This study presents a Reinforcement Learning (RL)-assisted Genetic Algorithm (GA) framework for the Multi-Objective Capacitated Vehicle Routing Problem with Time Windows (MOCVRPTW). In this problem, a set of homogeneous vehicles depart from a depot, visit all customers exactly once, and return back to the depot. The routes of the vehicles are constructed by considering three objectives: minimizing the total travel time, minimizing the number of vehicles, and maximizing the satisfaction obtained from the customers who are visited within their time windows. We propose using an NSGA-II-based approach that is assisted by Q-learning-based operator selection methods for this problem. Unlike traditional GAs that use fixed operators, the proposed approach enables learning-based selection of each operator (crossover and mutation) considering the current performance of solutions. We make tests with five different Q-learning-based operator selection strategies and compare their results to using fixed or randomly selected operators by nonparametric statistical methods. The results show that all Q-learning-based operator selection strategies outperform the fixed-operator approach, whereas the random selection strategy is outperformed by four. In addition, when the best operator for each state of solutions is found considering all solution approaches and used in NSGA-II throughout the algorithm, it consistently results in the best performance among all. Overall, the results demonstrate that the proposed RL-supported GA framework provides a competitive alternative in terms of Pareto-front solution quality for MOCVRPTW, and learning-based operator selection can be an effective mechanism for adaptively controlling the evolutionary search process.