My Skill Lab
Machine LearningAdvanced

Reinforcement Learning Fundamentals

MDPs, Q-learning, policy gradients and RLHF basics.

Taught by Rohan Maharjan · ML Engineer

What you'll learn

  • Write the agent-environment loop in Gymnasium and compute discounted returns from reward sequences
  • Implement epsilon-greedy, UCB and Thompson sampling for bandits and explain the exploration trade-off
  • Formulate a problem as an MDP and solve it exactly with value iteration and policy iteration
  • Implement tabular Q-learning and SARSA and explain the difference between on-policy and off-policy learning
  • Build a deep Q-network with experience replay and a target network, and a REINFORCE agent with a baseline
  • Train and evaluate PPO agents with Stable-Baselines3 and describe how PPO and DPO are used in RLHF

About this course

Reinforcement learning is how machines learn to act: to play games, control robots, manage inventory and, increasingly, to make large language models helpful through RLHF. This course builds the subject from first principles with runnable code at every step. You will start with agents, environments and rewards in Gymnasium, learn exploration with multi-armed bandits, formalise decision-making as a Markov decision process and solve it with value iteration, then move to model-free learning with Q-learning and SARSA. From there you will implement a deep Q-network and REINFORCE in PyTorch, train PPO agents with Stable-Baselines3, and finish with a clear account of how reward models, PPO and DPO align language models. It is an advanced course for engineers and researchers who know Python, NumPy and basic PyTorch and want a solid, practical grounding in reinforcement learning rather than a survey.

Course content

8 lessons
  1. 1Agents, Environments and RewardsThe reinforcement learning loop in Gymnasium, episodes and returns, and why discounting matters.Preview
  2. 2Multi-Armed Bandits and ExplorationBalance trying new actions against exploiting known good ones with epsilon-greedy, UCB and Thompson sampling.
  3. 3Markov Decision Processes and Bellman EquationsFormalise sequential decisions as an MDP and solve small ones exactly with value iteration and policy iteration.
  4. 4Q-Learning and SARSALearn action values from experience alone with temporal-difference updates, and see how on-policy and off-policy differ.
  5. 5Deep Q-NetworksReplace the Q-table with a neural network, stabilise training with experience replay and a target network, and solve CartPole.
  6. 6Policy Gradients: REINFORCE and Actor-CriticOptimise the policy directly with the policy gradient theorem, reduce variance with a baseline, and add a learned critic.
  7. 7PPO in Practice With Stable-Baselines3Understand the clipped objective that makes PPO stable, then train, evaluate and tune agents with Stable-Baselines3.
  8. 8RLHF: Aligning Language Models With Human FeedbackHow reward models, PPO with a KL penalty and DPO turn a pretrained language model into a helpful assistant.