RL Lab 2 — Interactive WEB EDITION

💡 The big idea in plain words

The lander learns by replaying memories. It stores what it experienced (situation → action → outcome) and, between flights, practices on random memories — like a pilot studying past landings instead of only flying live. Two brains help: the online brain decides and learns, while a frozen copy (the target network) keeps the scoreboard stable enough to learn against. Exploration comes from ε-greedy: mostly follow the brain, occasionally press a random button — so the lander discovers moves it would never try otherwise.

🎓 The Lab

Reinforcement Learning Lab 2 (KTH/UPM, 2021) trains a Deep Q-Network (DQN) on OpenAI Gym's LunarLander-v2. The Python implementation (problem_2.py, a 64-64 MLP with replay length 16384) is ported 1:1 to rl2-core.js; the web edition uses a tuned 32-32 network with replay 8000 so learning is visible in real time — the algorithm is identical.

🚀 The task

The lander must touch down between the two flags with low velocity. The state is a 5-vector of position, velocity and angle; four discrete actions (none, left, main, right thrust). Rewards: \(+100\) for a safe landing, \(-100\) for a crash, \(-1\) per time step, plus a small shaping bonus for hovering near the target.

🧠 Deep Q-Learning

The optimal action-value function satisfies the Bellman equation

\[ Q^*(s,a) = \mathbb{E}\!\left[ r + \gamma \max_{a'} Q^*(s',a') \right] \]

A neural network \(Q(s,a;\theta)\) approximates it by minimizing the TD loss on sampled transitions:

\[ \mathcal{L}(\theta) = \mathbb{E}\!\left[\left( r + \gamma \max_{a'} Q(s',a';\theta^-) - Q(s,a;\theta) \right)^2\right] \]

Three ingredients make this stable:

• Experience replay — transitions \((s,a,r,s')\) are stored in a buffer and sampled uniformly, decorrelating consecutive updates.
• Target network \(\theta^-\) — a frozen copy of the network, refreshed every \(C\) steps, so the regression target does not chase the moving estimate.
• ε-greedy exploration — \(\varepsilon\) decays geometrically per episode (default 0.9 · 0.99^ep), trading exploration for exploitation. Adam optimizes the loss with clipped TD errors.

The reward chart plots the 25-episode running average; the first safe landing typically appears within ~20-40 episodes with the default parameters.