Maze: imagine every game position glowing with a number — "how good is it to be here?" — where the exit glows +500 and being caught glows −1000. Value iteration spreads that glow through the maze like water finding its level: each position learns "my best move's promise, minus the cost of moving", over and over, until nothing changes. Then the arrows simply point to the brightest neighbour.
Mountain car: the car is too weak to climb straight up — it must slingshot back and forth. SARSA learns from its own experience: after every step it nudges its beliefs toward "what actually happened + what the next situation promises". The Fourier basis is a set of smooth wave patterns; the car's "brain" is a weighted mix of them, and λ is a memory dial — how much a past decision shares the credit (or blame) for what happens now.
Reinforcement Learning Lab 1 (KTH/UPM, 2021) solves two
classic RL problems: the Minotaur Maze with tabular methods,
and the Mountain Car with linear function approximation. Both
algorithms are ported 1:1 from Python to rl1-core.js,
with a matching node:test
suite (same state space, same Bellman/SARSA updates).
The maze is a stochastic shortest-path MDP. The state is the joint configuration of both agents and the key flag
\[ \mathcal{S} = \{(t_i, t_j, m_i, m_j, k)\} \quad k \in \{0,1\} \]
where \((t_i,t_j)\) is Thomas's cell and \((m_i,m_j)\) the minotaur's. Thomas chooses one of five actions (wait, left, right, up, down); the minotaur then takes a uniformly random legal move. With no key required, the lab's transition model is
\[ P(s' \mid s, a) = \frac{1}{|A_{\text{mino}}(s)|} \]
with rewards \(+500\) for reaching the exit, \(-1000\) if the minotaur catches Thomas, and \(-1\) per step. The optimal value function satisfies the Bellman optimality equation
\[ V^*(s) = \max_{a \in \mathcal{A}} \sum_{s'} P(s' \mid s, a)\bigl[R(s,a,s') + \gamma V^*(s')\bigr] \]
Value iteration performs synchronous Bellman backups until convergence: \(V_{k+1}(s) \leftarrow \max_a \sum_{s'} P(s'\mid s,a) [R + \gamma V_k(s')]\). The optimal policy is greedy w.r.t. the converged values. The Win prob readout is the Monte-Carlo estimate of the victory probability of the optimal policy from the current state — the lab's fixed-path analysis reports exactly \(P_{\text{vict}} = 1.0\).
The car is underpowered: it must swing back and forth to build momentum. The dynamics are those of gym's MountainCar-v0:
\[ x_{t+1} = x_t + v_t, \qquad v_{t+1} = v_t + 0.0015\,a_t - 0.0025\cos(3x_t), \qquad a_t \in \{-1, 0, 1\} \]
with bounds \(x \in [-1.2, 0.6]\), \(v \in [-0.07, 0.07]\) and goal \(x \ge 0.5\). Because the state space is continuous, the action-value function is approximated linearly with a Fourier basis of order \(N\):
\[ Q(s,a) = \mathbf{w}^\top \boldsymbol\phi(s,a), \qquad \phi_i(x,v) = \cos\bigl(\pi (i_x x + i_v v)\bigr), \quad i_x + i_v \le N \]
SARSA(λ) learns on-policy with the TD error \(\delta = r + \gamma Q(s',a') - Q(s,a)\) and eligibility traces \(\mathbf{e} \leftarrow \gamma\lambda\,\mathbf{e} + \nabla_\mathbf{w}Q\), updating \(\mathbf{w} \leftarrow \mathbf{w} + \alpha\,\delta\,\mathbf{e}\). The page defaults (\(\alpha = 0.005\), \(\lambda = 0.9\), \(\varepsilon = 0.1\), order 5) reach the flag in ~35 steps.
Move Thomas with the arrow keys (wait with Space). The minotaur takes a uniform random legal move after yours. Reach the green exit to win; if the minotaur lands on you, you lose. The orange arrows trace the optimal route (with step numbers), the red heat forecasts the minotaur's likely wanderings, and 🤖 Auto plays the optimal policy for you.
Click Run VI — the optimal V-values converge sweep by sweep, visualized as a heatmap on the slice where the minotaur currently stands. Arrows show the greedy optimal action per cell.
Learn runs the lab's SARSA(λ) with Fourier features live in your browser (α, λ, ε, basis order adjustable); the reward curve climbs as the policy improves. Drive lets you throttle manually (← / →, Space boosts).
github.com/alejp1998/rl_lab1 — Python lab, JS port, tests, notebook.