Paid Course · Evaluations & Reinforcement Learning

RLHF & Aligned Agents, From Scratch

From the state-vs-action gap to PPO, GRPO, and DPO, then a runnable car-navigation capstone where you train real policies and measure what is actually true. Built on the agi-eval package, so every claim is one you can check.

Modules

Lessons

Notebooks

Capstone

Pricing

Foundations

$29one-time · Modules 1-2

A low-risk way to start: the state-vs-action gap and the scoring-policy design that the rest of the course builds on.

Start reading ›

Complete Course

Best value

$199one-time · Modules 1-4 + capstone

Everything: foundations, the full RL formalism, the algorithms that train agents (PPO, GRPO, DPO), and the runnable car-navigation capstone.

Start reading ›

Every lesson is free to read right now. Sign-in, progress tracking, and certificates light up as the course goes live; Lesson 1A always stays a free preview.

Curriculum

Why Aligned Action Is HardModule 1 · 4 lessons

Tier A / Foundations

Live

The state-space vs action-space gap, what it costs, and the math that turns implicit decisions into measurable ones.

1AThe Hidden Problem (State vs Action Space)›1BWhat Misalignment Actually Costs›1CThe Mathematical Solution Framework›1DFrom Theory to Practice›

Designing the Scoring PolicyModule 2 · 5 lessons

Tier A / Foundations

Live

Derive value-aligned scoring from user values, then the two-stage score-then-generate architecture. With a runnable agent.

2AWhat "Aligned" Actually Means›2BDesigning for Alignment (the Three Questions)›2CFrom Design to Code (Three Hats)›2DImplementation Strategy›2EA Worked Example (Stock-Analysis Assistant)›

The RL Formalism, Built UpModule 3 · 6 lessons

Tier B / Core RL

Live

MDPs and observability, action spaces, stochastic rewards and risk (CVaR), trajectory alignment, preference learning, and curricula.

3AState, Observation, and Sufficiency›3BAction Spaces and Value Trade-offs›3CStochastic Rewards and Risk›3DTrajectories and Sequential Alignment›3EPolicies, Risk, and Learning Values from Preferences›3FCurriculum Learning›

The Algorithms That Train AgentsModule 4 · 6 lessons

Tier C / Advanced

Building

MC/TD foundations, value-based deep RL, policy gradients into PPO and GRPO, DPO and the direct-alignment family, reflective agents.

4AFrom MDP Theory to Real Agents›4BMonte Carlo, Markov Chains, and TD Learning›4CDeep Q-Networks›4DPolicy Gradients and PPO›4EDPO and the Direct-Alignment Family›4FReflective Agents (and the Course, Pulled Together)›

Capstone · LLM-Guided Curriculum for Car NavigationReal PPO · 5 seeds · runnable

Tier C / Capstone

Measure What Is Actually True

Train a status-quo PPO baseline and an LLM-guided agent on the same sparse-reward task, across five seeds, and report the real numbers.

The honest finding: on easy tasks the LLM guidance did not converge faster. The payoff was reliability on hard sparse tasks: cold-start solved 3/5 seeds, guided solved 5/5. No synthetic plots, the figure is the run.

Read the capstone ›

Multi-seed PPO comparison: status quo vs LLM-guided

FIG. C-1 · generated by the shipped harness