Capstone: Exploration, Curricula, and Why You Have to Measure

Track: Reinforcement Learning · Level: Applied capstone Time: about 60 to 90 minutes (roughly 30 minutes reading, the rest running things) You'll need: MDPs and PPO (Modules 3 and 4), Python, a working stable-baselines3. A tiny local LLM (Ollama plus qwen2.5:0.5b) is optional. The examples run fine without it.

Most tutorials hand you a technique and a happy ending. This one does the opposite. We're going to make a confident claim, then try to break it with experiments, and watch it break.

That's not a gimmick. It's the actual job. By the end you'll have a real result about curricula, a feel for which problems they fix (and which they don't), and the one habit that separates RL work you can trust from RL work you can't.

What you'll walk away able to do

Explain why a sparse reward turns a trivial task into a brutal one, and spot the tell-tale sign of it.
Build a reverse curriculum, and connect it to the paper it comes from.
Read an RL result the right way: across seeds, distribution first. And say exactly why a single run, or even a mean, can lie to you.
Tell whether a stuck agent has an exploration problem, a credit-assignment problem, or a stability problem. They are not the same, and they need different fixes.
Draw a straight line from this toy car to frontier reasoning models, and stay honest about where LLM training tooling actually earns its keep.

1. The claim we're going to test

Here's a pitch you've seen a hundred times:

"Use AI agents to optimize your RL training. Small LLMs reason about the reward and the curriculum and make it converge faster."

Sounds great. We built exactly that. A car learns to drive to a goal, and you get two ways to train it:

Status quo (examples/example_1_status_quo.py): plain PPO, trained on the task cold.
LLM-guided (examples/example_2_llm_guided.py): small open-source LLM agents tweak the reward shaping and run a curriculum (start easy, ramp up).

So does the LLM-guided version actually converge faster? Don't take my word for it. Don't take the pitch's word for it. Run it.

2. Try it: the easy task

First, set the task up the naive way: a nearby goal with a dense reward, where you earn a little something every step you move closer.

python examples/example_1_status_quo.py --seeds 3      # the easy, dense setup

Plain PPO goes from 0% to roughly 100% success in about 40,000 steps. Every seed. It just works.

Plot twist number one. On this easy task, the LLM-guided version is no better, and the curriculum can make it worse. Think about why before you read on. A curriculum that trains on easy near-goals is spending its budget not practicing the real target, so when you measure on the real target it lags (we saw 65% against the cold baseline's 100%). Lesson: a curriculum is not a free speedup. On an easy task it's pure overhead.

So the pitch is already wrong as written. That doesn't make curricula useless. It means we were testing them on the wrong kind of task. Let's fix that.

3. Try it: the hard task

We change three things to manufacture a real exploration problem:

Sparse reward. You only score at the goal. No breadcrumbs.
A far, tiny goal. Distance 1.4, radius 0.05.
A short clock. 120 steps to find it.

Now the agent has to stumble onto the goal by pure luck before it has any signal to learn from. Run both methods across several seeds. And multiple seeds is not optional here. You're about to see why.

python examples/run_comparison.py --seeds 5

This trains the cold baseline and the LLM-guided method on the same hard target, across the same 5 seeds, and writes examples/figures/comparison.png. Here's what actually happens:

Cold PPO vs LLM-guided on the hard sparse target

The numbers behind the picture:

method	per-seed final success	seeds solved
Status quo: cold PPO	0%, 0%, 100%, 100%, 100%	3 of 5
LLM-guided: curriculum	100%, 100%, 100%, 100%, 100%	5 of 5

Now look again, and read the variance instead of the mean. Cold PPO averages 60%. But not one run scored 60%. It's bimodal. It either gets lucky (early exploration trips over the goal, it learns, 100%) or it never finds the goal, learns to sit perfectly still to dodge the out-of-bounds penalty, and flatlines at 0% forever. Two of five seeds were total failures. If you reported "60%," you'd be hiding the single most important fact about this method: a lot of the time, it just fails.

The curriculum version starts the goal close, where random driving reaches it easily, learns to drive, then pushes the goal outward as it succeeds. It solves the task on every seed. That's the real finding. Notice carefully what it is and isn't:

It is not "faster on easy tasks."
It is "reliable on a hard task where the baseline only wins by luck."

Big difference. Most people would have shipped the first version and called it a day.

4. Why this happens, and the papers that already said so

None of this is folklore. It's textbook RL, and naming the parts makes them yours to reuse.

Sparse-reward exploration is the hard problem. When reward only shows up at a goal you've never reached, there's no gradient pointing you there. You have to find it by accident first. This is the whole reason agents face-plant on games like Montezuma's Revenge, where vanilla methods score about zero (Bellemare et al., 2016).

Our fix has a name: reverse curriculum. "Start the agent near success, then expand outward" is a published method, Florensa et al., 2017, Reverse Curriculum Generation for Reinforcement Learning. The broader idea traces to Bengio et al., 2009, Curriculum Learning, and the serious RL treatment is Narvekar et al., 2020, the JMLR survey. When you can't hand-design difficulty, the field learns the curriculum automatically (Jiang et al., 2021, Prioritized Level Replay; Dennis et al., 2020, PAIRED).

That dense progress reward? It's potential-based shaping. Rewarding "distance closed toward the goal" is a potential function, and Ng, Harada, and Russell (1999) proved potential-based shaping doesn't change the optimal policy. It's the principled way to densify a sparse reward, not a hack. That's what the reward-shaping agent is doing when it works.

And the variance you just watched is the RL reproducibility crisis in miniature. This is the most important link in the lesson, so slow down here. Henderson et al., 2018, Deep Reinforcement Learning That Matters, showed deep-RL results swing wildly with the random seed and that single-run claims mislead constantly. Agarwal et al., 2021, the rliable paper, argued the field should report distributions (interquartile means, bootstrap intervals) instead of single numbers, precisely because of bimodality like our "0, 0, 100, 100, 100." Your five-seed plot is a tiny replay of their whole argument.

The habit to burn in: in RL, run multiple seeds and look at the spread before you look at the mean. If you take one thing from this entire capstone, take that.

5. When does a curriculum help, and when is it a waste of time?

A curriculum fixes an exploration bottleneck, the "I can't find the reward" problem. It does nothing for two other problems people constantly confuse it with:

Bottleneck	What it looks like	What actually helps
Exploration	Never finds reward. Bimodal across seeds.	Curriculum, shaping, intrinsic motivation
Credit assignment	Finds reward but can't tell which action earned it (long, sparse episodes)	Better advantage estimation, denser or process rewards, shorter effective horizon
Stability	Learns, then diverges. Value estimates blow up.	The deadly-triad fixes from Module 4c (target networks, smaller steps)

Reach for a curriculum when your real problem is stability or credit assignment, and you'll burn a week for nothing. The skill is the diagnosis. And the diagnosis is, once again, a measurement: watch when and how learning fails across seeds.

6. Does any of this scale to harder, more involved tasks?

Three honest answers, because three different things are generalizing.

The measurement lesson scales everywhere, and it scales up. Across all of deep RL, frontier scale included, single-seed results are untrustworthy and plenty of methods are bimodal. This gets more true as problems get harder, not less.
The curriculum mechanism scales, and its payoff grows with the exploration difficulty. Our task is mild: cold fails 2 of 5 times. On genuinely vicious exploration tasks (long-horizon robotic manipulation with sparse success, Montezuma), cold RL reliably scores a flat zero, and a curriculum is the whole difference between learning and not. The catch: the literal "move the goal closer" trick needs a difficulty knob you can dial. When you don't have one, you reach for automatic curricula (PLR, PAIRED above).
It does not transfer to non-exploration problems. See the table in section 5. Different bottleneck, different fix.

The payoff: this toy is a peephole into frontier reasoning models

Here's why this capstone is worth your time beyond the car. Reinforcement Learning from Verifiable Rewards (RLVR), the method behind modern reasoning models (Module 4d and 4e; DeepSeek-R1, Guo et al., 2025), is a sparse-reward problem. The model only scores when its final answer checks out. The exact dynamics you just watched come back at billion-parameter scale:

If a training problem is so hard the model never solves it, there's no signal. Same as a goal the car never reaches.
The practical fix is the idea you just ran: curate and curriculum the training data by difficulty. Keep the problems the model sometimes gets right, then expand outward, often with rejection sampling on top. That's reverse-curriculum thinking wearing a different hat.

So when you read that frontier RLVR pipelines pour enormous effort into filtering and ordering training problems by difficulty, you now get why from first principles. A half-million-parameter car taught it to you in five minutes.

7. One honesty guardrail (this is the real meta-skill)

We proved two different kinds of claim in this lesson, and they deserve very different levels of confidence.

The RL science ("a curriculum makes a hard sparse task reliably solvable") is well-established and cited. Trust it.
The product pitch ("a small LLM picks the curriculum and reward better than a plain heuristic") is not established. In our own runs, a dumb deterministic heuristic drove the win, and a 0.5B model reasons imperfectly. LLMs are a flexible controller for the training loop (Eureka, Ma et al., 2023; ELLM, Du et al., 2023), but whether one actually beats a good heuristic on your task is something you measure. At tiny scale, it often doesn't yet.

That gap is the whole game. Separate what you've verified from what you're hoping, and never let a slick-sounding mechanism stand in for a number.

Pause and try this

Predict, then check. Before you run anything, guess: at what goal radius does cold PPO flip from "solves it every time" to "coin flip"? Sweep goal_radius over {0.12, 0.08, 0.05, 0.03} (edit TARGET in examples/common.py) at 5 seeds and find the cliff. Does the curriculum survive past the point where cold PPO snaps?
Kill the curriculum, keep the shaping. Make Example 2 use only the reward-shaping agent, no distance curriculum, by densifying the reward (progress > 0) on the hard target. Does dense shaping alone rescue cold PPO? What does that tell you about which lever is doing the work?
Write it both ways. Take your 5-seed numbers and write the one-sentence result a careful researcher would publish. Then write the misleading single-number version a marketer would tweet. Keep both, taped to your monitor.
Hunt the bottleneck. Make the task hard a different way: a very long horizon with a dense reward (that's a credit-assignment problem, not an exploration one). Does the curriculum help now? Why or why not?

The five things to remember

A sparse reward turns an easy task into a hard exploration problem. The fingerprint is bimodality across seeds: it works, or it totally fails.
A reverse curriculum (start near, expand) makes those tasks reliably solvable. That's reliability, not raw speed, and it does nothing on easy tasks.
Run multiple seeds, read the distribution before the mean. A mean can bury outright failures. This is the universal one.
Match the fix to the bottleneck. Exploration wants a curriculum or shaping. Credit assignment and stability are different animals.
The toy reaches all the way to RLVR and reasoning models, where sparse verifiable rewards make difficulty curricula over training data essential. Keep the verified science separate from the unproven pitch, and measure the difference.