AGI·EVALSSign in
← Docs/ Code

SWE-Lancer

Roadmap

Real freelance software tasks priced by their actual payout.

On the roadmap

SWE-Lancer is catalogued but not runnable yet, so there are no usage docs — we do not document what does not run. The fact sheet below is sourced from the paper; the protocols it will implement are stable today.

Paper
SWE-Lancer: Can Frontier LLMs Earn $1 Million from Freelance Engineering?
Citation
Miserendino et al., 2025, arXiv:2502.12115
License
Custom (OpenAI)
Homepage
https://github.com/openai/SWELancer-Benchmark
How an eval goes live
  1. Implement an EvalRunner against the stable protocols.
  2. Bundle a small real-schema sample so it runs offline.
  3. Point the catalog entry's runner at the class.
  4. Ship its docs in the same change — required to flip live.

pip install agi-evals