← Docs/ Code

SWE-Lancer

Live

Real freelance software tasks priced by their actual payout.

How it works

01
SWE-Lancer pays real freelance dollars: 1,488 Upwork/Expensify tasks worth $1M. It has two halves. We port the SWE-Manager half — 265 Diamond tasks worth ~$264k — which is fully text-gradable and faithful: the model reads the issue and several engineer-written proposals, then selects the one the original hired engineering manager actually approved. The gold answer is that manager's real decision.
02
The model is prompted as the engineering manager and replies with a proposal number; the runner parses the choice (explicit 'Proposal N' preferred, then any valid proposal id) and compares to the manager's real pick.
03
The IC SWE half (write code, graded by end-to-end Playwright tests in Docker) is catalogued separately and stays 'building' until the upstream container harness is wired — the SWE-bench pattern. This page is the manager half.

Scoring

01
score = selection accuracy (fraction of tasks where the model picks the manager's proposal). Binary per task.
02
detail.price carries each task's dollar value and detail.earned is the price when correct, 0 otherwise — so the benchmark's headline '$ earned' metric is the sum of detail.earned across the run.
03
Read accuracy and total-earned together: a model can be right on cheap tasks and wrong on expensive ones, which the dollar metric surfaces and raw accuracy hides.

Using it

01
agi-eval run swe-lancer --model echo --limit 3 # offline sample (3 manager tasks)
02
agi-eval download swe-lancer # full 265-task Diamond manager set
03
agi-eval run swe-lancer --model openai:gpt-4o --push

Troubleshooting

Every non-pass carries exactly one typed failure tag, so the diagnosis is mechanical: look up the tag in report.failure_counts, then the per-case detail.

Tag	What it means	What to do
NO_ANSWER	No valid proposal number was parsed from the reply.	Instruct the model to answer with just 'Proposal N'; verbose models that discuss every option without concluding trip this — a stricter prompt helps.
WRONG_ANSWER	The model picked a different proposal than the manager did.	detail.chosen vs detail.correct_id shows the miss. This is a judgment task — the 'best' proposal often isn't the most detailed one, which is exactly what the benchmark tests.
ADAPTER_ERROR / HARNESS_ERROR	Our side, not the model's: transport/auth failures or a harness bug. Excluded from the score automatically.	Check endpoint credentials and connectivity; if HARNESS_ERROR persists, file an issue with the per-case detail.error string.

Run SWE-Lancer →Leaderboard