RepoBench

How it works

• 01

  RepoBench evaluates code completion in repository context: given retrieved cross-file snippets, the file's imports, and the code written so far, the model predicts the next line. We run the completion task (RepoBench-C) on the cross_file_first split — where the next line is the first to use a cross-file dependency, the setting that actually tests repo-level reasoning rather than local pattern completion.

• 02

  The prompt assembles imports + in-file context; the model returns one line. The runner takes the first non-empty line (stripping any code fence) as the prediction.

• 03

  Bundled sample ships three real tasks so it runs offline; agi-eval download repobench fetches the full Python cross_file_first split.

Scoring

• 01

  score = exact match on the next line after whitespace normalization — the benchmark's headline metric, and an unambiguous scalar.

• 02

  detail.edit_similarity reports a character-level ratio in [0,1] (difflib SequenceMatcher, equivalent to the fuzzywuzzy ratio in upstream scripts) so near-misses are visible — a prediction one character off still fails exact match but shows ~0.95 similarity.

• 03

  Read score (exact-match rate) and the mean edit similarity together; the gap between them tells you whether the model is close-but-wrong or genuinely lost.

Using it

• 01

  agi-eval run repobench --model echo --limit 3 # offline sample

• 02

  agi-eval download repobench # full Python cross_file_first split

• 03

  agi-eval run repobench --model openai:gpt-4o --limit 200

Troubleshooting

Every non-pass carries exactly one typed failure tag, so the diagnosis is mechanical: look up the tag in report.failure_counts, then the per-case detail.