BigCodeBench

How it works

• 01

  BigCodeBench extends function synthesis to realistic, library-heavy programming: each task's hidden test suite is a unittest.TestCase exercising the completed function with diverse calls, often through popular third-party packages (numpy, pandas, flask, sklearn...).

• 02

  The model completes the function from its signature and docstring (the 'complete' split); the candidate is assembled with the test class and run in a fresh subprocess with a wall-clock timeout — the same execution model as HumanEval+, scaled up.

• 03

  Scope note on dependencies: BigCodeBench tasks import from 139 libraries. The bundled offline sample is restricted to stdlib-only tasks so it runs out of the box; the full set needs those libraries installed (the upstream bigcodebench package pins them). Missing imports surface in detail.stderr.

Scoring

• 01

  score = pass@1 (fraction of tasks whose full unittest suite passes). Supports pass@k via n_samples/k for sampled decoding (unbiased Chen et al. estimator).

• 02

  A task passes only when every test method succeeds — partial test passes count as a fail, matching upstream.

• 03

  Infra/import errors land as WRONG_ANSWER with the traceback in detail.stderr; read it before blaming the model — a missing library is an environment problem, not a capability one.

Using it

• 01

  agi-eval run bigcodebench --model echo --limit 3 # offline, stdlib-only sample

• 02

  agi-eval download bigcodebench # full set (problems + tests)

• 03

  # install the tasks' libraries for full runs, then:

• 04

  agi-eval run bigcodebench --model openai:gpt-4o --concurrency 4

Troubleshooting

Every non-pass carries exactly one typed failure tag, so the diagnosis is mechanical: look up the tag in report.failure_counts, then the per-case detail.