AGI·EVALSSign in
← Docs/ Code

BigCodeBench

Roadmap

Practical programming tasks chaining many real library function calls.

On the roadmap

BigCodeBench is catalogued but not runnable yet, so there are no usage docs — we do not document what does not run. The fact sheet below is sourced from the paper; the protocols it will implement are stable today.

Paper
BigCodeBench: Benchmarking Code Generation with Diverse Function Calls
Citation
Zhuo et al., 2024, arXiv:2406.15877
License
Apache-2.0
Homepage
https://github.com/bigcode-project/bigcodebench
How an eval goes live
  1. Implement an EvalRunner against the stable protocols.
  2. Bundle a small real-schema sample so it runs offline.
  3. Point the catalog entry's runner at the class.
  4. Ship its docs in the same change — required to flip live.

pip install agi-evals