Catalog

5 evals — Agent / Tool use

The same catalog/evals.yaml the CLI reads. Live means it runs end-to-end today; building and roadmap entries show exactly what is coming and welcome contributions.

All Agent / Tool use Code Reasoning Embodied Robotics Safety / Securitylive building roadmap

Eval	Category	Paper	License	Status
τ-bench Customer-service episodes (retail+airline) judged by final DB state and communicated outputs.	Agent / Tool use	Yao et al., 2024, arXiv:2406.12045	MIT	Live
SWE-bench Verified 500 human-validated GitHub issues resolved by producing a working patch.	Agent / Tool use	Jimenez et al., 2023, arXiv:2310.06770 (Verified subset, OpenAI 2024)	MIT	Live
SWE-bench Lite 300-issue subset of SWE-bench for cheaper, faster iteration.	Agent / Tool use	Jimenez et al., 2023, arXiv:2310.06770 (Lite subset)	MIT	Live
GAIA Real-world assistant questions needing tools, web, and multi-step reasoning.	Agent / Tool use	Mialon et al., 2023, arXiv:2311.12983	CC-BY-4.0	Live
Berkeley Function-Calling Leaderboard Function-calling graded by AST match (simple category live; multi-call coming).	Agent / Tool use	Yan et al., 2024	Apache-2.0	Live