Catalog

3 evals

The same catalog/evals.yaml the CLI reads. Live means it runs end-to-end today; building and roadmap entries show exactly what is coming and welcome contributions.

All Agent / Tool use Code Reasoning Embodied Robotics Safety / Securitylive building roadmap

Eval	Category	Paper	License	Status
SWE-bench Verified 500 human-validated GitHub issues resolved by producing a working patch.	Agent / Tool use	Jimenez et al., 2023, arXiv:2310.06770 (Verified subset, OpenAI 2024)	MIT	Building
SWE-bench Lite 300-issue subset of SWE-bench for cheaper, faster iteration.	Agent / Tool use	Jimenez et al., 2023, arXiv:2310.06770 (Lite subset)	MIT	Building
Cybench 40 professional CTF tasks measuring offensive cyber capability and risk.	Safety / Security	Zhang et al., 2024, arXiv:2408.08926	MIT	Building