Catalog

5 evals — Robotics

The same catalog/evals.yaml the CLI reads. Live means it runs end-to-end today; building and roadmap entries show exactly what is coming and welcome contributions.

All Agent / Tool use Code Reasoning Embodied Robotics Safety / Securitylive building roadmap

Eval	Category	Paper	License	Status
ManiSkill 2 GPU-parallel manipulation tasks with dense and sparse rewards.	Robotics	Gu et al., 2023, arXiv:2302.04659	Apache-2.0	Roadmap
CALVIN Long-horizon language-conditioned manipulation from undirected play data.	Robotics	Mees et al., 2021, arXiv:2112.03227	MIT	Roadmap
VIMA-Bench Manipulation specified by interleaved text-and-image multimodal prompts.	Robotics	Jiang et al., 2022, arXiv:2210.03094	MIT	Roadmap
BEHAVIOR-1K 1,000 realistic household activities in high-fidelity simulation.	Robotics	Li et al., 2023	MIT	Roadmap
RoboBench Broad manipulation suite spanning skills, objects, and scene variation.	Robotics	2024	Apache-2.0	Roadmap