Hugging Face, BitRobot, the Singapore Embodied AI Centre, NUS and others launched RoboColosseum on November 4, a distributed real-robot evaluation framework covering four embodiments: SO-101, Franka, Bimanual-YAM and the Unitree G1 humanoid. The first study ran 5,699 blinded pairwise comparisons at 124 sites across 5 VLA policies and 3 frontier models. Claude Opus 5.5 and GPT-6 Astra beat their VLA counterparts on average task progress by 12.1 and 11.0 points, while Grok 4.7 trailed by 7.5. All rollouts are open on Hugging Face, and the project pledges 10,000 free evaluation hours to academic labs.
Key Takeaways
- ✓Two tracks: the Open Track runs blinded zero-shot pairwise comparisons, and the Fine-tune Track gives each VLA the same 100 demonstrations per task across 20 tasks.
- ✓The VLAs tested are GR00T N1.7, pi0.5, MolmoAct2, G0.5 and LingBot-VLA v2; the frontier models are GPT-6 Astra, Claude Opus 5.5 and Grok 4.7.
- ✓After fine-tuning, GR00T N1.7 leads on SO-101 and pi0.5 leads on Franka and Bimanual-YAM; on the G1 humanoid no policy completes any task, with average progress of 13.3% to 23.2%.
- ✓Rankings agree only weakly across robots (Kendall W = 0.24), and zero-shot rankings poorly predict rankings after fine-tuning.
Key Decision Metrics at a Glance
Turn your technical choice into a development budget
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Background and industry pain points
Robot models are multiplying, but real-world evaluation is hard to reproduce: every lab has different robots, scenes and lighting, so results don't compare, and simulation scores don't always carry over. RoboColosseum, built by NUS, the University of Washington, Stanford, Ai2, Oxford, Hugging Face, BitRobot, the Singapore Embodied AI Centre and others, launched its paper and platform on November 4 to turn real-robot evaluation into shared infrastructure that scales.
Architecture highlights and mechanisms
The platform connects 100+ robots across the US, UK, Singapore, Malaysia and China to unified policy serving and standardized protocols, run by full-time evaluators in 140+ in-the-wild scenes. The Open Track compares two models blinded at the same site under matched conditions and ranks them with a difficulty-aware DCR rating on an Elo-style scale; the Fine-tune Track fixes 100 demonstrations per task. The paper also shows active matchmaking reaches similar ranking agreement with far fewer physical trials than uniform sampling.
Benchmarks and measured comparisons
In the first 5,699 Open Track comparisons, GPT-6 Astra has the highest DCR on DROID/Franka, while Claude Opus 5.5 leads on SO-101 and Bimanual-YAM. Across 901 paired task-progress evaluations, frontier models win 394, VLAs win 265 and 242 tie, an average agent advantage of 5.3 points; Claude Opus 5.5 and GPT-6 Astra lead by 12.1 and 11.0 points, while Grok 4.7 trails by 7.5. These are the paper authors' own evaluations.
Developer adoption and getting started
Every rollout, failures included, plus task datasets and fine-tuned models are public under the RoboColosseum organization on Hugging Face for download, reproduction or training. Embodied-AI teams can use it to check how their policies hold up across robots, and academic labs can apply for part of the 10,000 free evaluation hours.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.