X Debate: Artificial Analysis Says Gemini 4 Carbon Ran 7 Hours 33 Minutes in Its AA-Robotics Lab and Broke a Gripper, Sparking Questions Over Whether the Retry Should Count

Background and Industry Pain Point

AA-Robotics v0.1 is a research preview Artificial Analysis launched in October. Coding agents control a 6-axis robot arm through an HTTP API to complete eight tabletop tasks such as pressing a button, pouring a drink, building a tower and measuring a box. A software benchmark can be rerun endlessly after a mistake; a physical one cannot. Every error can damage equipment, and someone has to stand by. Gemini 4 Carbon is the strongest reasoning model Google opened to everyone on November 10, so its showing here drew extra attention.

Architecture and Mechanism

According to AA's methodology, each model runs zero-shot inside a coding-agent harness, watches four camera feeds, writes its own control code, and a human judge scores each task with partial credit. Carbon is positioned to spend more tokens digging into hard problems, which may explain why it ran so long: repeated attempts over many hours also mean more chances to drive the arm into things it should not hit. AA only said it bashed the arm around so much that it broke a gripper and did not say on which task.

Benchmarks and Measured Results

AA has not published Carbon's AA-Robotics score. AA figures for comparison: current leader Claude Fable 5.2 scores 75, GPT-6 Astra 69 and GPT-6.1 Sol 65, and at launch Astra averaged about 30 minutes per task. Carbon's 7 hours 33 minutes works out to roughly an hour per task across eight tasks; that is our estimate, not an AA figure.

Developer Guide

The main views in the replies: some argue retry results should be listed separately, or the comparison is unfair to models that never broke anything; others point out that Carbon is not a dedicated robotics model, so a general model sustaining more than seven hours of real-hardware control is informative in itself; and some joked about who stood next to the emergency stop for seven and a half hours. If you are building embodied agents, the lesson is to add torque and speed limits, an emergency stop and a per-task time cap before connecting an agent to real hardware, and to count equipment wear as part of evaluation cost. You can watch each model's task videos in the run explorer on AA's AA-Robotics article and compare once AA publishes Carbon's result.