AI data company micro1 released Dual-Arm Sim, which has 10 general-purpose frontier models control a two-arm Franka Panda robot in a simulated operating room built on NVIDIA Isaac Sim, using only camera images and robot sensors, across four tasks: swapping instruments, replacing a faulty drug cartridge, passing a sample between hands, and packing a surgical kit. Only 11 of 800 trials were completed, all by Gemini 4 Argon (9) and GPT-6 Astra (2). Models claimed "task complete" in 213 trials and were wrong every time. All 200 traces of the cartridge-swap task are public on Hugging Face.
Key Takeaways
- ✓Only 11 of 800 trials finished: Gemini 4 Argon 9 (6/20 cartridge swaps plus 3 surgical kits) and GPT-6 Astra 2; no model completed the instrument swap or the two-arm sample transfer (per micro1)
- ✓All 213 "task complete" claims were false, and in 94 of them the robot never touched the object; Gemini 3.1 Pro made the claim in 49 of 80 trials
- ✓Thinking longer did not help: Argon used only 315 tokens per decision and still led; its 9 successes cost $119 in total, while Claude Fable 5.1 spent $839 for none
- ✓Recovery decided success: Argon and Astra regained a real grasp after a miss in about two-thirds of trials, Grok 4.7 in 3 of 80; first position estimates were a median 33 cm off
- ✓Lifted faulty cartridges ended on the floor 14 times and on the trolley 11 times, versus only 8 correct swaps
Key Decision Metrics at a Glance
Turn your technical choice into a development budget
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Background and the problem
LLMs are increasingly wired to robot arms, lab gear and medical devices, yet most evaluations still test text or web agents. micro1's Dual-Arm Sim checks whether models can be trusted with careful physical work: see, grasp, carry, place and confirm, every step done right.
Architecture and how it works
The environment is built on NVIDIA Isaac Sim and Isaac Lab, with two Franka Panda arms on mobile bases at an instrument trolley. Each step, a model receives overhead and wrist camera images, fingertip positions and gripper opening, and returns relative moves, gripper commands, target estimates, a status note and a done flag. The simulation pauses while the model thinks, with up to 120 decisions per trial. All 800 trials are scored step by step against simulator ground truth, and every statement the model makes is checked against reality.
Benchmarks and measured results
All figures are micro1's own: Gemini 4 Argon and GPT-6 Astra tie at a 49.5 progress score, with Argon first on 9 completions. The other 8 models, including GPT-6.1 Sol, Claude Opus 5.5, Claude Fable 5.1 and Grok 4.7, completed nothing. All 213 "task complete" claims were false; Claude Opus 5.5 and Sonnet 5.5 made the claim in only 2 and 3 trials. Grok 4.7 described carrying a cartridge in all 20 cartridge trials while its gripper read fully closed.
Practical guide for developers
Teams building embodied or device-control agents can download the Hugging Face cartridge-swap dataset (200 trials with per-step reasoning summaries and raw replies) to study failure modes. micro1's advice is practical: never trust a model's own "done"; verify the end state with an independent sensor or a person, and pilot only simple single-arm pick-and-place tasks under supervision. micro1 says robot-trained VLA models will be added next.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.