Robbyant, the embodied AI company under Ant Group, open-sourced LingBot-Touch on November 9 under Apache 2.0. It uses JEPA-style latent prediction and a shared Haptic Transformer to turn piezoresistive, magnetic, and visuotactile sensor signals into one set of tactile tokens. Per the official technical report, adding the frozen 5.46M-parameter Tiny encoder raised average success of Diffusion Policy, pi0.5, LingBot-VLA, and T-Rex by 13.9 to 24.8 points across 26 real dual-arm dexterous-hand tasks, while feeding raw tactile signals gave at most about half that gain. Tiny and Small weights are on Hugging Face.
Key Takeaways
- ✓Pretraining corpus: 4,983 hours, 464K episodes, about 1.33B frames from tactile gloves and teleoperated robot hands, covering piezoresistive, magnetic, and visuotactile sensing
- ✓Vendor figures on 26 real-robot tasks: LingBot-VLA 48.3% to 73.1%, pi0.5 48.1% to 69.9%, T-Rex 45.4% to 68.6%, Diffusion Policy 21.6% to 35.6%
- ✓The Tiny encoder has only 5.46M parameters, adding under 0.2% to the policies it augments; Small has 21.76M, both are 12-layer Transformers
- ✓In 12 simulated dexterous RL tasks, success reaches 69% vs 46% with task-specific CNN features and 21% without touch
- ✓Caveat: it transfers to sensor models unseen in pretraining, but a new taxel layout still needs geometry calibration first
Key Decision Metrics at a Glance
Turn your technical choice into a development budget
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Background and Pain Points
Robot manipulation has long relied on vision alone. For contact-heavy actions like twisting caps, closing zippers, or ejecting an SD card, fingers block the camera and force is guesswork. Tactile sensors are also fragmented: piezoresistive, magnetic, and visuotactile sensors differ in layout, resolution, and channels, so teams usually trained one model per sensor and nothing transferred. As dexterous hands and VLA policies ship faster, a general tactile representation layer is missing.
Architecture Highlights
LingBot-Touch first maps every sensor's taxels into a canonical fingertip space, then feeds modality-specific shallow tokenizers into a shared Haptic Transformer with 5D RoPE. It is trained JEPA-style to predict representations of masked taxels in latent space instead of reconstructing raw signals. The team explains that most tactile frames are non-contact background, so reconstruction makes a model learn the background. Output tokens keep their time, finger, and spatial addresses and can be combined with vision and robot state.
Benchmarks and Measured Results
All figures are vendor-reported (technical report Table 2, 20 trials per task). The 26 tasks include 20 on Sharpa Wave and 6 on XHand1. Adding LingBot-Touch improved the four policies by 13.9, 21.8, 24.8, and 23.2 points on average, and it was best in 89 of 104 task-policy pairs. Simulated RL across 12 tasks reached 69% success. No head-to-head comparison with other open tactile models on the same real-robot tasks was published; FTP-1 is compared only on probing tasks.
Developer Quick Start
The GitHub repo Robbyant/lingbot-touch ships inference code plus Sharpa and XHand sample data. Download config.json and encoder.pt for lingbot-touch-tiny or small from Hugging Face, then run infer.py under Python 3.12 and PyTorch 2.11 to get token features; CPU works too. The team recommends cross-attention queries to read out features rather than mean pooling. It suits teams that already have dexterous hands and VLA policies and want to add touch cheaply.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.