AI evaluation nonprofit METR published a research update on November 20 covering public models up to Opus 5.5 and GPT-6 Astra. On its expanded TH1.1 plus TH2 task suite, frontier 50% time horizons double roughly every 3.6 months (95% CI 2.1 to 4.9), and naive extrapolation reaches the suite's longest 200-hour tasks in July 2027. METR says current public evidence is insufficient to rule out drastic AI R&D acceleration in the next couple of years, and that capabilities are getting harder to measure.
Key Takeaways
- ✓On the combined TH1.1 and TH2 suite, frontier 50% time horizons double roughly every 3.6 months (95% CI 2.1 to 4.9), close to the trend OpenAI reported on internal research tasks
- ✓Naive extrapolation has 50% time horizons reaching the suite's longest tasks (200 hours) in July 2027
- ✓On coding and optimization tasks, models usually match or beat the skilled humans METR recruited; on contest-style tasks with many entrants they place very high but not at the top
- ✓The new tasks are so hard that only 9% of human time estimates come from successful human baselines; the rest use records, expert interviews and extrapolation
- ✓METR introduces the METR Capability Index (MCI), which, like Epoch's ECI, estimates ability directly from scores; trend fits range from slight deceleration to moderate acceleration depending on methodology
Key Decision Metrics at a Glance
Turn your technical choice into a development budget
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Background
METR used to argue catastrophic risk was low because models' autonomous capabilities had broad limitations, as measured by time horizons. In an @METR_Evals thread on X it says that argument is now harder to make, so it extended its task suite.
How it works
METR added more cheaply verifiable tasks plus tasks that are hard to verify or require judgment, and tested public models up to Opus 5.5 and GPT-6 Astra. It also introduces the METR Capability Index (MCI), which, like Epoch's ECI, estimates task difficulty and agent ability directly from scores without human time estimates.
Numbers that matter
Per the METR post, frontier 50% time horizons double roughly every 3.6 months (95% CI 2.1 to 4.9), and naive extrapolation reaches the suite's 200-hour tasks in July 2027. Only 9% of human time estimates come from successful baselines. Agents improve even on hard-to-verify tasks but still show worse judgment than human experts there.
What it means for developers
Tasks with clear, fast feedback such as coding and optimization are increasingly safe to hand to agents, while work without fast feedback still needs human review. METR also notes continued reward hacking and growing eval awareness in frontier models.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.