On November 22 OpenAI published "Perplexity and Intelligence" by Chief Scientist Jakub Pachocki, disclosing for the first time the log-perplexity of its models on an internal held-out codebase dataset since GPT-3.5: about 0.99 for GPT-3.5, about 0.91 for GPT-4, transformative cybersecurity and math capabilities below 0.8, and breakthrough math results from a model at about 0.76. He proposes perplexity as a shared cross-lab metric and common limits not to be crossed until AI monitoring reaches a concrete milestone.
Key Takeaways
- ✓Source: an official openai.com post by Chief Scientist Jakub Pachocki, who introduced it on X; Sam Altman reposted it calling it the most important thing OpenAI has published in a while
- ✓Disclosed data: log-perplexity on an internal held-out codebase set is about 0.99 for GPT-3.5 and about 0.91 for GPT-4; recent models are below 0.8, and a model at about 0.76 began resolving open math problems
- ✓Forecast: more transformative capabilities are expected in the 0.68-0.7 window, and an avalanche of pretraining algorithm advances could come as soon as 2027
- ✓Proposal: frontier labs jointly set development limits based on perplexity and other measurements, not crossed until understanding and monitoring of AI behavior reaches a concrete milestone such as a successor to CoT monitors
- ✓Caveats: labs use different eval sets and data mixtures, and gains from chain-of-thought reasoning do not show up in perplexity

Key Decision Metrics at a Glance
Turn your technical choice into a development budget
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Background
Downstream benchmarks saturate quickly, so they track AI progress poorly over time, and labs lack a comparable shared metric. OpenAI Chief Scientist Jakub Pachocki posted on X on November 22 to introduce his article "Perplexity and Intelligence", saying OpenAI chose to share more about pretraining progress to move toward shared safety bars.
How it works
The post frames modern AI as training a compressor, or predictor, on a huge corpus and then eliciting its latent capabilities with prompting, finetuning and reinforcement learning. OpenAI has tracked this intelligence substrate with perplexity, or cross-entropy loss, on held-out data. Pachocki argues that AI taking part in pretraining algorithm research could cause a step change in pace that alignment and monitoring may not keep up with.
Numbers that matter
The disclosed log-perplexity on an internal held-out codebase set is about 0.99 for GPT-3.5 and about 0.91 for GPT-4. Below 0.8, transformative cybersecurity and math capabilities appeared, and a model at about 0.76 began producing breakthrough math results. The post expects new capabilities around 0.68-0.7 and notes a steep acceleration in 2026.
What it means
This is not a product launch and does not change the API. The key ask is an industry norm: frontier labs set common limits based on perplexity and other measurements and do not cross them until AI monitoring reaches a concrete milestone. Whether other labs will sign on has not been announced.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.