Confident AI’s deepeval 4.2.x (notes for 4.2.2/4.2.4; PyPI at 4.2.6) wires TypeSafe Jev into LLM-as-a-judge: evaluation LLMs still handle language tasks, while closed verdicts go to System One for typed decisions with calibrated probabilities. Claimed decision-stage pricing is about $0.042 per million input tokens with no output charge, ~100ms typical latency, and batched independent questions. Faithfulness, GEval, DAGMetric, and Classifiers are integrated; enable via `deepeval set-mode experimental` plus `typesafe-sdk`—missing keys fail loudly instead of silent fallback.
Key Takeaways
- ✓Notes: python-v4.2.4; blog Introducing Jev for Evals; PyPI deepeval 4.2.6
- ✓Official: ~$0.042/M input tokens for decisions, no output charge; ~100ms typical; batch independent questions
- ✓Faithfulness/GEval/DAGMetric keep language work on the judge LLM; closed verdicts use Jev Noul/Choice/Score
- ✓New Classifiers for categorical labels (e.g. RefusalClassifier)
- ✓Opt-in: deepeval set-mode experimental + typesafe-sdk + TYPESAFE_API_KEY (loud failure if missing)
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Core Background & Industry Pain Points
LLM-as-a-judge treated closed verdicts as generation tasks—label drift, output-token cost, and JSON repair—making full CI evals expensive and noisy. Decisioning needs to be separated from explanation writing.
Architecture Highlights & Internals
deepeval 4.2.x integrates TypeSafe Jev: language work stays on the judge LLM; closed decisions use typed System One answers with calibrated probabilities. Wired into Faithfulness/QAG, GEval, DAGMetric, and new Classifiers. Score formulas unchanged. See the announcement and integration docs.
Authoritative Benchmarks & Measured Scores
Official decision-stage economics: ~$0.042/M input tokens, no output charge; typical latency ~100 ms with batching of independent questions. No large public accuracy bake-off vs pure LLM judges is cited—validate on your gold set before mandating experimental mode.
Developer Hands-on Guide
pip install -U deepeval typesafe-sdk, run deepeval set-mode experimental, set TYPESAFE_API_KEY. Missing deps fail loudly (no silent LLM fallback). Defaults unchanged for existing users. Start from Getting Started.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.