JevAgentHarness & AgentDecision-4B: Sharing Control Between 4B Decision Models and LLMs Makes Agents 2.5x Faster and 89% Cheaper

Background and the Problem

Modern LLM agents capable of patching codebases, operating desktop GUIs, and handling customer workflows rely on agent harnesses that invoke a massive frontier LLM at every step. Yet the vast majority of these calls culminate in a discrete multiple-choice decision—such as selecting which tool to invoke next. While specialized decision models (like Jev) can answer multiple-choice queries at a fraction of an LLM's latency and cost, benchmarking them on the newly introduced AgentDecisionBench and DecisionGym reveals a stark gap: the decision model most accurate on isolated questions solved only 69.5% of live agent tasks, and delegating tool choices without a calibrated confidence threshold reduced task success by 19.0 percentage points because existing decision models never observe the off-policy states an agent reaches after its own mistakes.

Architecture and How It Works

To enable seamless shared control between lightweight decision models and frontier LLMs, researchers from UIUC, Yale, and Princeton developed JevAgentHarness:

1. Per-Choice Confidence Threshold Calibration: Fits an optimal confidence threshold for each individual choice directly on the agent's execution trajectories, allowing the fast decision model to act when confident while deferring ambiguous states back to the large LLM;
2. Training on Post-Mistake Trajectories (AgentDecision-4B): Collects the exact recovery states an agent encounters after making errors and trains an open 4B parameter decision model (AgentDecision-4B) using Jev's labels, immunizing the controller against compounding rollout drift;
3. AgentDecisionBench & DecisionGym: Provides a unified evaluation suite testing decision models from isolated classification up to full closed-loop control of running agents.

Benchmarks and Measured Results

Comprehensive evaluations across both in-distribution and out-of-distribution (OOD) agent workloads demonstrate massive efficiency gains:

• 81.1%–89.0% Cheaper and Up to 2.5x Faster: On in-distribution workloads, 4B decision models trained with JevAgentHarness reduce agent execution costs by 81.1% to 89.0% and accelerate end-to-end completion by up to 2.5x, with less than 3 points of success difference;
• 91.5% Success on Out-of-Distribution Tasks: On unseen OOD tasks unlike its training corpus, the agent solved 91.5% of tasks with AgentDecision-4B, decisively outperforming baseline Jev (82.0%);
• Eliminates the 19.0-Point Delegation Penalty: Per-choice threshold gating completely recovers the 19.0-point degradation suffered by uncalibrated tool delegation.

Getting Started for Developers

All models, datasets, and harness code are available at https://github.com/JevAgentHarness/JevAgentHarness:

• Drop-In Harness Acceleration: Integrate AgentDecision-4B into existing agent loops to offload deterministic tool-selection turns while reserving heavy LLM calls for complex code synthesis;
• Trajectory Calibration: Run the included per-choice threshold fitter on a small sample of domain trajectories before production deployment to guarantee zero accuracy regression.