Deconstructing Overconfident Failures in Code LLMs: Instruction Tuning Inflates Confidence on Failing Code

Background and the Problem

Code generated by large language models often appears syntactically plausible while failing execution-based correctness checks. While sandboxed testing and static analysis remain standard, they are costly and post-hoc. Consequently, model-derived uncertainty (such as token logprobs and entropy) is widely eyed as an early signal for selective acceptance. However, code LLMs suffer from severe overconfidence: models frequently generate broken programs with output probabilities indistinguishable from correct solutions.

Architecture and How It Works

This paper presents the first comprehensive empirical study characterizing overconfident failures across four open-source code models (CodeLlama, DeepSeek-Coder, Qwen2.5-Coder, and OpenCoder) and three rigorous benchmarks (HumanEval+, MBPP+, BigCodeBench):

1. Program- and Token-Level Uncertainty Metrics: Evaluates global log-probability, minimum token confidence, and localized entropy across syntax boundaries;
2. Selective Generation Feasibility: Tests whether rejecting generations below uncertainty thresholds improves accepted set pass rates;
3. Impact of Instruction Tuning: Compares base versus chat/instruct checkpoints to evaluate calibration shifts;
4. Latent Representation Probing: Investigates whether internal transformer layers encode execution signals suppressed during output projection.

Benchmarks and Measured Results

The study establishes four definitive empirical findings:

• Uncertainty metrics provide inconsistent signals: Standard token-level confidence metrics serve as fragile, model-dependent indicators of execution failure;
• Overconfidence persists across scales: Selective generation based on output confidence fails to reliably boost accepted accuracy, as wrong programs match correct code in probability distributions;
• Instruction tuning inflates false certainty: Post-training raises the model's confidence on erroneous programs without improving its discriminatory power between correct and buggy logic;
• Hidden states retain latent correctness signals: Probing intermediate hidden representations shows they encode execution correctness features that the output softmax layer fails to expose.

Getting Started for Developers

These insights offer vital architectural guidance for developers of AI coding agents and automated development environments:

• Avoid Logprob-Only Filtering: Never rely exclusively on token logprobs or sequence confidence to trigger agent retries or code approvals;
• Enforce Sandbox Execution: Mandate isolated execution validation and AST syntax passes as non-negotiable acceptance criteria;
• Leverage Hidden Layer Probing: For security-critical code synthesis with open-weight models, train lightweight probing heads on intermediate hidden representations rather than reading surface probabilities.