LLM Sequence Score Recovery Method
Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores
The method improves reasoning accuracy in existing models without retraining, making it valuable for developers working with constrained resources in ASEAN.
The paper introduces a diagnostic protocol for recovering correct answers from collapsed LLM sequence scores on reasoning tasks. Hidden-state probes successfully decode correct answers even when native sequence scoring collapses due to structural biases. The method recovers 9-34 accuracy points for Qwen3.5 models using just two parameters on 25 unlabeled examples. The approach transfers successfully to OLMo-2-1B and Llama-3.1-8B models.
Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores
When a large language model fails a reasoning task, it is often assumed to lack the underlying capability. However, this conflates a genuine absence of reasoning with a late-stage output bottleneck. We observe a consistent readout gap across diverse reasoning benchmarks: hidden-state probes successfully decode correct answers even when native sequence scoring completely collapses due to structural biases. To test whether instance-specific logic survives this collapse, we introduce a diagnostic protocol using a minimal, target-label-free additive correction. Fitting just two parameters on as few as 25 unlabeled examples recovers 9--34 accuracy points for Qwen3.5 models, transferring successfully to OLMo-2-1B and Llama-3.1-8B. Crucially, these recovered decisions persist on hard instances unresolved by simple lexical overlap and significantly exceed count-preserving permutation baselines. Our results show that many apparent zero-shot reasoning deficits are expression failures masking intact internal logic, urging a narrower interpretation of benchmark evaluations.
