BiG-SURE for LLM Uncertainty Estimation
BiG-SURE - Bipartite Graph for Semantic Uncertainty and Reliability Estimation of LLMs
BiG-SURE provides a practical uncertainty estimation method for safety-critical applications using black-box models, which is crucial for deployment in regulated industries.
BiG-SURE uses cross-temperature semantic agreement to estimate uncertainty in black-box LLMs and VLMs. The method constructs a bipartite graph using NLI-based entailment scores between low-temperature anchors and high-temperature probes. Experiments across text, multilingual, and multimodal QA tasks show BiG-SURE improves average abstention AUROC over prior black-box estimators. The approach remains simple, unsupervised, and applicable to inaccessible model parameters.
BiG-SURE - Bipartite Graph for Semantic Uncertainty and Reliability Estimation of LLMs
Reliable uncertainty estimation is a crucial requirement for deploying large language models (LLMs) and vision-language models (VLMs) in safety-critical settings, especially when the model parameters are not accessible (black-box). We propose BiG-SURE, an uncertainty estimator based on cross-temperature semantic agreement. The method samples low-temperature responses as stable semantic anchors and high-temperature responses as probes under meaning-preserving input transformations. It then constructs an anchor-probe Bipartite Graph (BiG) using NLI-based entailment scores and defines confidence through the normalized squared spectral energy of this matrix, with uncertainty given by its complement. This bipartite graph-based Semantic Uncertainty and Reliability Estimation (SURE) score measures whether high-temperature probes remain semantically aligned with the model's stable low-temperature belief or not. We evaluate BiG-SURE on text QA, multilingual QA, and multimodal QA tasks across multiple model families. In these experiments, BiG-SURE improves average abstention AUROC over prior black-box uncertainty estimators, while remaining simple, unsupervised, and applicable to black-box model settings.