Do small language models know what they don't know?

Computer Science > Computation and Language
[Submitted on 21 Jul 2026]
Title:Do small language models know what they don't know?
View PDF HTML (experimental)Abstract:We explore whether entropy-based confidence signals can be leveraged to improve the accuracy of Small Language Models (SLMs) with fewer than 3 billion parameters, running entirely on consumer hardware. We evaluate seven distinct approaches, including token-level entropy early stopping, semantic entropy estimation, and uncertainty-aware routing to larger expert models, across 7 model pairs and 5 standard NLU benchmarks. Our key finding is that token-level entropy is effectively blind in SLMs: in 91% of dataset-model combinations, mean token entropy is near zero regardless of answer correctness, rendering token-based confidence signals unusable at this scale. We demonstrate that semantic entropy, computed by generating multiple samples, clustering answers by meaning, and measuring distributional uncertainty, recovers a viable confidence signal. Using semantic entropy to selectively route uncertain queries to a larger expert model yields accuracy improvements of up to +50 percentage points. Notably, cross-family routing (e.g., SmolLM 360M to Phi-3.5-mini) averages +22.0% improvement compared to +6.8% for same-family routing, revealing that expert model quality matters more than architectural compatibility. Our results suggest that the value proposition for entropy-based methods in SLMs is not computational savings but intelligent compute allocation: spending more tokens where they matter most.
References & Citations
Loading...
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Connected Papers (What is Connected Papers?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
alphaXiv (What is alphaXiv?)
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Hugging Face (What is Huggingface?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Facts Only

* The study explored leveraging entropy-based confidence signals to improve Small Language Models (SLMs) under 3 billion parameters using consumer hardware.
* Seven approaches were evaluated, including token-level entropy early stopping and semantic entropy estimation, across seven model pairs and five NLU benchmarks.
* Token-level entropy was found to be near zero across 91% of dataset-model combinations, regardless of answer correctness.
* Semantic entropy, computed by generating multiple samples and clustering answers by meaning, recovered a viable confidence signal.
* Routing uncertain queries using semantic entropy to larger expert models yielded accuracy improvements up to +50 percentage points.
* Cross-family routing (e.g., SmolLM 360M to Phi-3.5-mini) resulted in an average improvement of +22.0% compared to +6.8% for same-family routing.
* The value proposition of entropy methods is suggested to be intelligent compute allocation rather than computational savings.

Executive Summary

Entropy-based confidence signals can be used to improve the accuracy of Small Language Models (SLMs) with fewer than 3 billion parameters running on consumer hardware by leveraging uncertainty. The research evaluated seven approaches, including token-level entropy early stopping and semantic entropy estimation, across seven model pairs and five Natural Language Understanding (NLU) benchmarks. A key finding was that token-level entropy is ineffective for SLMs, as mean token entropy is near zero regardless of answer correctness in 91% of tested combinations, making token signals unusable at this scale. Conversely, semantic entropy, calculated via multiple sample generation and clustering by meaning, along with measuring distributional uncertainty, proved to be a viable confidence signal. Using semantic entropy to route uncertain queries to larger expert models resulted in accuracy improvements of up to 50 percentage points. Furthermore, routing between different model families, such as SmolLM 360M to Phi-3.5-mini, showed an average improvement of 22.0% compared to a standard family routing improvement of 6.8%, indicating that the quality of the expert model is more important than architectural compatibility. The overall conclusion suggests that the value of entropy methods lies not in computational savings but in intelligently allocating compute resources based on where uncertainty is highest.

Full Take

The research introduces a critical distinction between surface-level statistical signals and meaningful semantic uncertainty in small models. The failure of token-level entropy highlights a known limitation: correlation (or lack thereof) between local token randomness and global answer quality is not a reliable indicator for resource management at this scale. This forces the system to look beyond immediate token probabilities to grasp the actual semantic uncertainty of the generated output, which requires more complex computation like sampling and clustering. The finding that routing between different model families is significantly more effective than intra-family routing suggests that architectural or training context differences are salient features for uncertainty management in multi-model systems, implying that optimizing inference involves understanding the latent knowledge boundaries across disparate models. This shifts the focus from simple efficiency gains to architecturally aware allocation of computational budget based on inferred epistemic gaps. The subsequent research must investigate whether this semantic routing benefit generalizes across different task types and model architectures beyond the specific benchmarks used.

Sentinel — Human

Confidence

The text presents a dense, technically specific summary of an experimental study, exhibiting high coherence and specificity consistent with peer-reviewed research presentation rather than general AI prose.

Signals Detected

Moderate sentence length variance; sophisticated vocabulary used appropriately for technical subject.

High logical flow tracing the experimental process and specific findings; exhibits clear, focused emphasis on the results.

Structured presentation of quantitative results referencing multiple model pairs and benchmarks, typical of academic reporting.

No immediate markers of LLM confabulation; the data structure strongly suggests a genuine research summary.

Human Indicators

The specific, detailed quantitative results (e.g., 91% blind rate, +50 percentage point improvement) and the nuanced interpretation of the finding ('intelligent compute allocation') suggest deep domain expertise and synthesis beyond simple generative phrasing.