Taming Visual Neglect: A Variational Information Bottleneck Framework for Adaptive Attention in Multimodal In

Computer Science > Computation and Language
[Submitted on 14 Jun 2026]
Title:Taming Visual Neglect: A Variational Information Bottleneck Framework for Adaptive Attention in Multimodal In-Context Learning
View PDF HTML (experimental)Abstract:Large vision-language models exhibit strong in-context learning (ICL) capabilities, yet when and why visual context helps multimodal ICL remains poorly understood. Empirical studies show a puzzling dichotomy: models sometimes effectively leverage visual demonstrations, yet often neglect them entirely. We propose VIB-ICL, an information-theoretic framework that resolves this dichotomy through the Information Bottleneck principle. We introduce the Cross-Modal Information Gain (CMIG), which quantifies the additional mutual information that visual context provides about the target beyond textual context. We derive a generalization bound showing that multimodal ICL's excess risk over text-only ICL is governed by the CMIG, proving that multimodal ICL provably outperforms text-only ICL when visual information is non-redundant. We further prove that visual context neglect, often viewed as a failure mode, is the Information Bottleneck-optimal solution when visual information is redundant, yielding a closed-form Attention Reallocation Principle that prescribes how visual attention weights should be adaptively adjusted. We instantiate this principle in the VIB-ICL algorithm, which estimates CMIG via variational bounds and dynamically reallocates attention. Experiments on five benchmarks demonstrate consistent improvements of up to 4.7\% accuracy gains and 35\% reduction in required demonstrations, validating our theoretical predictions.
References & Citations
Loading...
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Connected Papers (What is Connected Papers?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
alphaXiv (What is alphaXiv?)
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Hugging Face (What is Huggingface?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

* Large vision-language models exhibit strong in-context learning capabilities.
* Empirical studies show a dichotomy regarding the utility of visual context in multimodal ICL.
* The Cross-Modal Information Gain (CMIG) quantifies additional mutual information visual context provides about the target beyond textual context.
* A generalization bound is derived showing that multimodal ICL's excess risk over text-only ICL is governed by CMIG.
* Multimodal ICL provably outperforms text-only ICL when visual information is non-redundant.
* Visual context neglect is the Information Bottleneck-optimal solution when visual information is redundant.
* An Attention Reallocation Principle is derived prescribing adaptive adjustment of visual attention weights.
* The VIB-ICL algorithm estimates CMIG via variational bounds and dynamically reallocates attention.
* Experiments on five benchmarks demonstrated accuracy gains of up to 4.7\% and a reduction in required demonstrations by 35\%.

Large vision-language models demonstrate strong in-context learning capabilities, but the role of visual context in multimodal in-context learning remains unclear. Empirical studies reveal a dichotomy where models sometimes utilize visual demonstrations effectively and other times neglect them entirely. The proposed framework, VIB-ICL, utilizes the Information Bottleneck principle to address this by introducing Cross-Modal Information Gain (CMIG), which measures the mutual information visual context provides beyond textual context. The work derives a generalization bound indicating that multimodal ICL outperforms text-only ICL when visual information is non-redundant. Furthermore, the framework suggests that neglecting visual context is the optimal solution when visual information is redundant, leading to an Attention Reallocation Principle for adaptive attention weighting. Experiments on five benchmarks showed improvements of up to 4.7\% accuracy gains and a 35\% reduction in required demonstrations.

The research navigates the tension between apparent success and theoretical understanding in multimodal AI alignment. The central innovation lies in quantifying the information transfer—the CMIG—between modalities, moving beyond simple correlation to establish a principled measure for when visual context is truly informative versus redundant. This reframes the common failure mode of model neglect as an optimal decision under redundancy constraints, suggesting that what appears as a failure might be an inherently efficient state dictated by information economy principles. The resulting Attention Reallocation Principle suggests a mechanism for *how* to manage this balance adaptively, rather than simply observing the outcome. The implication is that future multimodal learning should incorporate explicit information flow metrics to resolve ambiguity regarding modality utilization, shifting the focus from post-hoc observation of neglect to proactive allocation guided by mutual information principles. What assumptions about redundancy are built into the framework's application across diverse visual and textual tasks? How can this principle be integrated into larger pre-training objectives without imposing excessive computational overhead?

Taming Visual Neglect: A Variational Information Bottleneck Framework for Adaptive Attention in Multimodal In

Facts Only

Executive Summary

Full Take

Taming Visual Neglect: A Variational Information Bottleneck Framework for Adaptive Attention in Multimodal In

Facts Only

Executive Summary

Full Take