Sources of Truth: A Multi-Platform, Multilingual Audit of Citations in AI Mental Health Information Queries

Computer Science > Computers and Society
[Submitted on 31 Aug 2026]
Title:Sources of Truth: A Multi-Platform, Multilingual Audit of Citations in AI Mental Health Information Queries
View PDF HTML (experimental)Abstract:Online health information seeking is shifting from keyword search, where users consider a ranked list of links, to conversational systems that compose a single answer and curate its citations. Source evaluation therefore passes from user to platform, yet what these systems surface is poorly characterized. We audited three free consumer products (ChatGPT, Perplexity, Google AI Overview) on twenty English mental health questions under two prompt conditions, with a subset of three also translated into six further languages of varying resource tiers. We recorded 15,942 citations across 1,140 responses and 1,713 unique domains, then classified every citation with a nine-category organizational typology applied by a deterministic classifier validated against human coding. Citations were heavily concentrated: the ten most-cited domains accounted for 43.6% of English citations, and government, commercial health, and academic sources were closely matched at roughly 22% each. Platforms differed little in typical citation volume but sharply in consistency and in the source types they favored. Explicitly requesting sources shifted composition only modestly. Non-English queries surfaced fewer citations and were routed to language-appropriate resources at significantly lower rates. We release the typology, classifier, and annotated corpus as reusable instruments for auditing generative health search.
Current browse context:
cs.CY
References & Citations
Loading...
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Connected Papers (What is Connected Papers?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
alphaXiv (What is alphaXiv?)
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Hugging Face (What is Huggingface?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Facts Only

Three free consumer products (ChatGPT, Perplexity, Google AI Overview) were audited on twenty English mental health questions under two prompt conditions. A subset of three queries was translated into six additional languages. The audit recorded 15,942 citations across 1,140 responses involving 1,713 unique domains. Citations were heavily concentrated among the ten most-cited domains, which accounted for 43.6% of English citations. Government, commercial health, and academic sources each made up roughly 22% of citations. Platforms showed minor differences in citation volume but significant variation in consistency and favored source types. Explicit requests for sources caused only modest changes in composition. Non-English queries generated fewer citations and were routed to language-appropriate resources at lower rates.

Executive Summary

Online health information seeking is evolving from keyword searches to conversational systems that generate answers and citations. An audit was conducted on three free consumer AI products—ChatGPT, Perplexity, and Google AI Overview—using twenty English mental health questions under two prompting conditions. The study included translations into six other languages with varying resource tiers. A total of 15,942 citations were recorded across 1,140 responses, involving 1,713 unique domains. Citations were heavily concentrated; the ten most-cited domains accounted for 43.6% of English citations, with government, commercial health, and academic sources each making up approximately 22%. Platforms showed minor differences in citation volume but exhibited sharp variations in citation consistency and favored source types. Explicitly requesting sources resulted in only modest compositional shifts. Non-English queries resulted in fewer citations and routing to language-appropriate resources at significantly lower rates. The resulting typology, classifier, and annotated corpus are released for auditing generative health searches.

Full Take

The process highlights a fundamental shift in the epistemology of health information seeking, moving from ranking links to synthesizing answers. The concentration of cited sources toward government, commercial, and academic domains suggests that generative systems effectively default to established institutional knowledge when tasked with providing health information, which establishes an implicit hierarchy of trust based on source type. The differential treatment of non-English queries underscores a systemic bias where linguistic diversity is penalized by the resource allocation of search interfaces, creating an information access gap for non-dominant languages. The development and release of a classification typology and classifier serve to move the evaluation process from subjective user assessment to measurable, replicable auditing, which introduces a layer of analytical control over the opaque mechanisms of AI synthesis in sensitive domains like mental health. This methodology asks whether optimizing for surface-level citation volume inadvertently entrenches existing institutional knowledge while obscuring the specific provenance that shapes user trust in these complex answers. What measures can be developed to ensure the evaluation frameworks themselves are equally accessible and context-aware across linguistic and cultural modalities?

Sentinel — Human

Confidence

The text presents dense, empirical findings in a structured manner typical of human-authored research summaries rather than general AI output.

Signals Detected

Sentence length variance is natural; the tone shifts from descriptive to technical without being overly mechanical.

The text maintains a consistent, objective, research-reporting voice appropriate for an academic submission despite technical content.

The flow is logical (setup -> method -> results -> implications) without heavy reliance on mechanical transition words.

The content reads like a description of an actual academic/experimental finding, with specific metrics cited. No immediate signs of LLM confabulation.

Human Indicators

Use of precise, domain-specific terminology (e.g., 'deterministic classifier,' 'organizational typology') suggests authorial expertise beyond general LLM synthesis.

The structure mirrors a formal research paper abstract/summary, indicating adherence to established academic writing conventions.