From Prompts to Constructs: A Dual-Validity Framework for Large Language Model Research in Psychology

Full text loading...
Abstract
Large language models (LLMs) are entering psychological research both as tools and as objects of inquiry. Yet many studies apply human instruments to LLMs without establishing that the outputs are reliable or interpretable, raising the risk of measurement phantoms—statistical regularities mistaken for genuine psychological phenomena. This review argues that robust AI psychological research requires integrating two methodological traditions: psychometric validation of what a score means and causal inference standards for what the results warrant. It develops a dual-validity framework in which evidentiary demands scale with scientific ambition: from tool use through behavioral characterization and human simulation to cognitive modeling. Classifying text may require only accuracy and reliability; claiming that an LLM simulates anxiety or illuminates cognitive mechanisms requires additional evidence, including construct validity evidence and experimental controls. Progress depends on developing computational analogs of psychological constructs rather than assuming human measures automatically apply to language models.

Facts Only

* Large language models are entering psychological research as tools and objects of inquiry.
* Many studies apply human instruments to LLMs without establishing output reliability or interpretability.
* Robust AI psychological research requires integrating psychometric validation (what a score means) and causal inference standards (what results warrant).
* A dual-validity framework is developed where evidentiary demands scale with scientific ambition across tool use, behavioral characterization, human simulation, and cognitive modeling.
* Text classification may require only accuracy and reliability.
* Claiming an LLM simulates anxiety or illuminates cognitive mechanisms requires additional evidence such as construct validity and experimental controls.
* Progress depends on developing computational analogs of psychological constructs.

Executive Summary

Robust AI psychological research requires integrating two methodological traditions: psychometric validation to determine what a score signifies and causal inference standards to assess the warrant of the results. This necessitates a dual-validity framework where evidentiary demands scale with scientific ambition, ranging from tool use and behavioral characterization to cognitive modeling. Classifying text may only require accuracy and reliability, whereas claiming an LLM simulates complex phenomena like anxiety or illuminates cognitive mechanisms demands supplementary evidence, including construct validity and experimental controls. Progress in this field depends on developing computational analogs of psychological constructs rather than assuming that human measurement standards automatically apply to language models.

Full Take

The framework calls for a fundamental shift in how psychological concepts are applied to artificial systems, moving beyond mere application to rigorous validation. The core tension lies between descriptive accuracy (reliability/accuracy for simple classification) and explanatory depth (construct validity and causal inference for simulating complex states). The implication is that current research risks conflating statistical correlations observed in language patterns with genuine psychological mechanisms. The necessity of developing computational analogs suggests that applying established human measures directly is insufficient; instead, researchers must build new constructs grounded in the model’s internal representation, necessitating novel methods for establishing equivalence between computational measurements and human-defined concepts. This requires asking: what specific causal pathways or predictive relationships are being measured, and what constitutes sufficient evidence to attribute a simulated state to a genuine psychological construct?

Sentinel — Human

Confidence

This text presents a sophisticated argument about the methodological pitfalls of applying human psychology to LLMs, proposing a specific dual-validity framework for future research.

Signals Detected

Moderate sentence length variance; sophisticated vocabulary used in a measured, academic tone.

High coherence; the argument flows logically from problem identification (measurement phantoms) to proposed solution (dual-validity framework).

No obvious verbatim repetition or template matching found; abstract argumentation structure is complex.

Claims rely on established academic terminology but the novel framework ('dual-validity framework') appears specific and original, suggesting human synthesis of existing concepts rather than pure LLM generation.

Human Indicators

The nuanced framing distinguishing between 'accuracy/reliability' for classification versus 'construct validity' for simulation demonstrates a specific, layered argumentative strategy typical of human scholarly writing.

The conceptual leap required to integrate psychometric validation and causal inference standards is intellectually rigorous and less predictable than standard LLM summarization.