Image: culturalanalytics.org · rights & removal

Multimodal LLM Categorization for Humanities Research on Social Media: A Case Study with Pepe the Frog

Reporting by Journal of Cultural AnalyticsRead the original at culturalanalytics.org

Executive Summary

The research investigated the utility of Large Language Models (LLMs) in interpreting politically sensitive, multimodal meme material, specifically the Pepe the Frog meme, using a dataset of 3407 tweets related to the January 6, 2021 riot. The study compared four models—Claude Opus 4, Sonnet 4, Mistral Pixtral 12B, and LLaVA-NeXT—against three human raters who categorized posts based on hatred/abuse, sexual content, and political positionality, using a detailed codebook.
The results indicated that the Claude models (Opus 4 and Sonnet 4) demonstrated strong agreement with each other and closely mirrored human raters, particularly in political positioning, though they were cautious in applying "encouraged" hate tags compared to humans. The performance varied significantly across categories: LLMs showed less accuracy in identifying specific forms of hatred than humans, but some models tended to apply referential tags instead of encouragement tags in cases where humans did. While Contextual References proved the least subjective category, political positionality was found to be highly subjective, with lower inter-rater agreement among humans and between humans and LLMs compared to other categories. The analysis also revealed that safety guardrails cause models to default toward neutrality, potentially softening or obscuring emotionally charged interpretations of sensitive discourse.

Facts Only

* A dataset of 3407 tweets related to the Pepe the Frog meme was used for experimentation.
* Four LLMs were tested: Claude Opus 4, Sonnet 4, Mistral Pixtral 12B, and LLaVA-NeXT.
* Three human raters coded each post using a codebook covering hatred/abuse, sexual content, political positionality, and contextual references.
* Human rater agreement was benchmarked by Cohen’s kappa scores of 0.55 (minimal acceptable) and 0.65 (strong agreement).
* Claude models showed strong pairwise agreement (0.722) for political positionality among themselves, while human rater pairs ranged from 0.286 to 0.887.
* LLMs tended to apply "Referenced" tags instead of "Encouraged" hate/abuse tags compared to human annotators in certain categories.
* The highest agreement scores (e.g., 0.936 for hate/abuse references among Anthropic models) were found in Contextual References, which concerned specific references to political events or figures.
* LLMs demonstrated a tendency towards a "bias towards neutrality" due to safety guardrails when processing sensitive topics.
* Claude Opus 4 and Sonnet 4 most closely mirrored human dynamics regarding political positionality surges related to major U.S. political events.

Full Take

The experiment reveals that the difficulty in applying nuanced, context-dependent categorization—especially concerning subjective emotional expression and moral dimensions embedded in meme culture—highlights a crucial gap between algorithmic safety engineering and genuine human contextual understanding. The finding that LLMs tended to moderate strong expressions of hatred (using "Referenced" instead of "Encouraged") suggests that the imposition of abstract, universal safety frameworks inherently leads to analytical flattening of real-world discourse, where the potential for raw emotional expression is suppressed in favor of a neutral assessment. This process mirrors the broader concern that systems designed for public safety might inadvertently dilute accountability or critical engagement by smoothing over the "jagged edges" of political reality. The distinction between how LLMs handle visual and textual context reveals that multimodal understanding is not just about recognizing objects, but about navigating the complex, often contradictory, ideological terrain where memes operate. This implies that when humanities scholars use LLMs for large-scale analysis of public discourse, their primary task shifts from simple categorization to critically assessing *how* the AI's emergent consensus shapes the interpretation of social responsibility embedded within volatile online interactions. The system risks establishing a new, sanitized reality by favoring high-level statistical agreement over the messy, morally weighted process of human judgment.

From the original · Journal of Cultural Analytics

Introduction In the days following the assassination of right-wing digital content creator Charlie Kirk, images circulated online of the suspected shooter from some years prior, wearing a Halloween costume that made it look like he was riding a Pepe the Frog figure that had Donald Trump’s hair.

Read the full story at culturalanalytics.org

Sentinel — Human

Confidence

The provided text argues that Large Language Models (LLMs) function as a new medium, following McLuhan's adage, and therefore their operation shapes information in ways that can be transparent or opaque. It uses an experiment involving LLMs (Claude Opus 4 and Sonnet 4) to show a qualitative shift: the models were more hesitant to apply strong moral judgments (like 'encouraged' vs. 'referenced' hate material, or 'pornographic' vs. 'sexual content'), preferring broader, less morally charged labels. The authors attribute this shift to overlapping mechanisms, including a bias toward neutrality in safety guardrails, policies discouraging intensified judgment, and multimodal oversensitivity to visual cues. This softening of judgment has the effect of watering down the sense of personal responsibility regarding the creation and circulation of harmful material. This dynamic is explored through the lens of 4chan greentexts, which feature anonymous users immersed in intense emotional narratives. The text then contrasts this with figures like Arthur Jones, Matt Furie, and YouTuber Mills, who exemplify taking public responsibility for their relationship to online culture amidst anonymity. Finally, it calls for Digital Humanists to work alongside cultural producers to tell data-driven stories about human agency online, especially concerning the complex, often anonymous production of online content.

From the original · Journal of Cultural Analytics

Introduction In the days following the assassination of right-wing digital content creator Charlie Kirk, images circulated online of the suspected shooter from some years prior, wearing a Halloween costume that made it look like he was riding a Pepe the Frog figure that had Donald Trump’s hair.

Read the full story at culturalanalytics.org