Computer Science > Computers and Society
[Submitted on 18 Sep 2026]
Title:Your Programming Students' Cognition with ChatGPT: Higher Performance, Lower Retention, and Reduced Ownership
View PDF HTML (experimental)Abstract:Generative AI can improve students' programming performance, but successful task completion may not reflect what they retain. We examined performance, retention, cognitive load, and ownership in a controlled between-subjects experiment with 59 undergraduate computer science students, 55 were retained for analysis. Participants completed three introductory C programming tasks with access to ChatGPT-4.5 or conventional web search without generative AI. We measured task performance, self-reported mental effort and difficulty, pupillary responses, heart rate variability, and ownership, and assessed cued recall immediately and 48 hours later. ChatGPT-assisted students achieved higher coding scores (89% vs. 69%) but lower recall scores immediately (41% vs. 53%) and after 48 hours (39% vs. 52%). There was no significant difference in the loss of recall information over 48 hours between the groups. Self-reported mental effort increased less across tasks in the ChatGPT condition (Holm-adjusted p = .047), and students attributed less of the submitted code to themselves (45% vs. 81%). Confirmatory physiological tests did not detect significant differences in trajectories between conditions; substantial data loss limits their interpretation. These findings reveal a gap between assisted task performance and subsequent recall and sense of ownership in this setting. They motivate the need for assessment practices and AI learning tools that require students to explain, retrieve, and contribute to the work they submit as active participants in their education.
Submission history
From: Jake Renzella Dr [view email][v1] Fri, 18 Sep 2026 01:20:34 UTC (2,851 KB)
References & Citations
Loading...
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Connected Papers (What is Connected Papers?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
alphaXiv (What is alphaXiv?)
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Hugging Face (What is Huggingface?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Facts Only
* 55 undergraduate computer science students participated in a controlled between-subjects experiment.
* Participants completed three introductory C programming tasks.
* One group used ChatGPT-4.5; the other group used conventional web search without generative AI.
* ChatGPT-assisted students achieved coding scores of 89%, while the web search group achieved 69%.
* Immediate recall scores were 41% for the ChatGPT group and 53% for the web search group.
* Recall scores after 48 hours were 39% for the ChatGPT group and 52% for the web search group.
* The ChatGPT group attributed 45% of submitted code to themselves, compared to 81% in the web search group.
* Self-reported mental effort increased less across tasks for ChatGPT users (p = .047).
* Physiological data included pupillary responses and heart rate variability.
* Physiological tests showed no significant differences in trajectories between conditions due to substantial data loss.
* The study was submitted on 18 September 2026 by Jake Renzella.
Executive Summary
Generative AI significantly boosts immediate task performance in introductory programming but appears to diminish long-term knowledge retention and student ownership. In a controlled study of undergraduate computer science students, those utilizing ChatGPT-4.5 outperformed peers using traditional web search in coding scores (89% vs 69%). However, this performance gain was decoupled from learning; AI-assisted students showed lower recall immediately after the tasks and 48 hours later.
The psychological impact includes a reduced sense of agency, with AI users claiming far less ownership of their work than those using traditional search methods. While self-reported mental effort was lower for those using AI, physiological markers—such as heart rate variability and pupillary response—provided no conclusive data due to significant data loss. These results suggest a "performance-retention gap," indicating that while AI can facilitate successful task completion, it may bypass the cognitive struggle necessary for deep learning and personal investment in the work.
Full Take
This study employs a standard between-subjects design, but a peer reviewer would immediately highlight the small sample size (N=55) and the "substantial data loss" in physiological metrics, which renders the biometric claims inconclusive. While the delta in coding scores and recall is statistically notable, the study focuses on "introductory" tasks. It remains unclear if this retention gap persists as students move toward complex architectural problems where AI cannot simply provide a snippet, but rather requires high-level integration.
The findings confirm a growing suspicion in pedagogy: the "fluency illusion." When AI removes the friction of problem-solving, students mistake the output's correctness for their own understanding. The data suggests that the "cognitive load" typically associated with struggling through a C programming task is precisely what encodes the information into long-term memory. By lowering the mental effort (p = .047), the AI effectively optimizes for the *product* (the code) while sabotaging the *process* (the learning).
If these results hold, the cost of AI integration is a degradation of professional identity; attributing only 45% of work to oneself suggests a shift from "creator" to "editor." The second-order consequence is a workforce capable of deploying code they cannot explain or maintain.
Bridge Questions:
1. Would requiring students to write a manual "proof of understanding" for AI-generated code close the retention gap?
2. Does this effect vary based on a student's prior expertise, or is the "ownership drop" universal?
Counterstrike Scan:
A coordinated campaign to limit AI in education would use this data to create a "cognitive decline" narrative, framing AI as a mental prosthetic that causes atrophy. However, the current presentation remains a neutral academic inquiry into pedagogy rather than a polemic.
Sentinel — Human
The text appears to be an accurate, fact-based summary of experimental findings, characterized by the presentation of specific quantitative results and measured cognitive outcomes.
