Economist Abel Brodeur was leading a group of colleagues examining a survey-research study when they noticed something that stunned them. Its data set, available in a public repository, contained names of people who had admitted to crimes and identified their accomplices, information never provided to police. The study authors had published the data set as a “replication package” to enable other scientists to check and build on their work—in keeping with an evolving movement to promote transparent, reproducible research. But the release threatened to violate the confidentiality participants had been promised.
It wasn’t an isolated case. Brodeur and David Valenta, both at the University of Ottawa, went on to examine 327 publicly available replication packages for human-subjects studies published in 11 leading journals in economics, political science, and psychology. Fourteen percent contained at least one instance of data directly identifying a subject, such as their name, address, phone number, or an internet address for a computer they used to take an online survey, Brodeur and Valenta report in a discussion paper published last week on the EconStor server. Another 7% included details such as age, occupation, and place of residence that when linked can point to one individual. Participants often disclosed the information when answering “free-response” questions on surveys, sometimes even providing their name and bank account number to ensure they received a payment promised for taking part in the study. Study authors inadvertently made it public.
The results are “definitely concerning, and it’s really important that we know this” to help inform remedies, says Karen Macours, an economist at the Paris School of Economics and editor-in-chief of the Journal of Development Economics, one of the journals identified as publishing problematic data sets. The paper “puts a light on a problem the magnitude of which was not widely known.”
The breaches are an unexpected consequence of a push to tackle a so-called reproducibility crisis. Because independent testers have failed to reproduce as many as half of studies in some fields, many social science journals require or encourage authors to publicly post the data used for their studies. But those replication packages often contain more personal detail than the papers they support. And many social scientists are still learning how to adequately protect privacy when assembling them, says Kirsten Morehouse, a postdoctoral researcher in computational psychology at Columbia University. “There’s a lack of education about how severe the consequences [of privacy violations] can be.” Brodeur, who directs his university’s Institute for Replication, and Valenta write, “Replication packages are essential for credible, reproducible research. But if open science is going to work well, it also has to protect the people whose information makes that science possible.”
Their study was confined to replication packages for papers published from 2020 to ’24 and created by the authors themselves, as opposed to those from third parties such as the World Bank that extensively scrub their data to protect privacy. Thousands more are estimated to be publicly available. But the 11 journals included in the study may indicate the scale of a wider problem: All had published at least one paper associated with a data set that revealed a person’s identity.
Brodeur and Valenta alerted all authors of problematic data sets, and 70% were fixed or removed within 2 weeks. But those steps may not have averted all harm from the privacy violations. Social scientists often examine sensitive topics besides crime, such as domestic violence, political repression, sexual behavior, or substance abuse. And the study found a troubling pattern: Data collected in low- or lower middle–income countries, which tend to contain especially vulnerable populations, were more likely to divulge identifying information. “We don’t know why,” Brodeur says. “It’s not something we expected. It’s very sad.”
Authors can take simple steps to reduce future unintended disclosures, he and Valenta suggest. These include disabling functions in online surveys that automatically harvest respondents’ internet addresses, and storing data about participants’ identities in files separate from other data collected about them—a best practice among social scientists that some evidently ignore.
Researchers, journal editors, and repositories could also run data sets through automated screening tools that flag for closer inspection any text that bears hallmarks of personally identifiable information. Brodeur and Valenta developed one such tool, the PII Checker, to conduct their study. They have made the code, which works in tandem with a large language model, publicly available for free, and some of the 11 journals have expressed interest in using it; others have deployed other software. To reduce the risk of indirect identification of study participants, which arises if multiple details together can point toward a single individual, researchers and editors may also need to employ other specialized software, Morehouse says. She helped develop one such tool, called MinBlur.
Journals may have to spend more to provide the required human oversight, Brodeur and Valenta say. Even before the new study appeared, Macours’s journal planned to hire a data editor to help make the data from its papers more widely available. Now, that editor will also be checking for sensitive data in replication packages. “We need to fix this,” Macours says.