This post is part 2 in a series about automated content moderation. Read the first post here.
When whistleblower Frances Haugen leaked a set of documents from Meta in 2020, among the revelations was a jarring statistic: The company’s algorithms designed to detect terrorist content incorrectly deleted nonviolent Arabic-language content 77 percent of the time, while failing to detect hate speech under the company’s own policies in many instances. Meta’s own transparency report released later that year demonstrated similar findings. Five years later, researchers in the region report that overzealous moderation remains a problem, while paths to remedy have all but collapsed.
Where these systems are faltering in Arabic, they’re positively failing in less-resourced languages. As a 2025 report from the Center for Democracy and Technology found, labeled datasets in certain languages and dialects such as Maghrebi Arabic and Kiswahili contain inconsistencies, bias, and inaccuracies due to the limited hiring of annotators who actually speak the languages as well as shifts in the languages themselves. An investigation into ChatGPT’s outputs in several low-resource languages demonstrates the depth of problem.
But language disparities are just one of several concerns as automated moderation becomes more widespread. From the systemic suppression of content from Palestine to the repeated misclassification of LGBTQ+ content as adult or explicit material, these varied examples demonstrate the risks of overreliance on automated moderation—and the need for stronger safeguards.
Transparency, Cultural Competence, Appeals
As we discussed in Part 1 of this series, automated systems can process content at a scale that humans never could, potentially enabling better moderation at scale and alleviating the psychological load on ill-paid moderators whose jobs require them to view incredibly disturbing content. But automated systems also reproduce existing biases, struggle to understand context, and often make mistakes that disproportionately affect journalists, activists, artists, and other vulnerable and marginalized communities.
As Rachel Griffin wrote in 2023, “Perfectly accurate moderation is not only technically out of reach but intrinsically impossible.” Despite those intrinsic flaws, there is a great deal companies, policymakers, and civil society can do to help ensure that highly-automated systems operate in ways that respect human rights, minimize predictable harms, and provide meaningful accountability when they fail. If companies are going to continue relying on automation to moderate users’ speech—and there is little reason to believe they won’t—then accountability must evolve alongside these technologies.
That evolution can start with committing to the Santa Clara Principles 2.0. These principles, first outlined in 2020 and re-launched in 2021 after substantial international input, reflect the needs and expectations of the global community and specifically address automation. The first Foundational Principle states:
Companies should ensure that human rights and due process considerations are integrated at all stages of the content moderation process, and should publish information outlining how this integration is made. Companies should only use automated processes to identify or remove content or suspend accounts, whether supplemented by human review or not, when there is sufficiently high confidence in the quality and accuracy of those processes. Companies should also provide users with clear and accessible methods of obtaining support in the event of content and account action.
Drawing on the Santa Clara Principles 2.0, international human rights standards, and years of research documenting the shortcomings of automated moderation, we propose eight recommendations for policymakers thinking about regulation and companies deploying AI-assisted content moderation systems.
- Automated technologies should help, not replace, human moderators. For example, automated systems can help flag and prioritize content for review, while humans can interpret context, handle sensitive cases, and refine system performance.
- Companies must be transparent about when and how automation is used in content decisions.
- Companies must regularly audit their automated systems for bias, with particular attention to low-resource languages, vulnerable and marginalized communities, and conflict zones.
- Users must have the ability to appeal, and to provide context when they believe human or automated moderation decisions have wrongfully removed their content. Appeals should be promptly evaluated and decided by human moderators.
- Companies should regularly assess the human rights impact of their moderation decisions, and issue public statements of the results
- If they rely on third-party vendors, companies should carefully (and regularly) audit those vendors for compliance with these same principles
- Lawmakers should avoid promoting and passing legislation that effectively or explicitly mandates automated moderation systems
- Policymakers should also refrain from attempting to dictate platforms technical and design choices to favor or disfavor particular expression.
These recommendations understand that automated content moderation isn’t just a technical problem for clever engineers and product teams to solve. Because content moderation shapes public discourse and fundamental rights, its design and oversight must respond to the concerns of policymakers, civil society, independent researchers, and the communities most affected by these systems.
This is the second post in a 2-part series on automated content moderation. Read the first post here.
Facts Only
* Meta's algorithms incorrectly deleted nonviolent Arabic-language content 77 percent of the time.
* Meta's systems failed to detect hate speech under company policies in many instances.
* A 2025 report from the Center for Democracy and Technology found labeled datasets in certain languages contain inconsistencies, bias, and inaccuracies due to limited annotator hiring and language shifts (e.g., Maghrebi Arabic, Kiswahili).
* Investigations into ChatGPT outputs show problems in several low-resource languages.
* Automated systems can process content at a scale humans cannot.
* Automated systems reproduce existing biases and struggle with context.
* The risks of overreliance on automated moderation include the systemic suppression of content from Palestine and misclassification of LGBTQ+ content.
* A Foundational Principle requires integrating human rights and due process into the content moderation process at all stages, and publishing information on this integration.
* Recommendations include ensuring automation helps humans rather than replaces them, requiring transparency, regular auditing for bias, providing user appeal methods, and assessing human rights impact.
Executive Summary
Automated content moderation systems have demonstrated significant failings across various contexts, including the incorrect deletion of nonviolent Arabic-language content and failure to detect hate speech under established policies. Research indicates that these systems exhibit disparities when applied to low-resource languages like Maghrebi Arabic and Kiswahili, due to inconsistencies in training data caused by limited annotator hiring and linguistic shifts. Beyond language issues, automated moderation is implicated in systemic harms, such as the suppression of content related to Palestine and the misclassification of LGBTQ+ content. The text argues that while automation offers potential scale benefits, it reproduces existing biases and struggles with context, leading to disproportionate negative impacts on marginalized communities.
The text proposes evolving accountability through principles like the Santa Clara Principles 2.0, which emphasize integrating human rights and due process into moderation processes. Key recommendations include ensuring automation supplements, rather than replaces, human review, mandating transparency regarding automated usage, requiring regular bias audits focusing on vulnerable groups, and guaranteeing accessible appeal mechanisms adjudicated by humans. The argument is that accountability must evolve alongside the reliance on these technologies to ensure human rights are respected.
Full Take
The narrative exposes a critical tension between the potential efficiency of large-scale automation and the inescapable reality of contextual nuance, linguistic diversity, and systemic power imbalances. The core pattern is the deployment of opaque, error-prone systems to govern public discourse, where the costs of these errors are shifted disproportionately onto already marginalized groups—specifically those speaking low-resource languages or navigating highly sensitive political topics. The failure mode is not merely technical inaccuracy but the abdication of human judgment regarding harm assessment; when perfect accuracy is deemed "intrinsically impossible," the focus must shift from achieving unattainable perfection to establishing robust, context-aware accountability structures.
The proposal for the Santa Clara Principles 2.0 functions as an attempt to re-anchor technological deployment within human rights frameworks, shifting the locus of control from purely technical optimization to procedural justice. The pattern observed is a systemic impulse to privatize moderation decisions while attempting to regulate only the input layer (the systems) rather than the outcome and process. This implies that accountability must address not just algorithmic bias but the very structure of who designs these systems, who controls the training data in endangered linguistic contexts, and who adjudicates failure. The implication is that technological progress without mandated procedural justice reinforces existing marginalization; thus, true resilience requires embedding cultural competence and due process as non-negotiable architectural constraints rather than bolted-on compliance layers.
What is the necessary next step in establishing genuine cognitive sovereignty when systems are designed to reproduce historical inequities? How can policymakers ensure that 'due process' for linguistic minorities is not treated as an afterthought but as a foundational requirement built into the architecture of AI, especially when dealing with the inherent instability of low-resource language data? If perfect accuracy is unattainable, what metric best reflects acceptable risk in systems governing fundamental rights, and who bears the ultimate responsibility when these imperfect systems cause demonstrable societal harm?
Sentinel — Human
The text presents a well-structured argument linking specific failures in automated content moderation to a set of principled policy recommendations, strongly suggesting human authorship rooted in research synthesis.