GPT‑Red: Unlocking Self-Improvement for Robustness
Training strong automated safety red-teamers to improve robustness.
Summary
Problem
Red-teaming is essential to discovering vulnerabilities and improving the robustness of our models. However, current approaches are not scalable, creating a bottleneck.
Commonly used robustness evaluations have already been saturated by our latest models.
We need to develop methods that allow safety and alignment to scale alongside model capabilities.
What we did
We trained GPT‑Red, an automated red-teaming model that scales our ability to find vulnerabilities so we can fix them before wider deployment.
GPT‑Red is a strong red-teamer, and our previous models are highly vulnerable to its prompt injection attacks.
We use GPT‑Red to adversarially train GPT‑5.6, making it much more robust to prompt injections.
We will continue to scale this approach alongside human and third-party red-teaming, layered safeguards, and real-time monitoring.
AI systems commonly encounter third-party data through browsers, connected apps, local files, and other tools. These affordances are necessary for performing real-world tasks, but they also create more opportunities for malicious actors to influence model behavior. For example, a third party might embed a carefully crafted instruction—designed to trick the model into uploading sensitive data to an external server—in an email, webpage, tool response, or code repository.
Human red-teaming is a critical part of our safety work, helping us uncover these vulnerabilities before deployment and put the right safeguards in place. But human red-teaming alone is difficult to scale. Designing and running these exercises is time-intensive, limiting how quickly we can identify new failure modes and incorporate them into stronger safeguards. Further, while these exercises produce valuable examples of successful attacks, they cannot generate the volume and diversity of adversarial data needed to improve model robustness through training.
Keeping pace with increasingly capable models requires red-teaming to scale as well. To this end, we have been training automated, internal-only red-teaming models that uncover vulnerabilities before deployment and generate attacks during model training to improve robustness. We believe automated red-teaming unlocks a crucial form of self-improvement for safety: using today’s models to directly help make future models safer.
GPT‑Red is the culmination of these efforts and our current best automated safety red-teaming model. Similar to how human red-teamers craft attacks, the model works toward a goal by sending a prompt, observing how GPT models respond to it, and iterating. We trained GPT‑Red at the compute scale of some of our largest post-training runs at OpenAI—an unprecedented amount of compute dedicated purely for improving safety.
We directly incorporate GPT‑Red into the training process of our production models. As a result, GPT‑5.6 Sol is our most robust model to prompt injections to date, achieving 6x fewer failures on our hardest direct prompt injection benchmark compared to our best production model from just four months earlier. The scalability of our approach leaves us excited for even stronger results in the future as we continue to train stronger red-teamers.
GPT‑Red is trained using self-play reinforcement learning, where the model and a collection of diverse defender LLMs are trained simultaneously on a broad set of red-teaming scenarios. GPT‑Red is rewarded for eliciting a valid failure, such as a successful prompt injection, while the defender models are rewarded for resisting the attack and completing their original tasks. As the defenders become more robust, GPT‑Red is forced to discover stronger and more diverse attacks.
To support self-play training, we build an expansive set of realistic scenarios where prompt injections might be inserted. Each environment has a threat model that specifies what GPT‑Red can control and what counts as a successful attack. For instance, GPT‑Red might control part of a local file, a webpage banner, an email body, or output of a tool.
At the end of its training, GPT‑Red is a very strong attacker: it can break nearly all models it is pitted against, both internal and production models up to and including GPT‑5.5. After GPT‑Red completed training, we used it to generate prompt injections for the training of GPT‑5.6, resulting in the model becoming highly resistant to GPT‑Red’s attacks.
We keep GPT‑Red separate from the models we deploy. This keeps the malicious capabilities we specifically train into GPT‑Red out of the hands of adversarial actors, while instilling robustness into our production models.
GPT‑Red is highly effective against the population of defender models and red-teaming scenarios that it was trained on. We also evaluate whether the model is useful as a general-purpose red-teaming agent to benefit safety broadly at OpenAI. To do so, we test GPT‑Red’s effectiveness on novel safety environments and target models.
We first evaluate GPT‑Red’s ability to generalize to novel red-teaming scenarios using a replicated version of the indirect prompt injection arena from Dziemian et al. (2025)(opens in a new window). In this challenge, both human red-teamers and GPT‑Red independently proposed attacks against GPT‑5.1 on a set of pre-specified environments. These red-teaming scenarios and goals are distinct from those used to train GPT‑Red. GPT‑Red achieves significantly higher attack success rates, finding success on 84% of scenarios compared to 13% for humans.
The ultimate test of a red-teamer is the ability to achieve targeted malicious goals against real-world agentic systems with incomplete knowledge of the system’s underlying model and harness design. Our first experiment in this setting pitted GPT‑Red against an AI-powered vending machine in the OpenAI office (similar to Project Vend(opens in a new window)) produced by Andon Labs. We gave GPT‑Red a description of the system and the ability to send attacks and observe tool calls from the simulated agent that closely mirrors the real world deployment. After iterating on attacks, GPT‑Red deployed its attack against the production agent, achieving all three of its malicious objectives:
- Change the price of an expensive in-stock item to the minimum allowed price of $0.50;
- Order a new $100+ item and offer it for $0.50; and
- Cancel another customer’s order.
We disclosed these vulnerabilities and new safeguards are actively being tested.
We also use GPT‑Red to attack a Codex CLI agent (based on GPT‑5.4 mini) on a suite of 10 held-out data-exfiltration scenarios. We compare the model to a prompted GPT‑5.5 baseline to study the impact of our training procedure on held-out red-teaming performance. GPT‑Red is both more effective, in that it can successfully get the agent to exfiltrate sensitive data in more scenarios, and is more token efficient.
The ultimate goal of GPT‑Red is to improve the robustness of our models. Over the last six months, we’ve trained progressively stronger red-teaming models (precursors to GPT‑Red) with increasing compute, and used these models in the training of each successive production model since GPT‑5.3. Over time, each subsequent GPT release has gotten more robust.
As one example, an early version of GPT‑Red found a novel class of direct prompt injection attacks known as “Fake Chain-of-Thought” attacks. These attacks achieved success rates of upwards of 95% on GPT‑5.1 but are now below 10% for GPT‑5.6 Sol. Similarly, several of our indirect prompt injection benchmarks that target attacks in developer tools and browsing have been saturated by our latest model (>97% accuracy).
Robustness to GPT‑Red itself has also improved substantially. On a broad set of robustness environments, GPT‑Red’s attack success rates have dropped monotonically over time. With our latest model release, GPT‑5.6 Sol fails on only 0.05% of GPT‑Red’s direct prompt injections.
A model can appear safer by refusing more requests or becoming less capable. A model that does less is naturally harder to attack, but that is not useful robustness.
We thoroughly evaluate both general frontier capabilities along with targeted over refusal tasks that we design. We find that all normal capabilities remain unaffected while significantly improving robustness. This suggests that the robustness gains came from better resistance to malicious instructions rather than improper tool-usage or refusing legitimate requests by default.
AI agents are already being used to improve the capabilities of our next-generation models. We believe with GPT‑Red that we have started to unlock a similar flywheel for safety, where today’s models can be used to make tomorrow’s models more robust, aligned, and trustworthy. We will continue to scale compute and data while making algorithmic improvements, to train future versions of GPT‑Red that are stronger than today’s model. And in turn, these models will help make future GPT releases safer.
We will be releasing a pre-print with more details later this week.
Facts Only
* GPT-Red is an automated red-teaming model.
* GPT-Red was used to adversarially train GPT-5.6.
* The training involved self-play reinforcement learning with defender LLMs.
* GPT-Red was rewarded for eliciting a valid failure, such as prompt injection.
* The process utilized realistic scenarios where prompt injections could be inserted into local files, webpages, emails, or tool outputs.
* GPT-Red demonstrated the ability to break nearly all internal and production models up to GPT-5.5 in testing environments.
* GPT-Red was used to generate prompt injections for training GPT-5.6.
* The training resulted in GPT-5.6 Sol achieving 6x fewer failures on a direct prompt injection benchmark compared to the preceding model.
* GPT-Red's attack success rates against human red-teamers were 13% across specified scenarios, compared to 84% for GPT-Red.
* GPT-Red successfully executed malicious objectives against an AI-powered vending machine agent in a simulated environment.
* GPT-Red was used to attack a Codex CLI agent on data exfiltration scenarios.
* The model’s performance regarding prompt injection attacks decreased monotonically over time with subsequent GPT releases.
Executive Summary
The development of GPT-Red involved training an automated red-teaming model to enhance the robustness of large language models against prompt injection attacks. This process uses GPT-Red, which operates through self-play reinforcement learning, interacting with defender LLMs across a range of adversarial scenarios to elicit successful attacks. The system generates diverse, realistic prompts that can target various components, including local files, webpages, and tool outputs.
This automated method addresses the scalability limitations of human red-teaming by generating the volume and diversity of adversarial data required for training. GPT-Red is directly incorporated into the training process for production models, such as GPT-5.6 Sol, resulting in improved resistance to prompt injections—achieving six times fewer failures on benchmark attacks compared to previous models. The approach involves keeping the red-teaming mechanism separate from deployed models, and further testing of GPT-Red against novel agentic systems demonstrated its ability to achieve complex malicious goals in simulated environments.
Full Take
The narrative pivots on the concept of shifting safety improvement from slow, labor-intensive human assessment to scalable, automated adversarial training. The pattern observed is a feedback loop: models improve robustness by attacking each other within a carefully constructed environment. This suggests that achieving advanced alignment may require systems capable of self-generation of high-quality stress tests, moving beyond passive defense toward active, internal adversarial evolution.
The distinction between robustness gained from resisting malicious instructions versus merely refusing requests creates a critical tension regarding the definition of safety itself. The finding that normal capabilities were unaffected while robustness increased suggests that optimization is occurring at the boundary of malicious instruction handling rather than general utility constraints. This implies a necessary philosophical shift: safety should be engineered as resilient capability against manipulation, rather than managed by explicit adherence to baseline requests.
The ultimate implication for agency lies in the delegation of adversarial skill. By training an agent to become a superior attacker, the system leverages its current understanding to enforce future security protocols on itself. The risk then shifts from external actors exploiting known weaknesses to the emergent properties of self-directed adversarial evolution. This raises questions about controlling the trajectory of these internally generated adversarial evolutions and ensuring that the goal of self-improvement remains tethered to broader human safety goals rather than purely internal optimization metrics. What mechanisms exist to ensure that the pursuit of robustness does not create novel, unaligned objectives within the red-teaming agent itself?
Sentinel — Human
This text reads like an internal research summary detailing advanced AI safety engineering methodologies, characterized by deep technical specificity and structured argumentation.