Since announcing Jalapeño, OpenAI’s first custom inference chip, we have been testing the chip and the system built around it. The results show a significant performance advance: Jalapeño can serve more AI work per unit of power while also returning responses more quickly. Jalapeño delivers both higher throughput and lower latency with one architecture, where existing hardware systems often have to make a tradeoff between the two.
For customers, that can mean faster responses, more responsive agents, and more reliable access as demand grows. Our mission is to ensure that artificial general intelligence benefits all of humanity. These gains will help make increasingly capable AI more affordable and more broadly available.
OpenAI models also accelerated Jalapeño’s development. Earlier generations helped the team design and bring up the chip, while our latest models are accelerating how we optimize and program it. Jalapeño’s performance extends across GPT‑OSS 120B, DeepSeek R1, and Kimi K2.5 1T, showing that the architecture works across models developed both inside and outside OpenAI. Across all three, Jalapeño delivered 1.5 to 1.9 times more AI work per watt at peak throughput and 1.7 to 3.6 times lower end-to-end latency than the comparison systems. For highly interactive workloads, it delivered 2.1 to 4.1 times higher performance.
Jalapeño is also evidence of a broader full-stack advantage. OpenAI can design models, products, serving software, chips, memory, networking, and systems together, using what we learn from real workloads to improve every layer of the stack. Jalapeño is working first-party silicon with measured results, and it is the beginning of a multigenerational platform. In the months ahead, we will ramp Jalapeño to deliver faster, more capable, and more efficient products for our customers.
We evaluate performance at a matched user experience, measuring how much useful AI work each system can complete per unit of power while meeting the latency customers and interactive agents require. This matters especially for agents, which need to complete many steps in sequence, so delays can compound across an entire task.
To understand how Jalapeño performs in practice, we tested it on InferenceX, a public benchmark from SemiAnalysis that measures the full process of serving an AI request. We compared Jalapeño with leading commercially available AI systems across the tested operating range, from high-throughput serving to highly interactive, low-latency use. Jalapeño delivered a better combination of throughput, power efficiency, and latency. Although performance is sometimes reported per chip, we believe the more useful standard is performance per unit of power.
Across all three public models, Jalapeño delivered a better combination of performance per watt and latency across the tested operating range, placing it on the Pareto frontier. To compare the systems consistently, we normalized the results using each accelerator’s published chip power rating. Jalapeño is rated at 700 watts, although its measured sustained power remained at or below 550 watts on the workloads tested.
Jalapeño performed strongly across GPT‑OSS 120B, DeepSeek R1 670B, and Kimi K2.5 1T. On Kimi, the largest public model we tested, it delivered approximately 1.5 times higher peak performance per watt and 3.4 times lower end-to-end latency than the comparison system. In our internal testing, Jalapeño’s advantage widened further on frontier OpenAI models, suggesting that the architecture becomes more valuable as workloads grow larger and more demanding.
Jalapeño was designed from the start by asking: what hardware would we build if its primary job were serving modern and future language models, especially interactive agents? Jalapeño’s gains come from designing the chip, memory, network, software, and rack-scale system together around real language-model workloads. Language-model inference moves through several distinct phases with different bottlenecks. Prefill, when the system processes a prompt, is compute-intensive, while decode, when the system generates the response token by token, is constrained more by memory bandwidth. Communication can also add latency when data must move between cores and chips, leaving some processing units idle while they wait. A system that excels at one phase can lose that advantage while waiting for data or moving model state between different resources.
We designed Jalapeño to minimize data movement and communication delays. This means that model state, including the KV cache used while generating a response, can be explicitly placed and kept local while the system activates the right combination of compute, memory, and networking for each inference phase. The network is integral to the architecture. Its large domain allows the entire workload to remain within one connected system, minimizing data movement and helping the complete request stay fast and efficient from beginning to end. The result is a balanced and fungible accelerator that can support changing model architectures, excel at both prefill and decode, and adapt as the balance between them changes, a defining feature of agentic workloads.
AI played a direct role in Jalapeño’s development, enabling the team to move from initial design to tapeout in nine months by exploring implementations, shortening design, measurement, and verification loops, and continuously iterating on model workloads. AI also helped optimize the chip’s arithmetic circuits, allowing the team to fit more compute performance into the chip on schedule.
Jalapeño was designed as a clear, predictable programming target for both humans and AI. Engineers can describe work through local tensors, explicit communication, and predictable synchronization. AI can then optimize how that work is mapped, placed, scheduled, and coordinated across the system. That clear, predictable structure gives AI a tractable way to tackle the traditionally difficult problem of parallel programming.
Supporting each new model family still requires new kernels and model-specific optimizations. Using Codex with GPT‑Astra, the team brought three open-weight models that were not part of Jalapeño’s original production plan to high performance within two months. This demonstrated both the flexibility of the architecture and the speed at which AI can help us program it. For selected GPT‑OSS attention and mixture-of-experts blocks, AI-generated implementations ran 1.5 to 1.8 times faster than the existing human-expert-written implementations. Those figures apply to the selected blocks, not the full model, but they point toward a powerful new development loop.
AI infrastructure is valuable because of the useful real-world work it enables. By producing more useful work from the same power and hardware, Jalapeño can help us serve more demand and lower the cost of delivering a successful result. For OpenAI, that can improve operating leverage by allowing useful work and revenue to grow faster than the cost to serve. It can also support broader adoption and continued investment in better models, products, and infrastructure. Faster inference can enable faster iteration and new use cases.
Jalapeño expands what is possible for efficient, low-latency inference:
- Ultra-fast-mode inference at efficiencies previously available only in fast mode
- Fast-mode inference at efficiencies previously available only in batched mode
- Higher efficiency for batched-mode inference
We plan to begin deploying Jalapeño within OpenAI’s compute infrastructure by the end of the year. It is the first generation of a multigenerational roadmap: Gen 2 is deep in development, and Gen 3 is taking shape. Each generation will build on what we learn and further advance both efficiency and speed.
Meeting growing demand for AI will require more compute from every available source. We will continue to widely deploy accelerators from NVIDIA and other partners for both training and inference workloads. Our mission is to ensure that artificial general intelligence benefits all of humanity.
As we prepare for deployment, we are continuing production qualification, maturing the software, preparing to operate Jalapeño at scale, and validating performance across more models. The results so far show what is possible when we design the full system together: more responsive, capable, and agentic AI delivered more efficiently to more people.
Higher peak mixed TPS / kW
≈1.9×
85,448 vs. 44,960 mixed / kW
Lower end-to-end latency
≈1.7×
1.03 s vs. 1.80 s
Lower min TBT
≈2.7×
0.69 vs. 1.87 ms (1,459 vs. 535 tok/s/user)
More throughput at previous TBT
≈53.7×
22,935 vs. 427 mixed / kW (at 535.28 tok/s/user)
Higher peak mixed TPS / kW
≈1.7×
19,641 vs. 11,781 mixed / kW
Lower end-to-end latency
≈3.6×
1.65 s vs. 5.99 s
Lower min TBT
≈4.1×
1.43 vs. 5.90 ms (700 vs. 169 tok/s/user)
More throughput at previous TBT
≈104.3×
12,258 vs. 118 mixed / kW (at 169.41 tok/s/user)
Higher peak mixed TPS / kW
≈1.5×
18,195 vs. 11,862 mixed / kW
Lower end-to-end latency
≈3.4×
1.56 s vs. 5.31 s
Lower min TBT
≈3.8×
1.44 vs. 5.48 ms (694 vs. 182 tok/s/user)
More throughput at previous TBT
≈56.1×
6,744 vs. 120 mixed / kW (at 182.46 tok/s/user)
Facts Only
* Jalapeño is OpenAI’s first custom inference chip.
* Testing showed Jalapeño can serve more AI work per unit of power while returning responses more quickly.
* Jalapeño delivered 1.5 to 1.9 times more AI work per watt at peak throughput compared to comparison systems.
* Jalapeño achieved 1.7 to 3.6 times lower end-to-end latency than comparison systems.
* For highly interactive workloads, performance was 2.1 to 4.1 times higher.
* Jalapeño performed across GPT‑OSS 120B, DeepSeek R1, and Kimi K2.5 1T.
* On Kimi, the largest tested public model, Jalapeño delivered approximately 1.5 times higher peak performance per watt and 3.4 times lower end-to-end latency than the comparison system.
* The chip was rated at 700 watts but measured sustained power remained at or below 550 watts on tested workloads.
* Jalapeño architecture minimizes data movement and communication delays by keeping model state local during inference phases.
* AI accelerated development by enabling faster design, measurement, and verification loops.
Executive Summary
Jalapeño, OpenAI’s custom inference chip, demonstrates industry-leading speed and efficiency in AI inference by achieving higher throughput while reducing latency simultaneously, addressing a historical tradeoff in existing hardware systems. The chip performs well across various models, including GPT‑OSS 120B, DeepSeek R1, and Kimi K2.5 1T, showing performance across models developed internally and externally. In testing against these models, Jalapeño delivered 1.5 to 1.9 times more AI work per watt and 1.7 to 3.6 times lower end-to-end latency compared to comparison systems. This advantage was particularly pronounced for highly interactive workloads, where performance increased between 2.1 and 4.1 times.
The architecture of Jalapeño is designed to minimize data movement and communication delays by keeping model state, such as the KV cache, local while optimizing compute, memory, and networking across inference phases (prefill and decode). This integrated full-stack approach allows for a balanced accelerator capable of supporting changing model architectures. Development was accelerated by incorporating AI into the design process, allowing teams to iterate faster on chip optimization and model adaptation. The platform aims to facilitate faster, more capable, and efficient products by enabling ultra-fast-mode inference and higher efficiency across batched and interactive modes.
Full Take
The narrative positions Jalapeño not merely as a piece of silicon but as evidence for a superior, holistic approach to building AI infrastructure—integrating models, software, chips, and systems from the ground up. The core implication is that traditional hardware design often forces an inefficient compromise between throughput and latency, whereas the multimodal optimization achieved in Jalapeño reveals a pathway where both can be simultaneously optimized through architectural coherence. This reinforces the argument that performance measurement must shift from isolated chip metrics to system-level efficiency per unit of work delivered, especially for complex agentic applications where sequential delays compound rapidly.
The description of how the architecture addresses inference bottlenecks (prefill vs. decode and data movement) suggests a deep understanding of the practical constraints of language model serving. The subsequent use of AI to optimize specific blocks further establishes a powerful feedback loop: real-world workload demands guide architectural innovation, which in turn is accelerated by algorithmic optimization. This cyclical process shifts the focus from incremental hardware gains to systemic design principles that can support future, more complex models and agentic systems efficiently.
The pattern observed is a deliberate framing of innovation as an integrated engineering challenge where AI acts as the accelerant for cross-disciplinary optimization rather than just a computational tool. The unspoken assumption is that efficiency gains derived from optimizing data flow are the most valuable form of advancement, suggesting that future competition in AI infrastructure will be defined by system coherence rather than component-specific benchmarks. What remains to be explored is whether this integrated system design translates consistently when scaling beyond current public model benchmarks to truly novel agentic workflows where temporal dependencies are highly dynamic and context switching is continuous.
Sentinel — Human
This text reads like a detailed technical update from a research or product development team presenting results for a new chip architecture, rather than typical journalistic reporting.