OpenAI’s Jalapeño is a custom accelerator designed specifically for AI inference, particularly the low-latency, multi-chip workloads behind interactive and agentic systems. Its central idea is that useful performance cannot be reduced to peak floating-point operations or memory bandwidth. What matters is how quickly and efficiently the complete request finishes. Accordingly, OpenAI evaluates the chip using end-to-end latency, time between tokens, and tokens per joule across the full latency-throughput trade-off.
The ASIC combines substantial compute and memory capacity: up to 13.4 petaflops of MXFP4 matrix compute, 216 GiB of HBM4, and 15.4 TB/s of memory bandwidth within a 700-watt package. A complete 2,048-chip system reaches 27 exaflops of MXFP4 compute, 432 TiB of memory, and 32 PB/s of aggregate bandwidth. Yet the design argues that raw specifications are not enough. Although aggregate HBM bandwidth theoretically permits thousands of full-model weight reads per second, practical token rates remain far lower because data movement, synchronization, communication, and late-arriving operands leave compute units waiting.
Jalapeño addresses these bottlenecks through a spatial architecture. Each core slice is paired with an HBM slice, creating a fast, high-bandwidth local memory view. Specialized collective networks handle common cross-core communication patterns, while a more flexible network-on-chip supports general traffic. Software places tensors explicitly, performs computation locally, and uses shared paths only when necessary. Distributed control also avoids the costly global fences that can arise when independent cores must synchronize through centralized resources.
This organization reflects the changing demands within a single inference request. Prefill is attention-heavy and largely compute-bound. A speculative draft model is small, operates at extremely low batch sizes, and is highly sensitive to network latency. Verification and decoding combine attention, memory-bandwidth-intensive mixture-of-experts execution, and bursty communication. Rather than assigning these phases to separate accelerators and repeatedly transferring the key-value cache, Jalapeño keeps state local and activates different combinations of compute, memory, and networking resources. Unused blocks can be power-gated. OpenAI’s guiding observation is that “dark” silicon inside one package costs less than idle accelerators that continue consuming baseline HBM, networking, cooling, and rack resources.
At system scale, 128 chips form a local domain, while as many as 2,048 chips are connected through a two-level Clos network. Bandwidth is deliberately tapered: tensor parallelism receives higher bandwidth, expert parallelism requires less, and all traffic benefits from low latency. This makes the interconnect part of the computer rather than an accessory to otherwise independent chips.
On the public InferenceX benchmark, OpenAI reports Jalapeño on the performance-per-watt Pareto frontier for GPT-OSS 120B, DeepSeek R1 670B, and Kimi K2.5 1T. Against the selected GB200 or GB300 configurations, the deck reports 1.5–1.9 times greater peak throughput per kilowatt, 1.7–3.6 times lower end-to-end latency, and 2.1–4.1 times better token interactivity. At the previous system’s time-between-token point, the reported efficiency advantage ranges from 8.6 to 104.3 times. These comparisons use 8,000-token inputs, 1,000-token outputs, four-bit weights, package-TDP normalization, and vendor-optimized software. Importantly, Jalapeño uses single-token prediction while some baselines use multi-token prediction, making the disclosed methodology essential when interpreting the results.
Bottom line: Inference is becoming the dominant operational cost of AI, while agentic applications may invoke models repeatedly before completing one user task. Improvements in requests per second per watt translate into lower energy consumption, greater datacenter capacity, and faster interactive systems. Jalapeño also demonstrates the value of hardware-software-AI co-design: OpenAI moved from initial RTL to tape-out in nine months, used AI to optimize hardware blocks, and generated kernels that reportedly exceeded expert-written implementations. The broader implication is strategic: model developers can now shape silicon around real workloads, shortening the loop between algorithms, compilers, architecture, and deployment—and making frontier intelligence more economical to serve.
Source:
Also Read:
Semicon West: Transforming Tomorrow One Chip at a Time
HBM’s Vertical Ascent: Why Packaging Is Becoming the Heart of AI Memory
Share this post via:
Comparing Advanced Packaging from TSMC, Intel Foundry, and Samsung Foundry
Facts Only
* OpenAI developed Jalapeño, a custom ASIC for AI inference.
* The chip provides 13.4 petaflops of MXFP4 matrix compute per package.
* Each package contains 216 GiB of HBM4 and 15.4 TB/s of memory bandwidth.
* Power consumption is 700 watts per package.
* A 2,048-chip system yields 27 exaflops of MXFP4 compute and 432 TiB of memory.
* Hardware organization consists of 128-chip local domains connected via a two-level Clos network.
* The design utilizes a spatial architecture where core slices are paired with HBM slices.
* Development from initial RTL to tape-out occurred in nine months.
* Performance was measured using the public InferenceX benchmark against GB200 and GB300 configurations.
* Testing parameters included 8,000-token inputs and 1,000-token outputs using four-bit weights.
Executive Summary
OpenAI has introduced Jalapeño, a specialized accelerator optimized for the low-latency requirements of interactive and agentic AI systems. Moving away from traditional metrics like peak floating-point operations, the hardware focuses on end-to-end latency and tokens per joule. The architecture employs a spatial design that pairs compute cores with local HBM slices and utilizes specialized collective networks to reduce data movement bottlenecks during the prefill and decoding phases of inference.
At scale, the system integrates up to 2,048 chips through a tapered Clos network, prioritizing bandwidth for tensor parallelism over expert parallelism. Benchmark data indicates significant efficiency gains over NVIDIA GB200/GB300 configurations in terms of throughput per kilowatt and token interactivity, though these results are based on specific 4-bit weight configurations and single-token prediction. This shift toward hardware-software co-design allows model developers to iterate rapidly on silicon, potentially reducing the operational costs of frontier intelligence.
Full Take
The strongest version of this narrative is that the "compute bottleneck" has shifted from raw FLOPS to data movement and synchronization. By integrating the interconnect as a first-class citizen and utilizing a spatial architecture, Jalapeño represents a transition from general-purpose AI acceleration to workload-specific silicon.
However, a pattern of Authority Game is evident; the performance claims rely entirely on a "deck" and internal benchmarking where the vendor controls the software optimization and the specific token-count parameters. The massive efficiency gains reported (up to 104x) occur at a specific "time-between-token point," a narrow window that may not translate to general use cases. The comparison is further complicated by the use of single-token prediction against baselines that may use multi-token prediction, creating an uneven baseline for "interactivity."
Patterns detected: ARC-0052 Authority Game
The driving paradigm here is the vertical integration of the AI stack. When the model developer becomes the chip architect, the "black box" of hardware is optimized for the "black box" of the model. This creates a powerful feedback loop that accelerates deployment but risks creating proprietary islands of efficiency that are untestable by third parties. The second-order consequence is the potential for "silicon lock-in," where the algorithm and the chip are so interdependent that switching costs become prohibitive.
If this were a coordinated influence campaign, the playbook would involve releasing highly specific, non-replicable benchmark numbers to signal technological dominance and discourage competitors or investors from pursuing alternative architectures. The actual content aligns partially with this by presenting "Pareto frontiers" without providing the raw data for independent verification.
Bridge Questions:
1. How would these efficiency gains change if multi-token prediction were applied across all tested architectures?
2. Does the nine-month RTL-to-tape-out timeline suggest a reliance on existing IP blocks, or a fundamental shift in how AI optimizes hardware design?
3. To what extent does this vertical integration move AI development away from open standards and toward closed, hardware-dependent ecosystems?
Sentinel — Human
The text functions as high-level technical analysis, synthesizing specific hardware specifications with broader implications regarding AI system design and economic efficiency, strongly suggesting human expertise guiding the narrative.