Deploying Artificial Intelligence (AI) and Machine Learning (ML) workloads at scale has become a primary objective for modern enterprises. However, moving these data-heavy, stateful workloads into cloud native infrastructure introduces massive data bottlenecks.
To help organizations navigate this fast-evolving landscape, the CNCF Technical Advisory Group for Infrastructure (TAG Infrastructure) has released its latest comprehensive white paper: Data On Kubernetes – Data Analytics and AI/ML Workloads
The Challenge: Storage at the speed of AI
Traditional storage architectures optimized for standard microservices fall short when tasked with feeding massive datasets into parallelized, high-performance accelerator hardware like GPUs. Infrastructure teams face unique hurdles across the data lifecycle:
- The Small-File Trap: Datasets consisting of millions of small files put immense pressure on storage metadata servers.
- Decoupled Bottlenecks: Compute-storage disaggregation scales efficiently but can introduce heavy API call overhead and low GPU utilization rates.
- Shifting Workload Profiles: High-throughput batch training jobs require sustained data movement, whereas production inference demands low-latency, spiky request-response profiles.
Key technical pillars inside the White Paper
The white paper breaks down the cloud native AI data ecosystem into critical structural layers:
- Data Lake Houses & Vector Databases: The guide explores the merging of centralized systems into hybrid data lake houses using open formats like Apache Parquet and Iceberg. It also dives into Vector Databases (like Milvus) that handle high-dimensional embeddings for similarity searches and Retrieval-Augmented Generation (RAG).
- Caching & Data Locality: To eliminate data transfer lag, the paper outlines data locality strategies, highlighting the CNCF project Fluid for orchestrating distributed caching within Kubernetes.
- Standardized Interfaces (CSI & COSI): Learn how the community bridges storage layers using the Container Storage Interface (CSI) for block/file storage, the Container Object Storage Interface (COSI) for object storage, and cloud-native FUSE CSI drivers.
- Modern Data Pipelines: The paper maps out structural roadmaps for moving from legacy batch blocks to real-time streaming using Change Data Capture (CDC) and event streaming platforms like Apache Kafka.
Storage profiles across the AI lifecycle
A major highlight of the white paper is its granular breakdown of storage footprints across three distinct phases:
1. Model training
A long-running, throughput-oriented phase focused on maximizing GPU utilization. Storage must tolerate non-sequential access due to random data shuffling and survive massive, synchronized write bursts during checkpointing (saving model states to protect against hardware failures).
2. Model inference
A latency-sensitive phase characterized by spiky traffic and rapid model-loading requirements. Production systems rely heavily on advanced memory architectures like KV Caching and Prefix Caching to eliminate redundant conversational calculations.
3. Agentic AI (AI Agents)
Emerging AI agents introduce a complex, closed-loop iterative reasoning architecture. Their storage footprint requires short-term memory (mutable state tracking and append-only event histories), artifact repositories for intermediate code or media blobs, and long-term memory to consolidate past sessions.
Get involved!
The TAG Infrastructure community welcomes your insights as we develop sustainable, practical architectural patterns for modern cloud native workloads.
- Read the full white paper
- Contribute by checking out the TAG Infrastructure Charter.
- Join the conversation on the #tag-infrastructure channel on the CNCF Slack workspace!