A few years ago, getting a model into production meant a data scientist, a DevOps engineer, and a narrow set of tools: train it, test it, ship it, watch the dashboards. Large language models broke that picture. The thing being “operated” is now a system that chains prompts, calls vector databases, and produces open-ended text judged on tone and safety, not just accuracy. That’s the gap LLMOps fills, and it lands on top of MLOps and DevOps, all competing for the same pipeline platform engineering is supposed to run. Get the ownership model wrong, and you recreate the shadow-IT problem platform engineering was built to prevent, just with prompts instead of Jenkinsfiles.
What Is LLMOps?
LLMOps, or large language model operations, is the set of practices, tools, and workflows used to develop, deploy, and manage LLMs throughout their production lifecycle: data management, prompt engineering and fine-tuning, deployment and serving, monitoring and evaluation, and security and governance. It isn’t MLOps with a new label. Google Cloud calls it a specialised subset of MLOps, and the gap comes down to scale and evaluation: LLMs cost more to fine-tune and serve, and their output is far harder to score than a clean accuracy number. An LLM has to be more than accurate; it has to be secure and trustworthy, and that’s a much harder thing to measure.
It’s also the ongoing care and feeding of a model once it’s live, not a one-time deployment checklist. Models drift, costs creep, prompts stop working the way they used to, and the integrations tying an LLM to a CRM or an internal knowledge base need constant tending.
The LLMOps lifecycle, end to end
The lifecycle runs from data prep and prompt engineering (treating prompts as versioned artefacts, not throwaway strings), through fine-tuning open foundation models with libraries like Hugging Face Transformers, to model and prompt versioning and lineage tracking, inference serving on GPU-backed endpoints, and monitoring with human feedback for drift and cost. Every stage needs infrastructure, access controls, and a place to run. This is exactly the territory platform engineering already occupies for every other kind of workload.
Who owns the pipeline: DevOps, MLOps, and platform engineering
Software delivery has fractured into three camps: DevOps teams buried under deployment tickets, platform engineering teams building self-service golden paths in response, and MLOps teams who built an entirely parallel stack (MLflow, Kubeflow, Weights & Biases) because DevOps tooling never anticipated data versioning or drift monitoring. Add LLMOps, and you risk a third parallel stack for prompts, vector stores, and RAG pipelines, invisible to whoever is supposed to be governing the rest.
Platform engineering is infrastructure-centric, and MLOps is model-centric. These are different disciplines solving different problems, with platform engineering serving as the backbone the others run on top of. The unhelpful question is “who owns the pipeline?” The useful one is “who owns which layer, and is anyone actually coordinating across them?”
Where LLMOps fits in the platform engineering layer cake
CNCF TAG App Delivery’s Platforms Whitepaper describes a similar three-tier split: Products at the top, Platforms in the middle (the thinnest reasonable integration layer), and Capability Providers at the bottom. Model fine-tuning jobs, vector databases, prompt registries, and inference endpoints are, functionally, just another platform capability, needing the same API, versioning, and clear ownership as anything else.
The tooling already exists across the CNCF ecosystem: Backstage surfaces golden paths at the product layer, Crossplane composes infrastructure at the bottom layer, and orchestration frameworks such as Kratix, KusionStack, and KubeVela sit in the middle, letting teams expose an LLM pipeline through the same self-service interface as everything else. Skip that, and the DevOps-versus-MLOps turf war resurfaces one layer higher.
The real risk isn’t the model, it’s shadow LLMOps
The bigger operational risk isn’t a hallucinating chatbot; it’s the sprawl around it: a team standing up its own RAG pipeline against an unreviewed vector store, invisible to whoever tracks what’s actually running. This “shadow AI” pattern is the same one that made the DevOps-versus-platform split so painful: a capability gets built outside the platform because the platform wasn’t ready, and it never gets folded back in. The fix isn’t slowing teams down. Instead, it’s making the platform say yes fast, with governance built in.
Governing LLMOps through the platform, not around it
Concretely, platform teams should implement:
- Governed APIs, not ad hoc scripts. Fine-tuning jobs, prompt deployments, and inference endpoints should be requestable through the same self-service interface developers already use for everything else.
- Policy enforcement at request time. Cost limits, data residency rules, and model access controls checked before a job kicks off, not discovered after the cloud bill arrives.
- Human approval where the blast radius warrants it. Not every prompt change needs a sign-off, but a model touching customer PII or making autonomous decisions probably does.
- An audit trail that answers “what changed, and why.” Regulators and security teams want to know what changed about the model, the prompt, or the data, and who or what approved it.
From a governance standpoint, an LLM pipeline is just another automated consumer of platform capabilities that needs the same guardrails a human developer or an autonomous agent would get. The teams getting this right aren’t picking a side in the DevOps-versus-platform-versus-MLOps argument. Rather, they’re treating the entire pipeline, LLM workloads included, like a product: versioned, observable, cost-aware, and built with feedback loops rather than handed down as a fixed set of rules.
Final thoughts
LLMOps isn’t a rebranding of MLOps; it’s MLOps under real strain: bigger models, higher costs, fuzzier evaluation, and a production surface that didn’t exist a few product cycles ago. But that strain doesn’t mean the ownership question needs reinventing. Build the golden path once, expose it as a governed capability, and let every consumer, be they developer, data scientist, or AI agent, use it through the same API or UI. LLMOps doesn’t need its own kingdom. It needs a well-run platform willing to let it in.
This is a live conversation inside CNCF: TAG App Delivery’s Platforms Working Group is where much of the cross-project thinking on golden paths and platform governance is already happening, and it’s open to anyone who wants to help shape it.
If you want to learn more, check out my deep-dive article, “What Is LLMOps and How Does It Relate to Platform Engineering?” on the Syntasso website.
Facts Only
* Getting a model into production historically required a data scientist, a DevOps engineer, and specific tools for training, testing, shipping, and monitoring.
* LLMOps involves practices, tools, and workflows for the entire LLM production lifecycle: data management, prompt engineering/fine-tuning, deployment/serving, monitoring/evaluation, and security/governance.
* LLMOps is a specialized subset of MLOps due to the higher cost of fine-tuning and serving, and the difficulty in scoring outputs beyond accuracy.
* Models require evaluation for security and trustworthiness, which is harder to measure than accuracy.
* The ongoing care required includes managing model drift, costs, prompt efficacy, and integrations requiring constant tending once a model is live.
* The LLMOps lifecycle spans data preparation, prompt engineering (treating prompts as versioned artifacts), fine-tuning open foundation models with libraries like Hugging Face Transformers, model/prompt versioning, inference serving on GPU endpoints, and monitoring with human feedback for drift and cost.
* This lifecycle requires infrastructure, access controls, and execution environments for every stage.
* Software delivery is fractured across DevOps (deployment tickets), platform engineering (golden paths), and MLOps (MLflow, Kubeflow).
* LLMOps introduces a risk of creating a third parallel stack for prompts, vector stores, and RAG pipelines if not integrated into the platform layer.
* Platform engineering focuses on infrastructure, and MLOps focuses on models, with platform engineering serving as the underlying foundation.
* Governing LLMOps requires implementing governed APIs, policy enforcement (cost limits, access controls), human approval where necessary, and audit trails for all changes.
Executive Summary
Large Language Model Operations (LLMOps) describes the set of practices, tools, and workflows for developing, deploying, and managing LLMs across their production lifecycle. This discipline encompasses data management, prompt engineering and fine-tuning, deployment, serving, monitoring, evaluation, and security/governance. The gap between LLMOps and MLOps stems from the increased scale of LLMs, higher costs associated with fine-tuning and serving, and the difficulty in scoring outputs beyond simple accuracy metrics, as LLM trustworthiness requires measuring safety and security alongside performance.
The LLMOps lifecycle covers everything from versioning prompts as artifacts through fine-tuning foundation models using tools like Hugging Face Transformers, managing model/prompt lineage, GPU-backed inference serving, and monitoring for drift and cost. This entire process requires infrastructure, access controls, and execution environments, aligning with existing platform engineering domains.
Ownership of the LLM pipeline is fractured among DevOps, MLOps, and platform engineering, creating potential duplication due to specialized toolsets like MLflow and Kubeflow operating in parallel. The core risk identified is the creation of "shadow LLMOps," where teams build out RAG pipelines and vector stores outside established platforms, leading to operational sprawl invisible to governance structures.
The recommended solution involves platform engineering providing governed capabilities through self-service interfaces, ensuring that deployment, fine-tuning, and inference endpoints are exposed via unified APIs. This requires implementing policy enforcement at request time, establishing human approval points where necessary, and creating comprehensive audit trails to track changes in models, prompts, or data.
Full Take
The narrative suggests a structural tension between specialized domain expertise and centralized platform governance in the context of increasingly complex AI systems. The core pattern identified is the emergence of "shadow IT" or "shadow AI" when specialized teams build capabilities outside the established platform structure, driven by the speed and complexity of LLM workflows. This dynamic mirrors historical friction points like the DevOps versus Platform engineering dispute.
The implication here is that treating LLMOps as merely an extension of MLOps fails to capture the unique demands related to governance, prompt lineage, and safety; it requires a shift from purely model-centric operations to infrastructure-centric product thinking. The risk is not the technical failure of the model itself, but the systemic failure of oversight resulting from this operational sprawl.
The proposed solution advocates for embedding LLMOps capabilities directly into the platform engineering layer by treating prompt registries, vector stores, and inference endpoints as first-class, governed platform capabilities. This shifts accountability upstream, suggesting that governance must be applied uniformly across all automated consumers of infrastructure, regardless of whether they are traditional ML tasks or LLM workflows. The challenge for any organization is not adding another silo, but enforcing a unified contract where specialized knowledge (data science, prompt engineering) consumes the governed infrastructure provided by platform teams.
Bridge Questions: How can existing platform tooling be immediately adapted to recognize prompt versioning and lineage as core asset types? What mechanisms can ensure that policy enforcement applied during development scales seamlessly into runtime monitoring across heterogeneous LLM components? If platform engineering successfully ingests these workflows, how is ownership formally transferred from the domain experts to the platform stewards?
Sentinel — Human
The text functions as sophisticated, well-structured thought leadership synthesizing complex industry concepts into a cohesive argument about operational ownership and platform engineering for LLMs.