Running large language model (LLM) workloads in-house is one of several patterns teams adopt alongside managed API services. Managed API services are convenient and well suited to many workloads. Self-hosting is a complementary option that some teams choose for reasons such as cost predictability at high request volumes, more control over latency, and controlling data-residency to meet contractual or regulatory requirements. A hybrid approach (running open source models locally for high-volume or sensitive workloads while using managed API calls for tasks that benefit from them) is a practical middle ground that combines both. This article documents what it took to set up that kind of self-hosted inference stack in a Kubernetes lab environment. In this example, we’ve chosen to use vLLM for inference and LINSTOR® for persistent storage.
Background on vLLM
vLLM is a high-performance, open source inference engine for large language models. It is designed for serving many concurrent requests efficiently in a cluster environment.
An important characteristic of vLLM for this use case is that it exposes an OpenAI-compatible REST API. Anything that already talks to the OpenAI API (LangChain, LlamaIndex, or your own code that calls the OpenAI SDK) can be pointed at a self-hosted vLLM instance with nothing more than a URL change. That compatibility is what makes the hybrid architecture described earlier a viable alternative.
Overview of the setup
Instructions in this article use a Kubernetes cluster with LINSTOR providing persistent storage through the LINSTOR Container Storage Interface (CSI) driver. Kubernetes is the orchestration layer for the entire stack, and the storage integration follows the standard CSI used across the cloud native ecosystem. LINSTOR is an open source software-defined storage solution built on DRBD® that provides replicated block storage across nodes. Replicated storage is a good fit for storing large model weight files that need to survive pod restarts and node failures.
The model used in this blog is meta-llama/Llama-3.2-1B-Instruct
, a small but capable model from Meta that has been fine-tuned to follow user instructions. At 1B parameters, it is lightweight enough to run on CPU (important for a lab without dedicated GPU nodes) while still being useful for testing the setup.
Prerequisites
Before deploying anything in Kubernetes, you need access to the model itself. Meta Llama models are gated on Hugging Face, meaning you need to request access before you can download them.
- Create an account at huggingface.co.
- Navigate to the Llama-3.2-1B-Instruct model page and submit an access request.
- After approval, go to your Hugging Face account settings and create an access token with read permissions.
- Keep that token nearby because you will need it in the next section.
You will also need LINSTOR deployed into Kubernetes along with a StorageClass for Kubernetes workloads to request PersistentVolumeClaims
. The open source Piraeus Operator deploys LINSTOR and the LINSTOR CSI driver into Kubernetes and documents the setup. In the next section you will create a PersistentVolumeClaim from a LINSTOR StorageClass named, linstor-csi-lvm-thin-r2.
Deployment
The deployment consists of three Kubernetes resources: a PersistentVolumeClaim
for model storage, a Secret for the Hugging Face token, and a Deployment and Service to run the inference server.
Creating the PVC and secret
The PVC uses the linstor-csi-lvm-thin-r2
storage class, which provisions a thin-provisioned LVM volume with two replicas across the cluster. This provides both redundancy and efficient use of disk space, which is important when model weights can easily consume tens of gigabytes.
The mountPath of the container is /root/.cache/huggingface
. This is where the vLLM container caches downloaded model weights. By backing this path with a persistent volume, the model is downloaded from Hugging Face only once, and later pod restarts skip the download entirely because they will persist on the LINSTOR-provided persistent storage.
Enter the following command to create the PVC and the Hugging Face token secret:
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: vllm-models
spec:
storageClassName: linstor-csi-lvm-thin-r2
accessModes:
- ReadWriteOnce
volumeMode: Filesystem
resources:
requests:
storage: 50Gi
apiVersion: v1
kind: Secret
metadata:
name: hf-token-secret
type: Opaque
stringData:
token: "REPLACE_WITH_YOUR_TOKEN"
EOF
❗ IMPORTANT: Replace REPLACE_WITH_YOUR_TOKEN
with your actual Hugging Face access token and change the StorageClass name if yours is different, before applying this configuration.
Deploying vLLM
Enter the following command to deploy the vLLM inference server and its service, adapted from the vLLM Kubernetes deployment documentation:
VLLM_IMAGE=public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:latest
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: vllm-server
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: vllm
template:
metadata:
labels:
app.kubernetes.io/name: vllm
spec:
containers:
- name: vllm
image: $VLLM_IMAGE
command: ["/bin/sh", "-c"]
args: [
"vllm serve meta-llama/Llama-3.2-1B-Instruct --gpu-memory-utilization 0.80"
]
env:
- name: HF_TOKEN
valueFrom:
secretKeyRef:
name: hf-token-secret
key: token
ports:
- containerPort: 8000
volumeMounts:
- name: llama-storage
mountPath: /root/.cache/huggingface
volumes:
- name: llama-storage
persistentVolumeClaim:
claimName: vllm-models
apiVersion: v1
kind: Service
metadata:
name: vllm-server
spec:
selector:
app.kubernetes.io/name: vllm
ports:
- protocol: TCP
port: 8000
targetPort: 8000
type: ClusterIP
EOF
📝 NOTE: Even on a CPU-only deployment, vLLM uses the --gpu-memory-utilization
flag to govern how aggressively it reserves memory for processing requests. Without this flag, vLLM defaults to 92% of available memory, which caused startup failures in my testing environment. Setting it to 0.80 provided enough headroom in my environment for the engine to initialize successfully.
Watching the logs
The first startup takes a few minutes. vLLM needs to download the model weights from Hugging Face (roughly 2.5GB for this model) and initialize the engine. Enter the following command to follow the logs.
kubectl logs -f deployment/vllm-server
After you see something such as INFO: Application startup complete, you are ready to test.
Testing the deployment
The simplest way to test from inside the cluster is to deploy a throwaway curl pod. Enter the following command to create one.
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: curl-client
namespace: default
spec:
containers:
- name: curl
image: curlimages/curl:latest
command: ["sleep", "infinity"]
restartPolicy: Never
EOF
Start an interactive shell environment in the pod:
kubectl exec -it curl-client -- sh
And send a request to the vLLM service by using its in-cluster DNS name:
curl http://vllm-server.default.svc.cluster.local:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.2-1B-Instruct",
"messages": [
{"role": "user", "content": "What is the capital of France?"}
]
}'
If everything is working, you will get back a response in the familiar OpenAI format:
{
"choices": [
{
"message": {
"role": "assistant",
"content": "The capital of France is Paris."
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 15,
"completion_tokens": 9,
"total_tokens": 24
}
}
When you are done testing, delete the curl pod:
kubectl delete pod curl-client
Pausing the deployment
If you need to pause the deployment while troubleshooting without losing your PVC or configuration, scale it to zero:
kubectl scale deployment vllm-server --replicas=0
💡 TIP: Scaling the deployment to zero keeps the PVC and cached model weights intact. The model will not need to be re-downloaded when you scale the deployment back up.
Bring it back up with:
kubectl scale deployment vllm-server --replicas=1
Conclusion
This lab setup demonstrates the basic pattern: a self-hosted LLM running in Kubernetes, backed by replicated LINSTOR-managed persistent storage, and accessible through a standard OpenAI-compatible API. Caching the model weights on a LINSTOR volume means that pod restarts are fast, and the DRBD-backed replication means that the volume is not a single point of failure.
From here, you could try adding GPU nodes to the cluster for significantly better performance, or training the model further on your own private data to make it more capable for your specific use case. If you are building toward hybrid AI, a natural next step is to layer in an inference router that dispatches requests between the local deployment and a managed API based on cost, latency, or capability requirements. The llm-d project is an open source request router for Kubernetes designed for exactly this kind of setup. These next steps are directions for further exploration rather than configurations covered in this lab.
If you have questions about running vLLM or any containerized applications with LINSTOR in Kubernetes, or about other use cases for LINBIT® software in your environment, you can join the LINBIT Community Forum.
Facts Only
* Self-hosting LLM workloads is an option alongside managed API services.
* Self-hosting allows for cost predictability at high request volumes and control over latency and data residency.
* A hybrid approach uses self-hosted models locally and managed APIs for different tasks.
* vLLM is an open-source inference engine compatible with the OpenAI API.
* The setup uses a Kubernetes cluster orchestrated by LINSTOR for persistent storage via CSI.
* LINSTOR is a software-defined storage solution built on DRBD for replicated block storage.
* The model used is meta-llama/Llama-3.2-1B-Instruct.
* Model weights are cached on a volume mounted at /root/.cache/huggingface within the container.
* A PersistentVolumeClaim named vllm-models, requesting 50Gi, is used for storage.
* A Secret stores the Hugging Face access token required for model download.
* The deployment uses an image from public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:latest.
* Testing successfully executed a curl request to the vLLM service via cluster DNS, receiving a standard JSON response.
Executive Summary
Running large language model workloads in-house offers an alternative to managed API services, providing benefits like cost predictability at high request volumes, latency control, and data residency requirements. Self-hosting is presented as a complementary option to managed services. A hybrid approach combines self-hosted, open-source models for high-volume or sensitive tasks with managed API calls for other functions, leveraging the OpenAI-compatible nature of tools like vLLM to enable this integration.
The setup details deploying an inference stack on Kubernetes using vLLM for serving and LINSTOR for persistent storage via a CSI driver. The specific example uses the Llama-3.2-1B-Instruct model, which is suitable for CPU execution in a lab environment. Persistent storage is achieved by mounting a PersistentVolumeClaim backed by a replicated LINSTOR volume to cache model weights, ensuring persistence across pod restarts and node failures due to DRBD replication.
The process involves setting up prerequisites like obtaining necessary access tokens and deploying three main Kubernetes resources: a PersistentVolumeClaim, a Secret for the token, and a Deployment/Service for the vLLM server. Testing confirms that an external request can be routed through the self-hosted instance using its cluster DNS name, receiving standard OpenAI-formatted responses. The deployment includes a mechanism to scale the service to zero for safe maintenance while preserving state.
Full Take
The narrative frames self-hosting as a pragmatic solution for teams seeking control and predictable costs in AI infrastructure, positioning it as an essential component of a broader hybrid strategy involving managed services. The technical execution demonstrates that open-source tools like vLLM can effectively interface with Kubernetes orchestration and distributed storage solutions like LINSTOR to achieve this goal. The critical pattern is the successful decoupling of model serving from the persistent data layer via CSI integration, which directly addresses resilience concerns through block-level replication.
The implication for cognitive sovereignty lies in understanding where control resides: moving computation and data residency into a self-managed Kubernetes environment mitigates external dependency risks inherent in fully managed services. However, the reliance on specific configurations—such as the necessity of setting `--gpu-memory-utilization` to avoid startup failures—highlights that achieving true operational autonomy often requires deep, granular understanding of the underlying engine mechanics, rather than simple API calls. The suggested next step of layering an inference router suggests a necessary evolution: moving beyond mere self-hosting to creating an intelligent control plane for workload distribution.
The pattern suggests a tension between convenience (managed APIs) and sovereignty (self-hosting). The technical success in this lab validates that the necessary components exist to achieve sovereignty, but it simultaneously underscores the complexity required to maintain it—the configuration nuances become the new barrier to entry. One must question whether the promise of complete control is truly attainable without incurring significant operational overhead; does the ability to self-host simply trade external vendor lock-in for internal complexity? How is the cost and complexity of achieving this sovereignty factored against the realized benefits in real-world organizational contexts?
Sentinel — Human
The text functions as high-quality, hands-on technical documentation detailing a complex setup, showing strong structural coherence and specific experiential knowledge.