How the lab ships code

The short version

We run a single-node k3s Kubernetes cluster in our homelab and manage it with GitOps. Argo CD keeps the cluster matching a folder in our Git repository, using an app-of-apps pattern. Our read-only AI endpoint, an MCP server, is packaged as a Helm chart. Argo Workflows is our CI: it tests every pull request that touches the server, posts the result back to GitHub as a commit status, and when a change merges it builds the container image with BuildKit, pushes it to a registry inside the cluster, and rolls out the new version. Nobody runs a command between merge and deploy.

The cluster: k3s in an LXD container

• k3s is a lightweight, fully certified Kubernetes distribution: one binary and a small memory footprint, a good fit for a single machine.
• It runs inside an LXD system container on our second server, capped at 6 GB of memory and 3 CPU cores at low priority, so that server’s real jobs (narrating stories, the media library, a local AI model) always come first.
• Why a container and not a virtual machine: that server’s firmware has hardware virtualization switched off. The container still keeps Kubernetes’ networking rules in their own network namespace, so the host’s firewall and security alerts don’t change, and one command removes the whole cluster.
• Nothing new is exposed on the network. The dashboards are reached through an SSH tunnel.

GitOps with Argo CD

• Git is the source of truth. Every Kubernetes object lives in a folder of our control-plane repository, and we change the cluster by merging a pull request, not by typing commands.
• App of apps: one root Application points at a folder of Applications, one per component: the image registry, the CI system and the MCP server’s test copy. Adding a component means adding one file.
• Automated sync with prune and self-heal: if someone changes the cluster by hand, Argo CD puts it back the way Git says, and if a file is deleted from Git, its object is deleted from the cluster.
• Argo CD reads the private repository with a fine-grained, read-only token that exists only as a Kubernetes Secret, never in Git.

Packaging with Helm

• Our MCP server (the read-only endpoint AI assistants use to search Arc’s articles and verify their signatures) is packaged as a Helm chart: a Deployment, a Service and a values file for its settings.
• The chart follows standard hardening: it runs as an unprivileged user with a read-only root filesystem, drops every Linux capability, sets a memory limit, and has readiness and liveness probes, so Kubernetes only sends traffic to a healthy copy and restarts a stuck one.
• Production still runs outside Kubernetes. The cluster runs a test copy that talks only to Arc’s public API, so it needs no access to production systems.

CI with Argo Workflows

• GitHub Actions is paused on our account, so CI runs inside the cluster instead.
• A CronWorkflow checks the repository every 10 minutes, except on weekday evenings, when our electricity costs the most.
• For each new pull request that changes the MCP server, it starts a Workflow from a WorkflowTemplate: download that exact commit, run the tests in a pod, and post the result back to the pull request as a commit status. Changes that don’t touch the server get an immediate “no changes” status, so nothing waits on work that wasn’t needed.
• When a change reaches main, the same Workflow also builds the container image with BuildKit, pushes it to the cluster’s own registry, and restarts the test copy so it runs the new image.
• Each step is its own pod with its own memory limit. A shared volume carries the code from step to step, and an exit handler always reports success or failure.

What went wrong the first time

The first automatic deploy crash-looped. Kubernetes injects environment variables for every Service in a namespace, so our Service named arc-mcp produced ARC_MCP_PORT=tcp://…, which collided with the server’s own ARC_MCP_PORT setting. The pod logs showed the error in one line. The fix was one line in the chart, enableServiceLinks: false, shipped as a pull request, and Argo CD rolled it out within minutes.

Supply chain and safety basics

• Every image in our own manifests is pinned by its content digest, not just a tag, so a rebuilt image can’t slip in unnoticed.
• Kubernetes, Helm and the Argo charts are pinned to exact versions, and the Helm download is checked against its published checksum before use.
• The CI’s GitHub token can read code and write commit statuses, nothing more.
• Changes to Arc itself still pass our full local test gate, the same steps GitHub would run, before they merge.

Beyond Kubernetes: how the rest of the lab ships

• A local test gate runs the same steps as our GitHub workflow (lint, type checks, unit tests, a production build and alert-rule tests) on a clean checkout of each branch before it merges, because GitHub Actions is paused.
• Frontend deploys build the new image first and swap only when it’s ready, so the site stays up, and the previous image is kept for a one-step rollback.
• The narrator runs on both servers. One deploy script ships the code to both, restarts them, and reports failure unless the files’ checksums match on both; an hourly health check catches any drift between deploys.
• One health check, a few dozen claims about services, fresh data, backups and security controls, is the go/no-go for reboots and kernel updates.
• Reboots follow a runbook: a pre-check must say safe, the second server reboots first as a canary, then the main one, and a post-check must say all good.
• Custom kernels are built for each server and installed for one boot only; if a new kernel fails to boot, the machine falls back to the old one by itself.
• Backups count only when proven: every night a check decompresses the newest copies, verifies their checksums and runs a git integrity check.

Monitoring

A small exporter reports the cluster’s health to the lab’s Prometheus every five minutes: whether the node is ready, how many pods aren’t running, how many Argo CD applications are out of sync or unhealthy, and whether the last CI run passed.

Words worth knowing

k3s

A lightweight, certified Kubernetes distribution.

Pod, Deployment, Service

A pod runs containers; a Deployment keeps the right pods running and replaces them on updates; a Service gives them a stable name and address.

Helm chart

A package of Kubernetes templates plus a values file for the settings.

GitOps

Running a system by changing Git; a controller makes the live system match.

Argo CD Application

Argo CD’s record of “this folder in Git belongs in that place in the cluster”.

Self-heal and prune

Undoing manual changes, and deleting what Git no longer has.

WorkflowTemplate and CronWorkflow

A reusable pipeline, and a schedule that starts one.

Commit status

The pass or fail mark a CI system posts next to a commit on GitHub.

BuildKit

The engine behind modern Docker builds, runnable inside a pod.

Image digest

The sha256 fingerprint of one exact image.

Runbook

Written steps for a routine or emergency operation, so it goes the same way every time.

Rollback

Returning to the last known-good version, ideally in one step.

Configuration drift

Copies of a system slowly becoming different; checksums and GitOps catch it.

Canary

Trying a change on one machine first, so a bad change hurts only that one.