Kubernetes ships with built-in awareness of CPU and memory, but most real-world scaling decisions depend on signals that live entirely outside that narrow window: how many messages are waiting in a queue, how long the last batch job took, how many active WebSocket connections a pod is holding. When the built-in metrics are not enough, a metrics exporter bridges that gap.
This post walks through writing one from scratch, packaging it as a container, and wiring it into a cluster so that Prometheus — and ultimately the HorizontalPodAutoscaler — can consume it.
What a metrics exporter actually does
An exporter is a small HTTP server with a single responsibility: expose
application state as text on a /metrics
endpoint. Prometheus scrapes
that endpoint on a regular interval, stores the time-series data, and
makes it available for queries, alerts, and autoscaling rules.
In some cases you can instrument your application directly — embedding
the Prometheus client library and exposing /metrics
from within the
same process — rather than running a separate exporter. A standalone
exporter makes more sense when the data source is external to your
application or when you do not control the application code.
The format Prometheus expects is plain text — one metric per line, with a name, optional labels, and a numeric value. Client libraries handle the serialization for you, so in practice you only need to decide what to measure and call the right function when that value changes.
Choosing what to measure
Before writing any code, it helps to decide what kind of signal you are dealing with. The Prometheus data model has three main types:
Counters only ever increase. They are the right tool for totals: requests served, jobs processed, errors encountered. Never use a counter for a value that can go down.
Gauges represent a current snapshot of a value that can rise and fall freely. Queue depth, active connections, and cache size are all gauges.
Histograms record the distribution of observed values, such as request latency. They let you calculate percentiles (p99, p50) rather than just averages.
Once you know which type fits your signal, choose a name that follows
the convention __
in snake_case
. A job
processor might expose worker_jobs_processed_total
(counter),
worker_queue_depth
(gauge), and worker_job_duration_seconds
(histogram). Clear names save everyone debugging time later.
Setting up the project
The Go Prometheus client is the most common choice for exporters in the Kubernetes ecosystem, largely because the same library powers most of the official Kubernetes components. Start by creating a module and pulling in the dependency:
mkdir my-exporter && cd my-exporter
go mod init example.com/my-exporter
go get github.com/prometheus/client_golang/prometheus
go get github.com/prometheus/client_golang/prometheus/promhttp
Registering metrics
Create main.go
. The first thing to do is declare the metrics and
register them with Prometheus's default registry. Registration tells
the library that these metrics exist so they appear in the output even
before the first observation is recorded:
package main
import (
"log"
"net/http"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
)
var (
jobsProcessed = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "worker_jobs_processed_total",
Help: "Total number of jobs processed, partitioned by status.",
},
[]string{"status"},
)
queueDepth = prometheus.NewGauge(prometheus.GaugeOpts{
Name: "worker_queue_depth",
Help: "Current number of jobs waiting in the queue.",
})
jobDuration = prometheus.NewHistogram(prometheus.HistogramOpts{
Name: "worker_job_duration_seconds",
Help: "Time spent processing a single job.",
Buckets: prometheus.DefBuckets,
})
)
func init() {
prometheus.MustRegister(jobsProcessed, queueDepth, jobDuration)
}
prometheus.MustRegister
panics on a duplicate registration, which
makes misconfigurations obvious at startup rather than silently at
runtime. If you are embedding this exporter inside a library that other
packages will also instrument, prefer prometheus.Register
and handle
the error yourself.
Collecting real values
With the metrics registered, the next step is to keep them current. You can either continually update the data as the data change, or run your own internal refresh loop. The pattern below shows a polling loop — a goroutine that periodically reads from whatever data source your application owns and updates the registered metrics. Replace the simulated values with real calls to your database, internal API, or message broker:
import (
"math/rand"
"time"
)
func collectMetrics() {
for {
// Replace these with real reads from your application.
depth := float64(rand.Intn(50))
queueDepth.Set(depth)
start := time.Now()
time.Sleep(time.Duration(rand.Intn(200)) * time.Millisecond)
jobDuration.Observe(time.Since(start).Seconds())
jobsProcessed.WithLabelValues("success").Inc()
time.Sleep(5 * time.Second)
}
}
The polling interval (here five seconds) should be shorter than Prometheus's scrape interval so that each scrape sees a fresh value. The default scrape interval in most cluster deployments is fifteen seconds, which gives you comfortable headroom.
Exposing the endpoint
Wire the collection loop and the HTTP handler together in main
. A
/healthz
path alongside /metrics
gives Kubernetes a liveness probe
target without exposing metric data on the health route:
func main() {
go collectMetrics()
http.Handle("/metrics", promhttp.Handler())
http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
})
log.Println("Listening on :8080")
if err := http.ListenAndServe(":8080", nil); err != nil {
log.Fatalf("server error: %v", err)
}
}
Verify the output locally before building the image:
go run .
curl http://localhost:8080/metrics | grep worker_
You should see three # HELP
and # TYPE
blocks followed by the
current metric values. If those lines appear, the exporter is working
correctly and is ready to be containerized.
Build a container image
A multi-stage build keeps the final image small and avoids shipping a Go toolchain to production. The first stage compiles a statically linked binary; the second stage copies only that binary into a minimal base. The example below uses Docker, but the same pattern works with any OCI-compatible build tool such as Buildah or Podman:
FROM golang:1.21-alpine AS builder
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /exporter .
FROM gcr.io/distroless/static:nonroot
COPY --from=builder /exporter /exporter
EXPOSE 8080
ENTRYPOINT ["/exporter"]
distroless/static:nonroot
contains no shell, no package manager, and
runs as a non-root user by default, which satisfies most cluster
security policies without extra configuration.
Build and push the image, replacing
with your own registry
address:
docker build -t /my-exporter:v1.0.0 .
docker push /my-exporter:v1.0.0
(Note: Using a CI/CD pipeline to automate this is generally a better pattern than running these commands manually.)
Deploying to the cluster
Two manifests are enough to run the exporter: a Deployment that manages the pod lifecycle, and a Service that gives Prometheus a stable address to scrape. (You might prefer to have Prometheus scrape from every Pod; if that makes sense for your use case, then it's OK to configure instead).
The examples below use the monitoring
namespace, which is a common
convention when running Prometheus and related components together. Adjust
the namespace to match your own cluster setup.
The Deployment sets conservative resource limits appropriate for a
lightweight sidecar-style process, and uses the /healthz
route for
its liveness probe:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-exporter
namespace: monitoring
labels:
app.kubernetes.io/name: my-exporter
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: my-exporter
template:
metadata:
labels:
app.kubernetes.io/name: my-exporter
spec:
containers:
- name: exporter
image: /my-exporter:v1.0.0
ports:
- name: metrics
containerPort: 8080
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
resources:
requests:
cpu: 50m
memory: 32Mi
limits:
cpu: 100m
memory: 64Mi
The Service names the port metrics
, which the ServiceMonitor in the
next section will reference by that name:
apiVersion: v1
kind: Service
metadata:
name: my-exporter
namespace: monitoring
labels:
app.kubernetes.io/name: my-exporter
spec:
selector:
app.kubernetes.io/name: my-exporter
ports:
- name: metrics
port: 8080
targetPort: metrics
Apply both:
kubectl apply -f deployment.yaml -f service.yaml
Telling Prometheus where to look
How you configure scraping depends on how Prometheus was installed.
Option 1: Prometheus Operator (ServiceMonitor)
If you installed Prometheus using the
Prometheus Operator
or the kube-prometheus-stack
Helm chart, the operator must be running
in your cluster before you create a ServiceMonitor. The release
label
must match the label selector configured on your Prometheus resource —
kube-prometheus-stack
is the default for a standard Helm install:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: my-exporter
namespace: monitoring
labels:
release: kube-prometheus-stack
spec:
selector:
matchLabels:
app.kubernetes.io/name: my-exporter
endpoints:
- port: metrics
interval: 15s
path: /metrics
Option 2: Annotation-based discovery
If your Prometheus uses annotation-based pod discovery instead, you will
need a matching scrape_config
rule in your Prometheus configuration —
check with whoever manages your Prometheus installation to confirm it is
in place.
You can add the following two annotations to the Pod template regardless of which scraping method you use. They are ignored by the Prometheus Operator but picked up automatically by annotation-based setups:
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8080" # omit if not using annotation-based discovery
prometheus.io/path: "/metrics" # omit if not using annotation-based discovery
If you are unsure which setup your cluster uses, the ServiceMonitor approach is more explicit and easier to debug.
Verifying the scrape
Port-forward to the Prometheus service and open the targets page to confirm the exporter has been discovered:
kubectl port-forward svc/prometheus-operated 9090 -n monitoring
Navigate to http://localhost:9090/targets
. The my-exporter
target
should appear with state UP. If it shows DOWN, check that the
ServiceMonitor's release
label matches and that the pod is running:
kubectl get pods -n monitoring -l app.kubernetes.io/name=my-exporter
kubectl describe servicemonitor my-exporter -n monitoring
Once the target is healthy, run a quick query in the expression browser to confirm data is flowing:
rate(worker_jobs_processed_total{status="success"}[2m])
A non-zero result here means the full pipeline is working: your application is producing data, Prometheus is scraping it, and the time-series are stored and queryable.
What comes next
A working exporter is the foundation, not the destination. The natural
next step is surfacing these metrics to the
HorizontalPodAutoscaler
so that your workload scales on the signals that actually drive load,
not just CPU. That requires a metrics adapter — the Prometheus Adapter
is the most widely deployed option — which registers your custom metrics
with the Kubernetes Custom Metrics API. Once registered, any
HorizontalPodAutoscaler in the cluster can reference worker_queue_depth
or worker_jobs_processed_total
directly in its metrics
block.
For a walkthrough of that setup, see Autoscaling on multiple metrics and custom metrics. For a catalog of ready-made exporters covering databases, message brokers, and cloud services, the Prometheus exporters and integrations page is a good starting point.