In event-driven Kubernetes architectures, CPU and memory utilization often fail to reflect real system pressure. A worker pod may sit idle from a CPU perspective while thousands of messages pile up in an Amazon SQS queue. In other cases, pods may continue running long after a traffic spike has passed.
For asynchronous, queue-based workloads, backlog is the true scaling signal – not infrastructure utilization.
In this article, you’ll learn:
- How Amazon SQS queue depth can work as an autoscaling metric for event-driven workers
- How to scale workloads on Amazon EKS using KEDA with AWS pod identity
- How KEDA calculates desired replicas from queue depth
- How to tune queueLength, activationQueueLength, cooldowns, and HPA behavior
- How to validate scaling behavior and troubleshoot common issues
Why this matters:
In queue-driven systems, delayed message processing directly impacts users and downstream systems. Scaling based on SQS depth aligns autoscaling with actual demand, enabling faster burst handling and lower idle costs when queues are empty.
1. Prerequisites
Before implementing KEDA-based autoscaling with Amazon SQS, ensure you have:
- An Amazon EKS cluster running a supported Kubernetes version
- KEDA installed in the cluster (for example, via the kedacore Helm chart)
- An AWS authentication method selected (IRSA or EKS Pod Identity)
- An existing Amazon SQS queue
- A worker deployment designed to consume messages from the queue
2. Architecture and Request Flow
This architecture relies on KEDA observing Amazon SQS queue metrics and managing a Kubernetes Horizontal Pod Autoscaler (HPA) based on backlog.
Request flow:
- Producers send messages to the Amazon SQS queue
- KEDA polls queue attributes to determine backlog
- KEDA updates the target metrics in the HPA
- The HPA scales the worker Deployment
- Kubernetes schedules additional Pods to process messages
- As the queue drains, replicas scale back down
This model keeps autoscaling decisions tied directly to outstanding work.
3. Implementation Steps
Install KEDA via Helm
The recommended way to install KEDA is via Helm:
Add the KEDA Helm repository
helm repo add kedacore https://kedacore.github.io/charts
helm repo update
Install KEDA into a dedicated namespace
helm install keda kedacore/keda \
--namespace keda \
--create-namespace
What this does:
Deploys the KEDA operator and metrics API server, along with CRDs such as ScaledObject and TriggerAuthentication.
Deploy the Worker with AWS Identity Attached
An SQS consumer typically requires permissions such as:
- GetQueueAttributes
- GetQueueUrl
- ReceiveMessage
- DeleteMessage
- ChangeMessageVisibility
Example IAM policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ConsumeFromQueue",
"Effect": "Allow",
"Action": [
"sqs:GetQueueAttributes",
"sqs:GetQueueUrl",
"sqs:ReceiveMessage",
"sqs:DeleteMessage",
"sqs:ChangeMessageVisibility"
],
"Resource": "arn:aws:sqs:us-east-1:123456789012:orders-queue"
}
]
}
What this does:
Enforces least-privilege access by granting permissions only for the required SQS queue.
Configure KEDA Authentication and ScaledObject
KEDA connects the deployment to the SQS queue using TriggerAuthentication and ScaledObject. Using queueURLFromEnv avoids hardcoding the queue URL.
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
name: sqs-processor-auth
namespace: workers
spec:
podIdentity:
provider: aws
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: sqs-processor
namespace: workers
spec:
scaleTargetRef:
name: sqs-processor
pollingInterval: 10
cooldownPeriod: 120
minReplicaCount: 0
maxReplicaCount: 30
advanced:
horizontalPodAutoscalerConfig:
behavior:
scaleDown:
stabilizationWindowSeconds: 60
triggers:
- type: aws-sqs-queue
authenticationRef:
name: sqs-processor-auth
metadata:
queueURLFromEnv: QUEUE_URL
awsRegion: us-east-1
queueLength: "10"
activationQueueLength: "1"
What this does:
Defines how KEDA authenticates with AWS and how replicas are calculated. Each pod targets 10 messages, and scale-to-zero is enabled when the queue is empty.
4. Replica Calculation Logic
KEDA calculates outstanding work as:
ApproximateNumberOfMessages
+ ApproximateNumberOfMessagesNotVisible
(+ delayed messages, if enabled)
Replica calculation:
desired replicas = ceil(outstanding messages / queueLength)
Example (queueLength: “10”):
- 0 messages → 0 pods
- 8 messages → 1 pod
- 25 messages → 3 pods
- 95 messages → 10 pods
Final replica counts are bounded by minReplicaCount and maxReplicaCount.
5. Verification
After applying the manifests:
1. Send 50 messages to the SQS queue
2. Inspect the ScaledObject:
kubectl get scaledobject sqs-processor -n workers
3. Watch pod scaling:
kubectl get pods -n workers -w
4. Confirm pods scale back to zero after processing completes
6. Troubleshooting Common Issues
No scale-out
- Cause: Incorrect queue URL or IAM permissions
- Check: kubectl describe scaledobject, KEDA operator logs
Too many pods
- Cause: In-flight messages counted
- Check: scaleOnInFlight, SQS visibility timeout
Never scales to zero
- Cause: Delayed or unacknowledged messages
- Check: Queue attributes and application behavior
HPA exists but no scaling
- Cause: Authentication or metric fetch errors
- Check: KEDA logs and fallback status
7. Production Tuning Tips
- Choose queueLength based on throughput, not guesswork
- Tune cooldown and HPA behavior separately – they control different scale-down paths
- Enable fallback replicas for resilience if metrics become unavailable
- Decide whether in-flight messages should count based on visibility timeout behavior
Conclusion
Queue-based workloads benefit most when autoscaling is driven by the amount of work waiting to be processed rather than traditional infrastructure metrics. Backlog-aware scaling enables applications to react more quickly to changing demand while reducing unnecessary resource consumption during quieter periods.
Although this article demonstrated the approach using Amazon SQS, Amazon EKS, and KEDA, the same design pattern applies across many event-driven architectures and messaging platforms. The key is selecting a scaling signal that accurately represents workload demand and tuning the autoscaling behaviour to match the characteristics of the application.
As organisations increasingly adopt asynchronous, event-driven systems, workload-aware autoscaling becomes an important part of building resilient, efficient, and cost-effective Kubernetes deployments.
Facts Only
* Worker pods may be idle from a CPU perspective while SQS queues have pending messages.
* Backlog in an Amazon SQS queue is presented as the true scaling signal for asynchronous workloads.
* The architecture uses KEDA to observe Amazon SQS queue depth.
* KEDA updates target metrics in the Horizontal Pod Autoscaler (HPA).
* Request flow involves producers sending messages to SQS, KEDA polling queue attributes, and HPA scaling the deployment.
* A TriggerAuthentication resource connects KEDA to AWS using pod identity.
* A ScaledObject defines the scaling behavior with parameters like `queueLength`, `activationQueueLength`, and replica bounds.
* KEDA calculates desired replicas based on outstanding messages divided by the configured `queueLength`.
* The system uses specific configuration values, such as `queueLength: "10"` for a target of 10 messages per pod.
* Troubleshooting involves checking IAM permissions, KEDA logs, and SQS queue attributes to diagnose scaling issues.
Executive Summary
Asynchronous, queue-based workloads benefit from scaling based on actual backlog rather than infrastructure utilization. This approach allows systems to scale in response to the amount of work waiting to be processed, enabling faster burst handling and cost reduction during idle periods when queues are empty. The proposed architecture leverages Amazon SQS queue depth as the primary autoscaling signal for event-driven workers running on Amazon EKS.
The implementation involves using KEDA to monitor the SQS queue metrics to drive scaling decisions in a Kubernetes Horizontal Pod Autoscaler (HPA). This flow connects message ingestion via SQS producers, metric observation by KEDA, configuration of target replicas via ScaledObject, and final scaling execution by Kubernetes. The system uses specific parameters like `queueLength` and `activationQueueLength` to define the relationship between queue depth and desired replica counts, with settings for cooldown periods and stabilization windows to manage scale-down behavior.
The core objective is to ensure that resource consumption aligns directly with outstanding work, allowing systems to scale down efficiently when demand subsides. The process requires careful tuning of KEDA parameters and validation steps to account for in-flight messages and ensure accurate reflection of the actual workload state.
Full Take
This approach shifts the focus of autoscaling from reactive infrastructure metrics (CPU/memory) to proactive workload demand (queue depth), which is a fundamentally more relevant signal for asynchronous systems. The pattern involves decoupling the consumption rate from raw resource utilization, recognizing that latency in message processing directly impacts downstream systems. The design relies on careful quantification: establishing an appropriate `queueLength` and accounting for visibility timeouts is critical because KEDA's calculation of "outstanding work" hinges entirely on how SQS visibility state translates into actionable metrics for the autoscaler.
The implication for systemic resilience is that scaling decisions become inherently workload-aware, reducing waste when queues are empty while ensuring swift reaction during spikes. However, this introduces a dependency: if the application or AWS messaging layer incorrectly reports backlog (e.g., due to unacknowledged messages), the resulting scaling behavior will be flawed. The challenge lies in tuning the interaction between Kubernetes scheduling logic and Amazon's eventual consistency in queue state management.
What is missing is a standardized, universally applicable methodology for defining the relationship between SQS visibility timeouts, KEDA polling intervals, and HPA cooldowns across diverse messaging platforms. Future investigation should explore how to integrate more nuanced latency metrics alongside simple depth counts to create truly adaptive scaling policies that account for processing time rather than just queue volume.
Sentinel — Human
The text presents a technically sound, highly structured guide on using KEDA for SQS-based autoscaling. The structure and depth suggest human technical expertise, though it is framed in an encyclopedic, instructional style.