The HPA has one job: keep resource utilization below a threshold. When it sees CPU pressure, it adds pods. This works great until the thing causing your CPU pressure isn’t your service being overwhelmed, it’s your service being blocked waiting on something else.

I’ve been on-call for incidents where the database connection pool saturated. Application pods queued up waiting for connections. The HPA saw CPU pressure and fired more pods. The new pods each opened a handful of database connections. The pool hit its limit. The database started refusing connections. Error rate spiked. HPA added more pods….You know how this ends.

The frustrating part is that the HPA did nothing wrong. It saw high utilization and scaled. That’s its job. The problem is that it had no way to ask whether adding capacity would actually help.

This is the problem I am building Narya to solve.

The options I looked at first

The most common approach is to encode the safety boundary directly into PromQL. Multiply your burn rate by a boolean — if connection pool utilization exceeds 85%, the query returns zero and the HPA holds steady. It works, technically. But when that boundary activates, there are no events, no status on the resource, no alert if it’s been holding for twenty minutes. During an incident you can’t tell whether your service is healthy or whether it’s deliberately starving itself to protect the database. That distinction determines whether you escalate.

KEDA is excellent and I use it for queue-based scaling. But there’s no native inhibitor mechanism. You can pause a ScaledObject via annotation, but making that dynamic requires writing a separate controller that watches your dependency and patches the annotation in real time. You’ve built the same problem one layer removed, with the same observability gap.

Service mesh circuit breakers are for a different problem. When Envoy sheds traffic because the upstream is slow, that increases the SLO burn rate. If your autoscaler watches the burn rate, the circuit breaker triggers more pods. The two systems compound each other rather than cooperate.

How Narya works

A SLOAutoscaler resource has two parts: a burn rate query and inhibitors.

The burn rate query is PromQL returning your current error rate relative to your SLO budget. Narya scales up when it crosses a threshold and scales down when it drops below a lower one. Inhibitors are PromQL queries for downstream dependencies — if any of them exceeds its threshold, scale-up is vetoed regardless of what the burn rate says.

sloRef:
  burnRateQuery: |
    sum(rate(http_requests_total{job="payments-api",status=~"5.."}[5m]))
    /
    sum(rate(http_requests_total{job="payments-api"}[5m]))    
  scaleUpThreshold: 0.14
  scaleDownThreshold: 0.05

inhibitors:
  - name: db-connection-pool
    query: |
      pg_stat_activity_count{datname="payments"}
      /
      pg_settings_max_connections      
    threshold: 0.85

scaling:
  minReplicas: 2
  maxReplicas: 20
  scaleUpCooldown: 5m
  scaleDownCooldown: 10m
  scaleUpFraction: 0.25
  scaleUpAggressiveness: 1.0
  scaleDownFraction: 0.10
  vetoAlertThreshold: 15m
  inhibitorStalenessTimeout: 60s

When the connection pool exceeds 85%, scale-up stops. A ScaleUpVetoed Kubernetes event fires with the specific reason: “inhibitor db-connection-pool exceeded threshold (0.92 > 0.85)”. A Prometheus gauge goes to 1. If the veto holds past vetoAlertThreshold, a separate alert fires — a signal that scaling can’t fix whatever’s happening and someone needs to look at the database directly.

The observable veto is what I care about most here. “Why aren’t we scaling?” is a question you want answered in seconds during an incident, not after digging through PromQL to figure out why the HPA seems to be ignoring a high burn rate.

The scaling math

Scale-up moves a fraction of the remaining headroom. At 4 replicas with a max of 20, a 25% fraction adds 4 pods. At 12 replicas, it adds 2. A single traffic spike can’t jump you straight to max replicas.

scaleUpAggressiveness amplifies the step proportionally to how far above threshold the burn rate is. At 1.5× threshold you get a bigger step than at 1.1×, without bypassing the geometric ceiling. It just gets there faster when things are actually bad.

Scale-down has a longer cooldown than scale-up — 10 minutes versus 5. This handles the case where the database just cleared a lock storm and you’d otherwise immediately flood it with new connections on the way back up.

If an inhibitor query returns stale data, Narya treats it as a veto. If Prometheus is unreachable, it freezes at the current replica count. It never makes a scaling decision without current data from both sides.

What it won’t do

It doesn’t protect you from writing a bad burn rate query. It doesn’t know your service’s scaling characteristics — those are your inputs. It currently only manages Deployments. And it requires Prometheus, so if you’re not already running it, start there first.

Narya is still being built. The name comes from Tolkien — Narya is the Ring of Fire, one of the three elven rings given to the keepers of the world. It felt right for something that is supposed to hold a line under pressure.