A cloud bill that grows faster than traffic is rarely a scaling problem. In almost every estate we review, the largest single line is capacity that was reserved and never used, and no autoscaling policy can reclaim it because the scheduler is doing exactly what it was asked.
Requests, not usage
A Kubernetes scheduler places pods by their requests. It does not place them by what they actually use. A pod that requests two cores and uses a tenth of one has reserved two cores, and the node it sits on is full as far as the scheduler is concerned.
Multiply that across an estate and you get the characteristic picture: nodes at ninety per cent committed and twenty per cent utilised, a cluster autoscaler dutifully adding more, and a bill that tracks requests rather than work.
Autoscaling on top of guessed requests scales the guess.
This is why request hygiene comes before any scaling policy. It is also why the cost conversation has to involve the people writing the application, because requests are written in the deployment manifest and nobody on the infrastructure side can set them honestly.
Measure first
Run a vertical autoscaler in recommendation mode, or read the percentiles out of whatever observability you already have, and compare actual usage to what each workload requests. Two numbers matter per workload: the ninety-fifth percentile of CPU and the maximum of memory.
Set CPU requests near the ninety-fifth percentile. CPU is compressible. A pod that exceeds it is throttled, which is survivable. Set memory requests at or slightly above the observed maximum, because memory is not compressible and exceeding it ends the process.
- Biggest line
- reserved and unused, nearly always
- Commit last
- discounting a wrong baseline locks it in
- Owned by
- whoever writes the manifest
This diagram as text
- 1 · Switch off non-production out of hours — no architecture change, no risk
- 2 · Requests from measurement — p95 CPU, max memory
- 3 · Let a provisioner pick the instance — across families and sizes
- 4 · ARM where the image supports it
- 5 · Spot for anything interruptible — with a disruption budget
- 6 · Committed use discounts — only once the shape has settled
Relationships
- 1 · Switch off non-production out of hours → 2 · Requests from measurement — buys time
- 1 · Switch off non-production out of hours → 3 · Let a provisioner pick the instance
- 2 · Requests from measurement → 4 · ARM where the image supports it
- 3 · Let a provisioner pick the instance → 5 · Spot for anything interruptible
- 4 · ARM where the image supports it → 6 · Committed use discounts
- 5 · Spot for anything interruptible → 6 · Committed use discounts
Lowering a request is a production change
Measurement tells you where the waste is. It does not make the correction safe, and this is where most request work stalls: a cut that looked obvious on a dashboard ends a process at month end, the team that owns the service puts every request back, and nobody tries again. Meanwhile requests drift upward on their own. A new workload copies its manifest from a neighbour or is sized for the worst input anyone can imagine, and an allocation that fitted one hardware generation or one release of the code is quietly wrong on the next.
Netflix has published how it handles this on Stratum, its media processing platform, where containers are resized in production without a person approving each change. Stratum runs batch functions rather than Kubernetes services, at a scale no estate we work on approaches, but the discipline transfers intact. Every change is a recommendation with a lifecycle, and it has to earn its way into production.
Treat a lower request as a deployment, with a canary, a pass mark and a revert.
Compare against a baseline, not against the fleet. Netflix runs the candidate configuration on a small share of live traffic beside an equally small baseline on the current configuration, and compares those two, because the main fleet sees a different mix of work. On Kubernetes the same shape is a second deployment behind the same service carrying the new requests and a handful of replicas, judged against a matching group on the old ones. Set the pass mark before the test starts: no more memory-driven restarts than the baseline, and median latency within a slowdown the owning team has agreed to accept.
Move memory in small steps. A CPU cut that goes too far makes the work slower, so CPU can go to its target in one change. Memory cannot. Netflix’s own account is the warning: a memory cut of half passed every check, ran cleanly for two weeks, then failed badly when the traffic shifted. Their response was to cap each memory or disk reduction at fifteen per cent and to wait about a week before the next one, so the workload meets a wider spread of real inputs at each size.
Keep watching after it ships, and revert to the last known good. A test proves the change against the traffic that happened to arrive during the test. Watch the same signals afterwards over a short window and a long one, and write down the threshold at which the change reverts, automatically if you can. Revert to the configuration that ran before, not to a newly calculated value. An incident is the wrong moment for the system to be clever.
Know what your platform does when the guess is wrong. Stratum re-runs a task on a container half as large again when it nears its memory limit, which is what lets everyday sizes stay tight. Kubernetes on its own does nothing of the kind. A container killed for exceeding its memory limit restarts with the same limit and is killed again. For a batch job you can build the equivalent by resubmitting with a larger request. For a long-running service the revert is the safety net, which is why it has to exist before the first cut.
- Memory step
- capped; Netflix uses 15% per change
- Between steps
- about a week on one workload
- Revert to
- the previous config, never a new guess
This diagram as text
- 1 · Recommend from usage — p99 CPU, p99.9 memory plus margin
- 2 · Take a small step — memory cut capped per change
- Baseline — current requests — small share of live traffic
- Candidate — recommended requests — equal share of live traffic
- 3 · Compare against the pass mark — OOM restarts, median latency
- 4 · Apply — manifest change, reviewed
- Reject — current requests stay
- 5 · Watch, then revert on breach — short and long window — back to the last known good
Relationships
- 1 · Recommend from usage → 2 · Take a small step
- 2 · Take a small step → Baseline
- 2 · Take a small step → Candidate
- Baseline → 3 · Compare against the pass mark
- Candidate → 3 · Compare against the pass mark
- 3 · Compare against the pass mark → 4 · Apply — passes
- 3 · Compare against the pass mark → Reject — fails
- 4 · Apply → 5 · Watch, then revert on breach
None of this needs a platform team or a tuning service. A weekly pass over the workloads with the widest gap between request and use, one manifest change per workload in the repository, a canary, and a revert rule written into the pull request is the same discipline at the size most estates are. What it does need is the application team’s agreement on the pass mark, because they are the people paged when it is wrong.
Scale on the signal that means load
The second common failure is scaling on CPU because it is the default.
For a request-driven service, CPU is a lagging and noisy proxy. By the time it rises enough to trigger a policy, latency has already degraded, and garbage collection or a neighbour on the same node can move it for reasons unrelated to traffic. Scale on requests per second, on concurrency, or on queue depth for a consumer. Those are the things that actually describe the work arriving.
For a queue consumer in particular, scale on the depth of the queue and nothing else. It is the only signal that says work is accumulating faster than it is being done.
Horizontal and vertical do not combine
Running a horizontal autoscaler and a vertical autoscaler on the same deployment against the same metric produces oscillation: one adds replicas, the other grows them, each reacting to the other’s effect.
The arrangement that works is vertical in recommendation mode to set honest requests, and horizontal in control of the replica count. One of them measures, the other acts.
The levers in order
Switch things off. Non-production environments outside working hours, ephemeral environments reaped on merge, and anything running at three in the morning that nobody asked for. This is the cheapest money in any estate and it needs no architectural change.
Fix requests. As above. This is where the largest sustained saving is, and it is slow because it is per-workload and needs application teams.
Let a provisioner choose the instance. Constraining node provisioning to a single instance family removes most of its ability to find capacity cheaply. Constrain on what the workload genuinely needs and let it choose across families and sizes.
ARM where the image supports it. Better price-performance for the same work. The effort is in the build pipeline producing a multi-architecture image and in actually testing the ARM build, not in the platform.
Spot for anything interruptible, with a disruption budget and graceful termination genuinely implemented rather than assumed.
Commit last. A committed use discount applied to a baseline you have not yet corrected locks in the waste for one to three years. Do the four steps above, let the shape settle, then commit to what remains.
Make the number visible to the people who can move it
A bill that arrives monthly to one person in finance changes nothing. Cost per service and per environment, visible weekly to the teams that own them, changes behaviour without anybody having to run a programme.
The attribution is the work: tags or labels applied at creation, enforced by the pipeline rather than by a policy document, so an untagged resource cannot reach production. Retrofitting attribution onto an estate that has none is several times harder than applying it from the start.
How STP approaches this
We measure before changing anything, because the first instinct is usually to add a scaling policy to a cluster whose requests are wrong. Each lowered request then ships as a canary, with a pass mark and a revert agreed with the team that owns the service. Non-production schedules go in first since they are free and buy time for the slower work. Commitments come last, after the shape has settled, and attribution is enforced in the pipeline so the number stays answerable as the estate grows.
More on cloud, hybrid and multi-cloud and Kubernetes platform engineering, or start a conversation.