SRE Throttling Resolution Blueprint
The Dashboard Illusion: Why Averages Lie
The P99 latency alert fires. You check Grafana, and every pod is sitting comfortably at 20% CPU utilization. Nothing looks overloaded. The spike settles after a few minutes, the ticket gets closed as a transient error, and the same pattern returns three days later. Welcome to the most misunderstood phenomenon in container orchestration: kubernetes cpu throttling with low cpu usage.
CPU utilization metrics average out data over 15 to 60 seconds. However, Kubernetes CPU limits are actively enforced by the Linux kernel's Completely Fair Scheduler (CFS) in microscopic 100-millisecond windows. Your application acts like a NASCAR driver in a school zone. It handles a burst of traffic and exhausts its 100ms quota in the first 10ms. The kernel then forcibly freezes (throttles) your container for the remaining 90ms.
Your dashboard shows "low average usage," but your application was completely frozen for 90% of that second. This triggers a deadly Retry Amplification Spiral where clients retry timeouts, tripling the load on an already frozen service.
Phase 1: Detecting Throttling via PromQL
When debugging k8s cpu limits 100m vs 500m, you must understand how limits translate to cgroup parameters (cpu.max in cgroups v2). A 100m limit grants only 10ms of runtime per 100ms window, triggering severe throttling for almost any active application. A 500m limit grants 50ms.
[Important thing] Stop Looking at CPU Usage
To find hidden throttling, you must stop looking at container_cpu_usage_seconds_total. Instead, track the exact percentage of 100ms windows where the kernel parked your application.
Execute this kubernetes cpu cfs throttled periods prometheus query in Grafana. If the resulting ratio exceeds 15-25%, your application is actively suffering from artificial latency:
# Throttling Ratio Query: Percentage of CFS periods where the container was throttled
sum by (namespace, pod, container) (
rate(container_cpu_cfs_throttled_periods_total{container!="", container!="POD"}[5m])
)
/
sum by (namespace, pod, container) (
rate(container_cpu_cfs_periods_total{container!="", container!="POD"}[5m])
)
* 100
Phase 2: Fixing Thread Amplification (JVM/Go)
Why does a container with a 1-core limit burn through its 100ms quota in just 3 milliseconds? The hidden culprit is Thread Amplification.
[Warning] The Host-Core Blindspot
If you deploy a Java or Go application on a Bare Metal node with 64 physical cores, the language runtime looks at the Host OS, sees 64 cores, and spawns 64 Garbage Collection or Worker threads. If your container limit is only 2 CPUs, all 64 threads will wake up simultaneously, instantly shredding your cgroup quota in milliseconds.
You must force your runtimes to respect cgroup limits natively, without hardcoding anti-patterns:
# 1. For Java / JVM (JDK 11+):
# Rely purely on UseContainerSupport (default) so the JVM automatically detects
# Kubernetes cgroup limits and right-sizes its ForkJoin pools dynamically.
# DO NOT hardcode -XX:ActiveProcessorCount=2 as it breaks when limits scale!
ENV JAVA_OPTS="-XX:+UseContainerSupport"
# 2. For Golang (Pre-1.25):
# Add Uber's automaxprocs library to your main.go to auto-tune GOMAXPROCS
import _ "go.uber.org/automaxprocs"
Phase 3: The 2x P99 Right-Sizing Rule
If you research kubernetes cpu limits best practices, you will often find developers asking why kubernetes cpu limit is bad. Because of the throttling paradox, 80% of blogs will tell you to simply "remove CPU limits entirely." This is extremely dangerous advice in an enterprise environment.
[Alert] The "Remove Limits" Security Threat & Eviction Myth
1. CPU Exhaustion DoS: In a shared cluster, removing limits allows a compromised container to consume 100% of the node's CPU cycles, starving critical DaemonSets like CoreDNS.
2. The Node Starvation Trap: It is a myth that high CPU usage causes the Kubelet to evict your pod (CPU is a compressible resource). The real danger is that an unconstrained pod can starve system processes (like container runtime), causing the entire node to freeze and enter a NotReady state.
3. OOMKill vs Throttling: Do not confuse CPU limits with Memory limits. Hitting a CPU limit causes latency (throttling). Hitting a memory limit causes fatal OOMKills. NEVER remove memory limits.
Instead of blindly removing limits, Elite SREs implement the 2x P99 Right-Sizing Rule using the Vertical Pod Autoscaler (VPA):
- Run the Kubernetes VPA in
Recommendation mode to safely monitor your app's true telemetry without restarting pods. - Measure the actual P50 (median) CPU usage over 7 days and set it as your
requests.cpu. This drives accurate scheduling. - Measure the P99 (peak burst) CPU usage over 7 days, multiply it by 2, and set it as your
limits.cpu. This provides vast headroom for micro-bursts while maintaining node security.
Phase 4: Eradicating Throttling with Bare Metal
Even with perfectly tuned CFS Quotas, running Kubernetes on Public Cloud VMs (AWS, GCP) introduces a fatal hidden layer of latency known as Hypervisor Steal Time.
In the cloud, you are renting vCPUs that are time-sliced by a hypervisor. When your application bursts, you suffer from "Double Throttling"—first by the Kubernetes CFS quota, and second by the hypervisor fighting for physical CPU cycles alongside noisy neighbors on the same hardware.
To achieve absolute, deterministic execution, you must deploy your clusters on ServerMO Dedicated Bare Metal Servers. Bare metal provides 100% exclusive access to physical CPU cores, unshared L1/L2 caches, and zero hypervisor virtualization overhead.
[Important thing] Static CPU Manager Policy (The Holy Grail)
On ServerMO Bare Metal, you can enable the Kubelet's cpuManagerPolicy: static flag. By placing the pod in the Guaranteed QoS class—meaning both CPU AND Memory requests exactly equal their limits (e.g., cpu: 4 and memory: 8Gi)—Kubernetes will completely bypass the CFS 100ms quota system. Instead, it will pin your container directly to exclusive physical CPU cores. No quotas. No stutters. Just raw, unthrottled, predictable power.
