Google officially defines this error as occurring when "the zone does not have enough resources available to fulfill the request" — distinguishing it from quota limits, which are a separate class of problem. The error, when you eventually find it buried in GKE cluster-level logs, looks like this:

Node scale up in zones us-central1-f associated with this pod failed:
GCE out of resources. Pod is at risk of not being scheduled.

The issue has been documented across Google's own developer forums since April 2026, with fresh reports appearing through August. According to community threads, it has affected n1, n2, and e2 instance types across all availability zones in us-central1 — zones a, b, c, and f — making the standard "try a different zone" advice ineffective when the shortage is regional.

What Engineers Are Reporting on Google's Own Forums The reports started quietly in spring and have grown louder through summer.

"For many years I've been deploying Compute Engine VMs to us-central-1 region but for the past couple days I've constantly received error ZONE_RESOURCE_POOL_EXHAUSTED."

— Kevin_Koehler · April 13, 2026 · Google Developer Community

"Assumed it was a temporary resource shortage but it's been happening consistently for the past week, which has been very frustrating."

— nju · May 18, 2026 · Google Developer Community

"I tried zones a, b, c, and f. Same issue. I tried e2, n1, n2. Same issue... I guess GCP doesn't want my money anymore 🤷"

— standev · August 13, 2026 · Google Developer Community

⚠️ No Official Response. As of the time of writing, no Google employee has responded to these forum threads, and no incident has been posted to the GCP Status Dashboard.

The Failure Mode That Looks Like Slowness

What makes this particularly dangerous for production systems is how the failure presents itself. When GCP cannot provision a requested machine type, it does not return an error to your application. The GKE cluster autoscaler keeps retrying silently. Your worker pods stay in Pending state. Your job scheduler marks tasks as still running. Your monitoring dashboards show activity. Nothing appears broken.

The only place the truth surfaces is in GKE cluster-level logs — not in your application logs, not in your job scheduler UI. Most engineers only look there when something is visibly broken, which by then is hours later.

In distributed batch workloads on GKE, a common pattern is a lightweight driver pod that coordinates the job and schedules work across executor (worker) pods that do the actual computation. When this capacity issue strikes, the driver pod starts successfully — it's small and fits on an existing node. The executor pods never arrive.

"The driver node got created around 19:30 UTC but the executors were allocated only around 00:30 UTC — 5 hours later. This is the case with all the tasks."

— Platform engineer's documented incident report

Which Machine Types Are Affected

Three Actions You Can Take Today

• Stop autoscaling from zero on SLA-critical node pools
  When your minimum node count is zero, a capacity crunch becomes an indefinite stall. Raising the minimum floor means your workloads always have somewhere to land.

# Terraform — raise minimum node count

# Before — zero floor, entirely dependent on autoscaler
min_node_count = 0 
max_node_count = 10

# After — warm baseline; autoscaler supplements from here
min_node_count = 4 
max_node_count = 10 

Always commit this through your IaC tooling. A manual GCP Console change will be silently overwritten on the next apply.

• Try a different machine type
  Google's official troubleshooting documentation explicitly lists "selecting a different machine series or type" as a recommended workaround. Some teams have had better provisioning success with n2d compared to n2.

# gcloud — update node pool machine type

gcloud beta container node-pools update YOUR-NODEPOOL-NAME \
  --project      YOUR-PROJECT-ID \
  --cluster      YOUR-CLUSTER-NAME \
  --region       us-central1 \
  --machine-type n2d-highmem-16

Plan for ~30 minutes of node pool downtime. Validate on a non-production cluster first.

• Alert on pod pending time — not just job failure
  This failure mode presents as slowness, not an error. Set a Kubernetes-level alert on pod pending duration.

# kubectl — find pods stuck in Pending

# List all Pending pods across namespaces
kubectl get pods -A --field-selector=status.phase=Pending

# Inspect a specific pod — look in Events for the root cause
kubectl describe pod <pod-name> -n <namespace>
# GCE out of resources  →  capacity problem (this article)
# Insufficient cpu/memory  →  resource request too large

• The Long-Term Answer: Migrate to C4
  The C4 machine series — powered by Intel Emerald Rapids processors and Google's Titanium offload chips — is Google's current-generation compute offering, and no documented community reports of ZONE_RESOURCE_POOL_EXHAUSTED for C4 have been found as of August 2026.

  

💡 Migration approach: Create a new C4 node pool alongside your existing n2 pool, validate your workloads, then drain and delete the old one. This avoids any single-pool downtime and lets you roll back cleanly.

Key Takeaways

• The GCP status page is not your monitoring ZONE_RESOURCE_POOL_EXHAUSTED does not appear as a GCP incident. Monitor your cluster directly — pod pending duration is a more reliable early-warning signal.

• Minimum node count zero is a risk for production batch
  For jobs with SLAs, maintain a warm baseline that can absorb workloads without needing GCP to provision new hardware immediately.

• Multi-zone helps — regional shortages need a different response
  When the shortage spans all zones, machine type diversity or capacity reservations are the only effective responses.

• Know your machine type fallback before you need it
  Have a tested alternative ready in your runbook. The wrong time to figure it out is during a production incident at 3 AM.

The frustrating reality of ZONE_RESOURCE_POOL_EXHAUSTED is that it sits in a grey zone: widespread enough to affect multiple teams across months, specific enough that Google doesn't surface it as a regional incident. The community has been documenting it since April. The status page has stayed green throughout.

If you're running SLA-critical workloads in us-central1 on n2 — especially with a minimum node count of zero — the fixes above are worth reviewing this week, not after the next unexplained multi-hour delay.

Sources & Further Reading

1. Sudden VM Unavailability in us-central-1 — Google Developer Community

2. Shortage of Compute Engine Resource in us-central-1 — Google Developer Forums

3. Troubleshooting Resource Availability Errors — Google Cloud Docs

4. C4 Machine Series Deep Dive — Google Cloud Blog

5. Troubleshoot GKE Standard Node Pools — Google Cloud Docs

The interesting part isn’t just that Claude had an outage.

The interesting part is how much this kind of outage matters now.

A few years ago, an AI chatbot going down would have been annoying.

Today, it can stop someone from writing code, analyzing a document, finishing a project, or even running part of a business.

And that’s a pretty big change.

So, What Actually Happened?

Reports started appearing Monday as users ran into problems with Claude.

Anthropic’s own incident updates described the problem as “elevated errors” affecting multiple models. The affected services included models such as Claude Mythos 5, Fable 5, Opus 5 and other Claude models. Anthropic said it had identified the cause and was working on a fix.

Users were also reporting problems with Claude Code and other parts of Anthropic’s ecosystem.

That’s important because Claude isn’t just a chatbot anymore.

For many developers, Claude Code is part of the development workflow itself.

When that stops working, it’s not the same as losing access to a website for a few minutes.

It’s closer to a developer losing one of their tools in the middle of a workday.

And Then I Started Thinking About Something Else

We’ve become incredibly comfortable with AI.

Maybe too comfortable.

Think about your own workflow.

You might open Claude to write something.

Then use it to review code.

Then ask it to explain an error.

Then use it to summarize a document.

Then ask it to generate another piece of code.

At some point, you stop thinking of AI as a tool you occasionally use.

It becomes part of the workflow.

And that’s where outages become interesting.

Because when Claude disappears, even temporarily, you suddenly realize how much of your work has quietly moved onto someone else’s infrastructure.

AI Is Becoming Infrastructure

This is probably the bigger story here.

We’re used to thinking about AI in terms of intelligence.

Which model is smarter?

Which one writes better code?

Which one reasons better?

Which one gives better answers?

But as AI becomes embedded in products and businesses, another question becomes just as important:

Can I depend on it being there when I need it?

That’s a very different question.

A model can be incredibly intelligent and still be a terrible dependency if your application completely breaks when the model has an outage.

This is something the cloud industry learned years ago.

And AI developers are now learning the same lesson.

One AI Provider Shouldn’t Be Your Entire Architecture

If your application depends entirely on one AI provider, an outage can become your outage.

That’s a dangerous position to be in.

A more resilient setup might look something like this:

Primary model → fallback model → cached response → graceful degradation

If Claude works, great.

If Claude doesn’t work, the application can switch to another model.

If neither is available, maybe the user can still continue using the non-AI parts of the product.

It doesn’t have to be complicated.

But you have to think about it before the outage happens.

This Is Especially Important for AI Coding

Claude’s developer tools make this even more interesting.

A growing number of developers are using AI not simply to autocomplete a line of code, but to work through entire tasks.

Give the agent a feature.

Ask it to inspect the repository.

Let it modify files.

Run tests.

Fix errors.

Repeat.

That’s a completely different relationship with an AI model.

The AI isn’t just helping you type.

It’s participating in the development process.

So when the service goes down, the impact isn’t necessarily:

“I can’t ask my chatbot a question.”

It can be:

“My development workflow just stopped.”

That’s a much bigger deal.

And This Isn’t Happening in Isolation

Anthropic’s own status history shows several incidents involving Claude models and services during August.

The company recorded issues affecting multiple models on August 12, August 13, August 17, August 18, August 19 and August 20, along with broader service disruptions affecting Claude services on other dates.

That doesn’t automatically mean Claude is unreliable.

Large-scale infrastructure is difficult to operate, especially when demand and system complexity keep increasing.

But it does highlight something that every serious AI user should start thinking about:

Reliability is becoming a feature. Not an afterthought.

The “Best Model” Might Not Be the Best Model

This is where I think the AI conversation needs to evolve.

We spend an enormous amount of time comparing models.

Model A beats Model B on a benchmark.

Model B writes better code.

Model C has better reasoning.

But imagine two models:

Model A: slightly better, but occasionally unavailable.

Model B: slightly worse, but your application can automatically fail over to it.

For a production application, Model B might actually be the better choice.

Because users don’t care about your benchmark score when your application returns an error.

They care whether it works.

What Should Developers Do?

If you’re building something serious with AI, today’s outage is a good reminder to check a few things.

Do you have a fallback model?

Do you retry failed requests properly?

Are you monitoring API errors?

Can your application continue working without AI for a while?

Are you storing useful results so you don’t have to regenerate everything?

And perhaps most importantly:

Do you know what happens when your AI provider goes offline?

If the answer is “everything breaks,” that’s something worth fixing.

There’s a Bigger Shift Happening

AI companies used to compete primarily on model capability.

Now they’re increasingly competing on something that looks much more like traditional infrastructure:

• uptime

• latency

• reliability

• capacity

• enterprise controls

• predictable APIs

• developer experience

That’s actually a sign of maturity.

When people start complaining about uptime instead of asking whether the technology is useful, it means the technology has become important enough to depend on.

And Claude is definitely at that stage.

So, Is Claude “Down”?

At the time of writing, the situation is better described as a service disruption/elevated-error incident earlier today, rather than Claude being universally offline.

Anthropic’s current status page shows its major Claude services as operational, although reports earlier in the day documented significant problems.

So if Claude is working for you now, that’s consistent with the latest status information.

But the outage itself is still worth paying attention to.

Because the real story isn’t that Claude had a bad day.

The real story is that AI has become important enough for its downtime to become a serious technology story.

The Takeaway

I’m not particularly interested in arguing that Claude is better or worse because of today’s outage.

Every major technology platform will eventually have failures.

The lesson is somewhere else.

If you’re a casual AI user, an outage is an inconvenience.

If you’re a developer, it’s an architecture problem.

And if you’re a business that has built critical workflows around AI, it’s a risk-management problem.

AI isn’t just a chatbot anymore.

It’s becoming part of the infrastructure we build on.

And once something becomes infrastructure, reliability matters just as much as intelligence.

That’s probably the most important thing today’s Claude outage reminds us.

Sources

1. Anthropic — Claude Status: Official incident history and current service status. Claude Status

2. Android Authority — Claude outage report: Coverage of today’s August 24 disruption and Anthropic’s response. Android Authority report

3. Notebookcheck — Claude outage: Reporting on elevated errors affecting multiple Claude models and Claude Code. Notebookcheck report

4. Economic Times — Claude disruption: Reporting on user complaints and disruption across Claude, Claude Code and related services. Economic Times report

If you’re dealing with something like this at work — or you have a war story of your own — I’m on LinkedIn and I actually respond.

I publish one production incident breakdown every week. Subscribe free if you want it directly in your inbox — no tutorials, no fluff:

Let’s Stay in Touch 🪂 Connect with us on: LinkedIn, Instagram, Medium

It doesn't matter if it's SQL Server, IIS, a background agent, or a custom app — when a Windows service goes down, the investigation is always the same five moves. Learn the pattern once, apply it to anything.

🚫 Don't restart first. Restarting a service without knowing why it stopped can hide a real problem — a failing update, an exhausted host, or an automation script that will stop it again in the next cycle. Spend 5 minutes on the evidence first.

Step 01 — Confirm Current State

Your first question isn't "why did it stop?" — it's "is it still stopped?" Monitoring systems have polling intervals; by the time an alert reaches you, the service may have already recovered. Establish ground truth before taking any action.

Get-Service "ServiceName1", "ServiceName2" |
    Select-Object Name, Status, StartType

Two things matter: Status (is it actually down right now?) and StartType (is it still set to Automatic? OS updates occasionally reset this).

🔵 SQL Server example:

Get-Service MSSQLSERVER, SQLSERVERAGENT, MsDtsServer130 |
    Select-Object Name, Status, StartType

All three came back Running / Automatic — already self-recovered before anyone looked. This told us the alert was valid but not actionable, and shifted focus to understanding root cause.

Step 02 — Map Service Transitions with Event ID 7036

Windows logs every service state change — stopped, running, paused — as Event ID 7036 in the System log. Pull the last 20 transitions and sort them chronologically. This gives you the precise stop-and-start sequence and exact timestamps.

Get-WinEvent -FilterHashtable @{
    LogName='System'; Id=7036
} | Where-Object {
    $_.Message -match 'Your Service Name'
} | Select-Object TimeCreated, Message |
    Sort-Object TimeCreated -Descending |
    Select-Object -First 20

Read the output bottom-up (oldest first). Look for the last time it went from running → stopped, and whether it came back on its own or stayed down. Multiple stop/start cycles in quick succession often mean a crash-and-restart loop.

🔵 SQL Server example:

Get-WinEvent -FilterHashtable @{LogName='System';Id=7036} |
    Where-Object {$_.Message -match
      'SQL Server Agent \(MSSQLSERVER\)|SQL Server \(MSSQLSERVER\)|SQL Server Integration Services 13.0'} |
    Select-Object TimeCreated, Message |
    Sort-Object TimeCreated -Descending |
    Select-Object -First 20

The output showed all three services stopped at 10:20 AM IST and returned to running at 10:27 AM IST — a 6-minute outage that overlapped exactly with the alert timestamp.

Step 03 — Look for System-Level Triggers

A service rarely stops in isolation. Before blaming the application, check whether the host itself caused the stop. Four event IDs tell this story:

Get-WinEvent -FilterHashtable @{
    LogName='System'; Id=1074,6005,6006,6008
} | Select-Object TimeCreated, Id, Message |
    Sort-Object TimeCreated -Descending |
    Select-Object -First 20

# For recent 1074s with full detail:
Get-WinEvent -FilterHashtable @{LogName='System';Id=1074} |
    Where-Object {$_.TimeCreated -gt (Get-Date).AddHours(-24)} |
    Format-List TimeCreated, Message

🔵 SQL Server example:

Event 1074 revealed two reboots in the same maintenance window. The first was triggered by o9ansibleuser via Ansible. The second was chained by TrustedInstaller.exe with reason "Operating System: Upgrade (Planned)" — Windows Update had quietly queued a second reboot on top of the Ansible reboot. Without checking Event 1074, this would have looked like an unexplained outage.

💡 If you find a 6008 (dirty shutdown) near your service stop, the host itself crashed — investigate the host before the service.

Step 04 — Read the Application's Own Logs

Windows Event logs tell you when the service stopped. The application's own logs tell you why. Every serious Windows service writes its own log — find it and read the entries immediately before the service stopped.

# Common log locations:
# SQL Server → C:\Program Files\Microsoft SQL Server\MSSQL{ver}.{instance}\MSSQL\Log\ERRORLOG
# IIS        → C:\inetpub\logs\LogFiles\W3SVC1\

# For services that write to Windows Application log:
Get-WinEvent -FilterHashtable @{LogName='Application';Level=1,2} |
    Where-Object {$_.ProviderName -match 'YourService'} |
    Select-Object TimeCreated, Message |
    Sort-Object TimeCreated -Descending |
    Select-Object -First 20

You're looking for one of three signatures:

• Error/exception before stop → crash

• Clean shutdown message → planned stop

• Nothing unusual → external trigger (reboot killed the process)

🔵 SQL Server example — ERRORLOG:

SQL Server rotates its log on every restart. Find the file that covers your outage window by LastWriteTime:

# Find the right log file
Get-ChildItem "C:\Program Files\Microsoft SQL Server\MSSQL13.MSSQLSERVER\MSSQL\Log\ERRORLOG*" |
    Select-Object Name, LastWriteTime, Length |
    Sort-Object LastWriteTime

# Read the last 50 lines of the relevant file
Get-Content "...\MSSQL\Log\ERRORLOG.4" | Select-Object -Last 50

The critical line at the tail of our log:

SQL Server is terminating in response to a 'stop' request from Service Control Manager.

That's a clean stop — no crash, no error. Case closed on fault; focus shifts to who stopped it and why.

⚠️ SQL ERRORLOG uses local server time, not UTC. Convert your alert timestamp before searching.

Step 05 — Classify the Stop, Then Decide

By now you have enough to classify into one of three categories:

🔴 Crash

• Exceptions/errors in app log immediately before stop

• Event 6008 (dirty shutdown) present

• Dump files in the app's log directory

• No clean stop message

Response: Don't just restart — find the root cause first. Check memory (Get-Process), disk (Get-PSDrive), dump files. Restarting a crashing service without fixing the cause fails again.

🟢 Clean Stop

• "Stop request from SCM" in app log

• Config/agent disabled just before stop

• No preceding errors — manual or scripted

Response: Safe to restart, but find out who stopped it. Was it automation? Is it expected to auto-restart?

🟡 External Trigger

• Event 1074 near the stop time

• 6005/6006 reboot pair visible

• Ansible, Windows Update, manual reboot

Response: The service is a passenger — the host was the cause. Verify the reboot was planned, confirm all services are back, close the ticket.

Going Further: From Reactive to Proactive

🔕 Maintenance Windows — Before any planned reboot, open a monitoring suppression. Automate it as a pre-task in your Ansible playbook, not a manual step someone might forget.

📋 Parse App Logs Continuously — Ship application logs to a central store. Alert on terminating, severity 20+, exception before a service hits the floor.

🧠 Watch Memory Before It Matters — Many crashes are preceded by hours of memory pressure. A service paging to disk for 4 hours before crashing gave you warning you didn't act on.

🤖 Tag Automation in Monitoring — Have Ansible write a comment to your alert system before rebooting. A ticket that says "Ansible reboot — expected" closes in seconds. One without context takes 15 minutes.

📅 Track TrustedInstaller Reboots — Windows Update's chained reboots are invisible unless you watch Event 1074. Add a post-patch check that logs TrustedInstaller-initiated reboots to your CMDB.

📈 Trend Restart Frequency — A service that restarts once a quarter is healthy. One that restarts three times in a week has a problem. Track restart counts over time — the trend tells you before a crash does.

Quick Reference: The Diagnostic Playbook

Save this to your runbook.

# 1 — Current state
Get-Service "ServiceA","ServiceB" | Select-Object Name,Status,StartType

# 2 — Service transitions
Get-WinEvent -FilterHashtable @{LogName='System';Id=7036} |
    Where-Object {$_.Message -match 'YourServiceName'} |
    Select-Object TimeCreated,Message | Sort-Object TimeCreated -Descending |
    Select-Object -First 20

# 3 — Shutdown events
Get-WinEvent -FilterHashtable @{LogName='System';Id=1074,6005,6006,6008} |
    Select-Object TimeCreated,Id,Message | Sort-Object TimeCreated -Descending |
    Select-Object -First 20

# 4 — App event log (errors/criticals)
Get-WinEvent -FilterHashtable @{LogName='Application';Level=1,2} |
    Where-Object {$_.ProviderName -match 'YourService'} |
    Select-Object TimeCreated,Message | Select-Object -First 20

# 5 — Server uptime
(Get-Date) - (Get-CimInstance Win32_OperatingSystem).LastBootUpTime

# SQL Server bonus — find the right ERRORLOG
Get-ChildItem "C:\Program Files\Microsoft SQL Server\MSSQL13.MSSQLSERVER\MSSQL\Log\ERRORLOG*" |
    Select-Object Name,LastWriteTime | Sort-Object LastWriteTime

💡 The three root causes — crash, clean stop, external trigger — each have a different recovery path. Classify before you act, and you'll never restart a service into the same problem twice.

If you're dealing with something like this at work — or you have a war story of your own — I'm on LinkedIn and I actually respond.

I publish one production incident breakdown every week — real commands, real dead ends, real fix. Follow on Medium if you want the next one in your feed: prateeksrivastav598.medium.com

Nobody plans to manage Kubernetes on three clouds at the same time. It happens gradually — an acquisition brings an Azure footprint, a new business unit commits to GCP, and you already run EKS. Then one morning a page fires and you're staring at three terminal windows, three different cluster behaviours, one very angry incident bridge.

I've spent the better part of two years in exactly that situation. What follows isn't a comparison chart you can find on any vendor blog. It's the stuff I learned the hard way — the silent differences that don't show up until something breaks in production at 2 AM.

{% callout type="warning" %} The Incident That Started It All We deployed the same Helm chart — identical values, same image tag — to EKS, AKS, and GKE within the same 20-minute release window. EKS came up healthy. AKS came up healthy. GKE's pods started, ran for about 90 seconds, then began OOMKilling in a loop. Same YAML. Same limits. Three different outcomes. Four hours to find why.

1. Networking: The Biggest Lie — "Pods Are Just Pods"

This is where most multi-cloud pain originates. Every Kubernetes networking tutorial tells you that pods get their own IPs and can talk to each other. What it doesn't tell you is that where those IPs come from completely changes your blast radius, your subnet planning, your security posture, and the failure modes you'll face at 3 AM.

EKS: Your pods eat your VPC subnet IPs

EKS uses the AWS VPC CNI by default. Every pod gets a real, routable IP from your VPC subnet. This sounds great until you have a /24 subnet and someone deploys a service with 200 replicas. The error you get isn't "out of IPs" — it's an ENI attachment timeout. The CNI pre-warms ENIs and exhausts capacity before you hit the IP limit.

The harder gotcha: each EC2 instance type has a max ENI count × max IPs per ENI cap. A t3.medium can hold 3 ENIs × 6 IPs = 18 pod IPs max, minus 1 for the node. Run 20 pods per node and you'll see Pending pods on nodes with plenty of CPU and memory — the constraint is invisible to the default scheduler.

# Check ENI and IP capacity on a node
kubectl describe node <node-name> | grep -A5 "Allocatable"

# Check aws-node daemonset for ENI attachment errors
kubectl logs -n kube-system -l k8s-app=aws-node --tail=50 | grep -i "err\|fail\|exhaust"

Fix: Enable VPC CNI prefix delegation — one ENI prefix gives you 16 IPs instead of 1.

AKS: Azure CNI pre-allocates IPs whether you like it or not

AKS with Azure CNI reserves IPs per node based on --max-pods (default: 30) — even before any pod runs. A 100-node cluster pre-consumes 3,000 VNet IPs. Plan your address space first. If you're IP-constrained, use Azure CNI Overlay mode.

GKE: Clean until you need to peer networks

GKE uses alias IP ranges, isolated from your main VPC. It's the cleanest story — until you need to peer your GKE VPC with an on-premises network and your pod CIDR overlaps your on-prem range, because you let GKE auto-assign ranges six months ago. Always explicitly define secondary ranges on GKE.

2. IAM + RBAC: Three Identity Models, One Misconfiguration Away from Disaster

EKS: The IRSA silent fallback trap

If your IRSA annotation has a typo, the assume-role call silently fails and the pod falls through to the node's instance profile — often over-permissive. You won't see an error. Verify assumed identity from inside the pod:

kubectl exec -it <pod-name> -- aws sts get-caller-identity

AKS: Workload Identity federation is fragile on setup

Three pieces must align exactly: Managed Identity, federated credential (with the exact OIDC issuer URL — including trailing slashes), and the Kubernetes service account annotation. One mismatch = silent 401.

GKE: Node rotation breaks implicit identity

Enabling Workload Identity on a node pool removes the node's default Google service account. Any workload relying on implicit node identity breaks on the next node rotation. Audit before enabling.

3. Memory + cgroups: Why the Same Container OOMKilled on GKE but Not on EKS

GKE had moved to cgroup v2 on newer node images. EKS and AKS were still on cgroup v1. Our Java service used JVM ergonomics to auto-detect heap size.

• cgroup v1: JVM reads from /sys/fs/cgroup/memory/memory.limit_in_bytes

• cgroup v2: JVM reads from /sys/fs/cgroup/memory.max

Our old JVM didn't handle cgroup v2 — it read host memory (64GB) instead of the container limit (2GB), allocated an 8GB heap into a 2GB container, and OOMKilled every 90 seconds.

# Check which cgroup version the container sees
kubectl exec -it <pod-name> -- cat /proc/1/cgroup
# cgroup v1: "12:memory:/kubepods/..."
# cgroup v2: single line "0::/"

# Check what heap the JVM allocated
kubectl exec -it <pod-name> -- java -XshowSettings:all -version 2>&1 | grep -i heap

Fix: JDK 15+ for native cgroup v2 support. Always explicitly set -Xmx and -Xms. Never trust JVM ergonomics in a container without verifying cgroup version on the target nodes.

4. Storage: The AZ-Pinning Trap

Block storage is AZ-specific on every cloud. Your PVC and its pod must be in the same AZ. When Cluster Autoscaler scales nodes in a different AZ, pods stay Pending — and the on-call engineer stares at CPU/memory graphs seeing nothing wrong.

# The event buried in pod describe that tips you off
kubectl describe pod <pending-pod> | grep "had volume node affinity conflict"

Fix: Use volumeBindingMode: WaitForFirstConsumer on all storage classes. This delays PV creation until the pod schedules, guaranteeing same-AZ placement.

5. Load Balancers: Same Service, Three Different Outcomes

Each cloud's controller uses completely different annotation namespaces:

annotations:
  # EKS — get NLB instead of legacy Classic ELB
  service.beta.kubernetes.io/aws-load-balancer-type: "external"
  service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"

  # AKS — internal (private) load balancer
  service.beta.kubernetes.io/azure-load-balancer-internal: "true"

  # GKE — completely different namespace for internal LB
  networking.gke.io/load-balancer-type: "Internal"

We once deployed a chart to GKE with only AWS annotations. GKE ignored them and created a public Load Balancer. An internal API was internet-reachable for 40 minutes before SolarWinds monitoring caught it.

Rule: Validate LB configuration on all three clouds in CI.

6. Autoscaling: Cluster Autoscaler Is Not the Same Everywhere

• EKS: Scale-up is 4–8 minutes minimum (ASG → EC2 launch → join → register → schedule)

• GKE NAP: Faster, but creates node pools with unexpected taints — pods without matching tolerations won't schedule even after "scale-up"

The pattern that works: Use KEDA for pod-level scaling (seconds), Cluster Autoscaler only as a last resort for node-level scale (minutes). Scale pods first, nodes only when necessary.

What I'd Tell Myself Before Starting All Over

Multi-cloud Kubernetes isn't twice the work — it's about six times the work, because every subtle difference compounds.

1. Standardise your observability first. Same dashboards, same alert expressions, same log structure across all three clouds. Muscle memory beats translation at 2 AM.

2. Document differences, not just similarities. Runbooks should say "on EKS check X; on AKS check Y; on GKE check Z."

3. Test Helm charts on all three before every major release. 20 minutes of CI is cheaper than one 4-hour incident.

4. Never trust "cloud-agnostic" on the label. Especially networking, storage, and identity — they break silently.

The day you stop being surprised by cloud differences is the day you've actually become a multi-cloud SRE. Everything before that is just surviving the learning curve.

🙏 Thank You for Reading!

Your time and attention mean a lot to us. We greatly appreciate your engagement with our content and would love to hear your thoughts and ideas. Let’s stay connected and continue our journey through the digital world together! 🌐📚💡

Let’s Stay in Touch 🪂 Connect with us on: LinkedIn, Instagram, Medium

⚡ TL;DR: An EKS worker node running 30+ pods crashed at 08:43 IST on Aug 16. EC2 Auto Scaling didn't act. EKS Node Auto Repair was never enabled. Cluster Autoscaler had no headroom (Min=Max=6). Our memory alert fired 82 seconds after the node was already dead. We fixed it manually at 12:30 IST — 4 hours later. Every fix is one or two CLI commands. Here's the full breakdown.

When an EKS worker node goes down, Kubernetes is supposed to handle it. That's the whole pitch — self-healing infrastructure. Pods evict, workloads reschedule, the cluster heals itself.

In a recent incident, a node running 30+ pods crashed at 08:43 IST. None of the four auto-recovery mechanisms we had in place did anything. The node sat NotReady for almost 4 hours. We discovered it 82 seconds after it died — from a memory alert that fired when the node was already gone — and fixed it manually at 12:30 IST.

The interesting part wasn't that the node crashed. It was that four different layers of protection failed for four completely different reasons.

Here's exactly what failed and why.

The Node That Looked Healthy Until It Didn't

ip-10---*.ec2.internal had been running since Jun 28. From Kubernetes' perspective, it was Ready. CPU requests at ~97%. Memory requests at ~69%. Nothing alarming.

The actual picture was very different:

Allocated resources:
  Resource    Requests       Limits
  cpu         ~97%           ~545%
  memory      ~69%           ~358%

358% memory limit overcommit. Kubernetes scheduled 37 pods there because requests showed 69% — but if those pods pushed toward their limits, the node had no chance. The requests were lies; the limits were the truth. And the truth said this node was 3.5x overloaded.

Underneath the Ready status, the containerd runtime had been in a degraded state since Jul 20, when the OOM killer had hard-killed a MongoDB process. The kernel had logged two containerd-shim deadlocks (Jul 1 and Jul 29) that the node had somehow absorbed. Neither showed up anywhere in monitoring. The control plane saw nothing wrong.

On Aug 16 at 08:22 IST, a service responsible for processing audit events started leaking memory. ELK showed node memory at 81.6% at 08:22:57 IST. By 08:43 IST the service had grown from 2.0GB to 6.2GB — a 4.2GB spike in 21 minutes — pushing the node from ~81% to ~96%.

The memory pressure triggered a third containerd-shim deadlock on the already-corrupted runtime. This time it was fatal. At 08:43:22 IST, SSM lost contact. Kubelet posted its last heartbeat. At 08:45:10, the node went NotReady.

Now watch how every auto-recovery mechanism missed it.

Failure #1 — EC2 Auto Scaling Health Check

The ASG protecting this nodegroup was configured with EC2-level health checks:

HealthCheckType: EC2 Min: 6 | Max: 6 | Desired: 6

EC2 health checks have exactly one question: is the VM powered on?

aws ec2 describe-instance-status \
  --instance-ids <id> --region <region> \
  --query 'InstanceStatuses[].[InstanceState.Name,SystemStatus.Status,InstanceStatus.Status]' \
  --output table

+---------+------+------+ 
| running| ok | ok | 
+---------+------+------+

The instance was running. System status: ok. Instance status: ok. The AWS hypervisor was satisfied. The deadlock inside containerd was completely invisible at the hypervisor layer — the kernel was alive, the VM was up, the NIC was responding to health probes.

EC2 health check never triggered. No replacement instance was launched.

The lesson: EC2 health checks only detect hardware failure or VM termination. They cannot detect OS-level hangs, kubelet death, containerd deadlocks, or any software failure that doesn't take down the underlying VM. For Kubernetes node health, EC2 checks are nearly useless.

Failure #2 — EKS Node Auto Repair

EKS Node Auto Repair is designed exactly for this scenario. When a node stays NotReady for a defined period, it automatically cordons, drains, and replaces it. No manual intervention needed.

aws eks describe-nodegroup \
  --cluster-name <name> \
  --nodegroup-name <nodegroup_name> \
  --query 'nodegroup.nodeRepairConfig' 

null

null. Never configured. Never enabled.

This feature was available. We had just never set it up. The node sat NotReady for nearly 4 hours while the feature that would have replaced it in 10 minutes was turned off by default.

The lesson: EKS Node Auto Repair is not enabled by default. Check every nodegroup right now with the command above. If you get null, you have this gap.

Failure #3 — Cluster Autoscaler

The cluster autoscaler was running. It's supposed to handle exactly this kind of situation. But our configuration made it powerless:

 Min: 6 | Max: 6 | Desired: 6

Min equals Max. The autoscaler has no headroom to operate. To replace a broken node, it needs to launch a new one first (going to 7, violating Max), drain the broken node, then terminate it (back to 6). With Max=6, it couldn't even start that sequence.

The autoscaler logs confirmed it saw the problem — and couldn't act:

I0817 12:17:57 pre_filtering_processor.go:67] Skipping ip----.ec2.internal — node group min size reached (current: 6, min: 6)

It knew the node was bad. It had no authority to fix it.

There was a second problem: the cluster-autoscaler pod itself was running on the broken node, stuck in Terminating state alongside 25+ other pods.

kube-system cluster-autoscaler-5********8 1/1 Terminating 0 6d5h

The thing responsible for replacing the broken node was stuck on the broken node.

The lesson: Min=Max is a zero-headroom configuration that paralyzes automated recovery. Set Max to at least Min+2 on every nodegroup.

Failure #4 — Memory Alert (Fired Too Late to Matter)

We had a memory alert configured — running on an 11-minute cron interval, watching for node memory usage above 95%.

ELK showed the node at 81.6% for hours before the incident:

IST timestamp Node memory % 08:22:57 81.6% 08:33:57 81.6% 08:44:57 80.6% ← node already down, stale value

The 4.2GB memory spike happened entirely between two check cycles. At 08:22 the node was fine. At 08:44 the check returned stale data because the node was already gone.

The alert fired at 08:44 IST. The node went unreachable at 08:43:22 IST. The alert was 82 seconds behind.

This is a structural problem: the metric source is the node itself. When the node dies, metrics stop. If the crash happens between alert evaluation cycles, you miss it entirely. The faster a node crashes, the more likely it dies between evaluations.

The lesson: Metric-based node alerts have a fundamental blind spot. If the node is the source of the metric, the metric disappears when the node dies.

What We're Fixing

1. Enable EKS Node Auto Repair (do this first)

   aws eks update-nodegroup-config
   --cluster-name
   --nodegroup-name <nodegroup_name>
   --node-repair-config enabled=true
   --region us-east-1
   

   When a node stays NotReady for the configured threshold, EKS automatically cordons, drains, and replaces it. The control plane makes this decision — it doesn't depend on the broken node reporting its own failure.

2. Set Max > Min on Every ASG

   aws autoscaling update-auto-scaling-group
   --auto-scaling-group-name
   --min-size 6
   --max-size 8
   --region us-east-1
   

   Max needs to be at least Min+2 to give the autoscaler room to launch a replacement before removing the broken instance.

3. Switch to ELB Health Checks

   aws autoscaling update-auto-scaling-group
   --auto-scaling-group-name
   --health-check-type ELB
   --health-check-grace-period 300
   --region us-east-1
   

   ELB health checks evaluate actual application-layer response, not just hypervisor state.

4. Replace Metric-Based Node Alert with Condition-Based Alert

   # This fires even when the node is completely dead.
   # kube-state-metrics reads from the API server, not from the node itself.
   - alert: NodeNotReady
     expr: kube_node_status_condition{condition="Ready",status="true"} == 0
     for: 90s
     labels:
       severity: critical
     annotations:
       summary: "Node {{ $labels.node }} NotReady for >90s — EKS Auto Repair should engage"
   

   kube-state-metrics watches the Kubernetes API server — not the node. It fires even when the node is completely unreachable.

5. Enforce Memory Overcommit Limits

   apiVersion: v1
   kind: LimitRange
   metadata:
     name: memory-ratio-limit
     namespace: your-namespace
   spec:
     limits:
     - type: Container
       maxLimitRequestRatio:
         memory: "4"   # limit cannot exceed 4x request
   

   This forces the actual memory ceiling to stay within a predictable range of what the scheduler sees. A pod with a 1Gi request cannot have a 16Gi limit.

The Broader Problem

This wasn't an unusual incident. A node accumulated damage silently for 46 days. Nothing alerted. Nothing flagged the degraded containerd state. The node reported Ready to the control plane while internally the runtime was already compromised.

When the fatal event came — a memory spike from an unrelated service — four recovery mechanisms failed in different ways:

• EC2 health checks were asking the wrong question (is the VM up?)

• EKS Auto Repair was never turned on

• The autoscaler had no room to maneuver (Min=Max)

• The metric-based alert fired 82 seconds after the node was already dead

Any one of these gaps would have slowed recovery. All four together meant 4 hours of a NotReady node and a manual reboot at noon.

The fixes are each one or two commands. None of them are complicated. They just require knowing the gaps exist.

Check Your Cluster Right Now. Run this on every nodegroup:

aws eks describe-nodegroup \
  --cluster-name <your-cluster> \
  --nodegroup-name <your-nodegroup> \
  --query 'nodegroup.nodeRepairConfig'

If you get null — you have this gap. Fix it before the next 2 AM alert.

The names, cluster/node identifiers, timestamps, instance IDs, pod counts, resource values, and other incident-specific details in this article have been intentionally changed or fictionalized for privacy and security. The failure pattern and technical lessons remain representative of the original incident.

🙏 Thank You for Reading!

Your time and attention mean a lot to us. We greatly appreciate your engagement with our content and would love to hear your thoughts and ideas. Let’s stay connected and continue our journey through the digital world together! 🌐📚💡

Let’s Stay in Touch 🪂 Connect with us on: LinkedIn, Instagram, Medium