The Empty DNS Record That Took Down 70 AWS Services for 14 Hours

We operate one of the largest cloud infrastructure platforms in the world, running hundreds of interdependent services across dozens of regions. Our DynamoDB service in us-east-1 — Northern Virginia — is the backbone of an enormous number of internal AWS systems, not just customer workloads. EC2 instance lifecycle management, network configuration state, Lambda, ECS, EKS, Fargate — all of them quietly depend on DynamoDB for state coordination. We have a fully automated DNS management system to keep regional endpoints healthy. It consists of two decoupled components: a DNS Planner that monitors load balancer health and generates DNS update plans, and a DNS Enactor that applies those plans via Route 53. We run three parallel Enactors, one per availability zone, for redundancy. The design assumption was that eventual consistency across Enactors would handle any timing gaps. We had never seen this assumption fail — until 11:48 PM PDT on October 19, 2025.

At 11:48 PM PDT on October 19, our monitoring systems detected a spike in DynamoDB API error rates across us-east-1. The first thing on-call engineers saw was a flood of DNS resolution failures against `dynamodb.us-east-1.amazonaws.com`. The DNS record was not slow — it was empty. No IP addresses. The endpoint had simply vanished from Route 53. Here is what had happened: DNS Enactor #1 was processing a routine DNS update plan but had fallen behind due to unusually high processing delays — the exact cause of that slowdown remains unclear from the postmortem. While Enactor #1 was working through its backlog, the DNS Planner, unaware of the delay, continued generating new plans at its normal pace. DNS Enactor #2, which was processing plans in parallel, moved through the queue quickly and then ran its stale-plan cleanup routine. It identified the plan still in progress by Enactor #1 as "old" and deleted it — along with all the IP addresses it was supposed to preserve. The cleanup routine's logic assumed any plan older than a threshold was safe to remove. It was not designed to check whether another Enactor was actively applying that plan. The result: a clean, authoritative, empty DNS record for DynamoDB's entire us-east-1 regional endpoint was published to Route 53 and propagated globally. Every system that tried to resolve the DynamoDB endpoint — internal or external — got back nothing. The cascade started immediately. The DropletWorkflow Manager (DWFM), which maintains state leases for the physical servers running EC2 instances, depends on DynamoDB to coordinate instance lifecycle transitions. With DNS gone, DWFM could not write or read lease state. Any EC2 operation requiring a state change — launch, stop, terminate, resize — began failing with "insufficient capacity errors." This was a lie; there was plenty of capacity. The instances just could not prove their state to DWFM. As DynamoDB recovered at around 2:25 AM PDT, DWFM did what any sensible system would do when reconnecting after an outage: it tried to re-establish all of its leases across the entire EC2 fleet simultaneously. This created a thundering herd. DWFM entered what engineers described as "congestive collapse" — so many lease re-establishment requests were in flight that timeouts began cascading, making the congestion worse. The system that was supposed to heal itself was now actively preventing recovery. The Network Manager, which handles routing configuration for new EC2 instances, had accumulated a large backlog of pending network state changes during the outage. As EC2 instances started coming back, Network Manager was so congested that new instances were failing health checks because their network configuration had not propagated yet. Network Load Balancers, seeing unhealthy instances, removed them from rotation — making the recovery look like a second wave of failures. At peak, over 70 AWS services were impacted. Signal, Snapchat, Starbucks, Coinbase, Reddit, Ring, Amazon.com itself, Amazon Alexa, and Apple's streaming services all went down for tens of thousands of users.

Root cause: a race condition between two parallel DNS Enactors caused one to classify the other's in-progress plan as stale and delete it, publishing an empty DNS record for `dynamodb.us-east-1.amazonaws.com` to Route 53. The fix for DynamoDB itself was to manually restore the DNS records and disable the automated DNS management system globally while a safeguard preventing cleanup of actively-applied plans is implemented. Recovery from the cascade took significantly longer than DynamoDB recovery itself — approximately 11 additional hours — due to DWFM's thundering-herd re-establishment behavior and Network Manager's backlog. Engineers had to manually throttle DWFM request rates to break the congestive collapse cycle and gradually drain the network configuration backlog. No data was lost. EC2 instances that were already running were unaffected; only state transitions failed. Customers experienced launch failures, invocation failures in Lambda/ECS/EKS, and widespread DNS resolution errors.