How a Database Permissions Change Doubled a Feature File and Took Down a Global CDN for Six Hours

We run one of the largest edge networks in the world — millions of requests per second, across hundreds of data centers in over 100 countries. Our network sits between users and the websites they visit, handling DNS, CDN, DDoS mitigation, bot management, serverless compute, and zero-trust access. One of our core products is a Bot Management system that uses machine learning to distinguish legitimate traffic from bots. This system relies on a "feature file" — a list of ML features and their configurations — that gets compiled from a database and distributed to every machine across our global network. The feature file is generated by a pipeline that queries a database, transforms the results into the format our ML runtime expects, and then pushes the file out globally. The runtime has a hard limit on the number of features it can load. We had never hit that limit in production. The system had been stable for years.

On November 18, 2025 at approximately 11:05 UTC, a routine database permissions change was applied. The change was meant to grant additional access to a service account. What nobody anticipated was that the new permissions caused the query that generates the Bot Management feature file to return duplicate rows — specifically, the response now included all metadata from an internal schema (referred to internally as the "r0 schema"), effectively more than doubling the number of rows in the output. The pipeline dutifully compiled this bloated result into a new feature file and pushed it to every machine on our network. At 11:20 UTC, the first signs of trouble appeared in our internal monitoring: proxy processes across the network began failing. The Bot Management runtime attempted to load the feature file, found that it exceeded its hard-coded ML feature limit, and crashed. But it did not crash quietly. The proxy process that handles HTTP traffic reads the feature file at startup. When the feature file was invalid, the proxy could not initialize correctly, and HTTP requests started returning 500 errors. By 11:28 UTC, customer-facing HTTP errors were spiking globally. Users across the internet saw error pages when trying to reach any site behind our network. Our initial response was wrong. The symptoms — a sudden, global spike in errors across all regions simultaneously — looked like a hyper-volumetric DDoS attack. Our incident response team spent critical early minutes investigating attack vectors before ruling out external causes. This misdiagnosis cost us time. The cascade spread beyond HTTP proxying. Our serverless compute platform (Workers), our key-value store (KV), our zero-trust access gateway, our bot verification challenge system (Turnstile), and even our internal dashboard all depend on the same proxy layer or on services that route through it. When the proxies went down, these services went down with them. Engineers trying to access the dashboard to diagnose the issue found the dashboard itself was unreachable — a circular dependency that slowed response further. Once we correctly identified the feature file as the culprit, the fix was conceptually simple: stop the propagation of the bad file and replace it with the last known good version. But executing that fix across a global network of hundreds of data centers, while the tooling you normally use to push changes is itself partially broken, took significant coordination. [inferred] Break-glass access procedures that bypass the normal deployment path were invoked, but some of these had their own dependencies on affected systems, adding friction to the recovery.

Root cause: a database permissions change caused a feature file generation query to return duplicate rows by exposing additional schema metadata. This more than doubled the size of the Bot Management feature file. When propagated globally, the file exceeded a hard-coded ML feature count limit in the proxy runtime, causing proxy crashes and HTTP 500 errors across the entire network. The immediate fix was stopping propagation of the oversized feature file and replacing it with the previous version. Core HTTP traffic was restored by 14:30 UTC, approximately 3 hours and 10 minutes after initial impact. Full restoration of all dependent services — including serverless compute, key-value storage, access gateway, and the internal dashboard — took until 17:06 UTC, roughly 5 hours and 46 minutes total. No customer data was lost. The impact was availability, not integrity. An estimated 2.4 billion monthly active internet users were affected across thousands of websites and applications that route through the network. In the aftermath, the company declared "Code Orange" — a company-wide initiative called "Fail Small" that prioritized resilience engineering above all other work. The plan mandated phased rollouts for all configuration changes (not just code), modeled on the existing Health Mediated Deployment process, with staged validation and automatic rollback. It also addressed circular dependencies in incident response tooling and break-glass procedures.