Two Silent Consul Bugs That Took Down a Gaming Platform for 73 Hours

We run a gaming platform with 50 million daily active players, 18,000+ servers, and 170,000 containers. Our entire infrastructure — service discovery, container orchestration, secret management, session locking, and health checks — runs on a single HashiCorp stack: Consul, Nomad, and Vault. Consul is the spine everything else talks through. If Consul gets sick, everything gets sick. We'd been running this architecture for years without major incident. The week before the outage, we made two routine changes in preparation for an upcoming peak traffic event: we enabled Consul 1.10's new streaming feature on our traffic routing backend, and we increased our node count by 50%.

At 13:37 on a Thursday afternoon, Vault started showing performance degradation. One Consul server showed elevated CPU. Players were unaffected — at first. By 16:35 the online player count was at 50% of normal. Consul was in a cascade. We called in HashiCorp engineers. The symptom was clear: Consul KV write latency had climbed from a normal 30–300ms to a consistent 2–3 seconds. Writes were timing out. Raft log replication was lagging. But the cause was not clear at all. Our first instinct was hardware degradation — the servers were aging. We replaced hardware. The latency did not improve. It got worse. We spent two days on hardware. We replaced all Consul nodes with 128-core machines with faster NVMe SSDs. We reset the Consul cluster from a pre-outage snapshot and blocked all traffic with iptables to give it time to recover. Performance degraded again within hours. We reduced non-essential Consul usage, disabled health check frequency, scaled down to minimal instances. Each attempt bought hours before the same failure mode returned. Meanwhile, our monitoring was partially blind. The telemetry systems we relied on to diagnose Consul were themselves running through Consul. Circular dependency. We were flying with gauges that read the health of the thing that made the gauges work. On the third day we went deeper — OS-level metrics, kernel-level blocking analysis. We started seeing write contention: many goroutines blocking on the same shared Go channel in the streaming implementation. The streaming feature in Consul 1.10 used fewer channels than the old long-polling model, which should have been more efficient. But under our write load, a single shared channel became a lock bottleneck. And our new 128-core NUMA machines had made it worse — more CPU cores meant more concurrent goroutines competing for the same lock on shared memory. We had upgraded our way into deeper trouble. At 15:51 on day three we disabled the streaming feature globally. KV write latency dropped from 2–3 seconds to 30ms within minutes. We had half the answer. The second half was on specific Consul leader nodes that were still showing elevated latency after streaming was disabled. We dug into BoltDB — the embedded key-value store Consul uses to persist its Raft log. BoltDB marks deleted pages as free in an internal freelist but never reclaims disk space. Each time Consul performed a snapshot (which it does regularly to compact the log), it deleted old entries. Those deleted pages accumulated in the freelist, which had grown to 7.8MB containing roughly one million page IDs. Every 16kB log write now required flushing the entire 7.8MB freelist to disk. We were doing hundreds of these per second. The affected leader nodes had 4.2GB of total log storage but only 489MB of actual data — 3.8GB of dead space. We prevented those nodes from winning Raft leader elections during recovery, bypassing the bottleneck. The fix long-term was upgrading from BoltDB to bbolt, which doesn't have the freelist issue. Recovery itself took another 12 hours. Cache systems had to be rebuilt cold. We re-admitted players via DNS steering in roughly 10% increments. Players found the DNS pattern and started sharing early-access DNS entries on social media. By 16:45 on the 31st, 73 hours after first detection, we were back at 100%.

Two independent root causes, both hidden in the Consul data layer. Root cause #1: Consul 1.10's streaming feature introduced a shared Go channel as a lock bottleneck under high write throughput. The channel serialised writes that the old long-polling model had handled in parallel. On our NUMA multi-socket hardware, the more CPU cores we threw at it, the more goroutines competed for the same lock — which is why hardware upgrades made things worse. Fix: disable the streaming feature entirely. Latency returned to normal within minutes. Root cause #2: BoltDB's freelist design accumulates deleted page metadata indefinitely. Consul's regular snapshotting behaviour caused the freelist to grow to 7.8MB. Every write flushed the full freelist to disk, turning a 16kB I/O into a 7.8MB I/O. Affected only certain leader nodes. Fix: prevent those nodes from winning leader election during recovery. Long-term fix: migrate to bbolt (the maintained BoltDB fork), which uses a hash-map freelist with O(1) operations instead of O(n). No user data was lost. Recovery time from root cause identification to full player access: approximately 26 hours.