Untangling Terraform State Drift 30 Minutes Before a Board Demo

At 1:30 PM, we had exactly the wrong kind of change window. We were an infrastructure tooling company, our production stack lived in Terraform with an S3 backend and DynamoDB locking, and the board was arriving at 2 PM for a demo of our new multi-region failover flow. One engineer was applying a change to roll out the new ECS task definition and ALB wiring for the demo build. A second engineer saw a stale-looking lock from their terminal, assumed the first run had died, force-unlocked it, and kicked off a different apply.

The state file was not corrupted so much as suddenly untrustworthy. After the overlapping applies, Terraform's view of AWS no longer matched what actually existed: a few resources had been created, some addresses had moved, and the next plan wanted to replace 47 production resources including parts of the app tier and database plumbing. That was the moment we stopped treating it like a deployment problem and declared an incident.

We pulled the previous state version from S3, compared it against what actually existed in AWS, and imported the three resources that had been partially created before the lock fiasco. Once terraform plan went clean again, we made the very adult decision to stop changing production and run the demo on what was already deployed. Afterward we moved all production applies behind CI, added a loud Slack announcement for every active apply, and agreed that force-unlock in production needs a second human in the room.