The CronJob That Quietly Saturated Our Kubernetes Cluster

This one started as a harmless cleanup task. We had moved our payments platform from Heroku to GKE about six months earlier and were still learning which safety rails Kubernetes gives you only if you ask for them. A developer added a CronJob to purge abandoned payment sessions from Redis. The job itself was simple, but it ran in the full application image, took 7 to 10 minutes on production-sized data, and nobody spent much time thinking about what would happen if it overlapped with itself.

The schedule was set to every minute instead of every hour. Because there was no concurrencyPolicy and no sensible resource settings, cleanup jobs piled up all weekend. By Monday we found dozens of active cleanup pods competing with the API for CPU and memory, plus a mountain of completed Job objects that made the whole namespace noisy to inspect. We never lost the control plane or "the cluster," but we absolutely made the cluster a bad place to run payment traffic for a few hours.

We suspended the CronJob, deleted the active Jobs, temporarily scaled the node pool up a notch to buy breathing room, and pushed a dedicated cleanup image that started quickly and did one thing. The replacement job ran hourly, used cursor-based scans instead of a giant key sweep, set concurrencyPolicy: Forbid, and cleaned up finished Jobs automatically. After that we added namespace quotas and alerts for abnormal Job growth so the next typo pages us long before Monday standup.