OpenAI Services Hit by Major Outage Due to Telemetry Service Deployment

OpenAI’s December 11, 2024 service disruption was caused by a new telemetry service whose configuration led every node in large Kubernetes clusters to perform resource-intensive API operations, overwhelming control planes and degrading or halting ChatGPT, the API, and Sora for over four hours; DNS caching briefly masked the issue, complicating detection and rollback, and remediation involved scaling clusters down, blocking admin API access, and scaling up API servers; the incident was not a security breach, and OpenAI committed to stronger pre-deployment testing, Kubernetes API load assessment, and rollout safeguards to prevent recurrence.