Autonomous hardware diagnostics and recovery at scale

Cloudflare presents Phoenix, an autonomous system that discovers servers in a Repair state, performs out-of-band connectivity checks and INAT diagnostics, reprovisions those that pass, and safely re-enables them without human intervention while updating tickets and logs. The system coordinates with other automations, applies an error-budget model to halt recoveries when failures exceed thresholds, powers off failed hardware to reduce energy waste, and provides transparency and metrics to improve efficiency and reliability across data center operations.