Eff Up Like It's Your Job
by in Editor's Soapbox on 2021-10-06This past Monday, Facebook experienced an outage which lasted almost six hours. This had rattle-on effects. Facebook's pile of services all failed, from the core application to WhatsApp to Oculus. Many other services use Facebook for authentication, so people lost access to those (which highlights some rather horrifying dependencies on Facebook's infrastructure). DNS servers were also strained as users and applications kept trying to find Facebook, and kept failing.
CloudFlare has more information about what went wrong, but at its core: Facebook's network stopped advertising the routes to its DNS servers. The underlying cause of that may have been a bug in their Border Gateway Protocol automation system: