OnsceneALERT
Oct 5, 20213 min
CRITICAL INCIDENT NOTIFICATION
OTHER SIGNIFICANT INCIDENT
Just as Facebook’s Antigone Davis was live on CNBC defending the company over a whistleblower’s accusations and its handling of research data suggesting Instagram is harmful to teens, its entire network of services suddenly went offline.
The outage started just before noon ET and took nearly six hours before it was resolved. This is the worst outage for Facebook since a 2019 incident took its site offline for more than 24 hours, as the downtime hit hardest on the small businesses and creators who rely on these services for their income.
After failing all tests for most of the day, a test of ISP DNS servers via DNSchecker.org showed most of them successfully finding a route to Facebook.com at 5:30PM ET. A few minutes later, we were able to start using Facebook and Instagram normally; however, it may take time for the DNS fixes to reach everyone.
Photo Credit: Wired.com
On Twitter, Facebook communications exec Andy Stone says, “We’re aware that some people are having trouble accessing our apps and products. We’re working to get things back to normal as quickly as possible, and we apologize for any inconvenience.” Mike Schroepfer, who will step down from his post as CTO next year, tweeted, “We are experiencing networking issues and teams are working as fast as possible to debug and restore as fast as possible.”
Inside Facebook, the outage has broken nearly all of the internal systems employees use to communicate and work. Several employees told The Verge they resorted to talking through their work-provided Outlook email accounts, though employees can’t receive emails from external addresses. Employees who were logged into work tools such as Google Docs and Zoom before the outage can still use those, but any employee who needs to log in with their work email was blocked.
Facebook engineers have been sent to the company’s US data centers to try and fix the problem, according to two people familiar with the situation. That meant the outage, already Facebook’s most severe in years, could be further prolonged.
A peek at Down Detector (or your Twitter feed) reveals the problems were widespread. While it’s unclear exactly why the platforms were unreachable for so many people, their DNS records show that, like last week’s Slack outage, the problem is apparently DNS (it’s always DNS).
Cloudflare senior vice president Dane Knecht notes that Facebook’s border gateway protocol routes — BGP helps networks pick the best path to deliver internet traffic — were suddenly “withdrawn from the internet.” While some have speculated about hackers, or an internal protest over last night’s whistleblower report, there isn’t any information yet to suggest anything malicious is to blame.
Instagram.com was flashing a 5xx Server Error message, while the Facebook site merely told us that something went wrong. The problem also affected its virtual reality arm, Oculus. Users could load games they already have installed, and the browser works, but social features or installing new games didn’t.
Update Oct 4th, 3:37PM ET: Added additional information about the outage.
Update Oct 4th, 4:15PM ET: Added statement from Facebook CTO Mike Schroepfer
Update Oct 4th, 5PM ET: Outage is still ongoing, added information about the 2019 outage.
Update October 4th, 5:35PM ET: DNS updates suggest Facebook is closing in on a solution.
Update October 4th, 6:08PM ET: Facebook.com is back online.
Source Agency (The Verge)
Sourcing Link (The Verge)
Alerting Agency: OnsceneALERT.com
Want to receive disaster & critical incident alerts like these on your mobile device?
Please visit www.OnsceneALERT.com
Alerts starting at $5.99 per month
OnsceneALERT is a multi-award-winning emergency notification provider when critical incidents threaten public safety;
such as disasters, violent threats, and significant events.
Our community-based mobile alerts provide critical incident awareness of emerging threats and approaching disasters throughout the United States so individuals can be better prepared to protect themselves and their loved ones. Our alerts also increase emergency preparedness and disaster readiness for individuals, families, businesses, schools, government, faith-based organizations, and many more.
"Disasters Don't Give Warnings, But We Do"
Incident Number
2-211004-7301
Alert Type
Critical Incident Notification
Incident Type
Other Significant Incident
Incident Occurred
October 4, 2021
Alert Posted
October 4, 2021
7:02 pm CST
Last Updated
October 4, 2021
7:02 pm CST
Incident Location
Nationwide
END OF ALERT