Widespread internet outages around the United States on Friday afternoon quelled productivity and sent irate customers to Twitter to complain. Comcast and Xfinity suffered the biggest service interruptions across its internet, cable, and landline products. The company, which has more than 29 million business and individual customers, said on Friday that the outages stemmed from fiber optic cables at two internet infrastructure companies that were cut or otherwise disrupted.
Like virtually all internet providers, Comcast relies on a combination of its own fiber optic infrastructure and that of other partner companies to seamlessly route data around the world.
“We identified two, separate and unrelated fiber cuts to our network backbone providers,” Comcast said in a statement to WIRED. “Our engineers worked to address the issue immediately and services are now being restored to business and residential internet, video and voice customers.”
‘We are profoundly and globally dependent on a fundamentally fragile infrastructure.’
Roland Dobbins, Arbor Networks
Comcast says the two internet infrastructure companies involved are Level 3 (now owned by CenturyLink) and Zayo, a fiber company headquartered in Colorado. Throughout the afternoon, the outage-tracking site Down Detector showed service interruptions at CenturyLink, Zayo, and Comcast, but the latter suffered the most severe consequences.
“While the CenturyLink network continues to operate normally, on June 29 we experienced two isolated fiber cuts in North Carolina affecting some customers,” CenturyLink said in a statement. “At this time, our technicians are working to restore the services.”
CenturyLink noted that its two fiber cuts would not have been enough on their own to cause the outage, indicating that another also occurred, as Comcast said.
“Earlier today, Zayo experienced a fiber cut in the New York area,” said a Zayo spokesperson in a statement. “We immediately dispatched our local team who quickly restored the cut. All impacted services in the area have been restored.”
Fiber cuts aren’t necessarily malicious, and can happen as the result of incidents like severe weather or construction mistakes. They’re also not terribly uncommon; when they do happen, the internet infrastructure community works to implement redundancies and traffic rerouting tactics so physical disruptions don’t cause digital ones. In this case, the combination of disruptions in New York and North Carolina were enough to turn off the internet for millions of people.
The underlying physical backbone of the internet is surprisingly fragile, and failsafes don’t always work. For example, in November, a tiny misconfiguration error at Level 3 caused outages around the US. And a digital attack on the internet infrastructure company Dyn famously caused major outages in 2016 because they were targeted at destabilizing one of the internet’s underlying routing protocols.
“We are profoundly and globally dependent on a fundamentally fragile infrastructure,” says Roland Dobbins, a principal engineer at the DDoS and network-security firm Arbor Networks, which monitors global internet operations. “Some redundancies exist, but many times they don’t.”
By Friday evening, Comcast service had come back for many customers. But the underlying message should resonate: The internet can be more frail than you’d think, and sometimes all it takes to shut it down is a couple of cuts.