That wasn’t the plan, and Facebook even had a tool in place to sort out commands that might cause such a catastrophic failure, but it didn’t work. “During one of these routine maintenance jobs, a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers globally,” Janardhan wrote.
#Facebook outage Offline#
To kick off the outage, Facebook was taking just part of the backbone network offline for maintenance at 11:39 a.m. In all, Facebook was down for seven hours and five minutes. “Individual data centers were reporting dips in power usage in the range of tens of megawatts, and suddenly reversing such a dip in power consumption could put everything from electrical systems to caches at risk,” Janardhan wrote. Restoring the customer-facing services that run over the network was another lengthy process because turning them up all at once could cause another round of crashes. It took time, but once the systems were restored, the network came back up. “They’re hard to get into, and once you’re inside, the hardware and routers are designed to be difficult to modify even when you have physical access to them,” according to a Facebook blog written by Santosh Janardhan, the company's vice president of engineering and infrastructure. That slowed things down, but they were slowed down even more because the data centers have safeguards in place to make tampering hard-for anybody. To make matters worse, the loss of DNS made it impossible for Facebook engineers to remotely access the devices they needed to in order to bring the network back up, so they had to go into the data centers to manually restart systems. Facebook says the root cause of its outage Monday involved a routine maintenance job gone awry that resulted in rendering its DNS servers unavailable, but first the entire Facebook backbone network had crashed.
0 kommentar(er)
