Twitter data center in Sacramento reportedly knocked offline by California heat wave
Last week’s extreme heat event in California caused an outage at a key Twitter data center in Sacramento, CNN reported Monday, leaving the social media platform vulnerable in the event of another outage at a different data center.
The platform on Monday, Sept. 5, “experienced the loss of its Sacramento (SMF) datacenter due to extreme weather,” an internal message to Twitter engineers sent Friday said in part, according to a story published Monday by CNN.
Sept. 5 marked the beginning of the most intense stretch within one of the biggest heat waves in Northern California history: downtown Sacramento reached 113 degrees, followed by a 116-degree high that made last Tuesday the hottest day in the city’s recorded history.
Data centers are warehouse-type sites with large volumes of computers, servers and other equipment, much of which must be closely temperature-controlled else risk overheating.
The internal memo, which CNN reported was sent by vice president of engineering Carrie Fernandez, said the Sacramento outage left Twitter in a “non-redundant” state.
This means that if one of two other sites were to experience an outage, said in the memo to be located in the Atlanta and Portland areas, Twitter “may not be able to serve traffic to all Twitter’s users.”
Twitter in a statement to CNN said there “have been no disruptions impacting the ability for people to access and use Twitter at this time.”
The former head of security at Twitter, Peiter Zatko, in a whistleblower complaint last month to multiple federal government agencies including the Securities and Exchange Commission, warned of Twitter’s “insufficient data center redundancy,” CNN reported.
Zatko’s complaint noted that in the period of 2020 to 2021, Twitter had data centers in Sacramento and Atlanta, with plans as of 2020 to add a new data center, according to recent reporting from the Washington Post, which obtained a copy of the complaint.
Zatko’s whistleblower complaint cites a February 2020 post by a blog called Data Center Dynamics, which said Twitter leases space at data center in Sacramento operated by a subsidiary of Japanese telecom giant NTT.
The exact location of Twitter’s Sacramento data center has not been publicly disclosed. Tech companies typically keep that information guarded for security reasons; likewise, data centers that lease out space tend not to reveal their clients.
NTT has three data centers in Sacramento’s Natomas area. The newest of the three is a 180,000-square-foot wholesale data center, according to the company’s website.
But in an emailed statement Tuesday morning, NTT said it did not experience an outage last week at its Sacramento center or any of its California sites.
Rather, in order to reduce load on the grid, the company said it switched to generator power during the height of the heat wave.
“Throughout last week’s extreme hot weather conditions in California NTT’s data centers remained operational, with 100% uptime and no disruption or outages for any of our clients,” the company’s statement said. “As the energy situation became more urgent, NTT worked closely with SMUD and the energy commission to move its data center load to back-up generators to reduce demand on the grid, freeing up critical energy resources for others.”
It is unclear whether Twitter currently leases space at multiple data centers in Sacramento.
Twitter has data ties with NTT America dating back to at least 2009, according to media reports as well as archived Twitter blog posts from the time, the latter of which said the social media giant would work with NTT to maintain its data center “footprint.”
Twitter around 2010 announced plans to build its own dedicated data center in Salt Lake City, parting ways at least in part with NTT, but the project failed for numerous reasons, according to a 2011 story by Reuters.
Twitter instead moved its data center, formerly in Santa Clara, into a Sacramento facility owned by a company called RagingWire, Reuters reported at the time. NTT acquired a majority stake of RagingWire in 2014 and 100% of the company in 2018, according to a company news release.
NTT on its website describes Sacramento as an optimal data center location because of its proximity to the Bay Area and Silicon Valley while offering less risk of earthquake damage.
Several other large data centers are located in California’s capital city, including QTS, also in Natomas, and Prime Data Centers in McClellan Park.
Zatko was due to give whistleblower testimony Tuesday regarding Twitter security to the U.S. Senate Judiciary Committee.
This story was originally published September 12, 2022 at 1:10 PM.
