(Dec 7): An outage that took down markets operated by CME Group Inc for more than 10 hours at the end of last week was caused by human error at a data centre owned by CyrusOne.
Onsite staff and contractors at the facility in Aurora, Illinois, failed to follow standard procedures for draining cooling towers ahead of freezing temperatures, a spokesperson for CyrusOne, a company owned by KKR & Co and Global Infrastructure Partners, said in a statement to Bloomberg News on Saturday. That resulted in the cooling system being overloaded and rising temperatures.
The data center serves as the hub for the CME, one of the world’s largest derivatives exchanges, which handles trillions of dollars each day across equity, currency, bond and commodity markets globally. The snafu at the facility some 50 miles (80 kilometres) from Chicago disrupted markets from Tokyo to London on Nov 28, bringing to a halt trading in everything from gold to oil and interest rates.
While CyrusOne said it took “extensive and decisive steps to restore the cooling systems,” CME issued a separate statement saying that the data centre’s “initial remediation attempts further exacerbated the problem, which eventually led to the failures of a number of chillers.”
The outage underscored risks associated with the CME’s reliance on the data centre, which it sold to CyrusOne in 2016 and leased it back on a 15-year deal. While CME’s disaster recovery plan calls for a move to a data centre in the New York area, the exchange opted against switching to the backup facility because the information it had pointed to a brief outage.
“We recognise the impact experienced by our clients around the world,” CME said on Saturday.
See also: Google must limit default contracts to one year, judge rules
CyrusOne said it brought in additional personnel to the site to handle the outage and its fallout. That included top engineering teams and executives such as chief executive officer Eric Schwartz. It also used additional equipment to restore cooling capacity. The data center operator said the facility never lost power.
To prevent a repeat of the outage, the company said it updated its procedures for cold weather, including boosting onsite engineers, and it’s also hardening its infrastructure in phases and increasing staff training.
The equipment failure has become a wake-up call for investors who’ve piled hundreds of billions of dollars into the artificial intelligence (AI) data centre boom, serving as a reminder that the facilities’ lease agreements are, in fact, breakable. Termination clauses are baked into many of them in the event of recurring outages.
See also: SpaceX to offer insider shares at record-setting valuation — Bloomberg
After the outage, Goldman Sachs Group Inc paused a planned US$1.3 billion ($1.68 billion) mortgage-bond sale for CyrusOne. The bank had been managing what would have been the data centre operator’s largest-ever commercial mortgage bond sale, with early feedback due on Nov 25, Bloomberg previously reported.
In a report on Friday, Citigroup Inc stressed that data centre leases can include clauses allowing tenants to terminate their contracts in the case of certain deficiencies, including cooling issues.
“We anticipate this is a one-off situation, and the owner-operator and tenant will fully resolve any redundancy deficiencies that caused the problem,” Citi said in the report.
Uploaded by Liza Shireen Koshy
