Coinbase to 'decompose' its application server in wake of recent outage

Coinbase says it will divide up its application server into separate services in an effort to “reduce the blast radius” following an outage last week that impacted the API used for the exchange’s website and mobile apps. 

The Coinbase team said Friday that by “decomposing” its application server, it could have different scaling profiles for the different functions the API serves. Thus, if one surface is affected somehow, it will only affect the APIs or functionality that the particular surface is responsible for, instead of taking down the entire system. 

The outage on January 6th and 7th was triggered by record-high traffic from crypto price movement, according to Coinbase, which led to “a thundering herd of requests.”

“Several of our downstream services were hit with traffic that they were underprovisioned for, and client retry behavior on the application server compounded the issue, rapidly issuing retry requests to the services,” the exchange said. “This retry storm prevented some parts of our app from functioning.”

The issue also happened because of problems with MongoDB clusters, which are responsible for handling the exchange’s high data volumes. Essentially, the MongoDB clusters were experiencing long delays when queried, causing newer queries to be further delayed. The team triggered a failover on these “unhealthy” clusters, which momentarily returned them to a healthy state. However, the clusters would regress back to unhealthy status within ten minutes. Thus, users were only able to use their apps for short periods of time. 

According to the announcement, the Coinbase team plans to break MongoDB clusters which carry “large discrete miscellaneous collections” into separate clusters, which will also make it harder to take down the entire system in the event something like this happens again.

Coinbase maintains that despite the server being overwhelmed, users were still able to buy, sell, and trade cryptocurrencies. 

The outage is one of several technical issues the exchange has faced in recent months, while also breaking its all-time-highs for volume. 

Most recently, on January 11, Coinbase Support announced that they were investigating an issue impacting transactions on their website and mobile apps. According to crypto on-chain analyst Willy Woo, buys on the exchange were not completing. “Coinbase is $350 lower than other exchanges right now, it’s throwing off derivative indexes and likely impacting trade algos,” he tweeted on January 11. However, Coinbase says this issue was not connected to the January 6-7 outage.  

In a separate statement also published Friday, Coinbase vice president of customer experience Casper Sorensen emphasized the exchange’s commitment to its users. 

“We know that we have let many of you down and we are doing everything in our power to turn the page,” he wrote. “Whether you’re a new user or have been with us since day one, we recognize this is frustrating. This is not the experience we want for you, our customers.”