BitCoins Lost, MongoDB and Eventual Consistency
The recent theft from several BitCoin operators has sparked a debate whether eventually consistent databases are useful or not for banking.
On March 2, 2014 Flexcoin lost all its bitcoins due to a code flaw. The attacker issued thousands of concurrent requests ordering transfers from one of his accounts to another. He then repeated the operation with other accounts until all bitcoins were withdrawn. This was possible because the code was not written to deal with multiple concurrent requests, and all the transfers happened before the balances were updated. If a balance is not updated in time, a new request could be granted even if the account normally is empty. As a result, Flexcoin shut down operations after losing 896 BTC valued at about half a million dollars.
The same happened with Poloniex two days later, but they lost only 12.3% of their bitcoins and the company covered the losses, managing to stay afloat.
Emin Gün Sirer, an Associate Professor at Cornell University, wrote a blog post blaming eventually consistent data stores for the lost bitcoins. He mentions MongoDB, Cassandra and Riak among the NoSQL solutions that are vulnerable to banking thefts because:
The problem here stemmed from the broken-by-design interface and semantics offered by MongoDB. And the situation would not have been any different if we had used Cassandra or Riak. …
Bitcoin coincided with a particularly dark time in distributed systems when people, armed with an incorrect interpretation of the CAP Theorem, thought that they just had to give up on consistency in their databases …
It is not clear if Flexcoin or Poloniex were using MongoDB at that time, and it should be mentioned that Sirer is involved in developing HyperDex, a competing key-value data store supporting ACID transactions. Also, this is not the first time Sirer blames MongoDB’s design. A year ago, he claimed that MongoDB’s fault tolerance is broken.
Polemics aside, it is worth pondering if an eventually consistent data store is suitable for banking operations. As expected, Sirer’s post triggered a large number of comments, and we extracted some of the most significant ones.
jrp noted that the update operation could have been done with MongoDB at the database level in an atomic way, but agreed that “this will not take care of other ACID properties.”
jakcharlton considered that while eventual consistency is problematic in such cases, an “ACID system does not solve the problem, it merely pushes it to an application boundary, where the problem re-occurs. Banking was a bad example domain to use, and shows the issue of developers thinking that as they have ACID in their database they are protected from issues of concurrency.”
Alex uses MongoDB for “everything except where transactions are needed and then I use the RIGHT tool for the job (which is not MongoDB).” He considered that the developer made a mistake by using MongoDB for a job that it should not be used for.
Robert Escriva, a HyperDex developer, considered that the culprit is not with the programmer but a general perception that eventually consistent systems can be used for banking:
The problem is not with programmers' understanding. There's a pervasive and wrong movement that encourages people to use eventually consistent systems. They will justify it with statements of the form, "If it's good enough for banks, then it's good enough for you." This thinking is dangerous.
At the end of the day, your application is the sum of its invariants. If the data store underneath your system is incapable of providing you with guarantees---or worse, requires extensive care, and a $100K support contract to provide you with guarantees---your application is broken and blaming the developers for this is a cop-out. We should be holding database vendors (especially those with hundreds of million in shiny VC cash) to a higher standard.
Eric Brewer, author of the CAP theorem, wrote earlier that banks give to consistency for their ATM operations in order to provide availability during partitioning. But they do this by taking certain precautions, including limiting the amount of money withdrawn at a certain small threshold. We should also mention that Stripe, a web&mobile payment system uses MongoDB, according to one of their blog posts.
Are eventually consistent data stores appropriate for general banking operations or should developers be aware of their limitations and look elsewhere?
Real banks of Laws, Bitcoins don't
Distributed systems and banks
So, moving to software and architecture, we have CQRS for this kind of business requirements. Too bad companies thinks that almost anybody can built complex distributed systems, so from time to time we can see Flexcoin case happening all over again.
BitCoins, Model of banking, the right tool
- nosql born also for some needs and not for all; relational world has more and more years of use cases, nosql world is younger so we need more use cases so we also need to use nosql
- are you sure that model of payments in banks, so near to db or mainframe, is the only way to do? Is C ap t he problem or there's more?
Relational BD does not make but storage saving
Pascual de Juan Núñez
There's no magic virtue in relational databases but normalization, which was needed in the seventies to save punched cards, tape or hard disk space (i.e. storing the relation/index to a common set of attributes like customer details in a customer table, instead of recording it in every account movement). De-normalization is cheaper nowadays, so we can use the right schema to the right usage (like faster readings). Even further, we can use several schemas (CQRS) to achieve the right goal.
Do you want to have a perfect ACID system acting as a single one? Well, maybe you'll need to get back to a single Mainframe or, if you think it twice, maybe you don't need a perfect ACID system for most of your use cases (and this also applies to banking).
Mainframe it is..
Banks do use eventual consistency, but carefully
The point: ATM machines have a max withdrawal amount within some timeframe, this means that while damage/theft might happen, it will be within some bound.
Ralph Winzinger Nov 25, 2014
John Krewson, Steve Ropa and Matt Badgley Nov 24, 2014