The recent theft from several BitCoin operators has sparked a debate whether eventually consistent databases are useful or not for banking.
On March 2, 2014 Flexcoin lost all its bitcoins due to a code flaw. The attacker issued thousands of concurrent requests ordering transfers from one of his accounts to another. He then repeated the operation with other accounts until all bitcoins were withdrawn. This was possible because the code was not written to deal with multiple concurrent requests, and all the transfers happened before the balances were updated. If a balance is not updated in time, a new request could be granted even if the account normally is empty. As a result, Flexcoin shut down operations after losing 896 BTC valued at about half a million dollars.
The same happened with Poloniex two days later, but they lost only 12.3% of their bitcoins and the company covered the losses, managing to stay afloat.
Emin Gün Sirer, an Associate Professor at Cornell University, wrote a blog post blaming eventually consistent data stores for the lost bitcoins. He mentions MongoDB, Cassandra and Riak among the NoSQL solutions that are vulnerable to banking thefts because:
The problem here stemmed from the broken-by-design interface and semantics offered by MongoDB. And the situation would not have been any different if we had used Cassandra or Riak. …
Bitcoin coincided with a particularly dark time in distributed systems when people, armed with an incorrect interpretation of the CAP Theorem, thought that they just had to give up on consistency in their databases …
It is not clear if Flexcoin or Poloniex were using MongoDB at that time, and it should be mentioned that Sirer is involved in developing HyperDex, a competing key-value data store supporting ACID transactions. Also, this is not the first time Sirer blames MongoDB’s design. A year ago, he claimed that MongoDB’s fault tolerance is broken.
Polemics aside, it is worth pondering if an eventually consistent data store is suitable for banking operations. As expected, Sirer’s post triggered a large number of comments, and we extracted some of the most significant ones.
jrp noted that the update operation could have been done with MongoDB at the database level in an atomic way, but agreed that “this will not take care of other ACID properties.”
jakcharlton considered that while eventual consistency is problematic in such cases, an “ACID system does not solve the problem, it merely pushes it to an application boundary, where the problem re-occurs. Banking was a bad example domain to use, and shows the issue of developers thinking that as they have ACID in their database they are protected from issues of concurrency.”
Alex uses MongoDB for “everything except where transactions are needed and then I use the RIGHT tool for the job (which is not MongoDB).” He considered that the developer made a mistake by using MongoDB for a job that it should not be used for.
Robert Escriva, a HyperDex developer, considered that the culprit is not with the programmer but a general perception that eventually consistent systems can be used for banking:
The problem is not with programmers' understanding. There's a pervasive and wrong movement that encourages people to use eventually consistent systems. They will justify it with statements of the form, "If it's good enough for banks, then it's good enough for you." This thinking is dangerous.
At the end of the day, your application is the sum of its invariants. If the data store underneath your system is incapable of providing you with guarantees---or worse, requires extensive care, and a $100K support contract to provide you with guarantees---your application is broken and blaming the developers for this is a cop-out. We should be holding database vendors (especially those with hundreds of million in shiny VC cash) to a higher standard.
Eric Brewer, author of the CAP theorem, wrote earlier that banks give to consistency for their ATM operations in order to provide availability during partitioning. But they do this by taking certain precautions, including limiting the amount of money withdrawn at a certain small threshold. We should also mention that Stripe, a web&mobile payment system uses MongoDB, according to one of their blog posts.
Are eventually consistent data stores appropriate for general banking operations or should developers be aware of their limitations and look elsewhere?
Community comments
Real banks of Laws, Bitcoins don't
by Fake Fake,
Distributed systems and banks
by Hrvoje Hudoletnjak,
BitCoins, Model of banking, the right tool
by stefano fago,
Relational BD does not make but storage saving
by Pascual de Juan Núñez,
Mainframe it is..
by Willem Goudsbloem,
Banks do use eventual consistency, but carefully
by John Heintz,
Real banks of Laws, Bitcoins don't
by Fake Fake,
Your message is awaiting moderation. Thank you for participating in the discussion.
One thing that this post, and most others miss, is that real banks have the law behind them. They can dip into negative balances because you signed a contract with them agreeing to pay back the balance, and in fact the bank makes some money off of this. Bitcoin exchanges do not have this benefit. So the question is not about if eventual consistency is acceptable for banking or not, one also has to take into account what happens if you do go negative. For Bitcoin exchanges right now, it might make more sense to sacrifice some availability because you don't have the government backing you up.
Distributed systems and banks
by Hrvoje Hudoletnjak,
Your message is awaiting moderation. Thank you for participating in the discussion.
Banks know how to deal those things from the beginning. They had offices in different locations, and when someone withdraw a money in one city, they had to "sync" balance on all locations. And that was before broadband internet connections and optical cables, there where horses:)
So, moving to software and architecture, we have CQRS for this kind of business requirements. Too bad companies thinks that almost anybody can built complex distributed systems, so from time to time we can see Flexcoin case happening all over again.
BitCoins, Model of banking, the right tool
by stefano fago,
Your message is awaiting moderation. Thank you for participating in the discussion.
- as already said here, bitcoins entities are not banks
- nosql born also for some needs and not for all; relational world has more and more years of use cases, nosql world is younger so we need more use cases so we also need to use nosql
- are you sure that model of payments in banks, so near to db or mainframe, is the only way to do? Is C ap t he problem or there's more?
Relational BD does not make but storage saving
by Pascual de Juan Núñez,
Your message is awaiting moderation. Thank you for participating in the discussion.
The only virtue of ACID databases relies in a single point of COMMIT, not in the relational representation of the information.
There's no magic virtue in relational databases but normalization, which was needed in the seventies to save punched cards, tape or hard disk space (i.e. storing the relation/index to a common set of attributes like customer details in a customer table, instead of recording it in every account movement). De-normalization is cheaper nowadays, so we can use the right schema to the right usage (like faster readings). Even further, we can use several schemas (CQRS) to achieve the right goal.
Do you want to have a perfect ACID system acting as a single one? Well, maybe you'll need to get back to a single Mainframe or, if you think it twice, maybe you don't need a perfect ACID system for most of your use cases (and this also applies to banking).
Mainframe it is..
by Willem Goudsbloem,
Your message is awaiting moderation. Thank you for participating in the discussion.
Actualy most banks still depend on the mainframe for transactional data... by choice.
Banks do use eventual consistency, but carefully
by John Heintz,
Your message is awaiting moderation. Thank you for participating in the discussion.
See this previous infoq article talking about ATM machines: www.infoq.com/articles/cap-twelve-years-later-h...
The point: ATM machines have a max withdrawal amount within some timeframe, this means that while damage/theft might happen, it will be within some bound.