BT

BitCoins Lost, MongoDB and Eventual Consistency

by Abel Avram on Apr 24, 2014 |

The recent theft from several BitCoin operators has sparked a debate whether eventually consistent databases are useful or not for banking.

On March 2, 2014 Flexcoin lost all its bitcoins due to a code flaw. The attacker issued thousands of concurrent requests ordering transfers from one of his accounts to another. He then repeated the operation with other accounts until all bitcoins were withdrawn. This was possible because the code was not written to deal with multiple concurrent requests, and all the transfers happened before the balances were updated. If a balance is not updated in time, a new request could be granted even if the account normally is empty. As a result, Flexcoin shut down operations after losing 896 BTC valued at about half a million dollars.

The same happened with Poloniex two days later, but they lost only 12.3% of their bitcoins and the company covered the losses, managing to stay afloat.

Emin Gün Sirer, an Associate Professor at Cornell University, wrote a blog post blaming eventually consistent data stores for the lost bitcoins. He mentions MongoDB, Cassandra and Riak among the NoSQL solutions that are vulnerable to banking thefts because:

The problem here stemmed from the broken-by-design interface and semantics offered by MongoDB. And the situation would not have been any different if we had used Cassandra or Riak. …

Bitcoin coincided with a particularly dark time in distributed systems when people, armed with an incorrect interpretation of the CAP Theorem, thought that they just had to give up on consistency in their databases …

It is not clear if Flexcoin or Poloniex were using MongoDB at that time, and it should be mentioned that Sirer is involved in developing HyperDex, a competing key-value data store supporting ACID transactions. Also, this is not the first time Sirer blames MongoDB’s design. A year ago, he claimed that MongoDB’s fault tolerance is broken.

Polemics aside, it is worth pondering if an eventually consistent data store is suitable for banking operations. As expected, Sirer’s post triggered a large number of comments, and we extracted some of the most significant ones.

jrp noted that the update operation could have been done with MongoDB at the database level in an atomic way, but agreed that “this will not take care of other ACID properties.”

jakcharlton considered that while eventual consistency is problematic in such cases, an “ACID system does not solve the problem, it merely pushes it to an application boundary, where the problem re-occurs. Banking was a bad example domain to use, and shows the issue of developers thinking that as they have ACID in their database they are protected from issues of concurrency.”

Alex  uses MongoDB for “everything except where transactions are needed and then I use the RIGHT tool for the job (which is not MongoDB).” He considered that the developer made a mistake by using MongoDB for a job that it should not be used for.

Robert Escriva, a HyperDex developer, considered that the culprit is not with the programmer but a general perception that eventually consistent systems can be used for banking:

The problem is not with programmers' understanding. There's a pervasive and wrong movement that encourages people to use eventually consistent systems. They will justify it with statements of the form, "If it's good enough for banks, then it's good enough for you." This thinking is dangerous.

At the end of the day, your application is the sum of its invariants. If the data store underneath your system is incapable of providing you with guarantees---or worse, requires extensive care, and a $100K support contract to provide you with guarantees---your application is broken and blaming the developers for this is a cop-out. We should be holding database vendors (especially those with hundreds of million in shiny VC cash) to a higher standard.

Eric Brewer, author of the CAP theorem, wrote earlier that banks give to consistency for their ATM operations in order to provide availability during partitioning. But they do this by taking certain precautions, including limiting the amount of money withdrawn at a certain small threshold. We should also mention that Stripe, a web&mobile payment system uses MongoDB, according to one of their blog posts.

Are eventually consistent data stores appropriate for general banking operations or should developers be aware of their limitations and look elsewhere? 

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Real banks of Laws, Bitcoins don't by Fake Fake

One thing that this post, and most others miss, is that real banks have the law behind them. They can dip into negative balances because you signed a contract with them agreeing to pay back the balance, and in fact the bank makes some money off of this. Bitcoin exchanges do not have this benefit. So the question is not about if eventual consistency is acceptable for banking or not, one also has to take into account what happens if you do go negative. For Bitcoin exchanges right now, it might make more sense to sacrifice some availability because you don't have the government backing you up.

Distributed systems and banks by Hrvoje Hudoletnjak

Banks know how to deal those things from the beginning. They had offices in different locations, and when someone withdraw a money in one city, they had to "sync" balance on all locations. And that was before broadband internet connections and optical cables, there where horses:)
So, moving to software and architecture, we have CQRS for this kind of business requirements. Too bad companies thinks that almost anybody can built complex distributed systems, so from time to time we can see Flexcoin case happening all over again.

BitCoins, Model of banking, the right tool by stefano fago

- as already said here, bitcoins entities are not banks
- nosql born also for some needs and not for all; relational world has more and more years of use cases, nosql world is younger so we need more use cases so we also need to use nosql
- are you sure that model of payments in banks, so near to db or mainframe, is the only way to do? Is C ap t he problem or there's more?

Relational BD does not make but storage saving by Pascual de Juan Núñez

The only virtue of ACID databases relies in a single point of COMMIT, not in the relational representation of the information.

There's no magic virtue in relational databases but normalization, which was needed in the seventies to save punched cards, tape or hard disk space (i.e. storing the relation/index to a common set of attributes like customer details in a customer table, instead of recording it in every account movement). De-normalization is cheaper nowadays, so we can use the right schema to the right usage (like faster readings). Even further, we can use several schemas (CQRS) to achieve the right goal.

Do you want to have a perfect ACID system acting as a single one? Well, maybe you'll need to get back to a single Mainframe or, if you think it twice, maybe you don't need a perfect ACID system for most of your use cases (and this also applies to banking).

Mainframe it is.. by Willem Goudsbloem

Actualy most banks still depend on the mainframe for transactional data... by choice.

Banks do use eventual consistency, but carefully by John Heintz

See this previous infoq article talking about ATM machines: www.infoq.com/articles/cap-twelve-years-later-h...

The point: ATM machines have a max withdrawal amount within some timeframe, this means that while damage/theft might happen, it will be within some bound.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

6 Discuss

Educational Content

General Feedback
Bugs
Advertising
Editorial
InfoQ.com and all content copyright © 2006-2014 C4Media Inc. InfoQ.com hosted at Contegix, the best ISP we've ever worked with.
Privacy policy
BT