New-age Transactional Systems - Not Your Grandpa's OLTP
John Hugg discusses high volume transaction processing applications with high and low frequency profiles, and how VoltDB can be used for that purpose.
The content has been bookmarked!
There was an error bookmarking this content! Please retry.
Posted by Chris Sims on Feb 02, 2009
Kent Beck wrote First One, Then Many to explain the application of Succession to software design. Succession is a technique for evolving the architecture of a system from 'just enough for now' to what will eventually be needed. The example given is for a system that only needs to process one transaction today, but will eventually need to process many.
In general, the Extreme Programming community prefers simple design and architecture that evolves, as needed. Examples of this include:
In Kent's example, the client doesn't yet know how multiple transactions should be handled. While the developers could make some reasonable guesses about what type of APIs and infrastructure would be appropriate for processing multiple transactions, these guesses probably won't be optimal. The team and client pay the price for developing functionality that isn't yet needed. Additionally, the team and client will have to pay again in the future, either by living with a design that was created based on guesses, or rewriting the code to correct the design. Kent points out that there is also a risk that future developers will incorrectly assume that the code already has the capability to process multiple transactions, based on the API.
Kent prefers to create a minimalist design today, and then use a process he calls Succession to evolve it. His article describes how he implements a particular type of Succession, namely One-to-Many Succession, which safely takes the code from handling single transactions to handling lists of transactions.
Would you design and build the multi-transaction system the first time? Why or why not? Leave a comment and share your thoughts.
Transforming Software Delivery: An IBM Rational Case Study
Case Study: IBM's Agile Transformation
In today’s hyper-competitive world, later may be too late to adopt Agile development and this Roadmap for Success will help you get started. Download "Agile Development: A Manager's Roadmap for Success" now!
I speak from a base of being a software developer for over 40 years. If its been done, I have likely done it - both good and bad. That is except for the getting filthy rich part.
Succession really does work even in quite complex situations. The challenge is to do the succession leveling so that a lower level does not freeze out a higher level. Doing this is often neither obvious nor easy but, if not done, you are in for a lot of restructuring.
The validity of the succession idea is based upon the fact that neither you nor your client really know what you are doing. All you have is a fuzzy idea of "wouldn't it be nice if....". You have a lot of "know how" and your client has a lot of "I wish and I want".
The method is a modification of the KISS principle: "Keep It Simple, Stupid". It should read "Keep It Stupid-Simple". Meaning the starting point is something so simple its obvious to the most casual observer and demonstratively true. Use it and learn something - especially the unintended consequences. Then build on what you have learned. Recycle as necessary until both you and your client agree that you are done - or have run out of time, money, or patience.
This is nothing really new. We have said "how good it is" for years. Yet how often do we leap over the Stupid-Simple stuff into the middle of our ignorance? I have been guilty of it and I bet you have too. Clearly, the leap that didn't work was stupid. That makes us stupid when we do it. We should start with that assumption and learn as we go. We will get to were we need to be a lot faster and with much less effort.
There is a very good guideline from the world of natural horsemanship: "Take the time it takes, and it takes less time." The same is true in software development. Shortcuts through ignorance don't work.
Absolutely agree, yes, evolve the architecture and pay tribute to your YAGNI sense. As I often hear it stated (and often state it myself): "Do the simplest thing that could possibly work".
Now, to those new to this approach (most likely, also new to agile in general), please be mindful to pay equal tribute to the implicit aspect of this statement: "Do the simplest correct thing".
In other words, I often see programmers do hacky things in the name of "YAGNI/Simplest-thing" that later makes life difficult when that change to handle "many" does come along.
Not going to go into detail with examples, etc here (maybe I'll blog about it?), but take the point seriously. Yes, there is an art to doing this well, and it's one that doesn't come overnight. (Another great instance of where having an experienced coach can be very handy)
But, at a minimum, do at least ask yourselves (as pair programmers!) - "are we doing the simplest correct thing, or taking a hacky shortcut?". You'll learn soon enough which is which.
Cheers
MB
To put it another way: Simplest thing != Fastest thing
Doing the simplest thing may require more refactoring of existing code in order to keep it simple. As your project grows and incorporates more and more functionality, doing the fastest, most expedient thing all the time will create a tangled mess in a very short timespan if your team is of any significant size at all. Or you may end up with nightmarish performance problems.
An example that I've run into recently:
Today, we need a method to return all of our Widget objects from the db, so we build Widget.GetAll. Simple enough.
Tomorrow, we need to determine if any Widgets exist so we can display an indicator in the UI. The fastest thing to do would be something like "indicator.On=Widgets.GetAll().Length>0" (pardon the confused pseudocode).
The problem of course is that the selection and return of all these objects is way more expensive in terms of both memory and time than doing a proper check against the db even though it sure is quicker to code.
The day after tomorrow, you may decide to start caching all these Widgets, but that requirement hasn't made it out yet, so ignore it!
Agree. All too often "simple" is misinterpreted to mean what is the quickest solution with the least amount of CURRENT effort. This does not keep things "stupidly simple", it just "simply stupid" (see Simple ain't Easy: Myths and Misunderstandings about Simplicity).
What we are trying to simplify and minimize is the TOTAL effort, and hence it is the EASE OF EVOLUTION that we want to optimize (and we do it by minimizing and removing complexity, not be adding it in anticipation).
When YAGNI and "Do the simplest thing that could possibly work" (a.k.a. DTSTTCPW) were introduced, there were only the first half of the story that XP and Kent were telling us at the time. The "Work" in DTSTTCPW did indeed mean "correct", and it was even okay if you did it the "simply stupid" way at first because you knew you would then subsequently "Refactor Mercilessly" to say things "Once and only once" (OAOO).
YAGNI, DTSTTCPW, Refactor Mercilessly and OAOO all went hand-in-hand together, not separate from one another. Now we sometimes see folks try to use YAGNI and DTSTTCPW in isolation from the effort for continual simplification and removal of complexity to ease evolution. It is simply WRONG to use those terms/phrases this to justify minimizing immediate/short-term effort! That is the opposite of what it is supposed to be.
John Hugg discusses high volume transaction processing applications with high and low frequency profiles, and how VoltDB can be used for that purpose.
Kevlin Henney examines code samples to see what can be learned from them starting from the premise that one won’t write great code unless he knows how to read it.
Jason Ayers share the observations he made watching a team of developers collaborating in real time on the same code base, pushing XP, pair programming and continuous integration to their extremes.
Michael Snoyman presents Yesod, a web framework written in Haskell and containing a web server, templating, ORM, libraries (templating, gravatar, etc.).
Richard Kreuter and Kyle Banker on how to avoid classical RDBMS transactional systems by using compensation mechanisms, transactional messaging or transactional procedures.
Attila Szegedi talks about performance tuning Java and Scala programs at Twitter: how to approach GC problems, the importance of asynchronous I/O, when to use MySQL/Cassandra/Redis, and much more.
One category of risk that project teams need to ensure they address is business value failure – delivering a product that fails to provide value for the business investor.
InfoQ spoke to the authors of Software Systems Architecture on a couple of new topics, the System Context viewpoint and Agile, which have been added to the second edition.
4 comments
Watch Thread Reply