At the simplest level, scalability is about doing more of something. This could be responding to more user requests, executing more work or handling more data. While designing software has its complexities, making that software capable of doing lots of work presents its own set of problems. This article presents some principles and guidelines for building scalable software systems.
1. Decrease processing time
One way to increase the amount of work that an application does is to decrease the time taken for individual work units to complete. For example, decreasing the amount of time required to process a user request means that you are able to handle more user requests in the same amount of time. Here are some examples of where this principle is appropriate and some possible realisation strategies.
- Collocation : reduce any overheads associated with fetching data required for a piece of work, by collocating the data and the code.
- Caching : if the data and the code can't be collocated, cache the data to reduce the overhead of fetching it over and over again.
- Pooling : reduce the overhead associated with using expensive resources by pooling them.
- Parallelization : decrease the time taken to complete a unit of work by decomposing the problem and parallelizing the individual steps.
- Partitioning : concentrate related processing as close together as possible, by partitioning the code and collocating related partitions.
- Remoting : reduce the amount of time spent accessing remote services by, for example, making the interfaces more coarse-grained. It's also worth remembering that remote vs local is an explicit design decision not a switch and to consider the first law of distributed computing - do not distribute your objects.
As software developers, we tend to introduce abstractions and layers where they are often not required. Yes, these concepts are great tools for decoupling software components, but they have a tendency to increase complexity and impact performance, particularly if you're converting between data representations at each layer. Therefore, the other way in which processing time can be minimised is to ensure that the abstractions aren't too abstract and that there's not too much layering. In addition, it's worth understanding the cost of runtime services that we take for granted because, unless they have a specific service level agreement, it's possible that these could end up being the bottlenecks in our applications.
2. Partition
Decreasing the processing time associated with a particular work unit will get you so far, but ultimately you'll need to scale out your system when you reach the limits of a single process deployment. In a typical web application, scaling out could be as easy as starting up additional web servers to handle the user requests and load balancing between them. What you might find, however, is that parts of your overall architecture will start to become points of contention because everything will get busy at the same time. A good example is a single database server sitting behind all those web servers. When that starts to become the bottleneck, you have to change your approach and one way to do this is to adopt a partitioning strategy. Put simply, this involves breaking up that single piece of the architecture into smaller more manageable chunks. Partitioning that single element into smaller chunks allows you to scale them out and this is exactly the technique that large sites such as eBay use to ensure that their architectures scale. Partitioning is a good solution, although you may find that you trade-off consistency.
As to how you partition your system, well that depends. Truly stateless components can simply be scaled out and the work load balanced between them, ideally with all instances of the component running in an active manner. If, on the other hand, there is state that needs to be maintained, you need to find a workload partitioning strategy that will allow you to have multiple instances of those stateful components, where each instance is responsible for a distinct subset of the work and/or data.
3. Scalability is about concurrency
Scalability is inherently about concurrency; after all, it's about doing more work at the same time. Technologies such as the early versions of Enterprise JavaBeans (EJB) attempted to provide a simplified programming model and encouraged us to write components that were single-threaded. Unfortunately, these components typically had dependencies on other components and this led to concurrency problems. If concurrency isn't thought about, you have systems where data can easily become corrupted. On the other hand, too many guards around concurrency lead to systems that are essentially serial in nature and limited in the degree to which they can scale. Concurrent programming isn't that hard to do, but there are some simple principles that can help when building scalable systems.
- If you do need to hold locks (e.g. local objects, database objects, etc), try to hold them for as little time as possible.
- Try to minimize contention of shared resources and try to take any contention off of the critical processing path (e.g. by scheduling work asynchronously).
- Any design for concurrency needs to be done up-front, so that it's well understood which resources can be shared safely and where potential scalability bottlenecks will be.
4. Requirements must be known
In order to build a successful software system, you need to know what your goals are and what you're aiming for. While the functional requirements are often well-known, it's the non-functional requirements (or system qualities) that are usually absent. If you do genuinely need to build a piece of software that is highly scalable, then you need to understand the following types of things up-front for the critical components/workflows.
- Target average and peak performance (i.e. response time, latency, etc).
- Target average and peak load (i.e. concurrent users, message volumes, etc).
- Acceptable limits for performance and scalability.
It might be that performance isn't critical, but you need to know this information as early as possible, because your approach to dealing with scalability will be driven by the performance requirements.
5. Test continuously
Once you understand the requirements you can start designing and building the solution. The design that we come up with and the code that we write is static in nature, so you can never quite tell how it will work until it is executed. It's for this reason, then, that all decisions on performance and scalability should be backed up by evidence, and this evidence should be gathered and reviewed from the start of the project and on a continuous basis thereafter. In other words; set measurable goals throughout the system, verify and measure the real performance and consider performance at all stages of the project.
One of the most frequently made mistakes is that our view of a system's performance and scalability can be clouded by our own experience or hearsay. Satisfying the non-functional qualities of a system is one reason why you may need to review the other decisions on your project. For example, the non-functional requirements might influence you to choose to not use a standard or to use something that might not be considered mainstream/fashionable. Non-functional requirements may invalidate religious dogma, and evidence trumps dogma.
6. Architect up front
Probably the most important principle for building scalable systems is that, if you need your system to exhibit this characteristic, you have to design it in up front. One of the pitfalls that many people (including myself) have fallen into, particularly during the early days of J2EE, was that you could build an application and have it automatically scale up and scale out. Applications designed to scale out will almost always scale up, yet applications designed to scale up will almost never scale out. Most applications can be scaled up by running them on more powerful hardware, but scaling out is a more complex problem. For example, how do you ensure that data remains consistent between application instances? And how do you make your singletons and synchronized code blocks work across processes?
Of course, thinking about this stuff up front is not necessarily the same as doing a waterfall style big design up front. Iterative and agile processes are there to help us, providing a framework in which we can do just enough design in order to be able to solve the problem. Just be pragmatic. Oh, and despite how good we think we are at designing scalable applications, it's always best to act as if you can't trust yourself and write/test code as early as possible.
7. Look at the bigger picture
Finally, remember to take the bigger picture view - look at the wood before looking at the trees. It's really easy for us to get carried away tuning components at the fine-grained code level, but ultimately it's the system as a whole that needs to be optimised. Focus on the end-to-end performance and scalability, sacrificing local optimisations if necessary. If you need to use a profiling tool to identify bottlenecks, then do so, but don't start doing this until you have a view of the end-to-end performance. Since performance is inversely related to the aggregation of all latencies throughout the system, any operation whose latency increases relative to load will become a problem. Having said that, if you're struggling to meet your performance and scalability goals, it's worth questioning whether you have chosen the right architecture. Again, look at the bigger picture and ensure that somebody is taking on the architect role.
Summary
This article has presented a number of principles and guidelines for building scalable applications, covering a number of different aspects of the software development process. The best advice I can give to anybody building scalable systems is that you need to explicitly think about and design your system. Scalability isn't magic, but it doesn't come for free. On a final note, while it might be true that faster hardware can save your ass, don't count on it!
About the article
The majority of the principles for this article have been sourced from some notes taken during a scalability discussion that took place at a private summit for architects held in London, UK, in late 2005. The summit was organized by Alexis Richardson, Floyd Marinescu, Rod Johnson, John Davies, and Steve Ross-Talbot. The video entitled "JP Rangaswami on open source in the enterprise & the future of information" is also from the summit.
About the author
Simon is a hands-on software architect who works within Detica's Global Financial Markets group. Simon has been involved in projects ranging from desktop clients and web applications through to highly scalable distributed systems and service-oriented architectures (SOA). His specialist technology is Java and, as a hands-on technical authority, he's called upon to advise and shape solutions; defining, delivering and assuring that the chosen architecture is fit for purpose and meets the non-functional requirements. Simon has written and co-written a number of books about Java EE web technologies, spoken at several conferences and founded Coding the Architecture - a website that presents a practical and pragmatic view of software architecture.