Article: Scalability Worst Practices

Impressive article. Some of these refer back to the documented anti patterns we find in the architecture community .

Hi Brian;



Nice job outlining some very salient issues.



Your comments on DNS lookups in Java hit a sore point I have been wrestling with. We have been doing some planning, preparation and testing for disaster recovery and business continuity fail over events and one of the issues that keeps coming up is the prototypical scenario of a Java app server that communicates with a remote service ( a DB, legacy service, messaging server etc.) that fails and the Java app must quickly switch over to what [usually] amounts to a new IP address.



Sounds simple enough, but there are some subtleties that I have been trying to definitively iron out as to what exact behaviour we can expect under different scenarios. As I understand it (and anyone please correct me if this is wrong) that the JVM absolutely does cache a DNS lookup so neither a host file update or a fully propagated DNS change will invalidate that cache. However, if the original IP address is detected as "invalid" because it is unreachable, the JVM will ditch the cached entry and refresh from the configured naming service. So in theory, provided your remote service has a cold, hard failure, your Java app should automatically pick up the new address. The problem is that frequently failovers are driven by partially impacted servers and not cold, hard failures. In these cases, the original address is still reachable and the only way to pick up the new address to to either deliberately kill the old service's IP connectivity (not always an option since this stalls triage and remediation) or recycle the Java app which extends the outage window. It would be nice if the JVM had a management interface (through JMX) that could flush all or specified cached addresses.



Having said that, even if the above was completely addressed, each app has its own issues. For example, hopefully you are using validating database connection pools which will automatically ditch failed connections and create new ones (to the failover target) in the process of handing out connections from the pool. Failing that, hopefully your connection pools have a management interface that allows you to flush the pool and repopulate.



Unfortunately, there are all too many apps around that do not have this richness built in and once a remote resource is impacted, a recycle is the only way to re-engage. The long and the short of all this is that business continuity plans, or more specifically, the detailed procedures to handle an event, are challenging to compile since they may be different for each individual application and the exact nature of the failure may further dictate what the procedure should be, making what should be an absolutely unambiguous set of procedures into a series of decision making and analysis at a time when one can least afford to do so.



Do you see it in any more black-and-white than I do ?



Cheers.



//Nicholas

When a security manager is not set, the default for networkaddress.cache.ttl is 30 seconds, at least, this is the default behaviour on modern Sun JDKs.

How do someone is able to run any modern application server with a security manager installed, when application server vendors don't even care to provide a starting policy file that will grant the minimal set of grant declarations to start their servers is beyond my understanding.

As for the failover options, as long as the libraries have configurable timeouts, you could go on with SRV records on your DNS, instead of As or CNAMEs. Google for "DNS SRV FAILOVER". SRV records were designed exactly to address this issue.

You can find a good description on SRV records and their usage here on:

www.zytrax.com/books/dns/ch8/srv.html

Anyway, it's a good article. It's good to see people starting to realize that caring for non-functional as performance and scalability can really make or save your day.

Other day, after a very long session of profiling and code archaeology, I crafted a phrase, which I pledge the readers to take with some sense of humour before crucifying me.



The Great Knuth said:
"Early Optmization is the root of all evil"
But Marcos Eliziario, who is a poor programmer, known of no one, said at 2 AM after two sleepless days:
"Reckless regard for optimization is the cube of an even greater evil"

Especially for .NET apps, apart from making a stateless app, it is always a best practice to host app/web service under IIS in windows 2003 where the platform enables technologies for scalability. This is pretty easy under Network Load Balancers for scale-out solution. Actually, developer has to focus on a cluster-aware applications to automate the configuration when a new node joins the cluster. In fact, more development effort is needed to provide scale-up solution rather than scale-out. On another ocation, I observed that the developers are spending time to do SSL with code. They should take advantage of IIS configuration which is easy to manage. After all, we don't want to write another framework or operating system. I think that developers should focus on functionality and improve presentation techniques.

I'd echo the praise of the previous commenters. Major work recently in my world has included introducing/improving/increasing monitoring, removing runtime dependencies which cause untold problems, pulling apart the big-ball-of-mud build and deployment process, even the way monitoring goes by the wayside and the JVM DNS-cache thing has come up recently during a disaster-recovery test as it did for Nicholas above - pretty much all the stuff you've described was familiar so I got that warm fuzzy feeling reading each paragraph.

Anyway, just wanted to say it hit the nail on the head for me - I look forward to your next article.

Regards.