XML Overload: Bad Design or Neccesary Evil?

| by Miko Matsumura Follow 0 Followers on Aug 18, 2006. Estimated reading time: 2 minutes |

The available styles of SOAP attachment and binary payloads are expanding and adapting, and SOAP messages themselves are achieving epic sizes. Many real world SOA implementations report XML documents reaching multimegabyte sizes and beyond.

A new survey of 570 developers, sponsored by Rogue Wave Software, finds that XML documents are getting huge. Thirty Six percent of respondents report they now need to parse XML documents bigger than a megabyte in size. Half of these say they are already dealing with XML documents that exceed five megabytes in size. 58 percent reported over 6 data formats within their organization, with 25 percent of respondents using greater than 20 different formats.

The take home message is that XML document size is expanding as SOA matures. but is this a problem? How big of a problem is it? Does the survey provide any clues about the magnitude of the problem?

35 percent of respondents see greater throughput speed and 26 percent see memory efficiency as the greatest challenge to XML parsing.

The survey participants were selected from the Rogue Wave developer community and also drawn from visitors to a Microsoft .NET community site. This selection process may have created some bias in the results of the survey, however it's not intuitively clear that customers of Rogue Wave or users of .NET would have substantially different XML document sizes than the general community.

These results suggest an increase in the magnitude of the documents and some side effects from those primary causes. However, the survey does not provide a detailed understanding of the extent to which this file size increase is adversely impacting organizations. The closest available survey data in the report showed 42 percent of respondents have requirements to specifically increase the scalability of their applications and of those 15 percent need to increase scalability by more than 10X, and an additional 23 percent need to increase by between 5-10X.

The typical responses to this include throwing more hardware and networking equipment at the problem (Gilder's law suggests that bandwidth grows at a rate three times faster than compute power). In some sense the emergence of software systems that rely on such large payloads is an evolutionary function of the availability of such bandwidth.

But of course a more architectural view would be that better software design would decrease the sizes of these transmissions substantially.

The question usually boils down to a simple equation: is the time and effort of good design worth the savings in terms of network latency, complexity, hardware cost and stability of the application?


Rate this Article

Adoption Stage

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

5 MB XML Means Not Using SOA by Kris Huggins

"The take home message is that XML document size is expanding as SOA matures"

I would speculate that these 5MB files are the traditional files you see in an highly-latent, batch-oriented system integration style except the file format is xml instead of proprietary.

If the file is transferred over port 80 it might make it a web service by someone's definition, but this certainly is not SOA. With SOA you will more likely see real-time, low-latency, message oriented system integration. This style by default means you'll have many more, much smaller xml files flying around as opposed to a single file with all of the "messages". This allows you to communicate things real time instead of all at once some time in the future.

Re: 5 MB XML Means Not Using SOA by Stefan Tilkov

I believe that larger XML documents are actually a good sign -- provided they don't include a simple structure that is repeated a few hundred or a few thousand times :-)

I get suspicious whenever I see lots of very small XML documents, as this is often an indicator for services designed with the wrong granularity.

Re: 5 MB XML Means Not Using SOA by j c

What else would they be?

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

3 Discuss

Login to InfoQ to interact with what matters most to you.

Recover your password...


Follow your favorite topics and editors

Quick overview of most important highlights in the industry and on the site.


More signal, less noise

Build your own feed by choosing topics you want to read about and editors you want to hear from.


Stay up-to-date

Set up your notifications and don't miss out on content that matters to you