InfoQ

News

XML Overload: Bad Design or Neccesary Evil?

Posted by Miko Matsumura on Aug 18, 2006 01:17 PM

Community
SOA
Topics
Messaging ,
Research
Tags
XML

The available styles of SOAP attachment and binary payloads are expanding and adapting, and SOAP messages themselves are achieving epic sizes. Many real world SOA implementations report XML documents reaching multimegabyte sizes and beyond.

A new survey of 570 developers, sponsored by Rogue Wave Software, finds that XML documents are getting huge. Thirty Six percent of respondents report they now need to parse XML documents bigger than a megabyte in size. Half of these say they are already dealing with XML documents that exceed five megabytes in size. 58 percent reported over 6 data formats within their organization, with 25 percent of respondents using greater than 20 different formats.

The take home message is that XML document size is expanding as SOA matures. but is this a problem? How big of a problem is it? Does the survey provide any clues about the magnitude of the problem?

35 percent of respondents see greater throughput speed and 26 percent see memory efficiency as the greatest challenge to XML parsing.

The survey participants were selected from the Rogue Wave developer community and also drawn from visitors to a Microsoft .NET community site. This selection process may have created some bias in the results of the survey, however it's not intuitively clear that customers of Rogue Wave or users of .NET would have substantially different XML document sizes than the general community.

These results suggest an increase in the magnitude of the documents and some side effects from those primary causes. However, the survey does not provide a detailed understanding of the extent to which this file size increase is adversely impacting organizations. The closest available survey data in the report showed 42 percent of respondents have requirements to specifically increase the scalability of their applications and of those 15 percent need to increase scalability by more than 10X, and an additional 23 percent need to increase by between 5-10X.

The typical responses to this include throwing more hardware and networking equipment at the problem (Gilder's law suggests that bandwidth grows at a rate three times faster than compute power). In some sense the emergence of software systems that rely on such large payloads is an evolutionary function of the availability of such bandwidth.

But of course a more architectural view would be that better software design would decrease the sizes of these transmissions substantially.

The question usually boils down to a simple equation: is the time and effort of good design worth the savings in terms of network latency, complexity, hardware cost and stability of the application?

Discuss.

5 MB XML Means Not Using SOA by Kris Huggins Posted Aug 18, 2006 3:40 PM
Re: 5 MB XML Means Not Using SOA by Stefan Tilkov Posted Aug 18, 2006 5:00 PM
Re: 5 MB XML Means Not Using SOA by j c Posted Aug 18, 2006 8:59 PM
  1. Back to top

    5 MB XML Means Not Using SOA

    Aug 18, 2006 3:40 PM by Kris Huggins

    "The take home message is that XML document size is expanding as SOA matures" I would speculate that these 5MB files are the traditional files you see in an highly-latent, batch-oriented system integration style except the file format is xml instead of proprietary. If the file is transferred over port 80 it might make it a web service by someone's definition, but this certainly is not SOA. With SOA you will more likely see real-time, low-latency, message oriented system integration. This style by default means you'll have many more, much smaller xml files flying around as opposed to a single file with all of the "messages". This allows you to communicate things real time instead of all at once some time in the future.

  2. Back to top

    Re: 5 MB XML Means Not Using SOA

    Aug 18, 2006 5:00 PM by Stefan Tilkov

    I believe that larger XML documents are actually a good sign -- provided they don't include a simple structure that is repeated a few hundred or a few thousand times :-) I get suspicious whenever I see lots of very small XML documents, as this is often an indicator for services designed with the wrong granularity.

  3. Back to top

    Re: 5 MB XML Means Not Using SOA

    Aug 18, 2006 8:59 PM by j c

    What else would they be?

Educational Content

Bindings, Platforms, and Innovation

This presentation focuses on the Internet and separating myth from fact, history from the future, and the mundane from the imaginative. Bob Frankston presents a vision of what could and should be.

Orchestrating Long Running Activities with JBoss / JBPM

This article explores the use of JBoss and jBPM to implement design solutions that effectively address the issue of orchestrating long running activities.

Neo4j - The Benefits of Graph Databases

This presentation covers the use of graph databases as an optimal solution for data that is difficult to fit in static tables, rapidly evolving data or data that has a lot of optional attributes.

Realistic about Risk: Software development with Real Options

This session introduces Real Options and shows how it can help in running your project. Real Options is a decision-making process that can be used to manage risk.

Communication Flexibility Using Bindings

This article discusses the use of bindings on services and references (including the instance of non-configured bindings) as the means to implement SCA communications in a Web and SOA environment.

Writing DSLs in Groovy

After a short introduction to DSLs, Scott Davis plays with the keyboard showing how to approach the creation of a DSL by typing working snippets of Groovy code that get executed.

Scaling Agile with C/ALM (Collaborative Application Lifecycle Management)

IBM Rational and InfoQ present, Scaling Agile with C/ALM, an eBook showing organizations how to become “finely tuned software delivery machines” by enabling team integration and scaling.

Concurrent Programming with Microsoft F#

Amanda Laucher presents a real life enterprise application written in F#. She shows actual code snippets, explaining design decisions and suggesting how to use some of the F# constructs.