Paul Fremantle, Sanjiva Weerawarana and several other WSO2 employees recently blogged about a performance benchmark which showed the open source Axis2 Web services stack’s performance to be roughly similar, and in some cases superior, to XFire. The language used in the article is rather timid:
This article shows the latest performance results of Apache Axis2 vs. Codehaus XFire, both Java implementations. The results demonstrate that modern Web Services engines can perform at very high transaction rates.
The blog entries have stressed the claimed improvement over XFire more clearly — for example, Sanjiva wrote,
Axis2 is consistently faster than XFire and is 50-70% faster as the data sizes get larger. Move over XFire, Axis2 is now the new hotrod.
XFire committer Dan Diephouse, when asked for a statement, pointed out that it’s more a benchmark of data bindings:
First, congratulations to the Axis team and their recent performance improvements in their last release. And I commend them for open sourcing their test suite so we can all run it ourselves. Second, this is really a benchmark of two different databindings - ADB vs JAXB, not XFire vs. Axis. If you look at the benchmarks they do seem to imply that XFire has less overhead when using the same databinding. But, the question many are probably wondering is - does XFire provide a higher performance databinding than JAXB? The answer is yes, JiBX. JiBX can provide significant performance improvements over JAXB as it does not use reflection, but instead does byte code generation to optimize databinding. I would encourage performance conscious users to try JiBX and to also run benchmarks for themselves.
A rather nasty exchange of blog entries was triggered by “Bileblogger” and XFire committer Hani Suleiman’s post, which prompted follow-ups by both Sanjiva Weerawarana and Paul Fremantle. Paul’s post contains results of benchmarking Axis2/ADB against XFire/JiBX:
The bottom line? Axis2 with JIBX is 2.5% faster than XFire with JIBX, and Axis2/ADB is 5% faster. These are averages - on some tests XFire was a little faster. I don’t want to make out that these are huge performance leads. They aren’t. Most user’s wouldn’t see any real difference between the stacks in terms of performance. In the article we were trying to point out that both stacks are fast and that SOAP is not slow. However, these numbers clearly show that there is no “sharp pointy plunger” in the vicinity of Axis2.
Steve Loughran linked to benchmark results for Sun’s JAX-WS 2.1 reference implementation, and questions the general usefulness of benchmarks:
When you build a service, what is your goal? Is it maximum throughput for a single node, minimum latency per call, or ability to scale well across multiple machines or multicpu systems? Similarly, when you choose a data representation for the incoming XML content, is it based purely on how well the SOAP stack parses it, or how easy is it to work with?
Community comments
Sketchy
by Dan Diephouse,
Re: Sketchy
by Davanum Srinivas,
Re: Sketchy
by Dan Diephouse,
Re: Sketchy
by Stefan Tilkov,
Re: Sketchy
by Dan Diephouse,
Both engines are very fast
by Paul Fremantle,
Re: Both engines are very fast
by Alex G,
Re: Both engines are very fast
by Sanjiva Weerawarana,
Re: Both engines are very fast
by Alex G,
Re: Both engines are very fast
by Sanjiva Weerawarana,
Re: Both engines are very fast
by Jesse Kuhnert,
enough with the nationality commentary
by Floyd Marinescu,
Re: enough with the nationality commentary
by Alex G,
awesome
by Jesse Kuhnert,
Sketchy
by Dan Diephouse,
Your message is awaiting moderation. Thank you for participating in the discussion.
I would also like to add that after looking at their benchmarks I think the latest claim about Axis2 being 2.5% faster than XFire with JiBX is not really true. I noticed that they added in a <soap:Header/> to every message. XFire is optimized for the non header case as that is what most clients will send. Sending a Header will trigger the creation of a JDOM Document to hold the headers in. This is why XFire looks slower in their benchmarks for small messages. For at least 90% of the people out there, XFire will in fact be faster.
If they want to look at scenarios that use headers, like WS-Addressing or user headers, thats great, and they might win out there, but this benchmark seems to be about the performance of moving data around without the bells and whistles.
Knowing that Axis2 had a bug for a long time where it wouldn't actually work without a soap:Header, I can only assume this was added so it Axis2 would actually work. (I don't know how you can even release a SOAP framework that doesn't work without a SOAP Header, but quality issues are something for another post...) Otherwise I would be quite suspicious that it was purposefully put there so XFire would look worse.
awesome
by Jesse Kuhnert,
Your message is awaiting moderation. Thank you for participating in the discussion.
It's almost like they thought everyone would be too lazy to verify their results.
I love it when people publish unsubstantiated FUD in an attempt to justify their own existence. That's super awesome.
Re: Sketchy
by Davanum Srinivas,
Your message is awaiting moderation. Thank you for participating in the discussion.
Dan,
I guess, you don?t know much about your own code do you? See the profiler stats on the soap:Header processing in XFire
ws.zones.apache.org/~dims/xfire-jibx-headers/
thanks,
dims
Re: Sketchy
by Dan Diephouse,
Your message is awaiting moderation. Thank you for participating in the discussion.
For me, header building takes 7% of the invocation time for the case of a simple invocation.
Maybe you are testing a large request?
Re: Sketchy
by Stefan Tilkov,
Your message is awaiting moderation. Thank you for participating in the discussion.
What's the practical relevance of this? Is there any case where the differences in speed measured in all of the benchmarks mentioned so far actually matters?
Re: Sketchy
by Dan Diephouse,
Your message is awaiting moderation. Thank you for participating in the discussion.
Who said benchmarks are relevant? :-) Personally I think Steve had some good comments in that area. It really depends on your application and what you're trying to achieve.
For StAX based web services frameworks (like XFire or Axis2), without lots of WS-* stuff enabled, performance isn't likely to matter in the context of most applications. And if it does matter, make sure you chose the proper databinding toolkit as that will have the biggest affect on your performance.
The people who should really care about web service performance though are the ones who are still using a DOM based web service framework (like Axis 1). By switching to a StAX based framework you're likely to see something like a 3-5x speedup (or something like 30x if you switch from rpc/encoded to doc/literal as well). This can make a significant difference in your application's responsiveness and load handling.
As pointed out in the original benchmark, both frameworks can handle a decent load. Its still not as fast as binary data over JMS, but for most applications it will work just fine.
Dan Diephouse, Envoi Solutions
Both engines are very fast
by Paul Fremantle,
Your message is awaiting moderation. Thank you for participating in the discussion.
The point I tried to make in the original article, which maybe got lost in the blogging, is that both XFire and Axis2 show that for most users SOAP is an acceptably fast solution. With older toolkits, that simply wasn't true. The other reason for benchmarking against XFire was that some users had somehow got the false impression that XFire was much faster than Axis2, which as you can see is not the case.
I still think there is lots more work to do here. Dims and I will be happy to improve the benchmarks anyway that Dan or any others like. In fact we are posting them on an SVN forge so that anyone can join in.
But the bottom line is that SOAP is no longer the major overhead it used to be.
Re: Both engines are very fast
by Alex G,
Your message is awaiting moderation. Thank you for participating in the discussion.
You also managed to accuse Hani of being racist. So, is it possible nowadays to criticize any software made by Sri Lankans without being accused of racism?
Regarding benchmarks: I do not care much about performance benchmarks, but I do care about software quality, development productivity and my own mental health. About half a year ago I had to choose one of the OSS web services frameworks and after evaluation of Axis/Axis2/XFire I was very happy to choose the latest as both Axis 'frameworks' were imho quite bizarre.
Re: Both engines are very fast
by Sanjiva Weerawarana,
Your message is awaiting moderation. Thank you for participating in the discussion.
Have you participated in the Axis community? If so you'd know that the project is not a bunch of Sri Lankans. Even if it were, if you want to critize the project go ahead but why do you need to refer to developers' national origin for it?
As far as Axis/Axis2 being "quite bizarre"- obviously you are welcome to own opinion but I'm sure you know there are hundreds of thousands of people who don't find it bizarre and use it quite happily for lots of different things.
Re: Both engines are very fast
by Alex G,
Your message is awaiting moderation. Thank you for participating in the discussion.
Why are you asking me this question? I do not care about such thing as nationality. But if Hani wants to mention "Sri Lanka" in a negative context he should be free to do that. BTW the statement "all Sri Lanka developers are bad" is racist, while the statement that "particular Sri Lanka developer is bad" is not. Do you agree?
Re: Both engines are very fast
by Sanjiva Weerawarana,
Your message is awaiting moderation. Thank you for participating in the discussion.
Sorry, I used "you" in the "one" sense. Didn't mean to imply anything about you in particular!
Axis2 is an open source project. IMO its not the right thing to pick on an individual developer of a project, whether the person is Sri Lankan, Japanese or American. In Hani's case he was picking on someone's ability to spell- while its great if everyone spelt the same but even Americans don't spell English correctly ;-) (or is it the English who haven't kept up with the times?!).
The world has declared that American English and the real English are the right spellings but if some other spelling is given its wrong. I'm very ok with that (its better to do things the "right" way) but at the same time you don't thump a young developer with all kinds of abuse because he spelt badly.
Maybe Hani was born into an English speaking environment. Many of the Sri Lankan developers who've participated in Axis2 came from very traditional homes with little English background. In fact, many spoke *ZERO* English until they entered university (where they are required to study in English). The right thing to do is to encourage people to improve their English, not ridicule them in public. If you knew the story of that particular developer you'd know he has already achieved far more in his life than most people.
That's what annoys me about Hani's racist nonsense. If not for those I'd never bother responding to his crap .. its just someone writing obviously intentionally inflamatory material for more PR. There's nothing novel in that strategy.
Re: Both engines are very fast
by Jesse Kuhnert,
Your message is awaiting moderation. Thank you for participating in the discussion.
You know - there is a very large difference between being prejudiced and racist. Everyone has built in prejudices of some sort whether they choose to acknowledge them or not.
Cutting out even more b.s. - let's be real here. We've all been witness to the overwhelming amount of mailing list posts of "I have problem X - help me solve it now please thank you!" type items that have unfortunately built up a certain prejudiced way of thinking of certain groups of developers....
If you took that and said "all Sri Lankan people are stupid with software" it would be racist...If you took it and said "the majority of Sri Lankan people I've encountered appear to be struggling with some basic development concpets" you'd be prejudiced...But only in the practical / pragmatic / obvious sense. ...(and please don't start calling me racist as I have a few "Sri Lanken" friends and know how they feel about this already...)
enough with the nationality commentary
by Floyd Marinescu,
Your message is awaiting moderation. Thank you for participating in the discussion.
Alex, Sanjiva, Jesse,
Politely, let's stop this discussion about nationality, it is best had offline of this site which is meant for serious technical discussion.
Thanks.
Re: enough with the nationality commentary
by Alex G,
Your message is awaiting moderation. Thank you for participating in the discussion.
Mea culpa. It wasnt my intention to start such a discussion. I was more concerned about the freedom of speech.