Tracking change and innovation in the enterprise software development community
IBM WebSphere
Virtualize apps
to lower costs,
Increase agility
Learn more!
Create REST
-enabled Web Apps
with Project Zero
from IBM.
QCon San Francisco
Nov 19-21:
+ 80 sessions & tracks
Register now
and Save $400!
Terracotta:
Scale your
Apps w/ease
Offload the DB:
Learn More!
Posted by Stefan Tilkov on Feb 08, 2007 04:00 PM
Paul Fremantle, Sanjiva Weerawarana and several other WSO2 employees recently blogged about a performance benchmark which showed the open source Axis2 Web services stack’s performance to be roughly similar, and in some cases superior, to XFire. The language used in the article is rather timid:
This article shows the latest performance results of Apache Axis2 vs. Codehaus XFire, both Java implementations. The results demonstrate that modern Web Services engines can perform at very high transaction rates.
The blog entries have stressed the claimed improvement over XFire more clearly — for example, Sanjiva wrote,
Axis2 is consistently faster than XFire and is 50-70% faster as the data sizes get larger. Move over XFire, Axis2 is now the new hotrod.
XFire committer Dan Diephouse, when asked for a statement, pointed out that it’s more a benchmark of data bindings:
First, congratulations to the Axis team and their recent performance improvements in their last release. And I commend them for open sourcing their test suite so we can all run it ourselves. Second, this is really a benchmark of two different databindings - ADB vs JAXB, not XFire vs. Axis. If you look at the benchmarks they do seem to imply that XFire has less overhead when using the same databinding. But, the question many are probably wondering is - does XFire provide a higher performance databinding than JAXB? The answer is yes, JiBX. JiBX can provide significant performance improvements over JAXB as it does not use reflection, but instead does byte code generation to optimize databinding. I would encourage performance conscious users to try JiBX and to also run benchmarks for themselves.
A rather nasty exchange of blog entries was triggered by “Bileblogger” and XFire committer Hani Suleiman’s post, which prompted follow-ups by both Sanjiva Weerawarana and Paul Fremantle. Paul’s post contains results of benchmarking Axis2/ADB against XFire/JiBX:
The bottom line? Axis2 with JIBX is 2.5% faster than XFire with JIBX, and Axis2/ADB is 5% faster. These are averages - on some tests XFire was a little faster. I don’t want to make out that these are huge performance leads. They aren’t. Most user’s wouldn’t see any real difference between the stacks in terms of performance. In the article we were trying to point out that both stacks are fast and that SOAP is not slow. However, these numbers clearly show that there is no “sharp pointy plunger” in the vicinity of Axis2.
Steve Loughran linked to benchmark results for Sun’s JAX-WS 2.1 reference implementation, and questions the general usefulness of benchmarks:
When you build a service, what is your goal? Is it maximum throughput for a single node, minimum latency per call, or ability to scale well across multiple machines or multicpu systems? Similarly, when you choose a data representation for the incoming XML content, is it based purely on how well the SOAP stack parses it, or how easy is it to work with?
Scaling a Massively Multi-player Server Casestudy: Terracotta on SmartFoxServer
Hibernate without Database Bottlenecks
Scale Your Application without Punishing Your Database
Why Should I Care About Terracotta?
Terracotta 2.6 - Download now for scalability without tradeoffs
I would also like to add that after looking at their benchmarks I think the latest claim about Axis2 being 2.5% faster than XFire with JiBX is not really true. I noticed that they added in a
It's almost like they thought everyone would be too lazy to verify their results. I love it when people publish unsubstantiated FUD in an attempt to justify their own existence. That's super awesome.
Dan, I guess, you don?t know much about your own code do you? See the profiler stats on the soap:Header processing in XFire http://ws.zones.apache.org/~dims/xfire-jibx-headers/ thanks, dims
What's the practical relevance of this? Is there any case where the differences in speed measured in all of the benchmarks mentioned so far actually matters?
Who said benchmarks are relevant? :-) Personally I think Steve had some good comments in that area. It really depends on your application and what you're trying to achieve. For StAX based web services frameworks (like XFire or Axis2), without lots of WS-* stuff enabled, performance isn't likely to matter in the context of most applications. And if it does matter, make sure you chose the proper databinding toolkit as that will have the biggest affect on your performance. The people who should really care about web service performance though are the ones who are still using a DOM based web service framework (like Axis 1). By switching to a StAX based framework you're likely to see something like a 3-5x speedup (or something like 30x if you switch from rpc/encoded to doc/literal as well). This can make a significant difference in your application's responsiveness and load handling. As pointed out in the original benchmark, both frameworks can handle a decent load. Its still not as fast as binary data over JMS, but for most applications it will work just fine. Dan Diephouse, Envoi Solutions
The point I tried to make in the original article, which maybe got lost in the blogging, is that both XFire and Axis2 show that for most users SOAP is an acceptably fast solution. With older toolkits, that simply wasn't true. The other reason for benchmarking against XFire was that some users had somehow got the false impression that XFire was much faster than Axis2, which as you can see is not the case. I still think there is lots more work to do here. Dims and I will be happy to improve the benchmarks anyway that Dan or any others like. In fact we are posting them on an SVN forge so that anyone can join in. But the bottom line is that SOAP is no longer the major overhead it used to be.
The point I tried to make in the original article, which maybe got lost in the blogging, is that both XFire and Axis2 show that for most users SOAP is an acceptably fast solution. With older toolkits, that simply wasn't true. The other reason for benchmarking against XFire was that some users had somehow got the false impression that XFire was much faster than Axis2, which as you can see is not the case.
You also managed to accuse Hani of being racist. So, is it possible nowadays to criticize any software made by Sri Lankans without being accused of racism?
Regarding benchmarks: I do not care much about performance benchmarks, but I do care about software quality, development productivity and my own mental health. About half a year ago I had to choose one of the OSS web services frameworks and after evaluation of Axis/Axis2/XFire I was very happy to choose the latest as both Axis 'frameworks' were imho quite bizarre.
Have you participated in the Axis community? If so you'd know that the project is not a bunch of Sri Lankans. Even if it were, if you want to critize the project go ahead but why do you need to refer to developers' national origin for it? As far as Axis/Axis2 being "quite bizarre"- obviously you are welcome to own opinion but I'm sure you know there are hundreds of thousands of people who don't find it bizarre and use it quite happily for lots of different things.
Have you participated in the Axis community? If so you'd know that the project is not a bunch of Sri Lankans. Even if it were, if you want to critize the project go ahead but why do you need to refer to developers' national origin for it?
Why are you asking me this question? I do not care about such thing as nationality. But if Hani wants to mention "Sri Lanka" in a negative context he should be free to do that. BTW the statement "all Sri Lanka developers are bad" is racist, while the statement that "particular Sri Lanka developer is bad" is not. Do you agree?
Sorry, I used "you" in the "one" sense. Didn't mean to imply anything about you in particular! Axis2 is an open source project. IMO its not the right thing to pick on an individual developer of a project, whether the person is Sri Lankan, Japanese or American. In Hani's case he was picking on someone's ability to spell- while its great if everyone spelt the same but even Americans don't spell English correctly ;-) (or is it the English who haven't kept up with the times?!). The world has declared that American English and the real English are the right spellings but if some other spelling is given its wrong. I'm very ok with that (its better to do things the "right" way) but at the same time you don't thump a young developer with all kinds of abuse because he spelt badly. Maybe Hani was born into an English speaking environment. Many of the Sri Lankan developers who've participated in Axis2 came from very traditional homes with little English background. In fact, many spoke *ZERO* English until they entered university (where they are required to study in English). The right thing to do is to encourage people to improve their English, not ridicule them in public. If you knew the story of that particular developer you'd know he has already achieved far more in his life than most people. That's what annoys me about Hani's racist nonsense. If not for those I'd never bother responding to his crap .. its just someone writing obviously intentionally inflamatory material for more PR. There's nothing novel in that strategy.
You know - there is a very large difference between being prejudiced and racist. Everyone has built in prejudices of some sort whether they choose to acknowledge them or not. Cutting out even more b.s. - let's be real here. We've all been witness to the overwhelming amount of mailing list posts of "I have problem X - help me solve it now please thank you!" type items that have unfortunately built up a certain prejudiced way of thinking of certain groups of developers.... If you took that and said "all Sri Lankan people are stupid with software" it would be racist...If you took it and said "the majority of Sri Lankan people I've encountered appear to be struggling with some basic development concpets" you'd be prejudiced...But only in the practical / pragmatic / obvious sense. ...(and please don't start calling me racist as I have a few "Sri Lanken" friends and know how they feel about this already...)
Alex, Sanjiva, Jesse, Politely, let's stop this discussion about nationality, it is best had offline of this site which is meant for serious technical discussion. Thanks.
Alex, Sanjiva, Jesse,
Politely, let's stop this discussion about nationality, it is best had offline of this site which is meant for serious technical discussion.
Thanks.
Mea culpa. It wasnt my intention to start such a discussion. I was more concerned about the freedom of speech.
Thin client paradigm characterized by web applications is a kludge that needs to be repudiated. Old compromises are no longer needed and it's time to move the presentation tier to where it belongs.
In this presentation filmed during QCon 2007, Jeff Sutherland, the creator of Scrum, talks about his visit at Google to do an analysis of Google's first implementation of Scrum.
In this article, Bryon Jacob and Chris Berry introduce AtomServer, their implementation of a full-fledged Atom Store based on Apache Abdera, which is now available as open source.
It is easy to think that virtualization applies only to servers. In reality the recent resurgence of the concept is also being applied to networking, storage, and application infrastructure.
In this article, Stefan Tilkov explains some of the most common anti-patterns found in applications that claim to follow a "RESTful" design and suggests ways to avoid them.
In this article, Adrien Louis and Marc Dutoo discuss the differences and relative merits of using orchestration vs. routing in a typical ESB setup, and discuss various implementation options.
Wayne Lund discusses batch processing, Spring Batch objectives and features, scenarios for usage, Spring Batch architecture, scaling, example code, failures and retrying, and the future roadmap.
Developer Jay Fields draws on his experiences as a ThoughtWorks consultant to describe effective user story estimation techniques.
14 comments
Reply