Enterprise Integration–The Seriously Nasty Stuff

product sales pitch than related to integration

There was a little bit of product pitch, but overall it was interesting to me. Having worked with accord specification and other financial specs, I found the talk entertaining. Back when I worked on trading systems, a new developer asked "why doesn't everyone implement FIX and SWIFT protocol?" Just as John says, these specs are ugly and complex. What I found interesting is how they are using spring integration to chain together transformers.

One could argue "why didn't john davies go into technical details?" Honestly, it's hard to get into details without diving into the specs. Do you really think that's going to appeal to the general public? I had to work with those specs for past jobs and honestly it wasn't joyful. Just about everyone that works with those specs finds them painful once you get into the nitty gritty. For example, try running .Net xsd.exe on accord spec and see what happens.

It was good watching a different aspect of financial systems. I wasn't aware of SWIFT/FIX/FpML etc protocols and after watching this video, i kind of started understanding the protocols & communication medium used by banks/traders. I think up-til this point he is trying to explain how to reduce the ugly details of these protocols/messages and is trying to promote Integrating Objects from C24.biz company. I think Integrated Objects api is paid too.
Will watch rest of the video and update

Interesting point was when somebody in the audience asked "How to process 2 lakh messages?". What i understand is they read and process headers and then route the messages to different channels and they are using Gemfire where messages are replicated to distributed systems and each system then might process the remaining part of the message.
Also he mentioned using C++ at one point. Increasing permgen size to load 5000 classes is eventually required. JaxB uses sax parser and is very fast but the object itself will require memory.
Anybody's opinion on this part? Was expecting more on this front....

Every single bank on the planet has to process these sorts of messages, there are very few open source options here as for SWIFT for example everything needs to be certified and that costs roughly $1m in human effort to implement the standards every year.

This presentation lasts 90 minutes, 60 of that was about integration in banking and the other 30 was a demo of a tool that anyone can download and use for free (from www.C24.biz). To use the SWIFT, ISO-20022, SEPA, FIX & FpML etc. in production you need to part with money, there is NOTHING in the open source realm that implement the SWIFT rules so I'm afraid you're stuck with sales pitches. This one is about as near as you can get to the open world as it was integrated with Mule, Service Mix, Spring Integration & Camel.

Thanks Peter, I try to entertain.

It's always a dilemma to know whether to dive into detail and walk through the code or cover the problem at a higher level hoping that the people who want details will engage afterwards, these integration issues are so specific to large enterprises (HL7, Accord, SWIFT, ASN.1, ISO-8583, ISO-20022 etc.) that unless you know what the problem space is you're going to end up walking out after the first 15 minutes and that's not good for the conference.

I have had a number of emails (to John dot Davies at Incept5 dot com) asking about this talk and I'm always happy to delve into details, supply code examples or reference to production examples.

I'm doing a shorter (60 minutes) version of this talk (without the "sales pitch") at SpringIO in Madrid on the 17th Feb if anyone's in the area.

-John-

Thanks Harsh, I think you got the idea, my goal was to educate the attendees and in this case the listeners about how we deal with the complex messaging and protocols in the financial world.

I've had a chance to do a little work outside the financial services and I found it very useful being able to apply my knowledge in these areas. The "Integration Objects" product was created as a result of our pain and those of people I worked with over the years (and decades). We must have done something right because it's been very successful and now used by the majority of the world's largest investment banks.

It is free to download and use but you will need a runtime license for production use of the complex libraries.

Regards,

-John-

For those that are not familiar with Indian numbers 2 lakh is 200,000 :-)

If we get something seriously complex to process then we often "pre-parse" the messages and then distribute them, an example would be processing a large number of complex SWIFT messages where we can only parse, validate & transform a few thousand per second per server. If we need a higher throughput we can simply distribute them but sometime it's more efficient to parse part of the message first and then distribute them.

When we need seriously low latency then we simply have to move to C/C++, our very fastest FIX engines, used by the lowest latency traders are all C/C++, it's the predictability as well as the latency that rules out Java. By the time we get to this stage we're into FPGA, RabbitMQ, ZeroMQ and LMAX etc.

Send me an email if I can be of further help,

-John-