BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage Presentations Event Processing at Massive Scale

Event Processing at Massive Scale

Bookmarks
47:30

Summary

Uri Cohen discusses several types of queues with their pros and cons used in financial and trading industries for highly parallelized data processing.

Bio

Uri Cohen is responsible for everything product related at GigaSpaces, and leads the definition and execution of the company’s product strategy. He has been with the company for the past four and a half years, serving in a variety of customer facing and technology roles from solutions architect through technology evangelist.

About the conference

Software is changing the world; QCon aims to empower software development by facilitating the spread of knowledge and innovation in the enterprise software development community; to achieve this, QCon is organized as a practitioner-driven conference designed for people influencing innovation in their teams: team leads, architects, project managers, engineering directors.

Recorded at:

Sep 13, 2012

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Community comments

  • Clarification

    by Cameron Purdy,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Interesting. Uri names the NYSE as the example of how large scale and high throughput these systems are, then later he claims that database technology can't keep up. To set the record straight, NYSE uses database technology to store every order and every trade -- it is the system of record for their real time market engines. I guess that means that database technology can keep up?

    There are obviously other markets (besides the NYSE) that do use different technology for their market engines. The last I heard, for example, the largest and the third largest foreign exchange markets did not rely on database technology for their real time systems of record, and instead used a combination of in-memory technology and a more custom form of durable storage.

    Peace,

    Cameron Purdy | Oracle

  • Re: Clarification

    by Ali Motaz,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Cameron Purdy, I *vaguely* remember your heated debates with the folks from gigaspaces since the days of theserverside.com

    I see you still like to debunk them (I mean that in a good spirited way)
    So does the NYSE uses oracle? what else do you know about their stack?

  • Re: Clarification

    by Wu Sheng,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Oracle is not fast enough for this.
    Most of the hedge funds are using kx KDB to process and store taq data from NYSE, but not sure if NYSE is using KDB

  • Re: Clarification

    by Cameron Purdy,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    I was trying to stay away from a vendor-specific response for two reasons: First, I don't want to disclose any confidential customer information; and second, I was trying to clarify an incorrect statement on technology, and not make it into a vendor versus vendor conversation.

    Obviously I have been known to skirmish a bit with the guys from Gigaspaces, but it's as friendly and as respectful as two long-time and fierce competitors can be ;-)

    I will try to find out what information is public, so I can respond with more detail.

    Peace,

    Cameron Purdy | Oracle

  • Re: Clarification

    by Cameron Purdy,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Oracle is not fast enough for this.


    I find that the level of one's surety is at times inversely proportional to the level of one's knowledge.

    (Needless to say, I am quite surprised that you would state such a thing unequivocally.)

    Most of the hedge funds are using kx KDB to process and store taq data from NYSE, but not sure if NYSE is using KDB


    There are many, many hedge funds. I've only personally worked with a few dozen of them, so I cannot speak authoritatively. I do know that KDB is a well-regarded solution in specific use cases in financial services, and even though I have not personally seen it in use at hedge funds, I am not surprised by the likelihood that it would be used. Here's a number of products in the general OLTP database arena that I've seen in use at hedge funds:

    1) Oracle database
    2) Microsoft SQL Server
    3) MySQL (surprisingly common)

    I haven't seen any Sybase in use at hedge funds, even though it is very common in the more established financial services businesses (because they all purchased unlimited perpetual licenses back in the early 1990s for almost nothing.)

    For analysis (OLAP etc.), I've seen a number of other more specialized solutions.

    For order books, trading systems, algo systems, etc., I've seen the use of in-memory data grids. (That's not surprising, since that is probably why I was there talking to them in the first place. ;-)

    Peace,

    Cameron Purdy | Oracle

  • The danger of quoting numbers without reading properly

    by Fernando Racca,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Uri says that calling a method in java would take 300 ms, do that with two threads contending for a lock, and you'll need a quarter of a second. Hold on a sec... a quarter of a second??? I work in a market maker, but not even the oldest computer i've ever used would take that amount of time!

    Oh, maybe if you actually go to the paper he's quoting, on the Disruptor, you can actually see that they are actually trying to increment a counter.... 500 million times!

    careful when quoting papers out of context.

    Fernando

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

BT