BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ
News Articles Presentations Podcasts Guides

Topics

Choose your language

QCon Plus
QCon Plus

Make the right decisions by uncovering how senior software developers at early adopter companies are adopting emerging trends. Register Now.

 InfoQ Live
InfoQ Live Roundtable

Learn how to implement and manage your API projects with a security strategy and a development mindset. Register Now.

 QCon London
QCon London

Adopt the right emerging trends to solve your complex engineering challenges. Register Now.

 The Software Architects’ Newsletter
The Software Architects' Newsletter

Your monthly guide to all the topics, technologies and techniques that every professional needs to know about. Subscribe for free.

InfoQ Homepage Presentations High Throughput Stream Processing with ACID Guarantees

How To Build Payment Systems That Scale to Infinity (Live Webinar December 13, 2022) - Save Your Seat

High Throughput Stream Processing with ACID Guarantees

Bookmarks
48:46

Summary

Terence Yim from Continuuity showcases a transactional stream processing system that supports full ACID properties without compromising scalability and high throughput.

Bio

Terence Yim has been a Apache Twill committer since its incubation. He is a Software Engineer at Continuuity, responsible for designing and building realtime processing systems on Hadoop/HBase.

About the conference

Software is Changing the World. QCon empowers software development by facilitating the spread of knowledge and innovation in the developer community. A practitioner-driven conference, QCon is designed for technical team leads, architects, engineering directors, and project managers who influence innovation in their teams.

Software Development Conference | March 27-29, 2023

I am in this industry, I do this every day and each time I go to QCon I learn about new things that I want to take back to my teams.

Randy ShoupConsulting CTO (formerly @eBay, @Google Calendar, @Stitch Fix)
Check out QCon London

Recorded at:

Dec 04, 2014

This content is in the Streaming topic

Related Topics:

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Community comments

  • I was disappointed by this talk...

    by Edward Sargisson,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    The speaker describes a system where they built a Multi-Version Concurrency Control layer on top of Hive and then implemented transactional queueing on top of that.
    They then claimed they could do a whole 100,000 events/s on 8 nodes.

    What is remarkable is not comparing that with just doing it on Postgres. While I don't have similar stats for Postgres I would expect that throughput on 1 or 2 nodes. Secondly, the speaker didn't seem to realize that queues have the feature that they're generally full or empty, rarely anywhere in between. This means that naive implementations suffer for performance because they contend on a lock that's protecting the start of the queue or the end of the queue.

    There was also not comparison with Apache Flume - which also does transactional, durable transport (but not replicated).

    Back to top

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

BT