Key Takeaway Points and Lessons Learned from QCon San Francisco 2015

| Posted by Abel Avram Follow 9 Followers on Dec 16, 2015. Estimated reading time: 41 minutes | NOTICE: The next QCon is in San Francisco Nov 5 - 9, 2018. Save an extra $100 with INFOQSF18!

At the start of November around 1,300 attendees descended on the Hyatt Regency in San Francisco for the ninth annual QCon in the city and the biggest QCon we’ve held in the US.

QCon SF attendees - software engineers, architects, and project managers from a wide range of industries including some prominent Bay-area companies - attended 94 technical sessions across 6 concurrent tracks, 18 in-depth workshops, facilitated open spaces and, as at all English language QCons, had instant access to all filmed presentations from the event on InfoQ.

The conference featured keynotes from Bill Buxton, Principle Researcher at Microsoft, a joint presentation from Ines Sombra, Engineer at Fastly, & Caitie McCaffrey, Distributed Systems Engineer at Twitter on large-scale distributed systems, Matt Ranney, Chief Systems Architect at Uber and Co-founder of Voxer, and Shuman Ghosemajumder VP Product Management at Shape Security on the evolution of AI based security attacks.

This article summarizes the key takeaways and highlights from QCon San Francisco 2015 as blogged and tweeted by attendees. Over the course of the next 5 months, InfoQ will be publishing most of the conference sessions online, including 10 video interviews that were recorded by the InfoQ editorial team. The publishing schedule can be found on the QCon San Francisco web site.


Tracks and Talks

Applied Machine Learning

Architecting for Failure

Architectures You've Always Wondered About

Browser as a Platform (Realizing HTML5)

Containers in Practice

Culture Reimagined

Design Thinking

Modern CS in the Real World

Modern Languages in Practice

Org Hacking

Taming Distributed Architecture

Unlearning Performance Myths

Taking Java to the next Level

The Amazing Potential Of .NET Open Source

China Tech Day

Solutions Track

Opinions about QCon





Avoiding the Big Crash

by Bill Buxton

Twitter feedback on this keynote included:

@ramrubio: Technology is not moving fast. There are just a lot of things moving slowly. #qconsf keynote @wasbuxton

@charleshumble: Think of Windows 8 on a large display. Actually please don't! Bill Buxton #qconsf

@philip_pfo: The extreme form factors have more in common than the middle. Engineer for the human first then the form factor. #qconsf @wasburton

@charleshumble: The cumulative complexity of a bunch of simple things exceeds are ability to cope with it. Bill Buxton #qconsf

@philip_pfo: The next big thing isn't an app/service/device, it is a change in the interactions of what is already there. #qconsf keynote @wasburton

@charleshumble: Every new product and service must provide great experience and excellent value - it works, flows and has demand. Bill Buxton #qconsf 1-2

@TonyPrintezis: I think your computer should be smarter than your toilet -- @wasbuxton #QConSF

@dberlind: Microsoft principal researcher @wasbuxton @ #QconSF: All $billion industries take 20 years to get to a $billion and are stealth for 1st 15.

@charleshumble: Devices need seamless aggregation and disaggregation, and graceful augmentation and degradation of capabilities. Bill Buxton #qconsf

@jessitron: Never beat yourself up for being human. Recognize it and take necessary precautions. @wasbuxton #qconsf

@NickVido: Who cares if we're connected if there is no sense of community? -The Legendary Bill Buxton #qconsf

@jitterted: Making things less complex for users is hard, making cross-device usage seamless is going to be really hard. (@wasbuxton keynote) #QConSF

So We Hear You Like Papers

by Ines Sombra & Caitie McCaffrey

Twitter feedback on this keynote included:

@infosec812: Reviewing #CAP theorem reminds us that designing for available systems is hard #qconsf

@craigsmitham: CAP told us to keep working on how to build more correct eventually consistent systems @caitie #qconsf

@craigsmitham: Applications must be aware of and integrally involved in conflict detection and resolution -D. Terry #qconsf

@jitterted: Actually have to ship stuff, not just sit around and think about concurrency  @caitie at #QConSF

@jitterted: Yes: don't recreate the wheel/re-research what's already been done, learn from history.  @caitie #QConSF

@bridgetkromhout: If you don't write tests cause you're a "good engineer", don't say this to @randommood or she may punch you. #qconsf

@randommood: Slides for our keynote "I hear you like papers" with the loveliest @caitie link  #qconsf

The Future of Build vs. Buy

by Matt Ranney

Twitter feedback on this keynote included:

@bridgetkromhout: The computing empires of today are also unlikely to endure forever @mranney #qconsf

@pbochynski: Today you can get millions of users overnight. @mranney #qconsf

@redsneaker: The cloud is a lot like renting mainframes from IBM in the 60's. #QConSF

@orrsella: Airbnb is not in the service discovery business! @mranney on Build v. Buy #QConSF. So good.

@bridgetkromhout: At a certain scale, there's no shelf you can get things from; the shelf is empty. @mranney

@__3ve: We would never make any progress if we didn't reinvent the wheel, says @mranney. #qconsf

@redsneaker: The biggest problem in making decisions is people. #QConSF

@sarah_shewell: Build or Buy - we're here because we want to create, not maintain someone else's code. If you're having fun, you'll work better #qconsf

@bridgetkromhout: The pace of our industry means facing our cognitive biases & emotions when making infra decisions. @mranney #qconsf

@sarah_shewell: If you build something and know how it works you will be the BEST team in the world to fix the problem @mranney #qconsf

@redsneaker: Build and easily support or buy and be at the mercy of the vendor #QConSF

@bridgetkromhout: Infrastructure software is going to be delivered as a service - the ultimate commoditization. @mranney #qconsf

@soumit: @mranney argues against a blind #NIH attitude towards buy over build. #qconsf

@jon_moore: .@mranney, in his #qconsf keynote: "People expect Netflix to be as reliable as TV." ;)

@jnorthcott: when you build, try to build up the stack and try to build open source. @mranney #qconsf

@ollie_ona: Cloud computing has made hardware a true commodity. @mranney at #qconsf keynote Build vs. Buy

The Imitation Game: The New Frontline of Security

by Shuman Ghosemajumder

Twitter feedback on this keynote included:

@jbillybytes: $110B/yr in damage done by botnet based attacks, Shuman Ghosemajumder keynote, #qconsf

@orrsella: "The Imitation Game" keynote by Shuman Ghosemajumder: "Sony was hacked so many times we need to say which year we're talking about" #QConSF

@DC_LAN: Every day humanity spends 17 person year solving CAPTCHAs #qconsf

@__3ve: Really wish I could ask all the devs if they've considered incorporating security auditing code when they develop their framework. #qconsf

@DC_LAN: Password is the worst kind of authentication, except for all the other authentication methods. #qconsf

Tracks and Talks

Applied Machine Learning

Beyond Ad-hoc Data Science

by Oscar Boykin

Twitter feedback on this session included:

@BigDataSc: Data scientists see themselves as Tom Cruise in Minority Report, using their hands to move data from side to side - @posco #qconsf

@BigDataSc: Twitter has 100s - 1000s of jobs with complex dependencies - @posco #qconsf

@BigDataSc: This is the typical data job input -> Function -> Output - @posco #qconsf

@BigDataSc: Whether you like it or not, data is functional programming! - @posco #qconsf

@BigDataSc: No one gets into data science because they are excited about Continuous Integration - @posco #qconsf

@BigDataSc: Machine Learning data flows don't have "private modifiers" like OO languages, someone can become a dependency of your flow - @posco #qconsf

@charleshumble: Data dependency has poor encapsulation relative to OO programming so you end up with deep data dependencies. @posco #qconsf

@BigDataSc: Rigor is not a bureaucratic process to make you run slower, it's about communication and will make you run faster - @posco #qconsf

@charleshumble: Twitter scale: O(1000) batch jobs, O(100) streaming jobs. Things break without some rigour. @posco #qconsf

@BigDataSc: Logic always requires some implicit or explicit schema, an explicit one is much better - @posco #qconsf

@BigDataSc: I'm a jack when it comes to SQL - @posco #qconsf

@charleshumble: At Twitter, all data has a schema which can be expressed using Thrift. Written to HDFS/Kafka, MySQL etc. @posco #qconsf

@BigDataSc: Data that is not changing only happens in our Utopian world of Data Hygiene - @posco #qconsf cc @dpatil

@r39132: Twitter's data pipeline only allows optional fields to support fault-tolerant schema evolution @posco #qconsf

@BigDataSc: Today's build tools are not equipped to compare changes between versions, only the last version - @posco #qconsf

@charleshumble: I'm showing a distributed data job that is not word count. Thank me after the talk. @posco #qconsf

@jessitron: Step 1 for reliable data science: all jobs are pure functions.@posco #qconsf

@jessitron: At Twitter, the system refuses to run a job if it hasn't been deployed in over 90 days.via @posco #qconsf #NoLegacy

Dino DNA! Health Identity from the Wrist @jawbone

by Brian Wilt

Twitter feedback on this session included:

@charleshumble: Woman sleep 20 more minutes on average than men across any section of the population. No-one is sure why. @brianwilt #qconsf

@charleshumble: 4 best data products - LinkedIn People You May Know, FB newsfeed, PageRank, and Amazon Related Items - are all about CTR @brianwilt #qconsf

@BigDataSc: The amount of data that comes from a single @JawboneUP3 band is GBs/day - @brianwilt #qconsf

@BigDataSc: It's a miracle if you can run Fourier transforms on a wrist band ... but we do it - @brianwilt #qconsf

@BigDataSc: Tesla could not ship a bicycle as their MVP product, they had to ship a car - @brianwilt #qconsf

@BigDataSc: Jawbone used their wrist bands to monitor heart rate at a company all hands @brianwilt #qconsf Genius!

Personalization in the Pinterest Homefeed

by Dmitry Chechik

Twitter feedback on this session included:

@BigDataSc: Pinterest Homefeed problems: Generating candidates, scoring and ranking, blending, creating the final feed - @dmitrychechik #qconsf

@BigDataSc: Pinnability - the name of the output of Pinterest's machine learning model - @dmitrychechik #qconsf

@BigDataSc: co-pins of images on Pinterest creates a very rich graph to extract information from - @dmitrychechik #qconsf

@BigDataSc: Origami, Russian posters and Places to visit in San Francisco is the definition of @dmitrychechik as a Pinterest user #qconsf

@BigDataSc: The scale of recommendations of Pinterest is several orders of magnitudes larger than those of Netflix and Pandora - @dmitrychechik #qconsf

@BigDataSc: Pinterest's content is evergreen, an image from years ago might be as relevant today. not true for social networks - @dmitrychechik #qconsf

@charleshumble: Pinterest working with around 1b pins. Many are evergreen. More than movies (Netflix) or Songs (Pandora). @dmitrychechik #qconsf

@BigDataSc: Pretty amusing to see how @Salesforce @Intuit and @Pinterest all have similar modeling frameworks #qconsf

@charleshumble: Pinterest has a DSL for working with end-to-end Machine Learning flow. It looks quite R-like. @dmitrychechik #qconsf

@charleshumble: Pinterest DSL allows same model to be used for off-line training and on-line classifier. @dmitrychechik #qconsf

@charleshumble: Interesting that Pinterest ML models might be different for international users vs. new users. @dmitrychechik #qconsf

@charleshumble: Encapsulate as much of the model process as possible, and separate code from model config. @dmitrychechik #qconsf

Takes a Village to Raise a Machine Learning Model

by Lucian Vlad Lita

Twitter feedback on this session included:

@BigDataSc: more smart data is better than purple data - @datariver #qconsf via @bigdatasheep

@BigDataSc: The next frontier of data intensive products is well designed software architerctures - @datariver #qconsf

@BigDataSc: An amazing % of US GDP flows through the @Intuit systems - @datariver #qconsf

@BigDataSc: Awesome way to present personalization - restaurant food choices - @datariver #qconsf

@BigDataSc: Having your data app connect directly to the database without middleware is not a great pattern - @datariver #qconsf

@BigDataSc: If you have many models running in production, you need model versioning - @datariver #qconsf

@BigDataSc: It appears that Intuit has hundreds of A/B tests running on their website at the same time #qconsf

@BigDataSc: Allowing an easy way for non-technical people to run experiments is a great step towards building a data driven culture @datariver #qconsf

@BigDataSc: Storing history is overrated for most data products - @datariver #qconsf

@BigDataSc: Storing history is overrated for most data products - @datariver #qconsf

@BigDataSc: Ask yourself whether there is a business justification to make all of your data products highly available - @datariver #qconsf

@BigDataSc: Common pattern for explanatory data models, use the best model for prediction, but decision trees for your customers - @datariver #qconsf

@BigDataSc: Personalize your personalization platform - @datariver #qconsf

The Lego Model for Machine Learning Pipelines

by Leah McGuire

Twitter feedback on this session included:

@BigDataSc: Spark and Scala is the way to go for production machine learning pipelines - Leah McGuire #qconsf

@BigDataSc: Scala was given to us as an instructions from above, "My boss said, you'll use Scala and you'll love it" - Leah McGuire #qconsf Guilty :)

@BigDataSc: Monolithic machine learning frameworks are not a good fit if you already have parts of it, which is most cases - Leah McGuire #qconsf

@BigDataSc: The generic Machine Learning pipeline - Leah McGuire #qconsf

@BigDataSc: The @salesforce Machine Learning pipeline named Optimus Prime, because it's a collection of bunch of Transformers - Leah McGuire #qconsf

@BigDataSc: We extended Twitter's #algebird aggregators to power our Feature Aggregators - Leah McGuire #qconsf cc @posco

@BigDataSc: The modeling, which is the step everyone thinks about in Machine Learning, ends up being the shortest step - Leah McGuire #qconsf

Architecting for Failure

Architecting for Failure Induced by Human Errors

by Amos Barreto

Twitter feedback on this session included:

@philip_pfo: Make changes with confidence by minimizing downside thru:1. simplicity2. isolation3. process4. anti-fragility@amos_barreto #qconsf

@johnscattergood: Uber creates isolation by not colocating critical and non-critical components on the same host or database #qconsf

@johnscattergood: Enforced peer review are good examples of good process #qconsf

@johnscattergood: 99.99% availability means that you have 5 minutes to solve a problem if you have 1 incident a month #qconsf

@johnscattergood: Every production outtage is an opportunity to learn, if you are willing to take it... #qconsf

@johnscattergood: Let's take the downside out of the risk #qconsf

@agonigberg: OMAR allows Uber to automatically rollback/take action based on anomaly detection. Removes humans from critical path. @amos_barreto #qconsf

Building Highly-resilient Systems at Pinterest

by Yongsheng Wu

Twitter feedback on this session included:

@orrsella: Pinterest at a glance: 10s of 1000s of AWS instances, Scala, Finagle, Go, 100 services, 1M QPs. Wow. #QConSF

@Y2F00: #qconsf @yswu @Pinterest has +100 services written in python, java, go, c++, we gain a lot of performance with @golang

@johnscattergood: Areas on concern for building resilient services at Pinterest #qconsf

@johnscattergood: Pinterest uses ZooKeeper for service registry and dynamic discovery #qconsf

@johnscattergood: Pinterest added McRouter component to delegate to different cache servers and handle failures #qconsf

@johnscattergood: Pinterest fronts databases with data services to provide availability through MySQL slaves in case of MySQL master failure #qconsf

@johnscattergood: Advantages of Async (Reactive) processing #qconsf

Crossroads of Asynchrony and Graceful Degradation

by Nitesh Kant

Twitter feedback on this session included:

@johnscattergood: Synchronous APIs simply do not scale, instead embrace Asynchronous interactions #qconsf

@philip_pfo: Empathetic systems have knowledge about their peers and adapt to those signals. @NiteshKant #qconsf

@johnscattergood: Key aspects of Async systems: lifecycle control, flow control, and function composition #qconsf

@johnscattergood: Async architectures extend to the network layer with multiplexed bi-directional communication #qconsf

@johnscattergood: is an approach to asynchronous HTTP request processing including "request leases" #qconsf

@philip_pfo: .@NiteshKant convinced me- async all the things to improve avail and reduce config complexity. #qconsf

Architectures You've Always Wondered About

Alibaba Mobile Infrastructure at "China Scale"

by Zhuoran Zhuang & Zhao Xu

Twitter feedback on this session included:

@orrsella: 80K payment transaction per sec on Alibaba’s 11.11 sale day #QConSF

@jkuner: Wow! 11.11 shopping festival generated 14.3Billion dollars, 249M online customers, and 68% on mobile! #qconsf

@Serginio89: Shopping.. Chinese scale - Alibaba alone is bigger than all US retailers combined - #qconsf

@__3ve: Just an example of the scale of China's 11.11 Day versus US's Black Friday... #qconsf #alibaba

@r39132: Taobao's 11.11 dwarfs Black Friday. $1.6B made in first 12 mins & 100 countries purchased in 1st min. #qconsf

How Netflix Directs 1/3rd of Internet Traffic

by Haley Tucker &Mohit Vora

Twitter feedback on this session included:

@agonigberg: Netflix has 3 main components: device, data plane and control plane. Implemented using hundreds of microservices. #qconsf

@orrsella: “We store the entire world’s routing information to optimize video delivery to our viewers” – How Netflix directs Internet traffic #QConSF

@rmacdonaldsmith: 2 out of 2 talks describe the use of a consistent hash to solve scale problems; Uber and Netflix #qconsf

Scaling Uber

by Matt Ranney

Twitter feedback on this session included:

@johnscattergood: Uber has 1500 engineers. 30% of there workforce! #qconsf

@orrsella: “More than half of our engineers have been with the company for less than 5 months” – Scaling Uber by @mranney #QConSF

@jitterted: No “casual” Uber app usage, very different usage than Twitter or Facebook. (Matt Ranney #QConSF)

@adidasGuy: #qconsf - - Uber - every year we just barely make it through New Years

@infosec812: Lots of @Uber engineers don't go out on New Years Eve @MattRanney during #QConSF

@johnscattergood: Uber's system has rapidly changed to support its growth, but this speed has created a lot of technical debt that needs to managed #qconsf

@r39132: @Uber is wrestling with 700+ micro services deployed to Production and call graph spaghetti #qconsf

@pbochynski: Should microservices become immutable after initial baking period?@mranney at #qconsf

@johnscattergood: Uber engineering tacitly supports 4 server side languages: JavaScript(node.js), Python, Go, Java #qconsf

@johnscattergood: Uber is moving away from HTTP and JSON for internal communication due to performance #qconsf

@infosec812: @MattRanney "#Thrift is OK, but generated code is bad" - Been there, done that! #qconsf

@chrisTesene: What do you do when http is too slow? Well if you are @uber you write your own rdp protocol #qconsf

@orrsella: “We opt for availability over consistency every time we can” – @mranney #QConSF

@johnscattergood: Uber culture changed to successfully deal with constant chaos #qconsf

Browser as a Platform (Realizing HTML5)

New in ECMAScript 2016 and Beyond

by Brian Terlson

Paul Krill attended this session:

Under a plan to have ECMAScript upgraded each year, the 2016 version would be available June 16 of next year, said Brian Terlson a Microsoft senior program manager  serving on ECMA Technical Committee 39, which is developing the specification. But Terlson, speaking at the QCon conference in San Francisco on Monday, cautioned that all feature plans are still tentative. …

"Async functions will drastically change the way you write JavaScript code," said Terlson…

ECMAScript 2016 also is on track to include a SIMD (single instruction, multiple data) numeric type, which would help exploit data-level parallelism. "It's strictly the ability to process more data and to process it simultaneously in a single thread," said Terlson. …

The 2016 specification will also include exponentiation operator capability, which takes a base number and offers the exponent of it. PadLeft and PadRight, meanwhile, would pad out strings to fill out a certain number of characters.

Features under consideration for future ECMAScript editions include class property declarations and decorators, providing a declarative syntax for properties….

An Observables type for modeling push-based data sources such as DOM events, timer intervals, and sockets may be introduced to the standard library in ECMAScript 2017 or beyond. "Observables really help lay the foundation for a new syntax," Terlson said. Value types, such as int64 bignum, decimal, and complex, are under consideration as well.

One capability that could be a long way off, though, is JavaScript types. "It's an active area of investigation," said Terlson, who estimated a time frame of five years before developers can start talking about it. "It's a hard problem -- a very hard problem."

Containers in Practice

Beyond Ad-hoc Automation: To Structured Platforms

by Bridget Kromhout

Twitter feedback on this session included:

@randommood: If you are building something local to your organization then your expertise becomes local to your organization @bridgetkromhout #qconsf

@jbillybytes: Sometimes you need to think value-add and choose 'boring software' over going the bespoke infrastructure route. #qconsf @bridgetkromhout

@jbillybytes: because you can doesn't mean you should. excellent point. @bridgetkromhout #qconsf

@bcantrill: Containers aren't going to fix your broken culture! - @bridgetkromhout breaking the tough news at #qconsf

@jon_moore: .@bridgetkromhout asks if your platform can offer these things... #qconsf

@jon_moore: .@bridgetkromhout : "Make the right thing be the easy thing" #qconsf < systems thinking!

Containers at Netflix - an Evolving Story

by Sangeeta Narayanan

Twitter feedback on this session included:

@dubmun: Netflix has some great culture ideas including highly aligned, loosely coupled development teams teams. #qconsf

@aiborisov: Freedom and responsibility is a key part of #Netflix Culture by @sangeetan. So many companies miss this! #QConSF

@philip_pfo: OpEx makes changes go live reliably, DevEx makes changes easier to make. Containers improve the dev experience. @sangeetan #qconsf

@agonigberg: Netflix data pipeline team has ~10k containers running at any given time. @sangeetan #qconsf

@agonigberg: Recurring themes for using Docker at Neflix are: infrastructure abstraction, local dev env, automation. @sangeetan #qconsf

Debugging Microservices in Production

by Bryan Cantrill

Twitter feedback on this session included:

@sangeetan: Think methodically to avoid debugging anti-patterns @bcantrill #qconsf

@ericturcotte: If you haven't seen Silicon Valley, you should. It's a great documentary. @bcantrill #qconsf

@erwin57: Software debugging is a pure distillation of scientific thinking.- Bryan Cantrill #qconsf

@_wsh: My mom used to tell me 'you can't play on that computer for the rest of your life.' Turns out you can! @bcantrill #QConSF

@khoubyari: The first non trivial program had 120 lines and 20 errors! @bcantrill #qconsf #docker

@jbillybytes: @bcantrill #qconsf Working software works forever because it is true.

@it_vegard: @bcantrill: Many "non-reproducable" bugs are merely unusual. #qconsf

@randommood: Taxonomy of software failures by @bcantrill #qconsf

@_wsh: If you draw one less about software engineering from Richard Nixon, it's 'don't cover anything up'. @bcantrill #QConSF

@redsneaker: The difference between my code and your code: My code is a Ferrari. Your code is rustheap. You should be ashamed of your code. #QConSF

@cpswan: The 3 commandments@bcantrill on top form at #qconsf

@_wsh: Let's not romanticize production. Production is war, and war is hell. It is you & us and we have no one else to blame. @bcantrill #QConSF

@jimternet: Distributed systems almost ALWAYS have to be debugged in production. Production is war. War is hell - bryan cantrill #qconsf

@klavergne: My therapist says it’s good for me to talk about this. @bcantrill on debugging at #qconsf

@portixol: 'You want the GC to take away your stinky sofa?’ @bcantrill’s talk on debugging microservices - I’ve never laughed so hard at a talk #qconsf

@dberlind: @Joyent's @bcantrill on what a microservice isn't (in his talk at #QConSF): "If you have a large microservice, then it's not a microservice"

Inside Yelp's SOA Infrastructure

by Kyle Anderson

Twitter feedback on this session included:

@dberlind: Yelp's Kyle Anderson (@solarkennedy) in his microservices talk @ #QConSF: "We call it the right-sized service not the microservice"

@ssasalatti: It's not done until it's S?h?i?p?p?e?d? monitored #qconsf

@bcantrill: Mesos out of the box is so un-opinionated that out of the box, it's almost not useful. -- Yelp's @solarkennedy at #qconsf

@sangeetan: Leverage what's out there and put it together to build a solution that works for you PaaSTA @yelp @solarkennedy #qconsf

Culture Reimagined

Creating Debt-aware Culture

by Aaratee Rao

Twitter feedback on this session included:

@WesCravens: 'If you spend a lot on production/maintenance of the software then you are paying interest on technical debt' Aaratee Rao #qconsf

@dubmun: Types of technical debt. #qconsf

@dubmun: Common symptoms of high tech debt. #qconsf

@dubmun: Consider separate technical debt backlog. #qconsf

@WesCravens: Senior business management are the least aware of Technical Debt. Aaratee Rao @ #qconsf

@dubmun: Culture needs to recognize that how things are done is as important as getting them done. #qconsf

@dubmun: Introduce a definition of done that encourages cleaning up technical debt. #qconsf

@dubmun: There may come a time, if technical debt is out of hand, to declare technical bankruptcy. #qconsf

Culture and the Games People Play

by Roy Rapoport

Twitter feedback on this session included:

@dubmun: Orgs want an outcome that result from actions. Actions are informed by what people hear and what they see. @royrapoport #qconsf

@jbillybytes: I like the idea of alignment in the cultural context - powerful idea that resonates with people who care about the company. #qconsf

@dubmun: If people see and hear conflicting things, it is difficult to achieve an expected outcome. @royrapoport #qconsf

@dubmun: Number one rule: do what is in Netflix best interests. Sensing a pattern here. @royrapoport #qconsf

@jbillybytes: The Override Bar - params for overruling a proposal: 1) am i certain it's wrong 2) what harm would come by proceeding @royrapoport #qconsf

@dubmun: Set context for what’s important and people will make the right decisions. @royrapoport #qconsf

@dubmun: It’s not necessary to dictate what needs to be done. @royrapoport #qconsf

@jbillybytes: balance agility and predictability. optimize for agility. be smart about things you must predict. @royrapoport #qconsf

@jbillybytes: ask: are our incentive structures motivating the right behavior? @royrapoport outstanding - this will be the 1st question I ask post #qconsf

@bm2yogi: By constraining the ability of people to make decisions, you constrain their ability to make *good* decisions -@royrapoport #qconsf

@jessitron: I try to hire for insubordination:Be hard to manage, easy to lead.@royrapoport #qconsf

@jessitron: As a technical manager, I don't get to make technical decisions. I get to make priority decisions. @royrapoport #qconsf

@jessitron: Agility and predictability are in opposition.Predictability is really important for surprisingly few things.@royrapoport #qconsf

Nordstrom’s 114-year-old Culture in the Tech Era

by Sarah Lake Hagan

Twitter feedback on this session included:

@fox_felippe: @QConSF #qconsf A well defined culture is not an option!

@sarah_shewell: Culture is what attracts, retains, and enables people to do amazing things Sarah Lake Hagen -@NordstromPLab #qconsf

@dubmun: Auditions for the Nordstrom’s team is a process to ensure skills and culture fit. #qconsf

@dubmun: Nordstrom’s Tech has walk-up service desk when user can bring tech problems. There is a dashboard with expected wait time. So cool! #qconsf

@fox_felippe: Your innovation relies on your culture pillars! @QConSF #qconsf @NordstromPLab

@slshee: Different divisions may have different culture values, but they have several common ones that binds them together - @thesarahhagan #qconsf

Design Thinking

Design Thinking, What’s in It for Me?

by Jeanine Spence

Twitter feedback on this session included:

@sarah_shewell: 1 size fits all never wins -@jeaninespence You have to really understand your customer #qconsf

Developing Customer Empathy

by Chris Witeck

Twitter feedback on this session included:

@sarah_shewell: Customer empathy mapping is not a one time event. It's a constant cycle for your product. @cwiteck #qconsf #designthinking

@jitterted: Many times what appears to be a technical problem can be reframed as a business/ux problem and better solved at that level. #QConSF

@johnscattergood: To develop empathy: Look below the surface (motivation), look for patterns of behavior, be observant #qconsf

@johnscattergood: Observation in practice #qconsf

@johnscattergood: Involving the technical team with Customers #qconsf

@itsjustneil: Don't hire engineers only on technical merit. Hire engineers who are excited to meet and learn about their customers. #qconsf

@jitterted: Invite customers (buyers, users, etc.) to your office, and get them to talk to your development team directly. Not just the execs. #QConSF

@jitterted: To most of your customers, your engineering team is *magic*. ‘You can do that?!?’ #QConSF

@dubmun: Find every way possible to get engineers in front of customers. #qconsf @stgarrity

@craigsmitham: Instead of modeling domain objects as actors, consider first modeling concurrent activities - emphasizing workflow for improved perf #qconsf

@DC_LAN: Everyone wants to work with super ninja pirate developers, but what I want is engineers that can talk to customers. #qconsf

Modern CS in the Real World

Contracts in Clojure: Settling Types vs. Tests

by Jessica Kerr

Twitter feedback on this session included:

@agonigberg: I'm not going to deploy code to production that I only tested in my repl using printlns! @jessitron #qconsf

@agonigberg: Clojure drove me nuts until I found prismatic schema. @jessitron #qconsf

@jitterted: “One test is an anecdote, generative tests are evidence“ - @jessitron #QConSF

@jitterted: Creating good sample/test data can be hard. Checking them can be harder, hence property tests (Schema tests!). #QConSF (@jessitron)

@jitterted: It’s not enough for my service to be great, I need to boost the greatness of your service (@jessitron interpreting @wasbuxton) #QConSF

It Probably Works

by Tyler McMullen

Twitter feedback on this session included:

@philip_pfo: The world is not perfect but is probabilistic; our systems should be so too. @tbmcmullen #qconsf

@caitie: A single source of truth in a system is also a single source of failure @tbmcmullen #qconsf

@_wsh: Heh. @tbmcmullen, the problem with RFC 2549 is that dropped packets get really messy. #QConSF

Modern Languages in Practice

Experiences Building InfluxDB in Go

by Paul Dix

Twitter feedback on this session included:

@fawad: Unit and integration tests won't save you. If you're operating at scale, you have to test as scale @pauldix #qconsf

@Y2F00: #qconsf @pauldix : just used the standard tooling with @golang to build @InfluxDB

@Y2F00: #qconsf @pauldix : we have shockingly few dependencies due to standard tooling of @golang @InfluxDB

Go GC: Prioritizing Low Latency and Simplicity

by Rick Hudson

Twitter feedback on this session included:

@charleshumble: A blink of an eye takes about 300ms so doing GC in the blink of an eye isn't that impressive. Rich Hudson, #qconsf

@charleshumble: Go GC has one tuning knob for space vs. time. Quite a contrast to Java. Rick Hudson #qconsf

@charleshumble: GC latency is not a barrier to Go's adoption. Rick Hudson #qconsf.

Org Hacking

Beyond DevOps: How Netflix Bridges the Gap

by Josh Evans

Twitter feedback on this session included:

@petesoder: @Netflix open sources spinnaker, continuous delivery platform - announced by @Ops_Engineering at #qconsf

@portixol: Just seven engineers built @netflix's Spinnaker - their continuous delivery platform, which is open source as of today #qconsf

@portixol: Production ready checklist @netflix #qconsf @Ops_Engineering

@oyvindma: There's a new boy in town! Spinnaker from Netflix. Check it out at #qconsf

Preparing PayPal for Launch

by Sri Shivananda

Twitter feedback on this session included:

@johnscattergood: PayPal has 270 scrum teams each about 10-12 members #qconsf

@jkuner: TIL Paypal uses 10MW of power for 53000 hosts (in Q1) - #qconsf

@johnscattergood: PayPal separation from eBay followed an 18 month effort to integrate systems #facepalm #qconsf

@johnscattergood: PayPal separation required 100s of projects to be completed simultaneously #qconsf

@johnscattergood: PayPal and eBay used a scalable agile process using scrum of scrims to separate the companies #qconsf

@johnscattergood: When starting initiatives, it's vital to start with small quick successes to build momentum #qconsf

Taming Distributed Architecture

Demystifying Stream Processing with Apache Kafka

by Neha Narkhede

Twitter feedback on this session included:

@caitie: So true "The Hard Part of Distributed Systems. Coding < Debugging < Operations" @nehanarkhede #qconsf

@agonigberg: In distributed systems coding for ease of operations is more important than for future debugging or coding. @nehanarkhede #qconsf

@SivaKalagarla: New interfaces coming up in Apache Kapka @nehanarkhede #qconsf

@mohitvora: @nehanarkhede Happy to hear abt SSL support! Great talk, learned a bunch! #qconsf

Engineering for the Long Game

by Astrid Atkinson

Twitter feedback on this session included:

@bridgetkromhout: “I was there when that ecosystem was born.” @shinynew_oz on being at Google for over a decade, with Borg. #qconsf

@bridgetkromhout: If things go well, you may end up living with the consequences with your decisions for a long time to come. @shinynew_oz on growth #qconsf

@bridgetkromhout: Software is easy. People are really hard. That truth we don't always acknowledge: realtalk brought to you by @shinynew_oz. #qconsf

@charleshumble: The goal is not 'to not get paged' - the goal is to make sure you're never paged for the same thing twice. @shinynew_oz #qconsf

@charleshumble: “If you can’t tell if you system is up, it probably isn’t.” @shinynew_oz #qconsf

@bridgetkromhout: “Have to cross tooling boundaries for people to work together? One more barrier you don’t need” @shinynew_oz #qconsf

@jon_moore: .@shinynew_oz : "Pick *something* and try to get most people on it" #engineeringatscale #qconsf

@bridgetkromhout: Think broadly about costs (including time), not just on your own team. Consider opportunity costs of not taking risks. @shinynew_oz #qconsf

@bridgetkromhout: If you must use a hot new thing, it should only be if nothing else solves your problems. Well-known open source = best. @shinynew_oz #qconsf

@bridgetkromhout: You can throw software at the problem of managing a system at a much greater scale than you can throw people at it. @shinynew_oz #qconsf

@charleshumble: On a large enough distributed system your chances of failure go up to 100%.Resilience is a property of system design. @shinynew_oz #qconsf

@bridgetkromhout: The basic philosophy of a well-designed distributed system is distrust and also verify. @shinynew_oz in her epic #qconsf talk.

@bridgetkromhout: Your goal is not to prevent failures but to handle them effectively. @shinynew_oz dropping real-world Google-scale experience at #qconsf.

@charleshumble: There's a significant velocity drag if you have to talk people. @shinynew_oz #qconsf

@bridgetkromhout: Monoliths are unfashionable these days, but there's also such a thing as too many microservices. @shinynew_oz on consolidation. #qconsf

@bridgetkromhout: “At Google, all of our biggest DOS attacks come from ourselves.” (Botnet? No, intern.) @shinynew_oz #qconsf

@charleshumble: Move your biggest customer first. When we were rolling out Borg the first customer we moved was web search. @shinynew_oz #qconsf

@jon_moore: Don't let the weeds get higher than the garden. @shinynew_oz #qconsf

@bridgetkromhout: “If your goal is to build a racecar, you don’t want to spend >10 minutes on a cardboard box.“ @shinynew_oz #qconsf

@bridgetkromhout: Software is made of people. The hardest problems are always the human interactions. @shinynew_oz #qconsf

@charleshumble: It is not the code that's important. It’s the mission. @shinynew_oz #qconsf

@philip_pfo: If your second system doesn't unlock new capabilities, don't do it. The cost will always be higher than estimated. @shinynew_oz #qconsf

Unlearning Performance Myths

Broken Performance Tools

by Brendan Gregg

Twitter feedback on this session included:

@bridgetkromhout: Broken performance tools are more dire than you’d think: even ‘top’ may be lying to you. @brendangregg #qconsf

@bridgetkromhout: “All disk metrics are misleading.” @brendangregg dropping some realtalk about how your i/o subsystem is complex. #qconsf

@bcantrill: 'Let's trace everything!' Now you have one million problems per second! - @brendangregg at #qconsf

@bridgetkromhout: I've done a lot of benchmark analysis... 100% of benchmarks are wrong. @brendangregg #qconsf

Java vs. C Performance

by Cliff Click

Twitter feedback on this session included:

@TonyPrintezis: #Scala is the C++ of #Java. -- @cliff_click #QConSF

@orrsella: 1/ Interesting insight from “Java v. C Performance” #QConSF session: HFT companies mostly use Java for quick algo iteration

Taking Java to the next Level

Netty @Apple: Large Scale Deployment/ Connectivity

by Norman Maurer

Twitter feedback on this session included:

@RichardWarburto: Apple running 10s of Petabytes of data/day through over 400,000 netty instances. @normanmaurer #qconsf

@orrsella: “Netty at Apple” by @normanmaurer: 400,000+ instances, 10s PB/day, 10s M req/sec. Wow. #QconSF

@charleshumble: “Please don’t roll your own connection pool - take a look at the Finagle one - its hard to get right.” @normanmaurer #QconSF

@charleshumble: Rules for the long game: 1. Imagine that you will be very, very successful. 2. Complexity grows over time.#qconsf

The Amazing Potential Of .NET Open Source

Net Machine Learning: F# and Accord.Net

by Alena Hall

Twitter feedback on this session included:

@jbillybytes: some data scientists say they spend 70% of their time preparing the data. interesting!@lenadroid #qconsf

@jbillybytes: f# integrates with the R language - that's really appealing. #qconsf @lenadroid

@jbillybytes: @lenadroid says, "Talk is cheap, show me the code". awesome. #qconsf

@jbillybytes: F# comes with data-source-specific data providers (example: World Bank). Cool! #qconsf @lenadroid

Space, Time, and State

by Amy Palamountain

Twitter feedback on this session included:

@iWriteSciFi: You can write UI code that is easy to reason about and predict. Just remove STATE. #qconsf @ammeep

@iWriteSciFi: state + mutation = complexity #qconsf @ammeep

@iWriteSciFi: Events are a snapshot in time. They are one method of changing from a pull to a push paradigm. #qconsf @ammeep

@iWriteSciFi: However, events are limited: they results in a large mental model and can’t be combined with other events. #qconsf @ammeep

@iWriteSciFi: Instead, create an immutable core with mutable adapters. Model states, not objects. #qconsf @ammeep

@iWriteSciFi: Write pure functions that do not change state. No side-effects. #qconsf @ammeep

@iWriteSciFi: Reactive extensions = time monads :) #qconsf @ammeep

China Tech Day

Architecture @ Eleme

by Haochuan Guo

Amos Xu attended this session: is the largest food online ordering/delivery service in China. This company started in 2009, and experienced a 10x+ growth in 2014. It is now serving over 3 million orders per day, and during lunch time this could peak at 300 orders per second.

The initial stack was a simple PHP application running on Nginx in a single server. When the load became heavy, more servers were added with an HAProxy server in front of them. This structure was capable of serving up to a few hundreds of thousands of orders per day. Then the 10x+ growth came.

To enhance scalability, started with decoupling its PHP application. The single application was broken down to a User Service and an Ordering Service, in which the User Service was re-written using Python. More machines were added and HAProxy servers were put in between these machines.

Business quickly grew to a million orders per day and the structure could barely cope. An F5 gateway was introduced to replace the front-end HAProxy, and HHVM was used to replace FPM in oder to enhance the performance of the PHP application. A large number of caches were used. Heavy and unimportant APIs were downgraded. Queries were optimized. Databases (MySQL and PostgreSQL) were partitioned based on domains. …

The team is now planning to simplify its architecture by putting all APIs under the management of an API Service Orchestrator, and to remove the legacy code from the stack.

JD Internet + Transformation

by Gang He

Amos Xu attended this session:

Currently, all applications run on top of their cloud. The cloud infrastructure has an IDC physical layer at the bottom, a software-defined datacenter layer (JDOS) above it, and a container cluster scheduling layer above the software-defined datacenter. JD cloud uses OpenStack and Docker extensively with 100K+ containers running in production,.  They also developed their own file system (the “Jingdong" file system) for storage.

JD OpenStack uses Docker as a hypervisor (docker virt driver) and uses the Nova scheduler as the Docker scheduler. On the network level, Neutron is used with integrated OpenVSwitch(OVS). The JD team worked on OVS to improve its latency for small network packets by 20%. Glance is used for image management.

Mobile Monetization: Scenario Design & Big Data

by Arther Wu

Amos Xu attended this session:

The Clean Master app has various ad formats. For user engagement without intruding into their behaviour flow, ads are usually placed after user actions as an extension of a behaviour flow (e.g. after  a user takes a photo, edits it, and shares it).

Precise audience targeting is crucial in mobile ads, so building user profiles is important. Through data mining of raw data, user insights could be generated (such as Google Play app categories, interest tags, and click preferences). Various algorithms are used for supervised learning and un-supervised learning.

Search Technology on Mobile App @Baidu Mobile

by Chao Han

Amos Xu attended this session:

The architecture of Baidu Mobile is a super client-server application for mobile app users. A client plug-in system is applied to both Web UI and native UI.

A super client-server application extensively re-uses web app concepts in the client UI implementation. The main design concept is a cross-platform template. As such, performance, compatibility and controllability work can easily be shared among different platforms.

Performance utilization focuses on two aspects - the server performance, and customer experience on the client side (display speed for example). In the end four optimization points were used: server page cache, async loader and displayer, request merging, and cache negotiation.

Technological Innovation at Alibaba

by Yuan Qi

Amos Xu attended this session:

Alibaba strengthened its technology teams and initiated various projects, including Aliyun (Alibaba Cloud) starting from Sept 2009 and ODPS (Open Data Processing Service) starting from Apr 2010. As a result, Aliyun and ODPS now carry hundreds of petabytes (PB) of data from dozens of business units. Aliyun is capable of running 5000 nodes in a single cluster, and ODPS is capable of processing 100 PB of data in 6 hours. …

There are lots of experiments on applied data science beyond search engines, advertisements and recommendation engines. An AI based customer service robot now serves over 95% of Alipay users, which heavily rely on speech recognition, deep learning, natural language processing and knowledge base construction. The micro-loans service provided to small enterprises requires a risk control model equivalent to those used in the financial industry, but with more agility and flexibility. A shipping insurance service for returned products on the Taobao platform always faces risks of incorrect pricing and fraud, and a dynamic pricing model significantly increased its accuracy.

The Philosophy of Mass Services at Tencent

by Bison Liao

Amos Xu attended this session:

Bison Liao, the Director of Tencent Social Network Group, summarized their philosophy of mass services into four key points.

1. Flexible availability. All features are decoupled and graded according to importance….

2. “Comfort notices". “Comfort notices” are especially important when a service becomes unavailable. A simple "login failed" or similar failure notice would usually cause confusion for users, users might even do multiple retries which further overloads the backend servers. …

3. Process crash. All processes need to be monitored, so that in the case of a crash they can be restarted in milliseconds. …

4. Overload protection. The system is designed to handle the case when the buffer becomes full. Requests that stay a long time in the buffer are discarded, and number of retries are limited.

Transformation from Call Center, Web to Mobile

by Eric Ye

Amos Xu attended this session:

As the biggest travel service platform in China, Ctrip now serves over 3 million bookings per day, with 72% from mobile phones. Yet in the year 2011, Ctrip only had 25% of bookings online, while the other 75% were from call center.s The Transformatuib from call centers to mobile businesses took a lot of effort.

Major work started in 2012 when the Ctrip website was re-architected, and the UI was re-designed. New architecture encouraged open APIs, and search was optimized. A centralized mobile business unit was built in 2013, but this caused a lot of trouble in 2014 - since many new business units were added in the year, and each business unit was fighting for mobile resources. Lots of work was repetitive and inefficient. The dev infrastructure was also a bottleneck.

As such, a re-organization was initiated in 2014 and mobile resources were decentralized. Mobile app types were reduced to iOS and Android platforms only, as compared to multiple device support (iPhone/iPad/Android phone/Android tablet/Windows phone). Everything was decoupled into Microservices. Mobile apps used to run on its own business modules, so for example the hotel app had its own business modules,as did flight app and other apps. Now all services can be accessed via the data/url bus.

Solutions Track

Deconstructing API Security

by Ian Goldsmith

Twitter feedback on this session included:

@dberlind: @Akana product VP Ian Goldsmith @ #qconsf "Anything you do from a security perspective is not going to be very useful if it's not automated"

@APIenterprise: 5 common #API weaknesses - and how to deal with them #qconsf

Getting Started with Spring Cloud

by Josh Long

Twitter feedback on this session included:

@bridgetkromhout: Production is the happiest place on earth for us - it's better than Disneyland. @starbuxman on life at @pivotal. (It's true!) #qconsf

@bridgetkromhout: If you're stuck in the distant past... choose WAR. If you're here today with me, in 2015, then please choose JAR. @starbuxman #qconsf

@bridgetkromhout: Remember, the move to microservices is all about smart services, dumb pipes. @starbuxman #qconsf

Microservices in Continuous Delivery

by Ken Mugrage

Twitter feedback on this session included:

@selimekizoglu: The ability to deploy independently is vital. If you don't, don't bother with #microservices and avoid the overhead - @kmugrage @ #qconsf

@itsjustneil: If you have long running feature branches that get deployed every few weeks it's not CI/D. May have good automation just not CI/D #qconsf

@itsjustneil: CD Means: Your team prioritizes keeping the software deployable over new features #qconsf

What Do Defects Really Cost? More than You Think

by Wayne Ariola

Twitter feedback on this session included:

@buildscientist: Attending a session on the cost of software defects only to learn that Sony lost $18B during their 2011 data breach debacle. #qconsf

Opinions about QCon

Twitter feedback on QCon SF 2015 included:

@dberlind: If there's a developer event that's not to be missed, #QconSF may very well be it. @floydmarinescu & his @Qcon team have done a killer job!

@SivaKalagarla: First day #qconsf rocks!!! Good food, great concepts, extraordinary speakers. Good job @QConSF.

@Yauvansuba: #qconsf has been very helpful in accelerating thought process required to solve engineering probs while building professional software

@jbillybytes: #qconsf is empowering engineers to innovate culturally as well as technically. Great holistic approach!


Kelvin Meeks’ takeaways were:

The confluence of diverse backgrounds, technologies, and business segments represented by a global breadth of attendees - merges into a synthesis of pure enthusiasm that provides one-and-all an opportunity to pause in our day-to-day professional challenges, and lift our heads up for a moment and embrace the realization that there are people doing extraordinary things in our industry - and as their knowledge is willingly shared - new plateaus of achievement are now within our own grasp….


The ninth annual QCon San Francisco brought together 1,300 attendees and more than 100 speakers in what was the largest ever QCon to be held in the US. QCon's focus on practitioner-driven content is reflected in the fact that the program committee that selects the talks and speakers is itself comprised of technical practitioners from the software development community.

Presentations and interviews from the event will be posted on InfoQ over the coming months.  QCon SF 2015 was produced by Other upcoming QCons include:

  • QCon London March 7-11, 2016. This will be our 10th QCon in the city.
  • QCon São Paulo Mar 28-Apr 1, 2016.
  • QCon New York Jun 13-17, 2016.

Rate this Article

Adoption Stage

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread
Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread


Login to InfoQ to interact with what matters most to you.

Recover your password...


Follow your favorite topics and editors

Quick overview of most important highlights in the industry and on the site.


More signal, less noise

Build your own feed by choosing topics you want to read about and editors you want to hear from.


Stay up-to-date

Set up your notifications and don't miss out on content that matters to you