Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage Articles Key Takeaway Points and Lessons Learned from QCon San Francisco 2018

Key Takeaway Points and Lessons Learned from QCon San Francisco 2018

In mid November around 1,600 attendees descended on the Hyatt Regency in San Francisco for the twelfth annual QCon in the city.

QCon SF attendees - software engineers, architects, and project managers from a wide range of industries including some prominent Bay-area companies - attended 99 technical sessions across 6 concurrent tracks, 13 ask me anything sessions with speakers, 18 in-depth workshops, and 8 facilitated open spaces.

We’ve already started publishing sessions from the conference, along with transcripts for the first time. The full publishing schedule for presentations can be found on the QCon SF website.

The conference opened with a presentation from Jez Humble & Nicole Forsgren, two of the authors of “Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations” – one of InfoQ’s recommended books for 2018.

Some members of InfoQ's team of practitioner-editors were present and filed a number of stories about the event, but the main focus for this article is the key takeaways and highlights as blogged and tweeted by attendees.


Building the Enchanted Land

by Grady Booch

Twitter feedback on this keynote included:

@danielbryantuk: "Most of contemporary AI is about pattern matching signals on the edge, and inductive reasoning" @Grady_Booch #QConSF

@danielbryantuk: "As a developer you can ask yourself what value AI projects will have for you. One answer is that you can plug in these new frameworks in order take advantage and add value within your overall system" @Grady_Booch #QConSF

@TDJensen: Right now AI and deep learning specifically are valuable as parts of other systems vs. as a stand-alone system #QConSF per Grady Booch, IBM

@danielbryantuk: "When using AI within software you will need to get data scientists involved early. You need to think about data usage and also ethics" @Grady_Booch #QConSF

@eranstiller: "The human brain runs at about 20 Hz, some run slower" -- @Grady_Booch, a talk about #AI and computation #QConSF

@randyshoup: Everything is a system @Grady_Booch at #QConSF

@danielbryantuk: "The disparity in computational effort required for training AI models versus inference of the models will effect software architectures, particularly around inference at the edge" @Grady_Booch #QConSF

@danielbryantuk: "Everything is a system. In complex systems, malfunctions may not be detectable for long periods" @Grady_Booch #QConSF

@cinq: Every line of code represent an ethical and moral decision - @Grady_Booch #QConSF

@danielbryantuk: "We are increasingly seeing social and ethical issues getting more attention when building systems. Modern systems can have a big impact on people" @Grady_Booch #QConSF

@gwenshap: "First question I ask is "Do you have a release process and can release on regular cadence?" If not, fix this first." &<- @Grady_Booch at #QConSF

@danielbryantuk: "When working with a new system I focus on 'can the system be easily and rapidly deployed' and 'is there a sense of architectural vision'. If you have these two thing right you get rid of 80% of your typical issues" @Grady_Booch #QConSF

@Ronald_Verduin: 'Grow a system through the iterative and incremental release of an executable architecture' Grady Brooch #QConSf

@danielbryantuk: "Software whispers stories to be interpreted and realised by the underlying hardware. You [as developers] are the story tellers" @Grady_Booch on the importance of making your work count #QConSF

Finding Purpose Through the Things We Build

by Kortney Ziegler

Bernice Anne W. Chua attended this keynote:

He shared his experience as a creator and a person in tech, who also happens to be a black trans-man. It seemed fitting as a final talk, because even if we are software developers, we are all human beings first and foremost. It’s nice to know that even though each of us has our own specific kinds of struggles and challenges, we are not alone in that all of us are struggling together. One of the things that we as people might find ourselves dealing with is at the end of the day, we want to know that what we build is going towards something meaningful to us.

Twitter feedback on this keynote included:

@tsunamino: Purpose is a force: it drives us, it's strong, it's the core of what we are building #QconSF

@jdkiefer89: Be the force that drives! #QConSF @fakerapper #purpose

@tsunamino: Being an openly queer black person is difficult, especially in academics #QConSF

@tsunamino: Kortney went to tech because academia wouldn't give him a job because he was trans, despite being the first person to graduate from his department #QConSF

@tsunamino: There was no space for trans people to get together and make tech to help one another, no community. So he made it #QconSF

@tsunamino: To apply a concept that is so simple (spare change programs) to combat a problem so big (combating cash bail programs) helps people feel they can contribute in real ways #QConSF

@tsunamino: To find your purpose, you can't do it in isolation. You have to find your community in whatever format makes sense to you #QConSF

@tsunamino: Remain open to growth - continue to study and meet new people #QConSF

@tsunamino: Tell your story - if you don't document your life, no one else will. Check back in on your journey so you know where you've been and where you've come from #QConSF

@tsunamino: Helping others will help you figure out your purpose. It's important to be more open, even if it makes you more vulnerable #qconsf

@danielbryantuk: Great #qconsf closing keynote by @fakerapper! Key takeaways:
- everyone intrinsically provides value
- find your community
- remain open to growth
- tell your story
- help others

If You Don’t Know Where You’re Going, It Doesn’t Matter How Fast You Get There

by Jez Humble & Nicole Forsgren

Twitter feedback on this keynote included:

@bridgetkromhout: Business likes to play golf, and ops doesn't want change: the classic model of dysfunction. @jezhumble & @nicolefv #qconsf

@tsunamino: Important keynote message at #QConSF: be skeptical of the data you see out there (eg the number of pool deaths are correlated with years Nic Cage movies come out)

@bridgetkromhout: Correlation is not causation; @nicolefv explains how cherry-picking data isn't enough, as we have logical fallacies and biases. #qconsf

@danielbryantuk: I always enjoying showing clients these software delivery performance metrics from The DevOps Reports and "Accelerate" -- these pics are from the @nicolefv and @jezhumble #qconsf keynote

@tsunamino: Capabilities that drive performance after 4 years of research #QConSF

@danielbryantuk: "Elite performers are 3.5X more likely to have strong availability practices" @jezhumble #qconsf

@danielbryantuk: "Transformational leadership is a key capability that drives high performance" @nicolefv #qconsf

@tsunamino: At the top is transformational leadership, good product development, and empowered teams and test automation #QConSF

@bridgetkromhout: Invest in capabilities that are predictive of high performance in an IT org. @nicolefv & @jezhumble #qconsf

@neilathotep: Software delivery performance predicts organizational performance. And continuous delivery makes our work better #qconsf

@danielbryantuk: "You can drive cultural change in many ways -- one way is by changing your technical practices" @nicolefv #qconsf

@neilathotep: You can be using mainframes and achieve the outcomes - @jezhumble @QConSF #qconsf

@tsunamino: Meaning the technical stack isn't the most important (cough kubernetes) #QConSF

@tsunamino: If you aren't seeing benefits from moving to *the cloud* you probably aren't hitting all these points #QConSF

@charleshumble: "High performance is possible you just have to execute." Dr. Nicole Forsgren, #qconsf

@bridgetkromhout: Cloud is "the illusion of infinite resources" according to @jezhumble. For me, not worrying about data center cooling or rack-n-stack is magic; I think moving up the stack is powerfully freeing. #qconsf

@danielbryantuk: Several great predictive findings from the 2018 State of DevOps Report for high performing teams, via @jezhumble and @nicolefv at #qconsf Architectural outcomes, adopting the cloud properly, and monitoring and observability are core predictors

@tsunamino: How do you actually measure effective automated testing? These three caused the most issues #QConSF

@tsunamino: Continuous testing practices that actually work #QConSF

@danielbryantuk: "Continuous testing is not all about automating everything and firing all of the testers" @nicolefv and @jezhumble dropping some testing wisdom at #QConSF

@tsunamino: Properly scoping projects is key to building features or updating org practices #QConSF

@randyshoup: Taking big bits of work and splitting them into smaller bits that actually provide value to customers is key to *everything* @jezhumble and @nicolefv #QConSF

@tsunamino: You will not be able to fix your tech stack without also fixing the trust within your culture #QConSF

@danielbryantuk: "When something goes wrong, ask yourself that if you had the same information and tools, would you have done the same thing? Don't automatically fire the person seen to be the cause of the issue" @nicolefv and @jezhumble at #qconsf

@tsunamino: It's not the mix of skills that makes the perfect team, it's psychological safety #QConSF

@randyshoup: Team *dynamics* are critical; team skills are not. There was *no* correlation at Google between the technical skills of a team and its performance. #psychologicalsafety @nicolefv and @jezhumble #QConSF

@bridgetkromhout: Leaders set clear goals and direction, then let their teams decide how to implement. @nicolefv & @jezhumble on how high-performing orgs work. #qconsf

@jessitron: If teams can't change the incoming requirements, you might be doing fast-waterfall. @jezhumble #QConSF

@vllry: @nicolefv @jezhumble Stability and velocity aren't a tradeoff. Easy to change systems, strong testing/monitoring/SRE reduce cost of change management. Architecture and paradigm changes are key, not just tools and policies. #qconsf

@vllry: @nicolefv @jezhumble On the subject of doing DevOps "right" - buy into the goals and not just the checkboxes of key success indicators, like cloud adoption. Using GCP/AWS/Azure and having a test suite don't inherently mean you're getting value from those things. #QConSF

@vllry: @nicolefv @jezhumble Enable autonomy to be agile. Teams (and members) should be able to work independently with little coordination and consultation. To enable this: fast change management and architectural isolation. I strongly recommend watching the talk when posted! #QConSF

Using Data Effectively: Beyond Art and Science

by Hilary Parker

Twitter feedback on this keynote included:

@wesreisz: @hspter Nice visual to represent how to think about working in #dataScience #QConSF

@danielbryantuk: "If data science is an art, why don't we teach it like an art?" @hspter #QConSF

@chemphill: Explore a problem through solutions #DataScience @hspter @stitchfix #QConSF

@randyshoup: Lots of things can go wrong in a data analysis @hspter at #QConSF

@randyshoup: ... and so is data science, via @hspter at #QConSF

@wesreisz: @hspter love this image... Generative organizations practice blameless postmortems #QConSF

@randyshoup: Statistics as an academic discipline is problem-focused. Data science as a product-oriented discipline is solution-focused. @hspter at #QConSF

@sangeetan: Data Science - Design, science and art come together @hspter  #QConSF

@danielbryantuk: "Blameless postmortems and sprints have a lot in common. They provide structure and rules for playing nice" @hspter #qconsf

@danielbryantuk: "Empathy is a useful skill within life and software development, and it's something that can be developed with meditation" @hspter #QConSF

Track and Talks

21st Century Languages

Go - A Key Language in Enterprise Application Development?

by Aarti Parikh

Twitter feedback on this session included:

@ag_dubs: what was fun 10 years ago isn't fun anymore- are we due for a new programming language paradigm revolution? #21stCenturyLangs #QConSF

@ag_dubs: i want to be able to leverage multicore processors from the *language* level #21stCenturyLangs #QConSF

@ag_dubs: the history of languages and platforms is tending towards simplicity #21stCenturyLangs #qconsf

@ag_dubs: @golang wants to be the simplicity that both developers and platforms seek #21stCenturyLangs #QConSF

@katfukui: Applying love languages to technology can shape and change how humans use what you build. For ex, use words of affirmation to add love and thought in notification and text-based workflows #QConSF

@ag_dubs: stability really mattered for Java - and @golang is paying attention to that, even as it looks to go 2.0 #QConSF #21stCenturyLangs

@ag_dubs: "i like programming in Go because jt is bridging the gap between application and systems programming." - @classyhacker closes us out with some inspiring thoughts about the legacy of @golang #21stCenturyLangs #QConSF

Kotlin: Write Once, Run (Actually) Everywhere

by Jake Wharton

Twitter feedback on this session included:

@maheshvra: There is a common concern in adopting #kotlin, that is owned by Jetbrains, not a community or any affiliated open source foundation. Though, it offers good value. What is the future? #21stCenturyLangs #QConSF

@ag_dubs: because passing state into a class is such a common pattern, @kotlin gives u syntax sugar for collapsing that data into a property #21stCenturyLangs #QConSF

@ag_dubs: kotlin has property delegates that make abstracting patterns like lazy loading or null checks easy and expressive #21stCenturyLangs #QConSF

@ag_dubs: now to talk about @kotlin on multiple platforms- there's platform-dependent AND platform-independent kotlin! #21stCenturyLangs #QConSF

@ag_dubs: toolchains are incredibly important for making a multiple language solution work- and kotlin has great solutions for this interop #21stCenturyLangs #QConSF

@ag_dubs: an architecture for a multiplatform application- can we rewrite it *all* in kotlin? @JakeWharton says yes! #21stCenturyLangs #QConSF

@ag_dubs: being able to have a small team target multiple platforms with a single #kotlin codebase gives you lots of options for folks to access your application #21stCenturyLangs #QConSF

WebAssembly: Neither Web Nor Assembly, But Revolutionary

by Jay Phelps

Twitter feedback on this session included:

@ag_dubs: asm.js was a great frontrunner in leading the #webassembly revolution - check out that bitwise or 0, it's functionally a type annotation for integer! #QConSF #21stCenturyLangs

@ag_dubs: #webassembly is revolutionary because it's the first open and standardized bytecode #QConSF #21stCenturyLangs

@ag_dubs: is #webassembly trying to kill javascript? NOPE! #QConSF #21stCenturyLangs

@ag_dubs: #webassembly is going to affect language and compiler design as we go forward- and it's because the web has very interesting and unique constraints! #QConSF #21stCenturyLangs

@ag_dubs: you may already be using #webassembly ! (and not even know it!) shoutout to @fitzgen's sourcemap impl in @rustwasm ! #QConSF #21stCenturyLangs

Applied AI & Machine Learning

Fairness, Transparency, and Privacy in AI @LinkedIn

by Krishnaram Kenthapadi

Bernice Anne W. Chua attended this session:

The most important takeaway from this talk is that “We should think of privacy and fairness by design, at the very beginning, when developing AI models; not just as an afterthought.”, and “Privacy and fairness are not just technical challenges. They have huge societal consequences. That’s why it’s our responsibility to include as many and as diverse a group of people as possible to be our stakeholders.”

Architectures You've Always Wondered About

Airbnb's Great Migration: From Monolith to Service-Oriented

by Jessica Tai

Twitter feedback on this session included:

@danielbryantuk: Nice overview of the migration and service creation principles from the @AirbnbEng team, via @jessicamtai

at #QConSF

- services should own read and write data

- service addresses specific concern

- avoid duplicate functionality

- Data mutations should publish by standard events

@danielbryantuk: Excellent overview of the @AirbnbEng migration to microservices by @jessicamtai at #QConSF "It's worked out well for us, but SOA (and distributed systems) isn't for everyone."

@danielbryantuk: The @AirbnbEng team have clearly thought about their approach and tooling as they've moved to microservices. @jessicamtai at #QConSF giving a shout out to @TwitterEng's Diffy for comparing migrated endpoints

Netflix Play API - An Evolutionary Architecture

by Suudhan Rangarajan

Twitter feedback on this session included:

@gburrell_greg: Distributed monoliths are worse than monoliths. @suudhan at #QConSF

@gburrell_greg: Beware of data monoliths! @suudhan at #QConSF

Paying Technical Debt at Scale - Migrations @Stripe

by Will Larson

Twitter feedback on this session included:

@randyshoup: "Cassandra is a Trojan Horse released by Facebook to ruin a generation of startups"@Lethain at #QConSF

@randyshoup: What percentage of the system should be migrated before we declare a migration successful and complete? @Lethain at #QConSF

Real-World Architecture Panel

by dy ShoupDavid BanyardTessa Lau,Jeff WilliamsColin Breck

Twitter feedback on this session included:

@danielbryantuk: Great #QConSF real world architecture panel by @randyshoup
- Automation and robotics is being driven by large orgs. Democratizing the tech is important (esp for small projects)
- People outside the industry have high expectations and think humanoid robots will arrive next year

@danielbryantuk: Important things to learn and understand when working with real world automation and robotics:
- Systems thinking
- Awareness of security, privacy, and regulation
- UX
- Algorithms
- Mechanics 
#QConSF architecture panel, organized by @randyshoup

@tessalau: Robotics doesn't have all the tools which general software has yet, we need more primitive building blocks like ROS for robotics - @tessalau at #QConSF @rosorg

Scaling Slack - The Good, the Unexpected, and the Road Ahead

by Mike Demmer

Twitter feedback on this session included:

@danielbryantuk: "Deploying is only the beginning. We're taking time to migrate data correctly while still running at scale. We're also conscious of the second system effect where we are tempted to sneak in new updates along with the migration" @mjdemmer #QConSF

What We Got Wrong: Lessons From the Birth of Microservices

by Ben Sigelman

Twitter feedback on this session included:

@JustinMKitagawa: "Those Sun boxes are so expensive!" and "Those Linux boxes are so unreliable!" Sentiments leading google to develop disposable commodity hardware infrastructure and their foundational platform constructs circa 2001.

@gwenshap: The only good reason to adopt microservices: scale human communication. @el_bhs at #QConSF

@gwenshap: There is a real cost to making network calls. Keep that in mind when going #serverless. Good for embarrassingly parallel problems, but keep an eye on serial network calls. #QConSF

@gwenshap: There are cross-cutting concerns like Monitoring and security are delegated to small microservices team. This is significant overhead. #QConSF

@raimondb: Ben siegelman: beware of too much independence with microservices #qconsf

@Ronald_Verduin: How is your scrum team independence organized? Ants are independent but there are some rules. #QConSF Ben Siegelman

@gwenshap: Monitoring is about detecting critical signals and refining the search space for "what's wrong". Giant dashboards don't help refine the search space. #QConSF

@gwenshap: Hardest part about Dapper: political reality of getting something to run as root on every machine at Google. #QConSF

@gwenshap: Biggest mistake in building Dapper? Not analyzing bottlenecks early and writing to disk. #QConSF

@MicroSrvsCity: Optimize for team independence not for computer science #Microservices #QConSF @el_bhs

Developer Experience: Level up your Engineering Effectiveness

Evolving Continuous Integration: Applying CI to CI Strategy

by Elise McCallum

Twitter feedback on this session included:

@sangeetan: Operability is important consideration for internal tools as well. Think how well your entire team will be able to support your users. Elise McCallum @WeAreNetflix #QConSF #DevEx

@sangeetan: You don't always have to replace your system to address gaps. Incorporate best parts of other tools into your own and customize to your use cases to provide optimal experience. CI evolution @WeAreNetflix #QConSF with Elise McCallum

@sangeetan: Connecting existing tools reduces the need for developers to learn yet another new tool. Incremental evolution and migration FTW. Elise McCallum @WeAreNetflix #QConSF #DevEx

Helping Developers to Help Each Other

by Gail Ollis

Twitter feedback on this session included:

@sangeetan: Automation doesn't just make things easier, it's executable documentation that gets tested regularly. @GailOllis #QConSF #DevEx

@danielbryantuk: Exploring the PhD research of @GailOllis at #QConSF, looking at the effect of good and bad software development practices. Some of the top responses from interviewees about the biggest impacts

@danielbryantuk: The dangers of working in isolation within a software system, via @GailOllis at #QConSF

@sangeetan: The moment anything is written, it becomes legacy and has constraints around it. @GailOllis  #QConSF #DevEx

@neilathotep: "I saw him create a legacy system in 3 weeks" @GailOllis on helping developers to help each other #QConSF

@danielbryantuk: "You see a lot of themes in software variable names, which are not helpful when trying to solve an issue -- at least 'wibble' is obviously meaningless" @GailOllis #QConSF

@sangeetan: Re inventing the wheel, adding elephant to it and a spaceship on top - what others' code can feel like to people who have to work with it @GailOllis #QConSF

Enterprise Languages

Building a Reliable Cloud Based Bank in Java

by Jason Maude

Twitter feedback on this session included:

@danielbryantuk: "Banking IT has traditionally been about stability and conservatism -- the opposite to what modern organizations do.  This is driven by the requirement that when you put your money in the bank you don't expect it to be stolen by child wizards." 

@danielbryantuk: The combination of @scsarchitecture and the DITTO architecture has allowed @StarlingBank to build a resilient online banking system

JavaScript & Web Tech

npm and the Future of JavaScript

by Laurie Voss

Twitter feedback on this session included:

@annthurium: "JavaScript is the most popular language in the world right now...50% of working programmers use JS at least some of the time." -@seldo #QConSF

@tsunamino: Turns out package managers are all about the same speed because OSS means sharing best tips so everyone wins! #QConSF

@annthurium: Did you knowwwwww that "npm ci" will double the speed of your builds? @seldo #QconSF

@tsunamino: Package authors must have 2FA enabled to prevent someone from publishing something under your name #QConSF

@tsunamino: Omg please upgrade your packages 🙊 #QConSF

@tsunamino: Or if you want to, let the robots fix their own security vulnerabilities! #QConSF

@annthurium: Studies show that people pick languages for projects based on what libraries are available, which lines up with npm survey data. @seldo #QConSF

@tsunamino: Frameworks never really die, they get maintained in legacy products and then people start new products in new frameworks #QConSF

@annthurium: React isn't a framework so much as an ecosystem. React solves the view problem, but decouples itself from other related problems such as routing, state management, and data fetching that are solved by other tools. @seldo #QConSF

@annthurium: Only 56% of npm users are using a testing framework. 😱 @seldo #QConSF

@tsunamino: Predictions of the future of JavaScript: learn GraphQL and Typescript and WASM because they aren't going away #QConSF

@tsunamino: The best frameworks are the ones that have the most people using it bc you get a community to help #QConSF

Microservices / Serverless Patterns & Practices

Crisis to Calm: Story of Data Validation @ Netflix

by Lavanya Kanchanapalli

Twitter feedback on this session included:

@danielbryantuk: "Adding circuit breakers for mitigating risks related to data propagation added a lot of value, but also a few problems. Genuine scenarios accidentally tripped the circuit. We learned to use knobs to fix this at first" @lavanya_kan #qconsf

@danielbryantuk: "The Netflix data propagation system is very fast, but this means that bad data is propagated just as rapidly as good data! This can lead to challenges" @lavanya_kan #qconsf

@danielbryantuk: "Data validation is key to high availability" great thinking points from @lavanya_kan about data changes and safety #qconsf

From Winning the Microservice War to Keeping the Peace

by Andrew McVeigh

Twitter feedback on this session included:

@danielbryantuk: Put circuit breaking into a microservices system by default. "This stops a single match creating a system wide fire." Andrew McVeigh #qconsf

@danielbryantuk: "Put an API gateway into a microservices system straight away, e.g. something that handles auth and rate limiting" Andrew McVeigh at #qconsf also mentioning the benefits of @EnvoyProxy Can I also subtly plug @getambassadorio which is an open source gateway also built on Envoy :)

Reactive DDD—When Concurrent Waxes Fluent

by Vaughn Vernon

Twitter feedback on this session included:

@randyshoup: Prefer fluent interfaces: By setting data on an entity, you are not conveying why you are doing that. Define a protocol instead of a pile of setters. @VaughnVernon at #QConSF

@randyshoup: In an asynchronous message-driven system, you need to model intermediate states explicitly. @VaughnVernon at #QConSF (Subtle and powerful. Think state machines, workflows, eventual consistency, etc)

@randyshoup: Software efficiency is a first-order environmental concern (cf. cryptocurrency) @VaughnVernon quoting @davefarley77 at #QConSF

Modern CS in the Real World

Artwork Personalization @Netflix

by Justin Basilico

Twitter feedback on this session included:

@charleshumble: "Over 80% of what people watch on Netflix is based on recommendations." #qconsf @JustinBasilico

@charleshumble: At Netflix "we get over 20M requests per second for images at Peak" #qconsf @JustinBasilico

Modern Operating Systems

Is It Time to Rewrite the Operating System in Rust?

by  Bryan Cantrill

Twitter feedback on this session included:

@danielbryantuk: "Moving into the C language era was like moving into modernity. Every language before was clearly written with a broken Caps Lock key -- it was screamed" @bcantrill #qconsf

The Operating System in 2018

by Justin Cormack

Daniel Bryant attended this session:

The biggest changes in this space include: performance driven improvement, such as eBPF and userspace networking; the changing role of operations, and how operators use and deploy operating systems; and emulation and portability. There are also areas with little change so far but with signs that this is on the horizon, for example: operating systems are effectively the "last monolith"; there is a lack of diversity in OSs, and OS programming languages and contributors; and security has not yet received the full attention it requires….

Modern computer storage and networking have become much faster; moving from 1Gb ethernet to 100Gb over the past decade has resulted in a communication speed increase of two orders of magnitude, and SSD can now commit data at network wire speed. Accordingly, modifications to operating systems have been required in order to keep up with these changes.

The first approach to mitigate the issue of increasingly fast I/O is to avoid the kernel/userspace switch latency. System calls are relatively slow, and so these can be avoided by writing hardware device drivers in userspace. For example, in networking DPDK is the most widely used framework, and for non-volatile memory express (NVMe) there is SPDK.

Another approach is to never leave the kernel. However, it is challenging to code for the kernel, and so eBPF has emerged as a new safe in-kernel programming language. Cormack suggested that eBPF is effectively "AWS Lambda for the Linux kernel", as it provides a way to attach functions to kernel events. eBPF has a limited safe language that uses a LLVM toolchain, and runs as a universal in-kernel virtual machine. The Cilium service mesh project makes extensive use of eBPF for performing many networking functions within the kernel at very high speed.

Twitter feedback on this session included:

@danielbryantuk: "The changes in hardware I/O has driven a lot of change in modern operating systems" @justincormack #qconsf

@danielbryantuk: "At some point, hardware has stopped becoming hardware, and instead you talk to the abstraction via an API" @justincormack on the move away from hardware drivers to userspace frameworks #qconsf

@rhatr: The future of Operating Systems in a nutshell: Unikernel/Userspace or eBPF - pick your poison #QConSF

@danielbryantuk: "Userspace can add too much overhead to manipulate I/O, and so the move is now to implement functionality in the kernel -- eBPF is kind of like AWS Lambda for the Linux kernel" @justincormack #qconsf

@danielbryantuk: Great example of using eBPF within @ciliumproject, via @justincormack at #qconsf

@danielbryantuk: "Operations has changed quite a lot over the past decade. There has been a move away from the Sun workstation model of the 90s, which was massively impactful throughout its time " @justincormack #qconsf

@danielbryantuk: Great overview of @Linuxkit for building minimal OSs, from @justincormack at #qconsf

@danielbryantuk: "The stable ABI of Linux made it an emulation target. What does this mean? It's a great abstraction, and increasingly this is being used to add security isolation" @justincormack #qconsf

@danielbryantuk: "OSs used to be all about multi user, but now (in the cloud) we often have a single user and we care more about tail latency than who is logged in and who owns what file" @justincormack #qconsf

@danielbryantuk: "We're heading towards an OS monoculture with Linux, which makes it easy to write code, but there are a lot of disadvantages..." @justincormack #qconsf

@danielbryantuk: "Linux has historically preferred going fast over security" @justincormack #qconsf

Optimizing You: Human Skills for Individuals

Incrementally Refactoring Your Habits With Psychology

by Tilde Thurium

Twitter feedback on this session included:

@tsunamino: Watching @annthurium talk about how to actually change your habits when coding: context matters when you're trying to figure out the the right reminder system! #QConSF

@tsunamino: Distractions: is it your IDE or are you actually distracted by the internet? #QConSF

@tsunamino: Pro tips on how to ask for help: set a timer for 15 min and have a backup plan if other people are busy #QConSF

@GailOllis: "Typing while someone else is watching is the hardest problem in computer science" Tilde Ann Thurium, #QConSF

@tsunamino: Cmd - r to find function names quickly and better than cmd-f #QConSF

Practices of DevOps & Lean Thinking

DevOps For The Database

by Baron Schwartz

Twitter feedback on this session included:

@jessitron: When a DBA is busy as a bottleneck for developers, that's a waste of specialized skills that could be planning for the next architecture. - @xaprb #qconsf

@jessitron: Q: How do I change my culture? A: make smart investments in tools and process.

@nicolefv: the first rule should be "don't bring down production." @xaprb #QConSF

Service Ownership @Slack

by Holly Allen

Twitter feedback on this session included:

@danielbryantuk: "Sometimes working in an agile team feels like being a racehorse on a treadmill -- going very fast, but not getting anywhere that provides real value" @hollyjallen #qconsf

@jpaulreed: "@SlackHQ lives and breaths this cycle. There is massive executive commitment to it."-@hollyjallen #QConSF

@danielbryantuk: "You need at least two things to drive real change: executive dedication to learning, and high trust teams. There are also several other technical practices that help too" @hollyjallen #qconsf

@neilathotep: Being on call like everything else - learn by doing. @hollyjallen #QConSF

@danielbryantuk: This pain seems familiar from my past experience... @hollyjallen dropping some knowledge on how a product team may often scale faster than ops, but ops is super important. Moving to dev on call requires change management and empathy #qconsf

@danielbryantuk: "Sometimes working in an agile team feels like being a racehorse on a treadmill -- going very fast, but not getting anywhere that provides real value" @hollyjallen #qconsf

@danielbryantuk: "You need at least two things to drive real change: executive dedication to learning, and high trust teams. There are also several other technical practices that help too" @hollyjallen #qconsf

@danielbryantuk: "Ops had to build and keep these crazy big mental models of who knew each part of the system the best, in order to be able to best fix production issues. They effectively became human routers. This needed to change" @hollyjallen #qconsf

@danielbryantuk: "The service engineers provide the platform and best practices. We empower service ownership for product teams" @hollyjallen on the @SlackHQ organisation structure at #qconsf

@danielbryantuk: "We run 'incident lunches' at @SlackHQ, which must use our incident resolution process to order the lunch. We also throw in random constraints too, to simulate the real world. Anyone in the company can attend" @hollyjallen on running gamedays at Slack #qconsf

@danielbryantuk: "The ops team spent lots of time looking down at the machine level metrics and alerts to detect outages. Giving this up was hard. But ultimately, user-driven alerting was the way forward" @hollyjallen #qconsf

@danielbryantuk: "We weren't ready for chaos engineering, and so instead we ran Disasterpiece Theater" @hollyjallen #qconsf

@danielbryantuk: Excellent shout out to what works for one organisation, won't necessarily work in another. Be careful in what you take away from conference talks. "Copy the questions, not the answers!"

@hollyjallen at #qconsf

@jessitron: Change is possible. Don't wait until you think you're ready. @hollyjallen #qconsf (And other notes)

Full Cycle Developers @Netflix

by Greg Burrell

Twitter feedback on this session included:

@danielbryantuk: "The Netflix Edge team work as full cycle developers. We are responsible for coding, deployment, and operation, and we rely on centralised teams for specialised tooling and input" @gburrell_greg #qconsf

@danielbryantuk: An example of some of the tooling at Netflix that facilitates full cycle developers, via @gburrell_greg at #qconsf

@danielbryantuk: "Creating a team of full cycle developers is *not* a cost cutting measure. It requires proper staffing and training, and commitment and prioritization" @gburrell_greg on the importance of leadership in the Netflix team #qconsf

@danielbryantuk: "The full cycle developer model is not for everyone or for every organisation" @gburrell_greg #qconsf

@danielbryantuk: "We don't do YOLO deployments at Netflix, but we do deploy independently with good tests and metrics. We also coordinate around some big events" @gburrell_greg #qconsf

Whispers in the Chaos: Monitoring Weak Signals

by J. Paul Reed

Twitter feedback on this session included:

@jessitron: On getting better at getting better at our work: learn about what we know about our systems during incident response. With @jpaulreed at #QConSF

@jessitron: Not just any experience develops expertise. @jpaulreed #QconSF

@vllry: Whispers in the Chaos: Monitoring Weak Signals at #QConSF by @jpaulreed Once there's an incident, how do we find the cause? Expertise vastly speeds up pattern matching. Target changes, relevant memories, and proximity to find the source.

Production Readiness: Building Resilient Systems

Building Production-Ready Applications

by Michael Kehoe

Daniel Bryant attended this session:

Kehoe … began the talk by referencing the book "Production Ready Microservices" by Susan Fowler and an associated quote:

A production-ready application or service is one that can be trusted to serve production traffic…

… We trust it to behave reasonably, we trust it to perform reliably, we trust it to get the job done and to do its job well with very little downtime.

… For the tenet of stability and reliability Kehoe began by arguing for the need for stable development and deployment cycles. In this context stability is all about "having a consistent pre-production experience". Within development this should focus on well-established testing practices, code reviews, and continuous integration. For the deployment practice, engineers should ensure that builds are repeatable, a staging environment (if required) is functioning "like production", and that a canary release process is available….

The discussion of the next tenet, scalability and performance, began with a focus on understanding "growth scales", i.e. how each service scales with business goals and key performance indicators (KPIs). Engineers should be resource aware, knowing what bottlenecks exist within the system and what the elastic scaling options are. Constant performance evaluation is required -- ideally testing this should be part of the CI process -- and so is understanding the production traffic management and capacity planning that is in place.

In regard to the tenet of fault tolerance and disaster recovery, engineers should be aware of, and avoid, single points of failure within design and operation….

The next tenet of readiness discussed was monitoring. Kehoe argued that dashboards and alerting should be curated at the service-level, for resource allocation, and for infrastructure. All alerts should be require human action, and ideally present pre-documented remediation procedures and links to associated runbooks. Logging is also an underrated aspect of software development; engineers should write log statements to assist debugging, and the value of these statements should be verified during chaos experiments and game days.

The final tenet to be explored was documentation. There should be one centralised landing page for documentation for each services, and documentation should be regularly reviewed (at least every 3-6 months) by service engineers, SREs, and related stakeholders. Service documentation should include key information, like ports and hostnames, description, architecture diagram, API description, and oncall and onboarding information….

The key learnings of the talk were summarised as: create a set of guidelines for what it means for a service to be "ready"; automate the checking and scoring of these guidelines; and set the expectations between product, engineering, and SREs teams that these guidelines have to be met as part of a "definition of done".

Building Resilience in Production Migrations

by Sangeeta Handa

Daniel Bryant attended this session:

Handa … opened the talk by discussing that there are several known risks with the migration of data that is associated with a change in functionality or operational requirements, including data loss, data duplication, and data corruption….

With a large-scale geographically distributed customer base, many organizations cannot afford to take systems fully offline during any maintenance procedures, and therefore some variation of a "zero-downtime" migration is the only viable method of change. Variants of this approach that the Netflix team uses within a migration that involves data include: "perceived" zero downtime, "actual" zero downtime, and no migration of state….

Summarizing the talk, Handa shared the key lessons that her team had learned: aim for zero downtime even for data migration; invest in early feedback loops to build confidence; find ways to decouple customer interactions from your services; build independent checks and balances through automated data reconciliation; and canary release new functionality "rollout small, learn, repeat".

Twitter feedback on this session included:

@danielbryantuk: "When thinking about data migrations with a high availability system we think about various options: perceived zero downtime; actual zero downtime; and choose another approach to avoid migration" Sangeeta Handa from @netflix at #qconsf

@danielbryantuk: Great advice on data migrations from Sangeeta Handa of @netflix at #qconsf 
- aim for zero downtime (perceived or actual)
- planning is vital
- aim for decoupling e.g. use queues to buffer
- implement observability
- build checks and balances
- use canarying and shadowing

@vllry: Finishing up the last #qconsf notes edits - "Building Resilience In Production Migrations" by Sangeeta Handa. Naive database migration involves a lot of downtime and hope, and that's usually unacceptable. We need strategies to migrate in the background and test the data.

Yes, I Test In Production (And So Do You)

by Charity Majors

Twitter feedback on this session included:

@aysylu22: "This is what production looks like in my mind. It's a lot messier" @mipsytipsy concludes the debut of Production Readiness track with the talk Yes, I Test In Production (And So Do You) #QConSF

@koldobsky: If your tools tell you that everything is up, your tools are not good enough - Charity Majors #QConSF

@eanakashima: Complex distributed systems make staging environments particularly worthless -@mipsytipsy #QConSF

@vllry: @mipsytipsy Only production is production. Reproducing it is asymptotically costly, and incidents will occur that you can't foresee. You need strong tooling and observability to understand production, rather than trusting that prod will always be fine. #QConSF

@eanakashima: Invest in tools that let you protect the 99% of users who are using the system in a reasonable way from the rest -@mipsytipsy #QConSF

@mikellsolution: "Software engineers spending too much time in an artificial environment is dangerous" #qconsf @mipsytipsy on the importance of testing in Prod.

@neilathotep: @mipsytipsy on "testing in production" - because when it comes down to it, you are always testing. #QConSF

Socially Conscious Software

Design Strategies for Building Safer Platforms

by Kat Fukui

Twitter feedback on this session included:

@tsunamino: User story of a blob trying to escape an abusive relationship: What is the stressor? How are they feeling? What does success look like? #QConSF

@tsunamino: User stories can help highlight the gaps of where your product fails a user #QConSF

@tsunamino: Define what safety looks like to your user base - one size does not fit all! #QConSF

@tsunamino: Starter kit for designing with user safety in mind if you don't have a product designer #QConSF (shout-out to @consentsoftware!)

@tsunamino: Asking for consent helps users feel in control of their experience on your platform #QConSF

@tsunamino: Blobbo has added explicit consent to a DM feature they're designing and makes it easier to update your settings #QConSF

@tsunamino: Make sure you are encouraging good behavior along with adding friction to destructive behavior #QConSF

@tsunamino: First time contributor badges help show when a user is new to a community so maintainers can adjust language accordingly #QConSF

@tsunamino: Encouraging rehabilitation in an online community has been shown to have far reaching effects, even on other platforms #QConSF

@tsunamino: Using color and hierarchy to distinguish destructive actions a person can use on the content and user levels #QConSF

@tsunamino: Acceptance criteria bridges the gap between user stories and the code engineers use to make sure everyone is on the same page #QConSF

@tsunamino: Don't forget to document! It helps scale your efforts for others down the line #QConSF


by Ram Mehta

Twitter feedback on this session included:

@tsunamino: SNAP benefits used to only be able to check their balance by calling (and entering a 16 digit card number every time), saving receipts, or using a website no one knew about #qconsf

@tsunamino: They made an app that scrapes the website but made it easy to read and use #QConSF

@tsunamino: Here's a tree of how the money for SNAP comes in #QConSF

@tsunamino: NOOOOOO. If you don't remember your security question, the error message is the same as if you answered the question incorrectly #QConSF

@tsunamino: If you have a tiny team and not everyone can code, making it easy to use JSON can empower non-coders to keep services going #QConSF JSON can turn into code here

@tsunamino: Some states (ahem TEXAS) doesn't have a web portal so they built a way to call and translate the automated phone trees #QConSF

@tsunamino: The Puerto Rico web portal gives you an error message that says "if you aren't using IE6-9, this web portal will close" o they changed the user agent to IE9 in the app and it was ok?? #QConSF

@tsunamino: FreshEBT had interesting user stories to work with: sharing a smartphone with another user, running out of data, and being afraid of running out of benefits at checkout #QConSF

@tsunamino: Making sure to focus on each one of these categories made it possible to scale with minimal engineers #QConSF

@tsunamino: Multi branch ci builds with Jenkins means that non-technical people can review and merge code #QConSF

Managing Values-Driven Open Source Projects

by Nick O'Neill

Twitter feedback on this session included:

@tsunamino: Comparing voting to high availability consensus machines with @nickoneill #QConSF

@tsunamino: Candidates look at the people who have voted and craft policy around those people, not the entire population #QConSF

@tsunamino: . @make5calls makes it easy to contact your representative to make real change between elections #QConSF

@tsunamino: How can you take something no one likes to do (call strangers on the phone) and have a big impact (influencing representatives in government) #QConSF

@tsunamino: 5calls also provides a script to help introverts like me call their representatives on specific topics #QConSF

@tsunamino: Tech stack is intentionally boring and inclusive to be opening to new people who want to help #QConSF

@tsunamino: Making it easy for people to get started on your project (I refuse to say it's "lowering the bar") means more people are able to start contributing faster #qconsf

@tsunamino: Figuring out the three types of volunteers can help them get up to speed faster #QConsf

@tsunamino: Experienced OSS contributors can get things done quickly but often want your project to support a niche side project of theirs and can lead to unsustainable projects #QConSF

@tsunamino: Solid coders who love the mission who need some guidance are great because you can plan ahead for them and love the project #qconsf

@tsunamino: Fresh coders who just want to learn are mission driven but need some extra love tend to stick around for a long time. Starter issues can be planned ahead for them #QConsf

@tsunamino: Since they're all volunteers, it can be hard to get things done. Focus on when they can get to something, not when it's going to get done. And people are going to flake frequently #QConSF

@tsunamino: Don't be afraid to check in on your volunteers! It's ok to see if they can't do something and reassign it to someone else #QConSF

@tsunamino: How do you keep your values in an open source project where someone can fork your project and use it for something you don't believe in? #QConSF

@tsunamino: It turns out the tech stack isn't the important part, it's the content in @make5calls and how often it's updated #QConSF

@tsunamino: Decide on your values and communicate it to your team. You'll attract the talent you want to retain #QConSF

@tsunamino: Licenses on open source software are permissive in terms of what your software can be used for #QConSF

@tsunamino: Having strong values can be a differentiator for your project! #QConSF

Software Love Languages (On Passion & Product)

by Isaac Elias

Twitter feedback on this session included:

@tsunamino: We are all building tools and environments for humans, so what are your products communicating to our users? #QConSF

@tsunamino: What are your products going to do to your users? How are they going to affect humans? #QConSF

@tsunamino: Casinos are built to capture your attention. Apps (like Fortnight) are also built this way #QConSF

@tsunamino: Someone didn't test this out #QConSF

@tsunamino: Treat people like people! The founders of @TrueLinkFin designed the product with their grandparents in mind #QConSF

@tsunamino: Have respect for your users and ask for their consent! Shoutout to @consentsoftware #QConSF

@tsunamino: Using the five love languages like words of affirmation in things like welcome flows to connect with your users #QConSF

@tsunamino: Twitter's sign up flow is helpful and gentle to affirm and build users up #QConSF

@tsunamino: Physical touch is hard in the digital world, but notifications and accessing user data are a way to invade a user's space #QConSF

@tsunamino: When a company advocates for better privacy, it challenges their peers and makes the world better for users #QConSF

@tsunamino: Each of us has a voice that can say "this isn't right". You have the ability to band together and advocate for user safety! #QConSF

@tsunamino: Strava released heat maps of people's workouts and inadvertently showed who was stationed at each base #QConSF

@tsunamino: Acts of service are why people give you money - so you can make something in their lives easier #QConSF

@tsunamino: Making boilerplate code templates are a way to help users ramp up sooner and with guidance #QConSF

@tsunamino: Building better phone trees can help keep call volume to an agent stable (red line) despite the number of calls of people who need something (blue line) #QConSF

@tsunamino: Little things like reducing the amount of time to make a pull request are acts of service that can make a person's day - and they remember it #QConSF

@tsunamino: Ride alongs, like with customer support, can really help give engineers context and empathy #QConSF

@tsunamino: Pro tip: people will think you're a wizard if you can help them with the little things like a copy/paste button #QConSF

Sponsored Solutions Track I

Observability: The Health of Every Request

by Nathan LeClaire

Twitter feedback on this session included:

@kelescopic: "More observable business will attract better engineers... more observable businesses will beat their competitors." @dotpem #observability #o11y #QconSF

@kelescopic: "Aggregates can be very misleading... but percentiles also have issues...." @dotpem #observability #o11y #QconSF

@kelescopic: "Traces offer a great view of requests as they flow through your system... they also offer access to your raw data."@dotpem #observability #o11y #QconSF

@kelescopic: "Events are comprised of key value pairs... because you can cross correlate any dimension... querying [high-cardinal data] is no problem." @dotpem #observability #tracing #o11y #QconSF

@kelescopic: "At scale you can't store everything forever... but historically, statistics have your back." @dotpem on the power of #sampling your data #observability #QConSF

@kelescopic: "Healthy requests are boring. Who cares?... sampling keeps more of the interesting stuff. [You can]be proactive about what's wrong in your system and explain why." @dotpem #observability #o11y #QconSF

@kelescopic: @dotpem talks about how things used to be more simple but we're never going back #QconSF #observability #o11y @julienbarbier42

Opinions about QCon

@Ronald_Verduin: Cool the keynote is started at #QConSf. Videos of the sessions can be shared up to 50 colleagues so let me know:-)

@tsunamino: Pretty cool that #qconsf allows you to iterate on the feedback you provide speakers and also delete and anonymize your history!

@santiagopoli_: I want to congratulate the organizers of the @QConSF for a great conference! Keep the great work going people, you are awesome. Today: workshops #QConSF

@yusbel: #QConSF conference end, very valuable conference for those that practice architecture, my number one reason for returning was the opportuinity to ask anything to the guys that make the magic happen. Now workshop begin, making the abstract concrete. #TD

@unfunky_ufo: Had an amazing first trip to SF this week. Thanks to the organizers of #QConSF for putting together a really solid conference.

@danielbryantuk: Yet another thoroughly enjoyable #QConSF in the books. Many thanks to all of the speakers, sponsors, attendees and organisers! See you in London for the next event :-)

@fabiale: I really liked #qconsf and all the language talks. One of them made me realise how cool #kotlin is.

@Ronald_Verduin: Okay #QConSF is a wrap. Lots of new information after 3 days conference and 2 days workshops. Cool I could be here, thanks #InfoSupport


Bernice Anne W. Chua’s takeaways were:

I liked that the topics covered in the talks and keynotes were thoughtfully curated, and are as varied as possible to represent most of the range of human experience, and that the staff took care of the needs of the attendees. I learned a lot and made some new friends. It’s a refreshing experience to have a conference for developers, by developers. And I thank the QConSF staff and speakers for this experience.

AWESOME/10, would attend again.

@NotMyself: While at #qconsf I had a chance to see a demo of @ballerinalang, which tags itself as a "Cloud Native Programming Language." Seemed kinda neat, might have to play with it when I get home.

@pavelmicka_xyz: The main themes of #qconsf: microservice architecture works + success stories. Challenges: its distributed system, circuit breakers & resilient design are a necessity. Recommendations: proper component ownership is the way to go (no throwing the app over the wall).


InfoQ produces QCons in 6 cities around the globe. Our focus on practitioner-driven content is reflected in the fact that the program committee that selects the talks and speakers is itself comprised of technical practitioners from the software development community. Our next QCon will be in London March 4-8, 2019. We will return to San Fransisco April 15-17 2019 with, our event focussed on machine learning and AI for developers.

Rate this Article