A large confluence of team leads, architects and project managers descended upon the Embarcadero in San Francisco this year for a technology pilgrimmage that has become an annual tradition: QCon San Francisco 2012 (QConSF). To accomodate its 950 attendees - 30% more than last year - the conference was forced to move to a larger venue this year.
Continuing in its tradition of providing presentations selected "by practitioners for practitioners," QConSF featured over 100 sessions, keynotes, and - for the first time this year - professionally facilitated open space sessions. Attendees this year also had the chance to attend the inaugural GrapchConnect, a co-located, 2-day conference covering important topics in the NoSQL and graph database community.
Case studies from Facebook, Google, and Pinterest drew large crowds at QConSF, as well as a wide array of talks on continuous delivery, Big Data & NoSQL, cross-platform mobile, dynamic languages, and more. The following article summarizes the key takeaways from QConSF 2012, including blog entries written by editors and practitioner attendees for all keynotes, tracks and sessions along with aggregated twitter feedback during the event. You can also see numerous attendee-taken photos of QCon.
Architectures You've Always Wondered About
Loose Concurrency & CAP Theorem Today
Tutorials
Agile Retrospectives by Aino Vonge Corry
Twitter feedback on this session included:
ernando Moribe
@fgmatwork :Agile Retrospectives tutorial from@apaipi was one of the best tutorials at#qconsf ever!!
Continuous Delivery by Jez Humble
Twitter feedback on this session included:
Fernando Moribe
@fgmatwork : Continuous delivery with@jezhumble in the first day of#qconsf was awesomeVlad Skvortsov
@wadcom :"The reason people become engineers is so they can avoid talking to other people"@jezhumble at#qconVlad Skvortsov
@wadcom :Definition: legacy code is a code that doesn't have automated tests.#qcon
Cross Platform Mobile Apps with PhoneGap by Christophe Coenraets
Twitter feedback on this session included:
desbiens
@desbiens : Build and deployed my first phonegap app thks to@ccoenraets#qconsfLou Sacco
@occasl :@ccoenraets#Phonegap preso pretty impressive with regard to what's available at the JS API level with decent performance.#qconsf
Domain Driven Overview by Eric Evans
Twitter feedback on this session included:
Franklin Amorim
@cyberelfo : "Reality is a distraction. A model serves some use." Eric Evans#QConSFHakeem Mohammed
@hakeemsm : Had a good refresher on#ddd by@ericevans0 at#qconsf having read his book some yrs back it was good to hear from the man himself
Micro Services - Evolutionary Approaches For Systems Of Systems by James Lewis
Twitter feedback on this session included:
Kris Trujillo
@kris_trujillo :"Tip 1 - Divide and Conquer" micro services#qconsfKris Trujillo
@kris_trujillo :"Tip 2 - Use Conway's Law to structure teams" - Micro Services#qconsfKris Trujillo
@kris_trujillo :"Tip 3 - The Last Responsible Moment.....don't make decisions at the moment you know the least" - Micro Services#qconsfKris Trujillo
@kris_trujillo :"Tip 4 - Be of the web, not behind the web" - Micro Services#qconsfKris Trujillo
@kris_trujillo :"Tip 5 - If something is important, make it an explicit part of your design" - Micro Services#qconsfKris Trujillo
@kris_trujillo :"Tip 6 - Favor services choreography over orchestration" - Micro Services#qconsfKris Trujillo
@kris_trujillo :"Tip 7 - Make it easy to do the right thing and hard to do the wrong thing" - Micro Services#qconsfKris Trujillo
@kris_trujillo :Clean looking application following micro services approach#qconsf pic.twitter.com/wuedfiN1Jeff Gothelf
@jboogie :.@tsharon presenting ux research for engineers#qconsf pic.twitter.com/o61ODM4xKris Trujillo
@kris_trujillo :"No Silver Bullets" - Micro Services#qconsf
Professional Javascript and Coffeescript by Peter Bell
Twitter feedback on this session included:
QCon San Francisco
@QConSF :“@PeterBell: Thanks for everyone who took my javascript/coffeescript tutorial at@QConSF - had a blast!” - We had a blast too!#qconsf
Keynotes
Cool & Useless by Kevlin Henney
Twitter feedback on this session included:
Lene Mejlby
@Mekkaz :Don't trust the compiler, just type it in yourself.@KevlinHenney talking about the code for a space shuttle#qconsfTherese Hansen
@qedtherese :Kevlin Henney on stage at#qconsf showing this http://m.youtube.com/watch?v=cYw2ewoO6c4 … My Danish friends made this#CoolAndUselessavid Whittaker
@rundavidrun :@kelvinhenney introduces us to COBOL on Cogs ala Ruby on Rails...of course written in JS http://coboloncogs.org#AtwoodsLaw#qconsfFernando Moribe
@fgmatwork :Funny party keynote from kevlinhenney at#qconsf http://instagr.am/p/RwGddhsDp3/Geoff Webb
@geoffnettaglich :Highlights from "Cool & Useless" keynote at#qconsf Cobol on Cogs (http://www.coboloncogs.org/ ) and JS/Linux (http://bellard.org/jslinux/ )
NoSQL: Past, Present, Future by Eric Brewer
Twitter feedback on this session included:
Kris Trujillo
@kris_trujillo :"Data outlasts any particular implementation" - Eric Brewer#QConSFDavid Whittaker
@rundavidrun :@eric_brewer says: "In the 70's we moved to higher level models, from assembly to C, and from navigational to relational databases."#qconsfTherese Hansen
@qedtherese :Eric Brewer on stage at#qconsf talking about NoSQL - and even some history of "NoSQL" before SQL http://instagr.am/p/RxvHfJngOe/David Whittaker
@rundavidrun :Want a clean model and ACID transactions? Go relational. Want elegance, modularity, and flexibility? Go NoSQL.@eric_brewer#qconsfErik Sowa
@eriksowa :Eric Brewer at#qconsf : “ACID was sacred.” CAP that!David Whittaker
@rundavidrun :@eric_brewer developed CAP theorem to explain why he had to make the trade-offs he did when designing shared-data systems.#qconsfMartin Thompson
@mjpt777 :"Jim Gray came up with 'ACID' transactions while stoned in his hot tub. Don't get too hung up on the letters..." - Eric Brewer#qconsfMartin Thompson
@mjpt777 :"A timeout is the point at which you make the choice between consistency and availability." - Eric Brewer#qconsfDavid Whittaker
@rundavidrun :@eric_brewer claims "For most businesses, availability is more important than temporary inconsistency, even banking, e.g. ATMs."#qconsfDan Cundiff
@pmotch :Eric Brewer, CAP creator says "In real world systems, availability > than consistency because being available is tied to revenue."#QconSF
Race Conditions, Distribution, Interactions--Testing the Hard Stuff and Staying Sane by John Hughes
Twitter feedback on this session included:
Geoff Webb
@geoffnettaglich :John Hughes (mr haskell) says "Don't write tests, generate them"#qconsfIsmail Elshareef
@ielshareef :Don’t write tests; generate them. Use QuickCheck. Keynote by@rjmh, who is totally fired up.#qconsf#tdd#qaDavid Whittaker
@rundavidrun :@rjmh implements queue as circular buffer in C that is 'obviously correct' and models intended behavior in Erlang to test it. Cool!#qconsfTherese Hansen
@qedtherese :John Hughes on stage at#qconsf talking about testing for race conditions and other nasty things. http://instagr.am/p/R0UXNjHgAD/David Whittaker
@rundavidrun :@rjmh models race conditions via ticket dispenser metaphor then fuzz tests, trying to interleave as sequential to check.#quickcheck#qconsfiliketoprogram
@iliketoprogram :Dear#Quickcheck - i need to research you you seem awesome!#qconsf
Real Software Engineering by Glenn Vanderburg
Twitter feedback on this session included:
Kris Trujillo
@kris_trujillo :"It started to go wrong when 'software' and 'engineer' were used together" - Glenn Vanderbug#qconsfKris Trujillo
@kris_trujillo :"Software engineering suffered from 'premature maturity' as a discipline" - Glenn Vanderburg#qconsfKris Trujillo
@kris_trujillo :"In engineering....people design through documentation" thinking that kills us as software engineers/architects#qconsfKris Trujillo
@kris_trujillo :"Wings on an airplane break at the exact point the engineers predicted"....if software could only be the same....#qconsfKris Trujillo
@kris_trujillo :"Different engineering disciples are different"#qconsfTherese Hansen
@qedtherese :Software engineering is science and art#qconsf http://instagr.am/p/RvNlfOngJm/Kris Trujillo
@kris_trujillo :"Software engineering will be different from other kinds of engineering" - Glenn Vanderburg on Real Software Engineering#qconsfKris Trujillo
@kris_trujillo :"In software building prototypes is essentially instantaneous and free"....prototypes over documents!#qconsfKris Trujillo
@kris_trujillo :"The code is the document in the software engineering discipline"#qconsfJohannes Carlén
@johannescarlen :"The code is the model in software engineering" - Glenn Vanderburg#qconsfadrian cockcroft
@adrianco :#qconsf Software Engineering keynote uses a caricature of other engineering disciplines to claim Software is different#strawman#failKris Trujillo
@kris_trujillo :"What works best? Some variant of agile development" - Glenn Vanderburg#qconsfDavid Whittaker
@rundavidrun :@glv#QConSF asks, "When you can build a prototype quickly and free, why spend a lot of time on formal methods?"
Architectures You've Always Wondered About
Cloud Computing at Google by Randy Shoup
Twitter feedback on this session included:
Jim Constantine
@jgc94131 :google's spanner - planet wide data store with globally synchronized clocks#QConSFHiroaki ONO
@hihihiroro :KVM-hosted virtual machines using Ubunts or CentOS@Google#qconsf
Scaling Pinterest by Yashwanth Nelapati & Marty Weiner
Twitter feedback on this session included:
rama roberts
@ramarob :Pinterest has 80 cc2.8XL MySQL servers. Yikes.#QconSFCyberPunkie
@CyberPunkie :#QConSF 100 Redis instances + 80 Memcache ones...#Pinterest
Big Data and Analytics
100% Big Data. 0% Hadoop. 0% Java by Pavlo Baron
Twitter feedback on this session included:
Alex Popescu
@al3xandru :Years ago: lots of "free" data, very few tools. Now: more tools, more data, but most of it behind walls. /cc@pavlobaron#qcon#qconsf
Appwatch - A Big Data Application Monitoring System for Ebay by Bhaven Avalani & Yuri Finklestein
Matt O'Keefe attended this session:
This is a Big Data talk with Monitoring as the context. The problem domain includes operational management (performance, errors, anomaly detection), triaging (Root Cause Analysis), and business monitoring (customer behavior, click stream analytics). Customers of Monitoring include dev, Ops, infosec, management, research, and the business team. How much data? In 2009 it was tens of terabytes per day, now more than 500 TB/day. Drivers of this volume are business growth, SOA (many small pieces log more data), business insights, and Ops automation.
The second aspect is Data Quality. There are logs, metrics, and events with decreasing entropy in that order. Logs are free-form whereas events are well defined. Veracity increases in that order. Logs might be inaccurate.
There are tens of thousands of servers in multiple datacenters generating logs, metrics and events that feed into a data distribution system. The data is distributed to OLAP, Hadoop, and HBase for storage. Some of the data is dealt with in real-time while other activities such as OLAP for metric extraction is not. …
The hardest part of distributing such large amounts of data is fault handling. It is necessary to be able to buffer data temporarily, and handle large spikes. Their solution is similar to Scribe and Flume except the unit of work is a log entry with multiple lines. The lines must be processed in correct order. The Fault Domain Manager copies the data into downstream domains. It uses a system of queues to handle the temporary unavailability of a destination domain such as Hadoop or Messaging. Queues can indicate the pressure in the system being produced by the tens of thousands of publisher clients. The queues are implemented as circular buffers so that they can start dropping data if the pressure is too great. There are different policies such as drop head and drop tail that are applied depending on the domain’s requirements. …
Insights
* Entropy is important to look at; remove it as early as possible
* Data distribution needs to be flexible and elastic
* Storage should be optimized for access patterns
Through The Looking Glass - Applying Analytics to Development by Michael Feathers
Twitter feedback on this session included:
ames Morgan
@jmrgn :Feathers is analyzing code complexity of files to number of commits. Cool stuff. Great way to identify refactoring opportunities.#qconsfDavid Whittaker
@rundavidrun :@mfeathers warns, "There's a bias in software towards a high change rate for some files. Keep the complexity low for these files."#qconsfErik Sowa
@eriksowa :Thought-provoking examples of analytics on codelines by@mfeathers at#qconsf - e.g. churn vs. complexity.Lene Mejlby
@Mekkaz :Working as a team of co-makers requires mind-reading.@jeffpatton#qconsf
Continuous Delivery
Adopting Continuous Delivery by Jez Humble
Twitter feedback on this session included:
Johannes Carlén
@johannescarlen :"Lean is not about reducing cost, lean is about reducing waste in the value stream" - Jez Humble#qconsfMatt Barcomb
@mattbarcomb : Continuous Delivery: your software should always be deployable...not deplorable /cc@jezhumble#qconsf
Changing Culture & Being a Force for Awesome by Jesse Robbins
Twitter feedback on this session included:
David Whittaker
@rundavidrun :Continuous deployment is the paragon when all things in a large system are finally working correctly and harmoniously.@jesserobbins#qconsflane halley
@thinknow :Small change, big impact. "No-one is in charge" --> "everyone is in charge"@jesserobbins on hacking culture change#QConSFLou Sacco
@occasl :@jesserobbins talk was really good on how to be a good steward of change in the continuous delivery segment.#qconsf
Matt O'Keefe attended this session:
Teams that do Continuous Delivery are much happier. Seeing your code live is very gratifying. You have the freedom to experiment with new things because you aren’t stuck dealing with large releases and the challenge of getting everything right in one go. …
Tools are not enough (even really great tools like Chef!). In order to succeed you have to convince people that you can be trusted and you want to work together. The reason for this is understood, for example see Conway’s law. Teams need to work together continuously, not just at deploy time.
Choice: discourage change in the interest of stability, or allow change to happen as often as it needs to. Asking the question of which do you choose is better than just making a statement. …
Cultural change takes a long time. This is the hardest thing. Jesse’s Rule: Don’t Fight Stupid, Make More Awesome! Pick your battles and do these 5 things:
- Start small and built on trust and safety. The machinery will resist you if you try sweeping change.
- Create champions. Attack the least contentious thing first.
- Use metrics to build confidence. Create something that you can point to to get people excited. Time to value is a good one.
- Celebrate successes. This builds excitement, even for trivial accomplishments. The thing is to create arbitray points where you can look back and see progress.
- Exploit Compelling Events. When something breaks it is a chance to do something different. “Currency to Make Change” is made available, as John Allspaw puts it.
Large-Scale Continuous Testing in the Cloud by John Penix
Twitter feedback on this session included:
desbiens
@desbiens :Google runs more than 50 Million tests cases everyday,@johnpenix,#continuousdelivery#qconsfdesbiens
@desbiens :3 roles and use cases : Build cop, release engineer, developer,@johnpenix,#continuousdelivery#qconsfDan Cundiff
@pmotch :Google's John Penix CI stats: 200k test suites in code base, runs 10m test suites /day, >60m test cases / day, >4k CI builds / day.#QconSFJez Humble
@jezhumble :“@LindroosKE: "People are not working in branches. Well, couple of teams are and they are not happy" <<@johnpenix of Google#qconsfstuartblair
@stuartblair :@johnpenix: "public humiliation is usually enough to stop people checking in broken code"#qconsfJez Humble
@jezhumble :CI and automated testing at scale at Google by@johnpenix#qconsf pic.twitter.com/yQLYO4JA
Product Development with Continuous Experimentation by Frank Harris & Nell Thomas
Twitter feedback on this session included:
lane halley
@thinknow :So cool.@Etsy 1-click deploy so well automated "VCs, dogs and PMs deploy."#qconsfIvan Linhares
@ivanlinhares :Etsy at#qconsf, doing 20 deploys a day and automated A/B testing.Dan Cundiff
@pmotch :Etsy deploys to production 20+ times per day across 200+ developers. 10,000 deploys in a year.#QconSFHans Thunberg
@hthunberg :Release small and often to get rid of scary releases.#etsy @#qconsflane halley
@thinknow :So excited to hear how@Etsy is using hypotheses and continuous deployment to be quick and confident.#QConSF@jboogie
Release Engineering at Facebook by Chuck Rossi
Twitter feedback on this session included:
Therese Hansen
@qedtherese :Facebook developer sandbox load time. ~ David Mortenson#qconsf http://instagr.am/p/RyN-ymHgII/Fernando Moribe
@fgmatwork :"Facebook has only engineers, it does not have a QA department". test automation and engineers that write test#qconsfKris Trujillo
@kris_trujillo :Impressed by the life of a Facebook engineer as described by David Mortenson.... sounds like a great culture#qconsfDante Briones
@dantebriones :.@chuckr destroying the myth that#continuousdelivery means#testing in production#qconsfPete Hodgson
@ph1 :Facebook error reports include a stack trace... with svn blame for each function so they know who to shout at.#qconsfPete Hodgson
@ph1 :Facebook dev's khama is only visible to the dev, the release team... and the dev's manager :-/#qconsfFernando Moribe
@fgmatwork :"Tools alone won't save you. You need the right culture (company)" Facebook about release process#qconsf
Matt O'Keefe attended this session:
You should always be testing. People say it but don’t mean it, but Facebook takes it very seriously. Employees never go to the real facebook.com because they are redirected to www.latest.facebook.com. This is their production Facebook plus all pending changes, so the whole company is seeing what will go out. Dogfooding is important. If there’s a fatal error, you get directed to the bug report page.
File bugs when you can reproduce them. Make it easy and low friction for internal users to report an issue. The internal Facebook includes some extra chrome with a button that captures session state, then routes a bug report to the right people.
When Chuck does a push, there’s another step in that developers’ changes are not merged unless you’ve shown up. You have to reply to a message to confirm that you’re online and ready to support the push. So the actual build is www.inyour.facebook.com which has fewer changes than latest.
Facebook.com is not to be used as a sandbox. Developers have to resist the urge to test in prod. If you have a billion users, don’t figure things out in prod. Facebook has a separate complete and robust sandbox system.
On-call duties are serious. They make sure that they have engineers assigned as point of contact across the whole system. Facebook has a tool that allows quick lookup of on-call people. No engineer escapes this.
Facebook does everything in IRC. It scales well with up to 1000 people in a channel. Easy questions are answered by a bot. There is a command to lookup the status of any rev. They also have a browser shortcut as well. Bots are your friends and they track you like a dog. A bot will ask a developer to confirm that they want a change to go out. …
When Chuck does the final merge, he kicks off a system test immediately. They have about 3500 unit test suites and he can run one each machine. He reruns the tests after every cherrypick. …
Chuck’s job is to manage risk. When he looks at the cherry pick dashboard it shows the size of the change, and the amount of discussion in the diff tool (how controversial is the change). If both are high he looks more closely. He can also see push karma rated up to five stars for each requestor. He has an unlike button to downgrade your karma. If you get down to two stars, Chuck will just stop taking your changes. You have to come and have a talk with him to get back on track….
The main point is that you cannot tool your way out of this. The people coming on board have to be brainwashed so they buy into the cultural part. You need the right company with support from the top all the way down.
Cross Platform Mobile
Building Social Apps For All Mobile Platforms by James Pearce
Twitter feedback on this session included:
alex-and-r
@alex_and_r :“@firt: Current Facebook stats Html5/web vs native traffic via@jamespearce at#qconsf pic.twitter.com/Bfr1VLoe” О_ОMaximiliano Firtman
@firt :After Facebook has updated the infinite list from html5 to fully natively, users are consuming 2x content. Via@jamespearce at#qconsfDavid Whittaker
@rundavidrun :@jamespearce#QConSF says "don't ask for all permissions up front, build trust with your user over time and ask for permission as needed."David Whittaker
@rundavidrun :@jamespearce#QConSF claims key factor to creating a viral app is to give users a way share their experience via social storytelling.David Whittaker
@rundavidrun :@jamespearce#QConSF says "the mobile phone is the ultimate social connectivity device enabling a personally adapted experience."Maximiliano Firtman
@firt :Facebook on HTML5: “our challenges are not everyone’s challenges”@jamespearce#qconsfMaximiliano Firtman
@firt :Performance on mobile is so important. For Facebook, 2x faster, means 2x content consumption and 1.5 stars to 4 stars in 3 weeks#QConSF
Chris Ferdinandi attended this session:
Facebook gets traffic from 7,000 devices a day via the mobile web, compared to what looks like about a third of that from both iOS and Android native apps. The Facebook app is currently one of the top 50 most downloaded apps in the Apple App Store, and is #4 under social networking. And yet they get more traffic from their mobile HTML5 website than from their Android and iOS apps combined.
Native apps are great. Personally, I’d rather use the Facebook app than visit their site on mobile Safari. But people are still going to websites, even sites they visit regularly, on their mobile devices.
The Mobile Web Developer's Tool Belt by Pete LePage
Twitter feedback on this session included:
David Whittaker
@rundavidrun :Thanks@petele for curating all these great web development tools for us!#qconsf -HammerJS, CodeKit, LawnChair, FastClick, JSConsole, etc.desbiens
@desbiens :Remote debugging w/ firefox, or chrome remote debug on android or ios w/safari, mobile tool belt#qconsfDavid Whittaker
@rundavidrun :@petele "You have the opportunity right now to 'make awesome.' Push the web and go make stuff!" http://m.chromeexperiment.com#qconsf
Loutilities has attended this session:
Pete LePage (@petele) gives a great overview of all the different tools you should consider when embarking on mobile web application development.
Sublime, Codekit
Great sites for good design patterns for mobile dev:
- www.mobile-patterns.com shows app screenshots to get ideas
- pttrns.com is also nice
Start with boilerplates from SenchaTouch, bootstrap, and maybe even jqueryMobile
github.com/ftlabs/fastclick for improving performance of button clicks (usually delayed by 300ms for pinch zoom, slide, etc.)
To debug, jsconsole.com can be used for remote debugging on other devices and will show stuff on site….coolio.
Hammer.js is cool for gestures to listen to those events
brian.io/lawnchair good to store some transient data by abstracting away for IndexDB, WebSQL or LocalStorage
@media, cool for high density resolution or using CSS -webkit-image-set for both 1x and 2x image densities for hi-rez icons
CharlesProxy good to test network latency or Network Link Conditioner on Mac
Chrome Dev Tools have cool features to override geolocation, user agent, and device orientation
Dynamic Languages for the Web
Polyglot Web Development With Grails 2 by Jeff Brown
Twitter feedback on this session included:
Gareth Rushgrove
@garethr :OH: "in dynamic languages it's important for the compiler to let you call things that don't exist" great point from@jeffscottbrown#qconsf
Using Node.js to Explore Holistic Internet Application Performance by Tom Hughes-Croucher
Twitter feedback on this session included:
Ismail Elshareef
@ielshareef :“How does latency affect our servers?” asks@sh1mmer. Node.js is great for addressing mobile latency. Good talk.#qconsfdesbiens
@desbiens :Event driven programming a la nodejs is well adapted to mobile architecture@sh1mmer#dynamiclanguage#qconsfdesbiens
@desbiens :Discovering how mobile app architecture is so much different from desktop web@sh1mmer#dynamiclanguage#qconsfstuartblair
@stuartblair :Kudos to@sh1mmer for delivering the most compelling case for node.js I've seen yet.#qconsf
Java Renaissance
Android App Anatomy by Eric Burke
Twitter feedback on this session included:
David Whittaker
@rundavidrun :In#Android push to server, add bundle to task queue, check network, remove from queue if available, repeat until empty.@burke_eric#qconsfDavid Whittaker
@rundavidrun :@burke_eric demos some of@square's best open source#Android libs: Tape (FIFO), Otto (bus), and Dagger (CDI) http://square.github.com#qconsf
Loose Concurrency & CAP Theorem Today
Exploiting Loopholes in CAP by Michael T. Nygard
Twitter feedback on this session included:
rama roberts
@ramarob :Nodes in diff. groups can reconstruct linear history using GPS time to get around CAP limitations. See Google Spanner. -@mtnygard#QConSFadrian cockcroft
@adrianco :#qconsf@mtnygard now talking about PACELC by@daniel_abadi Partition means trade off Avail/Consistency, Else trade off Latency/Consistency
Fear No More: Embrace Eventual Consistency by Sean Cribbs
Twitter feedback on this session included:
Alex Popescu
@al3xandru :"Any sufficiently large system is in a constant state of partial failure"@justinsheehy via@seancribbs#qconsf
The CAP Balancing Act: Building Reliable Data Stores by Jeremy Edberg
Alex Handy reported for SD Times:
Jeremy Edberg, manager of cloud operations and reliability engineering at Netflix, said he learned how to store data for quick access while he was working as an engineer at the news site Reddit.
Of the Reddit stack, he said, “There was Cassandra, Postgres and memcached. The biggest way Reddit does their resiliency is through the sharding of Postgres. Reddit splits writes across four master databases, which have different types of things. Each one has at least one replica in another zone. Votes on links only get one replica, while comments have 17 replicas spread across different zones. Reddit uses Postgres like a NoSQL store. There are just two tables: a thing table and a data table.”
Edberg said one of the keys to success was using consistent key hashing in memcached. “It's important. If you're using a naive hashing algorithm, which every memcached library uses by default, [you can run into performance issues],” he said. “If you add a server, it will shift three quarters of your data to a different server, so you invalidate three quarters of your cache. Consistent key hashing is the reverse: One quarter of your data will be in the wrong place. This makes it so you don't put a huge load on your database when you add a new node. Protip: Use consistent key hashing. If you're using memcached, you're probably using a naïve one if you haven't set it manually.”
Finally, Edberg said that designing for scalability from the start is key, but the initial number of servers is also an important decision. “Going from two to three is hard,” he said. “When you have one server, going from one to two is even harder to do, which is why I recommend going for three from the beginning. Scaling from three to six is a lot easier than from one to two or two to three.”
Mechanical Sympathy
JVM Mechanics: A Peak Under The Hood by Gil Tene
Twitter feedback on this session included:
Pavlo Baron
@pavlobaron :Gil Tene at#qconsf: "most of the people are measuring something that is not a problem"..."99.9% of the time your heart will keep beating"Pavlo Baron
@pavlobaron :Gil Tene at#qconsf: "backpressure doesn't let us measure bad things - they just don't get reported" - this guy is amazing!Pavlo Baron
@pavlobaron :Gil Tene at#qconsf: "percentiles are good, but always measure the maximum"
PHP on the Metal with HHVM by Keith Adams
Twitter feedback on this session included:
Martin Thompson
@mjpt777 :"memcpy will fill about 1/3 of the i-cache because it is 11KB of code!" - Keith Adams#qconsf
Running with the Devil: Mechanical Sympathetic Networking by Todd Montgomery
Twitter feedback on this session included:
Martin Thompson
@mjpt777 :"Batching is a key technique to making applications perform better..." -@toddlmontgomery#qconsfMartin Thompson
@mjpt777 :Apply "Application Level Framing" to greatly simplify your networking designs.@toddlmontgomery#qconsf
NoSQL Emergence
Architectural Patterns for High Availability by Adrian Cockcroft
Twitter feedback on this session included:
John Martin
@tekBuddha :“Things we don’t do at Netflix: plan for capacity.” -@adrianco#QConSFDan Cundiff
@pmotch :Listening to Netflix@adrianco talk about HA patterns. I knew about Chaos Monkey, but not Latency Monkey and Chaos Gorilla.#QconSFDan Cundiff
@pmotch :At#QconSF I've observed several big names state they're migrating to@ubuntu. Netflix is for example. Makes sense to me.Pavlo Baron
@pavlobaron :Netflix pushes devs to open source all the code they write. This creates natural pressure for quality. I so steal this idea...#qconsfCyberPunkie
@CyberPunkie :#QConSF Netflix and massive scale: 500 Cassandra instances!
MongoDB Large-Scale Data Centric Architectures by Kenny Gorman
Twitter feedback on this session included:
Jim Constantine
@jgc94131 :mongoDB jewels: your indexes must fit in memory / one write lock / ops nightmare / return code on insert doesn't tell you much#QConSF
Loutilities has attended this session:
Kenny Gorman presents on the gotchas with MongoDB and how to scale it appropriately.
Denormalize till it works to get to true scale; split vertically and then horizontally
MongoDB is an OPS nightmare
MongoDB challenges:
- Course lock scope; 2.2 has DB lock scope; 2.0 was global
- Visibility
- Schema – still need to business stuff
- When bad things happen…it goes down fast
Design for scale:
- Denormalize into diff. DB’s
- Tune your workloads
- NoORM; dump it
- Shard early (Think about your shard key in advance)
- Replicate (How to get availability)
- Load test pre-prod
Programmable Web
Caching Hypermedia APIs by Tim Stokes
Twitter feedback on this session included:
Kris Trujillo
@kris_trujillo :WS-* The Hypermedia Dark Ages - great way to describe it! programmable web #qconsfKris Trujillo
@kris_trujillo :Transfer state in the body not in the URI - caching hypermedia #qconsfKris Trujillo
@kris_trujillo :Learned a new trick with Content-Location playing into caching of POST responses - great stuff! - caching hypermedia #qconsfKris Trujillo
@kris_trujillo :Keys to remember when caching hypermedia #qconsf http://t.co/60igzTtVKris Trujillo
@kris_trujillo :Best approach to designing hypermedia with caching in mind#qconsf pic.twitter.com/EtnoVyXp
Real-Life Stories on Building Hypermedia APIs by Mike Amundsen
Twitter feedback on this session included:
Kris Trujillo
@kris_trujillo :Bash script as a client to a Hypermedia API - very cool!@mamund#qconsfKris Trujillo
@kris_trujillo :Media Type is a pre-defined collection of Hypermedia Factors - programmable web #qconsfKris Trujillo
@kris_trujillo :Message design, not object design - programmable web #qconsfKris Trujillo
@kris_trujillo :People who are use to serializing objects will have a hard time with hypermedia - programmable web #qconsfKris Trujillo
@kris_trujillo :APIs should be usable - programmable web #qconsfKris Trujillo
@kris_trujillo :Four important aspects to consider when applying affordances to hypermedia - safety, idempotent, mutability, transclusion #qconsfKris Trujillo
@kris_trujillo :If you get the design right your users will be able to do things you never even imagined - programmable web #qconsf
Alex Handy reported for SD Times:
Mike Amundsen, principal API architect at Layer 7, said that enterprises are extremely concerned with governance and policy enforcement on APIs. But he also said that these new focuses on SLAs and productizing APIs has resulted in some interesting shifts in how APIs are handled inside big businesses.
“More often now, I'm seeing people in sales meetings who say, 'I'm not the lead architect, I'm the product lead,’” he said. “They use that phrase. Even in the IT space, sometimes these product leads are inward-facing.”
In short, APIs are being tended to with the same care and feeding a regular software product would receive. This even extends to how companies approach API design.
“Other people showing up in the room are user-experience people,” said Amundsen. “I'm seeing people are using stories and personas for the API designs. User-centered design is a big thing for them for some of our customers.”Unlike many of the API management themes out today, Layer 7 is not simply pushing REST, said Amundsen. In fact, one of the best practices his company suggests is to allow the API to be loosely coupled with its transport.
“You've got this compositional layer already that's your business piece,” he said. “We're telling people to actually consider the API as this thin veneer—like a UI—and a representation layer. Your components give you data, you compose that into something interesting, then you're representing it to the client, either as a SOAP message or a JSON payload, or as REST.”
Realtime Web
Integrating the Classic MVC Architecture with Real Time by Daniel Erickson
Loutilities has attended this session:
MVC is here to stay because…
- It provides structure to a project
- Allows people to easily jump into a project and multiple people at that
- Easier to get started with
Real-time is great for:
- Getting instant feedback from your users
- Works well on mobile (Voxer)
- WebSockets over long/short polling
How can MVC and Real Time can be mixed?
Enter Geddy…if you know Rails, NoSQL/SQL ORM layers and templating languages than Geddy is a natural progression. PostGres, Riak, MongoDB adapters; EJS and Jade TL’s. Supports RT OOTB.
Geddy gives you a reach ORM/MVC framework that you can readily work into your Node.js applications.
The Realtime Web: HTML5 WebSockets, Engine.IO, Socket.IO, SPDY, HTTP2.0 & Beyond by Guillermo Rauch
Twitter feedback on this session included:
adrian cockcroft
@adrianco :#qconsf talk on websockets and Real Time Web Is a great introduction by Guillermo, author of socket.io and engine.io
Tales from Silicon Valley
Building for a Billion Users from Silicon Valley by David Mortenson
Twitter feedback on this session included:
Christian Sanz
@csanz :"Facebook doesn't have QA, only engineers, they are responsible for quality" - David Mortenson, Facebook Engineering manager#QCon#spoton
Alex Handy reported for SD Times:
To get to a thousand engineers, said Mortensen, Facebook set up an internal boot camp where it trained groups of incoming engineers to get them up to speed on the development tools, environments and practices they'd be dealing with. He said this directed effort for on-boarding engineers helped to ease the pain of rapid growth over the past seven years.
In the end, all of Facebook’s boot camp efforts were directed at improving developer efficiency. As Mortensen said, any discrete task that takes over two minutes for a developer might as well be a coffee break because it interrupts productivity in a significant way.
To this end, Facebook spent a lot of time and effort to streamline its development environments. Every developer at Facebook has his or her own development machine in the data center, which contains a complete running copy of the Facebook code. Developers can log in and manipulate the code, then glean the changes, and start pushing toward integrating them into Facebook proper.
The problem, said Mortensen, is that the development environments were initially extremely responsive because Facebook is written in PHP. That means changes would be instantly useable on a development machine without a need to compile. Unfortunately, as Facebook's code became more complex, load times for development boxes became longer and longer.
To solve this, Facebook created HipHop, a PHP-to-C++ compiler. Once compiled, the C++ code would run much faster, but for the developers, every change now required a recompile, which could take up to 10 minutes. For Mortensen, 10-minute wait times were deadly to productivity.
After building an interpreter to stand in for the compiler in development machines, the wait time was cut down to around 10 seconds, but for Mortensen and the developers at Facebook, this was still too slow. The answer was to use a JIT compiler.
“We turned this on this spring, and got some awesome results,” he said. “When we went to the JIT compiler, there was cheering in the streets. But as a reminder, it's never done. The site continues to get richer. This is one of those things that they continue to keep fighting on. It's been a fun saga. With enough effort, almost anything that matters can be made efficient. You really want to make your engineers fast.”
Engineering Elegance: The Secrets of Square's Stack by Bob Lee
Twitter feedback on this session included:
Kris Trujillo
@kris_trujillo :"Design is not just about pixels but the entire user experience" "To end users transactions never fail" - good stuff@crazybob#qconsf
Taming HTML5 & JavaScript
Javascript Performance Patterns by Stoyan Stefanov
Twitter feedback on this session included:
David Whittaker
@rundavidrun :Load JS in iframe to avoid blocking page load but use parent doc: this is not the document you are looking for!@stoyanstefanov#qconsfFaruk Ateş
@KuraFire :“There’s lies, damn lies, and performance advice.” —@stoyanstefanov at#qconsfDavid Whittaker
@rundavidrun :@stoyanstefanov demonstrates how to measure performance of your JS code using http://jsPerf.com#qconsfDavid Whittaker
@rundavidrun :Optimize JS by reducing the number of accesses to the DOM; think of it as crossing a bridge with a high toll.@stoyanstefanov#qconsf
SPDY, err.. HTTP 2.0 Is Coming! by Roberto Peon
Twitter feedback on this session included:
Ismail Elshareef
@ielshareef :SPDY talk: don’t focus on Page Load Time. It’s irrelevant to users. Users care about how “fast the page becomes useful.”#qconsfIsmail Elshareef
@ielshareef :SPDY’s Server Push: server knows what resources the client needs and sends it over w/o explicit req from client.#qconsf#webperf#spdyDio Synodinos
@synodinos :"It's not that we don't like proxies. We don't like transparent, poorly configured ones"#qconsf#spdy#tls
The Agile Team Field Guide
Building a Learning Organization From Any Level by Matthew Barcomb
Twitter feedback on this session included:
David Whittaker
@rundavidrun :Learning organizations help companies innovate, adapt, improve quality, increase employee satisfaction, and viability.@mattbarcomb#qconsfDavid Whittaker
@rundavidrun :When approached with any challenging activity: take a learning stance, admit ignorance, and allow failure.#agile@mattbarcomb#qconsfitsmemacelle
@itsmemacelle :Learning : action vs reflection, informal vs formal, input vs output, momentum, options, create an environment#qconsf pic.twitter.com/B5kaeYDQDavid Whittaker
@rundavidrun :Learning Momentum (n.) - the more you learn anything, the easier and faster it is to learn other things.#KeepLearning@mattbarcomb#qconsfDavid Whittaker
@rundavidrun :"Come to work, do your job, and be done is a myth." To keep yourself relevant you need to constantly be learning.@mattbarcomb#qconsfitsmemacelle
@itsmemacelle :there are key times during the day that're good for learning goal is not to finish, but to practice "learning together"@mattbarcomb#qconsf
User Experience (UX)
Better Product Definition with Lean UX and Design Thinking by Jeff Gothelf
Twitter feedback on this session included:
lane halley
@thinknow :Your requirements / More effectively expressed / As hypotheses@jboogie#QConSF talk introMike Long
@mblongii :Articulate your assumptions, define hypothesis, do lightweight tests to validate@jboogie#QConSFMike Long
@mblongii :Measure outcome, not output :)@jboogie#QConSFdesbiens
@desbiens :#qconsf value learning over growth@jboogieLene Mejlby
@Mekkaz :#qconsf Design thinking is about observations@jboogieLene Mejlby
@Mekkaz :#QConSF Lean UX and design thinking is not just for designers@jboogiedesbiens
@desbiens :Focus on outcomes ... Lean ux@jboogie ##qconsfesbiens
@desbiens :Test the value first.@jboogie#leanux#qconsf
Mathieu Lalonde attended this session:
I had a bit of a realization ("ha ha!") when Jeff clarified that requirements should be viewed as assumptions. Then Jeff explained how these assumptions should be proved or disproved in the cheapest way possible. In some cases that's using a minimum viable product (MVP) in other cases mock ups or paper prototypes might do the trick!
Another practical take away was how we could write user stories in the following style: "We believe that if we implement feature XYZ, our traders will be able to use it so they can make more trades per day."
Co-making Great Products by Jeff Patton
Twitter feedback on this session included:
lane halley
@thinknow :No more "who's to blame" / Solve the problems together / Co-making products@jeffpatton#QConSF talk introLene Mejlby
@Mekkaz :Design is a learning experience, as well as common sense@jeffpatton#qconsfHans Thunberg
@hthunberg :Jeff Patton - "we are all responsible for the things we create. Let's co-operate".#qconsfLene Mejlby
@Mekkaz :Design is not a product the designers produce, it is a process designers facilitate@jeffpatton#qconsfLene Mejlby
@Mekkaz :We are not here to build software, we are here to change the world@jeffpatton#qconsfTomer Sharon
@tsharon :The faster you build crap, the more crap you have. -@jeffpatton#qconsfJeff Gothelf
@jboogie :Just because it’s written on a sticky note doesn’t make it less valid. -@jeffpatton#qconsf#LeanUXLene Mejlby
@Mekkaz :You'll need to leave the office to learn what you don't know@jeffpatton#qconsfLene Mejlby
@Mekkaz :Telling your product's story will help you identify problems@jeffpatton#qconsf
High-quality, Impactful, Fast UX Research for Engineers by Tomer Sharon
Twitter feedback on this session included:
desbiens
@desbiens :Dont listen to user, observe behavior instead@tsharon#qconsfdesbiens
@desbiens :Attitude is not a predictor of behavior#leanux@tsharon#qconsfBhavik Joshi
@joshi_bhavik :Can't help but observe parallels between#custdev#leanlaunchpad & high quality,impactful & fast lean UX design#qconsf@tsharon@thelaunchLene Mejlby
@Mekkaz : Ask about experiences rather than predictions of the future@tsharon#qconsfdesbiens
@desbiens :Ask opinion only after user experience the thing or ask about recent past experiences#leanux@tsharon#qconsfdesbiens
@desbiens :Observe, observe, observe#leanux@tsharon#qconsfLene Mejlby
@Mekkaz :Eye trackers are not mind readers! They can tell you where people are looking not what they see...@tsharon#qconsflane halley
@thinknow :"Noticeability test" let the user use the page, then give them a cut up print out and have them recreate the page.@tsharon#QConSFdesbiens
@desbiens :Noticeability test#leanux@tsharon#qconsf cut the screen in pieces and observe user assembling the puzzleLene Mejlby
@Mekkaz :A standalone A/B test only tells you what happened, not why. Do A/B usability testing instead@tsharon#qconsflane halley
@thinknow :User research reports wasteful, don't support team learning. Asked "where is the report?"@tsharon asks "what do you want to know?"#QConSF
Mathieu Lalonde attended this session:
The talk was very interesting and really clarified some concepts about usability analysis/research. One of the key take away was: "Don't listen to users! Observe their behavior."
In less than 50 min, Tomer managed to explain the psychology behind his guiding principles and then discussed three techniques that could be used:
1. High-quality noticeability test
2. Impactful A/B usability study
3. Fast, colourful collaboration tool
I thought all three techniques were very practical and seemed effective. All in all, I found the presentation's delivery very effective and its content eye-opening. Good job Tomer!
Quora on Mobile: A Product-Centered Approach to Multi-Platform Deployment by Anne K Halsall
Twitter feedback on this session included:
lane halley
@thinknow :"An app ls like an accessory to your mobile environment, I think a lot about how to honor each viewpoint."@annekate#QConSFlane halley
@thinknow :"When you go to mobile you could probably lose 20% of the features in your product and no-one would notice."@annekate#QConSFLene Mejlby
@Mekkaz :Really understand the platform you're working on!@annekate#qconsfLene Mejlby
@Mekkaz :Tailoring the product for a new platform is worth it@annekate#qconsfMike Long
@mblongii :Tailoring your product for a new platform will pay off. Don't just port it and fake it. Customers will revolt.@annekate#QConSFMike Long
@mblongii :Build once, deploy anywhere is a myth.@annekate#QConSF#mobileLene Mejlby
@Mekkaz :Design new products for mobile before Web@annekate#qconsfdesbiens
@desbiens :The mobile deferentiator is not location@annekate,#ux#qconsfene Mejlby
@Mekkaz :Make sure you have the data you need to make your apps better@annekate#qconsf
User Experience - More Than Just a Pretty Stick by Lane Halley
Twitter feedback on this session included:
Mike Long
@mblongii :Focus on the problem, not the solution. Problems start with People ;)@thinknow#QConSF
Solutions Track: No SQL
Common Graph Database Use Cases by Emil Eifrem
Twitter feedback on this session included:
Peter Yu
@ThePeterYu :Twitter is running a graph database in a JavaVM with heap size 177GB.#yam#QConSF#GraphConnect
Loutilities has attended this session:
Trends in BigData & NoSQL:
- Increasing data size
- Increasing connected data
- Semi-structured data
- Architecture – a facade over multiple services
Categories of NoSQL:
Key/Value store (heritage from Amazon Dynamo) -
- Riak, Redis, and Voldermort are implementation examples
- Strengths is that it’s a simple but also weakness
Columnar Stores
- BigTable – every row can potentially have it’s
- Examples are HBase and Cassandra, Hyper Table
- Supports semi-structured data but weakness is nested or connections
Document Database
- Collections of documents, JSON that could potentially be nested
- 90% uptick on NoSQL in MongoDB but also CouchDB
- Strength is simplicity of database, but hard to do connected data
Graph Database
- Nodes and Relationships
- Examples are Neo4j, InfiniteDB, OrientDB, etc.
- Great for connectiveness of data and complexity but harder to scale to size
Sustainable Software Development by Alexander von Zitzewitz
Twitter feedback on this session included:
David Whittaker
@rundavidrun :Alexander Zitzewitz: "Universities don't teach students how to maintain large complex software systems." How can we fix this?#qconsfDavid Whittaker
@rundavidrun :Alex Zitzewitz: "Is software 'done' when it works? Imagine building planes this way...you're done when it looks like an airplane?!?"#qconsfDavid Whittaker
@rundavidrun :Alex Zitzewitz: "Technical quality manifests itself in every line of code. Highest cost impact: having a sustainable architecture."#qconsfDavid Whittaker
@rundavidrun :Alex Zitzewitz: "Circular dependency in code is like 'code cancer'. Reduce coupling by creating interfaces."#qconsf pic.twitter.com/0ucovQWRDavid Whittaker
@rundavidrun :Alex Zitzewitz points us to the essential software quality cheat sheet. Free download: http://refcardz.dzone.com/refcardz/designing-quality-software …#qconsf
The Design of a SQL Interface for a NoSQL Database by Stephen Buxton & Mary Holstege
Twitter feedback on this session included:
Kris Trujillo
@kris_trujillo :"Why SQL on NoSQL? Because we are in the enterprise" - NoSQL#qconsfKris Trujillo
@kris_trujillo :"Focus on structured pieces of data" - SQL on NoSQL#qconsf t.co/wRcwNnLb
Opinions about QCon
Twitter feedback on this session included:
Woody Zuill
@WoodyZuill :#QConSF staff: You are the best! Thanks for your hard work and attention to detail! What a pleasure working with you. Great Conference!Josh Warner-Burke
@hybrididentity :Food at#qconsf is solid!Jonathan Schlaepfer
@Schlaeps :This food is AMAZING.#qconsfKris Trujillo
@kris_trujillo :Agreed! "@tsharon: Awesome little idea to rate#qconsf talks. Vote as you go out of the room: green, yellow, red. pic.twitter.com/Rcpet7qv”Jeff Gothelf
@jboogie :How cool is this? Instant feedback on my#qconsf talk an hour after I finished. Love it. pic.twitter.com/snneCBdWJeff Gothelf
@jboogie :This is how feedback is collected at#qconsf after each talk. Everyone taps on their way out. Super cool. pic.twitter.com/T0gJ4Khkadrian cockcroft
@adrianco :#qconsf recipe for evening keynote. First feed free beer to everyone then put on an entertaining speaker. Good job!drian cockcroft
@adrianco :Problem with attending lots of inspiring#qconsf talks is that I'm now re-writing my slides for Friday's talk, but don't have time...Stoyan Stefanov
@stoyanstefanov :Exstatic that#qconsf folks liked my js perf talk. Reviews: 0 red, 30 yellow, 87 green, pro'ly the best in my speaker "career"kennygorman
@kennygorman :Had a great time at#qconsf 2012, great to see so many brilliant people!iliketoprogram
@iliketoprogram :So much input! Feels like my head is going to explode! Thank you#qconsf and presenters! this was an awesome week!Jussi Nieminen
@jussinieminen :Thank you@QConSF#QConSF for great conference! See you next year! Over and out!Kalle Lindroos
@LindroosKE :All good things come to an end. Thx@qconsf!#qconsf
Takeaways
Twitter feedback on this session included:
Burak Yenier
@BurakYenier :@tsharon talk on User Experience testing was stellar. The kind of talk that brings me back to#QConSFosh Warner-Burke
@hybrididentity :My mind is swimming with new ideas#qconsfPavlo Baron
@pavlobaron :"I've put it up on GitHub" as a phrase clearly dominates#qconsf talks I've attendedadrian cockcroft
@adrianco :Inspired by#qconsf and thinking about a new talk based on mining the data in Edda, which was released this week@NetflixOSSJonathan Calvert
@orion_quotient :Companies like Google and Facebook, Continuous deployment is part of their competitive advantage -- Great quote from#qconsfJonathan Calvert
@orion_quotient :Back in#dc after adventures at#qconsf - the real work of action from learning begins!
Conclusion
The success of each QCon lies in the fact that each is created virtually from scratch by the small team of experts that make up the program committee.
QConSF 2012 committee members include Jez Humble, Patrick Linskey, Steve Vinoski, Dio Synodinos, Aino Vonge Corry, Floyd Marinescu, and Nitin Bharti. The committee ensures that new tracks reflect current trends in enterprise software development, and that sessions are both educational and practical. InfoQ will be publishing video-casts of most presentations over the next few months. The publishing schedule can be found on the QCon SF web site.
QCon is co-produced by Trifork (GOTO and other Conferences) and InfoQ’s parent company, C4Media. QCon conferences include QCon New York, QCon San Francisco, QCon London, QCon Berlin, QCon Beijing,QCon Hangzhou, QCon Tokyo, and QCon Sao Paulo.