BT

New Early adopter or innovator? InfoQ has been working on some new features for you. Learn more

Key Takeaway Points and Lessons Learned from QCon New York 2016

| Posted by Abel Avram on Jul 12, 2016. Estimated reading time: 48 minutes | NOTICE: The next QCon is in New York Jun 26-30, 2017. Join us!

The fifth annual QCon New York was the biggest yet, bringing together over 850 team leads, architects, project managers, and engineering directors.

Keynotes were given by Etsy CTO John Allspaw, Research Professor and NASA JPL team member Dr. Anita Sengupta, VP Product Management at ShapeSecurity and former Google Click Fraud Czar Shuman Ghosemajumder, and CAP Theorem Creator Eric Brewer.

In total, over 140 practitioner-speakers presented 79 full-length technical sessions and 16 in-depth tutorials over the course of five days, providing deep insights into real-world architectures and state of the art software development practices from a practitioner’s perspective.

On Tuesday evening we hosted six different BOF sessions, including one from The Paper's We Love Meetup (New York City) who put together a series of 4 PWL "mini" (15~20 minute) presentations by Evelina Gabasova (@evelgab), Eric Brewer (@eric_brewer), Ines Sombra (@randommood), and Caitie McCaffrey (@caitie).

Videos of most presentations were available to attendees within 24 hours of them being filmed and we have already begun to publish them on the InfoQ site. You can view the publishing schedule on the QCon New York website.

InfoQ also reported from the event, and recorded podcasts with a number of speakers. This article, however, presents a summary of QCon New York as blogged and tweeted by attendees.

Keynotes

Design for Continuous Evolution

by Eric Brewer

Twitter feedback on this keynote included:

@philip_pfo: Immutable machine images aren't sufficient - you want an immutable service deployment (graph of resources). @eric_brewer #qconnewyork

@danielbryantuk: Construction as a step in application deployment (with deploy-time configuration) .@eric_brewer #qconnewyork https://t.co/ARTAxXYMlT

@philip_pfo: Construction graph is a deterministic repeatable process (deploy flow for cont service evolution). #qconnewyork https://t.co/DSDIxTfV1c

@danielbryantuk: Interesting thoughts for 'construction' of immutable app artefacts via type-driven config. @eric_brewer #qconnewyork https://t.co/3TngRCUiB1

@thenewstack: Construction is a step that should be added to deployment for system robustness @eric_Brewer #qconnewyork https://t.co/xYGxlvTfsH

@philip_pfo: The easy answer might not be optimal, but might get you the speed you need to launch. (@eric_brewer on flexible labels #qconnewyork)

@thenewstack: Reconciliation: A control loop for distinguishing intent from current state, and reconciling the two @eric_Brewer #Qconnewyork

@charleshumble: High availability just means the name is available.  We can start the service fast enough we get the request.@eric_brewer #qconnewyork

@danielbryantuk: Load balancing with labels enables (system) evolution @eric_brewer #qconnewyork https://t.co/A2ZVVSF8IV

@philip_pfo: Load balancing is a key abstraction for cont evol, decoupling reachable name from actual service. #qconnewyork https://t.co/hBmDW2KYO6

@danielbryantuk: Dealing with stateful services is still hard, but we're making progress @eric_brewer #qconnewyork https://t.co/5RkROnYxgp

Engineering the Red Planet

by Anita Sengupta

Twitter feedback on this keynote included:

@TechTargetFred: Anita Sengupta giving and amazing talk on the software engineering behind missions to Mars #qconnewyork https://t.co/3IcjI9lqv7

@jchyip: The Mars Curiosity landing stages were absolutely crazy #qconnewyork

@TechTargetFred: Did you know Mars rovers have lasers? Rad. #Lasers. ?? #qconnewyork https://t.co/xqj7Xz9gfc

@aspyker: Mars Rover has virtualization to restore backup computers by updating OS. Seems they bake & red/black for reliability too. #qconnewyork

@TechTargetFred: Anita Sengupta's talk about her work on Curiosity. Honestly...what am I doing with my life? #qconnewyork #mars #woah https://t.co/yTd8QMlpko

Incident Response: Trade-offs Under Pressure

by John Allspaw

Fred Churchville attended this session:

One of the first notes that Allspaw made is that organizations cannot simply rely on tools to make it easier to understand how and why incidents are occurring. Instead, teams need to rely on processes and reasoning in order to truly respond to anomalies. And they cannot, he said, treat these outages as a mystery that is constantly developing over time….

In order to properly deal with outage-causing anomalies, Allspaw recommended that organizations implement a “model of reasoning” that does not “distinguish between diagnosis and therapy.”

Listeners were also warned not to fall into the traps of “thematic vagabonding” and “cognitive fixation” – meaning that those debugging the code can become so wrapped up in simply fixing bugs and symptoms that they fail to delve further into discover the actually root cause of the issue….

He provided a list of “prompts” that teams can use to frame particular question, dividing the questions into four “stages” of incident response: observations, hypotheses, coordination and suggesting actions. By asking these questions, team members may be able to avoid “cognitive fixation” and get to the root of the problem.

Allspaw also talked about the importance of linking anomalies to any known, recent changes in the code or application and, more so, of having peers review your hypotheses.

Twitter feedback on this keynote included:

@charleshumble: Computers do not resolve outages. People do. @allspaw #qconnewyork

@charleshumble: Anomaly response does not happen the way we might imagine it does. @allspaw #qconnewyork

@danielbryantuk: Tools do not always make things easier to understand...@allspaw #qconnewyork https://t.co/Lg8HfPbqH0

@philip_pfo: Process for recognizing and analyzing issues - explorative interconnected loops. #qconnewyork @allspaw https://t.co/XyZh0Yao5H

@philip_pfo: Reframe incident response as anomaly response - detecting and acting on unexpected and evolving problems. @allspaw #qconnewyork

@danielbryantuk: Most IT incidents are not like murder mysteries. An incident (and env) is often not static @allspaw #qconnewyork https://t.co/ctPMGEBd3Q

@danielbryantuk: I didn't know the phrase before today, but I've definitely done my fair share of 'thematic vagabonding' during incidents... :)#qconnewyork

@charleshumble: Does the incident correlate with a known change? No. Freak out because it could be *anything*' @allspaw #qconnewyork

@aspyker: keynote - steps of dealing with anomaly (incident) response #qconnewyork https://t.co/L1KfoghKQv

@emregurhan_isik: Anomaly response does not happen the way we might imagine it does, we have to look hard at how it actually happens. #QConNewYork

@jchyip: Start with a low commitment to concepts. End with abstractions, don't start with them. @allspaw #qconnewyork

@caitie: The worst part of our domain is that we begin with abstractions, not end with them @allspaw #qconnewyork

@TechTargetFred: John Allspaw on incident (or anomaly, rather) response. "An outage is not a detective story. It's static, and it's there." #qconnewyork

Security War Stories

by Shuman Ghosemajumder

Jeanne Boyarsky attended this keynote:

  • “Computer security is like broccoli. You know you should have it but would rather have chocolate”
  • “My objective today is not to scare you”
  • “If I were legally responsible for the security of the software I wrote, I would quit my job immediately … and probably move to a non-extradition country”

How make completely secure:

1.     Get rid of business model

2.     Don’t connect to internet

Security is relative – really about tradeoffs….

Future [security threats]

  • Because the present isn’t scary enough, the future is scarier
  • Attack surface is related to complexity …
  • New attack surfaces:
    • autonomous vehicles
    • always on IoT devices like Amazon Echo could do surveillance
    • Apple Home – can unlock your home doors. Can turn on your lights/stereo at night randomly like horror movie.
  • New technology creates new attacks
    • Audio/video fidelity – could steal fingerprints from far away or record every conversation from far away
    • Battery life – run surveillance device for weeks
    • CPU power – defeat encryption
  • New attacks create unforeseen consequences – now we have to take off shoes before flying

Twitter feedback on this keynote included:

@danielbryantuk: Being responsible for security within software development is scary...#qconnewyork https://t.co/IMGQe866kV

@msheldon83: Security is never absolute. It's always about tradeoffs #QconNewYork. (The good news is that it makes for Great War stories)

@philip_pfo: Complete security could be achieved through isolation, but creating value requires participation. Balance with trade offs. #qconnewyork

@jeanneboyarsky: first hacker - Nevil Maskelyne? in 1903, morse code interfering with telegraph “rats” and more #qconnewyork

@philip_pfo: Early hacking story (1903), message injection via morse code. #qconnewyork keynote https://t.co/wvthdqkKsj

@jeanneboyarsky: 2002 - Bill Gates finally made security a top priority at Microsoft #qconnewyork - 14 years ago feels like so long. but still late

@charleshumble: Security wasn't a top priority at Microsoft until a company wide edict from @BillGates in 2002.  Shuman Ghosemajumder #qconnewyork

@charleshumble: With 1M stolen passwords, you can quickly take over 10,000 accounts without anyone knowing about it.. Shuman Ghosemajumder #qconnewyork

@jchyip: Credential stuffing pretty much means don't repeat passwords across sites / apps #qconnewyork

@rcb4m: A breach anywhere is a break everywhere.Shuman Ghosemajumder #qconnewyork

@jeanneboyarsky: Good - Netflix reset passwords based on ANOTHER company's breach #qconnewyork.  better than credit monitoring. plus not their site

@philip_pfo: The greatest attack is invisible. Credential stuffing is close- known passwords tried via a bot net. #qconnewyork https://t.co/vjpQacyFKL

@aspyker: Captcha defeat as a service #qconnewyork Security Keynote (CDaaS?) https://t.co/9yjXBeDjW6

@jeanneboyarsky: Every website has a tiny bit of attacks. These crawlers are just the background radiation of the internet. opportunistic probes #qconnewyork

@philip_pfo: Keyser Soze attacks: hidden in plain sight, hard to detect/harder to prevent, but not impossible. #qconnewyork https://t.co/WD9whJe8aC

@danielbryantuk: New technology creates new attack surface. We should be aware of this as tech becomes invasive at home#qconnewyork https://t.co/W7jZCT9fJr

@sangeetan: Security is about tradeoffs like everything else 80/20 rule applies #qconnewyork keynote https://t.co/IuyKn6DsLH

Tracks and Talks

Architectures You've Always Wondered About

DigitalOcean: Microservices in Your Datacentre

by Phil Calçado

Twitter feedback on this session included:

@danielbryantuk: Client libraries lead to hidden business logic. Why not use something like gRPC for an IDL? @pcalcado #qconnewyork https://t.co/bAa3X3ZNep

@randyshoup: Pay the same amount of attention to the schema for events as you do to your RPC interfaces. @pcalcado #QConNewYork

@danielbryantuk: What should be a service? We know the size should be related to @boicy's head... @pcalcado #jokes #qconnewyork https://t.co/oswYlA9zA1

@danielbryantuk: Hang on, is that a @swardley map in @pcalcado's #qconnewyork talk? Oh yes! "strategy and value are super important" https://t.co/zW3Bv3kWty

Scaling Container Architectures With OSS & Mesos

by David Greenberg

Joab Jackson attended this session:

“Scaling is painful,” admitted David Greenberg, who shared his experience of his time implementing an Apache Mesos cluster manager at hedge fund investment firm Two Sigma, in a talk at QCon ….

“Mesos scaled really well, it supported batch jobs and containers,” Greenberg said.

The company had to do a lot of work to prepare for the new Mesos-based architecture, however. Many applications had to be refactored, which is a job that is both complex and offers little in the way of immediate benefits. But the work is important because it sets the stage for faster advancement.

The company needed more sophisticated scheduling algorithms than what Mesos could offer through its Dominant Resource Allocation (DRF). So, the company extended DRF with Cumulative Resource Share (CRS), embedded into a home-built scheduler called Cook, which dynamically adjusts demands against resources.

Also, they learned that standard debugging techniques don’t really work for distributed systems. When something goes wrong, it’s not like you can just SSH into a machine and scan the logs, Greenberg pointed out. Monitoring is needed.

For this job, Two Sigma built its own monitoring system, called Satellite. … The company uses PagerDuty to alert administrators during off-hours. They also turned to Elasticsearch search engine and its Kibanavisualization package, for visualizations, ad-hoc queries and dashboards.

Despite Greenberg’s glowing recommendation, not everyone at QCon was so enamored of Mesos.

In another talkChien Huey, a DevOps engineer at the XO Group media company, walked through an evaluation process of DCOS (Data Center Operating System), an open-source package combining Apache Mesosand the Marathon orchestration framework both maintained by Mesosphere. … DCOS had some limitations, however. … Huey found no way to automate container operations that used private container registries. Mesosphere does offer a manual workaround for working with private registries, so Huey developed a script that would refresh the needed encryption key every 12 hours.

Huey was also not impressed with Mesosphere’s approach to auto-scaling, which involves installing an agent on the master node. “What if that node goes down?” he asked. Other companies have found workarounds to this issue, however. Netflix has developed Fenzo, for instance, to help with this issue.

Twitter feedback on this session included:

@philip_pfo: What was a feature in early days can turn into a bug as your system scales up (theme from @probst_kathrin @dgrnbrg #qconnewyork talks)

@aspyker: Schedulers seem to follow the organization that created them @dgrnbrg #qconnewyork

Scaling Uber to 1,000 Services

by Matt Ranney

Twitter feedback on this session included:

@randyshoup: The time that Uber services are the most reliable is on the weekend because Uber engineers are not changing them @mranney #qconnewyork

@DataMiller: Using a browser protocol [HTTP, REST, JSON] is not that useful at scale. #qconnewyork

@emregurhan_isik: Good perfomance is not required but "Known" is. says Matt Ranney. #QConNewYork

The Architecture that Helps Stripe Move Faster

by Evan Broder

Twitter feedback on this session included:

@danielbryantuk: Move fast and break things... unless you are dealing with looking after my money @ebroder from Stripe #qconnewyork https://t.co/3BgqQUFS0c

@danielbryantuk: We've had several new API features which were accidentally quadratic [in terms of complexity] @ebroder being brutally honest #qconnewyork

@danielbryantuk: Gating functionality to ease API evolution seems like a good idea, until time passes.@ebroder #qconnewyork https://t.co/QUSEjw2IZf

@IbraSho: [about Go] ... it's performance was going to be better than Ruby, because it's really hard not to do. @ebroder #QConNewYork

@danielbryantuk: Ragging on the 'loose' Ruby Rack parser for nested URI parameters at #qconnewyork with @ebroder https://t.co/ERt7IRKhAo

@danielbryantuk: Create a test zoo by capturing tricky API example case 'animals' e.g. Decoding issues...#qconnewyork https://t.co/39Bw57Uk0v

The Netflix API Platform for Server-side Scripting

by Katharina Probst

Twitter feedback on this session included:

@philip_pfo: Decoupled deployments enable increased developer velocity at Netflix. @probst_kathrin #qconnewyork https://t.co/E1UHya8u1d

@chobberoni: Make your API flexible. Make your API resilient. @probst_kathrin #qconnewyork

@danielbryantuk: Netflix edge API functionality can be changed at any time with the deployment of Groovy scripts #qconnewyork https://t.co/VpeCeJmXoF

@philip_pfo: Mitigating strategies expand your runway, but at some point you need to rethink if you need a runway at all. @probst_kathrin #qconnewyork

@danielbryantuk: Netflix are planning to deploy API edge scripts/apps in containers using node.js@probst_kathrin #qconnewyork https://t.co/zmqFzsNxm2

@philip_pfo: Netflix nextgen arch for device facing logic- containers/process isolation, JS/node. @probst_kathrin #qconnewyork https://t.co/urO8HYgr1Z

@danielbryantuk: Netflix edge API local development with containers... avoiding println statements :-)@probst_kathrin #qconnewyork https://t.co/aOCgcW9T77

@aspyker: .@probst_kathrin talking APIs containers use. Learn about Project Titus (the boat in the pic) tommorow. #qconnewyork https://t.co/iydp4jPGd6

@danielbryantuk: YES! –> "Keep thinking about the NFRs of your systems, and how they change over time, and how you should adapt" @probst_kathrin #qconnewyork

@danielgiri: Circuit breakers key on front-end server code (in JS), like in server-side m-services @probst_kathrin #QConNewYork @NetflixOSS

Better than Resilient: Antifragile

0 To 100 Days - Running DRTs at Dropbox

by Tammy Butow & Thomissa Comellas

Twitter feedback on this session included:

@philip_pfo: Resiliency testing tools and methods landscape (from @thomissa @tammybutow #qconnewyork) https://t.co/d98SJjhcTB

@philip_pfo: Theme underlying implementing DRTs at Dropbox: data capture & analysis around incidents, use that to inform teams. #qconnewyork

@KoltonAndrus: Communication is key for safe failure testing @thomissa @tammybutow #qconnewyork https://t.co/UQxx1kqAhk

@KoltonAndrus: Love the preflight checks before failure testing and verification at every step. Very disciplined! @tammybutow @thomissa #qconnewyork

Chaos Kong - Endowing Netflix with Antifragility

by Luke Kosewski

Twitter feedback on this session included:

@KoltonAndrus: Chaos is a sword, Traffic is a shield #qconnewyork #netflix https://t.co/i2HqwedwP1

@danielbryantuk: A quick case study for the benefits of @NetflixOSS Chaos Kong at #qconnewyork https://t.co/MfIxeKdr25

@KoltonAndrus: #Netflix flow visualization showing region failover #qconnewyork https://t.co/5NxekmON8T

@danielbryantuk: Netflix project 'Nimble' - provisioning hot standbys without all of the bad bits...#qconnewyork https://t.co/ddwQuBQ9Z4

@KoltonAndrus: DNS as a DB, enabling better DNS control #netflix #qconnewyork https://t.co/ma40Jbl1jg

@KoltonAndrus: Winning with Chaos Kong - region evac. Works when called upon because it's practiced often #netflix #qconnewyork https://t.co/LMuJhxLSVi

Improving Resilience by Creating Storms in the Cloud

by Michalis Zervos

Twitter feedback on this session included:

@KoltonAndrus: Go the extra mile to build resilient systems #qconnewyork @mzervos discussing Azure's failure testing https://t.co/VCAPmqFd6C

@KoltonAndrus: Injecting a fault is not enough, you must verify behavior. -@mzervos #qconnewyork

@KoltonAndrus: We built a 747, now we need to learn to fly it. - @mzervos on learning to use a fault injection system #qconnewyork

@KoltonAndrus: Changing dates on the OS to test certificate invalidation bugs. Great idea by @mzervos #qconnewyork

@KoltonAndrus: Many uses of fault injection. Training a big (nice) surprise @mzervos #qconnewyork https://t.co/XF7ARH7Up7

@KoltonAndrus: Gamification of fault injection - @mzervos #qconnewyork https://t.co/OAguFwlZHO

Containers: From Dev to Prod

An Approach to a Container-happy Tech Department

by Michael Venezia

Twitter feedback on this session included:

@sangeetan: Low appetite for change makes container adoption hard in large orgs - Mike Venezia #qconnewyork https://t.co/Uf4mdR1bS0

@tealagile: Most companies are already ready for #containers #qconnewyork https://t.co/cLFOkHggs3

Getting Towards Real Sandbox Containers

by Jessie Frazelle

Jeanne Boyarsky attended this session:

Today

  • Docker typically runs as a privileged user.
  • Containers are meant to limit the damage from a compromise. The world an attacker can see inside the container is a limited one)
  • Want unprivileged containers so don’t need sudo/privileged access to launch container in the first place.

Chrome sandbox on Linux

  • … doesn’t need to be run as root.
  • each tab is in its own namespace – process only knows about itself
  • if Chrome can do this, why not Docker?

Future

  • Won’t need to run as root
  • Can customize sandboxes from defaults, better UX for dealing with security policies.
  • “postgres should maintain a postgres profile”

Twitter feedback on this session included:

@sangeetan: Containers aren't the solution to your security issues but they can limit the damage @jessfraz #qconnewyork https://t.co/ionax1YKkl

@charleshumble: cgroups limit what the process can use. Namespaces limit what the process can see.@jessfraz  #qconnewyork

@charleshumble: User namespaces can be created with root if the (uid,gid)_map is mapped to the current user creating the namespace.@jessfraz  #qconnewyork

How Containers Have Panned Out

by Adrian Trenaman

Jeanne Boyarsky attended this session:

Projects

  • ION Roller
    • Immutable deployment – Destroy original cluster when done with this process for Docker upgrades.
    • Slow to setup/tear down environments.
    • Can be expensive for continuous deployment
    • Open source, but in house.
  • Nova
    • Uses yaml to deploy
    • No Docker registry. Base images are on Docker. Releases aren’t needed on there so go straight to Amazon
    • Less boilerplate
    • Immutable deployment on mutable infrastructure. Docker container is immutable.
  • Fighting bit rot, chaos-monkey style
    • Don’t want things to run forever in Prod.
    • What if there is a security vulnerability
    • Every day, kill oldest AMI randomly. This forces latest AMI with fixes and fail early.
    • Doesn’t solve vulnerability in Docker container. Would need new release with new base image for that. Hasn’t happened to Gilt yet.
  • Sundial
    • For running batch jobs
    • Automatically reschedules if fail
    • Define a process – group of tasks with dependencies between them…

Using Docker as a local build platform

  • Different projects use different versions of build tools
  • Docker can be used as a versioned build container.
  • A year from now, we will still have everything need to run code

Lessons

  • Containers let separate what deploy from how, where to deploy it
  • Still the wild west on how containers are deployed
  • Seek immutability in the container, not in the stack
  • The competitive advantage for Gilt is to be able to deploy quickly/frequently/safely to production and therefore can innovate faster. Gilt lets engineers deploy whenever they want without asking permission.

Twitter feedback on this session included:

@aspyker: 90% of all of our developers test in production and have abandoned staging @adrian_trenaman #qconnewyork

@sangeetan: 90% of Gilt devs test in Prod using canary pipeline @adrian_trenaman #qconnewyork https://t.co/Y8WbVwOpYK

@jeanneboyarsky: We could solve this now, or just wait six months and Amazon wil provide a solution #qconnewyork

@jeanneboyarsky: interesting idea – using Docker as build container so still have tools in a year if you need to a patch or run and old version #qconnewyork

@sangeetan: Seek immutability in container not stack - lesson from @gilttech @adrian_trenaman #qconnewyork

Scheduling a Fuller House: Container Mgmt @Netflix

by Sharma Podila & Andrew Spyker

Twitter feedback on this session included:

@kief: Over 1/3 of US Internet traffic in the evening is @netflix - @aspyker #QConNewYork

@philip_pfo: Containers address the impedance mismatch between what resources a job needs & what machine types are available. #qconnewyork @aspyker

@kief: Extra benefit of dev <-> prod consistency is being able to replicate & debug production issues @aspyker #QConNewYork

@danielbryantuk: Why containers for Netflix? A consistent and easier developer experience, and also easier debugging of prod deployed artefacts #qconnewyork

@kief: Most container orchestration solutions focus on datacentre, not elastic platforms @aspyker #QConNewYork

@philip_pfo: Containers @ NFLX-  easier to build on top of existing infra rather than rewriting the world. #qconnewyork https://t.co/YZouzj5Ehy

@danielbryantuk: Netflix buy vs build for their container management platform 'Titus'@aspyker #qconnewyork https://t.co/HxU2fNVlfz

@kief: Running Docker multi-tenant on AWS you lose lots of AWS capabilities (e.g. networking) @aspyker #QConNewYork

@philip_pfo: Practical & tunable outpaces perfect but long time to implement with container schedulers. @podila #qconnewyork

@danielbryantuk: Netflix are carving up AWS r3.8xl boxes (32 CPU, 244Gb RAM) for deploying containers via Titus @aspyker #qconnewyork https://t.co/dMtJKGO89P

@danielbryantuk: Coming soon from Netflix - SPaaS - stream processing as a service... Seems like a popular topic :-)#qconnewyork https://t.co/p3HKgKPSF2

@kief: Traffic team orchestrates services for teams, doesn't use a consulting model like a "traditional" SRE team, @podila #QConNewYork

@TechTargetFred: Netflix's container mgmt at #qconnewyork. "The real value here is we have a single, consistent cloud platform." https://t.co/4mYMQ5gF2K

Culture as Differentiator

How to Optimize Your Culture for Learning

by Sonali Sridhar

Twitter feedback on this session included:

@mindweather: “Fear is an obstacle to learning.”??@jollysonali #QConNewYork

@tealagile: @jollysonali fear keeps us from saying "I don't understand" #qconnewyork

@mindweather: “Being transparent about our beliefs re-enforces learning“@jollysonali #QConNewYork

@tealagile: Treat people like adults??@jollysonali #qconnewyork @recursecenter

@aspyker: Every one of your strengths becomes a weakness if you don't evolve #qconnewyork

Learnings from a Culture First Startup

by Sunil Sadasivan

Twitter feedback on this session included:

@tealagile: @sunils34 #culture is as important as our product #qconnewyork

@mindweather: You know you’re in a dysfunctional culture when your people file these kinds of tickets.@sunils34 #QConNewYork https://t.co/Q4ditCT30G

@jchyip: Crafting culture should be painful (if you're doing it right) @sunils34 #qconnewyork

@morten_moe: https://t.co/ca1VNkMFFL Transparency taken to a new level. #qconnewyork

@mindweather: “Culture is defined and tested in your low moments.”@buffer’s @sunils34 at #QConNewYork

@jchyip: Hiring for culture fit assumes your current culture is perfect @sunils34 #qconnewyork

@tealagile: @sunils34 @buffer need to shift from culture fit to culture contribution #QConNewYork

@jchyip: Buffer bootcamp is a 45 day full-time contracting period #qconnewyork

@jchyip: After open salaries, Buffer applications went from 300 to 4000 a month #qconnewyork

@jchyip: Candidates often say 'I know more about Buffer than my own company' @sunils34 #qconnewyork

The Psychology of High-performance Organizations

by John Willis

Twitter feedback on this session included:

@danielbryantuk: Making a strong argument on watching out for burnout in high performing organisations, @botchagalupe #qconnewyork https://t.co/eZdT4APgTS

@msheldon83: Psychology of High Perf Organizations => Beware of burnout #qconnewyork. What do you worship?

@danielbryantuk: Taking a 'system view' of burnout, and watching for mismatch between individual and org @botchagalupe #qconnewyork https://t.co/1YnZmAO2Wm

@msheldon83: Finding a company that matches your values is important to personal health and success #qconnewyork

@danielbryantuk: Anyone seen the Enron culture deck. It looks pretty good, but they didn't live it @botchagalupe on conveying cultural values #qconnewyork

@randyshoup: Psychology of High Performing Organizations: faster deploys, better reliability, lower burnout, growth mindset @botchagalupe #qconnewyork

@danielbryantuk: Do not underestimate the power of the mind...@botchagalupe #qconnewyork https://t.co/GbKUx2fgQq

Your Org Should Think About Diversity, Yesterday

by Swati Vauthrin

Twitter feedback on this session included:

@charleshumble: There will be 1.4 million technology jobs to fill in 2020.  70% of those jobs will go unfilled. @suenacita1 #qconnewyork

@charleshumble: Buzzed has 7 billion content views/month. @suenacita1 #qconnewyork

@charleshumble: Looking at different pools.  You don’t have to have an engineering background to be a successful engineer.@suenacita1 #qconnewyork

@charleshumble: One of Buzzfeed's top engineers was a wine sommelier. Another had a degree in sculpture #diversity @suenacita1 #qconnewyork

@TechTargetFred: @suenacita1 an enlightening talk on fostering diversity at #qconnewyork. "If we're all the same, we lose our edge." https://t.co/v3rrjV8gCG

Incredible Power of an Open-sourced .NET

C# Today and Tomorrow

by Mads Torgersen

Twitter feedback on this session included:

@TechTargetFred: @MadsTorgersen talking about the new 'null conditional operator' in C#. Thanks, Mads! #qconnewyork https://t.co/biki4mi9fI

@TechTargetFred: @MadsTorgersen leading into the addition of "patterns" in C#. #qconnewyork https://t.co/ydGQFMSU9T

@TechTargetFred: C# -- now with 100% more binary literals!!!!!! @MadsTorgersen #qconnewyork #.net https://t.co/6yMATfvLGd

@TechTargetFred: @MadsTorgersen talks "twoples" in C# - huge. By the way, they're mutable ?? "Twoples are useful now!" #qconnewyork https://t.co/vsLoXBzxF8

@TechTargetFred: No C# 7 release date yet, but will be released with visual studio 15. They're NOT cancelling pattern matching. @MadsTorgersen

Data Science Meets Star Wars: The F#orce Awakens

by Evelina Gabasova

Twitter feedback on this session included:

@TechTargetFred: @evelgab talks at #qconnewyork about why F# is great for data. "F# makes it easy to parse scripts," she says https://t.co/OKPM9pPhxX

@TechTargetFred: @evelgab showing data visualization at #qconnewyork "Whenever you do data analysis, always visualize it. Always." https://t.co/3vQXXjbL01

@TechTargetFred: @evelgab showing us type providers in F# at #qconnewyork. "Type providers are amazing!!" https://t.co/AOkp8wf8Nn

Patterns & Practices for Cloud-based Microservices

by Rachel Reese

Twitter feedback on this session included:

@danielbryantuk: Why microservices at https://t.co/u5iVwMsXZ6? They evolved from idiomatic usage of F# scripts @rachelreese #qconnewyork

@danielbryantuk: The pattern-matching functionality of F# looks great, particularly for event-driven apps #qconnewyork @rachelreese https://t.co/CVGCkDuMLS

@danielbryantuk: It is very important to be functional with microservices. Favor immutability etc @rachelreese #qconnewyork https://t.co/NSS0EDcdCM

@danielbryantuk: What do F# microservices at https://t.co/u5iVwMsXZ6 look like? #qconnewyork @rachelreeseLove the functional style! https://t.co/uO2s9WUESq

@danielbryantuk: Preach!! –> "Microservices should not control their own lifecycle..." @rachelreese #qconnewyork https://t.co/rAIBU2HhjU

@danielbryantuk: Fantastic #qconnewyork microservice talk by @rachelreese. Summary points -> https://t.co/qmLtjopZfe

@danielbryantuk: We hire those from a functional lang to learn F#, and C# people to learn functional. The C# people have a much harder time #qconnewyork

Safe Systems Programming in C# And .NET

by Joe Duffy

Pierre-Luc Maheu attended this session:

Several lessons in Joe's talk came from a research project called Midori. The project was about creating an operating system from scratch using C#, which led to new insights in compiler construction as well as new strategies for high performance code.

Using managed languages to build an operating system empowers one to use the security features of C# at the memory level. It offers a solution against the exploits based on memory like code injection due to buffer overflows or format string vulnerabilities, since the runtime takes care of bound checking and type safety….

Several optimizations made by native languages compilers weren't traditionally available to managed languages. … This led to C# having bad reputation in the realm of tight, efficient low level code generated. The following optimizations were implemented recently through RyuJIT:
- Inlining (replace a function call site with the body of the called function)
- Flowgraph and loop analysis
- Static single assignment (SSA) and global value numbering
- Common subexpression elimination
- Copy/constant propagation
- Dead code elimination
- Range analysis
- Devirtualization
- Loop invariant code hoisting
- SIMD and vectorization
- Generic sharing
- Stack allocation (work in progress)…

One way to improve [GC] performance is to use structs. Structs improve performance on the following levels:
- Less GC pressure, as structs are allocated on the stack
- Better memory locality, improving cache hits rate
- Less overall memory usage, avoiding the 8-16 bytes overhead of objects in 32-64 bit applications.

One caveat of structs is that copying them may lead to a memcpy when past a certain size. For optimal performance, structs should be kept small, under 32/64 bytes.

Some features of C# 7 are going towards making low level optimization using structs easier. The tuples of C# 7 are structs, instead of the previous version System.Tuple<> which is an object. Ref return is another feature for structs, where a struct can be returned from a function while avoiding copying it….

The error recovery leads to the strategy of fail fast. Fail fast is a mechanism included in .NET, where some exceptions like StackOverflow bypass exception handlers and crash the process. This makes finding errors easier, as the exception cannot be swallowed by an overly generic exception handler. The Midori team found that they had a 1:10 ratio of recoverable errors (exceptions) to bugs (fail fast).

Innovations in Java and the Java Ecosystem

Applying Java 8 Idioms to Existing Code

by Trisha Gee

Jeanne Boyarsky attended this session:

Before refactoring

  • Make sure you have high test coverage
  • Focus on method level changes – tests should still work
  • Have performance tests too to ensure not negatively impacting system
  • Decide on goals – performance? clearer code? easier to write new code? easier to read new code? use lambdas because team used to it? developer morale?
  • Limit the scope – small incremental commits. Don’t make change that will affect the whole codebase.

Straightforward refactoring of lambda expressions

  • Predicate, Comparator and Runnable are common places where you might have used anonymous inner classes and can switch to lambdas
  • IDEs suggest where you can replace code with lambdas …
  • You lose explicit type information when using lambda. You might want to extra to method to make it clearer and call the method from the lambda….

Collections and Streams API

  • Turn for loops into collect() or forEach()
  • Remember you can call list.forEach() directly without going through a stream
  • If do something like get rid of initial list size, test performance before and after to make sure not a needed optimization …

Optional

  • Be careful. Refactor locally.
  • Easy to accidentally change something that requires changing a ton of code.

Performance

  • Lambdas expressions don’t necessarily perform better than anonymous inner classes. Similarly for streams.
  • Using lambdas to add conditional logging is a big quick win performance improvement.
  • Streams didn’t need the “initial size of list” optimization
  • Adding parallel() to stream operations sometimes performs better and sometimes worse. Simple operation often performs worse in parallel because of overhead of parallelizing.
  • Introducing Optionals increases cost.

Low Latency Microservices in Java

by Peter Lawrey

Twitter feedback on this session included:

@AgileTester: Using Microservices? Look at this scorecard and see how you are doing #QConNewYork #microservices #java  https://t.co/snFO6YVGQi

@philip_pfo: Low latency is not defined absolutely, but is relative to the context. #qconnewyork @PeterLawrey https://t.co/YqYFZEsuIx

@danielbryantuk: A modern computer is a distributed system @PeterLawrey on private/public data stores computing at #qconnewyork https://t.co/CnBjQgEqCh

@philip_pfo: Microservices are evolution not revolution. You probably are doing some of the practices already. #qconnewyork https://t.co/J7QSNstJcP

@philip_pfo: Starting with reducing your allocation rate can get you surprising performance improvements in JVM apps. @PeterLawrey #qconnewyork

Reactive Programming for Java Developers

by Rossen Stoyanchev

Jeanne Boyarsky attended this session:

Reactive libraries

  • … Project Reactor – Similar to ReactiveX, easy to bridge to Java 8 streams. (ReactiveX is like XUnit – commonality for different languages)…

Abstractions

  • Flux – sequence of 0 to n – equivalent of Java 8 stream – can convert to/from Java 8
  • Mono – sequence of 0 or 1 – can convert to/from CompletableFuture

Java 9

  • Reactive streams included (the four interfaces)
    • Publisher
    • Subscriber
    • Subscription
    • Processor
  • … Classes
    • java.util.concurrent.Flow – the four interfaces
    • SubmissionPublisher – bridge to reactie streams
    • Tie-ins to CompletableFuture and Stream…

Reactor

  • GA release scheduled for July
  • Currently called 2.5. Might changed to 3.0…

Other reactive efforts

  • MongoDB – reactive driver
  • Couchbase – reactive driver
  • Thymeleaf – split templates into chunks and throttle to respect back-pressure

The Engineer's Guide to Hotspot JIT Compilation

by Monica Beckwith

Jeanne Boyarsky attended this session:

Compilation Techniques and Notes

  • Pre-compiled/ahead of time
  • profile guided – based on critical hotspots
  • Adaptive optimization (Java uses Profile guided and Adaptive optimization)
  • Identify root of compilation
  • replace method or on stack – depends on number of times through loop
  • Server compiler has a higher threshold than client compiler for the threshold at which you need optimizations
  • Tiered compilation – tier 1 is client compiler with no profiling info, tier 2 and 3 are client compiler with profiling info. Then comes server compiler
  • CodeCache - order of magnitude larger when tiered compilation is enabled. If need more can use -XX:ReservedCodeCacheSize
  • Inlining – many different parameters when figuring out when to inline …
  • OOP (ordinary object pointer) is a managed pointer. The size can be changed to optimize.
  • Compressed Class Pointers – part of the Metaspace. Class data is outside of heap.

If curious about details

To get information about what compiler thinks/did:

  • PrintCompilation – ex: what level instructions were compiled at
  • PrintInlining – use -XX:+UnlockDiagonsticVMOptions

Understanding Hardware Transactional Memory

by Gil Tene

Ralph Winzinger attended this session:

Tene starts by explaining the four states of memory caches:

  • invalid
  • shared
  • exclusive
  • modified

and points out that with regard to hardware transactional memory, there are now two additional states:

  • line was read during speculation
  • line was modified during speculation

A transaction has to be aborted if another CPU wants to write data, if it wants to read data that was modified and if a CPU decides to self-evict the cache.

According to Tene, the big advantage of hardware transactional memory is to get rid of blocks of serialization. The goal is to run fully in parallel and only rollback in case of an actual collision while accessing a data item. …

Tene goes on to introduce the difference between lock contention and data contention. … Usually, lock contention is much bigger than data contention, but only data contention is a problem for CPUs working in parallel. …

With regard to Java synchronized blocks, Tene explains that uncontented blocks would be executed as fast as before. Only in cases where an actual data contention happens there would be the need to rollback the transaction and let the code run again in parallel. …

Gil Tene concludes by pointing out that whereas using HTM is transparent for developers, they now need to start thinking about data contention in their applications. Multiple threads should not modify a single variable because that would lead to data contention and thus to loss of speed up because the advantages of HTM could not be leveraged then.

Microservices and Monoliths

Lessons Learned on Uber's Journey into Microservices

by Emily Reinhold

Twitter feedback on this session included:

@philip_pfo: Microservices don't prevent outages, but they can help contain the scope of impact. (Emily from Uber at #qconnewyork)

@philip_pfo: Data migrations in a nutshell (excellent pattern description from Emily from Uber / #qconnewyork) https://t.co/iSu5lpToDc

@danielbryantuk: The @UberEng request splitter/router that was used to strangulate their monolithic app.#qconnewyork https://t.co/ro2HNlKo1c

@tealagile: Emily Reinhold @UberEng: migrating consumers is the hardest part of monolith to #microservices #qconnewyork

@danielbryantuk: Love your monolith... a refactor of the core may make all the difference#qconnewyork https://t.co/iX7APT2Tt3

Microservices: State of the Union

by Adrian Cockcroft

Dennis Doomen attended this session:

  • It's incredible how many (open-source) libraries, products, and tools have emerged since then [2014]. Kafka seems to be a big one here.
  • Container technology seems to have become a crucial element in a successful migration to microservices. … But apparently the next big thing is Serverless Architecture. …
  • Domain-Driven Design is becoming the de-facto technique for building microservices. …
  • One of the next challenges the community is expected to resolve is the complexity of authorization, security groups, network partition and such.
  • Microservices is not just about technology. It also has a significant effect on the organization. …
  • Version-aware routing and discovery of services through an API gateway is being used by all those that moved to microservices and seem to be a prerequisite. … Examples of gateways that were used included Kong,ApigeeAWS API Gateway and Mulesoft.
  • With respect to communication protocols, the consensus is to use JSON over HTTP for external/public interfaces (which makes it easy to consume by all platforms), and using more bandwidth-optimized protocols like thrift, Protobuf, Avro or SBE internally. XML has been ruled out by all parties. Next to that, Swagger (through Swashbuckle for .NET) or DataWire Quark were mentioned for documenting the interfaces. …
  • A common recurring problem is overloading the network, also called retry storming because each service has its own timeout and retry logic causes an exponential retry duplication. A proposed solution is introducing a cascading timeout budget. Using event publishing rather than RPC surely can prevent this problem altogether.
  • If you have many services, you might eventually run into connection timeout issues. So using shared containers allows reusing connections. Don't use connection pooling either. Next to that, if services have been replicated for scale-out purposes, you should only retry on a different connection to a different instance. …

Twitter feedback on this session included:

@philip_pfo: The key driving metric is now time to value and that's achieved with dev driven infra. @adrianco #qconnewyork https://t.co/QYNA65l9fQ

@kief: Developers' responsibilities have expanded - @adrianco #qconnewyork https://t.co/l5N9aI2JV2

@sangeetan: CD with Containerized Microservices gives companies competitive advantage @adrianco #qconnewyork

@danielbryantuk: Modern software developer responsibilities: faster, cheaper, safer (with CD & microservices)@adrianco #qconnewyork https://t.co/9VNq8n52K9

@philip_pfo: Microservices short definition - loosely coupled SOA with bounded contexts. #qconnewyork https://t.co/dKtEcZwNnp

@kief: The platform is speeding up. Lifespan from years to seconds. @adrianco #qconnewyork https://t.co/6xb74wyC9R

@fe3pluz: Adrian Cockcroft at #qconnewyork mentioning #ING as one of the frontrunners in software dev https://t.co/Y4nLxnwFyc

@philip_pfo: Assumptions of a service==machine limits your speed. Containers and lambdas are disrupting that thinking. @adrianco #qconnewyork

@danielbryantuk: Services like AWS Lambda are causing issues for monitoring. Minute resolution is no good @adrianco #qconnewyork https://t.co/l3LPhGb2Xc

@sangeetan: Find the pioneer team within your org and let them loose to explore & innovate so other teams can benefit @adrianco #qconnewyork

@philip_pfo: Microservices require having a solution for each of the boxes. @adrianco #qconnewyork microservices sotu https://t.co/DYHiz7aOIz

@kief: The fixed costs for microservices is too high for an MVP (my take on what @adrianco is saying here) #qconnewyork https://t.co/2Vwys9kKKW

@danielbryantuk: The best architectures are the ones that evolve continuously. Build upon current work @adrianco #qconnewyork https://t.co/0wfDO1OlCb

@kief: I'm waiting for container orchestrators to narrow down so I can choose - "Sorry, 3 new ones came out this week" @adrianco #qconnewyork

@thenewstack: Over time, orchestration will just become a feature offered by cloud providers. No need to install #Kubernetes #Mesos @adrianco #qconnewyork

@thenewstack: Increasingly, it is developers, not operators, who control how applications get isolated for security @adrianco #qconnewyork

@danielbryantuk: In search of segmentation with cloud and containers: IAM, @projectcalico and @weaveworks @adrianco #qconnewyork https://t.co/MAoLh0Hq50

@danielbryantuk: An important point -> "The @NetflixOSS Simian Army enforces architectural principles" @adrianco #qconnewyork https://t.co/7lo5j8spkR

@philip_pfo: Failing well starts by defining what margin you have to fail without impacting customers; testing verifies that. @adrianco #qconnewyork

@kief: Version aware routing. Interesting idea. @adrianco #qconnewyork https://t.co/4L3iulH9zn

@sangeetan: Deserialization is the costliest - keep in mind when choosing protocol to communicate between services @adrianco #qconnewyork

@thenewstack: For data serialization, #SBE is about 10x more effective than #protobuf, though there are trade-offs @adrianco #qconnewyork

@danielbryantuk: Build a driver client for any service. Tooling like Swagger and Quark can auto-gen this @adrianco #qconnewyork https://t.co/TvdoYOC3kZ

@philip_pfo: Use lightweight deploys to change one thing at a time, easier to reason about what changed. @adrianco #qconnewyork microservices SOTU

@wesreisz: Nice slide showing downstream service inducing failure of a system #qconnewyork @adrianco https://t.co/cqqOBEDa3q

@danielbryantuk: Think about your timeout budgets when serving a request in a complex microservice system @adrianco #qconnewyork https://t.co/KH6gTET2fU

@danielbryantuk: Haha, nice -> "The 'tragic' quadrant of microservice monitoring" @adrianco #qconnewyork https://t.co/my6NYHR80h

@danielbryantuk: The serverless programming model...@adrianco #qconnewyork https://t.co/2Jhud1qXsz

@thenewstack: #serveless is great for financial systems in particlar because there is no server to attack @adrianco #qconnewyork

The Seven (More) Deadly Sins of Microservices

by Daniel Bryant

Twitter feedback on this session included:

@tealagile: Beware of cargo-culting: repeat 3x "we are not Spotify" @danielbryantuk #qconnewyork

@IbraSho: “Use the right tool for the right job” @danielbryantuk #QConNewYork https://t.co/KrnngmDaDp

@stonse: Don't do #Microservices like you are applying lipstick on a monolithic pig @danielbryantuk #QConNewYork https://t.co/5Bzz6eGKnv

@tealagile: @danielbryantuk: technology is no longer a cost center but now a competitive advantage #qconnewyork

@_nighthacking: Daniel Bryant (@danielbryantuk) on Deadly Sins of Microservices #QCon #QConNewYork https://t.co/Lyiscavqw9 https://t.co/SVlpp8ktF8

What They Don’t Tell You about Microservices…

by Daniel Rolnick

Twitter feedback on this session included:

@danielbryantuk: Another shout to the pioneers, settlers and town planners model at #qconnewyork https://t.co/dbN8wILEFJ

@danielbryantuk: Important thinking points for data store connection pooling changes required by microservices #qconnewyork https://t.co/FbOMSxhJZm

@plmaheu: Testing in a continuous delivery process. #qconnewyork https://t.co/AdcqsSOViE

@danielbryantuk: Every microservice exposes a /test endpoint, which verified functionality/deps during canarying #qconnewyork https://t.co/CxqZKHAvhT

@danielbryantuk: Etsy's Hound and Google's Repo sound like interesting tools for managing microservice codebases#qconnewyork https://t.co/MJlcTJjba6

Modern API Architecture - Tools, Methods, Tactics

ESPN Next Generation APIs Powering Web, Mobile, TV

by Manny Pelarinos

Twitter feedback on this session included:

@philip_pfo: ESPN API calls peak rps lowered (without reducing product experience) by converting to push over web sockets. #qconnewyork

WebSockets, Reactive APIs and Microservices

by Todd Montgomery

Twitter feedback on this session included:

@sangeetan: Asynchrony is the norm; but composability is hard. @ReactiveX simplifies things  @toddlmontgomery #qconnewyork

@sangeetan: Ultra thick clients vs under/over specified protocol for inter svc comm -no clear answer;you prob need both. @toddlmontgomery #qconnewyork

Modern CS in the Real World

The Verification of a Distributed System

by Caitie McCaffrey

Twitter feedback on this session included:

@danielbryantuk: We are all building distributed systems... @caitie #qconnewyork https://t.co/b0ytFPMZIS

@danielbryantuk: Amazon found utilising formal verification methods was a big success @caitie #qconnewyork https://t.co/3jFZk7TnQ5

@danielbryantuk: In one study 35% of catastrophic failures were caused by what I call 'dev laziness' @caitie #qconnewyork https://t.co/ay8ZRsprt8

@DataMiller: Three nodes or less can reproduce 98% of failures. @caitie on verification #qconnewyork https://t.co/T2oJXvjArv

@danielbryantuk: Simple unit and integration testing can catch a lot. Don't forget to test boundary and edge cases #qconnewyork https://t.co/WDv2Bhu8sy

@danielbryantuk: What we're doing now is incredibly hard - there is a lot of value in testing for correctness @caitie #qconnewyork https://t.co/XMl6MIFOWO

@pcalcado: But also... #qconnewyork (by @caitie https://t.co/S8S46D0cVc

@danielbryantuk: How to run a disaster game day...#qconnewyork @caitie https://t.co/JjuhdSwsZD

@danielbryantuk: Monitoring is not testing... monitoring is super valuable, but it's reactive @caitie #qconnewyork https://t.co/kzzsCN3OZK

@TonyPrintezis: Canaries are not the only means of testing you should be using. @caitie #QConNewYork

Optimizing Yourself

Syntactic Sugar for English: Pragmatic Eloquence

by Wendy Closson

Twitter feedback on this session included:

@DataMiller: You don't win friends with sarcasm. @wendypclosson helping us hack our communication #qconnewyork https://t.co/HEthy8VyAH

@jchyip: A bit of a right speech, right thinking, importance of ritual vibe to @wendypclosson's talk #qconnewyork

Unlocking the Secret Sauceof Great Teams

by Heather Fleming

Twitter feedback on this session included:

@philip_pfo: The secret sauce of great teams from @hfleming #qconnewyork https://t.co/YtzQCgSDnj

@aspyker: THIS!  is the reason why I love that we have one title for all engineers at Netflix @hfleming #qconnewyork https://t.co/w1mK7oSsMY

@TheHungryGeek: Awesome talk from @hfleming on how being yourself at work builds a great team #qconnewyork

Practical DevOps for Cloud Architectures

Access and Secret Management in Cloud Services

by Ryan Lane

Twitter feedback on this session included:

@aspyker: Lyft created the same metadata proxy as Netflix Titus.  Although we did it beyond docker0.  #qconnewyork https://t.co/Zp22SGzPoY

@kief: I hadn't been aware of confidant secret management from Lyft https://t.co/Uiucdev5Md #qconnewyork

DevOps'n the Operating System

by John Willis

Twitter feedback on this session included:

@kief: .@botchagalupe  apologized to me as he declared #InfraAsCode is dead on stage at #qconnewyork. He has a valid point.

Implementing Infrastructure as Code

by Kief Morris

Ralph Winzinger attended this session:

… Morris goes on with mentioning that creating servers with a single mouse-click is not where it ends. Usually this leads to a large fleet of servers and often some configuration drift. Since inconsistent servers are not easy to maintain automated, maintenance for those machine is likely to be done manually which leads to even more inconsistencies.

This is where infrastructure as code is introduced as a possible solution to the problem and a way to create well defined servers: using tools like Puppet or Chef in an mode of "unattended automation". These tools run on schedule with no way for manual changes. Even small things have to be fixed in the underlying templates and configuration, eventually producing a landscape of immutable or containerized servers with no more manual changes at all. This concept should also be leveraged when promoting servers between environments. Re-use as much of the templates and configuration between stages as possible….

But it's not only the benefits that Morris mentions, it's also some drawbacks or pitfalls that are part of his presentation. Just like in any other system that is maintained or developed by multiple teams you have to keep in mind that there will be integration points, bottlenecks or dependencies to keep track of. For example, you will need to provide test instances for depending services and leverage consumer or contract driven testing to ensure that all services cooperate in the desired way. If parts of the templates or configuration are used like shared libraries, they also need to be tested thoroughly before distributing them to other teams.

Security War Stories

Banking from the Future: Cryptocurrency Key Storage

by Olaf Carlson-Wee

Twitter feedback on this session included:

@charleshumble: Bitcoin is the perfect thing you can steal if you are a cybercriminal since when you gain access to it it is instantly gone.#qconnewyork

@charleshumble: “There are nearly 150 breeds of bitcoin-stealing malware in the wild.”Olaf Carlson-Wee #qconnewyork

@charleshumble: You can compromise any sort of computer and extract value via extortion - for example CryptoLockerOlaf Carlson-Wee #qconnewyork

@charleshumble: Securing private keys on an internet-connected device is a bad idea.Olaf Carlson-Wee #qconnewyork

@charleshumble: There’s a guy who goes by the name btcrobinhood,  He will steal your bitcoin for you and then return it.Olaf Carlson-Wee #qconnewyork

@charleshumble: Coinbase stores 10% of all bitcoin in a hosted wallet.Olaf Carlson-Wee #qconnewyork

@charleshumble: .@conibase uses shamir secret sharing for its bitcoin hosted wallet.  Depends on consensus. Olaf Carlson-Wee #qconnewyork

Modern iOS Application Security

by Dan Guido

Ralph Winzinger attended this session:

Guido starts by explaining the security mechanisms in place for iOS applications. …

  • Transport Layer Security: iOS provides support for securing network connections.
  • Data Protection: iOS offers strong encryption for nearly all files used by applications.
  • Code Signing: Since Apple requires every piece of code to be signed, memory contents in a granularity of 4kB pages can be traced down to an individual developer.
  • Runtime Process Security: iOS isolates processes via strong sandboxing. Processes are not able to access the memory of other processes.
  • Secure Enclave: Newer iOS devices with fingerprint sensor feature hardware based encryption keys that are uniquely generated for every device at manufacturing time and reside outside the operating system.

… there is one more thing that developers need to take care of: Jailbreaks. As soon as a device is jailbroken, all of the above security mechanisms might get rendered useless. … If a developer needs to provide high security she or he has to ensure that no jailbreak is active:

  • Jailbreak detection - developers can check for certain traces that are left behind when jailbreaks are installed. This included specific files and processes.
  • Anti-debug protection - developers need to make sure that their application won't run in debug mode because any jailbreak detection would be visible there.
  • Anti-reversing - developers need to make sure that their code can't be re-engineered in a useful way. Usually this is done by artificially making the code larger and more complex.

The Bad Things Happen When You're Not Looking

by Ryan Huber

Jeanne Boyarsky attended this session:

General Notes

  • Time to detect is an important metric
  • Credential theft is biggest/one of the biggest
  • Bad model – NetCool – train people to acknowledge all alerts and they miss things because bad habit …
  • The hypothetical malicious insider – a former security team member has a lot of knowledge. And an insider with credentials has access …
  • Don’t overwhelm users. Confirm bulk actions in bulk not one at a time.

Slack Security

  • Setup reliable logging platform
  • RELP (reliable event logging protocol)
  • steamstash/logstash -> Elasticsearch (Splunk is superior but costs more)
  • Two weeks of data is about 2 terabytes of logged data. Almost never sits on disk
  • auditd – part of Linux. Run auditctl commands and kernel looks for matching events.
  • audisp – works with auditd to transform data
  • osquery – Facebook project for system monitoring using SQL
  • ElastAlert – yelp project to pick up on ElasticSearch events. Does queries on a timer against Elastic Search.
  • AlertCenter – have SecurityBot looking at alerts. Security bot posts to Slack asking user to type “acknowledge” on phone to confirm action.

Twitter feedback on this session included:

@charleshumble: First Slack security employee.  My goal: Don't be NetCool.  It’s awful @ryanhuber #qconnewyork

@charleshumble: “Set up a reliable logging system.  Slack uses RELP, which stands for reliable….ELP.” @ryanhuber #qconnewyork

@charleshumble: Slack security logging chain: RELP—->StreamStash—> Elasticsearch@ryanhuber #qconnewyork

@charleshumble: If someone is attacking you and they don't know where the trip wires are you have an advantage.@ryanhuber #qconnewyork

@jeanneboyarsky: slack sends msg telling user type "acknowledge" on phone to confirm alert. that way not relying on potentially compromised acct #qconnewyork

@charleshumble: Don't let perfection be the enemy of good. Log everything and look at your logs. @ryanhuber #qconnewyork

@charleshumble: Git Gist for @ryanhuber #qconnewyork talk is here https://t.co/J29M4kIL0y

The Nihilist’s Guide to Wrecking Humans & Systems

by Christina Camilleri

Twitter feedback on this session included:

@danielbryantuk: Scarily awesome story of using social and technical hacking to compromise production servers@0xkitty #qconnewyork https://t.co/UBR9MMf3My

@danielbryantuk: We assume trust. We protect outer barriers while not paying enough attention to the inside @0xkitty #qconnewyork https://t.co/aMcjDtG0Mh

@danielbryantuk: Great talk on a timeless issue of hacking by @0xkitty at #qconnewyork. The 80s video showed how much hasn't changed https://t.co/uJqMrI03NO

Stream Processing @ Scale

Ingest & Stream Processing - What Will You Choose?

by Pat Patterson & Ted Malaska

Twitter feedback on this session included:

@danielbryantuk: It is possible that in two years time people will be writing stream processing pipelines without code #qconnewyork https://t.co/6x2iYr4y1p

@danielbryantuk: You may want to check out stream processing abstractions like Apache Beam, but watch for SQL support #qconnewyork https://t.co/2nOoewWe4t

@danielbryantuk: Minecraft can be used as a $27 data visualisation tool... @metadaddy #qconnewyorkVery cool! https://t.co/8FmRr6rLDh

Large-scale Stream Processing with Apache Kafka

by Neha Narkhede

Ralph Winzinger attended this session:

Narkhede starts by introducing the basic programing paradigms for working with data:

  • Request / response cycles
  • Batch processing
  • Stream processing

and continues by giving a practical example for stream processing from the retail domain: sales and shipments are basically unbound datasets and working on such datasets is effectively stream processing. Sales and shipments are a stream of events ("what happened") and a function for recalculating prices ("do something"); based on those events is the stream processor….

Narkhede continues by giving an overview on how important capabilities of stream processing systems are realized:

  • Scalability is automatically provided since the event log is partitioned. …
  • Fault tolerance is also provided out of the box. There is no master in a cluster of Kafka Stream nodes, just peers. …
  • Stateful processing is also supported as needed by joins or windowed calculations. In such cases the necessary data is pushed to processor to avoid remote access.
  • Reprocessing of data with changed business logic is supported by letting new consumers start event processing with an offset of zero (from the beginning).

Narkhede then continuous to introduce the duality of Kafka Streams as the basic principle for implementing the given features: basically, the concept of tables ("state of the world") and streams ("how did the state evolve") are combined. Therefore, Kafka Streams based applications can be reactive and stateful at the same time.

Twitter feedback on this session included:

@philip_pfo: Exactly once is the holy grail all stream processing engines are pursuing. It's semantically at-least-once + idempotence. #qconnewyork

@philip_pfo: Event centric thinking enables you to build a forward compatible system. @nehanarkhede #qconnewyork

@philip_pfo: Tables and Streams are alt views of the same data, with the changelog as the binding. Novel insight from @nehanarkhede #qconnewyork

Opinions about QCon

Impressions expressed on Twitter included:

@ddoomen: I love the name tags that emphasize first names here at #qconnewyork. It really helps to connect with people.

@frankgreco: The award for Best Food for a Technical Conference goes to... the envelope please... the winner is #qconnewyork by a wide margin!

@stevelun: First time in New York and Qcon.  Each were amazing, but put them together and it's a trip I won't forget.  Thx #qconnewyork

@caitie: Just some nerds talking about Computer Science at #QconNewYork https://t.co/oUQoNzzJVN

@fletchertinam: Congrats @qconnewyork on a very successful conference this week. Many new things learned and new friends made! #qconnewyork

Takeaways

Takeaways from QCon New York included:

@philip_pfo: Share what you know - that's how progress is made. @charleshumble #qconnewyork

@ddoomen: You want to ensure a prototype isn't kept as the real product? Use a language/platform that the organization can't support #qconnewyork

@ddoomen: You want to ensure a prototype isn't kept as the real product? Use a language/platform that the organization can't support #qconnewyork

@AgileTester: If you're not going to build a better version of something that exists, then don't build it - Google's buy borrow or build #QConNewYork

@AgileTester: It's time to start thinking of your team as a "“Symphony, not a factory” - @randyshoup #QConNewYork.

Conclusion

QCon's focus on practitioner-driven content is reflected in the fact that the program committee that selects the talks and speakers is itself comprised of technical practitioners from the software development community. QCon New York was produced by InfoQ.com. We will next be in Shanghai, Oct 20-22, 2016 and San Francisco, November 7-11 this year. QCon New York will continue to run around June of every year. However, this year was our final year in Brooklyn. Next year we will be in Manhattan.

Rate this Article

Adoption Stage
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread
Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Discuss

Login to InfoQ to interact with what matters most to you.


Recover your password...

Follow

Follow your favorite topics and editors

Quick overview of most important highlights in the industry and on the site.

Like

More signal, less noise

Build your own feed by choosing topics you want to read about and editors you want to hear from.

Notifications

Stay up-to-date

Set up your notifications and dont miss out on content that matters to you

BT