Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage Interviews Elisabeth Hendrickson on Building Quality In

Elisabeth Hendrickson on Building Quality In


1. Good day, this is Shane Hastie for InfoQ and we are here at Agile 2014. I’m talking to Elisabeth Hendrickson. Elisabeth, you and I know each other well, but there are probably few people out there that don’t know yet you, would you mind briefly introducing yourself for the audience?

Sure, I’m Elisabeth Hendrickson, I’m also known as @testobsessed on Twitter and these days I’m Director of Quality Engineering for Cloud Foundry at Pivotal.

Shane: So tell us a little bit about what you are doing here at the conference this year.

Sure, I’m very, very pleased that they invited me to the Agile Boot Camp Stage to give the basic introduction to Agile Testing, it’s the Agile Quality and Risk Management session.

Shane: But don’t we not bother with testing in Agile, isn’t that one of the…..

You are going to make me roll my eyes at you.


2. What is special about Quality and Risk Management in Agile?

Well one of the things that a lot of people don’t get when they are first introduced to Agile is that Testing and Quality go hand in hand with Agile, so if you are doing Agile right you are actually getting better quality and lower risk, however if you adapt Agile without any of the technical practices you can get situations where it’s more fragile than Agile and you end up with worse quality and slower feedback cycles and slower delivery times, and so it’s important to understand the technical practices that enable you to get all the real benefits of Agile.


3. What are some of those important technical practices?

Test Driven Development is a really, really big one because that’s part of being able to, first of all get as a side effect of doing Test Driven Development getting the very, very fast feedback cycles, as a developer I make a little itty-bitty change and I’ve got a fully automated set of Unit Tests that I can run that tell me if I broke anything. Second of all being able to stay up front what my expectations are for the code that I’m about to write, that level of discipline practice ensures that at the very base level the code behaves as we intended, the system may still not behave as we intended the system too, but at least as the code level we’ve got that. So that’s the lowest level of feedback cycles and then putting that into your continuous integration system so that with every single code check-in that full suit is going to be running, now we know that nothing broke in all of the code that goes into the overall system. Fully automating our system regression tests, so that again with every code push we could be running that longer cycle of full system tests, full integration tests and ultimately then being able to do multiple deployments to a production or a production like environment every single day, so that we are getting some road time on that code every single day that level of disciplined practice and putting in place all of the technical underpinnings to make that work is essential if you want to get the benefits of Agile.

Shane: That’s actually quite a complex technical environment to build.

Ok, it is!


4. Tell us a little bit about some of it? So what are some of the things that need to be in place to make that work?

Well some of the things that need to be in place, let’s start with just environments, so back when I was consulting because I only within in the couple of years took this role at Pivotal. So back when I was consulting and my job consisted of get on airplanes and go visit sites, I saw a lot of sites where there is somewhat a penny wise pounds foolish approach to environments, because environments can be so expensive, depending on the nature of the thing that you are shipping, they wanted to constrain the number of environments and so you need to have a lot of infrastructure available to you in order to do this form of testing because you are going to need an awful lot of environments to be able to have all of the things in place to deploy frequently to them, so you need to have environments.

Those environments needs to be automated, one of the things I’m super excited about these days is the confluence of the Agile Community and the DevOps Community and yesterday in the opening Keynotes, Sam Guckenheimer said: “DevOps is basically second decade Agile” and I totally agree with him on that because the ideal of being able to recreate the enterprise from a source Repository and Data Backup and raw hardware or virtual environment, being able to stand it all back up again, that’s essential to being able to replicate those tests environments over and over again, so having in place that culture of configuration is code, we version it, we can deploy everything over again and restore our data from backup is another essential aspect of it. Having in place the technical practices within your teams, so that your teams understand how to write good tests and having in place the frameworks that are appropriate to your environment, these days those are pretty much all OpenSource, but having them and using them are essential.


5. Thinking of the traditional Tester Community, this is not a skill set that a lot of them have?

Increasingly I think they do, more and more I’m talking with testers who just kind of assume that they are going to be working in a very, very technical environment and I would also say that with Agile the role of the testers had been changing anyway, so I don’t know that it’s necessarily true anymore to say that don’t have these skills. Of course I don’t know that it is true, it isn’t true? Anyway, yes.


6. So how do we build that quality in?

It all starts with a story, the user story. So in Agile we work in user stories and building quality in starts with being super explicit about what we expect the effect of that story to be. That means both acceptance criteria, it also means to me that since quality is not just about meeting our internal requirements or our expectations but also about fitness for use , did we achieve the business objective that we intended, that not only do we need acceptance criteria on our stories that reflect what we expect to be true in the product, we also need to have a notion of what we expect to be true with our users or our market after we’ve implemented this thing. So quality begins with a story and a conversation around the story then we get into the technical practices and making sure that we’ve got those fast feedbacks cycles for not only that we built what we intended to build but that we did not break anything else in the process.

Shane: As the Director of Quality Engineering for Cloud Foundry, you are responsible for making this happen and one of the things you’ve said to me is that you don’t have a testing team.

That’s right, there is no Quality Engineering Team for me to direct, so that’s not what I’m directing.


7. What are you doing?

So what do I do with my days? I like to say that I’m responsible for the care and feeding of our feedback cycles, but one thing I have to say is that this is one of the very few rarefied environments where I would be able to do what I do and have the results that we have because the company already had this in their DNA. So Pivotal is a spin-out from EMC and VMware, it borrows its name from Pivotal Labs which was a smaller consultancy and was part of the spin-out. Pivotal Labs has had an extreme programming practice for a long time, in fact I like to say I went home to the Mother-ship because Pivotal Labs is where I learned Agile way back in a day. I did my first project with Pivotal Labs in 2004-2005, somewhere in there, and there really where I had learned about TDD, I had learned about all of the various practices, Pivotal Labs is where actually got time practicing that on a real project delivering software, so I went home to the Mother-ship.

Anyway because this was already in the DNA of the organization I don’t have to start from the very beginning and say: “Hey guys, this TDD stuff is cool!”, instead it’s much more likely that I walk into a retrospective for the team and the developers will already be talking about: “You know our tests sweets is getting a little bit flakey and it takes a little bit long to run and we think these things are connected because the longer it takes to run the more likely we have timing issues, that’s one of our retro items that we are going to take that as an action next week and fix that”, and I don’t have to say anything, I just get to sit there and smile. So in terms of being responsible for care and feeding at the feedback cycles, I’m just looking across our 17 sub-teams on the Cloud Foundry project and making sure that we are not overly locally optimizing, that we are finding the right opportunity to globally optimize when we see multiple teams shaving the same yaks, there are all in the same yak herd then we are going to start centralize some of those practices so that we can get benefits of scale, but I don’t have to start from the very, very beginning.


8. For somebody who does, what advice would you give them?

You are in for a really long journey. Cloud Foundry had originally came from a team that was not following the Pivotal process because didn’t come out of Pivotal Labs, it came from, it was actually a VMware project and I’m not even sure where it came from before that. And so we had this interesting time period when we were shifting the process and I learned a really important lesson there, and that is that wasn’t till we had about 50:50, a one to one ratio of Pivotal Labs people who already knew the Pivotal Labs process, who already knew extreme programming, who are paring every day, who test drove probably just about everything in their lives they thought in terms of how I’m going to express this expectation. Those very, very experienced in XP people with people who are already very, very experienced in Cloud Foundry, they are all very, very competent, amazing engineers but until we had that one to one ratio we didn’t see the process really shift and so one of the realizations I had is when I was consulting and getting on airplanes and I be the solo consultant in an organization I wasn’t going to be able to affect change, not all by myself, it really does take an entire community to achieve critical mass, so if you are starting from scratch, my advice would be to figure out how are you going to build that critical mass because you are not going to be able to do it on your own.

Shane: One of the things that you said when we were chatting earlier as well was that the team owns quality. That’s also a mindset shift.

And it’s been interesting, because the team was responsible for their deliverables, partly that means that the Product Manager is responsible for making sure that what they are asking the team to build is in alignment with all of the other 16 sub-teams and what they are building, and so the Quality of the system as a whole has to came together as a result of the alignment between all of our Product Managers, and the team itself is responsible for the quality of their product; making sure that it’s under load, it’s not going to fall over, so our load testing is ultimately the responsibility of the team that’s responsible for the deliverable. All the performance benchmarking is the responsibility of the team. Security is again the responsibility of the team.


9. How do they do that, how do you get that thinking and instilled in the team?

Again I’m not starting from scratch with this, so the idea that the team owns quality was already part of the Pivotal Labs DNA, the Pivotal as a company is embracing, so I didn’t have to start from scratch but some of the stuff that we are doing, Cloud Foundry is a massive distributed, huge project and so figuring how to do this at that scale has been really interesting and raising awareness around security issues and raising awareness around performance issues has been interesting. Ultimately one of the things that I’ve learned in the last two years, year and a half, is that if it doesn’t get into the team backlog it’s not going to happen. So we’ll have conversations around stuff and until I actually can see the story reflected in the team’s backlog I know that I need to be tracking this as something that needs to happen. When for example, if we discover that we don’t actually know what the performance is going to be or we don’t know where the bottlenecks are in the system, we’ll have a conversation about that and then ultimately I need to see a thing in the team backlog and that’s what is going to trigger the work to actually happen.

So the work item isn’t going to happen unless somebody is actually taking the time to define it as a testable work item.

Yes, the work item is not going to happen unless it’s defined in the backlog.


10. Excellent, you did talk or you mentioned the tools team and you’ve touched a lot on the need for automation there, so what are some of the tools that you pulled together to do this?

I want to clarify because we don’t have a QA department and we also only have a little itty-bitty teeny-tiny tools team and really that little itty-bitty teeny-tiny tools team is doing some shared infrastructure, they are standing up a large distributed continuous delivery system, ThoughtWorks recently open sourced Go CD. So Go CD is kind of like Jenkins it’s kind of like any of the CI tools you might be familiar with, however the thing where it excels or the place where it excels is that if you’ve got a massive build pipeline and as you could imagine with a massive distributed system we have a whole pipeline, it’s not enough to say here is the build, there is like to get from a line of code changes in this component over here, do we have a shippable artifact takes like 7 to 10 stages and so that’s where build pipeline comes into place and ability to hand artifacts between parts of the pipeline before delivery.

So the tools team is standing up, Go CD and we are doing it in a very DevOps kind of way, we are using our deployment tool Bosh, Bosh it’s an open source deployment tool kind of like Chef or Puppet only it does not just deployment but also full life cycle management orchestration of the cluster. So we said you know we’ve got a cluster and just because it’s a cluster of CI kind of things doesn’t make it any less of a cluster, so we should be using our deployment tool to deploy that. So the tools team is responsible for using Bosh to keep our production continuous delivery system up and running at a production level of quality and making sure that all the dependencies that are shared between teams are installed in that so the teams no longer have to think about managing individual dependencies. So the way I like to phrase it is the tools team manages the plumbing system and the individual teams manage what flows through that pluming system.


11. So at the individual team level in there, they are choosing their own tools or common tools?

So one of the interesting things with Agile at Scale is, what is that balance between team autonomy and then needing to make everything work together in the entire system, and so organizational dictates. Our ideal is that the teams are completely autonomous, however if the teams were completely autonomous then we might end up with tooling that did not play nicely together so we’d had to figure out where we are going to mandate something versus where we just going to let teams run off and do whatever thing they want to do. So teams however, we err on the side of team autonomy which means that we have teams that they have to write their own unit testing tool. We’ve recently starting adopting the Go programming language, a quick aside about Go CD and the Go programming language, there was an interesting naming thing that happened where before either one was well known and each choosing their names, so it’s now way too late to reconcile that. The Go programming language, that’s a programming language, the Go CD was written in JRuby. So it’s actually valid to say Go was written in Ruby which makes people crack up; total digression.

Anyway so the Go programming language is relatively new, the ecosystem is relatively new and of course with any programming language the hard part isn’t the syntax of language itself but the ecosystem within which it lives, and what that meant was that the testing infrastructure wasn’t quite at the level that we’ve come to appreciate from say Ruby, for example. So we ended up writing our own behavior driven development, BDD tool because at Pivotal we like BDD more than traditional TDD, we like thinking about it in terms of behavior, and its behavior is all the way down and so one of our engineers, Ansi, wrote Ginko a BDD tool. So that kind of thing wouldn’t have been able to happen if we were an environment that mandated all the tools. So individual teams have the freedom to create the tools they need, to use the tools they need and in order to, because they own quality they also have to own the tools that enable them to get to that place.


12. This interesting dichotomy of team autonomy and playing nice in the big sandpit, scaling Agile, how do you manage that?

Starting with being conscious of the fact that there are 3 levels of things that we have to think about, so let me back up and explain. There are 3 Directors of Engineering, so I’m the Director of Quality Engineering, we have a Director of Engineering and then we have a Director or Engineering Services and the three of us work together as a team overseeing all of Cloud Foundry and some of the conversations that we have, because we make sure that we are in alignment all the time, some of the conversations we had are around RA - what’s a matter of policy so it’s something that we are saying teams will abide by this, what can we leave to individual team autonomy and then what’s up to individual discretion, and I’ll give you some examples of places where we’ve draw the lines: as a matter of practice Pivotal Labs historically started the day at 9, end the day at 6, we work at a sustainable base, there is no notion of “I feel like working from home today”.

There were good reasons for this, Pivotal Labs was doing client projects and the clients kind of expect you to show up, so we had an interesting decision to make, do we continue with that practice even when there is no client, Cloud Foundry is a product, there isn’t an external client coming in. We decided yes that it’s a very effective way to make sure that teams are running on all cylinders, that everybody knows that I can count on my team to be there all at the same time, that make it easier to pair up in the mornings, pairing is another one. Do we decide that we are going to keep pairing or do we decide that would be up to individual discretion or up to team agreements, no we decided that we would keep pairing so all the XP practices, core work hours 9 to 6, sustainable base, Test Driven Development, Continuous Integration, those are not negotiable, those are matter of policy and if a team came to us and said: “We feel like doing an experiment, we are not going to test drive anything” and we look at them very funny and say: “I think you are going to get to decide to do that”, but on the other hand anything that’s outside of that, that has to do with the specific tools they use or the way they decide to pair up in the mornings, that’s all of the team discretion level. Then at the individual discretion level, well pretty much because we pair all the time, you have to convince your pair whatever it is, is a good idea.


13. What are some of the other complications when you are doing Agile with 2000 people?

Well so anytime you are talking about something at that scale I don’t actually have a lot of visibility into everything that’s going on outside of Cloud Foundry in our company, so we are not doing everything exactly the same across all other parts of the company, so I would say that at the level above us there is, I talked about individual discretion, team working agreements and policy, that policy is really only applicable to Cloud Foundry and then at a higher level there is what’s mandated by the overall organization and just like on Cloud Foundry, we like to drive as much as possible down to the team level. Pivotal is a whole operates that way as well, other product lines make slightly different decisions and I don’t know exactly how they are working.

Shane: Ok, so it’s that devolution of power within a set of constrains.

Exactly, creating self organization within an overall structure.

Shane: Great, Elisabeth thank you for taking the time to talk to InfoQ today, it’s been really good to catch up and look forward seeing where this goes

You are very welcome, thank you so much for chatting, it’s always good to see you.

Oct 09, 2014