Agile Australia is the national Australian conference on Agile, attracting over 850 delegates in 2013. In its fifth year, the conference was themed Accelerate Innovation and featured thought leaders on innovation and business, including Dave Snowden, Bjarte Bogsnes and Ryan Martens. Agile Australia 2013 was sponsored by Rally, ThoughtWorks, IBM, Telstra, and Atlassian and is managed by SlatteryIT.
1. Hi my name is Craig Smith, I’m an Agile editor at InfoQ and it’s my privilege to have here with me from Atlassian, Andrew Prentice and Jo Cranford. So as I said, both work for Atlassian which is an Australian success story I guess in many ways, one of the very early software startups that kind of made it big, and you are both there from there, so Andrew you are test manager?
Andrew: Well I manage the QA and developer teams.
Andrew: Yes, that is right.
Jo: I’m a developer on the GreenHopper team.
3. Excellent. So for those who don’t know Atlassian, if there is anyone who is living under a rock somewhere who hasn’t used JIRA or Confluence or any of the other related tools, just quickly, could you give people an overview of what Atlassian does?
Andrew: Sure, we build software development and collaboration tools, so a lot about tooling is around for developer specific, so that is issue tracking, code, repositories, code views, that sort of things, but we also work on tools that are for less technical audience, so project management, workflow management, that sort of thing, and also make a wiki, for collaboration as well.
Jo: Yes, I’ve been in Sydney about 2 years now, I moved over here with ThoughtWorks, and then I moved to Atlassian just over a year ago, so I’ve been working on GreenHopper for most of that time, I also did a bit of work on Bonfire when I first started which was pretty fun, it’s been a pretty cool ride so far.
Andrew: I sort of got attracted to the big end of town when I left University, working for the large IT consulting companies which was great because I got to travel the world and experience different environments, but the thing that I got frustrated with, when we worked on a project, pretty much every time before we actually deployed it to production, I’d get reallocated to another project, so I never actually got to see the results of my work or whether what I was actually doing, decisions I was making were the right ones, so I really wanted to work for a product company and I was lucky enough to find Atlassian.
7. [...] How does someone fall into testing? What is your story? Because software development is something that you go to university and learn and it’s one of those things that you kind of go through, but what about testing, how you get there?
Craig's full question: Excellent. So one of the things I guess I wanted to ask you about testing, and we’ve known each other for a while, we share the same kinds of ideals in testing and you have spoken in the Agile space quite a bit, how does someone fall into testing? What is your story? Because software development is something that you go to university and learn and it’s one of those things that you kind of go through, but what about testing, how you get there?
Andrew: It’s definitely a profession that people fall into and that has pros and cons because some people discover it and actually love it and they are the kind of people that we try to hire at Atlassian. Unfortunately there is a lot of people who fall into the profession for the lack of anything better to do and as a result they kind of bring the whole profession down, so they are a bit of a hindrance as well, personally I started off in support and we were a level 2 helpdesk and through that got domain knowledge about some systems and then as a result of that I got called in to do some acceptance testing and that is when I discovered this realm and discovered that I really loved it and it took off from there.
Andrew: Once the opportunity presents itself to actually take ok or bad software and make it fantastic, it’s a pretty seductive proposition, for those people who are passionate about it.
Jo: I tried it out at university it’s funny, you should say that you can go to University and study because I fell into it as well, I was working in a part time job, ended up doing their website, and thought: “This is actually kind of fun, I think I might do a little bit more of this” and I ended up doing a little bit at university and then found a job luckily enough where I could learn on the job, so it kind of went from there.
Jo: I don’t know, I like puzzles and I like logic problems and solving those kinds of things and I think that is probably common to a lot of developers. I don’t know, it’s more fun than most girls think it is. I think when I was at school it never occurred to me and I do think there is kind of a gap in education where girls just really aren’t introduced to that kind of stuff, just you don’t even think about going to do Computer Science and for some reason guys do. I wish there were more girls in Software Development because it would definitely make my team more fun, but that’s just the way it is.
11. [...] Andrew you are here talking about quality assistance which is interesting, it’s a different take on what most people think the term QA is and how you help Agile teams develop faster at scale, so tell us a little bit about that because it’s sort of an interesting journey you take people on in that talk.
Craig's full question: So we are here at the Agile Australia Conference here in Sydney and both of you have talks here at the conference but very different ones, so Andrew you are here talking about quality assistance which is interesting, it’s a different take on what most people think the term QA is and how you help Agile teams develop faster at scale, so tell us a little bit about that because it’s sort of an interesting journey you take people on in that talk.
Andrew: When we think about quality, there’s are a few factors that can either enhance or impede producing quality software, and the one that affects Atlassian the most is growth, scaling, we are a fast growing company, and what that means is our customers numbers grow as our organization grows as our codebases grow, it can become more challenging to release better quality software. And so the QA team, its function is to mitigate those challenges and essentially the mission is to make it safer for development teams to move faster. And that is quite different to the traditional approach to QA where test teams would go “Well we’re are not really caring about the speed of development here, we’re just caring about making sure that we don’t ship bad code to customers” and often times that means that it will take a long time or slow things down on purpose as a result and that is a very different to our way of thinking. Hence why we call our QA team Quality Assistance rather than Quality Assurance, the role is really to assist the development teams, ship better quality faster.
12. So when we talk about Agile particularly and you’ve been on the journey on both sides of the equation, both traditional and normal, what is the difference, I mean our world has changed a lot and what does Agile testing actually mean?
Andrew: Well it means, recognizing that quality is the responsibility of everyone on an Agile team not just the testers, and if you accept that then that also means that testing is not just the responsibility of your testers or QA people, it’s the responsibility of everyone. And so that means encouraging and sometimes enforcing developers to learn how to be better testers themselves, because if they have those skills and that knowledge, then the likelihood of the code they are producing being of sufficient quality is much higher.
Jo: We use QUnit, it’s pretty standard in Atlassian, I used to use Jasmine before I came to Atlassian, and I think they’re both really good, I don’t know if you’ve used some other ones haven’t you?
Andrew: Yeah I have, like I think QUnit is good, I think personally my interest is more in the test runners and those headless browsers that are now available that really speed up test execution. On a slight tangent, one of the problems that we have at Atlassian is the cost of maintaining our automated tests. I think a lot of people feel like they need more automation, or need to start automating. A product like JIRA has around 30,000 automated tests and what we’ve found is that a lot of those tests were written quite some time ago using different technology stacks and the sort of work that Jo’s doing, means that we can actually write faster tests more efficiently with some of these new technologies.
Jo: I think with the slower tests what we’re trying to do with GreenHopper is just test the happy path, so we have a minimum number of tests that really test the whole application end to end, but we try and maximize the coverage in kind of integration and unit tests, so those real small fast ones, we have lots of, but the maintenance headache is another problem that we are trying to find ways to reduce the maintenance costs really, I guess.
Jo: With the tests that I have been writing recently we are trying to figure out the different user paths through the code, so I’ve been focusing more on testing the UI which I used to think was a really bad idea and I’m kind of coming around to the idea that you can, if you watch out for some of the pitfalls and you don’t try and replicate too much of the UI in the tests, then you can get quite a lot of value out of it, but it’s really thinking about what does the user do and what do they see? So if I click an issue and then I press a keyboard shortcut, what do I expect to see happen in the UI? So that I guess is where I’m focusing on.
Andrew: Well it can be but I think Agile offers a solution there, if you have people who are passionate and good at testing and QA embedded in the team with good collaboration and good oversight as to what everyone on that team is working on then they’re in an excellent position to be able to make those decisions about where to apply those limited resources. It’s in those situations where you have a separation between test teams and development teams and everything is happening behind closed doors, that is when that lack of visibility means you’ve got greater risk and tends to encourage people to mitigate those risks by trying to test all the things rather just the things that they need to test.
Craig's full question: Your role as the manager of a QA team, one of the things that if you read any of the Agile text books they say, the first thing you do is you dismantle the QA team. So obviously you still have that ideal of some sort of QA team at Atlassian, how does that work, how do you make that work in an Agile environment?
Andrew: If your QA team is made up of those people who fell into testing for all the wrong reasons and then by all means I encourage you to disband the team straight away, that is really not going to help you at all. What you really do want is find those people who are passionate about it, and one of the keys there is finding people who get frustrated about things being inefficient or slow, I mean you want people who are passionate about delivering great software to end users and getting satisfaction about those users getting value from that.
If you find those people, then they’re a great boon to have on your Agile development teams, because they can get rid of roadblocks, so the QA team at Atlassian spends quite a bit of time developing our own tools and providing solutions to some of those age old problems that bog down development teams. We have innovation hackathons every quarter and the QA team won not the most recent one but the one before with a tool that solves the “Why don’t developers test in IE problem”. We all like our Macs at Atlassian or Linux, very few people with Windows, how do we solve that problem. Traditionally the problem is developers will have VM’s with IE installed on them somewhere, but managing those is a problem.
So the QA team set this goal: “We want any person in Atlassian to be able to have access to their own virtual machine, with any version of IE connected through to their development or testing environment within 10 seconds” - that was the goal. So they built the tool that leverages EC3 to do exactly that, so from the command line now developers can just say: “Dev environment IE8”10 seconds later they’ve got a browser connecting back to their dev environment to do that testing.
Jo: And can I just get back to the original question as well, I just wanted to add something. So, on the GreenHopper team we have one QA, so although there is a QA team at Atlassian, they’re really the QA’s are distributed into the development teams. So I think we don’t have a QA team that we chuck stuff over the wall to, and I think that when you were talking about dismantling it, we do that and I think it’s works really well because our QA will work with the other QA’s to kind of bring these techniques into the GreenHopper team, so he is not just working on his own or reporting to the GreenHopper team manager for example, he’s part of this QA team and I think it brings the best of both worlds doing it that way.
19. So you have that idea of a community of testers that can do things exactly like the innovation you just talked about Andrew but also have a product that they are living and breathing and seeing to solve the other problem you are talking about which was actually seeing something come to life?
21. [...] How did that tool come about? It was, I guess that was kind of a frustration of testing really and a bit like these innovations things trying to put something together that helps testers, right?
Craig's full question:So you mentioned Bonfire before and I know Andrew, I was speaking to you very early, I saw some of the very early versions of Bonfire when it was I don’t even think it was called that it was called something else I actually think, but how did that tool come about it was, I guess that was kind of a frustration of testing really and a bit like these innovations things trying to put something together that helps testers, right?
Andrew: So I think to preface that, my answer, if you look at the Agile literature there is a common response to what does QA do in an Agile team, it automates all the tests, that is what you do. And don’t get me wrong, automation is extremely important, in fact if you are doing any manual regression testing then I think you are doing it wrong. But when it comes to new features, in an Agile environment, you want to be able to adapt as you are building something and respond to those sorts of things, and you want to be focused on building the best possible product feature. Often times having to write tests at the same time can be constraining on that and what we find is that manual testing of new features allows greater exploration and less time with often times less setup. And often when a feature is finally finished that is the point when we can actually look at the automated tests created today and decide whether we need to extend those or actually cut them back.
So the point there is we find a lot of value in manual testing of new stories, but we don’t use test cases the traditional way, because things change and we’d rather spend a lot of time testing the products rather than actually writing documentation. So we use a technical exploratory testing and we want to manage that in a way, we want to have the ability to actually look at what exploratory testing sessions that took place, who did them what kind of issues they found, that sort of thing. And we went and looked out in the marketplace to see what tools were available and there was none, so we built our own which was Bonfire. So effectively it was a tool that was created for our internal use, but speaking to customers who were on the same Agile curve as us, we have a use for similar tools so that is why we commercialized it and now sell it as a product.
22. And really the tool is just about simplifying the things that a tester really does on a day to day basis when they do flip back into exploratory or manual testing mode, it allows you to annotate screens and send tickets straight into JIRA, and those type of things?
Craig's full question: And Jo you, I guess we are at an Agile conference and it’s kind of ironic I guess that the product you work on is GreenHopper which is the JIRA add-on for Agile management, so I have to ask: working on GreenHopper as a tool, does the GreenHopper team work Agile, do you use your own product and eat your own dog food?
Jo: Yes, we absolutely do, and in fact the team used to work with Kanban and we switched to Scrum about this time last year maybe a little bit.
Jo: I know, and actually I think in some ways I would like to see us go back to Kanban maybe, but part of the reason for that was it was the point where they started building up the Scrum part of the product and there was a need for us to “dog-food” the product, if that’s a verb, and yes, so they wanted to be able to see how the Scrum product was building up and so using it day and day out, I think it gives you a much better idea of what you are building and making sure that we are kind of living by our values and giving the customers something that we like as well.
Andrew: Every team is free to choose its process and methodology, so there are some teams that are hardcore Kanban, hardcore Scrum and everywhere in-between, there is a lot of latitude there, but we certainly don’t have any teams that you would classify as traditional in any way.
Craig's full question: There’s lots of interesting, and if you go and look at the blogs at the Atlassian website, there is lots of interesting articles I know you’ve written lots about this Andrew in the past and I’ve seen some of your posts as well Jo on different things, I mean even down to the way you guys do technical writing and all the way through the entire stack of software development innovative approaches to tackling software development problems. It’s interesting that apart from where you actually use the word Agile in relation to GreenHopper, I never actually hear the word Agile really mentioned at Atlassian though and do you think that is part of the success of how you guys deliver, because you just do stuff?
Andrew: I have a little theory about this, to me when I think about Agile, I think that’s being an adult and traditional ways being a child. Some people, for whatever reasons, sometimes in maybe good reasons, prefer to be told what to do, but I think if you are going to be successful with Agile development, then you have to step up and be: “I want to be an adult, and I want to choose my own direction and I want to be accountable for the decisions I make”. And once you have that attitude then all of a sudden you are looking at everything saying: “Well I’m accountable for this and I can change things”, so you don’t have to put up with bad processes or bad tools and that sort of thing, and you can just go on and improve it, and fix it. And I think then when you get to that point, it doesn’t really matter what labels you apply to these things, this has just become natural and as a result, those kind of terminology begins to fall away.
Andrew: That is right, and I think also when you have two founders who dropped out of university and never worked anywhere else and never experienced say traditional Waterfall type approaches, then it was never that need to distinguish what we do with those other ways, it’s just the Atlassian one.
28. And it’s always been an innovative company and really pushing the boundaries anyway, I mean we talked about the hackdays before, and the 20% time, the equivalent to what Google, what most people would know, lots of things like that I guess in the organization, the Agile community kind of associates with Atlassian as a leader in those spaces?
Andrew: Scott, one our founders, often likes to say: “Don’t ask for permission, ask for forgiveness”. I think he is worried that the company is all going to want to funnel every decision through him but I think that is an ideal that a lot of people in the company actually hold dear, that we can have that freedom and that opportunity and shouldn’t let it go to waste.
29. From the software developer’s perspective in a company like that, what sort of freedoms does that give you, I mean do you use the traditional Agile development practices and things like Continuous Delivery?
Jo: We do and we release GreenHopper every two weeks which is fairly continuous, I think for our customers that is enough, like if we would try to do every single day, then they would probably get a bit upset. I think one of the main things for me is that the team is very self-managing, so we decide who is going to work on which stories that come up in the sprint between the team members, the actual developers, it’s not like our manager sits us down and says: “Right Jo you are going to do this, Michael you can do that one”, and that is just a great way to work, you can kind of look at the story list and think what’s interesting “I just want to do that one” and then you see the other ones coming up, and it’s much more motivating, and then you have the responsibility at the end of the sprint that you actually did it, so I think that is one of the main things for me.
30. Back on the testing front, one of the things that I do want to ask you Andrew, is that you talked about your QA’s and not the traditional ones, does that imply that testers at Atlassian have a technical expertise and then how does that fly in the face of you actually finding really good staff, because I assume that’s hard?
Andrew: We ask a lot of our QA engineers and we want them to be technical and we also want them to be good facilitators because they actually are trying to assist developers and that requires some soft skills that a lot of testers traditionally may not have. The technical side is a must have but it’s very difficult to find people strong in all those areas, so we do hire people who have good technical aptitude but perhaps they’ve never actually done anything with that from university. We will invest in courses, we are training to bring those skills up to speed, but yes, technical skills are a must. I mean for us, the ability for a QA engineer to actually just review the commits that the developer has made, and be able to say: “Right, I know what has changed, so therefore I know what to then test or what’s at risk”, it’s hugely powerful and enables us to make sure that we test the right things and not everything, and that way we can create that speed. In terms of finding people, it’s a big challenge, I think I haven’t counted recently because it’s a bit depressing but I think it’s, we hire one person in every 120 who apply for a QA role at Atlassian, but on the bright positive side, that is changing, I mean that is over a five year period. Five years ago these types of people with this experience were very rare, but as Agile gets more adoption, we are finding that there is more and more people who have now got the experience that we need.
Andrew: The area that I’m really excited about is analytics and finding ways to get analytics from SaaS customers and inserting that directly into the development process, I mean some of the things that are pretty common place I guess in terms of running A/B experiments that sort of thing. But I think there is also a possibility for us to actually test out more our decision making process in Atlassian. Sometimes when you are looking at something and ways to implement it, you might end up with multiple options and no clear winner as to which is the right decision. The ability to actually quickly run an experiment against a set of customers to get some information that actually allows you to answer that question, as you need it. I think could be hugely powerful, so that is the sort of things we want to start working on.
Jo: I’ve been playing around with some Scala so we’ve got some performance testing stuff for GreenHopper which is all in Scala, so that it’s a new language for me which is pretty exciting, and I’m getting quite involved in all the build stuff, obviously our builds are on Bamboo so I’m trying to skill up in that area, it’s pretty cool. And then, I don’t know really, other than that, that’s mostly it for Atlassian.
Andrew: Well the Atlassian developer blog is the place to go that is where we publish all our insights and thoughts, and that is the place to go.
Craig: Thanks very much for your time, it’s great to see you again Andrew, nice meeting you Jo!
Jo: Thank you!
Andrew: Thanks Craig!