This is Deborah Hartmann at Agile 2008. I'm with Diana Larsen and Jim Shore. Please tell us a little bit about who you are and what you do.

DL: I'm Diana Larsen, I am chair of the Agile Alliance Board, which has kept me pretty busy here at this conference, but in my "other life" I am a consultant with FutureWorks Consulting and I've been working with Agile teams and organizations on Agile adoptions for about 6 or 8 years.

JS: I'm Jim Shore and I'm also an independent consultant. My background is as a programmer. I've been working with Agile teams as a coach or trainer since 1999, I had my first team in 1999. My focus these days is on the whole team: what does it take to bring the whole team to success - that means the technical people, the business people, the testers and how to bring everything together. I'm also a coauthor of the book The Art of Agile Development and a Pask recipient of the Gordon Pask Award.

I heard some buzz around your presentation on Wednesday, which was called Pressure and Performance - The CTO's Dilemma. What is the CTO's dilemma then?

When we started that, when we proposed the session, we had the intent of creating a pattern language of CTO level Agile practices, starting down that road and presenting the beginnings of that research here at the conferences, which is what we did. Our expectation was that CTOs are human in face and they are often seen as the people in charge and have no problems whatsoever and we thought "I bet you that's not true! They have the same pressures that everybody else does, just in a different way" and that's why the title Pressure and Performance - The CTO's Dilemma came from. As we did a research and we interviewed many CTOs, we found some strong themes, but the title of the talk was not actually one of the themes we found, but it's a great title and we didn't want to confuse people, so we kept it.

Whom did you interview?

We collected about 12 and ½ hours of interviews from 8 CTOs or people with CTO level executive title, and they were very generous with their time. Our average interview was 1 ½ hour long and the range was really broad we had a great group of people in terms of getting a variety of perspectives. We had the smallest, we had somebody who's in charge of 6 people at an 8 person company and at the largest we had somebody who is in charge of 700 people at a 21,000 person company, with so many locations that they said "Lots and lots" when we asked how many locations they had. About half of them were in product-oriented companies and half were in a service role to the rest of their organizations in some way. What was really remarkable about it, to me, was that even with all this variety, some really strong themes came out of the interviews.

How did you interview these people?

DL: We created an interview protocol so that we knew that we would ask the same questions of everybody, so we could be able to aggregate the data. For a couple of people who were more time constrained, we didn't ask all the questions, but the questions they were asked were a part of the protocol, so we were able to compare the answers to the questions they were able to get to. We had questions in maybe 7 different broad areas, there were 28 question items altogether and some of those items had multiple subsidiary questions, that's why it took 1 ½ hours. It was quite extensive. One of the most interesting questions was the question we started the interview with and that one was "Tell us about 1 hour of your life that you would like your teams to know about, that you would like your staff people to be a fly on the wall or whatever and really know what that hour was like for you". We got wonderful answers to that question, as they told us the story of these different hours in their lives and what was remarkable about that was the hours ranged from "an hour of me at home checking my e-mails" -sometime between 9 in the night and 2 in the morning - to "an hour when I'm sitting in an executive team meeting with my peers" to "an hour with customers", they were all over. But when we asked them "Why that hour? Why did you pick that hour?" the answer was always the same.

JS: I have a great quote around this, I have some of the quotes from the interviews here with me. The interviews were not to be disclosed, but we got permission to use some of these quotes. We asked "Why that hour?" "I really wanted my developers to understand what I understand, see what I see". This quote exemplifies that beautifully. - "There is tendency for software developers to be highly motivated, but they also have a tendency to not understand why it's important that they do their jobs well. I talked about our organizational structure and the 5500 external advisors that we support. It becomes vitally important when software developers realize that what they are doing is helping those 5500 guys support their families, and by 'support their families' I mean basic things like send their children to college, to buy homes, to clothe their children. They are really depending on our system to be up, available and reliable every day for them, because they transact business using the tools we provide for them." Things like that were throughout the interviews and just that focus on people was fantastic.

DL: "I really think it would help my people if they could get the context that we are living in. By seeing this hour, they could get the context that I see and it would give them a way of understanding their jobs better" - and that was the theme for all the answers to that question. It was just remarkable.

You started out by getting into the vision of the person you were interviewing. What else did you do when you were interviewing them?

DL: We asked them - this was another area that was somewhat surprising - a couple of different sets of questions about performance measures for them and for their teams. Reporter: What did you find when you asked them about performance measures? [0466] JS: This one I think is going to be surprising to a lot of people because there is this vision of the CTOs being very numbers focused, very bottom line focused, so I often hear people ask when I say "I want to introduce Agile": "What metric can I have to beat my bosses and CTOs over the head with?" - so it's obvious that Agile is the right thing to do. That's was not the reason we asked that question, but that was part of why that's an interesting question.

DL: I think it was one of the assumptions that we had. We needed to understand that.

JS: Yes, what metric were they using. I don't have a direct quote on this one but the response was very uniform on this, which means that they're assessing the performance of their teams and assessing their own performance - which I can talk about in a moment -, but assessing the performance of their teams was not based on hard measurements in most cases. One person said "It's 30% 'balance score card in master project schedules delivering on time, in budget' and it's 70% 'did they understand the business, did they assess the situation, provide the best solution, is the customer happy and providing great feedback'?" That was really consistent across our CTOs. One person said "I don't even know how to measure productivity. I can't measure productivity." In the absence of that it's ultimately about - let me read you this quote - "Have you made every stakeholder outrageously happy or not? That's pretty much it. There is the routine part of did you do the stuff you said you were going to do and there is the making people happy part of, did you provide transparency, so nobody gets blindsided and they buy entry trade-offs that have to be made ". We'll focus on marketability - we call it - either internal customers for the service organizations or external customers and profitability for their product organizations.

DL: Always the metric part - delivering on time on budget, like that second quote - it didhave to be there. That's kind of a base line, the minimum and it's, in some ways, a Yes/No measure. Then there is the more subjective side of "What are the reactions we are getting to that?", "How are we delivering what we said we were going to deliver?" and "Are we getting feedback?" and "Am I noticing that peers or our stakeholders are giving us feedback on that?".

You mentioned the issue of how a CTO's performance itself is assessed, tell us about that.

JS: There was really not much in the way of formal assessment and there was certainly no metric. It was more a case of "My peers assess my performance" and "My personal perception assessment determines my performance". Ultimately, there wasn't as strong of a theme here, but what was strong was that there was no real formal measurement process and it was a very pure based subjective approach.

DL: Except in a couple of instances. There were couple of instances when there were some kind of 360 degree, but even then, the performance wasn't measured around the 360, it was what did they do with that feedback? Once they have gotten the 360 feedback, did they then commit to and take some action based on whatever the feedback was and then that's what people look at, not the results of the 360. Again it was not metric based, it was quite remarkable that way.

JS: I did not get the impression that the CTOs were doing what they did out of desire to improve a metric or fear for their jobs. It was really about "We want to serve our customers or the company", or in some cases, "It's important for us to make money so we can continue building a great organization". Now they did not say that, but the bottom line focus was that we expect that everybody else has, turns out that they are just like us and they have things that they care about and the bottom line is a means to the end.

This all sounds very rosy and it's making me wonder who you interviewed. Were your demographics varied?

DL: Yes, we had 6 men and 2 women. Some were in very traditional organizations, been around a long time. One was financial services and then there were some more recent kinds of industries - web based and those kinds of things. And a couple of start-ups. Then it wasn't all start-ups and they weren't touchy-feely industry or anything like that. Reporter: Were they all high performing organizations? [0818] JS: I don't think so. Some of them were doing really well. We asked them "What keeps you up at night?" They said "Well nothing at all. Pretty much got it solved". Some of them, were asked "What keeps you up at night?" and their answers were things like "I worry every day about having enough money to keep this company going" and that was obviously a start-up, or "There are things that we haven't figured out yet and that keeps me up at night" or "Risk, dealing with risk". There was not a strong theme around that.

DL: Each one was very situation-specific to what keeps them up at night. We didn't get a theme across that. It was quite a varied group which is part of what surprised us when we started seeing these strong themes emerge. This is coming from everyone or almost everyone of these people and yet they are in such different circumstances. All of them are at some stage of an Agile adoption, so that is something they have in common. We didn't intend to do it that way. It's just how the scheduling worked out, because we had names of folks, CTOs that we would want to interview and are still in our queue, that aren’t involved in Agile adoption. It was just coincidence that it worked out that way, but some of them are very early on and some other have been doing Agile for a few years, others were very early on in their adoption, so it's not like they had done a big culture change in their organization, not yet anyway.

It will be interesting to see what happens to your results when you interview the people who are not using the Agile.

JS: You said "It seems very rosy" and I find that interesting reaction because that's my reaction, but is it so surprising that CTOs are people too and that people are generally interested in the well being of other people? Is it really that surprising? Or is it just the image we have of the CTO in the big corner office?

We have stereotypes, don't we?

JS: Absolutely. That's one of the most valuable things that came out for me is the humanization of the person in this role. They are people and their concerns are organization wide concerns - I have some quotes about that - but they are still people.

DL: I think one of the things I was surprised by is that we realized that some of those assumptions that we had made - because of course my friends are my colleagues that I've already known were CTOs, that I already knew were human - were exceptions. Even in the interviews, as we interviewed them, some of these quotes would come out that were very much about caring about people and customers as people, so wanting to get results for customers, too. They would say things like "I know you probably don't hear this from a lot of people but this is what I do". Even they had the same assumption about each other, which I thought was very funny. There is this sort of concept of when you get to that C sweet level that everybody is very bottom line focused and it wasn't just our assumption, it was their assumption too, except of course, "I know myself and I am an exception". Sometimes they were a little bit sheepish about that "You know, I am really kind of thinking about people".

JS: They were bottom line focused and there were some things that people said that I can see that if you are working for that person it would seem perhaps like an ogreish thing, like "We expect that a minimum that you will deliver on your commitments - that's a very minimum level of performance". That seems actually like a hard expectation but also an expectation I can understand, probably because I know that with Agile development, we know more how to deliver on commitments than we used to, so it doesn't scare me as much as it used to. The expectation that "We really want you to understand what our customers need and make them happy", that was very strong, but it doesn't come out of the desire to make people's lives miserable.

Tell me about some of the other themes that emerged.

DL: Another theme that came out - and we actually did specifically ask that question about this and we got it in the answers to this question, but we also got it at other places during the interview - was we asked a question "When in your life do you encounter a situation where you have mutually contradictory goals and how do you deal with that?" Because we figured that would happen sometimes when you are at that level - dealing with paradoxes as a manager and as an executive. What was surprising was the answer, which ranged from "Oh, always/all the time/every day/constantly" or "Once every couple of weeks I am doing that" and it came across like "Well, this is just part of my routine". It's not like "I have to wrestle with this once in a while and because I am at this level of the organization I have to do this wrestling with this". It was "No, this is a daily part of my job. I know I am constantly going to be encountering this situation where we can have both of these things at the same time and I have to find a solution for that". That part of it was a bit surprising and how consistent it was among all of them. I mean that’s one of the things we really learned is that that is a really big part of their job - wrestling with paradox all the time.

JS: It really was routine that if we asked "What's notable about your job?" but that's not how we asked the questions. It wouldn't have come out, because it's routine, it's ordinary. I think I have some quotes along that line. Here is a great quote! When we asked about contradictions not all the quotes were this extreme, but this I think really demonstrates the point: "There are a lot of things that come across as just bad or stupid decisions that are made at the strategic level for the company, that are indeed bad or strategically poor decisions made by executives, but they are still made out of necessity. One example is that we had to sign and additional project with the customer that we didn't necessarily have the resources for, so we had to pull resources off of other projects. The reason for that was to get the cash flow going so we could meet payroll so we wouldn't go out of business.

And that was one of those things that was highly criticized in the company. 'Why are we signing another project when we have as much work as we can possibly do? We can't sign another project! You are harming the projects in existence by sending these other projects!'. We couldn't really explain it to them that 'we were trying to get your paycheck this week by signing this project'. It was a strategically bad decision to make, but it was a tactically necessary decision. We can't go to these people and tell them 'We can't afford to pay you this week' because we would lose half of them".

One of the topics you talked about in your session was emotional baggage. Can you tell me more about that?

JS: If the routines of contradictions is one thing, just an everyday part of the CTO's job, and unsurprising to them, another thing that's a routine part of the CTO's job - that I think many of the people we interviewed did not really think of it as a part of their job was dealing with - the term they used was - emotional baggage.

DL: That's their words, not ours. We didn't load that. That exact phrase came up several times in the interviews. JS: It could show up as something the CTO would actively manage because they were very focused. We asked them to prioritize what their most important topics were. Number one for everyone or almost everyone at least, was building an organization where teams strive and produce. Part of that is having to deal with these interpersonal relationships and some people managed to do this part of that job and some people said "I wish that people would take care of this on their own rather than bringing it to me". But either way, it was very present, and I have a quote about that.

This quote is from somebody who really wished that it wasn't part of their job, and I don't know if he necessarily thought of it as part of his job, but every CTO had this kind of thing happening in their lives. "Some of the issues that rise to the level they probably shouldn't are some of the interpersonal issues. I've seen these issues destroy projects, I've seen teams where certain people on the team refused to speak to other people on the team, even if they did talk, and even if they did talk to the person, it was so uncordial that it precipitated a bad atmosphere for the team. One of the more amazing things about that is I would have tolerance for it, if it was a 20 years old boy just out of college having problems with a 24 year old coworker, but I've seen instances where even people extremely mature, at least from an age standpoint have these difficulties working with other individuals.

I would like for a lot of those problems to work themselves out before coming to me, but those are a lot of the problems that are escalated to me in every job where I've been a leader in the organization". This emotional baggage was ever present and it was also something that the CTOs felt really could take a lot of time away, and I have a quote around that. This person said "The thing that makes my job easier is when we have teams that don't carry a lot of emotional baggage. One person who carries a lot emotional baggage can end up sucking all the management life out of the organization and ends up with their team spending so much time managing the situation, that there is very little margin left for exploration and some of these other things that we find so satisfying. I wish I had the answer to how you help team members with the emotional baggage become happy healthy satisfied people.

There is a clear desire! "Please, help me get my team happy, help me help them let go of the emotional baggage!"

JS: Yes, and even the people who did not express in that way, it was really clear that this was a major part of the life and nobody seemed to have a very clear answer of how to deal with that.

DL: This was one of the dilemmas.

What opportunities are you seeing? You went on this quest because you wanted to understand these people, to serve them better and because they impact the work that you do with other people in the organization. What opportunities are coming for you out of this new knowledge that you have?

JS: We started this with the intent of creating a pattern language of CTO level practices and using these themes I think we've identified some preliminary directions that look really strong. We have 4 general categories or directions to pursue that could lead to CTO level practices or practices for the CTO. We have another 4 or so categories that look like they are going to be good directions to pursue for the teams to help their CTO and help provide the CTOs with what they need.

When can we expect to see those patterns? Do we have to wait for the next conference?

DL: I hope not. We wanted to do a few more interviews, as I said earlier and then we will really start to consolidate and I suspect the first article that will come out will be one that gives an overview and then we'll start drilling down and dig each of these areas and explore them further, because each one is so rich. What's been a bit of a pleasant surprise for us is just how much rich data and information we are getting out of this and how many avenues to explore.

JS: When these themes come out they shout at us from the interview data - this is really a common issue, either common point of view, or as with the emotional baggage, a common concern. I think it will keep us busy for a while.

How much longer do you expect to be doing interviews? Or how many more do you need to round it out?

DL: We don't need a lot more. It will depend some on folks who have come forward to us and said "Now that I know about this, I really want to be a part of it. I really want to be interviewed". That actually was another piece that when we got to the end of the interview protocol, we asked folks "As a result of this interview, have you learned anything? What have you learned?" We got many answers that say "You know, I didn't realize this was going to be so valuable to me. I really had a chance to reflect on myself. I've learned this about my job." We have had other people come forward and say "I really want to be interviewed." I have to manage that queue a little bit. I think we want to do at least 4 more, maybe a few more than that, I'm not sure where the right number is going to be. Then I think we'll have enough data because the themes are coming out so strongly. If it was muddy we might need more, but it is just really clear what folks are telling us.

What's next for you two?

JS: We are very excited about pursuing this CTO research further. In the short term, Diana and I are delivering a course together in October, based around my book The Art of Agile Development. On October 27 and 28 we are delivering the Art of Agile Planning and in the remainder of the week, which I believe is 29-31 we are delivering a course called The Art of Agile Execution, which I think is going to be really neat, because it's going to be cross functional. It's going to be programmers and testers and customers - we are hoping to get a cross functional group and talk about execution for all these people.

These are public classes that you are running?

JS: That's right. There will be in Portland, Oregon.

I believe that you have an invitation you want to make?

DL: We absolutely do. We have formed a Yahoo Group - AgileCTO@yahoogroups.com and we invited all the people who came to our sessions to join that group with us. That's probably the first place where you would see further results out of this research, because we will have some discussion there and be learning more about what people want to learn. That will help direct the next steps of this, but also that's where we will be reporting out things as we find them before they are in article from.