Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage Podcasts Joe Levy on Engineering Effectiveness through the Pandemic and beyond

Joe Levy on Engineering Effectiveness through the Pandemic and beyond

In this podcast Shane Hastie, Lead Editor for Culture & Methods, spoke to Joe Levy of Uplevel about the results of their Engineering Effectiveness survey, the levels of stress and burnout in engineering teams and what managers can do to support their teams.

Key Takeaways

  • Engineering productivity cannot be measured in the same way as some other areas of an organisation can be
  • There is no one factor which indicates engineering effectiveness, it is a complex problem with many interdependencies, the job of the frontline manager is to figure out across a variety of factors, is my team healthy?
  • Software engineering productivity levels stayed consistent despite the massive disruptions in 2020.
  • As a discipline, software engineering is getting their work done but doing it at a great cost to their own mental health
  • Managers need to be very aware of what their people are doing, the levels of stress they are experiencing, the interruptions to their work and proactively support people to take enough time out to protect their mental and physical health

Introductions [00:15]

Shane Hastie: Hello folks. Before we get into today's podcast, I wanted to share with you the details of our upcoming QCon Plus virtual event, taking place this May 17 to 28. QCon Plus focuses on emerging software trends and practices from the world's most innovative software professionals. All 16 tracks are curated by domain experts to help you focus on the topics that matter right now in software development. Tracks include leading full cycle engineering teams, modern data pipeline and continuous delivery, workflows and platforms. You'll learn new ideas and insights from over 80 software practitioners at innovator and early adopter companies. Spaced over two weeks at a few hours per day, experience technical talks, realtime interactive sessions, asynchronous learning and optional workshops to help you validate your software roadmap. If you're a senior software engineer, architect or team lead and want to take your technical learning and personal development to a whole new level this year, join us at QCon Plus this May 17 to 28. Visit for more information.

Shane Hastie: Good day folks, this is Shane Hastie for the InfoQ Engineering Culture Podcast. I'm sitting down today with Joe Levy from Uplevel. Joe, thanks very much for taking the time to talk to us today.

Joe Levy: Thank you, Shane, appreciate the time.

Shane Hastie: Probably a good place to start is who's Joe?

Introductions [01:47]

Joe Levy: Great question. Who is Joe? Let's see. I grew up in the Seattle, Washington area, moved around the US for college and quite a few jobs. I was an engineer coming out of college, and my first four or five years at work were writing code and shipping product for a large consulting company called Accenture. Went on to do a couple startups and a couple big companies, and moved through various management roles, in product, and marketing, and even a small stint in sales. And, I think the culmination of my career that I've really enjoyed shipping products, and understanding the process to get there.

The fundamental shifts in the way software is developed [02:21]

Joe Levy: To play it right into what I'm doing now, the thing that I have continually noticed is this great transformation from when I started developing code, which was many moons ago, and you would wait perhaps what felt like six months for the product group to drop a phone book sized set of requirements on your desk. And then, you'd work for two years to ship that thing. Now, if you haven't shipped two or three features in a two week period, you're not moving fast enough. That's great, that's fantastic. But of course, that's fundamentally shifted the way engineers work.

Joe Levy: And, what's been really interesting to observe over my life is that I think the tools for other disciplines have helped enable their work, so we could talk about sales, or marketing, or finance and all the wonderful things they use, various SaaS products like Salesforce and Marketo, and those kinds of suites. I don't think there's tools that have really thought about helping the engineering discipline. If I think to the work I've done, to the teams I've managed, to the groups I've been at, there's been here's where you do your code, and we figured out how to deploy that faster in some kind of CICD pipeline. And then, there's the HR engagement survey that comes out once a year, and there's this huge void, in engineering land, between them.

Joe Levy: I think a lot about what makes a great engineering team be engaged at work and ship great product, and at some level, have fun in their jobs while working in this we need to ship something every week or two mindset. That, to me, always felt like a growing void in the market, and is something that I've built just a lot of passion and have fun thinking about.

Shane Hastie: Maybe we can dig into that one. What makes a great engineering team?

The desire to measure engineering productivity [04:03]

Joe Levy: Yeah, that's a great question. You know, I think there's been a lot of thought into can you measure the productivity of an engineering team. I think that is an elusive, almost evasive, "Surely, we can somehow figure out how to put a number on it." It has been the management quest ever since someone figured out that you could put in Salesforce and measure ROI per salesperson.

Shane Hastie: Fred Brooks with the mythical man month in 1980.

Joe Levy: Exactly. That's a perfect analogy, Shane, it really is. Again, it gets back to, at a C-suite level, the head of sales has to predict the future and says, "Well, if you want more revenue give me 20 more salespeople." And because I have 200 salespeople and they're all using some system, I can tell you the revenue per person, and I can give you the ROI of that investment in a very clean way. And then, you swivel chair over to the engineering head and say, "When are we going to ship the next features?" And they say, "Well, my burn down's pretty decent, I think I could use some more head count. I'm not quite sure where the bottlenecks are," and you get this very tough to answer question. You could argue it's for the tooling, which I think is a big part of it.

Engineering is an artistic discipline [05:13]

Joe Levy: But, I think to answer your question properly, it's an artistic discipline, that is the nature. If you're a sales rep, Shane, and I'm a sales rep, and you're kicking my butt because you're selling more product than I am, and let's assume it's properly set up where perhaps you have a different market than I do, I'll just copy what you're doing. That's lovely, and you're happy, and I'm happy, and the company's happy. Now in an engineering world, if you're particular piece of code has less bugs than mine, I can't copy yours, that's a ludicrous thought. It's different.

Joe Levy: One of our customers explained it very well. It would be like trying to measure who was a better painter, Picasso or Monet. Let's count the gallons of paint they used. It is a true statement they used paint, and so some number of gallons is probably a reflection of something, but it is a ridiculous statement to say that more gallons is better, or that it has anything to do with the quality of the shipped deliverable.

Joe Levy: My belief, to answer your question, is if you want to define a good engineering team, you have to think about a whole bunch of attributes which are complicated, and not any one of them is right. You have to look at are there clear requirements, is it an engaged team, is it the right level of skillset, is there a good time to work, how's the interruption factor, on, and on, and on. There's probably 20 or 30 things you have to study, and on any one of those things you're looking for healthy behaviors.

Management’s job is to nurture team health [06:37]

Joe Levy: Another one of our customers summed it up very well said, "It's almost like the blood report that you get from the lab that your doctor reads to you. It's filled with readings of a whole bunch of chemicals that you're not really sure what they are, or you didn't even realize your body had this many things. Some of them you might recognize like cholesterol, half of them you don't recognize. And, you're hoping that your doctor can make sense of them." That's the job of the frontline manager, is to figure out, across a variety of things, is my team healthy?

Characteristics of a great engineering team [07:06]

Joe Levy: So when you say, what's a great engineering team, I think about not are they productive by some arbitrary measure of commits, but are they able to deliver, do they have time, are there clear requirements, how's their burnout factor. Let's look at a variety of things and say, "Well, I have a team that's running effectively, and they are able to do their job in a proper way." That's probably the best you're going to get for saying, "My team's running well." There's not a universal number truth to that answer. People don't like that, and that's hard in business. But again, you go back to you try to put a number on it, you'll be wrong. Monet used more gallons than Picasso, doesn't matter. That's the perspective I have.

Shane Hastie: The engineering effectiveness survey that you did at Uplevel, tell us a little bit about that.

Overview of the engineering effectiveness survey [07:58]

Joe Levy: The theory we had was, at Uplevel in our business, and we can talk about that a bit later if you'd like, we look at actual real data of how engineers get their work done. We look at both their traditional engineering metrics, but also some of the more collaboration-based metrics. Things like time, and collaboration patterns, and those sorts of studies. As an analytics company, we always have a very strong desire to make sure we are accurately representing what's going on. We see what we see in our data, but what we wanted to know is how does this reflect people who aren't in our system. Is what we see in our dataset representative of the entire market?

Joe Levy: To try to understand that more deeply, we went and did a survey. We partnered with a leading company called Wakefield, and we surveyed a whole bunch of other engineers, specifically looking for very similar to the InfoQ audience, the first line managers or folks that are working level trying to keep the engineering teams effective and happy. We surveyed them, and asked them questions that were similar to the sorts of things we measure, trying to understand if there was good correlations in behaviors of what we saw.

Results of the survey [09:04]

Joe Levy: In summary, here's really what turned up. When we looked at it, and a lot of our lens was specifically around COVID and how people have been working in 2020, which has been, as everyone knows, a very challenging year. I'd say one of the biggest things people were concerned around is okay, there's this mass disruption in the world. Have software engineers still been able to get their job done? If you look at most of the raw metrics that most software teams would use, things like commits, and JIRA tickets, and those traditional measures of the thing got produced, the work got checked off, what's absolutely been shocking is most software teams have had almost no dip. I mean, very little, negligible amounts perhaps right in March, when everyone was shuffling around, and very quickly returned to normal.

Productivity stayed consistent despite the pandemic disruption [09:51]

Joe Levy: At one level you'd say, "Yeah, that's great." Software has always been viewed as this fantastic skill that you can do from anywhere, all you need's a decent internet connection and you can work from home, and it's this great malleable discipline in that way. Hooray to the software engineering discipline, we should send more people to computer science school, and that's the tried-and-true thing. What a lot of that misses, though, is to get that done, to keep that up ... one of our customers said this very well. There was a little bit of almost everyone was on steroids for the first two or three months. And, that is actually a psychological thing that happens during pandemics, there's been proven studies on this. When there's real fear and uncertainty in the world, shots of adrenaline rage into your body to go figure out what to do.

Joe Levy: People were working all night, and juggling how to deal with their kids on remote school, and set up their home offices, and do this stuff. Even if you were already a work from home person, you had to now collaborate in different ways. There were very few companies where the effects weren't big. The challenge with that was that lasted about three months, and then amazingly it started to settle into this rhythm where the deliverables were still going out the door pretty well, but that steroid, that adrenaline shot, had worn off, and then what you saw was at burnout factor started to get higher.

The increase in burnout and exhaustion [11:10]

Joe Levy: What we see, and leading all the way up until now, is a little bit of this pressure cooker where sure, I'm shipping, and sure I'm having my team meetings and doing my stuff, but I'm back-to-back Zooms, jumping around, taking breaks to walk my dog and deal with my kids, and fix tech support at home. And, making up for it by working nights and weekends because my office is 10 feet from my bedroom anyway, and I can't do much else. There's just no break in the day, and it's just really tiring for people.

Joe Levy: As a discipline, more than almost any other, software engineering is getting their work done but doing it at a great cost to their own mental health. That's been something that we saw in the survey, it's a top concern of the managers, it's one of the most hardest ones to deal with. If you're a technical manager and you see the team falling behind, the easy thing to do is, "Let me go help you code, or fix this issue, or unblock that." The harder thing to do is say, "Take two days off, you need a rest." Those challenges are very, very real today, and the problem is it's just becoming ... you think of it like a rice cooker, and it's just getting hotter and hotter, and drier and drier, and no one's turning the heat off on this thing.

Shane Hastie: So I'm an engineering manager, I've seen this, I'm intuitively feeling it myself, I'm seeing it in my team. What do I do?

Factors causing pressure and burnout [12:27]

Joe Levy: This is the hardest thing to deal with, so a couple things on there. And again, in our survey some of the specific things that come out, and this has been corroborated in our own data, some of the specific challenges that a lot of people had are things like being always on, or not having time for deep work, getting interrupted a lot more.

Joe Levy: There's a lot of studies out there that'll say, and you've probably seen some of this in your other podcast or work, an interruption to a software developer, if they're in their deep work flow state, will take them, on average, about 25 minutes to get back into it. Now, we've studied this a lot, there's obviously different levels. If you just say, "Where's the link to that?" And, I just send you the link, that's probably not too big of an interruption. But if I say, "Hey, I don't understand what you wrote here," and I have to stop and write you a paragraph back, for me to get back into my flow state, that takes a while.

Joe Levy: The challenge is, everything's digital now. I don't know what you're doing, for all I know you just came back from lunch, I feel totally fine to interrupt you. In fact, you will respond faster because it's your only communication path. When you're sitting at home, there's a fundamental, "I need to make sure everyone still knows I'm here, and I'm supposed to be working. If anyone Slacks me or anyone messages me, I should respond pretty quickly." That is a real feeling out there. It did not exist in the same way in the office, because you're in the office, you're here. The same pressure is not the same.

Joe Levy: People are responding faster, that's driving up the interruption factor, driving down your deep work time, making you pick that slack up in the evenings, or in the weekends, or times when the interruptions will be down and you, in theory, have free time because there's not much else you can do because you're supposed to be staying home anyway.

Advice for managers [14:08]

Joe Levy: A great engineering team, a great manager, has to first of all, be deeply aware of this. You've really got to be in tune with what's going on in your team. It's really hard to just say, "Shut down your notifications," if you don't have a sense of how big that problem is. Otherwise, it just feels like, "You should be healthier." Okay well, fair, we should all be healthier. But tell me, how bad is my problem? What do I really need to do here? We've seen in our data, we've seen in the survey, we've seen in our own teams, some devs that are getting interrupted three, four, five hours a day. How can that person possibly have a productive day, if all they're doing is responding to other people, answering interrupt driven questions?

Joe Levy: I think a lot of what we would say is the first thing you've got to do is just be aware of that. Be aware of that cultural change, that people are feeling like they have to be more always on, they are being more always on, and that's showing up in the data. Once you're aware of it, then you get into this okay, now what do I do to actually fix it? There's a variety of tips and tricks. I always think of the actual fixes, they're not easy, but they're a lot easier when you're aware of how big the problem is.

Joe Levy: And again, another one of our customers said this. "If you tell me yeah, you should probably lose three pounds, you should go out jogging a little more and be a little healthier, I get it, I'll probably do it. Maybe or maybe not. If you say you are at major risk of a heart attack if you don't start notably changing your diet and having significant different life choices here, you might take it a little more seriously and actually start doing something." I'm not trying to say that's the exact risk factor we're at. But if you don't know how bad the problem is, how can you even understand how urgent the solution should be?

Joe Levy: The kinds of things we've seen practically that works, it's some sense of the age-old stuff. Block good time, set times to be available, try to encourage an environment that doesn't require people to respond all the time. Notifications are just this incredibly double-edged sword. They're so wonderful because they pop what should be important, but if you feel like you have to reply to every one, it's no longer meaningful now. You've just killed your day. How can you think about your devs' time, really prioritize it, and try to set boundaries around that? It's got to be okay for people to not reply, even if it's coming from the senior exec with what seems like it might be a fire drill.

Joe Levy: We've had customers that say, "Well, all my interruptions feel like they're the top issues." Well sure, that's what an interruption is. But, someone's got to look back and say, "I can either ship these features, or deal with all the customer fire drills. I'm happy to do one or the other, but you have to help me prioritize." As a first level manager, have some data around that and forcing that conversation can be really valuable to protecting your team's time, and at least having some sympathy for the pressure cooker everyone's in right now.

Shane Hastie: Speaking of that pressure cooker, we've come through multiple rounds. I'm incredibly fortunate, I live in New Zealand, we're pretty open. But, most of the world is coming back into lockdown as we go into Northern Hemisphere winter. You said right at the beginning, there was the adrenaline rush. We certainly haven't got that with this round, and you can feel it. People are just tired. How do we help?

Joe Levy: Shane, it's such a good question because it's such a sympathetic question to the actual challenges that are in the world. And specifically to software engineering, which again, I'm not trying to say other disciplines aren't also very challenging, but I think there is a perception that okay, all you need's a computer and internet connection. What's the problem? Keep working. You're in a blessed situation. But of course, it is incredibly tiring, you're exactly right. Yeah, I live in Seattle and the lockdowns are not quite as bad as California right now, but it's not good.

Joe Levy: One thing I've been thinking about lately. While it's encouraging that there's a vaccination just entering the market, only just, I know myself, and I think a lot of folks in software engineering type roles are probably not, for just reason, but are not going to be towards the top of the line to get it. You're probably looking at, at least half the year, if not probably more towards the end of the year, before folks in software roles can get the freedoms they might have been used to beforehand. The negative side of this is you think you're tired now, you've still got the better part of a year to go. You need strategies to get through that, because you can just power through. It's not the one all-nighter. It is a marathon, not a sprint.

Experiments and strategies to help cope with the levels of stress [18:31]

Joe Levy: A couple strategies that we've done at Uplevel that have been fun to experiment with. We're a small company, and we do have people can essentially take vacations whenever they want, it's a very flexible vacation schedule. One thing we have experimented with lately that's been quite effective is hey, you can still do that, but we recognize no one's taking their big vacations. The trip to Europe, and the big hike to the Grand Canyon or whatever it is, those are all cancelled. People are just staying at home anyway, which is what you're supposed to be doing so that's good. But, it's not good from a vacation point of view. Even if you say, "Well, I'm going to take vacation and I'm going to sit on my other side of my apartment or house, and read a book, or plant some flowers in the garden," or something, the notifications are on the computer 20 feet away and it doesn't really feel that relaxing or refreshing.

Joe Levy: So one thing we've tried to do is say, "Okay, how about for once a month, let's pick a couple days and maybe try to align it with some other holiday as appropriate, and shut the whole office down more proactively." We've been doing that a few times. It's actually still been less vacation that most people would have taken on their own, but it feels like a better vacation because you shut everything down at once. The big effect on that is recognizing that teams are collaborative by nature, so taking one person out just causes that person stress when they come back, and slows everyone else down. But, taking everyone down for a little bit gives everyone an opportunity to refresh. I don't think that works great in a more normal world because people like to take vacations in different times and different ways, but in a world where you can't it actually works quite effectively.

Joe Levy: We've done quite a bit of that now, and I know a few of our customers have done that even on team levels. Perhaps the mobile app team, or the front end team, or an architecture team all decides, perhaps 10, 20 people in a group, let's all take the full week of something off, or even a long weekend. That can go a long way to feel more rejuvenating, because as everyone knows you're not coming back to endless messages to catch up on and you don't feel like you were blocking your co-workers when you were out. It's a little more relaxing in that way.

Joe Levy: The trick in all this, Shane, is how do you get someone rested and rejuvenated, when it doesn't feel like a natural break? It's a really tough thing to answer. There's just no amount of Zooming that'll make you really feel rejuvenated. And when you can't do that thing where you truly shut down, it's hard. The best things we've found are trying to do team level or company level shutdowns.

Joe Levy: The other piece that, again, is to go back to "Hey Bill, you've been working incredibly long hours with lots of interruption factors, very little deep work, for what seems like weeks, and weeks, and weeks on end. We need to reduce this, you can't sustain that." Everyone's always proud to pull a few all-nighters or work hard when they're shipping something they're really proud of, another great feature of software engineering teams. Everyone's had that adrenaline rush. You stay up late, ship something you love, gets out in the wild. It's a high, it's a total dopamine hit. But, you can't sustain that for four weeks in a row. Again, it's adrenaline, it's not sustaining.

Joe Levy: To be a manager and be conscious of just how folks are working and really work to say, "We've got to get you a break, you've got to shut off. You've got to do something in the evening. I don't care if you're walking 10 feet and binge watching some Netflix thing, do something that is not work because your brain requires it." Walk around your background and listen to an InfoQ podcast, something. That is something that we are saying is not normal for most managers to have to say so forcefully. So we're really trying to encourage, within our company and leaders in general, to push harder on really trying to tell your folks to shut down because they're not, and people can't sustain this.

Shane Hastie: That means, as a manager, I have to be very, very aware of what people are actually doing.

Managers need to be very aware of what’s happening with their people [22:16]

Joe Levy: Yeah. And, that's harder than ever because you're not next to them. And no matter what anyone says, when you sit there on Zoom, it's very hard to see the real expressions under someone's eyes, and you don't really get a sense for how tired someone might seem. It's made exponentially harder if it's a newer employee or a newer manager, and you haven't had a pre-COVID in-person relationship. Let's say it's someone that recently joined your team or recently got hired, or in some reorg came to your team, and you just don't have the same contextual history, knowing how hard you're really working is something that's extremely hard to do it.

Joe Levy: Our belief is we provide data to help with that. It's never the whole answer, but it can help shed a lot of light on things. We give that to both the individual developer, to bring to their manager, as well as the manager so everyone's on the same page with what that is. And, the manager needs to be aware of what that is, they need to work a little harder than normal specifically in trying to assess the burnout risk and velocity that developer's working at.

Joe Levy: A simple way to think of it is you can't solve the problem if you don't understand it. If you generally think it's bad but don't have some specific numbers and guidance on just how bad, or is it trending up or trending down, how are you really going to help? You can't improve what you can't measure.

Shane Hastie: In some ways, I think that cycles us back around to tell us a little bit about Uplevel and what you do there?

Introducing Uplevel [23:40]

Joe Levy: Yeah. Yeah, thanks Shane. Uplevel was founded to be at the intersection of people analytics and engineering work. Where there was no engineering tool that felt like it was focused on the people analytics side of engineering, and the people analytics tools felt like they were made for all information workers, with very little focus on the engineering discipline. Which again, is much more, I believe, artistic and has some very unique properties, relative to many other disciplines.

Joe Levy: What we do is we take data from systems that are classically more collaboration and how teams are working. Think about things like Slack, or Calendar, or even HR engagement data, those sorts of systems. We only take the metadata, we're not reading super personal stuff. We're just looking at behavioral trends, you'd call it. And then, we bump that up against data from your more classically engineering systems, code repositories, repos, things like GitHub and GitLab and Bit Bucket, as well as project management tools like JIRA. And, we put that all in one big platform.

Joe Levy: The idea is to say, "Hey, if Sally didn't commit a lot during the last spring, let's assume it's not because Sally's lazy and doesn't care. The more likely reason is Sally had four customer fire drills, she's super burnt out because she's been working 12 hour days since having to break in the middle of the day to deal with her kids' tech support with school from home. And, she's also had zillions of unanswered questions because she's trying to do everything over Slack, and her teams are also busy and distracted so she's got some blocked issues. That's why Sally's stuck." It brings you back to the studying the git commits, that is the output but it's not the reason, it's not the why.

Joe Levy: Our objective is to shine light on these areas that are the true why, give that data to Sally and to her manager, so they can have a data-oriented conversation on what could be frustrating her, how to improve, how to get that team to a really healthy condition. That's Uplevel.

Joe Levy: The other piece you asked me is what I do. I'm CEO, founded the business two and a half years ago. I spend most of my day talking to customers about this problem, learn a lot about how people are trying to solve it, how people are trying to assess it. I have not met an executive yet whose not deeply concerned about it, and yet has very little understanding of how to deal with it. Again, they all know how much JIRA burn down they have, or how much committing they're doing. What they're worried about is massive burnout on their teams, the survey shows this, our own data shows this, when I talk to other engineering executives I hear it. And yet, there's a big void in giving the organization data to help with that.

Joe Levy: Today, they usually wait for the HR engagement survey. Or we've seen, actually, quite a few companies that spin up some data analyst to try to figure this out by pulling Zoom logs or other things in a very one-off way. We have come in with a more scalable platform to that answer. So engineering leaders I talk to believe this is a real growing problem but it's a real blind spot in their org. And again, COVID's just added 25 degrees of heat to the situation.

Shane Hastie: Joe, that was really interesting, and I think some pretty useful advice for people in there. If our listeners want to continue the conversation, where do they find you?

Joe Levy: Our URL is, U-P-L-E-V-E-L And, there's obviously the contact us. I'm guessing we could put a link in your show notes, or something useful like that as well. Yeah, look forward to having a conversation with everyone. Obviously, we talk from developers to executives all the time. I think the classification of folks that we talk to most, that we help with the most, are frontline engineering managers, often people who are in that role a little newer in the tenure of their career. We'd love to talk to anyone out there listening, who has more concerns or considerations in this space.

Shane Hastie: Thank you so much.

Joe Levy: Hey, thank you Shane. Have a great day, and hope you're enjoying the summer down there.

More about our podcasts

You can keep up-to-date with the podcasts via our RSS Feed, and they are available via SoundCloud, Apple Podcasts, Spotify, Overcast and the Google Podcast. From this page you also have access to our recorded show notes. They all have clickable links that will take you directly to that part of the audio.

Previous podcasts

Rate this Article