Transcript
Stéphane Di Cesare: We're going to talk a bit about product thinking applied for engineers. I'm going to start with an anecdote I had a few years ago. I was working at that time for a consulting company and we had people, had customers, some were doing development, some were doing more operations. At that company we had performance reviews where you talk about the performance of everybody who is in the department. We realized that for the developers it's pretty easy to say that this year I've built this, this, and that, and it's reached that many customers and so on. For operations people it was like, it worked, nothing broke. We realized how much more difficult it is for operations people to show which value they're bringing compared to people who are building something. This was the inspiration for today to show to engineers how to better work with value.
My name is Stéphane Di Cesare. I work for the DKB, so the Deutsche Kreditbank. This is one of the largest online banks in Germany with about 5 million customers. I have an engineering background. I work in the platform experience team. We have an infrastructure platform, and in the experience team I'm working with the different stakeholders or the developers and also people like finance, business, all kinds of stakeholders.
Cat Morris: I'm Cat Morris. I'm Staff Product Manager at Syntasso, but originally, I was a failed engineer. I wasn't very good at it, but moved into product management pretty soon afterwards but never really left the engineering domain. I worked on internal tooling, MTL engines, things like that, before moving to Thoughtworks where I worked in their enterprise modernization platforms and cloud team. All about cloud migration, how do we do platform engineering. I specialized in product management and Platform as a Product as part of that. Now at Syntasso I help build a platform product, so help other people build their platforms.
Outline
We've got some key things that we think are quite important around product thinking. These are four areas we're going to talk about. We're going to talk about the what's and why's of product thinking for engineers. It's not just about making sure that you do well in your performance review, there's lots of other benefits. Stéphane's going to tell you a little bit about that. You're going to hear about some of my big failures around identifying problems before solutions, because I wasn't very good at that for a long time. Then talk about choosing the right metrics. A lot of talks have spoken about metrics and how do you measure things, and we're going to dig into what areas of those are specific for more developer focused tools, or if you are a developer on a team how can you think about the right metrics. Before telling you exactly what you can do, not every team has the luxury of having a product manager to define these things for you. How can you as a cloud-native engineer, someone building cloud tools, actually think about products a little better?
The What and Why of Product Thinking for Engineers
Stéphane Di Cesare: If you look back in time, so I started working with IT in the '90s, and there it was much more about what is called now workplace, about setting up computers, so hard infrastructure, nothing virtualized. IT was considered by many companies like a cost center. It's the people who install the printers, the value is in the end you can print. It doesn't change very much if you have the best printer or not. The most important thing is usually the cost. Now IT has evolved into being a vector for value for everywhere in the company, but still we see that for many companies it's still considered a cost center, and it's managed that way. The goal is IT should be the cheapest possible, and this is very often the way as engineers we think about value as well.
Then, also, as engineers we have a tendency sometimes to focus a bit more on the solution, to look at what are the cool things we can build. This is a very cool thing, for example. You can manage your Kubernetes clusters with Excel so you can just type the number of pods you want and it will update in real time, it's quite fun. Sometimes it's also important to think about, will the user see some value in that? I actually have to say I can think about users who see value in this, but then you have to work with the user to maybe look at, in that case, what is the best solution for the problem they're really having. What is important is to first show the value of your work, so what I was talking about with the performance review. How can you show that you're bringing something for the company? Focus on what the outcome of what you're doing is, so not the output. Not only how much work you do but what does it bring for the people who are using this work.
Here in Germany, we have to use the word outcome and output, because in German also in French where I come from, it's the same word, it's quite interesting. For some people it's a bit difficult to get the difference, but basically the outcome is what makes an impact for the people who are using your work. All this work with operations and also coordination work, so communication, making sure that people understand themselves well. It's only visible when it doesn't work. When it works well, when the communication is very good in the company, nobody is going to say that we have an excellent communication system. You only see it when people don't understand each other, then the scrum masters will be called in meetings and so on.
There are many ways to talk about product thinking. There are some sources here. In a nutshell what's important is to focus on the user value, both the value and the value for the user, which means as well that you have to understand who your user really is. Then you have to think about not jump into the solution but looking at which problems do your users actually have. As I said as well, so outcome before output. That's another big principle of product thinking that you think about not how much you do but what it brings. Also thinking about product before projects. The difference is project is something that has an end in time. Very often you have to think about the lifecycle that when you build something, automation, for example, it's not finished when you've coded it. You have to think about the lifecycle. How will it be maintained? How will it work in the future? It's a bit tricky especially when you're doing small coding activities like scripts, things like that. It all looks very simple. It's one of things. It's a few lines of code, but with time it becomes very difficult to understand what was this code really doing. It's really important to keep track of everything you've built. What was the purpose? Why are you doing it? How does it work, so that it can be maintained in the future?
We're going to go a bit deeper into some of these aspects. One is the user value. As I was saying, it's important to understand who your user is. When I think about the work I'm doing at the moment for an infrastructure platform, the obvious one, what everybody will say is the platform users are the developers. They are basically the builders, people who build something. It's only part of the picture, because the platform has other values for other people. You will have builders who are developers in product teams, in application teams outside your platform, but also internal ones. It's important to keep them in mind as well. People who are enablers who will help other people to use the platform, which will help you to grow. To talk with people who are not users yet but who will become users later. We're in a bank, so we of course have regulatory. We have regulations.
For us it's an important kind of user because we have regular audits. It's very important for us that the platform helps the audits to go well. The more we can save time for the auditors, the less time they will need to spend with people building the platform. If someone is a developer in the bank, I think the last thing you want to do is to talk to auditors. It's always good if the auditors can understand as much as possible on their own. You also have people I would call viewers, that are people who are using the platform for visibility, so people like finance. They want to know which application costs how much. This is something that the platform can help them with. That's an important use case to keep in mind. It's always a good idea to have finance on your side, first of all. It's an important impact also for management, for the business to use a platform to be able to understand better who is doing what. What are the costs of each application? Where do we have value? Where can we save money, and so on.
Then the value part. I think basically there are three aspects. The main one, which is the one that is usually talked about when people talk about IDPs, is efficiency. The developers work better, work more efficiently. There is also value in visibility. It's something that's a bit difficult to put a number on but that's important as well, because it enables decisionmakers to take better decisions because they have better data. If we were in our environment, so not that long ago we were a bit before DevOps. We were in a situation where developers code something and then it's being sent to operations. Then there is not a lot of communication.
In an environment like that, if finance goes to an application team and tell them, how much does your application cost to operate? I have no idea. Operation knows. Then you ask operations. They will tell, we have lots of stuff running on our infrastructure, we don't know what is which application. This is a big problem when you want to take decisions. That's where visibility is important, because you know what's going on. As a bank, avoiding risk is also important. That's important for everybody. As a bank we are of course more aware of it. It's a bit easy. You can put a number on. If you can think about what is the cost and the probability of something bad happening, this is something you can use to try to put a number on risks.
Identifying Problems Before Solutions
Cat Morris: This is a fun one for me, because the question I always ask when we think about problems before solution — everyone thinks they do it — but how many teams actually do it? I know a lot of my temptation is as soon as something's on fire and it lands on my desk, that's the first thing I want to deal with because it's right in front of me and it feels very urgent. That's not always the case. There are some patterns that I see quite commonly around problems before solution, and people may be not doing it when they could. One is, like I said, you get something that's burning, something that's really important that someone tells you you need to do immediately. That's the first thing you want to look at. Actually, maybe you're just thinking of solution mode of I need to solve this problem right now, rather than what are the problems we could solve.
The second one is the HIPPO effect which is the Highest Paid Person's Opinion. Someone who is particularly loud or particularly influential in the organization, or maybe your boss's boss's boss say that we need to fix this thing immediately. You want to fix that. You're thinking in solution mode again. The final one which is similar but I've seen a lot happen and I'll tell you a painful story around this was, if you think about, what are the things that I know how to solve? You've got all these problems that you've identified. You pick one that you know how to solve and solve it exactly the same way that you've always solved your previous problems. Then you're thinking about the solutions rather than thinking about the problem space a little bit. Like I said, you need to figure out what is the problem that is most important for you to solve as an organization before you go into, how do I solve this thing?
The second one which is also hard and people sometimes forget is deeply understanding the problem before you try and solve it so that you can figure out the best way of solving it. Coming back to some of the things that Stéphane said around user value. How do you understand the user problems or the user experience without actually understanding that problem itself? There are some ways you can do this. This is a classic thing that we used to trot out when I was at Thoughtworks all the time, it's called the double diamond. Lots of other product thinkers use it so it's not just a consultancy thing. It talks about the two different spaces. Often, if you're in engineering it's very tempting to go straight into the solution space. This is the classic build, measure, learn loop. What am I going to brainstorm? How do I build it? What am I going to build? Iterate over that to get more value. When actually we could be focusing a little bit more on this problem space so that we figure out what is the thing we want to solve first before actually trying to solve it. That's where you want to explore things and define things. I'll go into each of those areas a little bit more later.
First, I want to tell you a little quick story. My first official platform role was a long time ago now, and I was parachuted into a team as the platform product owner because the previous platform product owner had gone on paternity leave and quit to be a stay-at-home dad. I felt massively underprepared for this role, and looked at my team. I'd worked on platform adjacent things before and internal tooling but it'd never been called this. I was like, what do we need to do? We've got this budget, this team has been formed, what are we going to do? I looked at the SMEs and the team, and they said, "All of our pipelines are on Jenkins. They've been on Jenkins for 10 years. We need to fix this because Jenkins is so old. We want to move to Azure DevOps". I was like, "You're the SME. You know what you're doing. Let's figure out how to do this". We spent about three months, maybe a bit more, migrating all of the build pipelines from Jenkins to Azure DevOps because no one in the team knew Azure DevOps but they knew Jenkins really well. The translation and the migration effort was really hard originally.
By the time we got into Azure DevOps we realized we still don't know Azure DevOps, so maintaining it was really painful. The operational costs were really high at the same time. Actually, there was no user value. There was literally zero impact to the business because it was just as hard to maintain these things. The users had the exact same experience. They clicked a button. They got their builds. They got their database. Why did we bother? We've made negligible improvements to the business. Because we didn't explore the problem space properly, I just said, what should we do?
What is exploring the problem space? What should I have done? The goals of this phase is to really identify your top customer pains. Think of your users, what are they finding hard? What do they need to do instead? What can't they do today that they really want to that will add a lot of business value? We do that by building quite a lot of customer empathy and understanding. Customer empathy is our ability to sit in our customers and our users' shoes, feel their pains, feel the thing that they're experiencing day to day. The final one that often gets missed and I think is really important, is investigating key business aims. I'm probably a little bit too financially aware product manager. There is no point in you building something that your users love and think is great if it doesn't actually make money for the business or save you money or do anything that means your company is going to continue funding your team. Because you're internal, like Stéphane said right at the beginning, IT is often seen as a cost center. You need to justify your existence.
If you aren't thinking about the key business aims and what your business is trying to achieve, you're still going to fail. How do we do this? We do customer and stakeholder interviews. These are pretty simple. You find a key customer group that's really chatty, and I promise you they will exist. Engineers are some of the chattiest folks when you're talking about things that are going wrong. Sit and talk to them. Not just those engineers that you might be serving, also, like I said, key business aims. Speak to your financial teams. Speak to the other business stakeholders you've got, the CTO, the management in your organization. What are they trying to do? Data and process analysis. It's not all touchy-feely human stuff. Look at your pipelines. Where are the pain points? Where are people getting stuck? If you're measuring the DORA metrics in your organization, these are great because you can see which one am I an elite performer at, which one am I not so much an elite performer at? Where do I sit on the spectrum? Which of those should I look into? I don't think they use that categorization anymore in the latest report, but something similar. The final one is shadowing. I'll talk a little bit more about that later. That's basically where you sit behind them and just watch your users. What are they doing?
Back to my sad pipelines. What would this have looked like if I spent a bit more time actually exploring the problem space? I would have realized that onboarding was taking a really long time for new users to the platform. This was an insurance company. We were building a platform for it, but our core infrastructure was really important for making sure that you could onboard a new customer in their own segregated environment. That took a really long time. That really holds up the business.
Another thing that happened is, in the UK, Martin Lewis, who's like a financial guy, brings out all of these deals for people. Every time they introduced a new deal, the web traffic of this insurance company, because it'd be like, you can get cheap car insurance, would go massively high and massively ramp up. They had a back channel to this guy to figure out when it was going to happen so that they could scale everything up and scale it down again manually. What if we'd automated some of that behavior? Because that wasn't the only time when they were going to get a spike. There were other times that we might get a spike in traffic too.
Finally, a lot of it was very outdated that we were running on, a lot of this infrastructure, it was typically like on-prem VMs that were being used. These were going to go out of support. That's a very high risk to the business because we need all of our services to be supportable, to be audit compliant. What if we'd have looked into that instead? There were lots of different options. It wasn't just migrating the platform pipelines from Jenkins to ADO. There were lots of other areas that a platform could have helped with in this business.
They were the possible problems that we could have solved, things that we thought were important. Once we've figured out what those things are, how do we define this problem space and figure out which is the right one for us to solve now? The first one is identifying opportunities, which I've just shared. Some potential opportunities that we had in my team. We can prioritize those specific problems within those opportunities because opportunities are still quite big. Onboarding customers, that's like loads. Could be anything. How do we narrow that down into the specific part that we want to solve? Really get into that iterative, small, move fast thinking.
We need to gather insights and data, so all of that work that we did in exploring the problem space. How do we figure out which problem do we want to solve? We need to turn that into insights so that we can measure that improvement in the future. Some techniques. We've got value stream mapping or jobs to be done. This is a similar thing where you look at your users and think, how do they go from A to completing the job they want to do? Like from just deciding they want to do a thing. A great example here is like CI/CD pipelines. I want to get something into production. What does that look like? You can map out the steps that a user has to take. You can also map out the systems that they have to go through and figure out where the pain points are and where that is slow.
For prioritization, you've got a bunch of options there. We've got RICE, which is a framework for defining the prioritization based on reach. How many people does it impact? How much value will it generate? Cost. How much is it actually going to cost us to do? Effort. How hard is it? How long is it going to take? That's quite complicated, probably hard if you're internal because it requires quite a lot of data. Much more useful in a B2C context, so if you've got lots of customers or consumers. I prefer value versus effort. You just take all of the problems that you've got and map them against each other. Which one's going to generate the most value for the team and our customers? Which one's going to be the easiest to do? Pick the ones with high value, low effort. Or, which I'll talk about in a bit more detail, the Opportunity Solution Tree, which is a model that was created by Teresa Torres in her book, "Continuous Discovery Habits". It's really good. She's crazy smart and also has a lot of stuff of how to do product management with AI. She comes up with this concept, and I'll dig into that a little bit more later as well. Like I said, you're gathering insights and data from your exploration phase. Do some analysis on it.
Back to my sad pipelines again. What my team's goal, what our mission actually was from our leadership, which I found out later, was we really wanted to reduce that engineering toil by at least 50%. Because it was taking a really long time for engineers to get stuff done. Instead of migrating the pipelines as like, this is the thing we want to do. If we'd have thought about, we want to reduce our engineering toil by 50%, we'd have probably found that there were lots of different things we could have done. What about if we mapped that to the onboarding of new services, to our autoscaling that we wanted to achieve, to upgrading that infrastructure that was very high risk. These were all things that we thought might have achieved that goal.
Realistically, onboarding new services was really crucial to reducing that engineering toil. Because we had a lot of manual work to do it, not just from our platform team, but also from those application teams that needed to onboard those new services and onboard our new customers. Each time we had a new customer come on, every single team had to manually deploy things and manually look after stuff. If we'd have focused on that one, we could have made a significant improvement to that reducing the engineering toil. This is where the Opportunity Solution Tree comes in. I've retrofitted this and gone backwards. The way an Opportunity Solution Tree works is you start with the high-level problem that you want to start with and want to solve. In our case, it was, shipping valuable software is taking too long.
Then you make some bets, so some goals. How might I achieve this? We can ship more software by reducing the engineering toil by 50%, because engineers can write more valuable code if they're not manually doing rubbish. Other goals are available. Then once you've thought about the goal that you want to achieve, then you can think about how you want to solve it. It's really forcing you to step back and explore that problem space to achieve a goal that makes sense for the business before diving right into solutions. Like I said, exploring the problem space and defining the problem space leads to product discovery. Now you all know how to do product discovery in 15 minutes.
Choosing the Right Metrics
No point in doing the discovery unless you know you're building the right thing and it's made an improvement, so on to metrics.
Stéphane Di Cesare: Who here is working as an engineer at the moment? Who likes working with metrics? Typically, this is a topic that engineers have a tendency to flee somewhere else when we start talking about metrics. One importance of metrics is you will always have someone in your company who wants to measure how good is the work you're doing. One good reason to be involved is that this is better to do when you know what your job is, rather than if someone very far away who has read in the book about what an engineer is doing in his business management book. You don't want people like that to decide on the metrics on their own. It's better to be involved. What I want to talk about today is the product metrics. Product metrics means, how can we look at the outcome the product is delivering. Not once again something quantitative which is only showing how many Jira tickets you're doing or something like that, but really looking at, which benefit do you want your customer to achieve, and how can you look at how to measure it? An important user for when we're talking about platform are the developers. There's been a few frameworks to look at how effective the developers are. That's an important point.
As a platform you want to improve how effective your developers are. One of them is the DevEx framework which is quite a good starting point. In the DevEx framework you have basically three dimensions which are important for developer productivity. One is being in flow, so not being interrupted, being able to focus on what you're doing without other things disturbing you. One is to decrease the cognitive load, so not needing to know about many different topics. Since DevOps came in, this is becoming a bigger problem because if you want to do, you build it, you run it, then it means that as a developer you must know a lot more about security than before. You must know about infrastructure, about scalability, in the bank, about compliance. I've seen companies in the past as a consultant where they had all developer teams managing their own pipeline.
Then you need the developers to be Jenkins experts, for example, and you probably don't want to go that far. The last one is, as a developer you want to have good feedback loops, so when you build something you want to be able to see the result quickly and not having to wait for months before the software is installed and you can actually see what happens. Then it's also always important to think about, for your users, what is really important, which is the, why product discovery, as Cat described is a key thing. This is the general framework, the actual problems that your developers are having. They might be different. You might be in an environment where, for example, as we are there is a lot of compliance, so we have a lot more work than a typical company so that developers don't spend all their time on this and you have to focus on what your actual problem is.
There are some examples of possible metrics. In the DevEx framework, they divide it in two kinds of metrics, so the perceptions and the workflows. The workflows, they are more quantitative metrics, things like build time, things like how many steps you have in the deployment. There are perceptions, which are things that are more difficult to put a number on, but they are still important. What's important when you work with metrics is you have some aspects you will be able to easily put in a number and some not. It doesn't mean the aspects you cannot put in a number are not important. It's important to keep that in mind. Sometimes you have to evaluate, you will have to ask for customer satisfaction. It's not an exact science, but it's still important to follow that up, even if the numbers are not always exact. There was another interesting model, which actually came from Syntasso, which is looking specifically on platform engineering and looking at four aspects, the speed, safety, efficiency, and scalability. I think that these metrics are interesting as well. They might be a bit difficult to measure.
For example, this mean time to add a service in the platform, I think if you have a really big platform, which has a lot of services, it's ok. If you're adding a service every three months, it will be difficult to have good numbers. You have to think as well about, when does it start to add a service in the platform? Is it when the engineers start to do something? Is it when management has an idea? At least this is also important to make it clear which direction you want the platform to go, so that the people who are building your platform understand what we want to improve is this. Even if you cannot measure it exactly, it will be clear for them what is it that we want to make better in the platform. There are other frameworks as well. DORA was really the first one which was in this area, comes from this "Accelerate" book. I think it's the first book which really went into what is the value of DevOps and of trying to improve the speed of deployment, speed of feedback. One thing I recommend is Octopus Deploy, which is a British CI/CD company. They have very good documentation about the kinds of metrics for DevOps and how do they fit which use case.
Bringing Product Thinking to Your Role
Then we're going to talk about bringing product thinking to your role. Then you're probably all thinking, I'm not a product manager, I'm just an engineer, so what can I do with it?
Cat Morris: You can do a lot. I had someone talk to me recently and they were like, "You're a platform product manager. I don't meet many of those". I was like, "Yes, still a rare role". If you're on a cloud-native team or you're building tools for internal teams, developers, data scientists, there's still a lot you can do, even if it's not your formal job. This is all about like, how do I build customer empathy to make sure I'm doing the right thing? I said I'd come back to shadowing. This is a technique where you just sit there with the users and watch them and watch what they're doing. Generally, you'll go in and ask them to do a job. Maybe it's like, could you show me how you upgrade one of your services? Or, can you show me how you would request support if something's gone down? Or, how would you debug your service? Then you sit there and you watch.
As you watch, you will learn all kinds of things about how they use your tools and your systems. You might find that they redo the same step five times because they don't trust the data. You might find that it takes them a really long time to wait for things. They just sit there and don't do anything else with that time, they just wait because they're not sure whether it's going to succeed or fail, whatever job they want to do. You'll learn a lot about your customers and what their pains are and how they interact with their systems, and what kind of interface or user experience really adds a lot of value for them. Shadowing is one of my top recommendations if you can do it. Anyone in your role can do it. The best thing about working on a team that builds software for internal users is you have unprecedented access to these people. One of the things you find if you're working with customers, so actual paying customers, is they've got their own jobs. They're doing other things. They don't want to talk to you. They've got stuff they want to do. Internally, people always complain because they know they can get it fixed. You won't have any problems finding the people to talk to. Make the most of that fact that you have this access that you might not do otherwise.
Stéphane Di Cesare: It's really important as an engineer as well because it's not something that's natural to do as an engineer. It helps you as well understand, how are your users thinking? When you come with a solution, is it a solution that your users will actually understand because they have a different background than what you have. What you think is simple might not be simple for them. Every time I do shadowing, I always discover new things. You might have users who find workarounds to things that are not working well in your platform. Sometimes you get even inspiration.
Cat Morris: The big thing of this actually, is make sure you don't tell them what they're doing is wrong. Don't say like, no, you just click this button. If you can help them, great. Take that as feedback for you to improve your tooling, not to dunk on your users.
Stéphane Di Cesare: It's not technical support. You have to be in the background. You don't necessarily need to use the plant, I think. It's better if they know you're here. It's a bit difficult sometimes as an engineer to go into technical support mode, here is how you can do it better.
Cat Morris: Just watch. The next thing is ask why. I want you all to become the annoying toddler in the room that asks why way too many times as everything's happening. If someone tells you, "I want you to build this, I want you to fix this". "Why? What are you trying to do? Why should I fix that? Why are you trying to do that? Why aren't you using another tool?" Ask all the why's, not just to your users and customers, but also, if you do have a product manager, ask the product manager or product owner, why is this the most important thing to do? Why is this the top priority? Why are we solving this? Get really to the root cause of that customer pain that you're trying to solve, because then you'll be able to solve that problem better and more efficiently. You'll have more empathy for the user.
The next one, and this one I think is a little bit controversial because when I worked my first job, I became like the conduit for business updates. Part of my job was sit there on the quarterly calls for an hour, take notes to fill everyone in on what happens in the company. You should read your business updates. I know it's not the most fun job in the world, but this is going to give you really good context into what your business is trying to do. Is it trying to reduce operational costs? In which case you need to make sure that you're adding value. Is it trying to improve the speeds of delivery? Are you trying to invest in a brand-new region? That's really good information if you're building internal tooling because then you're thinking about different regions that you might need to deploy software into. Read those business updates, understand your business and what it's trying to do. This could be like the newsletters that come around. It could be the quarterly calls that leadership do.
At the very least, there will be some annual earnings call. Read them, look into them. Very useful. Then, finally, implement instrumentation. These are really helpful for proving the value of what you're doing. It could be real time. It could be that you build a monthly report that you export around what you're doing and the improvements that you're making in an organization. What value you're delivering. How much faster are people moving? How much cheaper are your services? These are things that are going to really make it much more easy for people to understand what value you're delivering. Everyone loves a good dashboard. Also works at a job level.
Stéphane Di Cesare: These are things that you can use personally as well, as an engineer for your career. You can think about, who are your users? There is probably your manager. Part of your job is always to help your manager reach his goals. You have to ask, what are your goals? When do you get a bonus? Things like that. Also think about more long-term, is there a specific position you are aiming at? What do you need for that? Then it's important to think also that way. What are the goals I want to reach and discover for the people who are controlling this? What are their goals? How can I help them reach their goals?
Cat Morris: I set my one performance review cycle. I set my goals for the year as OKRs, Objectives and Key Results. Basically, I set myself targets, and because I hit them, I could be like, "We agreed this was promotion criteria, can I have my promotion now?" It worked. I can recommend it for you. You can use some of these techniques in your own personal life as well, which is quite cool.
Summary
Product thinking, if you are a cloud-native engineer, maybe building tools for other developers, or data, internal tools. Think about user value. What is your user actually getting out of this thing that you're building? Think about the problems that you're trying to solve before you try and solve them. That's really linked with that user value. Choose your outcomes over outputs. What am I trying to achieve, rather than, I've delivered this many tickets? Consider that lifecycle. Your operational costs are part of your costs as well. You're not just building once and done. Explore, define problems to maximize that user and business value. Go into really thinking about what is that problem that I've thought about. Finally, measure your metrics to show your results. Because you'll earn the big bucks. You'll get the promotion. You'll save money. Your boss will love you. It makes it much easier to justify what you're doing.
Questions and Answers
Participant 1: Sometimes I've run into the problem that in the beginning when you're building out platform metrics, you maybe choose the wrong metrics or somebody from C-level enforces a metric that maybe works or doesn't work for the team. Do you have any experience in adapting those analytics or metrics over time? How do you then sell that to the business case if they become obsessed with a particular metric?
Cat Morris: I've had that a lot. Mine is generally pick one that you can measure that's close. Something fairly simple, the big risk that always happens. Thing I always do is you're like, this is the perfect metric. Let's pick this one, and use the perfect metric. Leadership love it. As soon as you try and measure it, nothing changes, or it's impossible to actually measure because your systems and services aren't built this way. Then finding something you can measure and showing that improvement has actually led to some quite interesting conversations with leadership. They'll generally go like, cool, but how much money are you saving me? Then you can dig into it a little bit more, "Actually, we've reduced these cloud costs by this much". That's a really easy one to show if you're internal. Pick something you can measure first and foremost.
Stéphane Di Cesare: I think you might have to do it in parallel as well because it usually doesn't work very well to say, we need a different metric. You can start measuring what they want and you start looking at the ones that really make sense for the product on the side. Then when you actually have some data, it becomes easier.
Participant 2: Outcome metrics are very hard. You might set up like a desired outcome and then the things that you do might move the needle, but other things move the needle in the other direction, and normally the shifts also take time. How do you really implement that tying, I did this and as a result this cost is outcome, and I did it in a timeframe that will affect my performance evaluation or whatever else. How do you factor all that in? What other things can you layer on top? Because the output metrics are so much easier. I make a decision to do this and I deliver it.
Stéphane Di Cesare: I think that's why it's important to think about the outcomes as early as possible. We've had the issue as well. For example, we automated our pipeline when we didn't take any numbers before. It's improved a lot the developer performance, but we had no hard data to prove it. We had to instead do customer satisfaction surveys. The developers say that they like the pipeline a lot, and then you can go back into more quantitative metrics. The clearer you are on which outcome you want to reach, the easier it gets.
Cat Morris: There's a big common thing around A/B testing, which is like, try one thing with one group and one thing with a different group. I find that doesn't really work at all with internal tooling because you just don't have the numbers, which is something you also have to consider. You have a finite set of users. Even if you are the biggest organization in the world, you're still only going to have a finite set of developers, but the majority of platform teams are going to be serving a much smaller group. Some of those techniques won't work, but really, it's about how you think of things on the whole. You might not be able to tie back to a direct thing, but you should be able to have a log of this is the work we've done and here's the change we've seen. We can't tie them directly, but this is the direction of travel that we're going in and these are the bets we're making. You might find that it doesn't make that improvement and it is different, but that's still good data and you can inspect a little bit closer.
Participant 3: My question is also about the metrics, but this time from the other way around, in terms of convincing. In Germany, we have lots of regulation with the works council and everything. How do you convince your works council that you should be measuring these metrics, and it doesn't have the criteria of you're tracking your employees doing whatever they are doing, because that could be against some regulations.
Stéphane Di Cesare: This is why it's important to think about product outcomes, to think at the product level, because you're right, especially in Germany, if you go into metrics that go too close from managing, looking at performance, you will have to talk to the works council and you don't want to do that. What's important at the end of the day is what the product is doing, not what individual employees are doing.
Cat Morris: What's the behavior of the overall platform or the tools that you're delivering? What's the usage of those tools? How fast is it delivering the services to people, not the individual person taking that amount of time?
Participant 3: Does that mean you track it on the product level and not on the team level?
Cat Morris: Yes.
Participant 4: For the engineering team working on infrastructure changes, it's often not very valuable in terms of the product perspective. It does not give you the money value, the dollar value. How do you sell that to the product team to be one of the more important things to do? Because it has intangible values versus their product own features that will generate X amount of revenue. The X amount of revenue always wins with the intangible. It's always a problem. What are some of the pointers that we should look at?
Cat Morris: I have never met a product team that doesn't want to do things faster or do them cheaper. If you can tie the value that you're delivering at an infrastructure platform level to, if you upgrade here, it'll take you two days. It means that every time you request a new service, it'll take you five minutes as opposed to waiting for us to do it in a week. They're obsessed with you. They love it. The other thing that I've found really helpful is you can donate people in your infrastructure team to help support them. I was having a chat at the Birds of a Feather session, and they found that the teams that didn't do the upgrade, some did, some didn't, those that didn't, they would just hand over like, here's someone for a week. They're going to help you do this upgrade to make sure that you're compliant, if that's the rule. Another thing is tying those to the business goals. If your business says we need to be 100% compliant and you've got one team that isn't because they won't upgrade, then you can escalate that.
Stéphane Di Cesare: I think that's the important point I would like to say as well, the business goal. I think like, for example, compliance is usually important so that you can show that you have less risk because the platform is bringing some level of guaranteed compliance. It reduces risk, and also the visibility that you can have a better understanding of what happens with your IT resources. I think in our case, that was the important part.
Participant 5: I have to ask about metrics as well. Would it be fair to say that monitoring the outcomes and monitoring what customers do is easy if you work on platforms because your customers are internal, but hard to impossible if you just work on the business application and your customers are end users. If you add a widget to the product that customers buy, you have no idea whether adding that widget caused more customers to sign up or not to join.
Cat Morris: I think I've heard the opposite. It's very easy to measure stuff with end users if you're doing like a B2C type product because you can put that instrumentation very easily into whatever interface you've got and track it. That's much more simple than the platform where you might have limited numbers. You tend to deliver the outputs, rather than the outcomes of like, what do people gain? Measuring what people gain in internal systems is very hard.
See more presentations with transcripts
