Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage Presentations The Good, the Bad, the Ugly: Making Teams Perform Better

The Good, the Bad, the Ugly: Making Teams Perform Better



Victoria Puscas talks about how a team of engineers and data scientists worked together for a year and became high performing by embracing change, improving day to day practices, breaking the walls between disciplines and overcoming many communication (and other!) challenges together.


Victoria Puscas is an Engineering Manager who has been working at Deliveroo for the past 3 years. The teams she looks after are building and scaling some of the most sophisticated machine learning systems the company has: solving problems such as dispatching orders, route optimization, forecast predictions, pricing and cost optimisations.

About the conference

Software is changing the world. QCon empowers software development by facilitating the spread of knowledge and innovation in the developer community. A practitioner-driven conference, QCon is designed for technical team leads, architects, engineering directors, and project managers who influence innovation in their teams.


Puscas: My name is Victoria. I'm an engineering manager at Deliveroo. Deliveroo is a food tech company. We feed hungry customers around the world in 13 markets. If you think that it is your PM's job to know what stakeholders want or what customers need, you're wrong. If you think that as a software engineer, your sole responsibility is to write software according to specifications and everything else is faff, you're wrong. If you think that it is someone else's job, perhaps a manager's job to make your team great, you're wrong. For that reason, stick with me for the next about 30 minutes and hear me talking about a case study about a group of engineers and data scientists that worked together for about a year and became a high performing team by embracing change, improving their daily practices, and overcoming a lot of communication challenges together.

Presenting the Case Context

I joined the company as a software engineer. A year later, I transitioned into a new role, and I became an engineering manager. I had a very smooth transition into my role. I was lucky. I still call my first team a happy bubble. Disappointingly, this talk is not about happy bubbles. It is about the next team I took on. The team was a smart bunch of well-seasoned engineers and data scientists. We had a product manager. The team also shared their engineering manager and their data science manager with a few other teams, which is why I was very excited for the opportunity to step in and dedicate my full time to that team. Also, there was a perception about that team that it was a bit sluggish, lacking drive, and somewhat headstrong. That team had a track record of using machine learning to build their product. I will refer to machine learning as models or algorithms. The team has just finished building and releasing a bunch of very sophisticated models to production and they were ready to pivot. In order to help the company with one of the most anticipated objectives for the year, which was profitable, sustainable growth through new pricing, quickly.

For customers that ordered food, we used to charge a flat fee as a delivery fee. If Al, as the customer, wanted to order a chicken wing from a shop downstairs, she had to pay £2.50 in delivery fee. If she wanted to order 15 pizzas for her birthday party or something, from her favorite Italian restaurant which was 5 miles away, we would charge her the same amount as delivery fee, £2.50, and an equivalent amount in all countries we operated in. This is suboptimal from many points of view. What we had to do is to find a way to charge customers a variable fee, but make you work for the company. On the other hand, every order is delivered to you by a rider. We used to have a very simple formula to calculate the price which we would offer on a per order basis to our riders. If Bob, the rider, was asked to deliver the chicken wing from downstairs and he was nearby, we would pay him a certain amount. If we ask him to deliver the 15 pizzas, he had to cycle 5 miles away, get the pizzas on his bag, so back perhaps on a hill. We would pay him a similar amount. You can clearly see how Bob might be pissed off. We had to find a way to pay our riders in a smarter, transparent, and sustainable way. Why was this all very important for the company? We operate in a hugely complicated market where we have a lot of competition. If we don't adapt quickly to the ever changing environment, as a company we're dead. This is how that team came to be the pricing team.

When I joined the team, I allowed myself some time to observe. I wanted to see a few things. First of all, I wanted to see, what is it that we're building? We had to come up with some brand new models, perhaps very smart ones. I wanted to see what was the ideation session like? How did we go about thinking as a team about the products we're going to build? I wanted to see how the team works as an entity. I wanted to see how engineers and data scientists work together. I wanted to see how the team interacts with the outside world, with other teams, with stakeholders. Ultimately, I wanted to know my reports. I didn't work with them before that, so I had to know them. I wanted to see, what were their contributions towards our project? Ultimately, how their individual work rolled up into helping the company grow.

The Wind of Change

I thought that we had to pick up momentum. Because we wanted to deliver things pretty quickly, we had to change a few things on the way. I thought to myself, if I tell people that we all wanted to be successful and deliver those projects, and everything's going to be great, people will be on board. They were, except the change part. Nobody really pushed back. Nobody jumped up with joy either. I thought, I'll just start with little things and it'll be fine. Our team came up with this idea of using a random forest for one of our algorithms. I was as confused as some of you now, when I heard about a random forest. I went on to do some research and reading, and educate myself about it. Apparently, it is a fancy regression. I still didn't understand how that fits our project. If I didn't understand I was part of the team, with the team every day. If I didn't get what is it that my team is building, and how are they doing it? Can you imagine how hard it was for the rest of the business to understand it? How are we going to charge customers, according to what principles? How are we going to pay riders? How and will we comply with different legal regulations in all markets we operate in? Perhaps improving our product algorithm and engineering specifications was a great way to help with that, so that at any point in time, if anyone in the business wanted to know what is going on and how are we thinking about the new product, they could have a plain, easy answer.

At the team level, some common agile practices existed but they weren't very efficient. Standups were free form. They were totally optional for managers. I don't think any manager has ever showed up until I joined. Team planning existed. They were an hour, or an hour and a half long meetings about what color our Jira epic should be. People worked on stuff. The priority and importance of the stuff was not super clear. Even if an engineer and a data scientist worked together on stuff, the desired outcome was not clear. I thought, there is so many easy wins we can have. Let's just get it over the line and move on. At individual level, people, in principle, used to have one to ones, but they were used to a manager that was running after another four, five teams. Poor guy. We started to have a very regular schedule. Think about agenda and just getting into the habit of having a regular session with each other. A few changes in. A few ideas I still had in my mind about improvements. It was the time for the retro. I was totally unprepared to enter a 2-hour long retro to hear about how disruptive the changes were. How inconvenient and annoying it was to come to standup every day, as early as 10 a.m. Why would managers need to know all those details? How algorithms work? How services are going to change? Why do they care? Don't they need to go to meetings? Don't they have stakeholders to touch base with? I was not prepared. After that retro, I locked myself in a room and I cried for an hour.

When you're 95% done you're Still Halfway Through

A month in, not much progress on the projects at all. A serious slap to my manager's face got me thinking, how can I fix what I've started? How can we get on the project, finally? I started honing my relationships with my fellow team leaders, the product manager and the data science manager. The good news was that we were aligned about how we want to work together and what we wanted for the team. I still needed to know if things progressed according to our expectations or not, because when you want to debug your team, you need to know where the problem is. I had to know if the team had a problem to begin with. I needed to know what is it that we're doing and how are we doing it? I needed to know the details. I needed to understand them in order to do my job properly. For that, I treated the norms that I had as black boxes. I started by questioning, what is it that we're building? What is the components that we're working on? What are they going to do? What are their inputs and outputs? Things started to clarify. We had to spot complexities and challenge the team to cut down on the complexities because we wanted to move fast.

I clearly have not done a good job of explaining why I'm introducing change. I didn't articulate very well how important it was for the company to get those projects done, and done on time. I fixed that. As an additional help, I worked with my fellow team leaders to recognize and thank people that were adopting new ways of working and that weren't that resistant to change. Most importantly, that had constructive feedback about how we're working and about all the changes. The ugly news was that my reports still didn't understand what my responsibilities and accountabilities were. For a very good reason, I never told them. We started working on that. I was very explicit about what my role was. By far, the most complicated conversation I had was with my team lead. In the end, he asked me, why was I assigned to the team? Because everything was working fine. Why do you have a full-time manager? It was great before I joined. I had to be very transparent and honest with him. I did say that our senior leadership didn't have trust that our team could deliver those projects on time. It clicked in his head, why I cared. Why we had so many questions. Why we had to keep the business up to date.

A few weeks in, and the team picked up some steam. We had pretty good momentum. We were clear about what we're building. Small features started to see the light of production. We were finally ready to test our models for the first time. For the consumer pricing model, we decided to test it in the UK, our biggest market. Before releasing it, and before trying it on, we looked at the data. We wanted to make sure that our customers paid the same on average. The data looked absolutely fine. We only had a small percentage of outliers but we were not alarmed at all. Nobody cares about outliers. The big day, we released the model. We turned it on. Within less than a day, our stakeholders came back to us absolutely pissed off. Some customers had to pay as much as three times more than they used to. Our stakeholders absolutely did not buy our outliers thing. Where we saw a small percentage of data that was just not fitting the overall story, they saw customers that had to pay more than they used to, and that were never going to come back to us. The model was unacceptable.

On the riders' side, we were a lot more careful. We thought to find a tiny area in the UK where we could test our model. We found a really small area with a handful of riders. We turned the model on. After 1 hour, and about maybe 10 orders placed, riders went on strike. Our riders are very knowledgeable about areas where they typically work. They deliver, usually from the same restaurants, typically to the same customers. They're used to seeing certain numbers in their app. They used to be paid certain money, even if in that hour when the model was on, they earned slightly more than they typically would. We compared this data with the same riders a week ago, same day, same hour of day, everything was very equivalent. They earned slightly more, but they hated the change. They hated the change too. How ironic could that be? We had to turn it back off again.

The Light at the End of the Tunnel

The ugly news was that, clearly, we didn't know our stakeholders. More than that, remember, in the UK, we charged a few customers three times more. All other countries that knew about this experiment did not want to participate anymore, until we proved that our model was bulletproof. We didn't know our stakeholders. That was a very bad situation. We were lucky to be given a second chance. Our senior leadership trusted us to do it again. It was very clear to us that we won't have a third try. We had a very fixed deadline to deliver those changes by. We had to pull up our socks or we didn't have a team. The bad news was that the previous iteration, our first trial, highlighted that we had a lot of complexity in our processes and in our code bases. For random forest, anyone remembers? We had to move fast. We had a fixed deadline. We had to come up with a way to deliver those models according to the expectations and get them right. We had to be creative because we didn't have enough data scientists to work on those models in parallel. I looked at other teams in the company that perhaps had similar issues in the past, but none of them had the issue of not having enough key people in the team to deliver those projects. Hiring is an option, of course, but we will never have the luxury of time to onboard those people, and get the work done on time. We had to find a different way.

One crazy idea came up to my mind, what if we ask engineers to do some of the things that typically a data scientist would do? They were extremely smart people. They knew a lot of things already and they were willing to help their colleagues that were under a lot of stress. We gave it a go. We looked at some of the things that data scientists did. Some of the models that this team has produced in the past had a few components that were part of the process all the time. We had to come up with a dataset. It's a very typical component in teams that do use machine learning to build their products. Our models had a few steps like training, data extraction, things like that. We also had a pipeline that stitched all of those components together. When we took a closer look at things, we figured that the dataset is nothing else other than SQL. The model was Python. The steps for the model were Python as well. The pipeline was just scripting. The bulk of the work was coding. Engineers are brilliant at reading and writing code. They do that for profit. While we poked at the data science backlog, we realized that we can share the majority of the work, and we could allow our data scientists to do the things only a data scientist could do and share everything else.

The pace started to go up massively. We quickly picked up a lot of the things. The good news was that, a real bonus, and one amazing thing that happened when engineers started poking around the data science code bases was that they started applying best engineering practices to those code bases. They added tests. They added linting. They added a few other tools to help with speeding up the pipeline and stuff like that. A few other useful things started to emerge. Some guidelines on how to write better SQL, so that the next person doesn't pull up their hair. A guideline on how to modularize the data science code bases. There are code bases. It's code. You can apply the same principles to most of the code anyway. We introduced the code review process that didn't exist between data scientists. That was good. One of the biggest artifacts that we produced was a machine learning readiness exercise, which is essentially a checklist of things to do before or after releasing a major version of an algorithm, or a brand new algorithm. This exercise is widely used by all of our teams that deliver or that use machine learning to build their products.

Another good thing that happened for individuals in that team was that they've learned a huge amount in that year. We started working together a lot better. We haven't even realized that we had such a barrier between engineers and data scientists. We never assumed that we could do each other's work, or we can help each other. Finally, the walls that we didn't even know existed, broke down. There was no more black magic concept around what data scientists are doing. As a result, no other iteration that we released on the models made riders to go on strike. We've never charged customers crazy delivery fees anymore. I'm very proud to tell you that a lot of those people were promoted. I'm very glad to be working with them because they're extremely smart and they worked extremely hard. They were rewarded for it.

The Journey Is the Destination

What are some of the takeaways that I wanted you to know and think about? At the product level, first of all, know who your stakeholders are. Think how they think. Give them what they need. You can have the most pristine and ideal code base. If the product doesn't do what the customer needs, it has no value. Start from the first principles. Find out where you can cut out on that complexity. Random forest, again. If you fail faster, you'll learn faster, what is the right product? What is the right approach? You'll ultimately get to the results that you wanted a lot quicker than you can expect. For teams, an efficient team is a lot more than each individual in it. You are working with very smart and talented people, I'm sure. No one team can achieve more if it's not working efficiently as an entity. Remember that. For individuals, don't think that people will know what you're doing. As you might not necessarily know what some job titles mean in your company, people might not know what you do. Don't assume they know. If there's an opportunity, explain that to them. Managers are not here to annoy you. There is a purpose in what they're doing. If it's not clear to you, just ask them. They'll gladly say it to you, I'm sure. Embrace change and be intentional about change. Because it will happen with or without you. It's the law of nature. Find what works for you and for your team. Ultimately, use your colleagues and especially your manager to help you grow and to help you achieve more and get what you want. Because your success is their success. Please, don't take my word for it. Try some of these things in your team and see how it goes.

Questions and Answers

Moderator: You've broken it down into product, teams, and individuals, which of one of those three things do you think is the most crucial to get right first? You can't say all three.

Puscas: Looking back at my case study, I would say the product. Because, eventually, you will get people to work better and more efficient together. They will perhaps accelerate their pace. There are things that are easier to control and improve. There's a faster feedback loop on that. On the product side, don't assume it's someone else's job to know what customers want and what product needs to be. It is also your responsibility to think how customers think. I think the product.

Participant 1: We're actually doing a very similar thing with QA and developers. We are trying to launch a QA automation framework. I think right now, there is still a missing part of the cultural thing to get buy-in from a developer. That's the next piece that we're going to try and figure out. Any tips on that?

Puscas: It is tricky to give precise recipes. If you assume that those people achieve the same things and they have shared outcomes, then there shouldn't be a barrier. There shouldn't be a verification of each job description to say, this is something you can do, this is something you cannot do. If I was in that situation, I would identify people that are most open to their collaboration and see how that goes, and use them as champions. Because these things are infectious. Nobody wants to try until someone tries. Once the process is kicked off, it's much easier. Identify the people that will be good examples. Then roll it further.

Participant 2: Did you have a debugging strategy when you joined the team?

Puscas: I wouldn't call it, I had a strategy. I can definitely think about the few things that I've done. I allowed myself some time to observe. Just be silent and see what is going on. That was my first iteration. I also had the manager that previously looked after that team, which was great help. I also asked people very openly what they thought worked well and what wasn't. Other things were some of the feedback that we could get from people outside the team. People that were using the product or were going to use the product, and leaning on them to tell me whether or not they felt there was something that they don't get, or there was something that we could improve. For the team itself, it is not a fixed science. Ultimately, you work with people. You have to see what works and what doesn't. As long as you are clear with what outcomes you want to get to and what your intention is, then hopefully, people can tolerate some small mistakes and some things that don't go as planned.


See more presentations with transcripts


Recorded at:

Aug 14, 2020