Developer Experience for the 99%

Transcript

Shane Hastie: Hey folks, it's Shane Hastie here. Before we start today's podcast, I wanted to tell you about QCon London 2024, our flagship international software development conference that takes place in the heart of London, next April 8 to 10. Learn about senior practitioners experiences and explore their points of view on emerging trends and best practices, across topics like software architecture, generative AI, platform engineering, observability and secure software supply chains. Discover what your peers have learned, explore the techniques they are using and learn about the pitfalls to avoid. Learn more at qconlondon.com. We hope to see you there.

Good day, folks. This is Shane Hastie for the InfoQ Engineering Culture Podcast. Today I'm sitting down with Jean Yang. Jean is the head of observability at Postman and has some strong opinions about building products for real developers. Jean, welcome. Thanks very much for taking the time to talk to us today.

Jean Yang: Thank you for having me.

Shane Hastie: My typical starting question with a guest is, who's Jean?

Introductions [ 01:17]

Jean Yang: I am the head of product and observability at Postman, as you said. I was previously the founder and CEO of a company called Akita Software, focused on API observability. We got acquired into Postman this past June, and prior to Akita, I was a professor at Carnegie Mellon University. I was working in programming languages and I was focused on building developer tooling, so language design, static analysis, dynamic analysis. And I started Akita because I realized that to get to the kind of impact I wanted for developers, I wanted to be building developer product, and that I came to believe that it's not at the language level that a lot of the upcoming innovations and impact are going to be. It's a lot of tooling around production and it's a lot of tooling on top of the languages and analyses that currently exist.

Shane Hastie: When we were chatting earlier, you mentioned that you feel there are three perspectives that are not very well aligned, that of the academia, that of the, do we call them thought leaders or influencers? And the work that's actually being done. So let's look at those three different perspectives.

The misalignment across academia, thought leadership and real development practices [02:28]

Jean Yang: Yes, so I came from the world of, we'll call, the software development futurists. So we were academics. We were looking at this is how programming could be and should be 5, 10, 15, 50 years from now. And a lot of programming practice has come from research labs. So if you look at all of OG programming languages and programming concepts, what we have today, C, C++, the Bell Labs artifacts, you had language research groups coming up with a lot of the previous things, a lot of the data types that we see entering languages today, they were developed in research in the late '70s and the '80s, when there was a lot of activity in how should people be programming in research, because not that many people were programming in practice yet. And in the futurist category, I'll also put the Bret Taylors of the world, who are really living in, how could programming be? What should it look like to construct software?

So that's one camp. Then there's a camp of, well, programming reality's not quite like that. You've got really messy heterogeneous systems, and you have all this stuff going on, and you have to do stuff at scales that a researcher, either in a academic lab, or a think tank, or art group, or what have you, there's just scale that you don't hit other places. There are people issues, you don't hit other places. So you have what developer influencers are going around talking about in their blogs at conferences. And so when I was in academia, I spent a lot of time reading what I could of this writing because it supposedly was a reflection of who we should be building for in the future. But I also spent a lot of time, when I was a grad student, I interned many places including Facebook and Google.

That was before grad school, but I also spent a couple of summers at Microsoft Research and we would talk to the non-research groups there as well. And so I started talking with practice groups there and I thought that was a good idea. So then I would go to conferences and see, well, what else is out there? And what I discovered was, maybe at a Microsoft, Facebook or Google, you would get people kind of close to what developer influencers are talking about because they could afford developer productivity armies, and tooling armies, and whole groups of people whose sole responsibility it was to make sure that they were scaling, developers had every single tool at their disposal and they could just focus on software problems. But everybody else that I was meeting when I was giving talks, in companies, at conferences, in other groups, they would come up to me after and they would say, "Jean, the things you're talking about assume that we're in a situation that's way more advanced than we are right now."

Real world software engineering is messy and constrained [05:17]

And when I started Akita, my company, as I was leaving Carnegie Mellon University where I taught, I went around to different software groups, and that's when I really realized people are saying one thing, but even some of these companies that proclaim to have certain practices, it turns out they only have those practices on one team. Or there's all these exception cases to the practices they're talking about, but that didn't make for as nice of a story. And so I realized there's just this real non-Photoshop, non-air brushed, very messy reality of software development and real software development has become more like a science than an engineering discipline in many ways because you're dealing with a lot of chaos, uncertainty and things that traditional engineering never taught us to really wrap our heads around in some ways.

Shane Hastie: Let's dig into what is the experience of what you've called the 99% developers, and how can we make that better?

Jean Yang: One thing I will say is that when I started out thinking about this term, 99% developers, a term I got from Scott Hanselman at Microsoft, my thought was, "Even though we say 99%, it's really the VBScript developer somewhere in the middle of America or very far outside of Silicon Valley." And what I realized was it's really most developers. All developers who are not at FAANG companies, Facebook, Amazon, Apple, et cetera, are more likely to fall into this 99% developer category than not. And I would say some of the hallmarks of being on a 99% software team is you're under constrained in some way that forces you to make trade-offs about developer experience and developer productivity.

So there are things that you can do that would make your developers more productive, like have a full team maintaining your DevOps instrumentation, but you just don't have that. Or if you had the bandwidth, you would have spent the two quarters migrating off of this one legacy infrastructure so that you could unify and have more homogeneous infrastructure, but you just had to make a call and you didn't have the resources to do that. And so I would say any company that has to make trade-offs based on developer productivity, I would put in the 99% developer category. And some common issues that I've seen are these teams can't do the best practices that require a lot of work. So some companies, they can afford to have a whole team spend a year migrating them onto Datadog, for instance, and then that team maintains their Datadog setup. A lot of teams out there in the industry don't have that luxury.

Some companies can afford to have a whole team of people migrate everything off of an old language or an old infrastructure, while most teams out there, they're really dealing with legacy subsystems, homogeneous subsystems, and you can't really make assumptions that, hey, with a snap of our fingers, we could all be using the same tech stack, or we could all be off of some legacy thing that would subvert the entire story of why this new thing we're adopting is nice. A lot of new things that people are adopting, the story goes, well, if you fully adopted this everywhere, then you get all the benefits. Well, what if you could only partly adopt it on a service or two? Too bad. And so to me, it's really important to talk about the issues that these teams face because a lot of tool developers are telling idealistic stories that are very idealistic compared to these realities.

I believe that if we were more realistic, even the people who are making decisions about these tools, buying these tools, can evaluate them in a more realistic light. And my hope is that this will give more space to tools that are addressing some of these realities.

Shane Hastie: How do we have that conversation?

Jean Yang: Well, my take is it's very similar to many other parts of life where, first, people are upholding some ideal, then people say, "Hey, wait a minute, that ideal is not true." And then they tell stories of the real version. A cheesy analogy I like to make is the Dove soap real women campaign, where most soaps, I guess, used to have very beautiful idealized women and now they have these real women of all body types and colors and all that stuff. But I think that's actually very powerful is to start highlighting stories of real software development.

So there's two parts of that. When someone tells a story of process or practice that seems too good to be true, it might be too good to be true. This is what I discovered from talking to teams about their real practices. I believe more questions should be asked instead of people going around and saying, "Hey, Spotify gave a talk and they said they did this and we should do it too." If you can't adopt it, maybe there's good reason, maybe there's a reason they didn't adopt it too. Asking more questions about what really did happen in practice. And then highlighting more stories of, this team had to wait five to 10 years to do this migration, and here's how they made do in the meantime. Those are not stories that are considered fun, or nice break from reality, or sexier, or any of that, but these are real useful stories. And so something I tried to do in a podcast that I used to host called PLTalk is, the last season was kind of realities, and I really believe more stories and acknowledgement should be told of the realities.

The more stories there are and the more narratives that are out there, people don't have to take the plunge themselves and come up with a cohesive narrative in order to tell it, if that makes sense. So the reason people can point to you, well, Amazon did this or Netflix did that and we should do it too, is because those stories are polished, they're circulated, they're highlighted. And if we could do the same for more real software development stories, that's very, potentially useful to a lot of teams out there.

Shane Hastie: Many organizations are not going to want to expose the dirty laundry like that.

Jean Yang: That's a really good point. I think a lot of the reason organizations approve of their engineers going out and saying, "Well, look at us. We innovated. We did this thing that's actually only true on half a team, but it makes us look good, so let's talk about it." I do agree with that. I think that there are organizations out there that would be okay with  it, especially because they might understand that attracting senior developers, telling stories transparently, can attract a certain kind of people. And my hope is that there are enough people and organizations out there who do believe in this, that we can start telling stories in a more realistic way about how software is created, how it gets maintained, and what the real problems are.

Shane Hastie: We can find a way to create a space for those conversations. Switching topics slightly, I know that you have some strong opinions about the design of tools for developers. Tell us more.

Designing tools for developers [12:07]

Jean Yang: I used to be very philosophical about tool design. I came from academic programming languages research. I had a colleague who called himself a computational theologian, or something like that. But there was a joke that a lot of programming languages research was religious battles. Some people think types should be used, some people believe types shouldn't be used. And traditional user research, if you did a user study, that wasn't going to tell you if half hour of using types, there's a lot of randomness in that user studies. The way people were doing them in human computer interaction research, wasn't really the way to test any of this. And so I've really departed from that because I felt for a long time that, surely there's been so much work done in software development practice and so many developers are programming now, there's enough data that we could be doing something different.

So there are many ways people have started using this data. The way I've really settled on is I think that developer products should be built like any other kind of product. You should be looking at what are the needs? You should be looking at user research, the way you do for any other product. What are the problems people have? What are they using today? What are they already doing repetitively or painfully that you could be replacing? And I think that one of the reasons I got really into thinking this way was, when I started grad school, I had all these cool ideas about wouldn't it be cool if we could do logic programming in this way, or there's so much boilerplate out there, wouldn't it be cool if we could just do this? And a lot of the time I realized that's not actually a problem people have or, as one of my early mentors told me, it seemed like I was running around with a hammer in search of a nail.

For better or for worse, I think that if a solution is cool enough, a lot of people are willing to try it out or that's been the case up to now when it comes to programming. I would say, in terms of professional software development, that's becoming its own thing different from programming, and more and more people are looking for value, especially tools that cost money. People are looking for how do I justify this to my manager and their manager? And so we've benefited a lot from tool developers who are very philosophical or religious about this is how programming should be. That's really given us the foundation for a lot of what we do today, but now we've entered a phase where I think we should be much more rigorous about following focused, disciplined user research, product research, product practice, and really focusing on who the user is.

Shane Hastie: Given your background, your experience, your knowledge across these different domains, what is programming becoming in the future?

Some ideas around the future of programming [14:51]

Jean Yang: Yes, that's a really good question, back to what I said before about how programming and software development are becoming really different. I think a lot of modern software development is calling APIs, maintaining code and building stuff on top of increasingly legacy code bases. And that's very, very different than, think about in the '70s what programming was. It was opening up an editor and typing code from scratch. And so I think that there are a lot of people still innovating from the point of view of if you're typing from scratch, this is what you could do better. And that's all very useful and good, but something that's always interested me is how do we actually help people build more reliable software in practice today? And to me, a lot of the interesting, important problems around there are, what do you do about legacy subsystems? They're really like corrosive, toxic waste. They're building up, they're full of bugs waiting to become a vulnerability or waiting to become some kind of meltdown. You can't really just forget about them. It's hard to quarantine them. What do you do with them?

Then there's the question of, how do you debug and maintain code that you didn't write, you don't have a lot of visibility into? A lot of the tools out there today are about, if you wrote some code from scratch, here's how you log it better, here's how you trace it better. Those solutions aren't actually a good fit anymore. And even just understanding code that exists already, people rely on documentation or someone existing on the team passing it down. I recently did a user interview with a guy who said that his department has said that he is a liability because there's so much knowledge only in his head that if something happened to him or even if you were out for a few weeks, they just wouldn't be able to understand large parts of their system. And so we need new practices for the time we live in, which is more complex systems, code that's older than ever before, and more turnover on teams than ever before. So the chance that whoever wrote something is not around anymore is actually way higher.

Shane Hastie: Where does Copilot AI fit in this?

Jean Yang: I love Copilot. I think I was one of the earliest people in the programming languages community just to embrace Copilot because I think that Copilot has taken this view that software systems are really complex. It's no longer possible to fully understand them end to end, and we just have to make do with what we know. And so it's very much like, well, you're going to have to read a bunch of code, and understand it, and throw some darts at the wall, why don't we make it easier for you to do that? And so I think solutions that just acknowledge that you can't control everything anymore, you can't know anything anymore, those are really the future.

Shane Hastie: In order to really embrace this future, what is it that programmers of today, and people coming into the programming sphere today, need to be comfortable with learning?

The skills needed for new ways of working [17:40]

Jean Yang: I believe the skills are really different now. When I was in school, when I was coming through, what was really valued was the ability to produce large amounts of code from scratch. It was algorithms, it was coding ability, it was what they test in US Computing Olympiad. How can you race to implementing a particular algorithm or something like that. And if you're looking at debugging, maintainability, understanding systems that you didn't write, that's a different skillset. There's more of a scrappy skillset of how can you parachute into a code base and quickly find your way around without understanding everything. That's a really different skillset than, can you implement an algorithm from scratch?

Can you excavate enough of a system to figure out how to call a service over there that's not really documented, and there's not a person who's responsible for doing it? So that, I don't even know what I would call some of these skills, but I would say these skills are coming more to the forefront. And so I think who is a good software developer, that profile, is shifting. There's also so many tools like Copilot. It used to be you're a better programmer if you had a lot of space in your head to hold all this information. You don't need that anymore because Copilot can just auto complete for you. And similarly, it doesn't even matter how fast you can implement core functionality anymore because you can just call a library function for that, or you can call an API. And so the ability to work with APIs, library functions, the ability to use some of these more modern tools, that's going to determine who's good software developer in the next five, 10 years.

Shane Hastie: Jean, some really interesting points here. If people want to continue the conversation, where do they find you?

Jean Yang: I am on X, formerly known as Twitter, a lot. My handle is jeanqasaur, J-E-A-N-Q-A-S-A-U-R. I'm also the same handle on Twitch. I had a live stream I did there, but I haven't done that in a while. And yes, I would say Twitter is probably the best place.

Shane Hastie: Thank you very much indeed.

Jean Yang: Thank you.

Mentioned

• Postman
• Copilot
• Jean Yang on Twitter