The Value of DevRel and Contributing to Open Source Initiatives

Transcript

Shane Hastie: G’day folks. This is Shane Hastie for the InfoQ Engineering Culture podcast. Today I am sitting down over a relatively short distance for a change. Our guest, Craig Box, and myself are both in the same time zone, which is quite unusual. We're both in New Zealand.
Craig, welcome. Thanks for taking the time to talk to us today.

Craig Box: Thank you Shane, it is a pleasure.

Shane Hastie: My starting point on all of these conversations is, who's Craig?

Introductions [01:16]

Craig Box: Oh gosh, that's a very existential question. You mentioned that we're both in New Zealand. We were having a chat beforehand to figure out how our paths have come and gone through various countries. I think that most people who grew up in New Zealand decide that they want to go off and see the rest of the world. It's a beautiful place here, but it's by virtue of its separation from the rest of the world.

I had started with a career in IT here because when I went through university, as a person interested in technology, I didn't want to be a developer. I didn't enjoy the idea of locking myself in a dark room and writing code and not talking to people. That led me to, first of all, customer-facing IT roles and then implementation using programming skills and ability, what would later on become what we think of as DevOps, but was called system administration at the time. Moved to Canada and did that for a couple of years there before deciding that it was far too cold and we wanted to go somewhere warm and ended up in London, which is of course famous for its warm weather.

But it was there that I got involved with cloud, first of all in doing deployments on cloud, and that led me to working at cloud consultancy companies and then eventually onto Google. I came at a very interesting time ,in that we were starting to talk internally about this secret project that was taking the internals of Google and the lessons that had been learned from many years of doing deployments internally and open sourcing them, and making it public to people in the form of a project which was later known as Kubernetes. I was at Google for nine years and that was basically through the entire life cycle of Kubernetes, from just before its launch to becoming the juggernaut that it is today.

Since leaving Google, I've worked at a couple of different companies, but now I'm working with a team of people that I was working with while I was at Google. One of the benefits of these community open source projects is that you can collaborate with people at different companies and quite often you'll find yourself moving between those companies as well. I’m working at Solo.io now; we're a vendor that is working largely with an open source project called Istio, which I was one of the leads on when I was working at Google.

Shane Hastie: So a technical person not interested in writing code ...

Craig Box: I know. It's heresy, isn't it?

Shane Hastie: [LAUGHING] But still deeply involved in the technology. You've fallen into the space that is often called developer relations. Tell us about that. Why do we need that?

Why do we need developer relations roles? [03:31]

Craig Box: It's interesting because to some degree I don't think we need it at all, but a lot of people feel that they need to have it as part of perhaps a marketing strategy or some sort of community outreach, and therefore it's just a thing that exists, especially at the larger vendors. It goes by many names. People don't necessarily like to use the word evangelism, so they use the word advocacy instead.

There is an idea that you are going out and having conversations with end users and saying, "This is what we're building, and this is what we see as a trend in the industry," bringing back insight from them to a development team. That's very much what the traditional approach was, in the world of boxed software perhaps. I think the first people who you might traditionally think of as a developer relations team were the Macintosh team at Apple around 1984. The idea there was that Apple would make computers and they would sell them to consumers, and the computer wasn't very much use unless the consumer also purchased some software. There was a different audience, a much smaller audience, of developers, who would make the software that customers would then go on and buy. There was a need for a different team to help those people find out what they needed in order to be able to make the software and go on, make the platform better for everybody.

That follows through to modern platforms. The teams at Google who make things like Chrome and Android for example, you could say it kind of makes sense to have that. But for a lot of the vendors we talk about today in the cloud space and anyone doing B2B software, you're selling a thing to developers, but those developers are your customers. You have a field technical team who go out and talk to them and if your thing is open source, your engineers are participating in these upstream communities themselves.

They know everything that's going on and they know what people are doing and where they're failing because that's all public. It's interesting nowadays in that we've got all these people who are nominally in the same-named role, but there's such a wide spectrum of things that they can be doing. I think it's very different depending on which organization you're at and where they are in the life cycle of the product they're supporting.

Shane Hastie: If we do need somebody, if we are in that position of almost the platform and we need somebody doing this, what are the skills that you bring to the table?

Skills needed for effective developer relations [05:35]

Craig Box: A lot of the people who I've worked with in the past, they are programmers with abilities that I could never hope [to have]. There's no point in playing that game. I look at some of the bug reports like, "Oh, I could probably try and work this out," but ultimately the people who built the thing are going to be so much better at it. But the idea of going out and explaining it to an audience, especially an audience who isn't deep in the technology or doesn't have that background, that's really a skill that I happen to have and a lot of other people don't; versus many skills that they have and I don't. I think that it's very useful and being able to do that with a technical background, even going out to audiences, it's very easy to go to a crowd and say, "Hey, I know the guy who wrote the software. I know the people who do this. I know the engineering team."

I remember saying, "I didn't get any smarter the day I joined Google, but all of a sudden I had access to the internal bug database and I could go and look things up and see things." That really gave me a better understanding of how things were done that I couldn't necessarily explain to the users at the end, but I could say, "Look, well there's more to it than this," and being able to explain to people what's going on and talk to them with this sort of shared history of the fact that I used to do this kind of implementation work. It's been a few years since I've actually done end-user work in terms of DevOps or anything like that, but I have done this, I have been through this cycle.

It's useful to be able to report back. Some teams use their DevRel people as a “customer zero” and say, "You go out and take our platform and try and build something cool with it yourself and figure out if it fits a need." Again, the people who build these things, they are very, very smart, but sometimes they're just a little bit too close to the problem.

Shane Hastie: This leans towards developer experience. What does good developer experience look like?

What does good developer experience look like? [07:16]

Craig Box: That's a tough question. Again, there is a sub-section of developer relations - at Google, it was called Developer Programs Engineering. You’d hear people talk about defining APIs and SDKs and all that kind of thing. That is a thing that I think should be considered the product of a team who are building a platform that's focused towards developers. I don't think you can say, "I'm going to just have my DevRel team do that for me." You need to be deep as part of that team and you need to assume that the product managers are having those kinds of conversations, and if they're not, it's a bit of a red flag really. I think they need to know because that's the audience they're building for.

I can't speak specifically to what makes a good experience for any given platform or product because the problem space is just so big. I think that even the audience of people having an idea of what it is that is relevant to them and what background they have, some people come in with a bit more knowledge and some platforms are an evolution of a previous idea. It's a lot easier to pick up newer programming languages if you have experience with similar older ones, for example. But then there are people who are coming into the industry fresh today and they don't have that background. They've never looked at these other languages or platforms. I think you need a combination of both.

I think you need to be able to explain something in a way that is easy to adopt for someone who doesn't have that, but expresses power in a way that is easy to peel back layers of. I saw a comment recently, which said it's easier to hide complexity. If you have a complex product that is in large part because the people who build it know that you might need this one day but you definitely don't need it on day one. You need to have a way of hiding it from people, versus not having the product be able to do these things, in which case people play with it for a couple of weeks and then they hit a brick wall because it doesn't help solve their problems anymore.

Shane Hastie: You touched on product management. What does product management look like in products that are being built for developers?

The nuances of product management for developer-focused products [09:06]

Craig Box: It depends, in the sense that if it's within a single company, it looks like building any other product, I feel. There's a modern idea of what we now call platform engineering, which is sort of the idea of building a product only for the internal team that you're looking at. Your developers are your customers, especially at a big company. You are now running a product organization that's saying, "I'm building out this infrastructure which I'm going to let my developers use." That's an interesting way of saying product management ideals and so on should be brought to this discipline of SRE or sysops or whatever you want to call it.

The thing that is interesting, especially when you start talking about developer tools, is the fact that so much of the tool space we deal with today is open source and a large amount of it is built by more than one vendor. If you are someone like GitHub and you're building something which is effectively proprietary, a service run by a particular company, they can do product management in the traditional sense - whether or not their product is closed or open source. But when you're looking at something like Kubernetes, that's built by hundreds of different companies and tens of thousands of people, I think in those cases you just kind of don't have product management.

You have people who apply that skill to things like GKE, which is Google's hosted version of that. But it's hard to say what that looks like when you've got so many collaborators. It's easy to say, "Here's some code, here's some engineering." It's a lot harder to say, "Here's documentation, here's planning, here's the rigor that you need around all of this stuff." It's harder for the people who are all contributing to this to find budget to apply that skill as well.

Shane Hastie: Let's dig into open source community. You've worked in and are part of a number of fairly large open source projects. How do those communities be effective?

What makes open source communities effective [10:56]

Craig Box: In a self-organizing fashion, I think. You get people who are interested in community and who want to deal with topics like governance and, especially with open source, licensing issues or legal aspects. You get a lot of people who are interested in technology and have that as their particular bent who are willing to participate in those ways. These multi-vendor communities have come up with approaches as to how they will deal with things that are fair in a company situation where you have people who are competitors who all want to come together. Quite often you'll have a foundation of some sort which will own the trademarks or the copyrights on particular things, so that all the vendors feel safe in participating.

But then you also find, as I mentioned before, there are things like documentation and administration of a project where it sort of comes down to luck, to some degree, as to whether you find someone who's willing to do that. In large part with the Istio project, which is the open source project that I'm primarily involved in, it was just that I knew that this needed to happen and I was interested in product management. I said, "Well, look, I'm going to apply the skill or this interest to not Google's internal version of it," because that wasn't my role there. But I had a little bit of rope, being in the developer relations department, to say, "Well, hey, I think that this needs to be applied to the project and that someone needs to come in and help run the meetings. Someone needs to figure out how the vendors communicate with each other and how their roadmaps align and how we plan everything."

If you find a person who's willing to do that and has those skills, then hold onto them, because they're few and far between. If you are one company building an open source project yourself, it's very easy because you just do it internally. But when you get to the point where you have these large multi-vendor things, not a lot of the vendors are going to staff that role. I think some people might say, "hey, it's missing and it would be useful to have it there," but it is definitely something that you notice the absence of.

Shane Hastie: Digging further into the open source communities and collaboration, this is the culture podcast, what is the open source culture?

Factors that influence the team culture in open source initiatives [12:56]

Craig Box: I think it's somewhat representative of the people. There's a paradox I can't remember the name of which basically says it's going to represent the culture of the organizations who contribute to it. If you start with something that is largely built by a single vendor, all of the norms in that project will largely reflect what the norms were of the company who contributed it.

Over time things will take something on largely reflecting the energy and effort that is put in. If we talk about Kubernetes for example, a lot of the early team there, was a lady named Sarah Novotny who was one of the first community managers who worked at Google on the Kubernetes project. She put a lot of processes in place around the project and the CNCF that allowed collaboration and allowed vendor-neutral contribution. That really carried through to the way that project operates. They have very strong norms in certain ways and they have a very big culture. You get a lot of people who come together and pick up the same T-shirt at the conference and go to the Contributor Summit.

Then you get some other projects which are sort of reflective of the fact that the companies who contribute to them are just that, they're companies. You get people who are building products based on this technology and they all want to treat it like - I heard someone call it a clearinghouse recently, or a trade body effectively, around a particular thing.

At the other end you get people's passion projects. You get apps, you get Linux and desktop software that goes on it that people just install and run and they find something broken about it. They say, "Hey, this is something I want to participate in." Those communities are largely just individuals. In the cloud space that I'm in, there's not a huge number of individuals. There are people who are looking to get involved and build up their skills and maybe be hired by one of these companies, but the reality is that these large open source projects are built by people who are paid to build them for businesses, because they are a complement to something that they do. That has been the case for many years now.

If you look at the Linux kernel, the people who contribute the most to it are vendors, people like Oracle for example, may be surprising, I think they were one of the most prolific contributors quite recently. Then you get vendors who make network hardware and storage hardware and so on. They are the people who are writing the drivers and moving things forward so they can sell more network hardware and storage hardware. It's not just people in their basement who are working on open source anymore.

Shane Hastie: You are on the governing board of the Cloud Native Computing Foundation. Tell us a little bit about what is the CNCF, and why?

The Cloud Native Computing Foundation [15:21]

Craig Box: Why indeed. Yes, so the CNCF was created as a way for Google to spread Kubernetes wider than just itself. If it was a thing that was only a Google project, then it would never get the people who are competitors to Google's cloud business - they wouldn't get behind that. Google obviously said, "All right, we want this thing to become an industry standard, and for that to happen, it needs to have a certain neutrality about it." They worked with the Linux Foundation to set up a project there, which is the Cloud Native Computing Foundation. It was seeded with Kubernetes as a technology, but it now has approaching 200 projects that are hosted in the CNCF.

There are a few different levels of the way the organization works. The largest sponsors get to have a seat on the business board, and there are a couple of seats that are reserved in that Governing Board for the projects and for the technical community. I was recently elected to one of those as a representative of the project maintainers. I'm saying, "I can come here and give the voice of the people who are working on building the software back to the CNCF in issues like how they spend their budget and how they market themselves and so on."

Largely the technical decisions are made by a group called the Technical Oversight Committee who are a group of 11 members, who are elected from the community to accept projects and to decide the direction of the foundation and should it be looking at projects in this space or that space, and how should they interact and relate. There are end user members as well.

The CNCF is the group who organize the conferences, the KubeCon conferences that have been going on for many years now, and a bunch of other events. They do great work internationally, making sure there are events in upcoming countries, making sure there are diversity programs. There are mentorship programs. There's a whole bunch of stuff that's done there made possible by the fact that there is such a commercial value to all of the software and that are companies who are willing to all put in and fund the organization.

Shane Hastie: How do you get such diverse stakeholders to collaborate?

Craig Box: You promise them that it's going to be a fair and neutral playing field, as I understand it. The goal was from Google's perspective to have this become the standard because it was a pattern that they knew how to operate and that they thought they would get more people coming to their cloud to run workloads at a time when they had a smaller market share. That obviously worked out very well. There are a bunch of other people who were interested in taking what was a more superior technology at the time, that had the sheen of Google's internals associated with it.

Then there are also other vendors who were like, "Well, we might not want to adopt the thing that Google built because we compete very heavily with them, but our customers are asking us for it and it has that demand."

Bringing it back to the beginning, working in developer relations for Google at that time was going out and trying to convince people bottom-up that they should want to run Kubernetes. Then they think, "Yes, I like that," and they go to Amazon who were their cloud provider at the time and say, "We want to run Kubernetes." Eventually Amazon builds out a very good hosted Kubernetes implementation and that gets them on board with the project and joining the CNCF. But it is largely down to customer demand and the way that Amazon operates. Then it is the political neutrality, and the reputation of groups like the Linux Foundation, that everyone will be treated fairly, that allows those things to happen.

Shane Hastie: Thinking of our audience, as a technical leader, technical influencer, technical practitioner, why would I want to get involved in one of these communities?

Why get involved with open source communities? [18:56]

Craig Box: You might not. It comes down to your own particular interests. Even within the communities that we deal with, there are people who just want to write code. They want to take a technical issue, solve an interesting technical problem. There are people who want to write documentation and don't want to understand the code, but who want to effectively ask one of those technical people how something works, and translate that. There is every possible opportunity, especially in something  as broad as the Kubernetes community. It is testament to how people like Sarah, who I mentioned before, set the community up, in that it's a welcoming space for everyone.

The CNCF, and the Kubernetes project especially, put a lot of effort into being welcoming for new users. There are a lot of mentorship programs. There's a lot of money from the Linux Foundation to sponsor people, programs like the Google Summer of Code that kicked off maybe almost 15 years ago. Now, there is a Linux Foundation program that's similar that gets new people in who don't necessarily have the opportunity to just walk into a job like you might if you were in California, but who are interested in building their skills in these spaces.

A lot of effort is put in to make sure it's a welcoming community, that there are opportunities for people at every level. There are things that you can do if your company wants to get involved and has a product based on this, but equally there are things you can do if you are a student and you want to get involved in technology.

Shane Hastie: What's the future direction?

Craig Box: That's a tough one, and I say that because it feels in large part that these problems have been “solved” within the space that we're in. If you think about the curves of disruption over the last few years, everything gets to a point - desktop operating systems are mostly done. It got to the point where everything was good enough and then the internet came along. needed to be a new thing. Cloud native was a new thing, replacing the idea of - building on top of also, but replacing the idea of virtual machines or servers. I'm not sure I could say, "Hey, there's some amazing new development that's going to happen in this particular space." What I think will happen is some other space will come along.

The basic unit of deployment, or the building block of cloud native technologies is usually referred to as the container,  the Linux container, and Windows now has containers as well. But then it's possible that something like WebAssembly comes along and says, "All right, well, we have a new way of taking code and running it." Ultimately, we're still doing what we were all doing 50 years ago, which is writing code and then deploying it to some computer somewhere. Maybe it's the one in front of you, maybe it's tens of thousands of machines in a data center in Virginia somewhere, but we're still just writing code and running it on computers. It might be that there is a new methodology for doing that, that gets better. It might be that the singularity and the AI stuff that we all promised ourselves that we wouldn't talk about, obviates the need for that somehow. But unless those things happen, we're probably still going to be writing code and deploying it on computers. The means may change, but whether or not it happens within this community or whether something else comes along - it's hard to say.

Shane Hastie: What haven't I asked you that I should have done?

Craig Box: Who does that lovely hair of yours? That's what I would've asked.

Shane Hastie: [LAUGHS] I polish it.

Craig Box: I can't come with a piece of wisdom that I would apply to everyone. I could say that there are now so many more things to do, like the things that I do and possibly yourself also, Shane, you couldn't say when you were in school, "You can study to do this thing," because it didn't exist. I'm not sure what to tell my children to learn today.

I saw a historian being interviewed recently who said, "In the Middle Ages, and even as recently as a couple of hundred years ago, you could teach your kids to farm and to sow seeds, and that is a skill that will be useful to them and see them through 20 years from now." We don't know what the skills of 20 years from now will look like. I can tell you that there were no developer relations people, there was no practical World Wide Web, and so on 20, 30 years ago, and that's the thing that's enabled my job and probably the jobs of many people who are listening to the show. It's really hard to say, "This is a thing I think you can do." But one thing that especially working in open source taught me is that it's the relationships, it's the people, it's the way everybody works together. All of that will still apply, and all of that will still be relevant no matter what the underlying technology changes are.

Shane Hastie: We're in New Zealand, the Māori term, "He tangata he tangata he tangata!". It's the people, it's the people, it's the people.

That's a great point for us to pull this to a close. Craig, if people want to continue the conversation, where do they find you?

Craig Box: This used to be so easy. I used to just tell people to follow me on Twitter, and I'm not sure that that's a thing that people are still comfortable doing. But I am on the remains of Twitter. I am @craigbox there, and you can find me by that name on most of the modern social networks. I am a lapsed podcaster; with a couple of young kids at home, I haven't been able to keep up with that as much as possible, but you can definitely find my work around, and Twitter is probably as good a place as any to look for me.

Shane Hastie: Wonderful. Thanks so much for taking the time to talk to us today.

Craig Box: Thank you very much.

Mentioned:

• Kubernetes
• Istio
• Cloud Native Computing Foundation
• Craig Box on Twitter and LinkedIn