Pat Helland on Software Architecture and Urban Planning

00:16 Introductions

00:16 Wesley Reisz: Hello and welcome to another edition of the InfoQ podcast. My name is Wes Reisz, co-host to the podcast and chair for QCon San Francisco. Today on the podcast, we are talking about urban planning (or at least urban planning and how it relates to software architecture).

00:36 Wesley Reisz: Our guest today is Pat Helland. Pat is extremely well known for his thoughts in distributed systems. He has been implementing transactional based systems, databases, application platforms, fault tolerance systems, and messaging platforms since 1978. He is currently a principal architect at Salesforce.

00:53 Wesley Reisz: Famous for his creative titles, some of which included titles like 'Mind Your State For Your State Of Mind', 'Space Time Discontinuum', 'Data On The Outside Versus Data On The Inside', 'Consistently Eventual' and 'The Best Place To Build A Subway', which he talks about being in a cornfield in Nebraska, not in the middle of a populated city. Today on the podcast, we're going to hit on topics like building software systems for the future. We're going to talk about this notion that services should be available that systems can plug into. We're going to be talking about the idea of constraints actually being empowering. And we're going to talk about devolving to the future. So, as always, thank you for joining us on the podcast. Let's get started with the show.

01:37 Wesley Reisz: Pat, thank you so much for joining us on the podcast today.

01:41 Pat Helland: Thank you Wes, it's good to be here. I appreciate it very much.

01:44 Wesley Reisz: This one's been a journey. You and I have been talking about doing this podcast since what, February, March, something like that?

01:51 Pat Helland: Something like that, but the COVID thing got in there and stuff and we all had our crap to deal with and I'm just fine. I'm actually getting more work done being locked up but it's got its downsides too. And I just want everybody to remain healthy. I worry about this virus and pandemic quite a bit, but then again, I've read books before about previous pandemics. And so, they've always concerned me.

02:11 Wesley Reisz: Yeah, it's amazing. And I'm glad you're doing well. It's not just the pandemic affecting... You're in the Bay Area. You've got the fires going on as we record this, right?

02:18 Pat Helland: Fortunately, none of them are close to our house, but they could be. We're adjacent to a state park, it could be, but the smoke is pretty annoying. And of course it's minor compared to what people are going through, losing their homes and the terrible battle, but we're fortunate to be doing okay.

02:31 Wesley Reisz: I'm actually, as we record this, I'm in Salt Lake City visiting my mother and it has this beautiful mountains that are often... This is first time I've actually ever been here, but you can literally see the haze, the smoke from California here in Salt Lake City. I'm just floored by it.

02:48 Pat Helland: And the interesting thing is, it goes from being excellent air to being horrific air based upon the way the winds blow. And so, we had some windows open, I got up this morning and it was smoky in the house so I had to close the windows. And it's not the biggest problem but it's just a concern. It's really, really awful what people are going through.

03:03 Wesley Reisz: And I was talking to Randy Shoup, I think, last week before... He was telling me that exact same thing, and it's the air quality in the Bay Area or whatever particular day he was talking about was the worst than any place in the world. It's amazing.

03:17 Pat Helland: Yeah. There's a website called Purple Air, which is interesting. And it shows you a whole raft of air quality sensors that people have signed up to get and use the internet to connect and it's just online and dynamic. So, it's really interesting, purpleair.com.

03:30 Wesley Reisz: That sounds like a properly geeky thing to look at.

03:32 Pat Helland: It's useful to see. 

03:34 What was the inspiration for ACM Queue paper “The Best Place to Build a Subway”?

03:34 Wesley Reisz: Yeah, totally. So, speaking of geeky things to look up, here's one that the audience can look up. It's the foundation for what I thought we would talk about, what we talked about talking about today. And it's something you wrote for the ACM Queue, and it's called 'The Best Place To Build A Subway'. So, this was March, 2020 that it was published at least, but it talks a bit about the importance or the relationship between things like some city planning concepts and how it relates to software. Specifically, it talks just like a city software system teams need to be built to evolve. Can you tell me a little bit about this paper and what made you want to write it?

04:11 Pat Helland: First of all, I write all sorts of things. Frankly, I've been so busy with other things that one of the reasons I selected that topic was because I knew it was relatively short and I could whip out a first draft in a day because this is the stuff I think about. I have seen repeatedly that when you are dealing with large companies, and I've spent 35 or so of my 42 years in the industry, working at companies with more than a thousand software engineers. And so, now you're thinking about, "How does stuff change? What makes it change? How does it work together? How does it not work together? What goes right? What goes wrong?"

04:42 Pat Helland: I've had a couple of years stints in my career a number of times, where you're working your butt off and then stuff doesn't happen, right? And then you better find a new thing to work on. And that's part of how it rolls sometimes. And I've learned to say at some point, "Well, that was not above my personal average for that year." And it works that way sometimes, but why? What connects things? What happens? And I've always countered when people are annoyed at the environmental constraints to do a big project. It's got to fit in with the stuff. And I've found that to be something that caused me to laugh because I started saying that the best place to build a subway is in the corn fields of Nebraska because you don't have those pesky historic monuments to work around. That does mean that you also don't have anybody willing to fund it. And so, if you want to build that, good luck, right, because that's not how it's going to roll. And there's there's this tension, "Can I plan and leave stuff in advance that allows for future growth, but basically costs nothing now?"

05:42 Wesley Reisz: And that led you to the discussion of Market Street in San Francisco, right?

05:46 Pat Helland: That led me to the discussion of Market Street in San Francisco where the city planner, Jasper O'Farrell, who laid out the city, left a 120 foot wide Boulevard, not to because he anticipated digging it up in the 1960s a hundred years later and to put in a subway to put in BART, but because he wanted to have a grand promenade. And it was the existence of the space for that grand promenade, which meant that you could dig a trench to build the subway down the middle of the Market Street without tearing down buildings. Now, it was very disruptive but it didn't involve tearing down buildings. And so, that was something where somewhat serendipitously the space was left to do something grand or you could argue whether it's grand or not, but I find BART to be pretty convenient down San Francisco's Market Street. And so, how do you think about what it means to evolve? How do you think about what you need to leave? And that then takes you to, how do you think about a big IT shop and how does stuff connect to other stuff?

06:46 Wesley Reisz: So, as you've been talking, there's a couple things that come to mind. So, as you're talking about building a subway in a cornfield to Iowa versus downtown where you have to tear down historical monuments, the things that first came to my mind is brownfield and greenfield applications. Greenfield is that Nebraska cornfield, right? Building something out there at brownfield is the reality. You've got to build on the system that you have today. That seems like a good comparison between the two.

07:13 Constraints Empower development

07:13 Pat Helland: It's a very good comparison. There is no doubt about that, but greenfield's actually harder because constraints empower. It's like, "I got to deal with this. This is not arguable. We're going to make this be the case. And so, all of these assumptions are now boxing me in. And so, the set of decisions I have to make is narrower. So, the problem space is smaller and so now I can actually resolve it," but there's still issues at the fringe of how do I deal with the edge, but in fact, the shape and the form of the mega project is less likely to take on what when I was a kid was called the second machine syndrome. Okay. You constantly saw companies building their first computer with compromise by hook or by crook, barely getting it out and then getting a customer base. And now they're getting rich and now they're going to build the new thing that's going to not have all the problems with the whole thing.

08:00 Pat Helland: And five, 10 years later, much money has been spent and it gets canceled because there's no constraints on it and so it continues to spiral out of control.

08:08 Wesley Reisz: It's analysis paralysis. Yeah, I like that: constraints empower. I can see it. It's where something like Kupernetes. A very opinionated framework, for example, is easier to install than just taking a look at the CNCF landscape and diving in to all the thousands of potential options that you have when you're deploying something like Kupernetes.

08:26 Pat Helland: They do. They just tell you what to do. It's so much nicer in so many ways but that's never a perfect concept. It's not because then the constraints mean that what you built may not cover a need and there's no perfect answer. And some of that's why they keep old farts around is to just have the ability to have that wisdom to point out the trade offs between those two.

08:46 Wesley Reisz: It's just wisdom from people that have been burned enough to say, "Don't touch this, it's hot."

08:50 Pat Helland: Correct.

08:51 How do you balance the ideas behind build for the future and not over-designing a system early in the lifecycle of a project?

08:51 Wesley Reisz: Okay. So, in the same paper, for example, you've got a couple of quotes though and I want to ask how you balance these two things. So, you've already said one, 'The Best Places to Build Subways in is a cornfield in Nebraska', got that one. But you're also fond of saying there's two kinds of software companies, those that start out building scalable infrastructure and those that are still in business. So, how in the world do you balance this notion of building a market street big enough to build a subway in however long it was since Market Street was built and the time subways went in? How do you balance those two things?

09:24 Pat Helland: It's not always easy, right. What you need to do though, is you must pay attention to the business needs. Where is the money coming from? I've spent my entire career working for people who want software so they can make money. No criticism, it's been a good career. I like it. I'm going to continue doing it.

09:43 Pat Helland: But that's what it's about, is what's a business need for that? So, the obligation when you make something, especially if you're starting a brand new company, which has not been something I've done, you have an obligation to figure out how to get the money coming in but you don't want to spend forever building plumbing that is dealing with what happens when you were the third biggest company in the world. But at the same time, you want to avoid API constraints that are going to prevent you from getting there. Well, let me rephrase it. APIs that do constrain. They push you into a model such that you will be able to grow and do well as time goes on. Pat Helland:

10:17 Too Much Flexibility is a problem?

10:17 Wesley Reisz: So, too much flexibility is a problem?

10:21 Pat Helland: Frequently, because then that means that some of the people are using these options and those options get in the way when you want to constrain stuff.

10:26 Wesley Reisz Okay. I'm tracking. I see it.

10:28 Pat Helland: Most successful platforms have been built in conjunction with an application that runs on them. You're building out the platform and the guys down the hall or the gals down the hall are building out the application that runs on it and it's hand in glove. And they're using the stuff which is natural for your platform and your platform is using the application to guide the features that must be there. If your interface between the platform and the API is too porous, then there will be stuff that stops you from doing the next improvement because people will use the features.

10:58 Wesley Reisz: Simon Wardley talks about pioneer settlers and in town planners when he talks about organizing engineering teams, for example, that you've got different mindsets than the engineers that you employ. And the pioneers like to go create these prototypes. They like to go settle new territories. Town planners like to build off that and actually build towns on it. And these are fortuitously... They feed and build each other.

11:25 Wesley Reisz How do you balance constraining a system enough that you're able to deliver on a product that you're building but still enable those pioneers, still enable that new R&D with enough flexibility to build new innovation?

11:40 Pat Helland: People want to do prototypes. People want to do trial things and there's value there. There really is. You can learn by experimenting, but it's hard in classic engineering environments because too much is on the bottom line. And so, that's one of the virtues and I'm a big believer in research organizations. I've never worked in one, but I'm a big believer in them, I think that they can actually help the industry and do well. But the question is, as you're designing something... We started out talking about there's two kinds of companies, those that are still in business and those that set out to build scalable infrastructure from the beginning. And the question is, how do you make sure to prioritize getting the revenue going when you're starting? That's just part of what we've got to do unless you are in a subsidized research environment. Pat Helland:

12:22 Software systems are built for the communication mechanism they were designed with.

12:22 Wesley Reisz: But how do you really balance this? For example, Slack was started as a game engine company, right? The chat engine that Slack uses was originally based on gaming that they were going to do. And you've got companies that have been around for a hundred years now. When we were talking just before we started recording, you talked about how cities were built on the prominent transportation mechanism of the time. So, donkeys and Jerusalem, for example, or cars in a particular city.

12:49 Pat Helland: Or in LA, right, in LA, but not in Downtown Boston and not so much in Downtown New York (or Manhattan).

12:55 Wesley Reisz: You're talking about today, you need to know where the money's coming from. How do you build systems using "where the money's coming" from isn't always the same, it may evolve, it may change.

13:05 Pat Helland: That happens. It's not the common thing. It's not common to start out in business A and then ended up in business B. That can only happen when there's enough funding to ride through the failure of A and maybe some salvaging of some of the infrastructure and the team. I'm not denigrating that but that's just not the common thing. It's not how you think about setting out to do stuff. I do think that we are still at the cusp of thinking about compositional systems and what they mean.

13:28 Pat Helland: I'm going to slightly diverge here for a second on the metadata. I've been fascinated by the evolution from tightly bound interfaces. How you doing components and interfaces and WSDL and all of that tightly bound stuff into the, "I'm going to call you with some REST in here we go, and you can figure it out. Go scratch around the JSON and see what you can find in there. It'll all be good," right? And what's fascinating is I believe we are devolving to the future. I'm not criticizing. I didn't anticipate it. I think it's hilarious and cool and appropriate. The devolution, the de-evolution of the crispness of that interface means that you don't always know precisely what you get when you work together. But it also means it's easier to find stuff to work together with that you didn't really think you were going to work together with.

14:13 Wesley Reisz: And then we add schemas and definitions to add structure to the file. It's coming back so that we can actually know how it's supposed to be used.

14:22 Pat Helland: Correct. I describe extensibility as scribbling on the side of the paper forms, right? It works sometimes and doesn't work on times, right. And it's fine. I'm actually very accepting of it but I'm amused by it and intrigued by how it will evolve moving forward. And it's, by the way, the same damn thing that you see in big data. I think of big data systems, most of them as Baleen systems like a Baleen whale scooping through the krill through the ocean, right. And you're just picking stuff up and seeing what you can digest. And that's just a fascinating concept that stuff is not so well controlled because it's coming from so many sources. Pat Helland:

14:56 What is the relationship between regulations and city governance to software?

14:56 Wesley Reisz: So, you touched on this though, way back in 2004, when we were talking about this topic. One of the things you said was a talk you did from, I think, VS Live in 2004 called Metropolis. And you started talking about regulations and governance with the city, at least in the same terms, right? That's the contract is, "Here's the standard. This is the standard for building a doorknob," or whatever. Right.

15:20 Pat Helland: So, yeah. Let's think about this for a sec. I have a company, it's got just a portfolio of software and I’m adding to it constantly. And things are communicating within that. And now we communicate much more handily across companies, right. And things get squirted back and forth. So, if I start thinking about an IT shop as being a city and I start thinking about the stuff that's going back and forth, it's goods, it's manufactured goods, it's commodities, it's stuff. I'm putting them in container ships. I'm shipping them over. I'm putting them on a railroad and shipping them over. And that's fine but that city over there might have a different policy towards sewer and utilities and electric and governance and where buildings are being built and how they're being built than my city over here.

15:59 Pat Helland: But at all levels, when you're building out a piece of software, I think of that as building an actual building in the city, a constructed building and it's got a hookup to the infrastructure. And what's interesting is the infrastructure continues to change. The system management, the monitoring, the Kubernetes, the deployment. Cloud computing is putting things under pressure, by the way, I believe in cloud computing, it's awesome. But it causes pressure on what you did for your app and how's it going to run in this cloud native environment, how you can make it work, and how do you secure it? How do you do all of these kinds of things are a big, big part of life within the IT environment. Pat Helland:

16:33 When do you decide it’s time to rebuild/rearchitect software?

16:33 Pat Helland: And so, that ends up being a real fascinating pressure. And you have this tension, "Do I throw away a piece of software because it doesn't hook up well?" And one of my anecdotes that I very much enjoy, I was sitting on an airplane 15 years ago or something like that, flying over to Europe and the lady next to me was from Germany and we're chatting it up. And I said "Hey, how old is your house?" And she said, "It's about 150 years old but the whole neighborhood's got houses that are eight, 900 years old." I said, "Why is your house so much younger than your neighbor's houses?" She said, "This city said we had to put in sewers back in the day. They said that everyone had to connect to the sewer and the house that was on the lot I live on was not of high enough quality to retrofit. So, they tore it down and put in a new house that was hooked up to the sewer."

17:19 Wesley Reisz: Just like what we do when we move applications from on-prem to the cloud.

17:22 Pat Helland: That is exactly what we deal with. As we're talking about software being managed in an environment run by a company. They have to monitor it, all this stuff. The sewer has to work, the power has to work, the fire codes have to be right. All of these things have to happen in a piece of software and sometimes it's worth retrofitting. And mostly it is and sometimes it's not when you have to do these changes. So, now you stand back and you say, "How do I organize a ginormous IT shop to make these changes," and the answer is very much like how you gradually get homes in a city hooked up to the sewer. A little bit at a time.

17:57 Wesley Reisz: So, as you were talking, I couldn't help but think of Kubernetes. We've both been around software long enough to remember building years and wars. And you talked about SOAP and WSDL building web services on top of zip files that were being deployed out to a server, for example. Today, the unit of deployment, more and more is a container. So, with Kubernetes as that infrastructure and building onto that cloud native containerized infrastructure, that's exactly what you're talking about, right? With containers being the unit of deployment, we're less worried about the plumbing inside because Kubernetes provides that grid infrastructure if nothing else, like a city might.

18:36 Pat Helland: Kubernetes is cool, but deploying it and managing it's got its challenges and furthermore, taking a piece of software and making it run inside kubernetes has got its challenges. And so, that's just how it rolls but it's all about moving forward that infrastructure and those utilities and the safety and the management and monitoring. And I think it's incredibly powerful, incredibly cool, but don't denigrate the need for the work that it takes to do that. And even the management of making sure that all 10,000 artifacts of software that were running are all hooked up in a consistent way. It's like painting the golden gate bridge. By the time you're done hooking it up, you've got to start back over because something else needs to be done differently.

19:14 Pat Helland: And so, there's a constant process of churn in what it means to participate in the community of a company's IT shop just like what it means to participate in the community of the city in which you're going. In city, building codes are fascinating. They're really different. I laugh every time I see a door opening outwards in Florida. The front doors to the homes open outward because a hurricane might mean that you can't get out, right. It's a mess. And that just never happens on the West Coast. The doors open inward, we almost never have basements. The list goes on of things that are different on the West Coast than in other parts of the country and it's for good reason, but that's similar to what goes on company by company, by company.

19:55 Wesley Reisz: So, I want to dig a little bit deeper. So, you say Kubernetes is complicated, totally agree. Thousands of choices, lots of configuration and it gives you tremendous amount of rope to hang yourself. A hundred percent agree but what it also gives you though, is a way to elastically expand resources and the unit of deployment is a container. So, whether that's Python, Ruby, java.net, doesn't matter, put it into a container. It can be deployed on the infrastructure in this case, the city. That's what we're talking about though, right? Is that idea of a plugable... I don't know if plugable is the right term, but if something that can easily deploy onto the infrastructure.

20:33 Pat Helland: Well, first of all, I'm all for that. I completely know this, right. I completely know that the job of someone managing all this stuff in a huge environment is uber challenging. I'm also quite complimentary of Kubernetes. It does a ton of stuff, but it's a huge artwork to figure out how you're going to use it in your particular installation. And so, in my mind, it would probably be better off with a few less options but that's not a major point I want to make. It's a challenge in all of these areas. What I'm actually really more interested in saying is that the environment imposes constraints on the thing deployed. And so, and it will evolve over time.

21:07 Pat Helland: And Kubernetes is a tool that's great to facilitate how do you manage the consistent application of that across things, right? If you're using Kubernetes for microservices, I have a thing I worry about, which is the state of the microservice against data. And how does one get a consistent behavior out of that is a concern of mine but that's somewhat orthogonal to this discussion about how do you plug into the monitoring and the management and all of the things that you need to have in (and security is important).

21:33 Wesley Reisz: The services, right? The city services.

21:36 Pat Helland: But look, all these services, you just stand back and you look at what's expected of a container today to fit into most companies. It is dramatically different than it was five years ago and 10 years ago. And I'm not criticizing, but it would be best to assume that change is the norm. That's the point I'm making is that this is constantly being upgraded. And when you look at what happens in cities and how they deal with the physical buildings that they have, they don't knock them down the minute somebody comes up with a new idea of a utility you need. Mostly they're retrofitted and you mentioned earlier, the fire escapes that are common in buildings from a hundred years ago, right? Where they're on the outside of the building. And that was a clearly a retrofit. People figured out buildings are dangerous, figured out people die, they can't get out fast enough so the city forced you to hang this steel structure on the outside of your building and it's ugly but it allows you not to have to start over.

22:28 Wesley Reisz: And this gets back to that comment with regulations though, right? And how regulations and standardization apply to software. Regulations are to city planning what standards are to software, that true?

22:40 Pat Helland: Mh-hmm.

22:41 What are the relationships between how cities fund new initiatives and how companies fund software projects?

22:41 Wesley Reisz: So, what are some of the other things that you see from software, even software teams and how it relates to city planning or urban planning?

22:50 Pat Helland: Again, it's that when you do urban planning you're trying to say, "I'm going to take some space and reuse it and I'm going to figure out how to make it be different. And what can I do to financially motivate that?" And I found it fascinating to see neighborhoods of San Francisco funded by selling skyscrapers on land that wasn't yet owned, that they were going to condemn from the existing owners and pay them what was fair market before the renovations, put parks, put transit centers, put all this cool stuff and then sell the skyscrapers for a lot of money. And that worked itself out to be a neutral thing with a 20 year cycle to make it work. And that's a fascinating area of thinking through what it means to invest.

23:29 Pat Helland: And companies are also needing to do that with their software. How do they think about their software systems? And do they talk about a replacement for some functionality and how do they fund it? And how does that work? And how does it integrate with the rest of the environment? And those are areas that also intrigued me, because if you don't do that, if you're a large bank, a large international company that... They need software. Software is a huge part of what they do. It's stunning to see how many scientific and commercial projects, how much money is software. And now if you're walking up and you've been up, you've been doing software since back in the early days of the sixties with the mainframes, now you're evolving it and now you're doing different services and so forth. How do you figure out what it means to change that? Because things are twisting, right. We're doing machine learning for a lot of things.

24:13 Pat Helland: The online systems are changing from what they used to be. The B to C nature of things are changing. And how do you do that? You typically need to wrap stuff around the old stuff, right, so that it looks new on the outside and then potentially then separately consider replacing the thing in the middle that's doing the actual logic.

24:30 Wesley Reisz: Back to the brownfield, greenfield, where we started. As you were just talking about that. Some of the things that came to my mind when you're talking about building skyscrapers to pay for some of the green areas, parks and different things in the city. It reminded me of some of the economic models now of some of the IOT advices where they have internet services that are always provided but you buy one little device and it's like, "There's whole new economic laws that you got to figure out how you can continue paying for software that's feeding updates to your device for example. I can see some relationship there.

25:04 Pat Helland: It's a great time to be alive. And it's a great time to be a nerd. It's just so much to think about, so many fun things to do.

25:11 Wesley Reisz : Fires and COVID with the exception though, right?

25:14 Pat Helland - Well, yeah. I'm as concerned about that as everyone and I'm of course really stressed about the people who are under pressure with their homes and their lives. I just want us to work our way past this and hope we can all cooperate to do so, but it's a road.

25:26 Wesley Reisz: Together we get through it, right? All right, Pat, while we're coming up on the end, we started with 'The Best Place To Build A Subway'. So, what is your one piece of advice to a software engineer, software architect who is building, we'll say it's a greenfield system as a lack of constraints? What is your advice to them to be able to apply some artificial constraints or to reduce the problem space, but still build for the future?

25:52 Pat Helland: You want to really narrow the functionality of what you're doing. You want to figure out something that meets the business needs and is hopefully extensible to do even more. That would be cool, but that has to be something which doesn't get in the way of shipping because if you get to your deadline and you haven't shipped, then you've just got a piece of ship and that's not good.

26:13 Wesley Reisz: Piece of ship, huh? Well Pat, thanks for joining us on the InfoQ podcast.

26:17 Pat Helland: Thanks Wes