Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage Presentations Architecting for Focus, Flow, and Joy: beyond the Unicorn Project

Architecting for Focus, Flow, and Joy: beyond the Unicorn Project



The panelists discuss some of the most fun and least fun moments when coding, how functional programming practices have helped, and how productivity can be unleashed at a team-of-teams scale.


Gene Kim is author of The Phoenix Project, The Unicorn Project & Accelerate. Michael Nygard is architect at Cognitect and Author of the Best Seller "Release It!". Carin Meier is data engineer at reifyhealth.

About the conference

QCon Plus is a virtual conference for senior software engineers and architects that covers the trends, best practices, and solutions leveraged by the world's most innovative software organizations.


Kim: I've had the privilege of studying high performing technology organizations since 1999. That was a journey that started for me back when I was a CTO and founder of a company called Tripwire in the information security and compliance space. Our goal was to study this amazing organization that had the best project duty and performance, the best operational stability and reliability, as well as the best posture of security and compliance. As you can imagine, in a 21 year journey, there were many surprises. By far, the biggest surprise was how it took me into the middle of the DevOps movement, which I think is urgent and important. In that journey, a particularly huge surprise, was my learning about the importance of architecture. That architecture was one of the top predictors of performance, even larger of a predictor than even continuous delivery.

I think what is so amazing is that, it is architecture that really dictates to what extent teams can work independently. To what extent can they develop, test, and deploy value to their customers without being coupled to potentially scores of other teams? I know that architecture is a particularly favorite topic here within the QCon community. When Randy Shoup asked me if I'd be willing to chair a panel on the subject, I jumped at the chance, because I wanted to explore these topics more deeply with two people whose achievements I deeply admire, Carin Meier and Mike Nygard.


Could each of you briefly introduce yourselves? Tell us what you're working on these days?

Meier: I am a data engineer right now at Reify Health. I'm also the author of, "Living Clojure." I'm very interested lately in the Clojure data science, machine learning areas.

Kim: It's now leveraging the Python amazing ecosystem of libraries. Mike, how about you?

Nygard: I am currently SVP of platform and architecture at Sabre, where I'm helping this company through a digital transformation and turnaround. The subject of architecture and coupling is very much on my mind every single day, at Sabre. I've also been a developer for a long time. In fact, Carin and I were colleagues at Cognitect, where we were both working in Clojure. We got to partner on a pretty amazing project, building a desktop IDE for building games using Clojure, which is not exactly considered the sweet spot for Clojure. It worked exceedingly well in that situation.

Kim: I saw that video. The name of that editor is called Defold.

When Coding Was Most, and Least Fun

We're going to talk about what productivity, focus flow, and joy look like in the large, but maybe we could talk about first in the small, at the individual level. Could you both talk about when coding was most fun for you? Maybe even let's explore the opposite, when was it also the least fun for you? Maybe talk about the factors of what made it so fun and what specific factors made it so not fun?

Meier: I've been doing programming for quite a while. It became really fun once I started coding in Clojure. I think that is a lot due to the nature of the language, and how fast you can get feedback, and how close you feel to the code. You can really interact with it and almost sculpt this, and the data. That really makes it a pleasure to work with during the day. It's very productive too, it gets rid of all the boilerplate and you can just focus on the real problems. When I'm working with Clojure, is one factor. I also like to explore and do a little bit of research. That really attracts me to the data science, machine learning realm because there's always something new to explore, and novel applications that can help businesses and just people.

Kim: In fact, one of the things I really admire about your work is it's so clear that you're really on the frontier of some of these problem domains in terms of solutions that are being developed across a very wide surface area of technologies. Am I overstating the case?

Meier: I do like to read the latest papers that come out, and old papers too. It's always nice to see, what problem could that solve? It's a new way of thinking about it and this new tool could open up possibilities that we haven't thought about before. That's really exciting to me.

Kim: I just want to concretize some of the things I heard. Some of the things I heard was fast feedback in your work. You get really fast feedback in terms of whether something is working or not. I know in one of our conversations, you said just how miserable you are when it takes a long time to get feedback. By long feedback cycles, you were talking about minutes, not hours, not days, the fact that it even took minutes. Am I recalling that correctly, Carin?

Meier: Yes. I get a little down when I have to do my Docker builds and deploys, that takes a little long. It's a far greater cause building out your pipeline, but I much prefer to meet my REPL.

Kim: Creating immutable build artifacts. That's important. It does amaze me that when you have 4, 5, 6, 7 minute build times that actually does suck the joy out of our work.

Mike, does that resonate with you? Tell us about specific times when you were having just a lot of fun, and versus not fun?

Nygard: First of all, waiting on a Docker build or anything else that's over that one minute threshold, like it's time enough to flip away and go look at social media and get angry and stressed out and depressed. It forces you off task. The times I've enjoyed most have been the ones like the project I worked on with Carin, aside from the fact she's just a joy to work with. It was a project that we were working on all the time. We were focused on it, really focused. The previous projects I've enjoyed most, there was a project for Photo Studios, where we were in a full XP lab situation. We controlled our equipment. We had our OS builds, they were ours. We controlled our build infrastructure. It's not that we were control freaks so much, it's just that it allowed us to cut off all those external dependencies, and really focus on the work itself. There were times when we would joke about whether this was a one can of Pringles problem or a two can of Pringles problem. What it really meant was, is this something we're going to get done in a morning, in the short block? Is it an afternoon, or is it an afternoon and an evening? Sometimes you'd get into that state of flow, and suddenly realize it was a two can of Pringles problem, because it was 9:00, and that was dinner. Not great on the health front, but it was a really enjoyable experience because we were able to work together, work closely, and create everything we needed to create. I think that's one of the ideals you wrote about around locality. We really had that. Those are the times that I've enjoyed the most.

Kim: Locality was really meant to suggest, to what extent can an individual or team do what they need to do by themselves, without having to open up tickets with potentially 15, 20 different teams? Mike, you had shared a story about your least fun moment. Can you describe what that looked like, and what are the factors that made that so frustrating and miserable?

Nygard: I don't even remember which example I used before.

Kim: Let me remind you. It was a floor with 300 people on it, tables.

Nygard: I figured it was either that or one other. No, it was a vast project. It was late and getting later, and so their solution to solving the late project was to add another 100 people. They had emptied out a floor of a bank and had rows of folding tables with all of us in there. The team was strictly organized by layers of the application, so we had a persistence team that had no idea what the domain needed. We had a domain team that had no idea what the users needed. We had a presentation layer team that also didn't know what the users needed. We were all trying to model the world while also managing these vertical dependencies that were just changing all the time. It was a tough slog.

Kim: Just to, again, concretize that. The fact that you had these teams organized, whether it was technology, it meant that anything that you need to implement from the user's perspective, from the customers' perspective was now smeared across those teams, and that involved communication, coordination, prioritization, sequencing. These are the things that suck.

Nygard: Exactly. Big documents, lots of meetings.

The Intolerable 4-Minute Build Times

Kim: I'll be the first to admit that I'm pretty fussy, and sometimes impatient. When Carin brings up the notion of, was it the intolerable 4-minute build times? Could we just talk a little bit about to what extent really that is tolerable or not tolerable? When Carin brings up that story, it reminds me of one of the most miserable experiences I've had as a developer, which was trying to get something working in OAuth. I'll admit, OAuth isn't my favorite thing to do, but what made it so frustrating was that the build and deploy times was about 7 minutes. To me, this was just an exercise in trial and error, in that every trial would take 7 minutes. I just found myself 2 hours later, in a pretty wretched state. I was no longer thinking clearly. I don't remember exactly what I did, but I got a more representative environment on my dev laptop, and suddenly, I could find out something works within 7 seconds. Suddenly, I could now be much more deliberate and productive in terms of actually fixing the problem. On a scale of 1 to 10, can you just react to that? One is [inaudible 00:11:21] that is creating childish impatience, and an inability to actually work mythology through a poem, or 10 is like, no, this is actually of concern, because those are the conditions that any developer would actually be able to be productive in. Mike?

Nygard: I don't want to belittle a 7-minute delay, but somebody has always got a worse story. I vividly remember C++ projects that would take 2 hours to compile, or even overnight. Sometimes you would commit your code, and you'd come back the next morning to see if the nightly build succeeded or not. To the extent that all of those delay feedback, they delay learning, they make it harder for you to adapt the software to what it wants to be, and to what your users need it to be. They're all bad. There's a really interesting cutoff point to me, somewhere between 10 minutes to 30 minutes, where people start to bend their whole workflow around the build times. If you know you only get three builds a day, then you plan your work around those builds, which has its own set of issues. There's something uniquely irritating about those medium-sized chunks, because they're too big to ignore, but too little to really switch modes into something else and get your head in a different space.

Kim: Carin, how about you?

Meier: I totally agree with it. Trying to minimize it, and whatever that means to you. You know when you're annoyed, when that time frame is for you. You should try to do something about it. One of the most recent times that we had to deal with that was an extremely annoying one that made me post a hawk GIF, because it enraged me so much every time it happened. It was in the middle of our Docker build, randomly, we would be building a JAR and the JAR would be cropped for just one of those, you have no idea why. It randomly happened but it was rage inducing, because then again, you'd have to lose 10 minutes. It got pushed to the backburner until it finally, as these things happen, you decide to do something about it. I think one approach is to timebox it. Say, I am going to take a day and try to timebox fixing this crop JAR problem. In the long run, it is going to make everything run a lot smoother, and make everybody a lot happier, less hawk GIFs in Slack. I think that's a good way to put the investment. We did solve it, which was nice. There's always something like that that gets pushed to the backburner because it doesn't happen all the time but it makes your cycle time longer, and you've got to weigh.

Kim: It impacts everybody. It will impact everyone who's incorporating code, benefiting from builds, benefits from this improvement.

Nygard: What I find really interesting about it though, is people don't get upset about it unless they know it doesn't have to be that way. If they think that that lost time is just the way the world is, then there's no emotion about it. It's just the way the world is. If you come into that project from somewhere else, you're like, "Your unit tests take 10 minutes to run? What are you, savages? How do you live like this?" You get that level of dissatisfaction. Then pretty soon something has to get done about it. Sometimes it takes an example from outside to say, actually, this thing is not a fact of the universe, it is an artifact of the way you're working, or the technology you're working in.

Kim: I love that. In the unicorn project, I think these concepts that are contrary and counterintuitive, maybe even crazy sounding, one of them, to me, was the notion of where we put our best developers. If you look at the tech giants, Facebook, Amazon, Netflix, Google, they put their best developers, the most experienced developers on dev productivity. The next most experienced developers on backend services. Then probably the most junior developers on the user facing features. Whereas in most large, complex enterprises, it's the exact opposite. They put the best developers on the feet first. The next most experienced developers on backend services. Then they end up putting their summer interns on dev productivity and build systems, which is, I think, exactly not what you want, for the reasons that you cited there.

How Important Functional Programming Techniques and Principles Are For the Everyday Programmer

One of the other things that made me so excited to talk to both of you is our common love for functional programming, and in particular, one functional programming language called Clojure. Can you talk about how important you think functional programming techniques and principles are for the everyday programmer? In other words, is it only for the senior engineers and architects, as represented here within the QCon community, or is there something in it for junior programmers as well? I'll put that out for both of you.

Nygard: I think it's much too powerful and fun for the front line developers, it should be reserved for wizards who've ranked up a few times.

Kim: That's right, because it's finite. Only so many people can use it at one time. Mike, why would you say such a thing? You said it's powerful, what makes it so powerful?

Nygard: We have to unpack a little bit what we mean by functional programming. I'm not going to try and pull up multiple academic definitions. Generally speaking, we're talking about languages that favor immutable values, which just make entire classes of problem disappear. There are also languages that favor building functions out of other functions. You get this effective leverage, where you can create increasingly powerful abstractions to avoid repetitive boilerplate code. For me, those are two of the features that I really cherish.

Kim: To what extent can they be benefited by even junior programmers, or is this only reserved for the level 30 definition?

Nygard: Yes, level 30, for sure. No, I'm being facetious of course. No, I think there's a certain amount of introspection or soul searching in the functional programming community saying, why is this not the default? Why is this not the mainstream? Maybe it's just time. Maybe it just requires enough of a critical mass to build up, or enough people to be exposed to it. Certainly, I think a lot of the FP principles are making their way into all of the mainstream languages, to an extent everyone's going to be a functional programmer. I think it does have a lot of benefits to offer even beginning level programmers. I can't tell you the number of hours I lost to race conditions mysteriously changing variables, deadlocks, all of the concurrency problems that you expect to experience. I probably left behind some horrifically gnarly dependencies, because I wouldn't have thought about building a function that can take any function from Int to Int. I would have thought about, this class knows that class which knows that class, and there's a point where you are impressed with your own ability to hold a lot of complexity in your head. Then, hopefully, you evolve past that and realize, no, the true mastery is not needing to hold that complexity in your head.

Kim: I'm thinking of the chief architect for Java. His name is Brian Goetz, and his amazing book, "Java Currency in Practice." I actually came from some background in concurrent programming. I thought I knew a lot about concurrent programming, then I read his book, and I cannot tell you how many shocks were, and these questions of like, I had no idea something actually worked this way. Then that feeling of, how many times did I make this mistake that's now planted in somewhere that will get triggered. I totally resonate with that.

Carin, how about you? Is functional programming accessible or should be accessible to junior programmers?

Meier: Yes, I think so. I think the only reason why people have trouble with it at first is because it's a completely different mindset, looking at the world, and that's hard to transition to. I've talked to people that their first exposure to programming was functional programming, and they actually have a hard time trying to then do object oriented programming. I don't think there's anything intrinsically hard about it, I think it's just that transition period, of changing your mind to be more data oriented and immutable, over the object oriented way of looking at the world.

Kim: I had a sobering moment when my friend, Cornelia Davis, she was talking about, working with her son in college, who was learning programming for genetic sequencing. It was an algorithm that could fit basically on one PowerPoint slide. She noticed a bug that he had introduced just on iteration, and it was an off by one error. That was a very sobering moment for me in terms of like just how fraught with errors even things like iteration are, and that there's actually a better way to do things.

How Leaders Can Bring Better Ways of Doing & Thinking into Daily Work

We've talked about some things that causes better ways of working, better ways of thinking. How does someone bring a better way of working and thinking into their daily work, which might be at the mercy of too many committees, managers who are quite happy to do things the way they've always done? Can you talk a little bit about how do leaders at all levels bring in better ways of thinking and doing into daily work? Mike, you had mentioned that sometimes it takes someone from the outside to see how savage and inhumane certain working situations are. We're here at QCon, where there are a lot of great ideas forged through tough experiences. What advice would you give to people trying to introduce better ways of working?

Nygard: I think the more locally you can begin, the better. If you're trying to bring in an idea that requires the whole company to get on board, you're not going to get very far unless it's a five person startup. Neal Ford used to talk about bringing Clojure into organizations as just a JAR file for a dependency for better unit tests. Bring it in at a level where it doesn't require a lot of approval, because you're not changing the production runtime, it's still the JVM. You're not changing the language that the production app is written in so you don't need new linters or new syntax checkers or formatters, or build tools, or anything like that. You are giving people a taste of where you can derive some real benefit and get some expressivity out of it.

If you're in a team where you control more of your own runtime environment, you're in a more DevOps style: you build it, you run it type of environment. Then you might say, "Within our boundaries, we're going to try out a new technique, a new language, a new architecture, but we're going to maintain all of our interfaces the same way so we don't require signoff or agreement from the parties on the other sides." I view it as try and get that seed crystal going in a way that doesn't trigger the organizational antibodies. As you do that, you also look for allies. You build your Rebel Alliance meeting at the beach bar at 5:00 every Thursday or whatever, and look for executives who have some enlightenment and can offer support for your Rebel Alliance, and help provide them some shielding and some air cover as you start to make larger changes. Of course, be aware that you're not the only party introducing changes, and other people are introducing their own waves of change simultaneously to yours. How they intersect and overlap, and constructively or destructively reinforce each other, can matter a lot. Keep your eye out for that. Don't try and stop all of the others because they may be making improvements too, and you'd rather have allies than enemies. Build that network for change laterally.

Kim: I love that notion of the other insurgent team. I heard at the DevOps Enterprise Summit, that the only thing worse than one DevOps team is two DevOps teams, who are all clamoring for attention trying to do the right thing.

Carin, how about you?

Meier: At least in my position now, I'm working on a much smaller team, so we don't have to deal with a lot of the organizational complexity. I think some of the same fundamentals are true, and that's communication, trust, respect, and feedback within people. Because fast feedback is not only with your REPL and not only with your program, it's with the people on the organization as well. There's various techniques to do that, if you want to explore a different way of looking or doing something, maybe you could do a lunch and learn. You can present your idea, educate people on it, and get their feeling of it. Is it something that you want to invest in like a spike day? That's just one day that you make some goals that you're going to spike something out and see if it has potential. Then share those results with everybody else, get feedback again. There's various ways that you can do this in an incremental fashion. Again, feedback is not only for the REPL, it's for people in the organization, too.

Reactions to JAR File Fix

Kim: In fact, can you talk a little bit about what people's reactions were when you fixed that terrible JAR file, long build time plus JAR file corruption, as if long build times weren't enough, 7 minutes, terrible. When you actually finally fixed the problem, what were people's reactions?

Meier: We're all remote, like most people now. In Slack, there were many GIFs of rejoicing. That's how we celebrated. Yes, celebrated wins too, that helps everyone.

Kim: I have to imagine this is exactly the reaction that one would hope for. It positively reinforces that brave and initially thankless task of looking with the build system and the guts of Dr. Build, to maybe skepticism, to genuine appreciation as everyone's benefiting from this improvement that you had made.

Meier: That goes also to new techniques. One of the things we're talking about, blending Python and Clojure using a bridge. We're using a Python library to do some differential privacy, so that was a new technique we were introducing in the organization. We did that with research, lunch and learn. Do a spike day, is it good? Then develop it out, and assess the results. That's a technique that you can use with just about anything.

Current Passions

Kim: I had mentioned how excited I was just to hear about your perspectives on things that cause focus flow and joy. One of the other things I wanted to learn from you is just, what are you working on these days? What are you most passionate about now in programming, given the fact that this is probably, I know, something that is inherently challenging? You both don't gravitate towards the easy. How would you recommend others to get into it who are also interested in this? Carin, I've been dazzled by all the work that you've been doing in the ML and the machine learning community, and then this incredibly audacious project to bridge the worlds of the JVM, Clojure and Python. When I first heard about it, my first reaction was like, that'll never work. Just to see all the achievements coming out of this group of people has just been dazzling. Carin, what are you most passionate about these days?

Meier: I'm very much into the machine learning and data science combining with the Clojure. That takes a blending of worlds because there's a lot of specialized libraries out there that are available in Python. Traditionally, it's very hard to meld those two worlds. There is this library that was created by a very smart guy, Chris Nuernberger, called libpython-clj, that allows you to actually bring Python into your REPL, and be able to use that library and then also have the same advantages of being able to integrate it into your larger Clojure infrastructure. I think that's really the best of both worlds. It's very much with the Clojure philosophy of being pragmatic. You're powerful, but then you want to get the job done. Yes, we've used that with differential privacy at Reify Health, and we're also looking at using various other data science techniques to improve the speed of enrollment and to drug trials.

Reify Health

Kim: Could just say a little bit what Reify Health does, because in this age, I can't think of a more noble mission than what you're working on these days.

Meier: It's really an exciting place, meaningful right now, especially. Reify Health has an application that is aimed at addressing the first bottleneck in a clinical trial, which is getting enough people enrolled. We all know, we've been all watching, are enough people enrolled yet? We want this vaccine. That's one of the things. We're involved in some COVID trials, also, cancer treatments. All aimed at trying to get cures to people faster.

Kim: I've got chills just hearing about that. Yes, that is amazing. All these breakthroughs that you're talking about are in service of these missions that you're talking about?

Meier: Yes, definitely. I'm on the data team, which is a new team that's been formed to build out our data pipeline, and be able to bring in all these data science techniques to accelerate this.

Current Passions

Kim: Mike, how about you? What are you most passionate about these days?

Nygard: For the day job, we are taking a 50 year old company that started with a mainframe, in fact, the name of the company was originally just the name of the machine. We're getting off of the mainframe and moving everything to cloud. That includes the engines that power most travel searches that you execute today. If you go on to the popular online websites, if you engage with the travel agencies, or the corporate booking agencies that's using our flight search, travel search technology. It's a company with a long and rich history, which can have good and bad aspects, or hindering and helping aspects. We're moving all of that to Google Cloud, and rearchitecting a lot of it along the way. That's the day job. I mostly am working at that nexus of organization and technology. I don't do a lot of hands-on coding during the day job. To keep myself grounded and sane in the off hours, my passion is doing Rust on bare metal, and stripping away all the layers of complexity as far as possible. Down to, I'm booting into some Rust code.

Kim: Is there a particular goal that is driving you to this area of interest?

Nygard: I got my start on 8-bit microcomputers where you knew where all the registers were, and had direct control over all of the memory. Partly, it's for fun, but partly, it's also because I think, in some areas, we have built up a Jenga tower of complexity for solving general purpose problems. We have some specialized problems that may admit specialized solutions, and gain pretty significant benefits from straying away some of the complexity.

Parting Advice, and Help Sought

Kim: Is there any advice that you would give this community here at QCon? Is there any help that you're looking for? Carin, advice and help you're looking for?

Meier: The advice is, have fun. You can bring fun into your daily job and with your coworkers, so whatever you're doing during the day, try to bring a little fun into it. Help that I'm looking for? Yes, I'm always looking for good ideas. If you have any good ideas, data science, or especially machine learning, or writing interesting articles that you think I ought to be aware of, please pass it my way.

Kim: Mike, how about you, advice you would give and help you're looking for?

Nygard: I love Carin's advice so much. I'd love to just say ditto on that. Fun and whimsy is sadly lacking this year. We've all gotten remote. We're largely isolated from each other and from the world. One of the best pieces of advice I've gotten recently is get outdoors, reconnect with nature. It doesn't have to be around people, get around plants. You can't get COVID from plants, I think. Stay grounded. We work in a lot of abstractions, and with a lot of abstractions, and sometimes we can lose sight of the real physical world that's happening all around us. That would be my advice.


See more presentations with transcripts


Recorded at:

Jul 31, 2021