BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage Presentations Coding Without Complexity

Coding Without Complexity

Bookmarks
43:07

Summary

Ellen Chisa talks about complexity in software and the need to remove it as much as possible. One of the possibilities is to remove something that is not absolutely needed as a way to reduce complexity

Bio

Ellen Chisa is the Cofounder & CEO of Dark, a holistic programming language, editor, and hosted infrastructure.

About the conference

Software is changing the world. QCon empowers software development by facilitating the spread of knowledge and innovation in the developer community. A practitioner-driven conference, QCon is designed for technical team leads, architects, engineering directors, and project managers who influence innovation in their teams.

Transcript

Chisa: Over the next 50 minutes, we're going to talk about what it means to code without complexity. To break that down a little bit, I'm going to tell you a little bit about myself and why I care about this problem. We're going to talk about what complexity in software development today does look like. Then we're going to do the fun part where we're actually going to get to write code for a while and look at what the future might be like. Then we'll break down some of those concepts and talk about how we can be removing complexity from the systems that we're all building.

Since it's the end of the day, let's keep in mind how we're feeling. I hope some of you are feeling like you've already had your mind blown, and you have something that you're really super excited about. I hope you're feeling adequately like you've spent a bunch of time talking about programming languages and infrastructure.

I'm happy to be here and talking to all of you, and I feel the normal amount of jitters that one feels when they're being attached to a microphone that they don't usually talk into. I started thinking about this a lot in my career, and as many of you as senior developers and leaders probably did as well. When I started spending less time staring at my laptop and writing code, and more time interacting with people. When I was thinking about how to handle my own feelings and keeping track of them and working with development teams, I realized that I wanted to use the tool that I'm most familiar with, which is software.

I wanted to use software to fix this. I sat down and thought, I'm just going to build myself a simple application. It will allow me to track my moods using emojis, which are what I used to track my moods anyway. It's what I send to my friends, it's how I think about it, and it was going to be pretty simple. I just needed to have a way to check-in and some aggregation. I thought this would be an incredibly simple backend. All I needed was a persistent data storage, an endpoint to post check-in, and endpoint to get all my check-ins back, and an endpoint to run some aggregation statistics.

When I sat down to do this over the weekend, it instead became a mess of lots of things I was interested in. I opened up Cloud Functions, I opened up the AWS console, there are 150 different options that I can think about, like we were talking about in the panel this morning. I thought, ok, I'm just going to go back to the way I would have done this a long time ago. I could get started in Rails, but Rails has an API generator now, and I'd be checking a bunch of dependencies and versions. Once I get this thing working locally, I still need to host it somewhere, so maybe I make a Repo and then maybe I run it on that cool new thing, Friender - but now I still need some data storage, and that's the problem.

I really only want to do four things. I want to be receiving some data, processing data, storing and querying data, and sending data. Yet, somehow I end up in this mess where I can spend weeks and never actually end up shipping an application. This seems wrong to be doing four simple things. This wasn't just in my side projects. In my previous professional life, I was a product manager for a long time, and infrastructure challenges came up across the board when we were working on things. I think many of you will face the same problems. When I worked at Microsoft on Office Mobile, it was great because we got to sit between Office, Windows Phone, SkyDrive, and SharePoint, but that also meant we were thinking about a lot of different teams.

At one point, while we were building Office Mobile, SkyDrive decided to update their API and we didn't have a formal agreement in place with them at the time. That meant that we actually ended up shipping a version of Office Mobile, where you couldn't see all of your documents, you had to then open the SkyDrive app separately and be able to open those. All because it was so hard to change API protocols.

Then over to Kickstarter, and that was exciting as well; there weren't as many problems of the previous type. Instead, what we had was we had our development environments, our staging environments, and our production environments. Pretty frequently, a lot of my work on backer experience meant having to check the checkout flow and understand how people were able to back projects and then process their payments. Unfortunately, frequently staging would get out of date, and there would be no live projects that you would actually be able to back.

Then as a developer, you would spend a lot of time thinking about, "How do I do move a bunch of stuff from our redshift system so we can have projects that are live on staging again?" Taking time away from thinking about "All I want to do is actually ship this improvement to the checkout flow."

Then I worked at Lola, a travel management company, and it was really important to us that our infrastructure worked well, and it was scalable. We knew that when we launched the product, we wanted to have it ready for a large number of users right away. I'm sure some of you have run into this problem before as well, and so we decided to start our databases, which seems like a good investment until we got to the point where we realized the core unit of our customer wasn't actually on the individual user, it was that we're going to be a B2B travel management platform. We had these challenges where we charted on the wrong thing, and it was, "Are we going to re-architect everything or are we just kind of going to keep cobbling this together?"

The really upsetting thing about this is, I feel like our general attitude within the industry is, it's got to be like this, and it's going to be hard. Fred Brooks wrote his essay in 1986, "No Silver Bullet," saying, "Until we get to the point where most of the things we're doing in software don't actually matter, we're not going to be able to make things an order of magnitude better." Right now, most of the code we write we need, and sure we do a little bit with memory management or garbage collection that's about the device that we're running on rather than what we're actually trying to serve, but it's not so much that we can get an order of magnitude better.

I think things have changed a lot since 1986, and one of the big ways that they have is that now we're all primarily running on distributed systems, rather than just writing our code that runs locally. When we did that, we kept up the same mentality that we'd always had for software development. When we think about the first tenant of the Unix philosophy, it makes each thing do one thing well, and that's what we still do. When we're adding tools to our ecosystem, we make a tool that solves one problem for us and does one thing well.

Where that's gotten us to is, we've actually gotten to the point where we are spending a lot of time, and all of you and your teams, are spending a lot of time on the things that we don't want to be. Stripe released their developer coefficient report in 2018 saying that nearly half of the time we're spending as developers is on dealing with bad code debugging, refactoring, and modifying rather than delivering new business value.

Accidental Complexity

When I look at that, that really seems like we have gotten to this place where it's no longer a world of 1986 where there's no silver bullet, we have accidental complexity absolutely everywhere in things we're doing. That's a little vague so let's break that down into three categories.

In particular, accidental complexity in software development today falls into three buckets. The first one, the one all of us in this room probably think about a lot, being infrastructure. The second one being deployment, and the third being APIs, and specifically how we talk to external APIs from within our applications.

Infrastructure-wise, have we seen this map before? My favorite thing about this diagram is that at the top, it has a little note that says, "Overwhelmed, please see our other map," even though this is meant to be an overview. I think a lot of you in this room probably actually know more of the things on this slide than I do, but I think very few people don't feel overwhelmed when faced with 400 choices of technology they could use when trying to move to a cloud-native infrastructure. The thing about this is, if we go back to our beginning of we're just trying to receive, and send, and query our data, and store it somewhere, yes, these things sort of help us solve these problems, but they aren't one thing that lets us do all of that. It's this giant ecosystem just to do those four things simple tasks.

When we think about deployment, I enjoyed the joke earlier that if you ask 10 people on a team how something gets to production, you will get 11 answers, which I think it's true. Similarly, coming from a product background, a common interview question to see if PMs were technical enough was always what happens when you type a URL into your browser and how do you get a response back? I think a more appropriate question today would be the same one of how do things that your developers that you're working with write get into production? When you think about this, really, all you're trying to do is ship something, and to think about how you're shipping it you have to remember these 34 different steps that you're going through, and someone has to maintain and think about this.

Then APIs - APIs used to work just like function calls. I don't know about all of you but today when I'm building with API's, I end up having one thing up with all of the documentation, and then my command line up so I can do a bunch of crawling, and then probably another API tool to be able to inspect things. Then a whole bunch of other stuff that I ended up needing for some reason. Today, it no longer works just like a function call. We end up in this world where we end up having to use a bunch of tooling just to do something simple, like talk to an external service.

Coding Without Accidental Complexity

That's what I want to think about is what happens if we don't think about these things? What if we don't need that slide of 400 things? What if we don't need a 34-step deployment process? What if APIs, once again, are like calling the native functions from within a language? That's what we've been doing with Dark. Over the last few years, we've been building a holistic language editor and infrastructure for doing this. I'm going to show you what's that like from the perspective of what does it mean to put all of these things together at once. Then we'll go back and talk about how some other tools are doing pieces of that.

Since this is a language track, I know that everyone always wants to get into the nitty-gritty of what is the syntax of this language, and how does it work? Before we actually demo things, it looks like a normal language. It just looks like code. This is a question that many of you may have been forced to answer at some terrible part of your life. From the perspective of this, Dark looks like any other language, you have conditionals, the language itself is functional.

Time to Write Code!

With that in mind, let's actually write code and talk about what a world without accidental complexity looks like. First thing, you will note that I am entirely in my browser. I have not done anything special for this demo before I got here, except I did pull out all of my tokens that I'll be using during the demo into function so I'll just be able to call them and not show you all of them.

First thing, in Dark when I'm setting up a new API endpoint, all I have to do is specify my route, the method it is responding to, and my response. This is now up and hosted, you could actually all go to it from your phones if you want to. Rather than my process before when I was thinking about setting up a simple and blue tracking application and then spend hours trying to figure out how to configure infrastructure and got down my rabbit hole of all the tools, I have something that's just up and it works for me. If you're wondering about how this works under the hood, it's actually a very standard thing that you build for yourself into the code. It's running IGKE and the data source are in Postgres.

Then if I want to make slightly more code, and want to greet visitors by name, I can add a URL parameter. I will say hello to QCon, and you can see that my response is still updating. Now some of you can create yourselves.

The cool thing about this is we can now see what our expression evaluates to for the current trace coming in. One of the things I don't usually show this in demos, but since it's this audience and I think this is fine, if we wanted to make a change to this code, and since we're writing live right now, if you had a bunch of end users, like right now yourselves, looking at your application, you wouldn't want things to just change without intentionally making a decision.

In Dark, I can add a feature flag, I can name it, "changeGreeting." Then we can set a condition - we'll change, say, only if the name is mine. Then I don't want all of you to be able to do this, and then we'll just change it to be "hi" instead. Now when I go back over and I greet myself, I get "Hi, Ellen," but the rest of you are still getting "hello" to whatever endpoint you wanted. From this perspective, we've moved from this idea of, I have an environment where I'm writing locally, and then I'm shipping to staging, and then I'm shipping to production, to, my application is working on production, but I am still doing all of my changes behind feature flags so I won't be creating a bad user experience.

What does this mean when you're building a real application instead of something that's just saying, "Hello World?" Let's make a new one where what we're going to do here is - frequently when we're developing, we have some piece of our application already built, in this case, a front end. What I'm going to do is, I have a sample front end that I built ahead of time, it's a react application. If you want to see the code, it is not very interesting. All that I've done in here is that I've set my API host to be pointed at Dark where we're using it. Then I use a command-line app to upload it to where we're working.

This is what that looks like, and you can see that because we're actually working live, we get this for our foreback. Instead of doing what we would usually do while developing, which I find to be extremely tedious and slow, and having to go inspect and figure it out, and then figure out our hosts' endpoint failed, and then go build that, and then deploy, and then hope that works, we're actually hooked up to it already, my dev tools. What I can do is when I go back into Dark, I can see that this endpoint is failing and create that endpoint directly. I'm building based off of what my application needs from the front end and from the end-user experience, rather than having to do a bunch of stuff and see if it works. Then we can actually try it out. If I add real data, I can go back and I have this here.

One of the things that seems a little bit silly about this demo is it feels like it's a toy application because it's relatively small, which is what you have to do to get it to work for this, but I want to make it as real as possible. Instead of just having a host endpoint, let's actually make a real data store. Let's give our hosts names and phone numbers. This way we can tell them when people are visiting them. The other piece of this is now we want to be able to put someone into this data store. Normally, I would have a bunch of database tools that I'm using with some database that I had to go set up completely separately from my browser. In this case, I can just make my tools side by side and it'll work exactly the same way my HTTP handlers are working.

What I can do is I can add myself. This is my Slack ID for our company Slack so will be able to send me actual messages. In Dark, it's a little bit hard to tell while I'm typing, you're editing in SAD, which is part of what makes us know you won't break your production infrastructure. When you have this, you can actually do a bunch of cool stuff because it knows you're not just editing text. I think there was a comment earlier about structure is text first code, and this is very much not just text. I'm going to extract this so I can reuse it. If I pull my same variable out here, and then I rename it, I get all the nice auto renaming stuff that you might get, which I think is very nice as a part of the built-in language.

Then, last piece, put this into the data store. Same as before, we could see the results of our incoming traces that were coming in. We can choose to execute functions that have side effects. In this case, if I want to insert something in my data store, I can do that. As soon as I've done so, my data store locks. Same thing with the handlers and needing to edit live code-behind feature flags. If you have real data in your data store, you want to be running a migration.

All right, so now that we have that, we just need to quickly edit our endpoint to be getting all of the hosts out so we can do the same thing here. We can run it, see what it is. We don't actually need every single host, so we can pull just their names. There that is, it'll be just me. We go back to our frontend application, that works. If we check in as QCon, same as well. Make my visits endpoint, and then we will work with that. The interesting thing about this is since we've passed in real data, we have it available to us, we have Dude here, who is apparently already checked in. What this means is that rather than having to guess the shape of our objects, we know exactly what they look like as we're doing it. Let me just generate a key and make our new data store, which is going to have the same shape, so then I can double-check this.

If I make these visits, I get this relational link to my business data store. I can go back and forth between them if I want but the interesting part about this is that I can compare that easily to the shape of the requests that I have coming in. In the future, you'll be able to use these to generate types and use types as the background. Again, we will run our request and we can see that that is now also locked.

Last piece of this, right now technically, if you go check and it'll work, you'll get a confirmation because you're getting a return value. I said I want to just show you what it might look like to be easier to operate with APIs, so let's do a little bit of that. In Dark, there is a built-in venting system, and so I can emit the request I got to be able to send a text message in a worker, then same thing will emit the same event to be able to send a Slack to you.

Let's do the text message first. You can see once I execute them, it basically makes a to-do list for me. When I'm doing my send text, much like I could work with my request directly in my HTTP endpoint, I can work with my events directly here. I have my hostname, and then let me get my host back out of my data store. Then again, we can run this and check that it works, which it does. You can see that I really have this feedback loop all the time because I have this data where I can make sure things are working the way I want them to. Then I want my host phone number, and you can see here I get that autocomplete for the fields like any modern editor. Then, last piece of this, I want to be able to send the host a text message to say that they've gotten a visitor. In Dark, I can just call Twilio send text, I will pull in my SID. This is the thing I showed you that I pulled up before. We are going to send it to me, the host phone. We're going to say event.visitor is here to see you.

Now is the point where you actually get to start sending text messages to my phone. The cool part is, much like we ran our DB query functions before, we can use the same thing with our API endpoint, run it, and then see the message. In this case, we actually get an error, and it's because I used the phone number wrong. Now it's easier for me to debug because I can see the phone number is this thing, which is not what it should be. It's because I put my SID there again. Run my text message again and I should get a text that says "JC is here to see you." I have a text on my phone. The iMessage syncing between one's phone and one's computer leaves a little bit to be desired.

Similar, you're not going to have every single API wrapped in a language out of the box, and so with Slack, I haven't done that yet. I'll show you the equivalent experience when you're using the HTTP client library instead, it's basically the same. Copy and paste works by expression, which is another nice thing. Then what we want is our channel, which will be our host.our Slack Id. Run this, make sure we actually get this back, we do.

In this case, we're going to want to build a header so Slack token, I also pulled out ahead of time. The API we're using, we're going to want to use the JSON content types, so we'll do that. Nice thing about this is we end up with a nicely formatted header. Let me just pull up the URL is postMessage and so now it should be, client::post url, token: slackToken, channel: channel, text: event.visitor "Is here to see you." We'll make this empty and we already built our header.

Same thing, we've now integrated with an API from scratch using an HTTP client library rather than having a nice wrapper. When we do this, we will get a bot message that selects Chisa from host "is here to see you," so I'd say that seems to be working for us. We can also see we have pending events coming in that will continue to process as people visit the web page and use this. For all of you who are not sitting there following along and resubmitting names over time, if I do this, I will get the confirmation, I will receive both a text message although probably not, and also a Slack message.

I wanted to show you that not because I love building all the sign-in applications, although I seem to do this a lot now, but because I think that's the power when you take away all of the complexity at once is you go from something where you would be spending way more than 10 minutes just to get an environment set up to being able to ship a working somewhat functional clone of what would have been an early stage envoy or other office check-in system startup. Projects go from being something that you need to spend a ton of time thinking about to being something where you can just spin it up, and try it out, and work with it whenever you want, whether it's a prototype or something you want to continue to work with. Let's hop back over and talk about that a little bit.

When we're thinking about no accidental complexity, it's just what you saw. You're getting compiled infrastructure where you aren't thinking about as you're writing code. You are getting an environment that I call deployless, much like when we talk about serverless, there are still servers, with deployless, there's still a deploy. It just happens in the 50 milliseconds after you type and before you can switch back to see what happened. You have something that acknowledges that the world that we're all developing in today is really focused on all these external services and connections that we need to make.

Let's look at what each of those pieces is compared to what we have today and some of the other ways people are approaching this as well.

Infrastructure Complexity

Infrastructure complexity, I think we probably all remember as being a giant slide of many services we can use to be able to do our basic needs for back ends of sending and querying and receiving data. We really have boiled down into a super small set of core use cases. I showed you four of these, but not the fifth. When you're building something without accidental complexity, you're thinking of what is it that I want to be doing?

It might be that you're having a persistent data storage. It might be that you are running something on a schedule cron. It might be that you have a background worker that you want to be responding to events. It might be an API endpoint, or it might be some sort of tool that you're using. I would encourage you to think about the applications you're building and figure out how many things you're doing can be boiled down into these categories, or don't. If you have a bunch that don't, I would be interested to hear about them because I found that most of my projects and most of the work that I've done over time is really a core set of services that I'm working from.

This might seem familiar if you've used other popular products like AWS Lambda. When you're thinking about this, even though Lambda technically has many, many options of what you can do, if you really think of it as the core of being your code runs on a trigger, you can also connect that trigger to be based on time or you can connect to that trigger to be based on an HTTP endpoint. You end up at a very similar set of core functionality that you're able to set up and keep working with.

Second part of this, one of the things we didn't talk about very much while demoing but is interesting is, when you have a bunch of independent services like this, it becomes really important to know how they connect to each other. I don't know how many of you have started a new job recently or have hired someone recently but I feel I spend a lot of time going to whiteboards and drawing out a diagram of, "This is how our code really works." You can't just put someone in front of their text editor and say, "Here's the file structure." That's not how people learn. That's not how people think about the impact they're building.

By having your infrastructure work like this, you can end up having something where you can see all of the connections between what you're building and how they relate to each other. In modern infrastructure technology today, a lot of other people are trying to build systems like this. CloudCraft is one example where you have a system that allows you to monitor how your infrastructure is connected together and how your services are interlinked.

Deployment Complexity

Ok, that's a big one - deployment complexity. We talked about this before. This is the big, scary one. Right now, we have these deployment pipelines, where most of the people on our teams don't necessarily know how the entire deployment pipeline works. We keep adding more things in the name of adding safety or hopefully making things better, but every time we add something, it's another thing for people to think about. Or, it's another couple of minutes they spend not being able to deploy. I spend a lot of time at Friday Retro meetings where people are going, "Just has to get through CI, and then it'll be there and then we can see it" and that comes with a real cost.

Of course, I just showed you this and how it works. In Dark, you can just deploy immediately. The scary thing about this is the fact that it is live and you're on your production infrastructure, and that feels weird. I think the really important thing to think about here, and that I would encourage all of you to think about and think about adding to your team, is when we came up with the idea of having development environments, and staging environments, and production environments, and creating safety by separating out "where things live," we didn't have the modern best practices of having feature flags. That was really the only way we could control who was able to access things.

Now that we have feature flags, that's a great way to control who is able to access things, and there's still limitations here. We get a bunch of extra safety features in the language where we can deploy and it's less scary. From my perspective, if you can set it up so the default action for thinking about how you're delivering to a user is behind a feature flag rather than behind your environment, you're actually thinking about the thing that matters. None of your users care about if something is on dev, or staging, or prod, they care about, "Can I see it or not?" If it's on prod, and they can't see it because it's behind a feature flag, that's fine, it's not there for them.

Similarly, that causes less risks for you. The longer you let something bake in dev, and you have this giant pull request sitting on your machine that isn't out there, the more likely it is that it's going to be hard and you're going to have a bunch of merge conflicts when you get it in. The longer something is in staging, the more secure people will think they feel about it because they've been testing it for so long. Staging isn't necessarily a proxy for the real world. Same thing is when I was talking about Kickstarter, and not ever having live projects on staging that means it's a very different place than the live Kickstarter site.

If you can work towards the world, where your primary thought about what is safe in your deployment process is operating behind feature flags, you've gone a big step towards not having to think about as much of this deployment complexity.

I would say that one person who has spent a lot of time thinking about this in the modern ecosystem is LaunchDarkly. They are specifically a feature flag provider. If you are trying to say, "Ok, let's spend less time on this deployment part and more time investing in feature flags," this is a great example of a provider who's really focused on that.

API Complexity

Last one is API complexity. My experience of APIs, opening many things to try to get something to work and it works in a terminal or it works in your API test provider and it doesn't work when it actually gets into your application. Plus, when it's actually in your application, then you need a bunch of stuff for back-office and retries and all of those other things you need to get it to work.

One of the things that's nice about having an environment, and particularly a language, is you can start to compose APIs such that you take more and more away from what individual developers need to use. There's the basic, you can talk to any API through a rest request, for the most part, and use HTTP client post. You could think about how you could build internal tooling for developers at your company such that they would be able to call them much more closely to functions like the Twilio send text I showed you.

The extension of this, if you want to go a step even further depending on the teams and their different needs, is you could even pull out your secrets and your credentials into another place so that developers are only really thinking about what is the phone number I'm sending a text to and what is the text that I'm sending? Not necessarily thinking about the actual configuration around the other requirements for the API.

A company that spent a lot of time looking at this space, and particularly the space of being able to think about, "I just want to get this API up and running" is Transposit. If you just want to play with it, they have a really fun sample where you can have your Google Calendar linked to your Slack very easily using a minimal amount of actually writing code. It'll say, "Ok, if you have vacation on your Google Calendar, I will set your Slack status to being away." People are really trying to move towards this world where we're not all writing the same logic for handling these external APIs, we're instead able to write the stuff that we care about.

The last piece of this, which is kind of cheating, my favorite thing about being able to integrate with external APIs is actually true for most of the code here. In this case, you get the text message response from your API, and you can debug like we did in real-time. In Dark, you get the same experience for other functions like we were seeing. This is actually the emoji application written in Dark, and I can look at check-in and debug what has gone wrong with it. There's certain emojis that break, which is really interesting.

From this perspective, the ability to have this real-time feedback loop, whether it's around the API's you're using, whether it's about the results or functions in your language, whether it's being able to see what would happen for a specific trace or debug a specific instance you have and being able to do that right away is actually the best way most of us learn. Right now, we think of those tools as being primarily for novices or people who are learning to develop. When we think about that, one of the really cool things about Glitch, if you haven't played with this yet, is you can go and you can build a JavaScript application. Then it just runs, and you can change it in real-time. You can just go and edit the code and then hit go and it just works, and you can actually see what's happening.

One of the challenges we've run into with backend systems today when you're sitting there is, you read a bunch of code, and then you go through this to deploy even or you build it locally and you hope it works, but you have this long lag between when you do something and when you see the potential results. I think one of the things that really removes complexity for people is not having to think about every single thing they're doing as they do it because they're going to get the real-time feedback.

Glitch is one company that's doing that. Another one, if you just want to play with it, is Replit where they spend a lot of effort on supporting a bunch of different languages, so you can get this real-time experience as you're writing software.

No Accidental Complexity

That kind of breaks it down. I think when you think about the future of the entire software ecosystem, it's really exciting when we can get to a world where you're able to build with no accidental complexity whatsoever, and we can all build things in 10 minutes, and then ship them to our users who will be happy about it. That's, of course, not the world we're necessarily all living in today. If you're at a small company, maybe it is, if you're working on a side project, maybe it is. For the rest of us, it's more about, we're going to need to make incremental steps towards having this better developer experience and not wasting so much of our time doing the same things over and over again.

Three tenants of that will be this idea that for our application developers, you will have this compiled infrastructure that just works for you instead of spending a lot of time thinking about it. You will be able to get this real-time feedback and not think about a 34-step deployment pipeline, it will work for you. You'll be able to use APIs like functions.

When you're leaving the room, and you're hanging out with each other tonight, and you're talking about it, I hope you'll take away two things from this. The first is when you are thinking of adding one of those many services from our cloud-native ecosystem, or something new to your deployment pipeline, or something new around connecting to external APIs, you'll think contextually about it and figure out what problem you're solving rather than just thinking about this is the cool new thing in our ecosystem.

The second, and perhaps more important part of that is, every time you're thinking about adding something new to your infrastructure tooling, when you're adding it, ask yourself, is this removing some existing complexity from what I'm doing that will make developers better able to deliver software as we're building? If that thing isn't removing complexity, ask yourself, is there something else from our toolchain that I'm going to be able to remove as a result of this to make sure I'm at least not adding to the complexity that our team is thinking about?

Questions and Answers

Participant 1: If I wanted to get started using Darkly, are there any talks or anything like that? Do you guys host any informational stuff to get started?

Chisa: Currently, we're in private beta. You can come to me. It's mostly about scoping the application that you want to build and making sure it's something the platform supports effectively today, and I'd be happy to talk to you about that.

Participant 2: My question is also based on what she asked. How do I get started on Dark and a few of the tools you just mentioned? Are all of them open source? When you talked about the infrastructure thing, I was amazed by how you have taken out almost 10 steps to deploy the application and get it immediately. How do we do more research on that and how is it happening without any deployment, container? I want to know more of that part.

Chisa: I'll repeat it back and see if I'm getting it right. You're interested in more of how it actually works under the hood. From that perspective, what happens when you're writing code in Dark is when you're editing a specific expression, you're never editing just a small piece of text, it has some actual implication where we say, we know what expression or what function you're adding to your code, we will deploy just that expression or just that parameter that you've added to that expression. Then what happens is that then goes through a very standard deployment process on our side. It's orchestrated using Kubernetes. It's hosted on Google Cloud, it's actually quite boring under the hood. The only interesting piece is the abstraction on top of it that allows you to not have to think about that. Participant 3: You mentioned Google Cloud, do you support AWS or Azure?

Chisa: This is not a "use Dark" marketing talk. This is just, I don't have another way to show all the things at once yet, which I hope other people build them too. Right now we don't, it's just on Google Cloud, just to scope what we're doing. There's no reason that someone couldn't build something like this, or this couldn't be multi-cloud.

Participant 4: How does testing work with unit tests, integration tests, that kind of thing?

Chisa: When you saw the requests coming in for when I had an endpoint, and you could see the live data, the idea would be that you would be able to save each of those incoming tests as a unit test if you specifically wanted to rerun that and have it automatically saved in that way. It's a general-purpose language so if you want to write other tests, you can.

Participant 5: This isn't about Dark itself, but in terms of reducing complexity and removing this stuff, how do you deal with developers or engineers not leaving well enough alone and always wanting to dig a little deeper into something and always having that one piece, "this doesn't work exactly how I want it"?

Chisa: This is half a CEO PM answer and half just person who is an engineer answer. I think the thing that I found to be most effective in this case is weighing the actual trade-offs. Yes, maybe this one-piece isn't exactly right, but if you're getting 10x the time savings and giving up that one piece, that means you can deliver 10 times more stuff to your users, you can learn 10 times more new things. I think it's always about framing the problem such that people can see the benefit in the context of the trade-off.

Participant 6: I've got this feeling it feels like the Visual Basic for Windows. It was very easy for users to create apps, but then it unleashed the whole mess that people are still trying to deal with it. Aren't you a little bit scared trying to create something like that again?

Chisa: I think the reason I'm not scared is I think when I look at all of the choices we're making, and the fact that we put a whole lot of very smart people in a room to try to talk about how we're going to have programmable infrastructure and the future of infrastructure. I think things have gotten to the point where the solution we're using probably isn't the right one and we have to come up with something else. When I look at that I go, what is the coolest thing I can possibly imagine for where we could go, and what would the best possible developer experience be, and what would it take to be able to do that?

Participant 7: With your feature flags and your elements, as you're promoting those from being behind the feature flag, do you have the observability and the metrics to look in to see what's being used from that standpoint, almost like an operational to define which code starts to kind of fade out?

Chisa: Yes, for sure. I think this is actually a really great point; while all of the infrastructure we're running on is accidental complexity that we have to do to be able to ship our systems, knowing how our systems are working, and who is using them, and how that works is really essential to businesses and people need to know what's going on. To that level of granularity with the feature flag, you can write completely custom logic for it. "This is only for my name", or "this is for this header," or "this is for the set of alpha users." Then you would want to look at how that's working across that group before you'd make the decision to promote to the next one.

Participant 8: Can you just flip them in?

Chisa: Yes.

Participant 9: Can you have multiple people work on the same code?

Chisa: This is one of the things that's really a bummer - it's hard to demo when you're by yourself. The other thing that being live and deploying everything incrementally gets you is you can end up having a collaboration experience with developers on your team that is much more like a Google Docs-like experience where you can both be editing things and working on things in real-time together rather than saying, "I'm going to go off and build this thing" and then come back with it. You can put those behind feature flags as well.

 

See more presentations with transcripts

 

Recorded at:

Dec 17, 2019

BT