Transcript
Chaplin: My name is Stefania Chaplin, aka devstefops. I'm going to talk to you about securing APIs and microservices in the cloud. In terms of the agenda, I purposefully keep it vague. I'm going to talk a little about myself. What are we here to talk about? The people involved. How we're going to fix it? Now what, why, and summary.
Who am I? I used to be a developer specializing in Python, Java, Rest APIs. I'm one of those people that if you give a JSON to, I can tell you exactly how many squiggly and square brackets it will take to get to the value you're looking for. Then I went to the wonderful world of security. DevSecOps, AppSec, CloudSec, API security, doing a lot with open source security awareness training. Now I work at GitLab, the DevOps platform.
Netflix
What are we here to talk about? Netflix. It is a streaming website for films, TVs. Netflix are very interesting, or at least I find them interesting, because they've done some really cool tech things. If you have never heard of Chaos Monkeys, I would go check it out. It's very much chaos engineering. They literally will turn off data centers or processes, or they try and break the system to see how advanced it is. How easy they can get it back up and running, or how seamless the user experience can be. They're also very cloud heavy. They are potentially AWS's biggest customer. That's quite an accolade. They weren't always on AWS. Actually, when they were doing that migration, it did take seven years. There is quite a lot of content there, and especially with the number of customers they have, the number of programs, just the entire system, they make it run seamless for users.
Evolution of an API Architecture
How do they get content to customers? This is their evolution of the API architecture. They started off with more of a monolith, so you can see a big black circle. You've got all your devices, and they are accessing the Netflix monolith. You can imagine those red lines as, for example, API calls. If one of them gets compromised, that is going to be a problem because it's almost a one-to-one mapping in terms of one device goes to the architecture. When you look at direct access, what you're seeing is, for example, a CDN, content delivery network. It's a bit more spread out in terms of you have multiple different calls, they may be going to different servers. Then there's also this internal communication as well, which is most likely done by APIs, and you're starting to see this microservices architecture. This is definitely an improvement. Likewise, if something gets compromised, you want to have multiple levels of security.
Then comes our gateways. We can see on the top right, we have a nice purple rectangle, signifying our gateway. This is gateway aggregation. It's a bit like a fortress. It's a bit like a front door. It's checking who needs to go, who needs what to be accessed, what needs to be called. It's a very good way of adding an additional security layer. Finally, what we see with a federated gateway is we have those layers of abstraction. If we had a malicious actor, and they managed to infiltrate a line in the federated gateway example, it would still be quite modularized in terms of, maybe they would get the API call between the device and the gateway. Or maybe it would be within the direct access between the microservices, but unlike with a monolith where you have one, if you got one lying down, then you're going to have a bad time. With federated gateway, you have a lot more security, availability, reliability.
Microservices
When it comes to microservices, why are we talking so much? Why are they so linked? Here I have my microservices example. I've got quite a lot of bugs spread out between my different services and directories. There's one particular one I want you to focus on, which is the red ladybug near the top, because unfortunately, this is an API vulnerability. This is where it comes back to the link between API security and microservices. Because with the API security, this is what is linking the microservices together. You really need to focus on your API security if you are going for a microservices architecture, or if you're doing APIs in general, it's a good idea to focus on API security. Because the reality is, you're probably going to have a lot of bugs in your system, but you really want to be as secure as possible, ideally, eliminate the bugs also. Martin Fowler talked about smart endpoints and dumb pipes. Very much when it comes to API security, you want to be doing your authorization, you want to be checking the context, what's going across. You want to have a permission model.
Common Pain Points
What are some of the common pain points, when we have microservices architectures? I was in a conference in QCon in April, and we asked the audience, how many people are using microservices? Nearly everyone raised their hands. How many have more than 10? Still lots of hands raised? How many have more than 50? The numbers dropped. How many have more than 150? I think that was one hand left, maybe two. What you're having is, when you were breaking down your monolith into a microservices architecture, you end up with many microservices, many API calls, because that's how these microservices communicate, but who owns security? Where does security come in? Are you doing secure design? Are you thinking about security as you are creating these APIs? It's very important. We talk about shifting left a lot in DevSecOps. It's now to the point where we need to shift left to design. The latest OWASP Top 10, the web one, top 4, is actually called now insecure design. It brings into, for example, threat modeling. Before you write any code, create any API calls, you need to think about the security angles of this. This can be challenging, because with a lot of organizations, you end up with siloed teams, which means there's a lack of ownership and accountability. Who owns what, and how? The person that is writing the API, are they responsible for the API security? Yes. Certainly, security, are they responsible for API security? Yes. Fundamentally, everyone's responsible for security. It is another quality test. It's a quality check.
No matter what your role is within the software development and API process, you need to be thinking about the implications of your work. Are you being as secure as possible? Because we end up with delays, fails, or worse. What I mean by this is, obviously, you have delays or fails. That can just be annoying when you end up with everything behind schedule, over budget. Everyone a little bit stressed. It's more than just that. If we look at security over the last five years, it used to be a case where you would get your personal details stolen. It might be credit cards. It might be address, or in America, social security number. What you're starting to see now, especially in the last year or two, is a lot more ransomware. While before, it would be like, we're going to take your data, and that's obviously bad. People can replace credit cards. It's a bit harder to replace your social security, there has been large rises of fraud. It's bad. Ransomware is worse, because then you are totally locked out of your system. Then you are, as the name suggests, at ransom. Also, it's not just about the hackers, if you are working in an organization where your developers are unable to do their job, there is bureaucracy, there are checkpoints, there are problems, there is no sense of flow or joy within their work. It's a very competitive market. What you might end up with, is, you get people burn out, then they leave, and then the remaining team are even more burnt out. It's really worth thinking about how you do your microservices architecture, how you do API security, and how you do security in general, because otherwise, you might end up with worse.
Who?
Who are we talking about? I'm going to tell you about my silos. I've got three specific ones. I have my developers. These could be API developers. If you are doing microservices, maybe they are using infrastructure as code. I want to say infrastructure as code developers, that sounds a bit weird, but I think you know what I mean. Maybe I'm using Terraform, for example. Usually, you have quite a lot of developers. If you look at the average organization, you have about 100 developers to each security. With security, they very much are numbered, so yes, 1 to 100. When we think about API security, it is not the responsibility of security to go through and check every single one, or check everyone manually. You can obviously get security tools that can help to automate, that can help to speed up the process.
Actually, what you want to do is you want to get the developers writing secure APIs to begin with. Or when they're writing their Terraform scripts, are you scanning them? For example, Checkov is a great tool out there. Are you using that? Because microservices are awesome. The fact that we can provision into the cloud is awesome, but now one typo affects thousands of servers. You also have a final group, DevOps. You end up, 100 devs, 1 security, 10 Ops, and they're just trying to keep everything going. Especially when we are doing microservices, with Ops, they not only have to maintain, but when things start going wrong, for example, incident response teams, you need to be able to isolate what's going wrong. Preventing vulnerability traversal. Then getting everything back running with a seamless experience to the user.
Who Owns Security?
This is the thing, if we have these numbers, so we've got 110 developers and Ops to 1 security, who owns security? If you haven't read nine components of microservices, I really would recommend it. It is a great read, if you want an intro to the what, and the how, and the why? What are the best approaches and best practices for microservices? Martin Fowler talks about cross-functional teams. Instead of having all of your 100 devs in one place, you're going to spread them out. One of his other components was around focusing on products. What that means, when you have a product mindset, it's like, you're doing econ, or you're doing mobile banking, or maybe you're just doing an environment, for example. What this means, if you're the product owner for the e-commerce, you're going to be responsible for the security associated with that. Because when we have these cross-functional teams, maybe we have developer, QA, Scrum Master, but who owns security? Where does security fit within these teams? Which is where you need to start introducing automated tools, automated scanning. I think it was Forrester who said quite famously, manual processes are doomed to fail. If you've got one security person manually checking, you're not going to have a good time. It's really about how we get there. Also, the scalability. Because you don't want to end up like this, these silos and not having as good a time.
It's not even just about security. I was working in an organization. I tend to join two Slack channels in every organization. One is the gaming one, because it's a lot of fun. Also, the engineering one. I used to be an engineer, it's interesting to see what's going on. One time, there was a graph that was very high. Then it went cliff face, and then it was very low, over time. One of the cofounders, CTO as well was in this thread and being like, we've investigated what has been happening. Within AWS, there was a misconfigured EKS, so something was spawning, falling over, spawn full, full spawn, spawn full. That sounds expensive. It was. They said, we've managed to reduce our AWS bill from $6,000 a month to $1,000 a month. This is the thing, it's not just about security. It's not just about getting hacked. It's also about basic hygiene. Don't unnecessarily spend money by having a misconfigured service.
How?
How are we going to do this? I'm going to talk a little bit about authentication and authorization. Okta had a great definition for this. Authentication confirms users are who they say they are. Think of this like the front door. This is my bouncer, are you on the guest list? Are you coming in? Are you allowed to be here? Your authorization, that's, what are they allowed to do when they get there? For example, if you're going to a night, and maybe there's a music act, maybe your music act gets free drinks all night. That would be the nice thing to do. Similarly, unfortunately, the flat above has flooded my apartment slightly against the wall, and I had to get a plumber in. For example, that plumber, I'm going to let them in. I'm only letting them in the specific room with the flooding, but actually, I'm giving them free will to do what they need to do, because I trust them because they are obviously experts. They're a plumber. The way I think of it, authentication, who's getting let in? Authorization, what are you allowed to do? Also, how are you doing it as well? What are the permissions?
OWASP API Security Top 10
There is OWASP, so Open Web Application Security Project. They do a security top 10. They do it for API. They do it for web. What's worth noting with the API? Number one and number five are about authorization. What can users do? Number two is about authentication, which is, who are we letting in? If you want to focus on securing your APIs and securing your microservices, go to the OWASP Top 10. OWASP is an awesome resource. You can read about them, and you can get a better understanding of what to look out for, what are the quick wins.
Tips for API Security
I'm going to talk to you about some of the tips for API security. For example, when it comes to authorization, what can we do? Treat APIs like humans when assigning permissions, aka role based access control. There will be a case where we need someone to be an admin, but do we need everyone to be an admin? No. Similarly, I'm sure that every organization in their HR department has a list of all the humans that work at the company. Maybe if you're best practice, your IT department knows how many laptops and knows about all the hardware, and hopefully knows about the software. With executive order, the requirements have an SBOM, software bill of materials. You might even have an idea of all the open source you're using. Do you know all the APIs that are being used? Do you know the permissions of the APIs? Do you know the usage of your APIs? Because if we think back to the slide I had on Netflix where we had all those different lines, if one line goes down, how bad is it going to be? What data does that API call, and what permissions as well?
Next one, content integrity, confidentiality, reliability, and availability. In terms of integrity, message or content integrity ensures that when you're sending the message, it's not compromised after transmission. When it's integral, it means it's not been intercepted by a third party after the sender transmitted the message before forwarding it to an API. When we come to confidentiality, which is similar, content or message confidentiality ensures that the message received is verified. That the journey from sender to API has not been witnessed by unwelcome spies who saw the details of the message. There are different ways to ensure this. For example, with message integrity, you can use digital signatures to record the authenticity of a transaction. In this case, an app can create a signature using an algorithm and a secret code. The API applies the same algorithm with a new secret code to produce its own signature, and compares it to the incoming signature. Another method is cryptography. Public key cryptography is a method of using an encryption of a message that's nearly impossible to code without a corresponding key.
Reliability and availability, so this one's slightly different, because today's apps, everyone's apps are in the cloud, some shape or form. You have integrations to countless other cloud or on-premise services. Data will be flowing from one service or microservice to another and from one user to another. This creates a multitude of attack surfaces. Your app needs to guarantee that it is always available to respond to calls. That once it begins execution on the call, that it can finish handling the received message immediately without losing data and leaving it vulnerable to attack. One way of doing this is horizontal scaling by the API across multiple services, and by handing off the processing of the message to a message broker, which will hold the message till the API has completed its processing. What's also worth noting is on my slide, the circle with the APIs, one of them is actually called lack of resource and rate limiting, because you're going to have all these API calls. That's great. What if one of these API calls is a hacker? What if someone is trying to brute force you? You also need to be able to tell, what is a sensible amount of reliability and availability, and what is someone nefarious?
This is when I come back to, what can go wrong? Think like a hacker, because it's about this secure by design. It's the security mindset. If we can all become 1% more secure, that's going to make a massive, incremental difference to our security, versus if we just have one security for every 100 developers doing their best to keep the organization secure.
Why?
Why are we here? Why have you been listening? Why am I talking about this? This is a great quote by Brian Foote, "If you think good architecture is expensive, try bad architecture." This applies so nicely to microservices, and by extension the way that you're doing APIs, and also security. If you think good security is expensive, try bad security. Try being on the front page, try being locked out because of ransomware. It's really worth the case. It really is worth investing in your designs of this system, designing them securely. Making sure they're reliable, available, they have integrity and confidentiality. Because, yes, that's going to be expensive, but it's definitely the lesser of two evils. Because if you've got a bad system, you're just going to have a bad time.
Summary
Think secure when you're designing your microservices and APIs. You're going to have bugs but you want to try and minimize them. Do as much as you can to reduce attack surfaces. Authentication and authorization, who can do what and how? If you're doing API security, or you can extend this to wider security as well, who are you letting in? I used to work in security awareness training. If we validated our inputs, and we sanitized, that would fix a lot of the bugs out there, and with authorization, permissions. If you think back to human examples in an organization, you're not going to put everyone's contract on the shared Google Drive that everyone can access. Maybe HR can access that, and the individual user reads contract, for example. It's very much thinking, who can do what and how? Finally, think like a hacker. What can go wrong? What can go wrong when you think like a hacker? I mean this more, what can go wrong in the system? What's the worst case scenario? There's a great meme, it's like a QA walks into the bar, and he asks for one drink, minus one drink, 999 drinks. A lizard. Then the user goes into the bar asks for the bathroom, and then the bar bursts into flames, because it is a different request. Definitely, on a personal level, think about how you can help to reduce attack surfaces, because security is a shared responsibility.
Questions and Answers
Losio: I really like that you put some numbers in the presentation, so even the example of the silos with 110. What's, for you, the ideal number of security guys in that scenario, it's basically 0, 100, or no signers at all?
Chaplin: I think of it like a spectrum. Everyone can be security, because in order to do your job, just thinking about it from a secure approach, because security people won't scale. I don't think you need a one-to-one for every security to developer. If each developer just gets 1% better, then it can cope with a more flexible approach. Maybe 20%.
Losio: If you are the CTO or a VP engineering of a small or medium company where anyway, you hardly have 100 people, at least in engineering, and you hardly ever have a security guy. Let's say that you're working with a team of 20, 30 developers, what's the best way to proceed? It's just keep sharing the responsibility or try to get a focus of one person in charge anyway. What's usually the standard process to deal with that, when you're growing? When is the point where you pinpoint where you should really have a security guy in charge?
Chaplin: What you'll find probably is, yes, if you're working within a smaller team, like I say, with the spectrum, some people will be very passionate, if not paranoid about security. Some may not have as high a priority. I don't know why. What you can do is empower those people and also think about empowering the whole team. Giving them the time to do basic security training within the language or tool that they're using, so then they can start to level up. Then what you'll find is when you do your temperature check of who's interested in security, it will normally be that person that will start maybe a security champion program or potentially move into a more secure engineer style role. Obviously, it depends on the organization.
Losio: You're basically saying that, yes, a nice path is if someone from inside the team takes the lead takes the interest and growth to that position. More than really, I've reached that point where I need to do the checklist and I hire a security manager or a security engineer. It's more like, try ideally to avoid the silos too.
Chaplin: If we think about it, my ratio, 100 developers to 1 security, we only got 20 developers, that's like one day of security time. Maybe you rent a CISO and have that one day a week. I'm all about incremental gains and empowering people. Especially with developers, if you can just get everyone to just level up, especially if you make security gamified, then you're going to have a better time than if you have only a top-down approach, and then it's all about gateways and barriers. I'm on the more like holistic yogi side.
Losio: One of the examples you used, was you mentioned, for example, I need to limit the number of requests to an endpoint, so just not securing in terms of authentication, authorization, but as well, it's like, I want to make sure that apart from distributed denial attack, or whatever, I want to make sure that someone is not sending me 100 requests per second. Maybe it's ok for my system. Maybe it's not. It depends. I'm a developer, pick up a language, Java, whatever you want. I'm building my application, my product, that layer like securing my API, I'm building the API of my system for another team or to expose my microservice, whatever. That part, thinking about, for example, rate limiting, those kind of features, something that I don't want to be, I feel like reinventing the wheel. I'm sure that there are open source packages, solutions out there. What's usually the right approach to go that way? It's just go to an expensive enterprise solution that maybe for a small company is pretty hard to jump into.
Chaplin: With that one specifically, yes, lack of rate limiting. You want to also prioritize what you're limiting, so especially stuff that has high authorization, prioritize in that sense. I had the slide with the OWASP circle that has a lot of information in terms of, you can use the tools that are out there. OWASP has, I think it's called ZAP. That's their APIs for API fuzzing. Also for, if you want to do dynamic, if you want to have your application and start pinging it and see what happens. OWASP is always my foundational starting point. Then there are obviously other enterprise tools out there as well.
Losio: It doesn't mean to develop inside their security on top of your API, but it's more like try to leverage what you can find already for doing that.
Chaplin: The open source community is awesome. There's usually some really cool solutions out there.
Losio: Can you say a bit more about federated gateways and how can they help security?
Chaplin: In terms of with federated gateways, I think it's more of an evolution, so if we think about how things are moving on. It's about having different checkpoints, which is a federated gateway access. You can get them within cloud service providers. For example, the main ones, AWS, Azure, GCP. I'm a big fan of reading documentation, so that can give you an understanding of what it is, why it exists, and best ways of using it as well.
Losio: Is there any holistic approach you can refer us to?
Chaplin: If we look at the shift left of security, first we would do pentesting, and then finding things in production. Then we're like, why don't we maybe take it into our CI and then we're building it? Then, it's like, why don't we have IDE integrations? My whole holistic approach is taking a more security awareness approach. That's why I reference the OWASP individual numbers, because if every single person, no matter what the role, whether you're a developer, Ops, security, if everyone just gets 1% better, that will have a massive gain.
Losio: I know it's a very controversial topic, so I bring it to you. You closed saying that, really keep in mind the cost of best security. I really follow you on that. I fully get that message. I think it's a key point to take home. Often, think about also like a small company, new startup, or a scenario where you feel like security is slowing you down. I am careful with that word, because I know that's a controversial part. It's like, I want to be the next unicorn. I know that the chance is really small. Most likely, I'm going to fail. I might fail due to security, or I might fail due to having no customer. Why should I prioritize and work on security when I can just basically gamble in it. Many companies don't do it. It's not that they don't know about security, but they know that they have a challenge, but they just leave it for a later stage.
Chaplin: Obviously, I'm not in that mindset. That's the opposing view. Look at, for example, compliance, because you're right, especially with the smaller startups, as you're scaling, you might not need to be as compliant, but those usually have a good basis level. Minimum, just make sure that you're compliant with what you're doing. Then also, that can be the foundations of the security program. Because obviously, the people that write regulations and compliance, have a good like, these are the lowest hanging fruits. That would be a good starting point, if you're not willing to invest too much in security at this point.
Losio: I'm thinking now, more in the mindset of a developer, like there are many tools on the cloud, whatever cloud provider you're using, that helps you in security, service that kind of encryption by default, or can enable encryption, or a service that can add security component. As a developer you can leverage them, but often, there's a cost associated with that. How do you usually suggest to approach that problem as a developer?
Chaplin: There's a couple of things, in terms of with what you've done to describe the transaction, say sending an API call, do you encrypt? I always talk about prioritization. Be like, ok, is this API call going to be holding really sensitive data? That it has a larger attack vector. Is this a fun app I made so that every day I get shown a different cat on the internet? Obviously, those are quite extreme examples. Prioritizing, what is your high risk? What are your attack vectors? What are your tier zero? If it goes down, do you not want to go down? Then start there, and then work your way down. Because as you do more, you get economies of scale. Hopefully, you can get everything better.
Losio: I just want to really suggest an action item, it's like I'm a developer. I love this presentation. I just have a few hours, couple of hours, half a day tomorrow, what should be my first action item, to read something, a video? What would you recommend?
Chaplin: It depends on what you like to do. If you like to read, I would check out OWASP. If you like to try things out, also go to OWASP, but then go on OWASP ZAP. That's a fun way to try API.
Losio: You can try them there, basically. You can use it as a sandbox scenario.
Chaplin: Exactly. It's also got a lot in it. If you want stuff other than API security, it also covers lots of other types of security.
See more presentations with transcripts