Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage Interviews Web Security with Aaron Bedra

Web Security with Aaron Bedra


1. Hi, we’re here at GoTo Aarhus 2013 with Aaron Bedra to talk about web security. Hi, Aaron, would you like to tell us a few things about yourself?

Sure, certainly. I work at Braintree Payments, headquartered in Chicago in the United States, also Venmo, which is peer to peer mobile payments application, I work between both of the companies, really with a focus on transactional processing and digital commerce.


2. Working for a company like Braintree, which handles online payments, I guess you should be probably at the cutting edge of what security is on the web these days?

Certainly, it’s a challenge. Whenever you’re moving money and especially large volumes of money, you become a target and folks want to see what they can do to take that money and in particular it’s not stealing any particular money but really stealing the credit card numbers and reusing them, because stealing a single sum of money once is not as effective as stealing a number that can gets you access to as much money or whatever that person’s credit card limit was.

Dio: I remember in the old days people would be trading credit card numbers.

It still happens.


3. What’s the latest trend?

They are actually very advanced, essentially organized crime around credit card fraud. They are very good at what they do, they adapt quickly, they have very complicated botnets that test stolen credit cards and relationships that they sell on black markets, credit cards, depending on the country they are from, will go for different rates, based on an average value of an account, that sort of things, anywhere from 10 to 50 dollars for a credit card number. And obviously bulk deals on all kinds of stuff like that. But the thing to take away is that they are very good at what they do. Obviously you have some outliers, but the professional groups are made up of very skilled people and they are very good at fraud.


4. [...] What are the stuff that you are usually dealing with in production and in reality?

Dio's full question: Traditionally, web security was all about protecting your server and making sure you didn’t have cases of SQL injections, some people worried about stuff like XSS but not many, and nowadays you have so much fractionality on the client side, so many ways to have security breaches there, what are the stuff that you are usually dealing with in production and in reality?

So, we have two pieces to deal with, we have our gateway which is more of a mechanical conversation, a server will talk to our gateway via a request, those are mechanic generated requests. So, as far as client side protection there is really just a quick negotiation over SSL and a PUT- POST request that goes to us to get a transaction. So, that is not the focus on the client side specific details, but there is one piece there which is SSL certificate pinning. We actually provide libraries that work with our API and it does a lot of things to help protect the client. One of them is re-bundle certificates and do certificate pinning to ensure, to help protect man-in-the-middle attacks. The other piece is that there is an option where you need to verify the peer and the hosts, basically valid certificate, valid hosts, a lot of times that will be disabled by accident or sometimes by default, depending on the library that’s used. Libraries we ship ensure that that’s enabled by default, so that’s actual protection against man-in-the-middle attacks. It’s a common mistake, it’s not on purpose, but if you’re testing your own servers maybe you’ll generate a self-signed certificate and then your mechanical code will break because it couldn’t verify the certificates, so you say “oh, don’t verify that, it’s ok, we’ll trust it for now” and then you’re in production and you forget to turn that back on, so it’s more of a mistake, it’s an accident but it still happens, so what a lot of times we’ll do is when we ship these libraries we turn that on by default, you can’t turn that off for our libraries.

So you are going to be talking to our servers, even our test servers have valid certificates so you don’t have to worry about that exchange in that particular place. So a couple of sane defaults is really what we offer and that helps the mechanical client side be protected. Now, the other part that our gateway has the control panel and merchants will come in and do these transactions, create users, the general management of the account, and that is web application. So, all the standard web application security applies to that, so there we do have lots of regular testing, we have third party code audits, we have our own code audits, we have our own penetration tests, that kind of thing, it’s very good to have validation from third parties, multiple third parties, that things are safe. Also, at the very backend, not with the client side but in our infrastructure fall into the PCI compliance mandate. We do processing, our latest numbers are about 12 billion dollars in transactions a year which is definitely a large volume, but we’re a level one service provider under PCI. So, that’s also very important to make sure that we do everything we can to protect our whole data.


5. You mentioned certificate pinning, is this something available across all browsers these days?

So, the certificate pinning is not as much browser based as it is client based. If I have a request, let’s say I take some language and I make a request against another peer, so I make a request to let’s say, what’s going to happen is I am going to be presented with a certificate and I need to verify whether that certificate is valid. Now, I can use a normal SSL channel, which will be a negotiation exchange, there will be certificate, the whole public infrastructure will hopefully do the right thing.


6. At least you have to make sure the first time you get it right?

Yes, and you provide extra security by actually bundling certificates with the browser, but also in particular for us we bundle them with our client libraries, because otherwise there is no additional verification mechanism, you have to trust PKI the way it is and that can be dangerous. We basically try to eliminate the x factor by using certificate pinning which is that man-in-the-middle attack.


7. You also mentioned getting certified by third party organization about your security practices, there are organizations like OWASP, which one of those do you think would apply to smaller companies and people doing web stuff that they care about, not necessarily transporting large amounts of money, but regular?

The OWASP program is not really a certification program, it’s more of a set of guidelines around web application security, it’s a really great place to start, it’s free information, it will always be free and open, there’s a lot of great defaults, there’s a language specific tutorials, they produce the top ten, which is a top ten of most common vulnerabilities found in web applications. It’s a really great place to start, they even have WebGoat, which is a vulnerable applications exercises that you can go through to learn how to attack a site, which helps you learn how to test your own, a combination of offense and defense when it comes to security is very valuable, going all the way, one way or another won’t get you the information you need to really be comfortable with your application.

So you need both sides. So OWASP is a really great place to start. When I say certification it means PCI certification in the context of Braintree, the third parties audits we do is not really is not really part of any certification, they are part of just our general awareness of what’s going on in our environment and making sure we’re not forgetting anything. It’s very easy when you see something a lot to eventually get tunnel vision or ignore something because you think it’s a certain way and it might have changed, so getting that additional verification from somebody else is very important. I encourage anybody to get third party security audits, maybe not as frequently as we do, we do it actually quite often, very regular schedule, but maybe before you release an application for the first time you have a penetration test done. There are lots of firms out there that do it, they can give you a lot of information on how to approach security maybe a little bit differently, or if they find vulnerabilities, that’s important too. There’s that separate class, there is vulnerabilities which are just bugs in your code and there are flaws which are just architectural design problems that are much bigger, it requires rewriting a portion or re-architecting the application to work a different way.

There are different classes of bugs, a penetration test won’t usually find a flaw in particular, it might find a security hole or vulnerability that you would eventually find as a flaw, but that source code review, architecture review, threat modeling, that stuff goes along with the black-box penetration testing. So if you really want a full package it’s source code audits, architecture audits, threat modeling and black-box penetration testing. You do all of that, you are off to a good start. So, it’s a tough world and things change every day. There are lots of exploits that are found every day, there are mailing lists for a lot of them. The CVE list has a mailing list especially for open source software, there’s an OSS-Security mailing list and every CV for open source software has an email and that’s what goes through there, you can keep track and see what tools they’re using as they announce their updates and patches and vulnerabilities. Keeping up with data that is really tough, but it’s not about patching servers all the time, sometimes it’s a library you use or a language that you use, and you need to keep up with all that, it’s very important.


8. With the evolution of server side frameworks and the new capabilities that the browsers have, I guess the threats have also evolved and changed, could you give us an example of something that wasn’t a threat in the past but is now?

It’s a good question. In general, the same threats are there that have always been around, maybe more sophistication around attacks, ways to bypass detection of attacks signatures and things like that, that’s evolved, it’s more the offer has improved just a little bit, the problem is we haven’t raised the bar. If you look at the OS top ten, which is that top ten list of vulnerabilities commonly found, 2007, 2010, 2013, you won’t see much of a change, things shuffled around in the list, but more or less it’s still the same, which tells us a lot. It tells us we are not getting better. So attacks don’t have to become more sophisticated because we haven’t caught up, we haven’t raised the bar, so in order for those to get more advanced, we have to raise the bar a little bit, which is both good and bad, right? We’ve pretty much been able to detect and isolate a lot of common attacks, so the information is out there on how to protect yourself it’s just not commonly used.


9. Having said that, are there parts of the web stack that you feel are particularly vulnerable and could evolve to become more secure or things that we are using and we shouldn’t be using at all?

Sure, I think when you are choosing a framework it really is about reasonable defaults, finding a framework that helps you protect against cross site scripting and SQL injection at least by using the framework defaults, it’s a good place to start, right. Something that has conscious choices, there are quite a few of frameworks that are doing that, most frameworks emerging at this point have started to move in that direction, they are getting a lot of pressure to have reasonable saying in security defaults because not everybody understands how this works, I obviously encourage everybody to read up on security, the attacks are not sophisticated, they are not advanced, they are very easy, the way you protect against them is using frameworks or even using very common mechanisms, if you can write a really simple web application, you can protect against a lot of the common vulnerabilities, it’s really not that advanced. So, I think there is kind of a weird stigma, a weird block that security is very difficult, very hard, and eventually it becomes a lot more difficult, it becomes a full time job but that base line is actually, usually pretty simple to accomplish.

Dio: Having talked about how to prevent security breaches, what is the state of the art in handling security breaches when they happen? Because the thing about security is, you need to be prepared about, things will happen.

Actually you said it, preparedness is the most important piece, I’m not sure it’s state of the art, but a plan is the most important thing you can have. So, when things go wrong you have to have a plan on how to deal with it. That’s something people ignore and seems trivial but that’s absolutely the most important part, that first moment you are notified or that you become aware that something is wrong, you have to react. And that first reaction is visceral reaction, the sinking feeling “oh, no, this is horrible”, you have to power through that. There’s a couple of things that work and that’s having a plan, documentation, a thing that you can fill out, something that guides you through the process so you don’t forget. You’re reacting, you tend to forget things, you tend to maybe not do everything you should do in the process of responding to a breach or potential breach and it’s very important to have a checklist that you can follow, that way it helps keep you sane, it helps check you. Because hopefully you’re not dealing with this a lot and dealing with it a small handful of times means you are not good at the process, you haven’t practiced a lot, which is great if you don’t have to practice a lot, that plan helps replace that lack of practice. If you can practice, that’s even better, but then everybody gets there. So, having a plan is the absolute most important part.

Second is having contacts that can help you, having forensics teams and other third party security teams that you can pick up the phone and call and say “help, I think we’ve been compromised”, they also have plans so they can also help, they can also help you create plans, if you don’t already have them. I think the main point is it’s great to seek advice and seek help, it helps you be stronger, but I can’t say enough about the plan. That’s the absolute most important part. The second part is really grace under pressure, understanding that yes, it’s very bad, but you have a job to do and you need to not let the craziness and emotions get the best of you because it can affect your ability to do the right thing. And also act ethically. When it happens, you usually have an obligation to your users, especially if there’s financial information involved, to let them know it’s happened. And that’s really important to act ethically if anything was to happen. Trust is earned, not demanded, and you have to know you can trust the company to protect the data and in the event that they can’t and something happens, which, you know, it does happen, let’s be realistic about this, you need to be able to have trust that they will work with you to fix it if it does happen, I think that is super important as well.


10. [...] Could you give us a rough overview of how the black market is, what dangers people should be worrying about?

Dio's full question: I guess in order to be able to protect yourself from all sorts of security issues you actually need to be very much aware about what security threats are out there, and I guess most people if they are not focused on security, most developers and administrators, might not have a clear view of how the world is shaped with respect to people trying to get credit card numbers. Could you give us a rough overview of how the black market is, what dangers people should be worrying about?

I think it’s very unique to your business, everybody’s business is a little bit different, but it can be categorized into types of commerce or types of business if it’s not commerce driven and you want to be aware what your area of business is valuable for. You’re creating a business, you’re creating an application, the intent is to provide some sort of service or some sort of value, so in that group you end up falling into a fraud or a theft or some other kind of attack. Those are more focused attacks, isolated attacks are coming after you in particular versus the unclassified drive-by attacks, which are scripts run on the internet just seeing if they can break down the front door, and a lot of them work. There’s not intent to attack you, there’s an intent to attack an opportunity and a lot of those they’ll control a server, they’ll make you part of a command and control network. What they really want is access to all the data on your computer, see if it’s valuable and also to use that computing power as extra power to keep attacking, to keep scouting the internet, because IPs are getting shut down every time they’re detected as malicious by ISPs, by people who detect they’ve been compromised and so you have to have that constant battle of acquiring new machines to help propagate attacks. And that’s very important.

Dio: Thank you very much, Aaron.


Dec 13, 2013