Security Assessment Techniques: Code Review v Pen Testing

Almost all the security vulnerabilities such as sql injection, cross site scripting, CSRF etc can be uncovered by web app/web service scanning tools and should comprise the majority of the code review and penetration testing coverage. There also exist static code analysis tools which reduces the cost of fixing code later in the software lifecycle. These tools spit out metrics based OWASP top 10 reports by categorizing the vulnerabilities in the appropriate buckets. The challenge is more intelligence in static code analysis tools to discover more vulnerabilities upfront.

Yes, every HP product should comprise the majority of the code review and penetration testing coverage! WebInspect is flawless in its false positive reporting. Fortify is a point-and-click static code analyzer that have 100% coverage in all scenarios! And, stop using well-trained analysts to assist in the review. Just use HP professional services to run the tools. Your vulnerability count will go to 0 in no time at all. Please contact an HP rep today for all your lulz.

Thanks Srini for the nice post about my talk at the OWASP conference.

I believe that Jeevak is from HP and so he might be a little biased on the capabilities of automated analysis tools. Such tools have their value, but they also have serious limitations as well, and require much more expertise to use effectively than most people understand.

The sarcastic reply from 'HP Rules', whoever that is, is just pointing out that the previous poster is most likely from a product vendor. I'm sure HP isn't promising anything as wildly fantastic as this sarcastic reply, but customers of such tools do need to clearly understand the strengths and weaknesses of such tools and the level of expertise, integration, and customization required to get good value from any investment in automated analysis tools.

Aspect Security has worked with numerous customers who have first invested large amounts of money buying tools, but didn't invest much in training, integration, customization, as well as making sure they had reasonably skilled staff using these tools. It's certainly appealing to imagine tools can handle the bulk of the effort required to discover vulnerabilities and determine the severity of the risks they represent but the amount of false positives and negatives is still so high that a significant amount of expert user involvement is still required to triage what was found, find the flaws the tools can't find, and then triage all this stuff so that the most important risks rise to the top, rather than simply the risks that are easiest to identify with automation. After we worked with these customers to integrate and customize these tools for their environment, they started to receive much more value from their investment in automation than they were getting before.

One other thing I want to point out is that finding flaws, which was the focus of my talk, is still the least expensive part of the application security problem, and yet it is getting the most focus today. The expensive part is building secure applications in the first place, or remediating issues that are found through application security assessments. Training developers, arming them with reusable security controls, introducing security architecture reviews and threat modeling, and secure coding guidelines is where you will get the most bang for your buck, and yet automated analysis is getting the lion's share of many naive organizations attention because it's so appealing to imagine that this will solve their problem, or at least be the best first place to invest.

In summary, I think automation clearly has a useful role in both code analysis and application penetration testing, but it requires far more expert human investment to get significant value out of them than most people realize, and these tools are simply unable to discover some of the highest risk issues like authentication, access control, and context aware input validation and output encoding.