Report reveals overconfidence in application penetration testing

Fortify Software has released a report, entitled “Misplaced Confidence in Application Penetration Testing,” that details overconfidence in application penetration testing.

The report highlights that users of application penetration testing poorly understand how to gauge the effectiveness of their penetration tests and is comprised of two parts: a survey of security testers and an in-depth experiment to validate survey results. While the survey revealed high expectations of application penetration tests, the experiment showed that automated and manual tests often reached only 25 percent of an application’s security critical APIs, leaving large portions of the code untested. In addition, the tests failed to identify critical vulnerabilities within the parts of the application they did cover.

Application black box testing uses various inputs to probe an application while it is running in order to simulate attacks and identify potential vulnerabilities. For this report, a member of Fortify’s Security Research Group conducted automated and manual application penetration tests on five common test applications. The researcher used two of the top three market-leading application penetration testing tools, as well as manual efforts. Fortify then used its own product, Fortify Tracer, and deployed it inside the test applications in order to generate detailed data on the effectiveness of these application penetration testing tools.

This study exposed a significant gap between the expectations of consumers of application penetration testing and the reality of the results when measured in a systematic and objective manner. The results showed that at best, one of the tools achieved 29 percent coverage averaged across five applications. Knowing that most companies augment automated testing procedures with manual testing, the tester attempted to increase the coverage percentages by adding manual efforts. Although these results showed an average increase in coverage of 19 percent, they still missed more than half of the vulnerable APIs in the applications.

Furthermore, of the APIs that were examined, Fortify Tracer showed several instances where automated and manual efforts failed to uncover key vulnerabilities; particularly SQL injection and cross-site scripting related vulnerabilities. In addition, Fortify Tracer discovered types of vulnerabilities that the black box testing approach is not set up to detect, such as a privacy violation, which indicates the application has written sensitive data to a log file.

At the onset of the report, Fortify surveyed dozens of penetration testers at various organisations to gauge their confidence in and expectations of the effectiveness of application penetration testing. More than 58 percent of the respondents said the security tests they run are adequately testing their applications for vulnerabilities, (defined as reaching at least 61 percent of the security related APIs in any given application). In fact, 46 percent of the respondents estimated that their application penetration tests were able to reach at least 81 percent of their application’s security related APIs.

The full report is available online (registration required) and is an update from “Taking the Blinders Off Black Box Security Testing,” released in October 2006.