Quantitative Look at Penetration Testing
by Nick Baskett - Managing Director of Matta - Wednesday, 1 August 2007.
Another consultant on our test, ran the password cracking tool, John the Ripper, on a system he was required to treat as production. He used 100% of the CPU for 24 hours on our 'production' server trying to crack the password. The sad thing was that the password was blank, and he never cracked it. His report stated that our password policy was very robust.

A further example with passwords was someone who spent hours trying to crack a password on an application, when the objective was privilege escalation, and the username and password were given to him in the briefing document. If only he had read it!

Most consultants of course, actually do read the briefing notes, and follow the instructions as you would expect, but if you're engaging with a new vendor, it certainly pays to make no assumptions.

Third, every vendor has a methodology statement, and clearly some follow it, but actually we find many do not. This is one area, I believe we as an industry can do much better. The old UK government CHECK approach is a good one, and anyone can follow it regardless of whether you have CHECK accreditation or not. I believe that many vendors are not active enough in ensuring their adopted methodology is followed. Typically, some of the issues we have seen include:
  • missing issues, because the consultant has not stepped through it in a logical and progressive manner
  • going in too 'deep' because the consultant gets excited about some vulnerability they've found, but then forgets, or runs out of time to do some of the basics
  • running exploits, changing passwords, and failing to clean up afterwards. In the real world we have been on incident response calls where the 'hacked host' was just the result of a previous security consultant failing to clean up after an assessment.

As I mentioned before, there are companies out there who we admire and respect. We have worked with companies who were pinging our network, waiting for us to open the firewall to them and start the test. They worked round the clock, were courteous, communicated with us when necessary, and didn't stop until we closed the connection at the end of the test. Then there were those that started late, and finished at 5 p.m. on the dot, even though they still had much more to do. There were those that read the briefing notes, and those that didn't. Those which scanned all 65k+ ports, and those which did a quick scan only.

All consultants and vendors are not equal. Some of the less competent vendors are nevertheless good at selling their services to clients who may not be aware how to judge the difference. More often nowadays we see companies choosing their Penetration Testing vendors based on incorrect metrics, such as accreditations of varying value, and of course on price. My hope is that an independent body of technically competent people with experience in Penetration Testing, but who are not vendors, set up a program which works in a way similar to how we have run Sentinel, and to award technical accreditations to individual consultants, not companies, in a range of technical security assessment areas. Until then, as a vendor, we'll continue to be put under pressure to 'buy' every new PCI, CISSP,CREST, CEH, et all accreditation to be competitive in the market, and most companies will continue to operate in the dark without a set of good, industry standard, technical metrics to guide them.


Harnessing artificial intelligence to build an army of virtual analysts

PatternEx, a startup that gathered a team of AI researcher from MIT CSAIL as well as security and distributed systems experts, is poised to shake up things in the user and entity behavior analytics market.

Weekly newsletter

Reading our newsletter every Monday will keep you up-to-date with security news.

Daily digest

Receive a daily digest of the latest security news.

Tue, Feb 9th