In some cases though, we also feel that the consultants would probably have missed the vulnerabilities regardless of the time limit. We believe that the output from some common tools is not so easy to read for those with less experience. Second, and for me the most baffling observation, is that some consultants - a small minority - but enough to be significant, don't seem to read their briefing notes. This is really concerning. Each test we run has an engagement protocol. The vendor is given a set of briefing notes, with key information, including perhaps some login credentials to a web application, or a request to treat a database exactly as if it were a production system.

So whilst most consultants had no trouble executing the tests with these instructions, one consultant repeatedly crashed the database to get debug information. Not something you would want do on a production database! On a similar note, we did hear a real life story from a client, in which a penetration tester had tried to drop a database to prove he had effected a compromise. Fortunately, due to mitigating factors, he was unable to drop it, but the client was less than happy, and I don't believe they required his services again.

Another consultant on our test, ran the password cracking tool, John the Ripper, on a system he was required to treat as production. He used 100% of the CPU for 24 hours on our 'production' server trying to crack the password. The sad thing was that the password was blank, and he never cracked it. His report stated that our password policy was very robust.


A further example with passwords was someone who spent hours trying to crack a password on an application, when the objective was privilege escalation, and the username and password were given to him in the briefing document. If only he had read it!

Most consultants of course, actually do read the briefing notes, and follow the instructions as you would expect, but if you're engaging with a new vendor, it certainly pays to make no assumptions.

Third, every vendor has a methodology statement, and clearly some follow it, but actually we find many do not. This is one area, I believe we as an industry can do much better. The old UK government CHECK approach is a good one, and anyone can follow it regardless of whether you have CHECK accreditation or not. I believe that many vendors are not active enough in ensuring their adopted methodology is followed. Typically, some of the issues we have seen include:

• missing issues, because the consultant has not stepped through it in a logical and progressive manner
• going in too 'deep' because the consultant gets excited about some vulnerability they've found, but then forgets, or runs out of time to do some of the basics
• running exploits, changing passwords, and failing to clean up afterwards. In the real world we have been on incident response calls where the 'hacked host' was just the result of a previous security consultant failing to clean up after an assessment.