Since 2004 Matta has been running a project to test the technical competence of security consultants. With probably the largest collection of data on the methodology's and technical approach taken by some of the worlds best known security companies, some interesting conclusions can be drawn. The tests started as a result of a client asking for our assistance in running an RFP. The program grew from there as other companies became interested in doing the same thing. In 2006, Matta provided Royal Holloway University with a version of Sentinel to make the country's first Penetration Testing vulnerable network for its students.

As there is a lot of data, the conclusions you reach will really depend on what you are looking for in the first place. I've documented in this article some of the things which I feel may be of general interest.

I would like to preface this article by saying that it is human nature to find the bad news more interesting than the good news. In our tests, we saw many impressive consultants. We tested companies which acted professionally and competently throughout, and there are consultancies who we admire and respect as a result of working either directly or indirectly with them. Many other companies could have set up the Sentinel program, and we don't place ourselves higher than our peers. It just so happened that it was Matta that was asked to do it. Typically, the clients who have run Sentinel programs, are either looking for a global Penetration Testing supplier - which Matta is not - or they are running internal accreditation schemes. Our reports have always been considered objective, and if we have something subjective to say, it goes on a separate page in the report, which is marked as a subjective observation.


Looking at some findings then, the first, and perhaps the most startling fact of all is that every consultant who has gone through the test has always found vulnerabilities with their tools, which then failed to make it on to their final report.

We sniff and log all the network traffic during the test, and are often required to demonstrate to the vendor that they did indeed find the issue, which was then absent on their report.

Clearly, there is a real problem with time limited tests, and the work required to go through reams of unqualified data to sort out the real issues from the false positives. Things just get missed. Importantly it seems, at least in our tests, something gets left out on every occasion. Our tests are intense, and time limited, so perhaps a fair conclusion to reach is that if the consultant is similarly under pressure, either internally, or from the client, then expect to get incomplete results.