Audits fall with autumn leaves

We've just been visited by one of the many audits to which a regulated firm is subject. We didn't come out as well as I would have hoped but the point for me was different and more worrying.

These were competent people. They were clear about their wants: Evidence that the controls we publish and claim to adhere to are actually working. And they knew what "working" meant -- that the circle is closed with human escalations and choices on exceptions. So that was good (and a lot of work for us) except for one teeny issue.

"Working" also means that the control environment will actually stop trouble. And these guys had essentially no interest in the technical effect of the controls. If I said "this is a report that shows yesterday's changes to all application admin groups", that was the truth. No test that we have the same reporting on all production DCs. No enquiry about alternative ways to get the privilege. No test that our installations actually adhere to the admin group conventions. If I listed a firewall policy, or handed over the perimeter network diagram, that was it. No enquiry about how often I checked the cable patching....

Now I know that they can't check everything. And I wouldn't want them to.... I know that they're at the wrong end of a crushing knowledge asymmetry. But all the same, it reminds me of the drunk searching for his keys under the lamp post: not because he lost them there, but because the light is so much better.

In the mean time, remember:

• A big four signature on a statement of controls -- SAS70 or whatever -- means less than you think.
• Somewhere in the big city, a security guy is neglecting controls that expose trouble in favour of those that'll audit well.