Coverage doesn't cover it; Why we need delinquency.

I don’t think “coverage” targets and metrics for update-led systems like AV and patching do the job. It’s just as important to measure the AGE of non-compliance: the Delinquency. I want the reporting packages that come with AV and patch products to offer that number, and I’m vexed - really quite upset just now - that they don’t.

The purpose and justification of compliance targets is to ensure that "enough" machines are current without wasting effort on too many fixes. In theory, “enough” means

• In the machine population as a whole, on average an installed malware infects (much) less than one other machine, so outbreaks are stifled ("herd immunity")  and
• The probability of any individual machine being compromised is acceptably small considering its sensitivity.

These factors could be measured and calculated, but in practice they aren’t. There are too many unknowns. In fact we adopt more or less arbitrary targets.

Almost uniformly, targets for anti-malware signature systems and patching are routinely measured and set in terms of coverage. What percentage of the population is “out of date” or not installed at all?

My view is that that suits the manufacturer’s model where installers and updating systems work reliably and easily, where the bulk of the effort lies in the initial setup and maintenance is mostly a matter of ensuring the new-builds all carry the agent. When that’s true, why not aspire to 100% and set a target of 97%? The real world is somewhat different.

Not so long ago, I faced a situation where a major vendor AV product was struggling to attain 80% coverage. The hardware was regular, the OS was XP, the builds were coming off a restricted set of images. We got plenty of high-quality support, but it was all about rectifying individual machines – try this, if that doesn’t work, try this. In that situation, the coverage metric was very unhelpful because it just doesn’t say what to do next, how to prioritise limited effort. Coverage can even be a negative guide if the support teams learn to focus on the easy wins, as the troublesome builds which may never have had a current scan will never be fixed. If you have a coverage target, it’s natural to fix the easy problems first – would you fix the AV on a dozen standard-build workstations, or that flaky build that runs that special system that nobody really understands? Are you going to reach out to the laptop users who never update? Or if you’re patching every server you have, except the domain controllers, the coverage looks fine, but the situation is dire…

The measure we want – and none of the reporting tools support it – is delinquency. Delinquency measures how long devices – servers, workstations – have been out of compliance. Admins faced with a delinquency target will be more motivated to fix the hard cases, or escalate them out of the system.

Delinquency is the percentage of machines which are out of compliance now and have not been in compliance before some cut-off time. If you scan compliance on Monday, then the machines that were first noticed the previous Monday or before are your one-week delinquents. Of those, the ones that first showed up the Monday before that are your (yes) two-week delinquents.

The timescales you use depends on how tightly you intend to ride rectification for that particular population. For example, for workstations, I would say that a target might be 10% of one-day delinquents, and zero% of one-weekers. I’m saying that we can accept quite a high percentage of non-compliant hosts, provided that we have confidence that all of them are getting fixed within the week – rebuilt or updated by hand if necessary.

Servers, naturally, are different.  For servers, the route to live is six weeks long and we get one reboot window per month. Many rectification processes involve a reboot. That’s part of the reason why coverage targets fail harder for servers. But for delinquency, we can say that our target is zero% of six-week delinquents – everything has to be fixed in the first reboot cycle after it goes bad – and all of a sudden we are getting somewhere.

I’m not against coverage reporting. It’s good, and it tells a good story at management level. And coverage targets are necessary to control some very obvious ways to game delinquency! But delinquency allows you to manage:

• It gives you a clear “next action” – pick the oldest – to prioritise your rectification effort, and
• It’s compatible with a zero target – you just have to set the age of non-compliance to match your environment, available effort, and risk appetite

On the downside, auditors tend to panic or look blank when you describe it to them. More seriously, it requires a history, and I guess it’s that dependency which means that it seems to be impossible to get figures out of the reporting packages.

And that’s why I’m ranting! I’ve just had to give up delinquency reporting as the hand-built tool I used became too hard to maintain with a change of platform. I’ve had to move back to checking coverage and keeping private little lists of troublemakers, and it feels like a real step backwards.

The non-Build Build

From time to time we issue non-build laptops to people who want to use the SSL VPN but don't have a suitable personal machine . It's not a practice that gives me much pleasure, as the temptation will always be to assume that it's OK to put firms data on one of these. And it's not.

So I've been developing a little list: what we should do to a standard manufacturer's XP install so that it can be placed in the permanent, unmanaged care of a regular user. Here's what I have:

• Truecrypt set up for the system partition. (If there's and I386 on d: or e:, leave that in clear.) Why encryption? 1) because I don't trust them not to put firms data on it, and 2) it's an immediate downer for a thief.
• The MVPS hosts file. It doesn't auto update, but it's a good start.
• Default Browser: Chrome. It's not IE so it's under attacker's radar, but it does auto update even if you never run as admin
• Microsoft AV -- seems to have difficulty with non-admin updates, but better than nothing.
• The default log in takes you to a non-admin account.
• Default settings on the Windows firewall, and Windows update.

It's not much -- in particular you can't make Adobe and Java auto update on a non-admin machine, but it is better than nothing.

In Favour of Delinquency

Anti Virus software doesn't work if it's not installed, running, and updating signatures. What with one thing and another, it's hard to keep AV installed and running on every machine, and so we need a metric to manage by.

It's conventional to measure coverage: "90% of our machines have updated their signature file within the last week". The number and the age are arbitrary -- it could be 80% or 99% or whatever within a day or a month. (But it certainly seems hard to stay above 90% with McAfee....)

But I think coverage is an inadequate target, especially for servers. You have to watch it, certainly, but it's not enough. The problem is that a coverage report says nothing about how long machines are out of compliance -- you risk being satisfied that some machines never, ever, have current AV scanners. Imagine a network with a thousand machines -- if everything is up to date except for two file servers and and the DCs, then your coverage is over 99%, but your overall situation is not at all pretty.

Worse, coverage isn't a good guide to the best next action. Are you going to fix the agent on that critical server with its rare maintenance window? or patch up a couple of workstations? If you just want to get the coverage up you're going to choose the workstations, and you'll be wrong to do so.

Delinquency is a different metric. It measures the proportion going unfixed. It's the percentage of the non-compliant machines in the latest snapshot that were also unfixed at an earlier one, and haven't been fixed in between. The lower the delinquency the better -- a high delinquency means that AV installs are breaking and not getting fixed, a low one means that you are keeping up with the workload.

The levels I like are these:

• For servers, I think the delinquency should be zero, but the lookback period should allow for the time taken to get a maintenance slot on a server. For us, that's seven weeks. It's simply a claim that everything should be fixed in one maintenance cycle, so you can't leave those DCs without current AV.
• For workstations, some delinquents are acceptable. So we say 10%, with a lookback of one week.

It's not ideal. It's harder to compute as you need historical data. But it does tell you what to do first.

And coverage? Well, if you're fixing the breaks, it hardly matters. Like all metrics, delinquency can be gamed if it's your only target, so the best plan is to set something easy like 90% and leave it at that.

Password-Stealing Spam

Big current spam trick: The stolen webmail account.

Hotmail etc. make it hard to register accounts for spamming, so a lot of mail out of their relays isn't spam. And that means that spam detectors mod up mail coming through those gateways -- if it's truly from Hotmail, it's much less likely to be spam. So we're seeing a resurgence -- it feels like 1998 -- of spam from public webmail services. Examined, it turns out:

• To be from a real MSN/Hotmail/Yahoo account (they're not just spoofing addresses -- that wouldn't work)
• To be pushing Chinese electrical goods (if it was stiffy lollies, the language would push the spam balance back to "block")
• It's all sent from Chinese IP addresses. Whether it's .fr, .co.uk, or whatever, it's all pirated from China.

I wrote about this, from the other side, last year. But this is more sophisticated, going to big lists, not just address books.

Just another penalty of being spywared.

I Got Spywared

I ought to go into detail about this, but it's late so I think I'll go straight to the takeaways:

• Don't browse as an admin. Resolving this has taken about fifteen hours over three days. I would rather have spent that time asleep. You can resolve a lot of LUA issues in fifteen hours. The problem here is that Firefox needs to be used as an admin to update, and I wanted 3.06 ....
• It can happen to you. I was using Firefox, I didn't click on anything I was aware of, and the MS Antispyware 2009 installer ran. Arguably it's time to get into Noscript -- I've always put that off because I can't face setting up the exclusions.
• It took me a long time to figure out what was going on. I was able to dump the overt spyware without too much difficulty, but the blocking of anti-malware domain names and the re-writing of Google search results in Firefox and IE to go via windows click dot com had me puzzled. It wasn't the hosts file: they've moved on -- it's device drivers now. I needed to get clear understanding becuase I couldn't get any tools to run -- of course.
• I needed help to figure out what device drivers were the problem. I found it at www.myantispyware.com which appears to be a guy called Patrik publishing instructions. God bless him. His advice didn't quite fit the condition of my machine -- no surprise after all the work I'd done -- but it gave me the names of the files to remove, and that did the job.
• Everyone needs a boot disk. I could have used my Backtrack key, or anything else that could mount NTFS to write, but I had a copy of the Ultimate Boot CD for Windows so I tried that. It was slow to boot, but easy to use. If I wasn't really comfortable in Linux, UBCD would be my first choice. Without it, I would have had to follow Patrik's laborious instructions , and I might have chosen to re-install instead.
• Everyone needs a fabulous hosts file. I got the Winhelp2002 version -- it seems pretty comprehensive.
• Wow! A lot of competent sounding people discuss malware in terms of removal, detection utilities etc. This seems insane to me -- it's really a question of not being admin. This is my first in years, and I don't have any of those tools.

Chinese Hackers are Real, I Tell You...

... And they're planning to flood the world with cheap telephones.

Al sits close to the Head of IT -- a position that reflects his operational centrality, and the affection in which he is held. But he came to me with a puzzle about his Hotmail. It seemed that he'd managed to send himself, and all his contacts, an email advertising http://www.feixiangyu.com -- an electrical distributor.

Well, we looked at things like his spam folder, and whether it was just in fact a particularly artful non-delivery notice. But soon he had replies from his contacts congratulating him on his new business venture.....

Now the beauty of Hotmail is that it's easy to attribute. The X-Originating-IP header gives just that -- the IP address of the originating computer, which is the IP that Hotmail saw as the browser that "got" (GETed?) the send links. This one was 123.53.119.162 and Sam Spade plumps that in the middle of the Middle Kingdom. The ISP is Chinanet, and the PoP is Zhengzhou -- capital of Henan, a respectful distance from the Yellow River -- seven million people in a few square miles, and at least one dodgy marketing guy.

On the whole, I'd rather be hacked by Chinese shopkeepers than the Russian Mafia -- you're less likely to have your bank account emptied. I told Al to change his passwords, check his bank statement, and run an online AV check on his home PC. I sure hope that shows something, otherwise I'm going to have to wonder whether it happened on his office machine, and that's something I just don't want....

ProxySG Appliance Event 3E0003

Here are some messages you don't want to see:

• 2008-06-24 10:13:30+01:00BST "Virus warning! The ICAP service 'proxyav' detected 'Mal/Badsrc-C' while accessing http://news.bbc.co.uk/sport1/hi/golf/default.stm.

• 2008-05-27 14:47:23+01:00BST "Virus warning! The ICAP service 'proxyav' detected 'Mal/Badsrc-A' while accessing http://www.citrix.com/English/ps2/segments/v/vertical.asp?contentID=1415."

I don't know what Mal/Badsrc-[AC] are -- Sophos are vague -- but I don't want to see them on Citrix.com and the BBC. If this is a sign that the malware distributors are moving up from the loweapline.com and the nla.co.uk, we may possibly all be in big trouble.

It's not easy running a website

I think I have this right: There are currently two large-scale ways in which you can have lost control of your website.

• If your site admin was compromised, there's an Apache "rootkit" (not actually rooting the host, but invisible to the admin) that serves a stealthy javascript downloader to all your visitors. As the site admin, you don't see this in the site content.
• If you deliver content from an ad network, you may end up serving malware.

Either way, you won't know about it until the customers are complaining.

Pollen Allergy (The Attack of the Online Florists)

I was talking to the helpdesk team meeting about safe browsing yesterday. I went round the table asking for guesses about the site category that caused the most virus blocks this week. All the usual categories came up: social networking, webmail, blogs and one wag offered the BBC. All good tries except the last, and all wrong.

The real answer was online florists.

Well, that was my route into saying that no site is really safe, (in fact it's a really good security story) and that's why I was going to have another review of their privilege, but I didn't really give it the thought it deserved.

Happily, Mary Landesman has. But I wish she had been able to figure out what was going on.

UPDATED 16/1/2008

It's being reported that all these sites were on Fasthosts when they had that mass site admin password reset in October (and then waited till December to enforce it). Looks as if the malware dropped at that time was left quiescent until last week which makes this a really good security story: Hackers are willing to wait, and there really is no logical end to the consequences of a root compromise.

Paid-for Malware

I sometimes get asked what anti-virus software I recommend for use on the home PC. I've tried a number of possible answers but my heart isn't in any of them: I know McAfee is a pain; bouquets for Norton outweigh the complaints, but not by much, so I've been recommending Kapersky -- I know it works and and the price is closer to reasonable. So a story like this one is a bit disconcerting. What are the lessons?

1. Don't trust software more than you need to. We had all the warning we needed when McAfee pulled this same stunt on a bunch of system files a few years ago. Don't delete: Quarantine.
2. It's time to start getting more assertive about my true answer....

Which is this: I don't run AV software at home. I never have. I don't do stupid things, mostly, and I don't let the children or Mrs U have administrator accounts. I know how to use autoruns (though I've never needed it) and there are the web scanners. I've never had any trouble, even on Windows, and my truly personal computer runs Linux.

Even just writing that, I can see how eccentric and impossible it seems.... really I should just say that I've no useful advice to give.

Quickest Compromise

Browsing round Ikea today I saw sales workstations left logged on to a Windows console, and that set me thinking. Our AUP requires users to lock their workstations on leaving them because the default screensaver lock of fifteen minutes is easily long enough for a malicious passer by to compromise the whole network, and I think that's fair enough. But I wouldn't have fancied standing in front of one of those screens trying to hack Ikea for more than about ten seconds. "Hey you..." So what's the quickest possible way to carry out an opportunistic compromise?

1. It's a real console -- a PC screen keyboard and mouse.
2. The logged on user is not an admin or a power user.
3. You can reboot (but not change a password), but the only boot device is the HD. USB, floppy etc. are all closed.
4. Internet access is through a proxy server running a business-access-focussed site category policy

Extra credit for universal applicability, and evading basic security precautions:

• ICAP server running signature checks on downloads
• No access to root of C:\ or anything other than the local profile
• Mo command line, regedit, ....
• Minimal profile in the event and proxy logs
• Hacked user can return to the console and notice nothing

I suppose the key points here are the exploit itself and the phone-home to control it. My mind is running to a binary exploit file, customised enough to pass signature checks, uploaded somewhere innocuous, and renamed after download to the desktop. The phone home is tougher.

The Man in the Middle for All Purposes

I love simple ingenuity, and FormSpy is ingenious. From the McAfee writeup:

... a malware that is installed as a Mozilla/Firefox component extension.

Upon execution, it registers Mozilla event listeners to the malware and sends information submitted by the victim in the web browser to a malicious website. These information can include, but is not limited to, credit numbers, passwords, e-banking pin numbers etc. The main executable is also capable of sniffing passwords from ICQ, FTP, IMAP and POP3 traffic.

This malware was modified from the "NumberedLinks 0.9" which is an open source Mozilla component available off the Internet. To the victim, he or she would only notice the "NumberedLinks 0.9" extension being installed

Minimum effort, maximum effect. Nice.

The Scent of 1995

This summary is not available. Please click here to view the post.