Maxims

These (http://www.ne.anl.gov/capabilities/vat/seals/maxims.html) from the security team at the US Argonne National Laboratory are worth a three-day seminar...

If You've Done Nothing Wrong, You've Got Nothing to Fear

So we're told. But it didn't work for Jacqui Smith.

Classic Fail

Went to pick up my printout from the printer and there it was: The biggest secret in the firm, and one from which I am firmly excluded. Ninety colour pages which someone had collected, collated and left prominently displayed to be picked up.

A few minutes in the event log of the print server gave the answer -- the same document printed twice in succession: the signature of a user losing track of what they've done. He's back on track now.

The report had been out on display for thirty minutes when I found it. I imagine the person who tidied it up will be one of the three people who used that printer between me and the inadvertent leak. But who else saw it is much harder to tell.

Naming More Risks

On the theory that risks need names, here's a couple more from the recent Sumitomo bank job.

• O'Donoghue (Kevin) risk: Bent security guards.
• Rodley ("Lord" Hugh) risk: Dealing with stereotypical peers who aren't in Debrett's. Check the photo in the BBC report....

There are some lessons there as well.

• First reports are generally wrong. On the morning the arrests were made, I was told to drop everything and check out all machines with access to SWIFT for keyboard loggers. Which would have made sense -- probably does always make sense -- but wasn't relevant to the facts of this attack, which was based on software loggers.
• Access control around documentation is not security by obscurity. Or if it is, then SbO works. Because what allowed Sumitomo to keep its funds was the mild complication of the fund transfer setup.
• Business-hours limitations would have made sense, too.

I Got Spywared

I ought to go into detail about this, but it's late so I think I'll go straight to the takeaways:

• Don't browse as an admin. Resolving this has taken about fifteen hours over three days. I would rather have spent that time asleep. You can resolve a lot of LUA issues in fifteen hours. The problem here is that Firefox needs to be used as an admin to update, and I wanted 3.06 ....
• It can happen to you. I was using Firefox, I didn't click on anything I was aware of, and the MS Antispyware 2009 installer ran. Arguably it's time to get into Noscript -- I've always put that off because I can't face setting up the exclusions.
• It took me a long time to figure out what was going on. I was able to dump the overt spyware without too much difficulty, but the blocking of anti-malware domain names and the re-writing of Google search results in Firefox and IE to go via windows click dot com had me puzzled. It wasn't the hosts file: they've moved on -- it's device drivers now. I needed to get clear understanding becuase I couldn't get any tools to run -- of course.
• I needed help to figure out what device drivers were the problem. I found it at www.myantispyware.com which appears to be a guy called Patrik publishing instructions. God bless him. His advice didn't quite fit the condition of my machine -- no surprise after all the work I'd done -- but it gave me the names of the files to remove, and that did the job.
• Everyone needs a boot disk. I could have used my Backtrack key, or anything else that could mount NTFS to write, but I had a copy of the Ultimate Boot CD for Windows so I tried that. It was slow to boot, but easy to use. If I wasn't really comfortable in Linux, UBCD would be my first choice. Without it, I would have had to follow Patrik's laborious instructions , and I might have chosen to re-install instead.
• Everyone needs a fabulous hosts file. I got the Winhelp2002 version -- it seems pretty comprehensive.
• Wow! A lot of competent sounding people discuss malware in terms of removal, detection utilities etc. This seems insane to me -- it's really a question of not being admin. This is my first in years, and I don't have any of those tools.

Spam Counter - 2008 November: 852

This month's drop is the the famous McColo effect. It'll be interesting to see how a whole month without McColo looks on 12/12.
The content seems the same as ever.
Two interesting papers about email-delivered nuisances: spam and phishing. Each offers methodologies which finally give realistic estimates for the return from penis spam and phishing. Both agree there's very little profit in it -- mugs and losers are, after all, a limited resource. Which is nice.

Chinese Hackers are Real, I Tell You...

... And they're planning to flood the world with cheap telephones.

Al sits close to the Head of IT -- a position that reflects his operational centrality, and the affection in which he is held. But he came to me with a puzzle about his Hotmail. It seemed that he'd managed to send himself, and all his contacts, an email advertising http://www.feixiangyu.com -- an electrical distributor.

Well, we looked at things like his spam folder, and whether it was just in fact a particularly artful non-delivery notice. But soon he had replies from his contacts congratulating him on his new business venture.....

Now the beauty of Hotmail is that it's easy to attribute. The X-Originating-IP header gives just that -- the IP address of the originating computer, which is the IP that Hotmail saw as the browser that "got" (GETed?) the send links. This one was 123.53.119.162 and Sam Spade plumps that in the middle of the Middle Kingdom. The ISP is Chinanet, and the PoP is Zhengzhou -- capital of Henan, a respectful distance from the Yellow River -- seven million people in a few square miles, and at least one dodgy marketing guy.

On the whole, I'd rather be hacked by Chinese shopkeepers than the Russian Mafia -- you're less likely to have your bank account emptied. I told Al to change his passwords, check his bank statement, and run an online AV check on his home PC. I sure hope that shows something, otherwise I'm going to have to wonder whether it happened on his office machine, and that's something I just don't want....

Real Financial Insecurity

Prostitute's postcard seen today in a phone box in King William Street:

• A very conventional picture of a youngish woman in partial undress, and
  
• A site: "London Bridge" -- in reality that would be far out in the Borough, but never mind...

All ordinary. But what struck me was the caption. It wasn't "Maid for Pleasure." It wasn't "New 19 YO Swedish." It wasn't even that perennial City favourite: "Fully Equipped Dungeon."

No. The caption was: "Kisses and Cuddles." If that isn't the clearest sign of financial calamity, I don't know what would be.

God bless her, though. It's wonderful to imagine that there's a living in snogging.

The Current Status of the Pound

I'm writing this on 28/10 but I'm back-posting to the day it happened, right in the middle of the (first?) UK banking turmoil.
I had occasion to use the toilet in the headquarters of a big four bank. As I reached for the paper I noticed a little blemish on the white(ish) sheet. Being unsqeamish about this sort of thing, I gave it a little scratch and a shred of coloured paper came away on my finger nail. I pushed back my specs for a closer look and found a tiny fragment of a £10 Bank of England note -- barely a millimetre across, but the engraving and colour so fine as to be unmistakeable. That bank had been wiping their collective arses on thousands of pounds in fine rag paper -- and they never knew.
I do wonder whether it's co-incidence, or whether support from the BoE comes with an unpublished obligation to help them get rid of their pulp....

Two Shiny Stories

So now we have the Google browser with a name that proves that the Septics do get irony. It's not chromy at all, but it does have two interesting stories:

• It is possible to keep big secrets for a long time. I don't know how long it took to go from concept to (beta -- surprise) release, but it can't have been less than eighteen months and even though a Google browser is a juicy story, all the news services all seem to have been taken by surprise. That's impressive. I'd like to know how they did it, and I wonder what other secrets they are keeping.
  
• Now that Google is just as wicked as Microsoft there's lost of fuss about the browser's licence and potential to phone information to Google (i.e. under the pretence of checking if the site you are visiting is phishing. But the source is under BSD so it ought to be possible for a forked "clean" version to appear on sourceforge any time. We'll see whether open source can still function without corporate support....

Naming Risks

Jerome Kerviel seems to be on the edge of getting a risk named after him. This is not the sort of distinction that will make his mummy proud, but it is a distinction nonetheless. About the only other named risk I can think of immortalises the otherwise obscure Herstatt Bank closed by regulators in 1974 before it had paid out on its forwards settling that day.
Kerviel's activities are set out in the Mission Green report, and if you were following the story at the time, it's interesting to see how wrong the initial spin was: He wasn't stealing passwords, he wasn't modifying control spreadsheets. He was exploiting his back office knowledge, but at a higher level: he knew how to use cancellations and corrections -- all the points where control can't be watertight because trading isn't -- to get his positions off the records, and he'd been doing it for some time. (It was only right at the end that he started to fake forwarded email -- nothing complicated, just editing a real forwarded email.) So this gives us a useful term: Kerviel risk is exploitable vulnerabilities -- uncompleted cycles of review and follow-up -- in a control system. A short name for a rather complicated concept, so maybe it'll stick.
No this definition means that Kerviel's name is not correct for authentication-abused-to-approve-fraudulent-actions risk. But Jagmeet Channa has come along just in time to help us out. He stole a couple of passwords to approve his multi-million pound transfers to his accomplices in N. Africa and Manchester.
The problem is figuring out what risk we're naming here. Channa's not talking so we can't tell if it's:

• Password stealing? -- he certainly did, but maybe that's not the point
• Inserted Insider?
• Coerced Insider?
• Criminal Mastermind who recruited outside help?

I'm going with the authentication, for the present. Channa Role Risk.....
And what makes this a security story? Well, the investigation started by interviewing the colleagues whose passwords Channa used. Don't fancy being in an interview like that? Then guard your password.

ProxySG Appliance Event 3E0003

Here are some messages you don't want to see:

• 2008-06-24 10:13:30+01:00BST "Virus warning! The ICAP service 'proxyav' detected 'Mal/Badsrc-C' while accessing http://news.bbc.co.uk/sport1/hi/golf/default.stm.

• 2008-05-27 14:47:23+01:00BST "Virus warning! The ICAP service 'proxyav' detected 'Mal/Badsrc-A' while accessing http://www.citrix.com/English/ps2/segments/v/vertical.asp?contentID=1415."

I don't know what Mal/Badsrc-[AC] are -- Sophos are vague -- but I don't want to see them on Citrix.com and the BBC. If this is a sign that the malware distributors are moving up from the loweapline.com and the nla.co.uk, we may possibly all be in big trouble.

Grepping the IE cache

I had to do an investigation the other week. I'm not an investigator and so naturally I screwed up. Here's what I learned.

Complaint was that some abusive hotmail-sent mail had arrived quoting the outside address of our firewall. After a bit of to-ing and fro-ing, I was allowed to see the headers, and that told me a good deal:

• Hotmail does indeed quote an originating IP in the header. Who knew?

• The earliest relay in a hotmail relay list is a name like bay99fd.bay99.hotmail.msn.com. Any hotmail user knows that the bay appears in the URL on the hotmail home page and throughout the user interface. And for any particular account, that bay number is fixed.

• Timezones were going to be a problem. We were in local DST, the victim's mail infrastructure was in their DST and four hours behind, his MUA was working in another zone still and a lot of the Hotmail infrastructure is on Pacific time. Still, given headers, I could convert everything to UTC easily enough.

OK. Time to see if we can knock this out in a single step and get back to proper work. The Log appliance appliance has been gathering proxy logs all year. We're a pretty relaxed site and I've not been asked to report on usage of a named site before, so I have to code up a report with wildcards for client IP, domain-name and the page name. A bit of experimenting gives me a report of access to that Hotmail bay.

Now this is the first place a real investigator would have done it differently: First step should have been a summary report of all the users of the bay over the last three months. That might have been enough to get HR off my back. As it was, I spent a week dipping in and out of the proxy logs data to look at alternatives as the mails emerged from the complaining firm.

That initial set of headers fingered a single user. I could only see two users of the bay, and at the right time only one of them was active. And guess what? Within the two-minute precision of the log upload batch, he used pages on the bay called "compose" and "premail". A bit of experimenting with my own hotmail showed that that is the characteristic signature of sending Hotmail.

This is the second point I did the wrong thing. I've got a budget for investigations and I should have used it. For UKL 1,000 + expenses and VAT, Kroll Ontrack (it used to be Vogon) will send midnight engineer to take a swearable image of a workstation hard disk, leaving you with a handy USB disk copy for your own investigation and the user none the wiser. I was focussed on our local, more rough and ready process, which was a bit too public for HR. It wasn't a total screw-up though. I'd only looked at proxy logs through a read only interface -- I knew enough not to touch the workstation, and so the purity of its evidential status was preserved, even though the Internet cache timeout was ebbing away.

Part of the delay was at the far end. HR can't and won't do anything on a complaint like this without the offending text, and the complainer was a bit coy. HR's reason is good: it might not be offensive in our context. Still, I thought it was a bit silly -- the headers showed that the hotmail address was obviously a real name, and not the name of our user, and he is, or was, a regulated person.

In the middle of that argument, I got a second set of headers for a much more recent mail. Same accounts, same bay, same user matches.

It all went a bit off course at that point. What I got next were not proper headers with that incriminating source IP and lovely times plainly referred to UTC. It was the nesting of headers in the body of a reply/forward dialogue, and the "on" times there are converted into the time of whoever received the mail. By that time, I was so focussed on matching the time to activity on the proxies that I set to work trying to infer the timezone of each recipient and reconstruct the offenders side of the dialogue. A proper investigator would have realised that this exercise was difficult enough to make uncertain results, and insisted on headers or nothing. As it was, I made mistakes and spent a lot of time wondering how the original mail could have been sent when our target definitely was busy and wasn't on Hotmail. I went as far as trying to rope in the other user of the bay as an accomplice -- that didn't work either. Looking at the times again, I can se my mistake: It wasn't five PM, but seven, and the mail was sent from home.

I'm not privy to the discussion that went on in the business. It's called reputational risk and I guess we were asking the board to trade a reputational compromise with a non-customer against possibly losing an expensively-hired fund manager and telling his customers that their money had been in the hands of a stupid person with weak morals. Glad I don't have to make that choice, but they did the right thing and I was told off to get the dirt.

The Kroll visit was simplicity itself, mainly because I didn't have to stay up all night -- the HR guy did that!

Lunchtime next day I got an urgent package with a 40GB USB hard disk which mounted first time on my non-build laptop. That was another mistake -- if I'd used a Linux laptop, or a regedit fix, I could have controlled the mount to be read only. It didn't really matter as the forensic copy is on Kroll's servers -- the supplied disk is just a playpen. The idea is that you hunt around any way you like, but any defence witnesses or advisers can still work with a guaranteed untouched copy.

This is important -- a lesson I learnt long ago. Never give in to the temptation to take a quick look at a workstation via the admin shares or however. Unless you are collecting them automatically, don't even look at the event logs. Right at the beginning of any question, figure out -- ask -- if there's any possibility that anyone will be held to account for what you uncover. Consider whether (for example) you could work from restores, or with a reporting tool. Tell your interlocutor that if it's possibly going to get as far as swearing evidence, you are less likely to be overturned if you work throughout with a trained investigator.

If you really have no choice, make sure you get a crytographically secure hash of each and every file you access. Make it clear in your notes that you obtained the hash before looking at the file. Print the hash out, note the time it was obtained, sign it and date it. Make sure that the file you keep will generate the hash you print. That way you can swear that it was that way when you found it.

However. I had a scratch copy of a workstation disk and I could do what I liked. There are tools for this sort of thing and I ain't got aught of 'em. Not necessary at my level. You can download an excellent Windows grep from the FSF and anything else is overkill. Remember to put the GNUWin32\bin directory in the path.

With the disk mounted, you'll find the IE cache is at (name changed to incriminate the innocent)

\Documents and Settings\umacf24\Local Settings\Temporary Internet Files\Content.IE5

Make your way there in the command line and issue a carefully chosen grep:

grep -irl madeupname@hotmail.com *

will search for the address in all the cached files in that directory. That matters because hotmail puts the logged on account on every page, so you can see right away whether the user has actually been active on that account -- the one thing the proxy logs can't give you. Those options mean: -i case insensitive, -l list the matching files (as the content isn't much use, as text) and -r recurses down the directories.

I was surprised to find that the IE cache went back a lot further than I expected. It looks as the the "retain for n days" setting only takes effect if space is tight -- this man's cache went back months.

Now the beauty of the IE cache compared to Firefox is that there's no complicated database format. The files cached are the files downloaded. Names are modified, and there's a directory structure to avoid having one huge folder, but the pages can be displayed in the browser. I have an account which doesn't have Internet access, so using that account, I just started IE and browsed to the appropriate files. It was one of my happier moments to see a hotmail folder listing -- looking a bit dodgy, admittedly -- listing times and subjects of the complained-of emails. Access to compose pages actually gave me content of mails which the complaining party had been relectant to reveal. I gather that the colour prints of those pages were particularly unsettling when the confrontation occurred.

I can't write a story like this without a few lessons.

• Serious investigation would have been overkill. We didn't need deleted files, we didn't need to to search for concealled media or executable content. It was just those emails

• Think. Of course he was doing it from home.

• Ask for what you need. I needed headers.

• Don't be afraid to search a PC. I've bought an imaging machine so we can do our own. I could have got those unarguable Hotmail reconstructions much earlier and saved a lot of time.

• You want to keep proxy logs for ever. The depth of context is invaluable when you need to do a lot of learning about what your users get up to.

• Remember that users can't protect themselves. Using gmail over SSL would have made this offence effectively uninvestigatable without bugging his PC. But who knows that?

Good luck to you

Himmler Murder Memos in the NRO

This story in today's FT magazine is interesting in its own right, but it makes a good security story as well:

1. Don't despise paper. Everyone quoted in the fake memos is dead, the empire they served is one with Nineveh and Tyre, and the record-keeping system was obviously not designed to detect this fraud, but the fakes can still be totally discredited. There is no shadow of a doubt that those notes are inauthentic, and the rest of the bundles they came from are real. What's the IT angle? Well, consider what you can prove with a signed page of printed hashes....
2. The sideband rules. Laser printing on a document that purports to be twenty years older than xerography. Every suspect document having the file hole torn. These circumstances talk directly to the investigator.
3. Listen to the language. The public school types who ran the war didn't talk like that and they certainly didn't write like that.
4. Keep access records. One man only, ever, was recorded as accessing all those bundles....
5. And of course, follow the motive. He wrote a sensational book...

Wasted Time

I spent some time going through the security morning checks with Internal Audit.

Report on event logs every morning, examined every morning, security incidents found in three years: none. Firewall traffic logs, examined ad-hoc over four elapsed years, security incidents found: one - an agobot infection on a bad build.

Hours wasted -- hundreds.

We're doing the wrong thing.

What's the right thing? There's too much novelty and too few admins in our network for IDS to be worthwhile. Just retain the logs but stop looking for trouble? The trick will be to do that, but keep looking responsible.

It's not easy running a website

I think I have this right: There are currently two large-scale ways in which you can have lost control of your website.

• If your site admin was compromised, there's an Apache "rootkit" (not actually rooting the host, but invisible to the admin) that serves a stealthy javascript downloader to all your visitors. As the site admin, you don't see this in the site content.
• If you deliver content from an ad network, you may end up serving malware.

Either way, you won't know about it until the customers are complaining.

Pollen Allergy (The Attack of the Online Florists)

I was talking to the helpdesk team meeting about safe browsing yesterday. I went round the table asking for guesses about the site category that caused the most virus blocks this week. All the usual categories came up: social networking, webmail, blogs and one wag offered the BBC. All good tries except the last, and all wrong.

The real answer was online florists.

Well, that was my route into saying that no site is really safe, (in fact it's a really good security story) and that's why I was going to have another review of their privilege, but I didn't really give it the thought it deserved.

Happily, Mary Landesman has. But I wish she had been able to figure out what was going on.

UPDATED 16/1/2008

It's being reported that all these sites were on Fasthosts when they had that mass site admin password reset in October (and then waited till December to enforce it). Looks as if the malware dropped at that time was left quiescent until last week which makes this a really good security story: Hackers are willing to wait, and there really is no logical end to the consequences of a root compromise.

Unnatural Selection

I've just started on my Christmas present, Kolmya Tales, by Varlam Shalamov. The introduction describes Shalamov's time in the mid-century Soviet labour camps and mentions the strokes of luck which allowed him to survive. I've read a little of the popular history and translated literature of the camps and the system that required them, and it struck me that all political survivor's stories have one feature in common: the amazing luck -- a necessary skill leading to a warm job, awakened sympathy in a guard or criminal, wasted food found or any of a host of other things -- that saved the witness' life. You could end up thinking that the penal labour was a happy-go-lucky setting where something would always turn up in time....

Of course that's wrong. There are no first person stories that don't have that lucky break, because all the potential authors -- without the lucky breaks -- died. By focussing on eyewitness accounts -- the best possible sources -- we've gone wrong. The camps were not about misery overcome by good fortune; they were about misery closed with death. Everyone is telling their truth, but the sample is bad, and so the picture is false.