This last week, a thread in Spiceworks discussed a rogue sys admin that had come back to haunt this company. I read through the different threads and one in particular jumped out at me: “You people sound like a bunch of cops. Don’t you trust anyone? In order to trust someone, you must be trustworthy yourself.”
I almost posted back that I wished I could be that naïve. I wish I could live in that world, not that I’m a pessimist. I’m an optimist with experience. And my experience tells me that the only possible answer to that is “I trust everyone . . . But only to a point.”
So let’s talk about how trust is a big problem in the IT security game. First, we hear so much how some company has been penetrated by hackers, and information stolen such as credit card numbers and so on. Let’s be honest about it, this is serious stuff, and it makes people afraid. Fear sells newspapers! What we don’t hear about is the thousands of little hacks that happen because they didn’t penetrate any site. They didn’t have to. They were already inside.
Once upon a time I worked for a company. They made software, and they were very good at it. I was brought aboard mostly to get their VMware environment off the ground and running. They were taking their product and tweaking it for all kinds of different electrical utility companies. What they’d done in the past was when a developer would go to that utility to help tweak the software and so on, they’d take a laptop, and in some cases, up to dozen different hard drives. The hard drives had an OS, all their tools, and the software for that environment. When they got to that utility, they’d take their hard drive out of their laptop, pop in the one labeled for that company, and they’d just keep on working. Great idea in theory, terrible idea in execution. Drives were getting lost, broken, and in some cases stolen. Indeed, when I got all the drives back, some had nothing to do with the software, but instead were full of music, movies, and so forth.
So, in comes VMware. Now the idea is we build their dev environment, we send them to a utility, and then all they have to do access their work in order to make tweaks and tests via VPN. Also a good idea, but every once in a while, we had to send them off with a hard drive anyway. In this case, we simply tossed VMplayer on their laptop, and I used VM Convertor to copy the VM over. This way they could do their work and not have to worry about a VPN connection. I controlled the hard drives, they became a receipt item, and when the developer left, I issued it to them along with the VM that they needed. When they got back, I was sitting there waiting for them.
Enter Donnie (which was not his real name)! Now he was a brilliant developer, a very charismatic individual, and one of those guys that should have been born with a warning label. His biggest gift was to be a nice guy, and he could turn the charm on full blast. He could take the most outrageous, preposterous idea and make it sound reasonable.
So he saw that I handed and controlled the VMs, and he approached me with an idea. “Hey, Rich. I know you’re an incredibly busy person,” he said and after praising me for a good five minutes continued: “Why don’t you give me the permission to hand out the drives and clone the VMs and so forth . . .”
Did I happen to mention that I was a police officer when he was still in diapers, held political office, and I’d developed a very finely tuned Nonsense Meter! It was pegging out and a little voice was warning me not to trust him as far as I could throw him.
“Thank you, but no,” I said.
“But . . .”
“Which part of NO didn’t you understand,” I asked.
He went away in a bit of a huff.
He didn’t come back and I thought he’d gotten the point.
If you run a virtualized environment, one thing you should be doing is watching over it like a hawk. You need to know how it preforms, and you always need to know what happens in it, how it preforms. So, I always take a virtual walk through my environment and check things out. And one day, I’m going through and I noticed a couple of machines that had snapshots. Not a big deal you say? Well, snapshots are a bit of an issue to leave lying about, and in some cases they’re expected.
Here’s how that works. In the world of VMware, the so called hard disk for a VM is really nothing more than a file called a VMDK file (and yes, you can have a whole bunch of VMDK files). A lot of backup software works by taking a snapshot. This puts the VMDK into a state we called quiescence. That means it’s gone quiet and any changes made (like file uploaded) don’t happen to the VMDK file. At the end of the backup, the software merges the snapshot into the VMDK file, and the snapshot no longer exists.
But here I was, looking at a snapshot where one shouldn’t be.
OK, I thought. Couple of possibilities. One, the backup software screwed up and didn’t clean up after itself. Incidentally, if you get random failures of backup software, you might want to make sure it’s merging the snapshot back. Possibility two. I made a snapshot and forgot all about it. I didn’t remember making a snapshot, but I am over 50 and you know, the mind is the first to go . . .
There was a third possibility, but I’d kept pretty close rein on password etc. Just for kicks, I checked the log for snapshots, found where one had been created and thought, huh, maybe I did do something stupid.
So I cleaned up the snapshot, and went on.
A week later there was another snapshot, and I went what the heck! I dug deeper and this time I realized something unexpected had happened. I went to my boss and asked if he’d done a snapshot. Of course not. Did you give the root password out? Of course not.
I went and changed the password, and considered my next move. Clearly I had a hacker. Somehow, the password had been compromised so this time I created a new password and made it incredibly long and incredibly complex for the root account. But I also knew from my years as a cop that Burglars like to come back and shake doors that have yielded up a haul in the past.
I also needed to track whoever was doing this. So I downloaded and built myself a SNORT box. For those of you, who have never played with SNORT, let me tell you that this is one of the best Intrusion Detection Systems out there. The fact it’s free makes it even better.
Now, I watched and waited. Incidentally, there were several things I could have done to really lock it down, but I wanted to catch the guy. I was a bit suspicious, and what I was thinking was the password had been compromised the old fashioned way. Someone gave it to the hacker. If my guesses were right, they would again.
I week later, I had not one, but two snapshots. And now I went hunting. Snort spat up connection information for that time and I tracked the successful attempts and unsuccessful attempts to one IP address. I ran off the information, found which workstation had leased that IP address and went and paid the operator a visit. Guess what, it was my buddy, Donnie.
“OK,” I said, slipping back into my old cop self. “Why are you doing what you’re doing, Donnie?’
I showed him my evidence and he began talking. “I wanted to do what I told you, take the pressure off you . . .”
“Sure you did, and that’s why I’m still handing out disk drives while you’re not.” I’d already done a remote check of his machine and asked, “Where are the VMs?”
“On an external hard drive,” he answered after a second. “It’s at home.”
“Why home?” I asked.
“I put them on a server,” he said. “And I’m letting my friends play with them!”
I felt the color drain from my face. The software, the data, all of it was proprietary stuff. And now it’s on the net! “Show me!”
He opened Firefox, typed in an address, and the next thing I saw was a web page showing an XP Box witha vSphere client and a number of the machines listed. He hadn’t even bothered to lock it down good and anyone and everyone could access it.
I nodded. My worst nightmare was unfolding before my eyes and ears. “How did you get the VMs?” I asked.
He looked uncomfortable for a moment and then explained. “I logged in through the vSphere client and made a snapshot. I was then able to just copy the VMX and VMDK files to my external hard drive. I took them home and opened them up in VMPlayer!”
“And so we’re having this conversation because you forgot to clean up after yourself.” It was time to ask the question for which I already knew the answer. “Where did you get the password?” I asked. “I asked them not to tell you because you’d said no.” He looked around and sighed. “I told them what I wanted to do, they thought it made sense, and they gave it to me.”
“Who’s ‘they’?”
“The company president and your boss!”
By this time I’d done a head slap, shook my head, and snarled, “Come with me!”
We walked into the president’s office and told him what I’d found and how his proprietary software was now out on the internet for anyone in the world to steal.
Later I got the whole story. He’d done the razzle dazzle, gee-whiz doesn’t this make sense dance, and they swallowed it. “We trusted him,” the company president lamented.
Needless to say, he wasn’t working there much longer. With the police in tow, we acquired his web server, and external hard drive and the mere fact he didn’t go to jail is because this security incident would have been too embarrassing for many.
Things quieted down a bit after that. As for Donnie, I haven’t seen any evidence he’s even in IT anymore. For all I know, he’s selling used cars someplace and is living happily ever after.
