The story goes that there was a Fortune 500 company that woke up one morning and found that they had over 5000 virtual machines. Right away, the question came up, if they are being used. So they did a very simple thing to find out. They shut them all off and waited for someone to scream. Almost no one did.
I heard the story years ago, and truthfully, I’ve never tried to verify, if it was true or not. But my own experience with virtualization says, it could easily be rooted in truth. Most people would look at this and say, so what. VMs don’t cost anything to build.
And nothing could be further from the truth. In the above story, each VM uses CPU resources – even if it isn’t doing much, it is still wasting a tiny bit of it. It’s using RAM, again, maybe not much, but still it’s some . . . Disk Space, now that it definitely is using. Even 40 or 60 GBs means space that can’t be dedicated to another machine (unless I want to go buy more storage). Licenses – definite expense.
It all boils down to TANSTAAFL. What’s that you say? Oh, it’s only “There Ain’t No Such Thing As A Free Lunch”. You’re paying for it someplace, in this case in performance, and yes, cold hard cash for something that isn’t making you money. So keeping control of the production and distribution of VMs is an important business. And the only way you’re going to do it is with a system. Someplace, somewhere, you have to come up with a way to regulate how they’re built, and that means paperwork, and that means a system of checks and balances.
Before you ever start building your first VM, people need to know that there is a system in place, and that system starts with a form. What’s on this form? Well here are a few things to look at:
- Purpose of the machine
- Name of the machine
- Number of CPUs
- Amount of RAM
- Disk Space, how many disks, and how much
- Operating system (Windows, Linux)
- Networking (any special VLANs)
- Backups
- Monitoring (if any)
Part of this is to encourage only serious customers to apply. The rest of it is cue card for us. For example, unless you work in a small shop, someone else probably handles SANS, and someone else is the Network guru. Again, someone else might be tasked with backups, and someone else with monitoring. This tells us who our approval people are, and they need to be at the table when the request comes in.
What we do is setup a provisioning meeting, and all the owners of the various processes are at that meeting. The requestor has to then explain what he/she needs and the rest of the meeting types kick it around, and see what they need to do to make it a reality. Keep in mind, that everybody at that table can veto the project, put it on hold, or dismiss it.
Another thing, if you’re the VMware Administrator, you should have a good idea at all times what each VM is doing. Always keep in mind, there is absolutely nothing wrong with making a VM pull a double or even triple duty (we do it all the time in the real world). That’s always an option, especially if resources are starting to close in.
But all the paperwork and meetings in the world are pointless unless you have a way of policing the system. There’s a lot of things you can do from actively watching to passive reconnaissance.
First, one of the things you should have done from the beginning was ensure only a certain number of people can administer the Host machines, and do things like build VMs or change their resources. Realizing this is going to be a nice way to really annoy people, who are used to doing things their way, but if you want to make sure there are no unauthorized people building VMs, then you have to authorize only a handful of people to do so. And make sure everyone is using their user account to access the host. One thing SOX auditors will look for is an audit trail, and if they’re all using root or administrator to access the host, trust me you’ll take a ding and have it coming.
So, how do we keep an eye on things? There’s a lot of ways to do this, one is plain old fashioned scripting.
I’m staying with PowerCLI (I’m mostly a VMware guy) and while I’m sure there are ways to do this for Hyper-V or Xenserver, I’m the guy that doesn’t know them. PowerCLI has a lot of capability to give you some reporting features, and very few people tap that potential better than the good folks out at virtu-al.net. Here you’ll find a script, that will show you the last ten VMs built. Incidentally, this is something you might want to run every couple of weeks (more often if you’re a busy house), and then the report becomes part of your auditing process. Of course what you’re doing is checking new machines against your requests for builds, and this becomes part of your SOX audit report.
There’s tons of other scripts out there, that will help you out, many will show things like snapshots made and give an age for them, and will even give you 20 cent version of Capacity Planning.
Oh, if you’re an IT Administrator and you don’t know PowerShell, my advice is to learn it. It will be the most career enhancing move you’ll ever do.
Now if the command line is something that terrifies you, you can try simple reporting software, something that will alert you anytime a VM is built. VMware vCenter comes stocked with a great tool that will help, and it won’t cost you a dime.
To do this, Log into your vCenter server, and in datacenter, go to Alarms:
Right click and select “Add Alarm”.
In the General section, give the alarm a name. Since we want to know when a VM is built, I’d suggest something that describes that. Also, in Description, say exactly what it’s supposed to do (alert us when a VM is built, which implies we need to configure VMware to send Email out). We’re monitoring Virtual Machines and we want to know, whether a machine has been built or not, so we select the second in the row there.
Under triggers, “Creating VM” is what we want to select.
Under actions, have it send you an email.
The downside of doing things this way is that there are a lot of things to watch, so you’ll have to create several rules or pieces, but at the end of the day, you can really help guard against unauthorized VMs being built.
The easiest way to do this is go and use a free tool like Netwrix Change Notifier for VMware. Any time a change happens to the environment, it will send you a report the following day.
Here’s a screenshot of what the report will show you:
Note, that it showed the addition of the datastores, and what exactly happened. The bought and paid for version (Netwrix Auditor for VMware) will also show you who did it and if you’ve someone rogue running about, it will certainly show you who they are, or at least how they’re getting in.
