Recently, Disaster Recovery was a topic of several blog posts. So far we’ve identified the four pieces, and they are:
• Mitigation: lessening or eliminating the impact of an event
• Planning: putting together a document that will act as a script to help you get back up and going
• Response: the things that must happen in case of a disaster
• Recovery: putting it all back together again
Now we’ll be talking about something that we do every day but probably don’t give it a second thought, and that’s patch management. It’s something most folks look at as a major inconvenience simply because it involves rebooting of servers, workstations, and God knows what’s going to happen when the systems come back on. But it’s also as necessary as taking the next breath because without it, bad things can and usually do happen!
A quick story. Some of you will remember a little Code Red virus that came out years ago. Now, I’ve always taken patching seriously and I recall Microsoft releasing its round of patches. One thing I always do is read the information put out concerning the patch, what it’s supposed to do, what Operating System it’s meant for, and of course, potential impact and how to fix it if things go wrong. Anyway, I recall reading about this one patch and went, ‘Oh, yeah. That applies to us.” I installed the patch. A month later, Code Red hits. I still remember going out front for a break, and all the other System Admins and IT Managers were standing there chain smoking. All of them had that look in their eyes like they were being chased by wolves or worse. One of them said, “My God, Rich. We’re down completely. Totally dead in the water and I don’t know how long we’ll stay that way.” He took a puff and then asked, “How you guys doing?” I shook my head. “We’re fine,” I said. “No problems at all.” That statement drew an audience. “How?” someone asked. I took a breath, and said, “Because the patch that stopped this from happening came out a month ago.”
If there was ever any doubt in my little mind concerning the need to stay on top of patches that settled it right there.
Patches come in a couple of flavors. There are “Security” patches that are meant to fix some flaw in the OS that could be exploited by a virus or person. There are “System” patches that are meant to fix some little issue with the operating system or application that could cause it to fail. Then there are “Enhancements” which are designed to bring new features, or to make something better.
They also usually fall into a couple of other categories. “Critical” means just that you need to apply this patch, and yesterday would have been a good day to have done it. “Important” might wait a week or so. And then there’s “Low” and you can apply them when you wish, and in some cases might not want to.
And it’s not just Microsoft that needs patches. Linux has patches, and they need to be installed, Apple, UNIX, all of them. And then there are applications (remember, not all apps come from Microsoft), and these also have bugs or flaws that could make them a security risk. Patching them is just as important as patching the OS.
Welcome to the world of being overwhelmed! Needless to say, there are a lot of great tools out there to help you. Small shops, WSUS still does wonders for the Windows side of the house. Bigger shops, think SCCM. SCCM would, of course be the weapon of choice simply because you can do third party apps with it, and it keeps a great audit trail that of course your auditors will want to see.
But suppose you aren’t a big shop. For some of us SCCM is overkill, so patching third party apps is a chore. But a simple google search will turn up dozens of products that cater to this area. The good news, most of them will provide you the MSIs you need to use to update your 3rd party apps, thus saving you a ton of work.
On the subject of auditors. One thing they’ll ask about is formal plan on how you handle patches, and then of course they’ll see how close you come to actually meeting it. This is where your change management comes into effect and these are a few points it should reflect:
• Name of patch (KB name, etc.) – a brief description of what it’s supposed to fix
• Procedure for installation
• Evaluation of how you know the patch was installed (Microsoft Security Baseline Analyzer (MBSA) or similar)
• Your back out plan. Yes, you read that right. What to do if the patch doesn’t work the way you expect or causes other issues
• Identification of potential impacts such as reboots etc
• Approval process. This is important because you never know when your patch management might go up against something else (like processing checks for pay day)
One thing you should keep up on, and that’s vulnerability scanning. We’re not so much looking for problems as we are to show what we’re trying to do. An example of what I did was before the monthly round of patches, I’d scan each server with MBSA. This showed issues, patches I needed and so forth. After I applied the patches, all the servers were scanned again. These before and after scans become a part of the SOX process and the results should be kept on file. Some folks don’t like keeping them around because it tends to get auditors asking questions. My own personal Princess of Darkness loved asking questions like “Why are there more than two administrators on this box”. After a while I learned it wasn’t so much a criticism as it was to make me think about security. It was OK as long as I knew it, and was doing something about it (like getting rid of old admins or limiting who had access by using security groups).
A last word on patching. If possible, test them before you deploy them. We should all have some kind of sandbox environment to do just that. I call my test environment “Vegas”. It has absolutely no contact with the outside network which should give you your hint why I call it “Vegas”. There’s a Domain Controller and a few basic servers. If a patch is going to bust something, better there than anywhere else. Bad patches are a fact of life and better to weed them out ahead of time.
All this to say one thing, patching is a part of your Disaster Plan. After all, you’re mitigating (there’s that word again) against possible threats or problems.
So, till next time, “Happy Patching”!
