Compliance Unfiltered is TCT’s tell-it-like-it is podcast, dedicated to making compliance suck less. It’s a fresh, raw, uncut alternative for anyone who needs honest, reliable, compliance expertise with a sprinkling of personality.
Show Notes: Your Patch Management Process is Hiding Vulnerabilities
Quick Take
On this episode of Compliance Unfiltered, Adam and Todd put on their sleuth hats to help the listeners uncover the vulnerabilities hiding in their patch management process.
Would you consider yourself someone who thinks turning on automatic patching solves everything? Are you the type that’s pretty sure your IT department, “has this covered?” Does your organization, “not really have anything worth protected?”
Then this is the episode for you. Listen as Todd and Adam highlight the perils of dodgy patch management, and how you can best protect your organization.
All this on this week’s Compliance Unfiltered!
Read The Transcript
So let’s face it, managing compliance sucks. It’s complicated, it’s so hard to keep organized, and it requires a ton of expertise in order to survive the entire process. Welcome to Compliance Unfiltered, a podcast dedicated to making compliance suck less.
Now, here’s your host, Todd Coshow, with Adam Goslin.
Well, welcome in to another edition of Compliance Unfiltered. I’m Todd Coshow, alongside the tri-tip to your compliance barbecue. Mr. Adam Goslin, how the heck are you, sir? Oh, that sounds tasty. I’m doing good. Doesn’t it? tri-tip, it’s never a bad day for tri-tip, if you’re a fan of that type of thing. A barbecue is what I was thinking. Just saying. I had the opportunity as a younger man to go to the original Terry Black’s barbecue in Lockhart, Texas. And that is an experience unlike anything else, Cicadas in the trees, you’re standing literally in the middle of the street, waiting in line to have somebody cut some fresh barbecue for you right in front of your face. Crazy stuff. We’ve been through the winter months, shall we say. I’m looking forward to being able to fire up a barbecue. That’ll be good. Yes, sir.
Speaking of firing it up, we’re talking today, Adam, about how your patch management process might actually be hiding vulnerabilities.
Now, a lot of companies, especially these days, tend to run into issues with their patch management. Tell us more about that. Well, for a lot of organizations, they seem to have this notion that they’re absolutely positive they’re all over their vulnerability patching, and it’s working like clockwork, all of our patches are happening when they’re supposed to happen. The reality is usually much different and as a result, it introduces risk for the organization. The ongoing health of the company is depending on confirming and addressing your vulnerability patching procedures. Even though I haven’t had the opportunity to go dig into everybody’s organization, I’ve found few that actually have it completely together. Even if you’re confident that the company is banging it out of the park, in terms of their vulnerability patching, it’s definitely an area that every organization should go and take a closer look at. I think it’s definitely worth the review, if you will. Yeah, it makes complete sense to me.
Now on that point, many organizations believe that turning on automated patching solves everything. Why is this not an accurate statement? Well, there’s a lot of companies that they’ll lean on automated patch applications or automated alerts from systems. So just take the fabled Microsoft Patch Tuesday. The system wakes up and says, hey, time to apply patches. And you just assume that everything’s good. The problem with just a direct lean on the automated patching procedure is, that it can leave some dangerous gaps in the patching practices of the organization. The automated systems certainly aren’t going to cover every patch for every system, and every piece of software for the typical organization. There’s always going to be some remaining gaps that your team has to basically plug the holes on.
So something as simple as your antivirus, making sure that the engine and definition files are getting updated. Even with dialing on the automated updates to those things, sometimes those updates will come multiple times a day, as new vulnerabilities are getting thrown into the definition files that get distributed across organizations. There’s a lot of companies that go under this assumption that, hey, we’ve turned it on and all our machines are automagically updating. So we’re good. And unfortunately, well, no, because you start digging in a little bit deeper and usually where I’ll trip across this stuff is, as I’m trying to help the organization, whether it’s attempting to get it’s act together for going through an assessment, whether they just care about security and compliance, whether they’ve been doing this for years, it’s irrelevant. But invariably you go in, you look at the master console and sure enough, there’s one or more machines that they’re not updating properly. And there’s usually a myriad of different reasons for that being the case, not the least of which is that the machine is offline. There’s something wrong with the auto updating software, something changed with permissions, between the main host and the target host. But the bottom line is, most organizations have these similar points of failure within their company.
Well, for a lot of organizations undergoing compliance, they tend to believe that their IT people are also the security experts, right? Well, it’s one of the things that I’ve been saying for a decade plus, I’ve been saying it’s the biggest bad assumption that organizations make. Your IT people have this process that they’ve used for years and hasn’t let them down, but nobody’s perfect. Nobody can do a perfect job all the time. There’s a lot of pressure on the internal IT teams for organizations to really live up to unrealistic expectations. How likely is it that IT is then going to turn around, raise their hand, and shoot off red flares that they may not be the security experts that everybody thinks they are? It’s common for the executives in an organization to just assume that because these folk know how to do IT stuff, that they know how to do IT stuff in alignment with good security and compliance practices, even though they’re not security and compliance experts. In every engagement that I’ve been a part of, I’ve seen a broad spectrum of IT teams. Some absolutely did not have it together. Some were doing a pretty solid job. But across my career of assisting and helping organizations deal with security and compliance, I mean, it’s literally been, sub five companies that actually had it together with great vulnerability patching, every single other company needed some form of improvement. And so, there’s always room for things to improve. But the real key here, that I really want leaders to understand, whether you are middle management, whether you are C level, whether you are the owner, whether you’re on a board. Do everybody a gigantic favor, and get out of this notion that because of the fact that your people know how to do IT, that they also know security. Because quite frankly, it’s a presumption that is endemic in organizations. It really takes the folks at the top to get the light bulbs to go on, and realize there’s a difference between these people. That makes sense.
Is it appropriate for organizations to have the folks responsible for patch management, and also be responsible for validating that everything is okay? Well, I’m a gigantic fan of trust but verify, right? I mean, I’ve worked with some organizations that they were really, really freaking good organizations. They had good people, the people cared, they had skills. But I’ve never done an engagement ever where you know, we basically got to the backend of the engagement, prepping them for some type of either internal and external assessment, and come to the conclusion that, that was a gigantic waste of time because everything was perfect. It doesn’t happen. We’re talking about human beings, we’re talking about hundreds, if not thousands of controls. We’re talking about dozens, to thousands of systems. We’re talking about, tens to hundreds of different pieces of software involved in the scope. I mean, there’s a ton of stuff going on. Nobody’s perfect. So think about it from a company finance perspective, right? You’ve got a series of checks and balances that are put in place. You’ve got your internal accounting people doing the day by day accounting functions. You’ve got some type of a CFO that sits above them, or various layers of management to watch out for things, making sure things are being done correctly. But at the end of the day, most organizations would view it as negligent if they didn’t have day-by-day people doing the accounting stuff with some level of oversight. For some organizations, at some level of oversight it’s layers of management. Some organizations, it’s a controller. In some organizations, it’s a third-party financial audit. You look at it this way, for organizations that are perfectly willing to apply that level of rigor to their financial health, but they don’t apply that same standard against their IT functions.
IT is more like the Wild West in most organizations comparatively. In a lot of cases, there aren’t built-in checks and balances for the internal team to make sure that the systems are actually protected, and as a result that introduces risk. So it isn’t a stretch to say to organizations that, while no offense to the financial people, but if I get a one or a zero wrong or something, yeah, okay, we thought we were in one place, but we really aren’t, it’s not the organization changing generally, but it certainly could be in the security and compliance arena. The mistakes in that space are definitely things that could potentially put the organization at substantial risk, or out of business. Every organization, they’re too close to their own IT activities, even to their own security-related activities to be really capable of stepping back and seeing gaps in their security. It’s almost like you can’t see the forest for the trees. And so there’s a definite value to having some third party consultant come in and evaluate your security program, partnering with your IT folks, or your vendors of choice. And just like you’re doing financial audits on a regular basis, your security and compliance needs the same type of regular auditing. And both of these are arenas for an organization, they’re both too critical not to do it. Even if your organization isn’t subject to a third party assessment that you have to go through each year against fill in the blank standard, there’s nothing forcing the oversight of your existing IT personnel. And certainly having that sanity check in place, having that third party consultant to provide some measure of oversight, that’s not part of the team type of a deal, it’s a huge way to be able to take a good step in the right direction for protecting the organization from a data breach. That’s for damn sure. Yeah. I mean, that makes sense.
A lot of companies don’t believe that they have anything worth protecting, despite their contracts mandating a strong security and compliance stance. Now, why is that not the right approach? Well, I mean, I’ve heard the expression from a number of organizations, oh, we don’t have anything sensitive, any sensitive information for somebody to take. And you’ll hear that comment from organizations.
It’s almost like they’ve convinced themselves that, because of the fact that they don’t believe they have anything worth protecting. Thereby they don’t need to spend any money on security and compliance, and doing it properly. A lot of times I’ll get the, well, all we have is first names, last names, phone numbers, addresses and emails. And this is publicly available information. So, why do we need to spend our hard earned dollars impacting our bottom line, by putting things in place to go protect that? I mean, I would counter with, okay, well, then those same organizations, if they do some digging and research on the companies that have information that was breached that only had PII, they’re going to very quickly discover just how pissed off their customer base is going to get when it happens to their company. You can find your business, very quickly, in really deep trouble. I was working with an organization, they were doing this back in the day, they were literally doing rebate processing, okay? So you go to the store, you buy fill in the blank piece of technical equipment, and then you get five or 10 bucks back. And of course they’d have to write down their name, address, phone number, and email, right? And so, that was it. It wasn’t state secrets, it wasn’t some proprietary methodology, it wasn’t nuclear codes, it wasn’t health data, it wasn’t credit cards, literally that was it. And this company found themselves in a situation where their data, because of security holes, got exposed. And I’m gonna tell you what, that company, I literally sat there and watched them. I stopped counting when they passed putting in a quarter million dollars trying to put the pieces back together. That was just the out of pocket expenses, that didn’t even count how many of their clients ended up hearing about it and began jumping ship.
So the light bulb that I really desperately want to go on for organizations is, you can very quickly find yourself in very deep trouble, even when it’s publicly available information. It’s your company’s responsibility to maintain a safe place for your clients, vendors, partners, and employees to entrust their information and data with you. I guarantee you, there’s probably things in your agreements that require certain things of your organization. It’s almost an assumption at this point in the game, that they’re trusting you with their information and data. And, they fully expect that you are going to put your best efforts behind protecting it. And quite frankly, the longevity for some of these organizations is literally on the line, even when it comes to just PII.
Well, how should organizations go about validating their patch management approach correctly? Well, there’s a couple of different steps. I’ve broken it out into four different phases for organizations to go through this exercise, and really get their arms around things. So first, go in and create and maintain accurate inventories. And when I say inventory, what I mean is a full-scale inventory of any, I’m going to use the term loosely hardware in today’s virtual world, but all of your hardware, whether it’s physical or virtual, your laptops, your workstations, any equipment that you’ve got that’s on your network, knowing what you’ve got is one piece of the puzzle. Better yet, you also want that inventory list to include software that’s being leveraged. So go look at each of your servers, each of your workstations, and laptops, figure out what’s all the software we have that we need, and putting that together. It sounds like an easy task, but legit for giant companies, this can be an absolutely daunting challenge. Not only do you need to create the inventory, but you have to actively maintain it as you’re deploying new assets, and deprecating assets. Imagine you’re in an organization with whatever, 1,000, 2,000, 5,000, 15,000 different assets. Yeah, that is a lot. a lot of moving pieces and parts.
So, phase two of the process is going through and doing validation. So, I’ve now compiled my inventory, and my inventory is ostensibly reflecting, what you actually have installed on each asset, not what should be installed. Now you can go through and do several validations, review every item in the inventory, make sure it’s receiving the various updates that are needed, and verifying that you have a source for provisioning those updates for each of these various items. Also, verifying whether that update is going to be automatically applied or manually applied. You may have a mixed bag there, where automatic updates are available for 60% to 80% of the stuff you’ve got, but there might be 20% to 40& percent where you have to go in and actually go pull an update on a periodic basis. If you’ve got manual updates, which quite frankly, most organizations will have at least one, then making sure that you’ve got a schedule for how often we’re going to check to see that there’s updates. In some cases with the ones where you have to apply the patching manually, the vendor may have some type of an alert that they’ll send out that you can subscribe to, but in other cases I’ve seen it where the vendor doesn’t even notify people that they’ve got security patching, and you have to go check their website. You want to make sure you’ve got that together. You also want to make sure there’s an audit trail for tracking the updates for each of the assets, and pieces of software that are in your inventory. Just being real, don’t expect that this process that you’re going to embark on, this isn’t a few day or a few weeks extravaganza. This can be a huge project for organizations, which quite frankly, is the reason why a lot of organizations don’t bother going down this path. They’ll just be like, hey, we’ve got the automatic updates. It’s all good. I was wondering if you were going to call it out. Depending on the size of the organization, the current state of your inventory and vulnerability patch management, there can be a really deceptive amount of complexity that’s involved in this arena.
The next phase of this process is plugging holes, and remediating issues. Not only do you need to validate that your patches are happening with regularity and properly, but that they’re actually getting applied. But then you’ve got to go and plug holes, fix issues and gaps that you find along the way, and you’re going to find them. Now I’ve got the right list of stuff that I need to go do something with. I’ve gone down and assessed where all my patching is happening. But when you go look at the systems and go, okay, let’s see if this is working, yeah, you’re going to find problems. In some cases you may discover that your current version of software is so far out of date, and you can’t just go download the latest version. I’m currently on version four point something, but I need to get to eight point something. Well, I can’t just go and install version eight. I have to instead, go step by step to version five, then go to version six, and then go to version seven. So there’s a ton of things that you’re going to trip across as you go down this process. So for anybody that’s been involved in the application of the patches arena, you’ve also got regression testing as you’re going through this, making sure that by applying these patches, we didn’t break something else functionally and lose functionality along the way.
And the last phase of this process is just the rinsing and repeating. Once you’ve gone and actually accomplished this initial exercise, that’s great. But, you want to do the trust but verify routine. Have some type of ongoing validation. Don’t just assume your AV is real. Hey, we dialed on all the updates for AV, so we’re good. No, go in and periodically do a quarterly pulse check. Is AV actually getting applied everywhere it’s supposed to? Do that periodic sanity check of the application. Because the problem is when you find that you do have issues, there can be ripple impacts that start happening within the environment that you’re not even in the know about, you know what I mean? I do.
Now, what should be done about the lower level vulnerabilities? A lot of companies deprioritize those, right? So why is that not a good idea? Well, yes there are organizations that will only patch. So as an example, depending on what security framework you’re leveraging, you could get this rule that says, well, we’re only going to have to patch medium and higher externally, or you only have to patch high in criticals internally, right? That’s the lowest bar that you can possibly hit in accordance with fill-in-the-blank requirement. Well for a lot of organizations, that’s typically where they’ll start. And especially if they’re walking into this for the first time, the problem is that they typically stop there. They’re like, oh, we made it through that process, now we’re going to go back to normal daily life. And they’ll deprioritize those minor vulnerabilities, which for a lot of organizations, they’ve just gone through the ringer and it seems to make sense that, oh, we don’t need to waste more time with patching all this other crap. Well the problem is, is that approach of leaving these mid to lower level vulnerabilities there is, you can leave your organization open to a disaster. Even if you don’t have any critical vulnerabilities, if you have the right or wrong combination of minor vulnerabilities, then bad actors now have the capability to go in, and take advantage of them, combine those vulnerabilities, and create a much larger problem. Maybe they’re aware of a, I don’t know, a not life threatening zero day, but the non-life threatening zero day vulnerability now is amplified. Because now, I can take advantage of this plethora of options I have of moderate to low level vulnerabilities, that I can now double into the new issue that’s been discovered. So the bad actors will definitely take a shot at taking advantage of combining all of those things. It’s like throwing together some hydrogen peroxide and vinegar. They’re both harmless on their own, but you put them all in the same container, and give it a shake, and you’ve got some lethal acid on your hands. But to the degree that you’re able, my recommendation to organizations, and I understand that they’re going to start right out of the box with, we want to do our medium and higher externalities, and our highs and criticals internally. When you get there, start working your way down the rest of that list. If you’re able to go in and get those applied, you’re now eliminating possibilities for the bad guys to go in and take advantage of.
What types of issues can application of patches cause, like the actual process of it? Well, one of the downsides when you’re going through the patch management, I alluded to it earlier, is you go turn on a particular patch and poof, it goes and just breaks your existing functionality. Any time that you’re doing patching, you need to consider your risk tolerance level, determine if the types of patching that you’re doing should be tested out first, apply those patches within a test system, perform validation making sure everything still works, run your regression testing before you just throw off all of the guardrails, and just let the patches loose right to production. For some organizations, depending on what you do and your dependencies. I’ve seen organizations that have done that, especially back in the day, oh my God, the Microsoft Patch Tuesdays back in the day. I know there’s people that are chuckling, oh my God, it used to be a bloodbath for the IT folks. All of a sudden, all the systems just automagically got updated with whatever effing garbage just came out of Microsoft, and all hells breaking loose. I saw a lot of organizations back in the day where they would literally have a subset of systems that would get the Patch Tuesday patches, and a lot of them would not apply them on Patch Tuesday. So what they would do is the week after Patch Tuesday, or later that week, Friday or something, they would apply it to a subset of systems and then they would roll them out to the rest of the systems early the following week. If you can, go through and choose victim systems to go in and get the initial round of patches, do your validation and then roll out to the rest. It’s usually a pretty good way to go about doing it. Generally today, the AV software, the patching for the engines and virus updates, those don’t tend to cause problems with organizations anymore. But certainly, testing out your patches for your web server software, or third party components that are supporting your web server, and especially if you’re doing custom coding, that’s definitely an arena where you want to take it slow and make sure that you’re doing all of the accompanying testing, so that you’re not running into production issues through the process.
Good stuff. Parting shots and thoughts for the folks this week, Adam. Well, you definitely want to go down the path of reducing risk. I’ve told the story before. I was an IT leader at an organization, this is when I first cut my teeth in compliance. I had to go through a full scale security compliance engagement for the very first time. I had to prove out that we had, just this litany of controls in place so we could pass the assessment. And that process of going through it, man, as a career long IT professional, it taught me a ton. I learned how little I knew about security and compliance. The scarier part for me as a leader is, I was under that assumption that all my IT people knew what they were doing in security and compliance. I was just management, right? I don’t need to like actually know this stuff, but these guys do, right? I mean, I had day by day IT people, I had network administrators, I had varying levels of developers, I had DBAs, I had a hosting company. They all knew what they were doing, right? Yeah, no, I was astronomically wrong. And, that’s where I come full circle to that whole don’t make the bad assumption, and I was doing it. I was doing it myself, which is how I learned this lesson, right? Do me a giant favor, do not assume your IT people have their arms completely around their vulnerability patch management, and it’s wholly buttoned up. Cybersecurity isn’t IT. Your company needs to have highly skilled IT people, and don’t get me wrong, they are good people. Don’t flame them because they don’t know security stuff. It doesn’t make sense. It’s like saying, I can drive my car, so I should know how to rebuild the engine. Well, I don’t know what to tell you. I drive the hell out of my car, but I don’t know squat about rebuilding an engine. So, that’s the similar assumption that folks are making. So, don’t be that guy or girl. You don’t want to find out the hard way that your patching isn’t getting applied properly. Simple mispatches can have the capability to put companies out of business, and it happens too often.
So bottom line is, that we’ve got the capability to provision assistance to organizations that are struggling with getting it together. We’ll be happy to have a discussion with you and see how we can help. That’s part of the reason why we got into this space. We wanted to help make managing compliance suck a lot less for people that are facing it. So, we’re definitely here to help you as you go through the process.
That right there, that’s the good stuff. Well, that’s all the time we have for this episode of Compliance Unfiltered. I’m Todd Coshow.
And I’m Adam Goslin. I hope we helped to get you fired up to make your compliance suck less.
