The Folly of a Scheduled Patch Release Cycle

A number of years ago, Microsoft led the charge by moving away from a dynamic patch release schedule to a monthly patch release schedule, essentially creating an imposed monthly patch cycle for their customers.  Since then, many other vendors have followed suit.  There are opinions and arguments supporting both a release schedule philosophy as well as a release upon completion philosophy, and today I’m going to outline where I stand on the issue.

If you couldn’t tell from the title, I am 100%, wholeheartedly, against vendor-scheduled patch release.

That said, let me outline why:

First, regularly scheduled patch release cycles create a known window of opportunity for attackers.  If an attacker has an exploit for a new 0day vulnerability in a Microsoft product, and they know that Microsoft generally releases patches for vulnerabilities they’ve become aware of on the second Tuesday of the month, when do you think an attacker is likely going to start using said exploit, potentially disclosing the vulnerability, to maximize the amount of time that systems are likely to be vulnerable to it?  You guessed it, on the second Tuesday of the month.  This behavior on the part of attackers has been proven many times, as recently as this last Patch Tuesday.  This potentially puts customers at greater risk as the vendor may be sitting on a patch for a vulnerability that is being actively exploited as that patch waits in their queue for the next regularly scheduled release.

Second, regularly scheduled patch releases lock customers into, at best, the same patch frequency as the vendor release schedule.  This is actually tied to a lot of proponents argument for such a release cycle, as it supposedly allows customers to schedule predictable change management windows during which to apply the patches.  I would argue that most enterprises do this anyway; when a patch is released has little bearing on when it is likely to be applied, unless it is an extremely critical patch and warrants an emergency out-of-cycle deployment.  Regardless, shouldn’t these types of scheduling decisions be made by the administrators of the systems and enterprise management, based on factors relevant to them and their individual situation, rather than by the vendor?  Some enterprises may themselves impose a more infrequent quarterly patch cycle, while smaller companies may be dynamic enough to patch once a week, or without any schedule at all.  Release of the patches alongside information about the vulnerabilities, as soon as they are developed and tested by the vendor, would seem to be the only responsible course of action.

Third, regularly scheduled patch cycles also create enormous, unbalanced workloads once a month for pretty much everyone involved, except the vendor releasing them.  Releasing all of the patches once a month at the exact same time causes systems administrators to scramble  to test and prepare for deployment the ever-increasing number of patches released in order to get as many as possible into the next maintenance window.  Microsoft itself even acknowledged this problem in 2004 when it began it’s Security Bulletin Advance Notification Program, which releases some details of the patches a few days early to help alleviate the inevitable scramble every second Tuesday of the month by allowing IT patch testers to prepare early the test systems and applications that the patches will be applied to.

Industries and companies that are built on or rely on patches for content, such as my current and a number of my previous employers, are also affected.  The McAfee Remediation Manager content team (formerly Citadel Security Software) heavily relies on vendor patch releases for content.  TippingPoint, and other IDS/IPS vendors, rely on vulnerability and patch analysis for development of some of their vulnerability and attack signatures and filters.  As a Security Researcher at BreakingPoint Systems, I regularly reverse-engineer vendor’s patches to uncover the vulnerabilities being patched in order to write exploits, or “strikes”, for our BPS testing product.  The point is, ever since Microsoft switched to a monthly patch cycle, every second Tuesday of the month has been at best hectic and at worst, dreaded, for any number of IT departments and patch-related content shops around the globe.  It’s not surprising that many in the industry now refer to this day as “Black Tuesday.”  Microsoft attempted to address this problem with their patch cycle by recently launching the Microsoft Active Protections Program, which essentially gives the program member organizations a lead on vulnerability information with an associated embargo on discussing or releasing anything related until they themselves release the associated bulletins and patches.  This program alleviates much of the scramble of the security content shops by changing their hustle to get content out as fast as they can after release on Patch Tuesday to a development window prior to Patch Tuesday, but this program does absolutely nothing for the systems administrators that still must scramble once a month.

It is my opinion that while structure and predictibility are usually good things, when it increases risk to customers, forces behavior that may not be what’s best for those being forced, and creates bursts of unbalanced workload for your customers, it may not be the best course of action.  I hope that one day Microsoft and the other vendors that followed suit recognize that what works for them may not be in the best interests of their customers and move back to a more dynamic and responsive patch release system.

Tags: , , ,

2 Responses to “The Folly of a Scheduled Patch Release Cycle”

  1. Bob O'Malley Says:

    Hey Dusty,
    One counter point that I believe you are overlooking is that Microsoft (and probably others) switched to the scheduled release cycle not to convenience themselves, but at the behest of their customers. Before the scheduled cycle, IT shops had no idea (or a short lead time at best) when potentially significant changes would be introduced into their environments. They would get paged unexpectedly at 3 in the morning when their systems no longer functioned as expected. Additionally, because the releases happened haphazardly, it would often not be obvious what was causing their trouble.

    Sure the scheduled release cycles can cause a monthly upturn in workload, but this is an _expected_ spike, which is much better than being blind-sided.

    You are spot on about attackers having a prescribed window of opportunity, but if the vulnerability is that significant, the vendor is free to do an emergency release (which Microsoft has done on occasion).

    Anyway, good article. Hope all is well down your way!

  2. Dustin D. Trammell Says:

    @Bob O’Malley: Perhaps I should have elaborated this point better, but one of my points was that if you’re doing patch management correctly, when the vendor actually releases a patch should be largely irrelevant to the customer’s process. If it has no other effect, then obviously sooner is better than later. If you’re Doing It Right(tm), any patch that comes from a vendor should be going through your enterprise’s patch management process; the only aspect that the release timing the vendor chooses should affect is you being able to start this process sooner should you choose to, and that is of course entirely up to you as the customer. Thus, it should be the vendor’s responsibility to get the patch to customers as quickly as possible once it’s ready, not their prerogative to release whenever they feel like it.

    To directly address your points, admins should never get paged at 3am because their systems are all-of-a-sudden no longer functioning due to patching issues, because the patch should neither be deployed automatically upon release by the vendor nor without testing it first, both of which are essential in any patch management process.

    Your second point actually supports my position. When all of the patches for the month are released at the same time, go through the patch management process at the same time or in batches, and are applied at nearly the same time, assuming some issue does appear afterward, this situation provides much more noise when determining which patch is causing the problem. When patches are released individually as they become ready, go through the patch management process individually or with fewer siblings, and are then applied, when a problem shows up it is much more evident which patch introduced the problem.

    As I mentioned in the original article, and in response to your third point, it should be entirely up to the enterprise when to begin a patch management process. The timing of patch arrival from the vendor shouldn’t blind-side you, because you can choose to queue the patch until your regularly scheduled weekly, monthly, quarterly, etc. patch management cycle. The primary point of the article was that this scheduling should be, and is, entirely up to the customer. Since Doing It Right(tm) involves such a process cycle, it should be the vendor’s responsibility to get patches to the customer as quickly as possible so as to get them into the customer’s next earliest scheduled patch management process.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: