In 2006, the CERT program at Carnegie Mellon's Software Engineering Institute reported upward of 8,000 application vulnerabilities that required software patches--that's 30% more than in 2005. We've had years to get this process down, yet patching continues to cause a great deal of angst. We frequently see organizations that are more than a month behind on patch applications--and open to viruses and security violations. Why take that risk? Too many IT groups lack the tools, processes, and resources to patch effectively.

No fewer than 14 vendors are looking to rectify that situation. Each product has strengths and weaknesses, and we're hoping to get most of them into our Real-World Labs in the near future. See our automated patch management Rolling Review invitees and requirements at Rolling Reviews.

Ideally, patch management will be just one element of a comprehensive configuration management or software distribution system in larger shops. Smaller companies can get by with standalone tools, but many need several point products for different types of apps and devices. But however you manage it, automation is critical, as are documenting changes, testing to ensure that patches won't break other apps, and deployment policies to avoid bogging down networks.

(click image for larger view)

WHAT HAPPENS IN REDMOND ...
While the need to patch applications is as old as computing, the volume of Windows updates coupled with Microsoft's market dominance have focused attention on the issue. Since the introduction of Windows 98, Microsoft has looked to automate patching of Windows servers and desktops. Its current incarnation, the Windows Server Update Services, or WSUS, provides a locally managed software update service alternative to the local Microsoft Update system. Using WSUS, IT can automatically distribute patches and updates to clients from a central server.

The current version expands on the range of software it can update and is a big improvement over using the Microsoft Windows Update Web site. You'll save bandwidth, time, and disk space, because individual computers don't have to connect to an external server. In Windows Server 2008, this capability will be native to the application; currently, WSUS is a free download from Microsoft's site.

Free's nice, but most organizations have more to their worlds than Windows desktops and servers. And Microsoft's free tools don't provide the flexibility or scalability required for larger organizations.

Top 7 Questions To Ask Vendors

1. Is your software focused on desktops or servers? For those devices, what specific operating systems and versions are supported?

2. Does the software have the ability to control network bandwidth usage when patching?

3. How does your product deal with the failure to apply patches?

4. How does the software discover what applications and operating systems are in your environment?

5. Can the product integrate with other inventory or configuration management products?

6. Can it report on successes and failures and provide me with a full audit log of all my patching?

7. How does the product become aware of and prioritize patches that become available?

Patch management tools often are found in software distribution, provisioning, and configuration management suites that, while relatively expensive initially, grow as your needs expand. Whether they're worth it depends on what type of patches you need to deploy. If you're responsible for network devices as well as servers and desktops, a tool that has the ability to handle all of them, such as HP's Opsware, may be worthwhile. If you're on the hook for desktops only, consider a narrower offering, like CA's Patch Management. If you need server patching, do you run only Windows, or do you have Unix, Linux, or virtualized systems?

Automation is critical. Manual patching is an unacceptably labor-intensive process. Develop a detailed list of every step in your patching process, including gathering patch information to determine severity and priority, doing detailed staging to uncover if the patch will affect other systems, and deciding what endpoints will be updated. Ask whether all those steps can be automated in the software you're considering.

Change control, as it relates to patch management, also is important. How often and when do you apply patches? Who can deploy and/or authorize updates? How are patches tested? What problems will trigger a rollback? Knowing how many managed devices you have, and will have in the foreseeable future, also is critical when looking at software. Offerings range from managing 50 to 100 devices up to scaling into the hundreds of thousands. Larger companies considering configuration-related products, such as software distribution or configuration management databases, should ensure that robust patch management capabilities are included. If you have an asset and inventory system, check that the patch management function integrates, or you'll end up having to do discovery.

Given the negative impact willy-nilly patching can have on users and the network, look at how products deal with utilization and devices that aren't connected at the time of the patch. Can it employ multicast distribution, advanced compression, and checkpoint and restart capabilities? If a communication link goes down, the end device is an offline laptop, or for some reason application of the patch fails, what happens? Ideally, the software will have a methodology to attempt to patch the application again and escalate notifications and alerts based on repeated failure.

Reporting is important, too. Ensure that the product can support auditing as well as notification. In many public organizations covered under Sarbanes-Oxley, this is a strict requirement, and neglect could result in substantial penalties.

SOFTWARE VENDORS TO THE RESCUE?
In general, there are four classes of patch management products, based on what they patch and how they use agents: Windows desktops and servers with optional agents, Windows desktops and servers with required agents, multiplatform systems with required agents, and multiplatform systems with required agents and a virtualization support/data center focus. Here's a preview of some products we hope to test.

Windows desktops and servers with optional agents:
Many organizations are willing to devote specialized resources to patching their hundreds or thousands of Windows desktops and servers before a virus or malicious code spreads throughout the network, resulting in substantial downtime. Shavlik Technologies' NetChk Protect combines patch and spyware management with the option of using an agent-based or agentless architecture; it can patch nearly 700 applications as well as Windows operating systems. Many vendors, including BMC, Microsoft, and Symantec, use Shavlik products within their patch management suites.

Like NetChk Protect, Ecora Software's Patch Manager gives IT administrators the option to use agents or run without them. With a concentration on discovery, patch assessment, and patch installation on both Windows workstations and servers, Ecora uses bandwidth throttling to limit the network resources dedicated to patching. Critical functions such as patch rollback, wake on LAN, and the ability to designate a test environment for patching before production deployment are all promised in Ecora, and a variety of reports are included that will aid in auditing and compliance.

Top 7 Patching Mistakes

1. Not testing the patch prior to rollout.
Some patches will be incompatible with, or even totally break, other applications.

2. Not having a rollback version.
If the patch fails, you need the ability to retreat gracefully.

3. Killing the network.
Deploying multiple patches at the same time across the network will grind other applications to a halt and raise the ire of users.

4. "Reboot required" during production hours.
Many patches require users to restart systems. When possible, deploy these after hours.

5. Missed patches.
Often a user call or virus initiates a frantic search for a missing patch. Stay on top of releases to minimize downtime.

6. No patch audit trail.
If you patch regularly, you need to keep track of what fixes were applied, and when, for auditing and reporting.

7. No formalized patch process.
A defined policy that specifies who may approve patches and the procedure to deploy them is critical to patch management success.

Windows desktops and servers with required agents:
Agents are typically used in environments where IT may not have dedicated access to managed devices, such as laptops that connect sporadically to the corporate network. Agents may also come in handy when you need to distribute network traffic related to patching, and they tend to provide tighter control over devices.

Kaseya's Patch Management software automatically discovers missing patches and updates, and it can automate deployment and installation of patches on a defined schedule. Once initial scans are completed, IT can review results for each machine and decide if, when, and how each missing patch or update will be applied. IT administrators also can track and approve patches for auditing and reporting.

Novell's Zenworks Configuration Management focuses on notifying IT when a new security update exists and ensuring that the update has been staged for distribution. Novell provides a team of security experts to track software vendor support sites and update feeds to organizations.

Multiplatform with required agents:
If your environment includes Unix, Linux, and/or Mac OS, you'll turn to vendors that support cross-platform applications and operating systems. In addition to offering a broader suite of policy management products, BigFix provides patch management and security update delivery for major operating systems as well as common applications, and it can handle more than 50,000 devices. You will need to run the BigFix Server on Windows and deploy an agent on every managed node.

LANDesk's Patch Manager includes a subscription service that will collect and analyze patches for heterogeneous environments. Like other suites, it scans managed devices to identify application and operating system vulnerabilities; when it discovers an issue, you can download the associated patch and research requirements, dependencies, interactions, and known issues. LANDesk monitors the status of each installation and provides bandwidth throttling, staging, and detailed policy and compliance reporting.

BMC Patch Manager, formerly Marimba, provides testing capabilities that allow administrators to minimize risk by analyzing the impact a patch will have on an endpoint. The BMC Patch Manager Policy Engine facilitates the initial patch installation and continually monitors patches to ensure that they stay installed. Lumension's PatchLink Patch Management suite automates the collection, analysis, and delivery of software patches to a wide range of operating systems. It also focuses on reporting.

Other vendors deploy configuration management databases to manage and control patches. Configuresoft's Enterprise Configuration Manager automatically discovers new systems and tracks configuration changes at scheduled intervals to ensure that the latest patch information is available. It groups machines by function or role and supports patch testing across different configurations. The software continually updates the patch status of all machines and maintains an audit history of patch deployment, extremely useful for compliance reporting. Similarly, Symantec's Altiris Patch Management is focused on a central, extensible repository.

Multiplatform with required agents, virtualization support/data center focus:
As organizations move toward data center consolidation and virtual machines for cost reduction and improved efficiency, ensuring that IT can maintain those consolidated servers is key. Patch management is critical, and care must be taken to work with vendors that can support complex, heterogeneous operating system environments that include servers running Windows, Linux, and Unix on VMware. HP's Opsware/SAS and BladeLogic fit well here.

BladeLogic is focused more on the data center server environment than the desktop world and supports operating systems, server components such as middleware, utilities, system software, and multitier apps in virtualized environments. BladeLogic Configuration Manager uses a policy-based approach where changes are applied to a policy, then synchronized with target servers. The company says this bidirectional method significantly lowers the costs and errors associated with managing servers. Configuration Manager also features a cross-platform command line interface that supports single sign-on using a range of authentication protocols. All communication is encrypted, and all user actions are logged and can be authorized based on role, which is key for highly secure environments and something that may not be available in midmarket products.

Opsware SAS will automatically discover server hardware, configurations, and software. With broad configuration and provisioning capabilities, SAS can identify and patch a large number of servers as well as create and enforce patch policies. SAS also uses best practices in audit and remediation definitions to enable fast response to security or compliance vulnerabilities that require patches.

Welcome to
TechWeb, the IT professional's online resource for news coverage of the
information technology industry. We know technology news. Our mobile
and wireless news coverage moves as fast as wireless technology itself.
We follow all the devices you depend on to stay connected. Our software
coverage follows the multi-faceted software industry from every angle.
We've got a lock on network security and computer security issues.
We're all over the business of the Web--the Internet business--and the
engines that run it. We have our eyes and ears tuned to the players who
make and run the tools that tie us all together--Google, Microsoft,
eBay, Cisco, Yahoo, Oracle, Apple, Sony--and scores of others. And we
keep close tabs on the backbone of information technology, PC hardware.
We know PCs and Apple computers inside and out. We cover computer
technology, computer news, software news, search engine news, business
software, operating systems, and software development. Our coverage of
tech news includes a strong focus on the security business, its
attendant spyware and viruses, how security relates to wireless
technology and business networking and the security issues surrounding
RFID technology. We closely follow developments in Internet news and
Internet technology, including the spread of broadband and its effect
on Web browsers and the Web business. We watch the VoIP business, and
how VoIP technology is affecting the state of telephony in the
enterprise. And if all that isn't enough, we also track developments in
the IT industry that affect IT jobs, IT careers, and outsourcing.