Key software patch testing best practices

Why is software patch testing important?

Software is never going to be completely bug free, even when software development best practices are followed. That's why we have vendor patching, a post-release update that fixes any newly discovered bugs in software, whether it's operating systems, network equipment firmware, applications or mobile apps.

Although software vendors rigorously test patches before releasing them, they cannot possibly evaluate the infinite possible configurations and scenarios where every patch will be deployed. The responsibility for checking that there will be no issues when a patch is installed rests with each organization deploying it.

Patches can cause a device or application to fail immediately, with the scale of the impact correlated with the importance of the device or application. However, some issues only appear when a particular combination of events triggers an error.

To avoid these potential outages and disruptions, you need to test patches comprehensively before installing them on production systems. Testing patches before deployment is perhaps the most critical step in the enterprise patch management process to avoid the risk of patches destabilizing the IT environment.

How do you test a software patch?

The aim of patch testing is to determine if a patch will cause problems for an organization's unique combination of hardware, software and configuration settings. The best way to do so is to create a test environment that mirrors the production environment as closely as possible. This can be expensive if it entails maintaining duplicate real-world environments. At scale, it becomes impossible, so often a representative sample of assets is used.

For example, a patch might be installed only on a test group of employee workstations, so if it doesn't install fully or causes problems, most employees won't be affected. Another alternative is a virtual test environment, discussed below. The ideal scenario is to be able to perform comprehensive tests when needed and at an acceptable cost.

Patch testing involves more than just ensuring that devices reboot correctly. You also need to check that a patch has been deployed successfully, and run smoke tests to determine if all the main functions of the software appear to work correctly.

Tests involve comparing application performance before and after patch deployment and ensuring that other applications running on the target environment are not negatively affected. It requires simulating usage cycles and seasonal peak workloads because some problems only appear above a certain threshold or when certain functions are executed -- monthly payroll, for example.

Automating these kinds of tests can reduce gaps in security updates. It is also important to confirm that a patch actually fixes the identified problem, whether it requires a system reboot to take effect and that it can be removed or rolled back without causing problems.

Finally, details about individual patches, test results and rollouts should be documented in the change management logs.

Security patch testing best practices

The patching process is a key part of vulnerability management, so it's important to get it right. Employing the following software patch testing best practices will reveal any conflicts with existing configurations that are unique to the systems on which a patch will be installed, thus avoiding disrupting business operations.

Determine what's most critical

Identifying which security issues and software updates are relevant to your environment requires a strong asset and software inventory. In complex environments, the best way to manage the process is to use an automated patch management tool.

Assign all assets a criticality level to help assess their importance in business processes, optimal downtime periods and vulnerability risk levels. Prioritize patches and test, schedule and deploy critical patches before less essential ones.

Change management is vital and should include tracking every update and each step of the patch management process. The change management policy should describe the processes used to identify and deploy patches, as well as who owns each step in the workflow.

Vendors routinely issue patches, and it's important to make sure you know about the most critical patches. A helpful tool is the Common Vulnerability Scoring System (CVSS), which rates the severity of security vulnerabilities across a wide range of software products.

Because it is application- and vendor-neutral, security teams can use the CVSS to more easily gauge the impact of vulnerabilities on their systems and prioritize which ones to fix first. For internally developed applications, use a software composition analysis tool, which tracks all the open source and third-party components used by an application. It's the best way to keep track of vendor update and patch announcements related to those components.

Once a patch is downloaded, verify its source and integrity. A digital signature is typically provided for this purpose. Unless you're facing an imminent threat, take time to see what the relevant forums are saying about the patch and whether anyone is reporting problems post-installation. When following patch testing best practices, it's not always best to be first.

Replicate environments through virtualization

Virtualization is a valuable part of a software patch testing strategy because you can replicate production environments on one computer, preferably using the same hardware. Running several OSes virtually saves time, space and money.

Virtualization lets you verify that applying patches will not result in unexpected or undesirable system behavior. However, infrastructures have become more complex, and exposing a patch to as many scenarios and state spaces as possible is difficult. Public cloud services, such as Microsoft Azure and AWS, are a cost-effective way to create a dedicated patch testing environment that's identical to your production system. Online services that replicate all or some of the production environment are another option.

Software patch testing involves more than just ensuring that devices reboot correctly. You also need to check that a patch has been deployed successfully, and run smoke tests to determine if all the main functions of the software appear to work correctly. Look closely for unanticipated changes in the test environment, such as the following:

• Application failures.
• Changes in permissions.
• Newly disabled or enabled services.
• Disrupted services.
• Negatively affected code.

If testing produces an unsatisfactory result, identify the root cause of the problem before going any further.

Production rollouts -- if conducted in stages -- are another part of testing. Apply the initial rollout to less critical systems, and if they perform as expected, continue the rollout until every system is updated. The testing process is finished when the full rollout is complete and no issues have been reported within a week. If any legacy system cannot be patched, it's vital to devise an alternative strategy to mitigate resulting threats.

Be careful when rebooting

At least 90% of patch deployments will require a system reboot. Plan a maintenance window to apply the patches to the production system to eliminate an unexpected reboot that could interfere with business operations or cause other problems. Consider a variety of patch strategies, such as conducting updates on a regular monthly schedule, a big weekend event to patch all systems at once or even deploying small updates throughout the month. The goal of software patch testing best practices is to avoid unexpected patching problems appearing all at once.

Even with a thorough testing program, it is wise to have a contingency and rollback plan in case something goes wrong so systems can be restored to their pre-patched state. Make a backup or image snapshot of every system before starting patch deployment. If staffing is an issue, consider an outside vendor. There are companies that provide patching services, such as testing patches for commonly used applications and preparing scripts for deploying patches to production systems. Be wary, however, of using auto-update tools on mission-critical systems because you will have far less control over when patches are applied.

Additional testing considerations

Keeping a large IT environment fully patched is a time-consuming job which, if not managed correctly, can result in unnecessary downtime or vulnerable systems. To make patch testing and rollout processes easier, consider consolidating and standardizing the software used throughout the organization. This will greatly reduce the number of applications that you need to monitor, test and patch.

Not every patch update will go smoothly, so learn from these situations and document what went wrong so you can prevent the same mistake from happening again.

There may be instances when a device or application can't be updated with an important security patch. In these situations, it's vital to test alternative methods for mitigating the vulnerability and deploy them as soon as possible.

Patching plays a key role in ensuring that systems remain secure. Various audits, standards and regulations require reports on patch deployment. Fines and other post-breach costs can be extremely high if it's found that an organization's systems and customer data were exposed due to running unpatched software.

Michael Cobb, CISSP-ISSAP, is a renowned security author with more than 20 years of experience in the IT industry.