At Praxair Technology Inc., software security patch management has become a collaborative effort. The manufacturer of industrial, process and specialty gasses, based in Danbury, Conn., works closely with its control vendor, Austin, Texas-based Emerson Process Management, to determine what patches need to be applied now, which ones can wait and which can be ignored. “We get an auto-mailer from Emerson that has a spreadsheet attached that includes data sheets,” explains Jason Solomon, Praxair associate control systems engineer. “They have links on the data sheets that tell you what version you’re running and what patches they’ve approved.”
The spreadsheet sent by Emerson has the a link to the patch supplied by Microsoft Corp. at the top, and indicates whether Emerson has tested the patch, and whether the patch is OK or not OK. “With most of the Microsoft patches, you can just install them and keep going,” says Solomon. “Seldom does a Microsoft patch require a shutdown.” Emerson uses the same style auto-mailer for control system patches as well. Because Praxair has a continuous system, Emerson gets involved in helping with control system patches. “We can download most of the control system patches locally, but a local Emerson rep will come out to back us up,” says Solomon.
Control system vendors such as Emerson are working closely with plant managers to develop collaborative patch management strategies. System security has become more of an issue now that plant systems are connected to the Internet. Plant managers are faced with a flood of patches coming from Microsoft, based in Redmond, Wash., as well as from application software vendors. The Dow Chemical Co., based in Midland, Mich., applied more than 69,000 patches on more than 2,200 Windows-based servers in the past couple of years, for example.
This is territory plant managers cannot easily navigate on their own, and they have been forced to develop patch management strategies.
Who’s responsible?
As security becomes a more important issue for plant mangers, they are looking more to their control software vendors for help. Because they have no idea whether an individual patch from Microsoft will interfere with the control system, plant managers are increasingly turning to their control vendors for guidance in determining whether to apply a patch. “In the past, manufacturers did it themselves,” says Mike Bush, program manager, Rockwell Automation Inc., the Milwaukee-based automation controls suppler. “As part of deploying Microsoft patches, there is some level of testing before applying it to the control system.”
This summer, Rockwell is launching a lab committed to testing Microsoft patches to find out how they operate in conjunction with control systems. “We’ve built a patch qualification test lab that will run 24/7 tests on the latest patches,” notes Bush. “When we see notification about a Microsoft critical patch—usually on a Tuesday—we’ll start testing immediately and notify customers within seven days.”
Controls vendors across the board are taking more responsibility in ranking the importance of Microsoft patches and testing to see if a particular patch affects the control system. “This all came up a couple of years ago. A big part of the issue is whether Microsoft fixes will break the control software,” says Bob Mick, vice president of emerging technologies, ARC Advisory Group Inc., in Dedham, Mass. “Independent software vendors (ISV) have many versions of products running, so it becomes a complex problem. So the ISVs are doing a cursory test to see if the patch interferes with anything.”
Mick expects that patch management will become an increasingly important issue in what plants expect from their control vendors. “Patch management will become a competitive factor in the future,” Mick predicts. “It’s not part of a service agreement now, but companies are becoming aware that it should be part of the service agreement instead of the annual upgrade.”
In some cases, plant managers are also turning to their control vendors to centrally manage updates and upgrades. “People are looking at how to manage patches and upgrades centrally. You can’t do it individually when you have 500 computers,” says Rashesh Mody, chief technology officer at Wonderware, an Invensys company based in Lake Forest, Calif., that is an automation software supplier. “They also want to know how to do it without shutting down the plant.”
Ranking patches
Some patches are more important than others. A critical aspect of patch management is the ability to determine which patches are “show stoppers,” patches that fix problems in the control system itself. These can correct problems that might cause serious calamities. Some patches—especially those Microsoft patches related to e-mail—don’t need to be deployed at all. Plant managers are turning to their control system vendors to help determine how patches rank in importance.
Emerson sends out detailed information to its customers to help them determine whether a patch should be applied. “The challenge is to assess whether to do something about this now, wait, or consider not applying the patch at all,” says Kim Van Camp, Guardian support program manager for Emerson, in Pittsburgh. “We give that information to our customers in an executive summary that is matched to the content of the customer’s system, so the burden of determining the importance and safety of the patch is not on them.”
Occasionally, a patch for the control system corrects a potential serious problem. “Every once in a while, a patch becomes mandatory,” says Rockwell’s Bush “You don’t get to choose whether to apply a patch if life and limb are at stake. That’s pretty rare, though.”
Determining the importance of a patch becomes a risk assessment challenge—the plant risks production loss if the patch requires a shutdown, while the lack of a patch might cause system problems affecting production. “You have to decide when to apply patches. Some say schedule some down time, some say wait for a down-time opportunity, and others say there is urgency and we have to do it before scheduled down time,” observers ARC’s Mick. “We say you have to prioritize the patches according to the risk you have. That brings in risk assessment and looking at the impact on costs if something goes wrong.” This is not a territory plant managers can easily assess without close collaboration with their system vendor.
Another critical aspect of patch management is making sure the patch doesn’t affect the control system. You don’t want to correct a minor problem with Windows only to find out the Microsoft patch causes problems in the control system’s ability to communicate with programmable logic controllers (PLCs). This is an area where the plant managers are blind. So they turn to their control system vendors for help in determining whether a Microsoft patch is safe to use in conjunction with the control system.
Patch testing services
Most control system vendors have assigned teams dedicated to testing patches before those patches are given an OK for their customers to deploy. “At Honeywell, we evaluate every security update from Microsoft. We either set it up for our end-users or take an exception,” says Kevin Stagg, engineering fellow for cyber security at Honeywell Process Solutions, in Phoenix. If Stagg takes “an exception,” that means end-users can ignore the patch. “Most of the time when we take an exception, it’s because the patch is for an e-mail application, and you shouldn’t be running e-mail on your control system,” says Stagg.
Vendors are now competing on how quickly they can test a patch and send out word to customers on the patch’s importance, and its safety in regard to the control system. “We make a commitment to our customers that we will make an assessment within 24 hours of a Microsoft security update,” says Stagg. “We publicly state that we will test it within seven days, but we’ve been consistent in testing in one day.
When it comes to patch management, the key to success is collaboration. Plant managers don’t have the time or expertise to test each patch to make sure it doesn’t crash the control system. Plant managers also can’t easily determine whether a patch is important enough to put the brakes on plant operations. The control vendor needs to work closely with the plant to determine the safety of a patch, and also determine whether deployment can wait until scheduled down time.
For more information, search keywords “patch management” at www.automationworld.com.
