Open networks have helped many companies to significantly boost productivity, but they’ve also opened up the potential for problems that come with broader access. Safeguarding industrial networks from viruses, hackers and even employees unauthorized to get into protected areas continues to get more focus, but with a twist.
Security is now being examined with more of an eye toward controlling costs. As companies race to upgrade their protection technologies and processes, their actions are being examined more closely, with less chance that information technology managers can dictate in the name of security what others can or can’t do. For many, that’s a significant change from a time when security was considered something of a black art that couldn’t be quantified for cost justification.
Concern about security is still new for many companies. The transition from proprietary industrial networks to Ethernet and Transmission Control Protocol/Internet Protocol (TCP/IP) that cut costs and opened communications throughout the enterprise didn’t bring all the issues of Internet security to the production environment. But it did open the door for far more security issues than plant managers were used to.
As with physical security, giving more people access to a network raises a number of complex issues. “Companies need to think about why they need a door. By opening it up, what are they trying to do and how far should they open it?” says Jeremy Bryant, networking specialist at automation vendor Siemens Energy & Automation Inc., in Alpharetta, Ga.
Controlling who comes in through these portals and keeping track of what they do once they’re in is coming into the spotlight at many companies. Security in plants has often been run by the Information Technology (IT) staffs that handle front-office networks. That trend has increased as the use of Ethernet and TCP/IP protocols expanded from the business side to the production facility.
In today’s uncertain economy, these groups are coming under more scrutiny. Executives who have given IT staffs a fairly loose leash are now reining them in, demanding more accountability for expenditures.
“After the Y2K and dot-com era, people got away with a blank check mentality. But lately, companies are saying they need better cost justification. Unless management sees benefits, IT projects won’t get budgeted,” says Bryan Singer, vice president of professional services at Wurldtech Security Technologies Inc., a Vancouver, British Columbia, Canada-based provider of industrial cyber security solutions.
Shifting focus
Now that Ethernet is taking over the industrial networking space, the focus for industry consortia and other groups is beginning to shift to security. Compatibility and other issues are closer to being resolved, freeing time to address other facets of networking. Most observers feel that there’s high potential that extortionists and other criminal types may take aim at industry. Making sure that industrial control networks aren’t shut down is becoming a major effort for many industry groups.
“We’re taking a look at the spectrum of cyber security tools. We’re also examining the language to put into requests for proposal so that companies can get some commonality in their proposals,” says Michael Torppey, technical manager for the Process Control Systems Forum. Torppey is senior principal at Noblis Inc., the Falls Church, Va., company that manages the PCFS.
There are myriad tools and a number of different processes that provide varying degrees of success in different environments. But there aren’t many ways for companies to know whether their investments in either time or effort are paying off.
Some techniques and tools overlap, and processes that are very effective in one organization may have little impact in another. Picking the right elements is a key factor when security initiatives begin. Once managers have determined their strategies and made their moves, it’s often difficult to determine how to invest in further upgrades and improvements. “Measuring the success of security is a very big challenge,” Singer says.
Though it’s not easy, it’s a necessary part of efficient management. When money is tight, companies need solid rationales for investing time and money in security. That begins with determining which aspects of the business warrant extra attention. “Companies need to perform a risk analysis, taking a look at revenue-producing activities. They need to determine how much impact each type of attack could have on their company,” Torppey says.
Many companies haven’t really looked at their production facilities with the same focus on risk aversion that’s been given to front-office operations. “Disaster recovery is part of any business continuity plan, but nothing that mirrors that exists in many manufacturing sites,” says Brad Hegrat, senior network security engineer at vendor Rockwell Automation Inc., in Milwaukee..
There are a few ways to determine how well new security tools are working. One is to keep close track of problems that arise from unwarranted network activity. When issues are logged in regularly, it becomes easier to tell what impact upgrades have made.
“If you’re able to show activity through logging, you can apply risk avoidance analysis techniques. When you improve logging, you can say that you’ve had fewer incidents after a firewall was installed, for example,” Singer says.
Another approach is to simply keep track of how well equipment is being used. “If you can improve the uptime of a line, reducing planned or unplanned downtime, you can measure that. People usually know the value of making a machine 0.1 percent more efficient,” Singer says.
Measuring overall equipment effectiveness (OEE) can help companies understand the benefits of security. Networks are often overlooked when OEE is analyzed, but they play a key role in keeping equipment running. Without network controls and feedback, equipment and inventory problems will eventually occur.
“Companies that use OEE can watch lines go up or down, seeing the changes after they put security steps into play. If you find that your network was saturated and went down, say, 10 times, and you can show that now you don’t have as many network problems, who cares whether the problems were caused by hackers or staff,” Singer observes.
Managers with solid security programs can also benefit by using honeypots to detect unwarranted activity. These easy-to-install systems serve no purpose other than to spot suspicious or problematic network activity (see sidebar.) Honeypots are named after either Winnie the Pooh’s habit of getting stuck in honey jars, or a cold war technique for trapping spies by using sex.
When companies are establishing or revamping their networks, protection schemes need to take many things into account. Some aspects of manufacturing security are specific to the factory floor, so special tools are needed. For other aspects of security, many IT techniques for the front office can be carried over.
Some of the suggested steps are fairly universal. “We all recommend intrusion detection on process control systems. If you see events that shouldn’t occur, you can stop them before they progress,” says Kevin Staggs, global security architect at controls vendor Honeywell Process Solutions, in Phoenix.
Though most every facility can benefit from running intrusion detection programs, installing them isn’t as simple as running software configured for the office environment. The communications that occur on systems used for e-mail are dramatically different than the message patterns found in industrial environments. Managers and engineers familiar with control systems must work together with their IT staffs to make sure software is set up correctly.
“Setting up a true intrusion detection system on a process control network is a lot of work. The signatures and protocols used on process control networks are usually not understood by IT departments,” Staggs says.
One way to reduce the potential for problems is to isolate the automation network from the front office. That can be accomplished by using an Ethernet-based intranet in the factory. This intranet can be tied to front-office networks so inventory and order data can be moved easily.
Access can be limited to only a few people who have the ability to move between the two segments of the enterprise. “In the industrial world, you need to think about ways to make things simple. You can just let certain IP addresses come into the intranet. Then you don’t even need to worry about passwords,” says Rockwell’s Hegrat.
Another benefit of this approach is that it reduces the levels of security. “If the automation system works at the cell level, you generally want to open access through an intranet. That network is already protected, so the big concern is access. You may only need a firewall,” Bryant says.
Some aspects of these security schemes should account for the same type of protection regardless of whether equipment problems are caused by network intrusions or by general equipment failures. “The first thing to do is to add back-ups, and also to establish plans to restore PLC (programmable logic controller) codes and HMI (human-machine interface) systems,” Hegrat says.
These back-ups can play a critical role as factories evolve and system settings change. Control software and other applications packages undergo many changes over the course of time as operators tweak parameters to improve efficiency. If companies don’t keep track of these changes, the results can be as disastrous as when front-office files are hit by viruses or other attacks.
“Often, there are small changes and applications that migrate throughout the plant. You can find that if you lost a PLC, replacing it is not as easy as starting from scratch. It’s more complex than, say, replacing a Web server,” Hegrat says.
One bright spot for plant managers is that the availability of tools that meet industrial needs is growing. “Last year, we saw a number of improvements in technical areas and introduced new concepts in the crossover space between control systems and IT,” Torppey says.
People and processes
It’s rare that an in-depth discussion of network security doesn’t get around to human factors. If employees don’t take security seriously, doing things like picking easy-to-guess passwords or taping them to screens, even the best strategy and equipment won’t do much good.
Just as business and manufacturing processes separate market leaders from the rest of the pack, the processes for implementing network security will have a key role in reducing the number of problems caused by network disruptions.
“Most problems are not necessarily technically related. Many are process- and people-oriented, so buying technology won’t have any impact. Addressing that comes down to organizational structure and training,” Hegrat says. He notes that just because an HMI has the ability to browse the Web doesn’t mean that operators should use that function.
Limiting access to equipment within the plant is another way to reduce the potential for error. Most studies say that a fair number of network problems are caused by employees, often those who feel they’re doing something for the benefit of the company. An employee who thinks he can improve quality by altering a couple of machines can cause serious problems. “In control systems, security can include anything that causes a disruption in the process, whether it’s internal or external, malicious or not,” Hegrat says.
Keeping well-meaning employees, as well as those with malicious intent, away from most systems in the plant is an important aspect of protection schemes. Most employees only need to talk to one or two machines in the facility. “Some people need to talk to certain equipment. It’s easy to limit them to only that equipment,” Hegrat says. One way to accomplish that is to limit password access to specific devices, he adds.
Though using Ethernet throughout the entire enterprise makes it possible for front-office personnel to inadvertently impact operations on the production line, most observers say that’s not much of an issue in most facilities. In reality, few office people will be able to do much damage in the factory. “People in the front office usually don’t have the software to access a programmable logic controller. If they do, the firewall shouldn’t have them in the IP addresses allowed, so they shouldn’t be able to get in,” Hegrat says.
“Corporations don’t treat someone who sticks their password onto the computer with a sticky note the same as someone who hangs their access card next to the front door, but it’s the same thing,” says Shaye Shayegani, senior field applications engineer at Lantronix Inc., a device networking solutions provider based in Irvine, Calif.
Companies can also use these passwords to establish a tiered structure for human access. Many companies give most employees similar access levels, but that’s often not the best approach. “Passwords are about authorization for what you can and cannot do, like applying domain policies that determine things like who can upload or download programs,” says Bill Lewins, a requirements analyst at Rockwell Automation in Milwaukee.
Alternatively, companies can restrict personnel to specific systems or limit the ways they can communicate. “We limit people to certain computers and protocols,” says James Davis, senior application engineer at Opto 22. He notes that as a way to provide additional security, the server automatically records everyone who accesses control programs.
While controlling people’s access is a big aspect of security, many companies do their best to automate communications so that people aren’t involved unless their decision-making input is required. “Machine-to-machine communications are easier, since the machines always follow procedure. They make sure they only send and accept messages from authorized equipment,” adds Shayegani.
