Downtime is a pretty big issue when you’re running an offshore platform that gathers and processes natural gas and oil. When system integrators at Cimation (www.cimation.com) set up the network on a high volume platform, one of their primary goals was to minimize downtime caused by accidental or malicious forces.
The platform is among the largest of those that sit on the U.S. continental shelf, so the potential for malicious attacks is higher than for smaller platforms. These facilities collect crude oil and natural gas from many nearby well heads and do some early processing, so they have far more programmable logic controllers (PLCs), remote terminal units (RTUs) and other control systems than platforms that simply drill for oil.
“They’re not just piping oil on these platforms. They’re also doing rudimentary treatment, such as separating liquids from gases,” says J-D Bamford, control room management and SCADA security engineer at Cimation, a system integrator focused on the process automation, industrial IT and enterprise data solutions for the energy sector. “That’s where a lot of the PLCs and RTUs are used.”
Unlike drilling platforms, which are moved once the well is set up, production platforms may collect and process oil and gas for years. The complexity of the systems, coupled with the long lifetime, prompted the owners to put a premium on security.
They picked Cimation, an integrator owned by Houston-based Audubon Co. (www.audubon-engineering.com), to set up physical and logical security that protects the operations from assault by extortionists or hackers, among other threats. Given the high cost of downtime, Cimation’s goal was to minimize shutdowns associated with networks and electronics.
“These large facilities cost millions to design and build. The cost of installing security equipment is a drop in the bucket, especially when you compare it to the loss of production for a day,” Bamford says. “Downtime easily costs hundreds of thousands of dollars for a day.”
The decision to invest in security highlights a change that’s occurred over the past few years. Until recently, industrial managers have found it difficult to get funding for security technology that will prevent unplanned downtime. Many in the industry say that the change occurred after the Stuxnet virus got publicity around the globe, proving that industrial attacks were not an obscure, unlikely possibility.
“Stuxnet was the poster child proving to people that cyber attacks can cause damage,” says Dan Schaffer, industrial security specialist at Phoenix Contact (www.phoenixcontact.com). “Before it happened, people only paid lip service to security. Now they actually do something.”
Though awareness has increased, change in the industrial field comes slowly. There’s still a lot of need for more education that will help companies start improving their security programs. That prompted PI North America (www.us-profibus.com), the trade association for Profibus and Profinet, to recently expand the security portion of its one-day Profinet seminar. It’s also updating its security guideline.
“The sad thing is a lot of people haven’t made it to step 1 yet. There’s still a major need for many companies to do something about security,” says Carl Henning, deputy director of PI North America. “One problem for many people is that when they’re writing capital expenditure requests, it’s tough to quantify security.”
Cimation didn’t have any trouble convincing its customer that it would be cost effective to install security based upon the Tofino Security tools provided by Belden. Cimation stressed that security isn’t just designed to prevent malicious attacks.
“From a reliability standpoint, PLCs and RTUs are not necessarily designed to talk to a lot of devices,” Bamford says. “People are always concerned about folks with malicious intent, but there are also issues with traffic when there are too many messages on the network. You want to protect devices from things they should be able to handle but can’t.”
There are many cases where excessive traffic has caused major problems. For example, the Browns Ferry Nuclear Plant in Alabama had to shut down a few years ago after one faulty node started spewing messages, overloading the network and causing problems that threatened the plant’s stability, says Eric Byres, chief technology officer (CTO) for Tofino Security at Belden Inc. (www.belden.com)
Focusing on this dual need for protection against attacks and improving reliability is part of the reason that there’s been a shift in attitudes about security. Stuxnet highlighted the dangers of one threat, while the additional improvements in reliability help companies justify the expense of buying security hardware and software.
“A general comment on return on investment is that you need to look at what happens if a virus gets through,” says Shawn Gold, global solutions leader, industrial IT solutions at Honeywell Process Solutions (www.honeywell.com). “People also have to remember that if you do security right, it’s not just protecting against attacks; it can also improve reliability.”
Defense in depth
Protecting networks and physical assets isn’t just a matter of installing one component. Whenever strategists in any field construct defenses, they plan for a failure or two. Networking security is no different. Experts all suggest that companies employ a few different technologies so weaknesses with one tool will be covered by strengths of a complementary technology.
“You need defense in depth. You can’t just have a front line,” Henning says. “It’s only a starting point to create a demilitarized zone between the office and factory networks.”
Connectivity between the front office and industrial operations is one of the factors that helped move Ethernet onto the plant floor. Firewalls are typically used to isolate mission critical industrial networks from the commercial side, where PCs continuously receive e-mails and go to Websites that potentially have malware. These firewalls can be set to limit traffic that passes from the industrial network to the office system that connects to the Internet.
“Managers can set rules, like blocking equipment from any Web traffic so people can’t go on line,” Schaffer says. “You can also configure the firewall to let this type of traffic go through but not that type of traffic.”
Firewalls for 40 machine networks
Most companies will want to install a few firewalls. That helps ensure that any viruses or attacks that get through the first firewall are stopped. It also prevents the spread of problems that begin on the plant floor, whether it’s a PLC that’s spewing messages or a virus from a USB stick.
“You can put a firewall in front of an end point like a PLC or an HMI [human-machine interface], or you can put it at the point where the IT [information technology] network meets the industrial network,” Schaffer says. “In the latter configuration, it can protect hundreds of devices that sit behind it. Adding firewalls in other areas gives you defense in depth. In the auto industry, they put security systems around each cell, like a welding or painting station.”
That’s the approach taken by ZF Sachs (www.zfsachs.de), a German manufacturer of drive and chassis components for the automotive industry. Its Schweinfurt plant is divided into 40 machine networks. Individual Phoenix Contact mGuard firewalls protect each of these Profinet networks.
“To ensure that the decentralized architecture with 40 individual machine networks did not lead to greater configuration and operative effort, we first developed a basic set of common firewall rules for all sub networks as an overriding control. The implementation was relatively simple,” says Asmund Hey, head of automation technology for ZF Sachs technical services.
During commissioning, the master parameters were applied to the subnet upon start-up. This covered most of the plant’s requirements, so additional rules only had to be added for special cases. Hey noted that taking the time to set up a well-structured architecture and tweaking it during setup eliminates many headaches later on.
Defense in depth also requires running a range of different software tools to help reduce the likelihood that networking issues will cause unplanned downtime. Unlike hardware, which can remain in place untouched for years, this security software needs to be upgraded as programmers adapt to new threats.
“You need to run multiple tools, like antivirus software and whitelisting,” Honeywell’s Gold says. “You should also install patches for the intrusion detection system.”
Keep it simple
Establishing the overall protection scheme will require a fair amount of thought and effort, but implementing the security system can’t be overly complex. Most equipment operators don’t want to spend time worrying about intrusion detection and other factors that don’t help them meet their goals for the day.
The firewalls that form the basis of many security schemes are a good example. Over the past few years, developers have made a concerted effort to make these devices easy to install and set up. Once they’re in, users can typically run them without much effort or assistance.
“You can install a self-contained appliance, one that’s hardened from a physical and logical standpoint, on the network,” Schaffer says. “You don’t need an IT guru to set the system up and maintain it.”
While suppliers and users both want to avoid the need for IT personnel, most security specialists say that it’s often important for industrial managers and front office IT staffs to work closely together. IT staffers usually keep up to date on the many technologies associated with Internet security, while industrial personnel know their way around the diverse plant floor environment.
These industrial environments have many technologies that will be quite foreign to IT teams. Front office equipment typically runs TCP/IP protocols, while industrial networks run Modbus, Profinet, DeviceNet and others. Facilities that require real time communications will also run additional protocols that provide higher performance and determinism.
“You’re not dealing with one protocol. You’re dealing with stacks of protocols, and every vendor does it a bit differently,” Bamford says. “Ethernet is a base for handling all these technologies.”
Though all these protocols run on the same cable as TCP/IP communications, systems still need to be configured to ensure that the various industrial protocols communicate effectively. These processes are fairly straightforward, requiring little setup time.
Industrial network managers must also make it easy to add and relocate equipment. Production lines continuously change, and new equipment is often added. Too often, third parties may install equipment and not let the manager of a large facility know what they have installed unless there’s some sort of problem. But not being able to install it at all is also a problem.
Cimation’s Bamford, who handles system integration for many companies, notes that systems must be configured to make it simple to add new hardware.
“In a lot of facilities, third parties install equipment without telling anyone,” he says. “The firewall may not let it start up. Tofino lets us set rules so these products can start up.”
While firewalls and other security components must be simple, they must also be fast. Security systems typically scan huge volumes of data over the course of a day, and it’s not unusual for some of the control data to be very time sensitive. A security scan can’t prevent a signal that adjusts the temperature or turns off a valve to arrive even a few milliseconds late.
“It’s important to handle security without sacrificing performance, you can’t have the scans slowing down communications. That’s why we use dedicated hardware,” Schaffer says.
>> Click here to read Security Appliance Protects against USB-Stick-Delivered Malware
