Lanny Gibson doesn’t want just anybody to be able to wander freely around his control network. If someone were to disturb the devices on it—either accidentally or maliciously—the results could be devastating for the refinery that Total Petrochemicals and Refining USA operates in Port Arthur, Texas. Nonetheless, he wants automation vendors to have some level of remote access so they can give his small crew of four control engineers a helping hand.
As the process control group leader at Port Arthur, Gibson could see that his crew was simply too small to implement the latest corporate cybersecurity directives by themselves. For this reason, he joined the growing ranks of engineers taking advantage of the remote maintenance and production-monitoring services that are gaining a measure of popularity today.
Gibson elected to use Managed Industrial Cyber Security Services from Honeywell Process Solutions, which control all access originating from outside the plant’s control environment, and manage the firewall, patches and other updates. Not only do the experts have more experience from monitoring and maintaining firewalls every day, but they also are specialists in industrial networks.
Key to the delivery of Honeywell’s services are two nodes: a service node inside the firewall surrounding Total’s process control environment, and a relay node that serves as a proxy server. “To comply with the Purdue model, we put the service node at level 3 and the relay node at level 3.5,” says Mark Littlejohn, Honeywell’s global manager of industrial managed security. All communications with anyone outside the plant’s control environment occur through this single, outbound port leading through the firewall to a specific IP address at Honeywell’s highly segmented, firewalled Security Service Center.
The service node continually checks in with the service center, asking whether there are any updates, service requests or other tasks requiring action. If there is a virus-definition update, for example, the node grabs it and pushes it down to the appropriate devices, where they are immediately installed to keep this mode of protection as current as possible. Patches, on the other hand, are stored at the device level until a maintenance window opens. “You never know how a patch will affect a crucial piece of equipment,” Gibson explains. So his team usually does the installation during the next scheduled downtime in case any difficulties arise.
Knock before entering
Remote access for routine maintenance and troubleshooting occurs through the same node. All remote technicians enter Total’s control environment through Honeywell’s service center. This includes the petrochemical company’s own process control engineers and senior operators who are off site. It also includes competing automation vendors, including one that recently resolved a problem on a tank farm by accessing the controllers through the service center.
These technicians—whether from Total, Honeywell’s service team, or a third party—must log in using two-factor authentication on an account approved by Gibson and set up in advance to have either read-only or modify access to specific devices. Before gaining access, the technicians must submit an electronic request describing who they are, what devices they want to access, what they are going to be doing, and when they will be doing it. The service node picks up these requests and presents them to Gibson’s team for approval. “They are knocking on the door, and they don’t get in if we don’t open it,” he says.
Even if Total’s control engineers do let them in, they can just as easily throw them out by terminating access. A monitoring feature helps the engineers make such decisions by allowing them to watch every mouse click and keystroke that occurs on the network. The feature also records the activity to provide an audit trail and replay the sequence of events.
Besides monitoring remote access coming across the firewall, the security services also monitor the activity of all devices on the network. Perhaps the most reassuring bit of intelligence for Gibson is the reporting on USB ports—his biggest security fear. He now receives an email whenever someone plugs in a USB stick.
Demand for more access
As more users are discovering, cybersecurity today requires defending industrial networks on many different fronts. “In the past, remote access was essentially used for troubleshooting,” notes Neil Peterson, DeltaV product marketing manager at Emerson Process Management. “Over time, remote access to control systems was demanded for other applications too, such as mobile operators, management access to production data, remote collaboration, and unmanned operations.”
One reason manufacturers are using remote access more is that they’re asking their experienced people to mentor less experienced employees, who are learning to cope with complex control applications elsewhere in the world, Peterson adds. Some companies are even using retired employees as consultants.
Plus, the increasing popularity of concepts like the Internet of Things (IoT) has only broadened the demand for access to industrial networks. “It is part of a continuing trend towards implementing open standards-based Internet technology in automation and process control applications,” reports Matt Newton, technical marketing director at Opto 22. These open standards certainly streamline access, but they also make things easier for hackers.
The good news is that many general cybersecurity measures still apply in remote access applications, according to Allen Tubbs, product manager for electric drives and controls at Bosch Rexroth. “Closing unused ports, controlling the IP addresses of devices allowed to connect, implementing user accounts and passwords that expire with disuse—these are not new,” he says.
Even so, industry has learned some important lessons along the way. The Stuxnet attack, for example, taught industry not to house the control software and user interface on the same device. “It might be more costly to keep them separate, but is probably worth it to protect against similar attacks,” Tubbs says.
He also thinks that, for data security, equipment designers should establish the same kinds of habits that have become routine for designing operator safety into machinery. A basic best practice is a layered design that segments the network and uses firewalls between the segments, as well as around the network’s outer perimeter. “Secure remote connections must come with a solid defense-in-depth network architecture to prevent access in one zone from migrating to others,” says Mike Werning, field application engineer at Moxa Americas.
Newton urges engineers to restrict access to human-machine interfaces (HMIs) and other assets with physical barriers and locked keyboards. He also suggests specifying devices that have Internet security features built into them. “Devices that will be connected to any type of network must have SSL/TLS [secure sockets layer/transport layer security] encryption built in and must implement some form of authentication,” he says.
Other tools can go beyond simple authentication to vet devices before granting them access. Peterson, for example, points to how Emerson’s DeltaV distributed control system (DCS) uses Cisco’s Identity Services Engine (ISE). “It works with the domain service active directory to ensure the remote user meets or exceeds the site’s security posture before access is granted,” he says. It checks that the computer’s antivirus definitions are up to date and that other security configurations meet the user’s standards.
Isolationism: A safe policy
Another best practice is to keep the business network isolated from the industrial network. “This can be tricky if the business network needs to access information on the automation network and vice versa,” Newton says. To help, Opto 22 designed dual Ethernet interfaces built into each of its SNAP PAC programmable automation controllers. “Each interface is logically isolated from the other on different subnets. The controller can communicate across both the business network and the automation network, but no traffic can be routed across the interfaces.”
In networks using fieldbus, another way to keep control networks separate is to use servers as protocol converters or gateways. “Fieldbus protocols were not designed with security embedded in them,” Werning explains. “Protocols such as OPC UA have a client-server architecture secured by encryption.” Others, such as message queuing telemetry transport (MQTT) and TLS protocols, provide security for wireless networks.
The most common method of establishing secure communications with fieldbus is through a virtual private network (VPN). “This builds a secure tunnel through public and private networks, which prevents eavesdropping on the conversation or allowing other parties to gain access to the network,” Werning says.
Not all automation vendors agree that VPNs offer industrial networks adequate protection, however. “With a VPN, the connecting computer becomes a member of the computer system that it’s connecting to,” explains Patrik Boo, ServicePort product manager at ABB Process Automation Service. “An outside person is connecting to your network and gaining access to everything without any real checks and balances on what that person can do.” Consequently, he thinks that VPNs can often expose industrial networks to unnecessary risk.
Another problem with VPNs is that they are general-purpose IT solutions. “In the IT world, the main focus is to protect the data,” Boo explains. “You want to deny access to credit card numbers, for example.” For industrial networks, on the other hand, the security concerns tend to revolve more around availability and data integrity.
To mitigate risk, users should establish remote connections to industrial networks with solutions designed for industrial settings, Boo says. Besides the one used by Gibson at Total, another example is ABB’s ServicePort, which connects to ABB’s services through its Remote Access Platform. Boo describes ServicePort as a computer with its own database. “Besides having the normal login authentications and other security measures, it also resides in the DMZ [demilitarized zone] on the network,” he says. Remote technicians look at production data and key parameters in ServicePort, not on the control system itself.
The cloud is another means for exchanging data outside the control system. “This is largely through the introduction of software agents that use the latest encryption methods to get data outside of the control network to a cloud server,” says Alicia Bowers, senior product marketing manager for automation software at GE Digital. While these applications secure communications with the content servers, hardware fiber isolation solutions keep the data communications pointed outbound.
Back to basics
As helpful as these technologies and practices can be in bolstering cybersecurity, Boo stresses the need for the basics. For him, that means performing tedious and mundane tasks like instituting and enforcing policies and procedures for who is allowed to do what and when. It requires developing an asset inventory so you know what you have and how things are connected. You also need to know what applications are running and disable unused ports. “If you just buy a solution without doing the hard things first, you’re just wasting money,” he says.
To ensure that they have a solid foundation, more plants are seeking help from automation vendors. For example, the Boliden Group, a Stockholm-based mining and metals company, has taken advantage of a growing number of services introduced by automation vendors for this purpose. Management contracted ABB to perform a non-invasive benchmarking service that the vendor calls a Cyber Security Fingerprint. The goal was to augment the existing security by customizing it for the ABB 800xA control system at one of Boliden’s plants in Sweden.
To implement the service, ABB’s consulting engineers spent a few days on site collecting the necessary information. Not only did they conduct structured interviews with key plant personnel, but they also deployed a high-speed software tool called Security Logger. The software collected system settings and other information from more than 100 points on the plant’s network.
Upon returning to their offices, the engineers took a few more days to analyze the data and compare it to industry best practices and standards, such as ISA/IEC-62443 and those promulgated by the National Institute of Standards and Technology. Using another tool called Security Analyzer, they calculated key performance indicators to quantify the relative strengths and weaknesses of the control system’s cybersecurity. After completing their analysis, the engineers compiled a report suggesting ways to strengthen already sound defenses.
