Six hours into an eight-hour process at a pharmaceutical plant, production hit a fatal snag and shut down, essentially flushing $100,000 down the drain. The problem: A software update came at just the wrong time. What would have been a routine update in the IT world proved a costly mistake in the world of operational technology (OT).
Welcome to the brave new world, where IT and OT intersect in ways that plant operators couldn’t have imagined just a few years ago. That intersection is allowing companies to streamline processes and maintenance, and connect vendors and suppliers with data to save time and money. Except when it doesn’t go as planned.
In the case of the pharmaceutical company, the problem was a window that popped up on an interface to ask the operator if he would like to update the software. Choosing to update resulted in a reboot of the batch server that was running a medicine-making process—and the loss of vital genealogy required by the U.S. Food and Drug Administration (FDA).
It was, as Gregory Wilcox, global technology and business development manager for Rockwell Automation, later put it, “a really bad day.” The batch was ruined and the pharmaceutical company could only call in Wilcox and other experts to recommend policies, procedures, technology and training to help make sure that kind of mistake was never repeated.
Despite the potential pitfalls, the benefits of connected automation—machines and processes that share information with each other and the businesses that operate them as well as with customers and suppliers—are legion, and growing all the time.
“Connecting plant-floor assets with the enterprise, and connecting manufacturers and suppliers can offer tremendous value,” says Scot Wlodarczak, a manager for industry marketing at Cisco and a spokesperson for Industrial IP Advantage, a trade group dedicated to education about industrial information architectures. “In fact, it’s estimated that four out of 10 companies will be disrupted in their market position by companies fully embracing connected factory solutions.”
Key to realizing those benefits is mitigating the potential risks—which, fortunately, can be done with proper planning and use of already-established best practices. It starts with getting off on the right foot.
Connecting the dots
Laying the groundwork for connected automation starts with an evaluation of what an automated process or factory already has to work with, and where managers want to go with it, says Tony Shakib, vice president of Cisco’s IoE Vertical Solutions Engineering organization. The starting point is what Shakib calls level one—just getting the component pieces, including machines, connected and sharing data.
An important consideration here is how tightly to link IT and OT systems, says Ryan Lepp, director of business development for industrial automation and the Internet of Things (IoT) for Panduit. “Is your end goal a completely converged network, where IT and OT coexist?” he asks his customers. His recommendation is for as unified a network as possible to reduce costs.
Once machines and processes are sharing data, Shakib says, companies deploying connected automation can proceed to level two—making use of all that data. Predictive maintenance is one benefit to be achieved at level two. “By having a constant connection monitoring the health of these devices, quite often you can predict when something’s going to go down months ahead of time,” he explains. Reducing or eliminating downtime is an obvious benefit, saving millions of dollars for manufacturers.
Also at level two, data can flow the other way, back to the machines. “Rather than having to spend hours changing a machine over manually to work with a different product, the machine has devices onboard that automatically can get changed almost just with a recipe,” says Robert Miller, senior manager of strategic collaborations and partnerships at Mitsubishi Electric Automation. The recipe (information about how to build a new product) instructs servos and other parts of each machine to reconfigure themselves to handle products of different sizes, shapes and weights.
Level three connects a factory with outside suppliers and customers, potentially extending the benefits of connectivity to the entire supply chain. But greater connectivity presents greater security risks. “End users need to adapt and embrace these new business models to remain competitive,” Wlodarczak says. “However, turning traditionally siloed industrial networks into borderless industrial Ethernet networks shared with suppliers can open up new attack vectors.”
Fortunately, careful planning and best practices can prevent a bad day.
Securing the network
Proper security practices operate on multiple levels, Wilcox says. “We always recommend to customers that they use a holistic defense-in-depth approach,” he says, which should address security at the physical, electronic and administrative levels.
Security at the physical layer can be as simple as restricting physical access to certain areas of a plant to only those who need to be there. That’s an approach all too often overlooked, Wilcox says. “Unfortunately, sometimes our customers have what’s commonly referred to as an M&M approach to security,” he says. “It’s hard candy outside and it’s soft and gooey inside. Once you get past the perimeter, whether at the receptionist or even a guard, at times there are no procedures to actually track visitors.” Access control provided by locked doors opened by ID badges can go a long way toward mitigating this potential security risk.
Physical security can also extend to physically preventing machines and controls from connecting to the wrong networks or devices. This can be ensured with cables that will not connect to the wrong places. Panduit, for example, makes cables and connectors that foster this level of physical security. “We have an entire line that can be used to configure and construct the physical security of a network,” Lepp says.
At the electronic level, says Miller, the right kind of network can ensure that only known devices are able to share data, Miller says, noting that CC-Link IE is an Ethernet-based network that provides this level of security. “CC-Link IE is inherently deterministic and inherently secure because of the technology and the communication that it uses,” explains Miller, who serves as director for the Americas for the CC-Link Partner Association. “Unless the network controller knows about a certain device, that new device will not be able to communicate across that network.” In other words, he explains, “You couldn’t just walk up to a CC-Link IE network, plug into it with a laptop and hack into the system.”
Finally, administrative access controls should restrict users to only parts of a network or to software that they have been authorized to use. Packages like FactoryTalk Security from Rockwell Automation can help system administrators establish the appropriate levels of access to software and hardware based on who is logging in to the system from which locations.
Stabilizing the network
As the example with the rebooting batch server at the pharmaceutical plant exemplifies, greater connectivity also can present challenges to maintaining uptime. Uptime is often less critical in the purely IT world than it is in the operational world, and bringing operational-level uptime to a converged network is the name of the game for many plant operators.
Step one in ensuring network uptime is simplifying wherever possible, Lepp says. That requires planning. “If there’s no strategy or plan,” he explains, “you develop this nest of communications, where you may have critical points of failure.”
Planning should include such factors as ensuring that switches have enough capacity to handle the volume of data passing through them. “That is something that’s going to become more and more of an issue,” Miller says. “As more devices become available to be put on a network and to monitor through the network, there’s going to be more and more data.” And more data increases the risk of network congestion, which can bring a process to a grinding halt.
Lepp cites the case of a food and beverage plant whose network teetered on the brink of collapse at any given moment because of too much network traffic. “If you added anything, it would crash the network,” he recalls. “If you took that extra device out, the network could recover and you could start the machine again.” Lepp and his team solved the problem by carefully assessing the network and then redesigning it to handle more data.
Such reconfigurations could include what’s known as zone architecture, Lepp says. “What a zone architecture does, is it pulls the switches out of a control panel and puts them into a rafter or higher level, then you disburse the backbone network off into individual zones,” he explains. That way, even if a control panel does go offline for any reason, the distributed switches keep the network and the systems that depend on them up and running.
In addition to reducing data loads, zone architecture promotes redundancy, another key to enhancing uptime. This can extend to cables as well as to switches. Lepp and his team make sure that there are backup data lines connecting switches, controls and machines. Equally important is that the lines don’t all follow the same route. “If you have got redundant fiber lying in the same pathway, then the physical location isn’t redundant,” Lepp says. “As soon as you hit that with a forklift, your network is down.”
The future of connected automation
Cisco and Rockwell Automation have collaborated on the Converged Plantwide Ethernet (CPwE), an evolving set of reference architectures for connected automation. Each partner maintains its own labs where best practices are designed and tested before being added to a growing library of reference materials. For example, a white paper released in June outlines use cases for deploying industrial firewalls. Panduit also recently introduced physical infrastructure recommendations for the reference architectures.
All of which should help plant engineers and operators in the future avoid the kinds of problems faced by the pharma manufacturer and its errant software upgrade process. In that case, Wilcox says, plant managers were able to keep the problem from happening again through additional operator education (don’t accept a system upgrade while a batch is running), improved communication between IT and OT departments (don’t try to upgrade production servers while they’re running), and preventive controls (critical systems are only upgradable on maintenance days). “To my knowledge, that customer has never had an incident like that again,” he says. “A little pain upfront, but it was a happy ending.”
Call it growing pains on the way to a new world of connected automation.
