John Gracia thought he’d seen the last Yokogawa Centum XL distributed control system (DCS) long ago. That is, until he got a call about one still in operation at a Pemex oil refinery in Monterrey, Mexico. After decades of faithful service, the Centum XL’s hard disk had crashed, taking engineering operations at the Pemex site down with it. Gracia, a systems consultant for lifecycle services at Yokogawa, got on a plane to see what he could do about it—which, as it turned out, was quite a lot.
Gracia’s experience highlights both the advantages and the challenges of supporting legacy controls. In the case of the refinery, keeping the engineering control station up and running with the old equipment allowed plant managers to upgrade the refinery’s field control stations one at a time without having to suspend operations. Pemex is just one of the many plant operators that have found it more economical to keep legacy controls running, rather than undergo costly upgrades that might not be necessary.
Making the decision to stay with legacy systems requires balancing the costs and risks of maintaining old systems vs. the risk and expense of upgrading. It’s not always an easy choice, as many of those old systems can keep running long after their intended lifecycle. But fail they inevitably must, eventually in ways that can’t be fixed. And if failure comes without warning, the results can be costly.
“If you’re having more and more failures on a device, and you’re running out of parts,” says Bain Ashworth, product marketing manager for industry solutions at Phoenix Contact, “that’s when things get critical.”
Staying with the tried and true
Legacy controls are particularly prevalent in the oil and gas industry, says Rich Clark, principal application consultant for Honeywell Process Solutions. That’s because the hundreds of thousands of wired connections at a typical oil refinery represent a significant investment. “Often times the single biggest expense of putting in a control system is just wiring it up,” Clark says. “A lot of people want the actual controllers to last a very long time. One of the reasons they want them to last a very long time is that wires last a very long time.” The algorithms developed to run legacy controls also represent a major investment in intellectual property, Clark adds, providing further incentive to maintain legacy controls.
So, how can plant operators best mitigate the risks of maintaining legacy systems while taking maximum advantage of the benefits of deferring the cost of upgrades? It’s all about planning, says Lonnie Morris, senior manager for services at Rockwell Automation. Planning is best done in conjunction with the original manufacturer of a plant’s controls, he says.
“We want to try to start the planning as early as possible with these customers,” Morris says. “Before we discontinue a product, that’s the best time to address the issues.” Just before a product is discontinued might be the best time to stock up on spare parts, he adds.
Proper planning should take into account the unique requirements of each installation, Morris says. He cites three factors to consider when deciding whether to maintain a legacy system or upgrade to a new one.
The first consideration should be the availability of parts for an old system. Ideally, plant managers should have a big enough supply of parts on hand, or within easy reach, for the intended length of the continued life of the system. Next to consider, Morris says, is whether you can get repair support for the old systems. Many manufacturers, including Rockwell, offer legacy repair services, and can be a good place to start when considering options. Finally, Morris advises, look for remote support options—that is, phone or online help.
Rather than choosing to stay with a legacy system or upgrade all at once, plant operators often choose a hybrid approach, where some elements of a legacy system remain in place while others get upgraded. Many manufacturers offer a variety of adaptors—devices designed to act as an interface between legacy components and new ones. This can help operators get the best of both worlds: the cost savings of maintaining the existing wiring and other infrastructure, along with the enhanced features of updated critical components.
Phoenix Contact, for example, offers an extensive line of media converters. These converters include options for connecting serial devices to Ethernet networks and for connecting copper-wired devices to fiber-optic networks. “There’s also what we call comm servers,” Ashworth says. Also known as protocol converters, these devices can take, for example, Modbus instructions from a legacy system and convert them to newer standards like Profibus.
Such devices can even put wired devices on wireless networks, which is just what one of Ashworth’s customers did when upgrading gas chromatographs at a Texas oil refinery. These two dozen or so monitors were plumbed into the pipes spread out over the plant’s several acres. Ripping them out would have been prohibitively expensive. At the same time, they required plant workers to visit each one to download the data they were producing to laptops, and from there to control stations.
Phoenix Contacts’ solution: Plug a radio into the serial port of each monitor and transmit the data to the control network over the air. “It’s an example of not really upgrading anything, but you’re upgrading the infrastructure to get your data back,” Ashworth says. In this case, the plant operators injected new life into 15-year-old monitoring devices while avoiding the expense of installing yet more wiring.
Ashworth cautions that a wireless system that sidesteps the cost of additional wiring—not to mention the savings in man hours spent manually collecting data—can safely get the job done only in situations where the data gathered is not mission-critical. “If the wireless link goes down and the system can’t think on its own, that’s what I consider mission-critical,” Ashworth explains.
Going to the source
The manufacturer of legacy controls could be a plant operator’s best go-to resource when it comes to maintenance and repairs. Rockwell Automation is among those offering service contracts for legacy systems. Under some of these contracts, Rockwell will build up its supply of parts for products before they are discontinued, warehouse them, and then ship them out to customers on a predetermined schedule. This helps ensure a steady supply of spare parts for a discontinued system, while at the same time reducing a customer’s upfront maintenance costs.
Conversations with potential “last-time buy” customers begin with working out an estimate—based on the number of years a plant operator expects to maintain legacy systems—of how many and what kinds of spare parts will be needed. Rockwell also gives priority access to spare parts to companies holding certain types of contracts, Morris says. “We put together something we call parts management agreements,” he says. “A customer doesn’t have to own these parts outright, but has access to these parts.”
While careful advance planning may work well for ongoing operations with a single plant owner, new owners who have inherited the legacy controls of old plants may not have that luxury. Even so, DCS manufacturers can often still help.
Many manufacturers offer dedicated help lines for customers using legacy systems, even if they are not the original purchasers of the equipment. Rockwell, for example, maintains a support desk for legacy systems, called Legacy TechConnect, but customers must contract for it. For more than remote support of their legacy controls, plant operators can, in many cases, also contract for on-site repair and maintenance of legacy systems with their original manufacturers.
For those wishing to go it alone, the supplier websites can also provide valuable resources, often for free. These include libraries of legacy equipment support documents, including scanned versions of the old three-ring binders containing legacy manuals—even those of equipment made by companies that were acquired by larger companies and no longer exist as separate entities.
Forums on DCS manufacturer websites are another source of support, where users can ask each other for help and information on maintaining legacy control systems. There, they can even gather information to help them decide whether or not to upgrade, says Stefan Werner, marketing manager for Siemens Factory Automation. “These things are discussed heavily in these forums,” he says.
Getting outside help
When all else fails, outside sources of support can help operators keep legacy equipment running. DCS user groups on LinkedIn are a particularly good resource, says Yokogawa’s Gracia.
Along with information, outside sources can provide access to replacement parts and repairs. A quick search on eBay, for example, reveals no shortage of complete discontinued DCSs and parts. Classic Automation, based in New York, is one of several third-party suppliers of discontinued control systems and parts. The company also offers repairs by mail and, through sister company Automation Evolution, on site.
Despite an operator’s best efforts to keep legacy equipment going, however, all good things must eventually come to an end. Eventually, the last available circuit board gets fried or the last hard drive for an old system crashes. That’s when an upgrade becomes inevitable, and someone like Gracia gets a call.
The Yokogawa Centrum XL DCS with the bad hard disk that Gracia was tasked with somehow rescuing was produced by his company in the 1980s. Spare parts and the expertise needed to use them were extremely hard to come by. And until it could be fixed, no engineering functions could be performed at the refinery where it had been installed (although control functions could still be handled at new operator stations).
More critically, however, Pemex engineers couldn’t recover the data from the dead drive that was needed to get a replacement up and running. That’s where Gracia came in. Working with a backup that was more than 10 years old, along with up-to-date data from a field control station, Gracia was able to patch the old DCS back together. “I got very lucky,” he says.
But plant operators threw in the towel a month after that, moving ahead with a planned migration to a new Yokogawa DCS. In fact, the engineering station was the last control station to be upgraded. “It’s no longer a legacy system,” says Gracia with a laugh.
