The software must control and monitor toxic gases used in depositing and etching of silicon circuitry on silicon wafers. False alarms and downtime can easily spoil hundreds of thousands of dollars of product, and leaks can be lethal.
For this reason, Staff Equipment Engineer Chris Roggenbuck has the job of testing and managing the frequent changes made to the hundreds of programs controlling these gases and other aspects of production. Like most companies these days, though, the company wanted to tighten its security. So it joined the growing ranks of manufacturers using the tools that automation vendors have developed for controlling and documenting program changes in controllers, human-machine interfaces (HMIs), and manufacturing execution system (MES) platforms.
Driving the decision was the fact that ensuring the reliability of the programs is not easy at NEC. The more than 100 programmable logic controllers (PLCs) in the facility receive input from thousands of sensors and interact with each over of the network. These programs include more than the control logic for whatever task is assigned to the PLCs. They also include the software that governs all of the associated co-processors and touch screens, as well as communications among them and the PLCs.
This nest of software is always in flux. Unlike other kinds of manufacturers that bring a process to steady state and leave it alone, NEC must constantly change its processes to keep pace with the explosive rate of innovation in its industry. Not only does it need to fit its processes with the latest innovations to remain competitive, but it also must make the chips that its customers want for their products.
Another challenge to controlling revisions is that the engineering staff is not always there. An engineering consulting firm about 10 minutes down the street does nearly all of the programming. Three engineers from the Roseville office of Barry-Wehmiller Design Group Inc. provide their services on an as-needed basis, averaging about two days a week at NEC. Maintaining continuity in this environment requires strict discipline.
When on site, a consulting engineer retrieves a copy of the most recent version of the program at hand and goes to work making the necessary changes off-line in the control room. After completing the changes, the consultant stores the new program on the server and documents the changes in a text file. Roggenbuck then installs and tests it, and has the consultant make any necessary changes. “Maybe a timer value needs to be increased, or a sensor tag is changed at the last minute,” offers Joe Gruber, director of process control at BW Design Group.
Reinforcing due diligence
Although the current team has developed the discipline to be diligent about observing the internal controls, there is always the chance of human error. “Sometimes, the test engineers get sidetracked, and maybe the changes don’t get put back on the server until the next day,” says Gruber. “Meanwhile, if another engineer comes in with another change, he naturally would retrieve the program on the server, which would not be the most recent program.” And the chances for errors will only increase as the team evolves and new members are added.
To avoid these problems, BW Design Group installed Proficy Change Management software from GE Fanuc Automation Inc., Charlottesville, Va. Its engineers integrated it into Visual SourceSafe, project-oriented file server-based version-control software from Microsoft Corp., Redmond, Wash., which developers use to manage changes to source code.
The software controls access to code. Only people with clearance can log into the server and check out a program to work on it. “And then, no one else can touch the program on the server until that person checks the program back in,” says Gruber. “Not only are we avoiding more than one person changing the program at the same time, but we also have the time and date that someone checked the program out.” At check-in, the software prompts the user for comments for the documentation log and records them with the log-in information.
Because the information is no longer stored in text files, it is easier to retrieve for troubleshooting. At a minimum, the software reports who made changes when, no matter what the brand is. For GE Fanuc controllers, however, the software also can exploit embedded features to flag which rungs changed in a version. “The logic associated with one change might be scattered throughout the program,” explains Gruber. “Finding all of those instances takes a lot of hunting. This will tell you exactly where to look: rung 10, 169, 1500.”
This capability is coming in very handy, because about 90 percent of the controllers in NEC’s fabrication process come from GE Fanuc. Another handy feature is the back-up and restoration function. “If a controller were to go down and lose its program, we can retrieve the program from our server and get back online immediately,” says Gruber. “We have confidence that we’re going to load the most current validated program.”
First line of defense
As NEC and many others have recognized, a manual system relying solely upon the discipline of employees is not enough for controlling software revisions. “It is risky and fallible,” explains Tad Palus, product manager, FactoryTalk AssetCentre, at Milwaukee-based automation vendor Rockwell Automation Inc. So, he recommends installing some sort of automated change and revision management software to reinforce internal discipline.
He offers four basic attributes that such software should have in order to be effective. First is a security system that gives only authorized users access to the code and the ability to revise it. Not only do most of the revision-management packages available today offer the kinds of log-in security used by NEC, but many also take the added step of providing electronic signatures.
An increasing number also can fit inside the security apparatus of existing infrastructure. “The vast majority of users want to use the log-in procedures, ID cards, and biometric controls that they already have,” says Phil Aponte, HMI marketing manager at Siemens Energy & Automation Inc., the Alpharetta, Ga.-based automation supplier. “Their people have already logged in once and they don’t want to force them to log in multiple times.” The second attribute that Rockwell’s Palus recommends for revision-management software is a means to record activity and establish an audit trail. The ability is a crucial productivity tool for regulated industries, such as pharmaceuticals.
Daniel Huot, for one, was glad that Rockwell’s FactoryTalk AssetCentre had this ability. After Aventis merged with Sanofi-Synthelabo to form Paris-based pharmaceutical maker Sanofi-Aventis, the new executive management team wanted to use a different set of metrics to evaluate the performance of the company’s assets. At the plant in Laval, Quebec, Canada, it was Huot’s job to implement the automation necessary to calculate them for the production of Altace, a drug for treating certain cardiovascular diseases.
Not only did the change management tools in FactoryTalk AssetCentre let him document the necessary changes to comply with governmental regulations, but they also tightened the company’s control over its control algorithms. All of the device configuration code sits in a secure database. “I can now catch every change in our facility,” says Huot. “Nothing is left to interpretation and nobody has the permission to modify anything their FactoryTalk AssetCentre user profile does not allow.”
The remaining attributes recommended by Rockwell’s Palus deal with storage. “The third is a source control library that manages the versions of the control-system configuration files,” he says. Fourth is a mechanism to back up the current configuration automatically for later restoration in case of a failure.
The right mindset
Despite the power of change-management systems that have these attributes, Palus recognizes their limitations, and urges users to rely on them only as the first line of defense. “An overall change-control strategy also should include such considerations as physical controls to prevent unauthorized access to the facility, information technology policies, and procedures to reduce the likelihood of external penetration of facility network resources,” he says.
In the end, though, the success of any revision-management program will depend more upon the mindset of the people in the company. Users, for example, can undermine it by entering Xs and Ys in the comment field. “No off-the-shelf software or hardware tool is going to protect against that,” notes Aponte, at Siemens. “You must maintain good practices internally.”
Opto 22, a Temecula, Calif.-based automation products supplier, minimizes the temptation to skip the documentation by designing its programming package to be a flowchart-based program that uses plain English commands. Users create programs by selecting blocks that have a predefined function and laying them out in a logical string. Because the blocks are labeled, a level of documentation is part of the design process, rather than something that occurs afterward.
“Compare this to ladder-logic control programming, for example, where I/O points are often indirectly addressed via numerical registers, and commands are expressed with a set of cryptic symbols understandable only to those experienced with ladder logic,” says James Davis, senior application engineer at Opto 22. “This makes it more difficult and discouraging for ladder-logic programmers to document their programs.”
MES brews better beer
Software for managing revisions to PLC algorithms and HMIs are not the only tools available for this task. Often, the MESs that operate above PLCs and below the enterprise resource planning (ERP) system have this ability built into them. “Tools built into our infrastructure let you manage the HMI graphical code, alarm-and-event detection, and the integration with devices and other software, such as databases and ERP applications,” says Steve Garbrecht, program manager for infrastructure and platforms at Wonderware, a Lake Forest, Calif.-based software unit of Invensys.
SABMiller Plc, the world’s second-largest brewing conglomerate, decided to exploit this fact when it built its Ibahyi brewery near Port Elizabeth, South Africa. As it went about building a showcase facility, it tested a number of technologies, including Wonderware’s ArchestrA architecture. The company learned to use it to synchronize changes both locally and globally so the company could improve the efficiency of its operations, yet ensure that its beers taste the same everywhere in the world.
ArchestrA’s interoperability was crucial “because of the variety of different platforms and systems we use in each brewery,” explains Thinus van Schoor, automation manager at SABMiller. Built on Microsoft’s .Net infrastructure, the architecture ties disparate systems together and propagates revisions in a controlled manner.
Software developers can define standards and create templates that allow reusing common sets of code for the next set of applications. “If you had several batch reactors, for example, you would create standards for monitoring and controlling a batch reactor,” says Garbrecht. This would include such things as defining feedback mechanisms, reports and overall management of the operation. As users create these standards, they write the documentation, which becomes embedded in the application’s code.
As is the case with systems used at Sanofi-Aventis and NEC, the software requires users to log in before they check an object out. Each object has a set of permissions associated with it, giving only certain people access to specific chunks of code. “Only some developers will be able to modify the base template, which is the source of the application,” says Garbrecht. “The others may only be able to modify objects that are derived from that standard.” In each case, the software asks for comments on whatever changes were made when the developer checks the changes back in. The historical archive keeps track of when changes were made and who made them.
Creating the standards and templates took a little extra time, but the company reaped significant dividends from the investment. After testing and refining them at a couple of breweries, the company was able to install them in what Garbrecht calls a cookie-cutter approach that cut installation time by as much as 70 percent in some plants.
For more information, search keywords “version control” at www.automationworld.com.
