The software must control and monitor toxic gases used in depositing and etching of silicon circuitry on silicon wafers. False alarms and downtime can easily spoil hundreds of thousands of dollars of product, and leaks can be lethal.

For this reason, Staff Equipment Engineer Chris Roggenbuck has the job of testing and managing the frequent changes made to the hundreds of programs controlling these gases and other aspects of production. Like most companies these days, though, the company wanted to tighten its security. So it joined the growing ranks of manufacturers using the tools that automation vendors have developed for controlling and documenting program changes in controllers, human-machine interfaces (HMIs), and manufacturing execution system (MES) platforms.

Driving the decision was the fact that ensuring the reliability of the programs is not easy at NEC. The more than 100 programmable logic controllers (PLCs) in the facility receive input from thousands of sensors and interact with each over of the network. These programs include more than the control logic for whatever task is assigned to the PLCs. They also include the software that governs all of the associated co-processors and touch screens, as well as communications among them and the PLCs.

This nest of software is always in flux. Unlike other kinds of manufacturers that bring a process to steady state and leave it alone, NEC must constantly change its processes to keep pace with the explosive rate of innovation in its industry. Not only does it need to fit its processes with the latest innovations to remain competitive, but it also must make the chips that its customers want for their products.

Another challenge to controlling revisions is that the engineering staff is not always there. An engineering consulting firm about 10 minutes down the street does nearly all of the programming. Three engineers from the Roseville office of Barry-Wehmiller Design Group Inc. provide their services on an as-needed basis, averaging about two days a week at NEC. Maintaining continuity in this environment requires strict discipline.

When on site, a consulting engineer retrieves a copy of the most recent version of the program at hand and goes to work making the necessary changes off-line in the control room. After completing the changes, the consultant stores the new program on the server and documents the changes in a text file. Roggenbuck then installs and tests it, and has the consultant make any necessary changes. “Maybe a timer value needs to be increased, or a sensor tag is changed at the last minute,” offers Joe Gruber, director of process control at BW Design Group.

Reinforcing due diligence

Although the current team has developed the discipline to be diligent about observing the internal controls, there is always the chance of human error. “Sometimes, the test engineers get sidetracked, and maybe the changes don’t get put back on the server until the next day,” says Gruber. “Meanwhile, if another engineer comes in with another change, he naturally would retrieve the program on the server, which would not be the most recent program.” And the chances for errors will only increase as the team evolves and new members are added.

To avoid these problems, BW Design Group installed Proficy Change Management software from GE Fanuc Automation Inc., Charlottesville, Va. Its engineers integrated it into Visual SourceSafe, project-oriented file server-based version-control software from Microsoft Corp., Redmond, Wash., which developers use to manage changes to source code.

The software controls access to code. Only people with clearance can log into the server and check out a program to work on it. “And then, no one else can touch the program on the server until that person checks the program back in,” says Gruber. “Not only are we avoiding more than one person changing the program at the same time, but we also have the time and date that someone checked the program out.” At check-in, the software prompts the user for comments for the documentation log and records them with the log-in information.

Because the information is no longer stored in text files, it is easier to retrieve for troubleshooting. At a minimum, the software reports who made changes when, no matter what the brand is. For GE Fanuc controllers, however, the software also can exploit embedded features to flag which rungs changed in a version. “The logic associated with one change might be scattered throughout the program,” explains Gruber. “Finding all of those instances takes a lot of hunting. This will tell you exactly where to look: ...