The technologies encompassed by the term “manufacturing IT” are very broad and touch on everything from industrial networks to document management. Underlying it all is a skill known as software engineering.
Since most manufacturing, chemical, and electrical engineers are far from proficient in the field of software engineering, Automation World asked Dennis Brandl, founder of BR&L Consulting (www.brlconsulting.com) and well-known manufacturing IT expert, to explain this topic at The Automation Conference 2013 (www.theautomationconference.com). Here are the key points—presented in a “good, bad, and ugly” framework—that Dennis shared in his presentation.
Software architectures
Good software architecture uses a formal model such as Unified Modeling Language (UML) or Entity Relationship Diagrams (ERDs) as well as flowcharts and process models. These architectures are easily recognizable because all the pieces fit together, there is no duplication in functionality, integration is not complicated and it is designed for extensions. UML and ERDs are good because they provide a way to review requirements, review systems, and ensure that someone reading them can understand it. After all, English is a fuzzy language when it comes to software architecture descriptions.
Bad software architectures are usually things Dennis refers to as “marketectures”—in other words, an attempt by the marketing department to create an architecture. These PowerPoint presentations, developed by marketing for sales engineers to explain the main elements of a system, look pretty but don't have any of the detailed information of a real architecture.
Really ugly examples of software architectures are where there isn’t even an architect associated with the design. This often arises when someone unfamiliar with IT is assigned to create software for an operation. These people, often engineers of one stripe or another, look at the project and see that an architect doesn’t generate code, write detailed design documents and test cases, or execute tests so they think having an architect on the project is just overhead. They’re wrong.
Architecture is a very defined skill set. It can’t be assumed that the project manager can be the architect or that the coders will figure out the architecture as they go. In such cases, the architecture often looks as if it was designed by 16 different people—and in some cases you find out it was. As a result, stark inconsistencies often exist within the architecture, and integration is a pain because there is no architectural pattern behind it.
Dennis advises buyers that any vendor should be able to show you the architecture for any software system you are considering. He adds that the documents should show a detailed architecture design, not a “marketecture.”
Document management
Most manufacturing engineers do not have any formal source code control system (SCCS) or document management system (DMS) experience. As a result, they may not realize the productivity losses that occur when shared files and emails are used to manage project information. Some people think a SCCS is just for code, but it can be used for any artifact you’re trying to capture in an automation system.
A good example of a DMS or SCCS is when these tools are used in shared files for check in, check out, version management, change tracking, meta data search, split and merge applications, or backup and recovery.
A bad example is when you’re using a tool like SCCS only for code and not for HMIs or other automation software applications.
The really ugly DMS or SCCS example is when you aren't using either for your shared file systems. This means there’s no check-in/check-out system. When everyone owns it — no one owns it, Dennis says. It also means there’s no enforced version controls on your software and no protection against worms, which often use shared directories to attack a system. It also means workers waste hundred of hours searching for files.
SCCS have metadata, so you know everything about a file and can search for them easily. They also have split/merge functions and can be used for backup and recovery.
Development sandboxes
Sandboxing is a term that refers to a testing environment that isolates untested code changes and experimentation from the production environment, thereby protecting live servers and the data on them.
The worst thing you can do on a software project is the big bang theory of integration — where everyone works on all the various pieces separately and you then spend several months trying to bring them all together. The preferred way is to do nightly builds where you have source code control, you check in and out, and build the entire system every night. To do that, you have local sandboxes where nightly build directories are passed to user tests before going to final builds.
To do this requires multiple copies of the software you’re running. So you have to talk to your vendor to get multiple copies. If you only buy one copy, how will you make changes without impacting production?
Bad examples of sandboxing involve the use of monthly builds, no dedicated test systems, and long development cycles.
A really ugly example is when someone just goes in and makes the change online. Sometimes you have to do that, but it’s always much better to test changes out in advance.
Industrial networks
The generally accepted good practice for industrial networks involves the use of segmented networks, establishing firewalls between systems, and having separate management rules for switches, routers, etc. ISA-99 lays this out in detail. Work with the IT department to employ these network procedures in your company.
A good strategy Dennis notes having seen used with manufacturing devices such as switches, PLCs, SCADA systems, etc., is to have all the Ethernet cables be one color and the Ethernet cables for the business systems be a different color. By doing this, if there’s a manufacturing-colored cable coming into a box, you can set up a rule that only someone trained in manufacturing IT can work on it. You don't want business IT people coming in to change the router or switch rules, Dennis says.
Not having segmented networks is a bad example of industrial networking. Consider what happens during March Madness if your networks aren't segmented. Your network is completely loaded; and without segmentation you impact production.
The systems on your plant floor (HMIs, PLCs and DCSs) are often quite old, so the accepted practice in business IT of just applying the latest patch when it comes out is not acceptable on the plant floor. As a result, you have to protect manufacturing systems from the business systems. And you need to protect the databases on both sides since they are increasingly sharing more information — that’s where the DMZ and firewalls come in. Without a DMZ or firewall, if any system is compromised, they're all compromised.
It gets really ugly when there are no separate IT management policies for business and manufacturing. This happens a lot at large companies with centralized IT support. In these situations, their answer to the problem is usually to reboot the system. Alternately, if a centralized IT group decides to run a port scan, they likely don't realize that PLCs and DCSs can’t handle all that traffic, the systems then shut down and production stops.
Software documentation
Formal writing has been around a long time, it's too bad most engineers have never learned how to do it, Dennis says with tongue planted firmly in cheek. As a result, many engineers don't document their projects properly and, in the place of that documentation, instead we have online meetings where we tell each other how the systems work.
If a system isn’t documented, you may as well throw it away now because you're going to throw it away eventually, Dennis says.
Signs to look for that indicate a really ugly software documentation process:
• Campfire equivalents where engineers travel around to tell others about how the system works with nothing written down;
• No user guides to help with installation and maintenance of the system;
• Document reviews do not include reviews for readability and clarity;
• No feedback mechanism exists for writers to improve their skills; and
• There is a lot of rework because someone “missed the meeting where we talked about that.”
>> Watch all of Dennis’s presentation from The Automation Conference at http://bit.ly/brandlmfgit.
