The system relies on automation spread out over hundreds of remote sites, from sewage lift stations to 1000-ft. or more deep fresh water wells. This part of the country is far from flat, and the distribution of remote sites requires a vast network of fiber optics, wire and radio technology. It is an impressive feat of engineering, and keeping it running efficiently and reliably is the job of automation engineers who rely on a collection of controls, infrastructure components and SCADA technology, all working in harmony day in and day out. When there is a problem, it needs to be diagnosed quickly, to keep the system in balance and to avoid any disruption of service to customers. That is where SNMP comes in.
SNMP stands for Simple Network Management Protocol, a standard for monitoring and managing a wide variety of information technology (IT) components like routers, switches, Wireless Access Points, and radio modems. While SNMP is commonly used in the commissioning of an infrastructure, or sporadically as IT professionals go about their business of supporting their Process Engineer brethren, SNMP is becoming more commonly used in real-time monitoring and management with the automation applications that are responsible for the operation of the core business.
For this water company, infrastructure components include BreezeNET® products by Alvarion, Cisco switches and routers, and GE MDS radios and Access Points. These components make up the spider web of connectivity to over 100 remote sites across almost 200 square miles. These products all support SNMP and offer a wealth of information suitable for real-time monitoring. Data within a device is defined by the device vendor in an MIB (Management Information Base) file. Through the Kepware iSNMP (Industrial SNMP) product, water companies can import a device MIB and immediately access the information they need to continuously monitor. In addition to configuration information, devices provide real-time data for network traffic, connection details and diagnostic information. The water company automation system consists of Wonderware’s Intouch™ SCADA solution. Automation engineers have added several HMI Screens (see image) enabling them to monitor their infrastructure both for troubleshooting and historization.
A variety of data from these devices enables new alarms and decision capabilities. In the screen image, a green background means everything is good. Technical issues are immediately visible by means of a red background. Alarms associated with the iSNMP variables are managed with supplemental InTouch alarms – a major benefit in that additional applications and personnel are reduced, and the alarm management procedures for notification and follow-up fall within standard procedures. Radios and routers are monitored by a standard bandwidth measurement. It is helpful to see the rate of bytes traveling through I/O ports rather than the percentage of port utilization with respect to capacity of the port. This offers more succinct measurement to be reviewed when planning any network enhancements or corrections. If the bytes transferred ever becomes zero, it is considered an alarm. This calculation is performed by doing math on the (iRouterPortThroughtput) value. Doing it this way provides a more understandable method to measure bytes over a selected time period, usually measured in seconds but calculated every minute or two.
The SCADA application has communication fail alarms built in to describe communications to PLCs; however, this does not explain which component in a multi-node path communication system requires service. The main screen shows all the network nodes required to be in service to propagate the data through the network. If one of the network nodes downstream of the I/O server becomes nonresponsive, it is graphically represented and offers a major clue to how the repairs will be planned. Before this application was created, staff would employ a system of pings, telnet, traceroute, wireshark, Solar Winds and drawings with IP numbers to try to determine which component failed. This process is subtle, tedious and time consuming and requires a good amount of guess work.
This water company is continually expanding their automation system, including their infrastructure. The addition of infrastructure health monitoring has substantially reduced troubleshooting time and allowed them to plan system expansion in a controlled and predictable way.