Formally organized in 1834, Waterford Township is located geographically in the center of Oakland County, Michigan, and is home to over 72,000 residents. With 360 miles of water main and 355 miles of sanitary sewer, water management in Waterford is no small task. The Department of Public Works (DPW) operates and maintains 19 production wells, 3 storage tanks, 11 treatment plants, and 63 sewer lift stations.
To run all this, they invested years ago in integrating core applications, including geographic information systems (GIS), asset management systems (AMS), enterprise content management (ECM), and supervisory control and data acquisition (SCADA).
That system delivered a lot of value over the years, but nothing lasts forever.
Time to upgrade
In 2017, Russell Williams, director of public works, and Frank Fisher, engineering superintendent, at Waterford DPW started on a project to upgrade their core SCADA infrastructure. The next year, they attended a conference announcing the release of Opto 22’s groov EPIC and were excited by the potential of MQTT Sparkplug to eliminate some long-standing systemic limitations.
With more than 90 controllers on their network, the polling mechanism they used, combined with the limited bandwidth of their radio network, meant that data from each site would update only every 3-5 minutes. Sometimes a lift station would run briefly in between polling cycles, creating gaps in their reporting and inhibiting operators’ ability to accurately detect issues until alarms eventually made their way through. And for each I/O point they added to the system, this latency only grew worse.
It seemed clear that MQTT’s report-by-exception behavior could significantly reduce bandwidth usage and ensure delivery of important system actions.
“We have many lift stations that will spend most of their time sitting,” Williams explains, “[So] why transfer data all the time?”
And with no dependence on a central polling program, they saw the possibility to eliminate a systemic bottleneck and potential point of failure.
From proof-of-concept to production
To help them execute their vision, Waterford DPW engaged Perceptive Controls, a Michigan-based system integrator specializing in industrial and process control applications for the water/wastewater, food and beverage, and oil and gas industries. But building an MQTT system for the first time came with a learning curve, according to Kevin Finkler, software engineer at Perceptive.
MQTT’s publish-subscribe communication model is a definite departure from that of traditional industrial protocols in a few key ways:
- Each field device connects only to the broker, regardless of where its data needs to go.
- When using Sparkplug payloads, each device publishes (sends) a list of its available data items, called topics, upon establishing a connection to the broker.
- Other MQTT clients may also connect to the broker, see the available topics, and then subscribe to (request) updates on those topics when published by field devices.
After experimenting with a few popular SCADA packages, Perceptive Controls decided on Ignition by Inductive Automation because it offered very tight MQTT integration, including the ability to serve as an MQTT broker itself.
Even though understanding the MQTT communication model took Finkler some work at first, establishing communication was straightforward in the end.
“It kind of happens automatically,” Finkler says. “You basically define a few parameters [in Ignition] to set up the broker. And each of the EPIC devices was pretty simple. You just point it at the broker and it starts sending tags.”
“I love that both of these sides have embraced MQTT,” adds Fisher. “It makes the connection seamless.”
Seeing an opportunity to leverage cloud computing for greater fault tolerance and scalability, Fisher decided to deploy Ignition directly on Amazon Web Services (AWS), and he and Kevin began building out the mechanisms to secure the new infrastructure.
First, Fisher configured the firewall on AWS to accept traffic only from his groov EPIC controllers and specific Ignition clients in Waterford’s and Perceptive’s offices. Firewalls on the cell modems and EPICs were also configured to accept only trusted IPs.
He then installed a client SSL certificate on each EPIC so that Ignition could authenticate and encrypt the connection, protecting against man-in-the-middle attacks that could expose data or permit unauthorized control.
Every authorized user is required to create strong passwords to access any groov EPIC controller or Ignition client in the system. In addition, every user login is tracked and reported throughout the system as well.
Fisher and Finkler even integrated physical site security into Ignition. Each lift station is secured with an outer door under lock and key, and a physical switch on the door is connected to the local EPIC. Ignition monitors the switch state to detect when someone enters. If a user login is not registered within a specific time with access privileges for that specific room, Ignition then generates a global alarm.
After completing upgrades on all 63 of its sewage lift stations and six of its clean water sites, the new groov EPIC/Ignition MQTT infrastructure has reduced field updates from multi-minute cycles to sub-second event-driven publications. With that kind of speed, Waterford never misses a system action or alarm notification anymore, and with cell-enabled tablets, operators can stay connected from anywhere through Ignition’s mobile-ready HMI client.
Because of MQTT’s report-by-exception behavior, in combination with analog I/O deadbanding in each groov EPIC, the new infrastructure has also reduced bandwidth consumption, allowing Waterford to publish even more data than before. They have access to communications and controller diagnostics—such as update latency, connection time stamps, message size, and firmware version—which simply wasn’t possible in the old system.
Waterford’s cloud-based infrastructure also enables greater flexibility and reliability. If there is ever an issue connecting to the data center in Ohio that hosts the new SCADA server, Fisher can have the entire system up and running in a different data center in 30 minutes. In time, he will likely set up full server redundancy.
In fact, a recent internet outage at the Department of Public Works offices provided an unexpected test of their new system, which kept on working without interruption.
“We only lost the old system,” says Fisher. “Our internal stuff couldn’t reach out, of course, but our iPads could connect through Verizon... and I was able to get back in touch. In a situation like this, the old system couldn’t send out alarms because it depended on a local connection. The new system didn’t even notice or care because it’s not running anything local.”
More to come
With huge increases in bandwidth, the low administrative overhead of MQTT Sparkplug, and EPICs providing spare data processing at the edge, Waterford can continue expanding its system for a very long time. Each new device or application they add only needs a connection to the MQTT broker to produce or consume data for/from the whole system.
“We are still trying to figure out what else we can do with this,” says Fisher. “We have a lot of other instrumentation that we want to be able to pull data from out in the field that wasn’t really feasible before… not just at our lift stations and our treatment plants but throughout the organization. Where can we use [MQTT] with flowmeters? Where can we use it throughout all of our assets to give us a better overview? We’re just beginning that journey.”
For more information, visit www.perceptivecontrols.com or contact Frank Fisher at [email protected].
