Formally organized in 1834, Waterford Township is located geographically in the center of Oakland County, Michigan, USA, and is home to over 72,000 residents. It is known regionally for its 34 lakes, from which it earns its name.
However, within municipal public utilities, Waterford is known for its leadership and persistent innovation in water/wastewater management. With 360 miles of water main and 355 miles of sanitary sewer, water management in Waterford is no small task. The Department of Public Works (DPW) operates and maintains 19 production wells, 3 storage tanks, 11 treatment plants, and 63 sewer lift stations.
To run all this, they invested years ago in integrating core applications, including geographic information systems (GIS), asset management systems (AMS), enterprise content management (ECM), and supervisory control and data acquisition (SCADA), all of them sharing data to enable seamless operations.
That system has delivered a lot of value over the years, but nothing lasts forever.
Time to Upgrade
In 2017, Russell Williams, director of public works, and Frank Fisher, engineering superintendent, at Waterford DPW started on a routine maintenance project to upgrade their core SCADA infrastructure. At the time, they used a serial polling program to request updates from their many sites through a network of RTUs (remote telemetry units) that communicated over licensed radio frequency (RF) transmitters.
A year later, they had begun replacing these RTUs and radios with Opto 22 SNAP PAC S2 controllers and DIGI 4G LTE industrial cellular modems communicating through a private Verizon network.
However, that same year, they attended a conference announcing the release of Opto 22’s groov EPIC edge programmable industrial controller, and it changed the scope of their plans. As Russ tells it, “We were talking about it on the ride back and said, ‘If this does what it’s supposed to, then it changes the whole layout of everything.'”
Removing Systemic Limitations
They were particularly excited by the idea that the EPIC’s native support for MQTT Sparkplug publishing could help them eliminate some long-standing systemic limitations. With over 90 controllers on their network, the polling mechanism they used, combined with the limited bandwidth of their radio network, meant that data from each site would update only every 3-5 minutes.
Sometimes a lift station would run briefly in between polling cycles, creating gaps in their reporting and inhibiting operators’ ability to accurately detect issues until alarms eventually made their way through. For each I/O point they added to the system, this latency only grew worse.
Waterford DPW’s legacy infrastructure relied on a network of RTUs and RF transmitters communicating to SCADA workstations in the office.
It seemed clear to them that MQTT (formerly IBM’s MQ Telemetry Transport, now available in open source) could significantly reduce bandwidth usage and ensure delivery of important system actions. That’s because, as opposed to cyclic polling, MQTT follows a strict report-by-exception publishing rule. Instead of waiting to be commanded by the central server (called, in MQTT parlance, the broker), field devices and other MQTT network clients send data on their own, if and only if there is a change in a monitored value.
“We have many lift stations that will spend most of their time sitting,” Russ explains, “[So] why transfer data all the time?”
With no dependence on a central polling program, they saw the possibility to eliminate a systemic bottleneck and potential point of failure. In fact, MQTT’s Sparkplug payload format takes resilience one step further by enabling edge devices like groov EPIC to store updates locally, in the event of a network interruption, and forward them to the broker as historical tags when the connection is restored.
“It just looks too simple. You’ve got to question it,” recalls Russ.
But, willing to test the premise, Russ and Frank purchased three EPICs to play with over the summer, and soon they had the evidence in hand.
“We disconnected a controller and within a millisecond the system reported the failure. It really is that easy: change a variable and it shows up in the broker, then on your mobile phone,” says Frank.
“It is that simple!” confirms Russ.
A comparison of one of Waterford DPW’s lift station control panels showing the old RTU layout and the new layout with the groov EPIC controller and DIGI modem.
From Proof of Concept to In Production
To help them execute their vision, Waterford DPW engaged Perceptive Controls, a Michigan-based system integrator and long-time partner of Opto 22, specializing in industrial and process control applications for the water/wastewater, food and beverage, and oil and gas industries. But building an MQTT system for the first time came with a learning curve, according to Kevin Finkler, software engineer at Perceptive.
“This was the first time I had done something that wasn’t strict client-server,” says Kevin. “The topic system and how you can subscribe to a particular topic is pretty different…. When you first jump into MQTT, you understand that clients connect to brokers, but how do you actually send data? … You can browse through the broker and see it there, but understanding how it’s functioning is hard.”
MQTT’s publish-subscribe communication model is a definite departure from that of traditional industrial protocols in a few key ways:
- Field device connections originate from the device, not the broker.
- Each field device connects only to the broker, regardless of where its data needs to go.
- When using Sparkplug payloads, each device publishes (sends) a list of its available data items, called topics, upon establishing a connection to the broker.
- Other MQTT clients may also connect to the broker, see the available topics, and then subscribe to (request) updates on those topics when published by field devices.
- When a field device publishes an update, the broker forwards that update to all subscribing clients.
Understandably, the first challenge that Waterford faced was integrating this new communication model into their existing SCADA system, but this ultimately proved to be a nonstarter. At the time, Waterford had two workstations running an older version of GE Proficy iFIX, and the system simply lacked the ability to work with the MQTT protocol.
After experimenting with a few popular SCADA packages, they decided on Ignition by Inductive Automation® because it offered very tight MQTT integration, including the ability to serve as an MQTT broker. Even though understanding the MQTT model took Kevin some work at first, establishing communication was straightforward once he had the right tools.
“It kind of happens automagically,” Kevin says. “You basically define a few parameters [in Ignition] to set up the broker. And each of the EPIC devices was pretty simple. You just point it at the broker and it starts sending tags.”
No “send data” commands to worry about after all.
“I love that both of these sides have embraced MQTT,” adds Frank Fisher. “It makes the connection seamless.”
Waterford DPW’s modernized infrastructure publishes data from groov EPIC controllers to a cloud-hosted Ignition SCADA and MQTT broker over a 4G LTE cellular network.
Building Defense In Depth
Earlier, as Frank searched for the components to build Waterford’s new SCADA infrastructure, he experimented with hosting an MQTT broker on Amazon Web Services (AWS). With the new cellular network already under construction, it seemed like an opportunity to leverage cloud computing for greater fault tolerance and scalability.
Having successfully tested the concept, when Waterford decided on Ignition as their broker and SCADA, they decided to deploy the new system directly on AWS. With that done, Kevin and Frank began building out the mechanisms to secure the new infrastructure.
First, Frank configured the firewall on AWS to accept traffic only from his groov EPIC controllers and specific Ignition clients in Waterford’s and Perceptive’s offices. Firewalls on the cell modems were also configured to accept only trusted IPs.
Then Frank installed a client SSL certificate on each EPIC, so Ignition could authenticate and encrypt the connection, protecting against man-in-the-middle attacks that could expose data or permit unauthorized control.
Every authorized user is required to create strong passwords to access any groov EPIC controller or Ignition client in the system, but over and above this, every user login is tracked and reported throughout the system as well.
Frank and Kevin even integrated physical site security into Ignition. Each lift station is secured with an outer door—under lock and key—and a physical switch on the door is connected to the local EPIC. Ignition monitors the switch state to detect when someone enters, and if a user login is not registered within a specific time with access privileges for that specific room, then Ignition generates a global alarm.
One of Waterford’s new lift station control screens. The station security panel is shown at center left.
Return on Investment
After completing upgrades on all 63 of its sewage lift stations and 6 of its clean water sites, the new groov EPIC-Ignition MQTT infrastructure has reduced field updates from multi-minute cycles to sub-second event-driven publications. With that kind of speed, Waterford never misses a system action or alarm notification anymore.
In Kevin’s opinion, “Ultra-low latency is probably the biggest benefit. The latency between the controller and the Ignition gateway is less than 200 ms. That’s across the cellular network with all of [the EPICs] communicating to a server in the cloud.”
And in fact, for most sites, it’s closer to 50 ms.
Waterford’s new infrastructure increased the speed of data updates from minutes to less than a second.
Because of MQTT’s report-by-exception behavior, in combination with analog I/O deadbanding in each groov EPIC, the new infrastructure has also reduced bandwidth consumption. This reduction allows Waterford to publish even more data than before. They have access to communications and controller diagnostics, like update latency, connection timestamps, message size, firmware version, and more, which simply wasn’t possible in the old system.
Benefits for Operators
Ignition takes advantage of all this data with a more user-friendly look and feel, highlighting critical elements like wetwell level, run time, and pump flow totals in each lift station, so operators can quickly spot problem indicators.
With cell-enabled tablets, operators can stay connected from anywhere through Ignition’s mobile-ready HMI.
Waterford’s new Ignition overview screen clearly shows any problems in the system.
Waterford’s cloud-based infrastructure also enables greater flexibility and reliability. Perceptive is able to perform controller updates over the air, which has reduced travel time and allowed them to continue project development unabated throughout the COVID-19 pandemic. If there is ever an issue at the Ohio, USA data center that hosts the new SCADA server, in 30 minutes, Frank can have the entire system up and running on a snapshot of the same server hosted in an Oregon, USA data center. In time, he will likely set up full server redundancy.
Russ Williams recognizes that Waterford is leading digital transformation in the public sector.
“I was at a FEMA training not that long ago, and they were adamant about not having an internet connection on your SCADA system,” he says, “but everything we are looking at will be more secure than we could do from [the office because then] you make a building a single point of failure.”
In fact, a recent internet outage at the Department of Public Works offices provided an unexpected test of their new system, which kept on working without interruption.
“We only lost the old system,” says Frank. “Our internal stuff couldn't reach out, of course, but our iPads could connect through Verizon... and I was able to get back in touch. In a situation like this, the old system couldn't send out alarms because it depended on a local connection. The new system didn't even notice or care because it's not running anything local.”
More to Come
Waterford will continue to manage a few sites through their legacy SCADA system until the end of 2022, by which time they expect to complete all remaining upgrades.
With huge increases in bandwidth, the low administrative overhead of MQTT Sparkplug, and groov EPICs providing spare data processing at the edge, Waterford can continue expanding their system for a very long time. Each new device or application they add only needs a connection to the MQTT broker in order to produce or consume data for or from the whole system.
By integrating residential meter data, for instance, they could help the system stay balanced against demand. If they can talk other agencies and neighboring counties into sharing data, they see the potential to build an advanced warning system that would improve their reaction time to system disturbances.
“We are still trying to figure out what else we can do with this,” Frank says. “We have a lot of other instrumentation that we want to be able to pull data from out in the field that wasn’t really feasible before… not just at our lift stations and our treatment plants but throughout the organization. Where can we use it with flowmeters? Where can we use it throughout all of our assets to give us a better overview? We’re just beginning that journey.”
When asked what he thought other engineers needed to understand most about MQTT, Kevin Finkler pointed out that “the client-server communication is essentially handled for them. Before there was [a lot of code] that handled all the communication. It was a lot of work to maintain. Now you just mark a tag as ‘public’ [in groov EPIC] and that’s all handled for you. It’s less work, so it’s less money spent on engineering time.”
And less money spent on engineering time means teams can tackle bigger challenges, moving critical infrastructure closer to a true digital transformation.
For more information, visit www.perceptivecontrols.com or contact Frank Fisher at firstname.lastname@example.org
Download case study pdf