May-June 2011

The New Face of SCADA

New technology and breakthroughs offer great rewards for water and wastewater industries, and, with the constant needs for regulatory compliance plus demands for higher efficiency and constrained budgets, the timing couldn’t be better.

Supervisory control and data acquisition (SCADA) systems are evolving so rapidly that many operators are having trouble keeping up. Areas with rapid growth include Ethernet networking, automation, smart phone applications, energy efficiency, and enterprise merging. Each one of those subjects is an entire field within itself. How are water agencies handling the tidal wave of technology?

“You should be overwhelmed,” says Kip Edgley, vice president, Automation/Integration, Veolia Water North America, Technical Direction Group, Chicago, IL. “SCADA encompasses an entire variety of automation services, hardware, software, and people factors. Even if you pull apart the acronym, the first part being supervisor control, that is completely different from data acquisition. So just the term alone is confusing.”

Staying ahead of the confusion is a large part of Edgley’s job, and the fact that Veolia recently launched the Technical Direction Group, to focus largely on integration issues, reflects one of the industry’s greatest challenges—the use of Ethernet technology (the hardware, software, and protocols used by computer to communicate on networks) in water industry SCADA systems. According to ARC Advisory Group, Dedham, MA, a research firm for industry and infrastructure, the industrial Ethernet infrastructure market has more than tripled since 2006 with worldwide sales growth, making it a nearly-a-billion-dollar market for 2011.

“Integration is becoming more important because of the burgeoning and complete acceptance of Ethernet as the protocol of choice within the industrial facility,” explains Edgley. “It’s very robust and reliable, and, because of its relative ease of use and cost-effectiveness, it’s taken great leaps forward, and that has driven a much stronger focus on integration.”

But Edgley notes that when applied to SCADA, one of the single most complex issues with the Ethernet is its simplicity. Although it sounds like a paradox, simplicity allows it to be distributed in ways that can be easily compromised.

Smart Phones Talking to Scada
Security issues will grow more complex with the developing trend known as “view anywhere strategy.” The generic term refers to remote access to SCADA systems from computers, smart phones, and personal data assistants (PDAs).

 “Yes, view anywhere strategies are here to stay, and it’s great for people that are separated by distance from facilities,” says Edgley. “But, the standards of core platform reliability and robust operations necessary to ensure 100%, or as close to 100% compliance with alarm initiation, require the fastest response possible. Now it is not quite robust enough to be considered a primary technology, so it’s an add-on.”

Veolia takes the technology seriously and is working with a cellular and virtual private network provider at a test facility. “We’re pounding on it to see whether or not it’s robust enough to put out into a facility in Fulton County, Georgia, where we have 35 lift stations and problems with radio communications.”

View anywhere technology is a definite growth area for ProSoft Technology, a Bakersfield, CA developer of communications solutions for water and other manufacturing industries. In fact, the company’s growth in product lines over the last 20 years mirrors the rapid changes in SCADA technology, with over 400 communication interface modules supporting more than 60 different protocols. Two recent additions include an industrial cellular service package for AT&T and Verizon 3G, US markets, and ProSoft i-View, a mobile phone applications for Apple’s iPhone, iPad, and iPod touch platform devices.

“Phone communication is a big upcoming trend,” says Aditi Kulkarni, strategic product marketing analyst, at ProSoft. “Water plants rely on remote communications and, often, don’t have enough manpower for all the stations and pumps in the field, so remote management of unmanned sites is common in the water industry. If you have flood weather, you can’t have people starting and stopping the pumps, and, in those cases, wireless is the answer. Now the trend is to access programmable logic controllers [PLCs] and software with the help of dedicated carrier lines and GSM [Global System for Mobile Communications], and all of these protocols can be done on wireless. ProSoft supports all these wireless and iPhone applications. One application that’s really high-end uses the iPhone and a dedicated carrier so you can you can turn your pumps on and off remotely from anywhere.”

Kulkarni notes that while the view anywhere trend is on the rise, many water districts have a much higher priority in just modernizing their outdated SCADA systems. Yet it’s not a trivial decision to scrap the huge investment that the legacy systems represent, nor may it be affordable to replace an entire system. Not surprisingly, ProSoft has found a rewarding market in protocol conversion modules for communications between new and old equipment, and retention of data.

“Water agencies upgrade typically for the portability of the existing database,” says Kulkarni. “When upgrading, you have to take into consideration the existing devices that are talking to the SCADA system. When you choose a new system, will these devices need to talk to the system? If they are talking, what kind of changes with the SCADA have to undergo, or what kind of time will it take?”

He continues, saying, “Then, how will the response of the new system compare to the old system? Will it be slower or faster? Or will the throughput change because all of these things affect the process? Many times when the SCADA system is changed, the graphic screens improve if it’s just an upgrade from the same manufacturer. But if you’re in a new environment, sometimes it cannot be ported.”

Another consideration when porting to a new environment is the trend of accessing data from the SCADA system for use in enterprise analysis, planning, and strategy. According to the ARC Advisory Group, the SCADA market is transforming due to the far-reaching impact of technology, and SCADA systems can empower a quantum-change in the way companies restructure business processes to respond to real-time knowledge flow. The group notes specifically that SCADA can provide water utilities with migration planning and management solutions to enable required business process changes.

“Beyond the SCADA layer there is the enterprise layer, and the database can be used by management for business logic,” notes Kulkarni. “So, when you upgrade to a newer system, you have to make sure that you’re not losing any data.”

Photo: ProSoft RadioLinx Intelligent Cellular (RLXIC) radio for Ethernet Communications

20/20 Hindsight
Obviously, maintaining the integrity of data is a large part of a successful upgrade, but there are plenty of other areas for caution, says Grant Van Hemert, P.E., water wastewater applications specialist for the Schneider Electric Water and Wastewater Competency Center, Palatine, IL.

“The average water and wastewater agency updates a facility once every 20 years,” states Van Hemert. “Look back at the amount of time. Did you have DSL or cell phones, or even e-mail around 20 years ago? So, from an automation standpoint, here we are working in 2010 technology talking to a client base that may still be thinking in 1990 terms. That presents a challenge with microchip manufacturers, because it’s five years from the time that they manufacture and release a chip to the time they consider it obsolete.”

Then, too, don’t forget the culture shift in the workplace and, further, the spatial shift. Thanks to the Internet, Van Hemert’s experience with water treatment operators and engineers shows that getting people up to speed is not necessarily as difficult as it may sound, because of the widespread use of Ethernet technology for consumer use in the home. Nonetheless, there are often some surprises inside a facility’s hardware cabinets.

“If you had a PLC that you bought in 1993, it takes up a certain amount of real estate and has a footprint within the control panel,” says Van Hemert. “And all the wires are cut to the length of the equipment that was installed. If you take that PLC out and rip out the heart of that control panel, all of a sudden the wires won’t reach the input and output cards; so what do you do? Do you rip the whole panel? Or do you actually find a way to keep the wires the same length and put your new equipment in within the same footprint?”

One available solution is a migration kit that allows for a modern PLC with the same footprint and connections for the new input and output cards to the existing wiring. Also, standardizing on the right technology can make a difference for both convenience and reliability, adds Vernon Madrid, maintenance supervisor water treatment facility, City of Fayetteville, NC.

“At our wastewater and water facilities, there’s a lot of data, control, and functionality needed, so we have standardized on the Modicon Quantum processor, because they are proven and we’re very particular,” explains Madrid. “Because we are centralized, the standardization issue is a big deal. We have Modicon processors at all the plants, and we’re actually modifying our Cross Creek Wastewater plant with a total of seven brand-new Modicon PLCs and new software.”

In 1992, the city purchased its first Modicon PLC and placed it in service at the Hoffer Water Plant in an air compressor application. It’s still working flawlessly, reports Madrid.

Photo: Schneider Electric Schneider SCADA control room console layout

The SCADA system is also standardized on the Ethernet networking protocol, thanks to a fiber optics network established in the 1990s by the Public Works Commission of the City of Fayetteville.

“We ended up with facilities that had Ethernet drops everywhere there was a RTU [remote telemetry unit],” recalls Madrid. “So, we have most everything on one Ethernet, and we can cross information from water plants to wastewater plants and report information from lift stations.”

New Channels for Video
Fiber optics turned out to have an additional benefit in the fact that it can support the bandwidth needed for video. After 9/11, security became a huge issue and mandated the need for cameras on all elevated tanks and lift stations.

Madrid explains, “We had a trunk line out there, so it wasn’t a big deal to run some small fiber, and when that happened the floodgates opened. We got rid of our radio system on the water distribution system and brought it all in on the open protocol modbus TCP [Transmission Control Protocol] with fiber. Now instead of dealing with the latency issues of radio, we have near real-time monitoring and data.”

The use of video continues to expand as security remains a high priority for water districts and agencies. But most aren’t quite as fortunate in having the fiber network seen at Fayetteville. Moreover, fiber isn’t an affordable option for large areas, such as the 125-square-mile, Otay Water District, San Diego County, CA. According to Bruce Trites, Otay’s network engineer, the district had a radio system for SCADA installed at 24 sites, but it wouldn’t support video.

“We have a diverse geographic area with a lot of hills and reservoirs and pump stations,” says Trites. “We wanted at least one or two cameras at each site, so bandwidth in the range of 25 to 30 megabytes per second would allow us to keep an eye on things. The SCADA is very low bandwidth so that wasn’t an issue, but we also had security onsite, plus local Wi-Fi access for operators.”

The solution was a wireless mesh video network from Firetide, Los Gatos, CA. According to Ksenia Coffman, senior marketing manager at Firetide, Otay water deployed a mesh with backhaul connectivity to connect their main sites with their assets.

“A 900-MHz near line of sight technology collects the data, so they’re actually using a combination of various frequencies that we offer,” says Coffman. “They were getting quotes from local service providers, and leased lines were going to run hundreds of thousands of dollars in recurring cost. With this system, they own the infrastructure and don’t have to incur the costs of dealing with a service provider.”

“We considered a local cable company, but they were only able to service about three quarters of our sites, and they wanted a 10-year commitment from us at the rate of about $10,000 a month,” adds Trites. “An optical or fiber network from the local phone company was a similar situation with limited site access and much trenching and long-term commitments with even higher monthly fees.”

Coffman notes that the mesh offers very low latency and high redundancy, factors that are critical for real-time applications. “There are a lot of physical security risks that utilities are faced with because it’s not just vandalism, it’s an issue of homeland security, such as someone tampering with the city’s water supply or disrupting the electrical grid. So higher bandwidth allows real-time video alert or access control data a couple of times a day, and with high-performance network they can do real-time video streaming 24/7 and access control information to other sites in real-time. So it’s also running the SCADA application on this network, but it offers so much more for other applications.”

Gauging the Rain
Sometimes there are networks with situations where data needs to be accessed, viewed, and imported to a wide variety of programs, but it doesn’t need to be controlled. Such is the case with Seattle Public Utilities, Seattle, WA, where Laura Reed manages a program for combined sewer overflow monitoring and prevention. Reed needed a way to manage problems and track their resolution at sites throughout the city. The solution was FlowWorks Analysis Software, a suite of online data collection, monitoring, analysis, and reporting tools. Based in Seattle, FlowWorks has created reporting and analysis software that enables users to efficiently manage their monitoring systems from a customized webpage.

“We don’t sell software or hardware and we don’t do any field servicing, we just manage data,” says Timothy Hicks, president of FlowWorks. “So we can take data from any system, flow meter, or instrument or SCADA system and also manual data and historic data, then pull it together at one location. Agencies can have 30 years of data in logbooks, in people’s heads, and in file cabinets, and we gather it together and give customers the tools to actually do something with it.”

For Seattle, that means data from flow meters and rain gauges that help the city in preventing combined sewer overflows. According to Brian Morganroth, a senior civil engineer and data steward for rainfall data in Seattle, “We like to see data and what’s happening in near-real-time or real-time, and FlowWorks is very open-ended. So we are able to take signaling from all the equipment we have in the field, and as data is collected it can be saved to FlowWorks and presented with a simple interface using Google Maps. We can see hydrographs and other representations of something that has happened, or as soon as the system updates. Because it’s a Web portal, you can access it from home or your laptop, so it’s very desirable.”

The Big Data Wave
The FlowWorks solution is another example of data access getting easier, but also more detailed. Higher-data volume with more depth would seem to be an advantage, but there are two sides to this benefit, according to Hany Fouda, Business Development Director—America for Remote SCADA & Telemetry, Control Microsystems, Ontario, CA.

“One of the bright spots we’re seeing is that operators have the ability to access far more information and how the system is working,” he says. “The widespread use of microprocessors combined with an extreme reduction in price has given us the ability to embed intelligence and collect far more data than we could 25 and 50 years ago.”

But are we to a point of too much data? Yes, but Fouda sees it as an opportunity for the industry.

“This is something of a data tsunami, because every sensor in every piece of equipment and infrastructure is now talking to you, and the computer’s brain is not able to analyze all that data,” he adds. “It has to be done by a combination of software and hardware to provide operators with the status of the system, yet not overwhelm them with alarms and sensor data. We are trying to build more intelligence into the software and hardware in order to eliminate the clutter and actually pinpoint the insights that the operator needs.”

Veolia Water North America With SCADA, recording and saving data is just one mouseclick away.

Another consideration in the programming of these approaches is the fact that the workforce that will replace the retiring generation of Baby Boomers is a different breed.

“You’re seeing our company and other vendors moving into an appliance mode where applications are already built into the hardware and ready to talk and implement,” say Fouda. “I think you will see that this trend grows over the next five years because of aging workforce and, additionally, because the talent is scarce. The current talent is not interested in learning logic and C-language because of the way everything is being done for them by Google and Facebook.”

ProSoft Hydroelectric Modules

For networks that require rapid deployment with low cost and simplicity, Rugid Computer, Olympia, WA, has developed WiSI a new family of wireless data acquisition products that address remote access with a unique design. The outdoor sensor is offered in two battery-powered 12-V DC versions, or with an integrated solar panel and maintenance-free super capacitor. All versions can mount inside a typical 2-inch pipe—leaving just an inconspicuous antennae on top—and, for the solar option, a small solar panel.

In an application such as water tank level measuring and reporting, the WiSI mounts right inside the 2-inch pipe present for the pressure transducer that drops down into the tank.

“The solar-powered unit is self-contained, and all you have to do is run the two signal wires down to the transducer, and this device will generate an 18-volt supply,” explains Brady Melchior, vice president of embedded design and engineering at Rugid. “The onboard solar panel and integrated super capacitor allows us to eliminate batteries so customers can expect a life of 10 years or more for the capacitor, and the solar panel can last 50–100 years, so there really is no maintenance or equipment to change.”

Data is secured with a 128-bit Advanced Encryption Standard with user-defined keys. Star networks with up to 16 nodes have a range of up to 2 miles. For greater distances or dense data collection, tree networks can have up to 200 nodes per network.

“It’s fair to call this a plug-and-play solution,” says Melchior. “The configuration software allows you to set your values and time intervals, and you just use modbus to gather your data. It’s a self configuring network, and you can have repeating nodes in the system for redundant paths.”

Any modbus-enabled device can pull information from the WiSI and control digital output for pump operations or other devices. The WiSI comes with four analog inputs, four digital inputs, four digital outputs, and 5-V and 18-V DC instrument supplies capable of sourcing up to 25 mA.

More Work, Less Energy
Although the data has to be simplified, that hasn’t slowed the SCADA industry from searching for ways to ring more value out of the numbers, and one of the most promising trends is in the area of energy efficiency, says Van Hemert. “It’s becoming an overused term, but Katrina has really driven a focus on energy efficiency.

“The water industry has process SCADA systems that look at the actual process, but, until recently, the energy use software hasn’t been tied to it,” he adds. “So I can go into a wastewater plant and ask the operator, ‘What is the dissolved oxygen in your basin for the last six weeks?’ They can pull it right up from their SCADA system, but can they tell me what the amount of kilowatt usages are at any particular data point? Or for each blower? Now if energy use has doubled within the last seven days the operator has a problem, and that might be because he has more load on his plant or a leak in the pipe, but who knows? Because there’s no tracking at most of these facilities for power usage. I would like to say that Schneider’s SCADA system is unique in accommodating energy use tracking, but, truly, all systems can.”

Obviously, energy efficiency tools can have a major impact on a water system or wastewater plant’s operations, and Schneider reports that energy savings of 30% are not unusual.

Along with the progress seen in automation, smart phone applications, and enterprise merging, it’s clear that SCADA is becoming even more valuable as one of the most effective tools available to help water agencies meet their operating challenges.