2001 - California Water Authority Upgrades Their SCADA Network

California Water Authority Upgrades SCADA Network With Redundant Ring Ethernet Switches

Larry Seifert, CCWA Maintenance Superintendent.
Larry Seifert, CCWA Maintenance Superintendent.

Redundant Ring Ethernet Switches are increasing in popularity as Utilities upgrade older networks. In addition to Ethernet benefits of simplicity, expandability and maintenance, these sophisticated Switches maximize reliability by automatically switching over to a secondary data path if a fiber or device failure occurs.

California’s Central Coast Water Authority (CCWA) recently upgraded its 160-mile SCADA fiber optic communications network with Redundant Ring Ethernet Switches. Faced with equipment and vendor support issues, it began the planning process in late 2003.

California’s Central Coast Water Authority
California’s Central Coast Water Authority.

According to Shannon Sweeney, Staff Engineer, the initial focus was on replacing 21 older ModBusPlus modems in a fiber optic network that linked Programmable Logic Controllers (PLCs) to its primary network, a "PremNet 500" self-healing ring fiber optic backbone transport system manufactured by Milgo.

"About two or three years ago we began experiencing fiber modem failures," Shannon said. "We had some spares, but we also began looking for new modems since these modem failures temporarily disrupted small segments of the system."

Then, another issue surfaced when NextiraOne announced that it was going to stop supporting the PremNet system in the near future (NextiraOne acquired Milgo in 2001). Installed in the mid 1990’s, the PremNet System functioned effectively; however, the CCWA was concerned that future equipment failures could seriously impact its network once the current inventory of spare parts was used up.

The timing of this announcement actually turned out to be a positive.

"Instead of looking only at solving the modem failures, we also looked at replacing the PremNet [10Mbps] with a higher speed fiber communications system," she explained. "This analysis led us to ask whether we needed to have all of these separate networks. We also took a hard look at how we could improve network performance and keep it cost effective."

The TC3720 Redundant Ring Switch supports temperature ranges from -40°C to 80°C.
The TC3720 Redundant Ring Switch supports temperature ranges from -40°C to 80°C.

Redundant Ring Switches

To meet these requirements, the CCWA installed Redundant Ring Ethernet Switches manufactured by TC Communications in Irvine, California. Specifically, 22 Model TC3720 10/100 Switches replaced the existing 28 fiber modems and 9 PremNet nodes.

The TC3720 Redundant Switch features a switchover time of less than 250 msec. and supports an extreme temperature range of -40°C to 80°C. It provided two Fiber and Six 10/100M copper ports. The CCWA can monitor the fiber ring, alarm conditions, and fault locations for local and remote units with a web browser. If there is fiber breakage on one remote unit, the local master ALAM LED will flash and the Dry Contact will close. The exact location of the faulty connection can be pinpointed by checking the Ring Status on the Monitoring PC.

Before: Block diagram of the original PremNet & Fiber Optic Modem Network installed in the early 1990s.
After: Twenty-two TC Communications Redundant Ring Ethernet Switches provide SCADA and Voice communications between the various pumping facilities, tanks, valves, water treatment plant and administration building along the CCWA’s 130 mile fiber optic network.

Installation

The installation went smoothly, but Shannon admitted that "we were concerned" despite successfully performing various pre-installation tests. "At first, it was daunting to think that a switch was going to place cabinets full of equipment." As an added precaution, the Premnet System remained intact as a backup.

Larry Seifert, Maintenance Superintendent, oversaw the installation.

"Basically, we took the TC3720s out of the box, configured them…then shoved them in," Larry said.

"We did have to do some minor network configuration. Actually, converting the PLC interfaces [from ModBus] to Ethernet was the most difficult aspect.

"We also tested the redundant switchover capability to the secondary fiber ring and it worked flawlessly."

Close-up of PremNet5000 chassis which was replaced by a Redundant Ring Ethernet Switch.
Close-up of PremNet5000 chassis which was replaced by a Redundant Ring Ethernet Switch.

Network Performance

The improvement in network performance was noticeable. For example, computer system users at the Water Treatment Plant commented that they experienced no delays when uploading files compared to sporadic delays with the older system.

Larry estimated that the new switches were providing a minimum of 40Mbps of throughput compared to about 4 Mbps with the older system. Shannon noted that the new network is "much cleaner and simpler" than the previous one. "By limiting the number of separate networks, we were able to flatten out and basically simplify the overall network," she said.

Ethernet Phone Extenders

The new Switches also simplified adding telephone service via the CCWA network. This was accomplished by using Ethernet Telephone Extenders, Model TC1910s, also from TC Communications.

Equipment enclosure with various communications equipment. The TC3720 Redundant Ring Switch is mounted vertically on the right side.

The Ethernet Phone Extenders had two purposes: one, they provided a backup emergency phone connection between the Water Treatment Plant and the Administration Office, a distance of more than 100 miles. (For example, regular telephone service between these two important sites was interrupted during a recent earthquake).

Two, the TC1910s added telephone links between designated remote sites (e.g. Tank Site 2) and the Administration Building in Buellton. They can also be used to dial outside the Ethernet Network when connected to an analog PBX.

The CCWA Network

The primary CCWA SCADA Communications Network is installed along 130 miles of pipeline between the Water Treatment Plant near Shandon, CA, and the Santa Ynez Pumping Facility in Santa Ynez, CA. The longest distance between the 22 Ethernet Switches is about 38 miles (see diagram).

The network is comprised of twelve strands of single mode duplex fiber optic cable. Multimode fiber is used for shorter runs on the sub-networks. Currently, 8 strands of fiber are used for the PremNet system and 4 strands for the Redundant Ethernet Switches.

The SCADA software system monitors the PLCs located at the various pumping stations, tanks, valves etc. sites along network. The PLCs connect to a variety of critical telemetry equipment such as flow meters, valve controls, water level sensors, contact alarms and pump starts-and-stops.

NOTE: It is interesting to note that special precautions were taken to compensate for the extraordinary number of earthquakes that occur each year near the Water Treatment Plant. Eighteen above-ground couplers have been installed to buffer earthquake effects where the pipeline crosses the San Andreas Fault. In addition, the fiber optic cable has extra coils to add flexibility and prevent breakage in the event of earthquake related stress or impact.

Ethernet Telephone Extenders (Model TC1910s) provide various maintenance phone links and emergency telephone service between the Admin Building and the Water Treatment Plant, a distance of more than 100 miles.
Ethernet Telephone Extenders (Model TC1910s) provide various maintenance phone links and emergency telephone service between the Admin Building and the Water Treatment Plant, a distance of more than 100 miles.

Future Plans

Future network plans could include IP video cameras, according to Shannon.

"We are seriously considering installing IP cameras at remote locations for general security surveillance," she said. The cameras under consideration would be activated by pre-designated event or motion parameters.

They also include disk storage capacity. As a result, these types of IP cameras require minimal bandwidth and are ideal for 100mbps Ethernet networks.

CCWA Pipeline

The CCWA is responsible for treating and delivering reliable, high quality supplemental water to its project participants in San Luis Obispo and Santa Barbara Counties. Participants include the cities of Santa Barbara, Carpinteria, Santa Maria, Vandenberg Air Force Base, Montecito, Buellton, Solvang, Santa Ynez, Orcutt and Guadalupe.

Designated the Coastal Branch Phase II pipeline, it began operation in 1997. It is fed by a California State Water Project aqueduct that originates in the Sacramento-San Joaquin Delta and processes up to 43 million gallons of water a day at its water treatment plant at Polonio Pass. After undergoing a process to remove the disinfectant at the Santa Ynez Pumping Facility, the excess water that is not distributed is then pumped into Lake Cachuma.