As Power Utilities continue to migrate to Ethernet, many continue to use both Ethernet and SONET for their communication network needs.
Because of its extensive hands-on experience with both SONET and Ethernet, Northwest Iowa Power Cooperative (NIPCO) has successfully taken advantage of each technology's strengths.
Recently, NIPCO simplified its network by putting most of its Ethernet traffic on a common platform by using TC Communication's JumboSwitch® Multi-Service Ethernet Switch.
NIPCO installed its first SONET ring network in the mid 1990s and its first redundant ring gigabit Ethernet network about six years ago. Today, it uses a combination of Ethernet, microwave and SONET for its communications transport including:
- A Gigabit Ethernet backbone as its primary communications network for SCADA, substation and office applications.
- An OC-3 SONET system for T1s and as a network back-up.
- An OC-48 SONET for network back-up and T1/T3 to customers on the western Iowa corridor (Omaha to Sioux City, IA, to Sioux Falls, SD).
- A microwave network for connecting remote substations to the primary network via T1 circuits.
These communication technologies complement each other in many ways, according to Keith Schiltz, NIPCO’s Metering, Digital SCADA and Telecommunications Services Manager. "Depending on the application, both SONET and Ethernet work well" he said. "And, depending on the cards, both offer an on ramp and an off ramp."
Primary Communications Network
In 2006 NIPCO made a decision about the direction of its communication networks that will resonate for years to come.
NIPCO’s 2006 decision to install a Gigabit Ethernet backbone rather than build out the existing SONET network was based on several factors.
Most important, Ethernet gave them additional bandwidth at an acceptable cost.
Second, Ethernet gave them scalability for future network expansion. Doubling or tripling the performance of a SONET ring is more complex (and expensive) than a comparable Ethernet network.
Third, Ethernet gave them the inherent flexibility of Ethernet. For example, they could move servers anywhere on the ring network and still be able to plug into VLAN ports while connecting to all points in the system.
On the basis of performance, expanding its OC3 and OC48 SONET rings and installing additional Ethernet cards would have served NIPCO well for most applications. This option, however, was ruled out because of the major costs associated with upgrading to OC192 to accommodate the increased demand for bandwidth.
Driven by the need to add more network flexibility and bandwidth to its fast-growing telecommunications business, NIPCO designed what would eventually evolve into its primary communications network. Moreover, this design obliged NIPCO to choose between SONET and Ethernet for this application.
After extensive analysis (see sidebar article for details), NIPCO opted to go with Ethernet and create a 650-mile Gigabit backbone network.
NIPCO continued to expand this Ethernet network over the next few years and eventually made plans to upgrade to a more robust multi-service Ethernet platform. To this end, NIPCO installed the JumboSwitches and began moving applications to this network in early 2011.
Benefits of GigE Backbone Network
Upgrading to JumboSwitches provided NIPCO with several immediate benefits including the ability to:
- Move AMR, Teleprotection and telecommunications over to IP
- Simplify maintenance, monitoring and repair
- Simplify Ethernet rate limiting to customers
- Quickly provision and configure core T1 links
- Quickly build out new Ethernet links for new customers
Most important, the JumboSwitch’s TDM over IP/Ethernet capability gave NIPCO an extra option to consider when adding or changing network applications. One example of this option was when NIPCO needed to replace unlicensed microwave at one of its substations because of an unusual problem.
The problem surfaced when the microwave network section between the J8 and J10 substations (see Figure 1) suddenly started having transmission errors. NIPCO technicians and subcontractors triple-checked all of the obvious potential problems, but the situation continued.
After a few days, a contractor who was re-peaking a dish noticed that wind turbines had been disrupting the signals because they had been installed directly in the path of the microwave dish. To get the network back on line quickly, NIPCO arranged to re-route its microwave link through the wind power company’s network.
Transporting T1 over Ethernet
Later, after discussing alternatives to bring the T1 capacity from substation J8 back to J10 and ultimately to the central office in Le Mars, NIPCO chose the JumboSwitch.
"The need for some type of network connectivity was obvious when the unlicensed microwave was replaced at the J8 substation," said Keith. "We researched using either a SONET, fibermux or Ethernet based technology as the new transport."
"The JSW [JumboSwitch] always came out on top after all discussions because of cost, ease of use, efficiency, types of circuits needed [Ethernet, T1]. This site is now a spur off of the main fiber ring and the transition from the main ring onto the spur was a simple SFP Ethernet card."
The cost of a JumboSwitch node with main, management, Ethernet and T1 cards, was about one half the cost of a comparable SONET node.
"Cost is always an important factor," said Keith, "but we also wanted to continue our network migration to Ethernet. It is easier to go forward in the Ethernet world. More and more telecommunication devices are becoming IP based which makes the JSW platform choice a logical one."
Installing the JumboSwitch Network
NIPCO staff implemented the entire JumboSwitch upgrade without outside contractors or consultants and were able to configure, test and commission the new JumboSwitch network in about six weeks. This included, Bench Testing, Field Acceptance Testing (FAT), Site Acceptance Testing (SAT), all field installations and a 4-day hardware and TCView™ Network Management Software training course.
"The training facility and instructor were top notch," said Keith, "and the training was based around our needs."
Both NIPCO's highly qualified technical staff and extensive training played important roles in NIPCO accomplishing so much in such a short time. At the TC Communication's corporate training center in Irvine, California, NIPCO's network technicians learned on JumboSwitches configured and set up to NIPCO network specifications. They worked through many "what if" and troubleshooting situations.
"The focus was always on NIPCO's needs and TC's engineering staff was always willing to listen to our ideas and thoughts," said Keith.
As a result, technicians were able to simulate actual NIPCO network situations and pinpoint how TCView could help streamline their installation and commissioning process by managing network connections and provisioning.
"TCView was definitely designed with the user in mind," said Keith. "It's good at monitoring network performance and with features such as 'Unit Discovery,' we were able to build the system network in a snap. We really like the optical launch and receive power displays along with the unit temperature."
Creating VLANs for Applications
The JumboSwitch network is used for several applications including SCADA, Teleprotection, Automatic Meter Reading (AMR) and office LAN traffic. These applications are differentiated by core Utility VLANs such as "SCADA VLAN" and "substation VLAN." In addition, NIPCO created individual "Guest VLANs" for several of its Ethernet data link service customers. Potential future customer service could include Digital Mobile Radio (DMR) traffic.
"The JumboSwitch makes it easy to provision or configure the Ethernet and T1 circuits that we use," said Keith. "Because the JSW is less demanding for building out Ethernet circuits it is therefore quicker to establish new customer traffic."
"One of the biggest benefits has been how easy it now is to 'plug and play' out in the field," said Keith. "Today, a tech can call up any site from his desk. No more 2 ½ hour drives out to J8 [substation]."
"Ten years ago a technician had to load the truck up with all kinds of spare parts and equipment to be able to deal with on-site problems. Now, he only needs a laptop to conduct health checks, reconfigure or check alarms."
For future expansion to the JumboSwitch network, Keith noted that NIPCO will add more cameras (Ethernet cards) and will consider adding VoIP cards for emergency communications during violent thunderstorms, tornados and ice storms.
"During bad storms the end users start pounding us with telephone calls," said Keith. "They don't realize that they need to call the local co-ops instead of us. This creates a problem because they tie up all of our incoming lines and the co-ops can’t get through to our Control Center. A JumboSwitch VoIP card would allow us to establish direct 'hot phone' lines."
Although NIPCO's JumboSwitch network has only been online for a couple of months, Keith is excited about future possibilities.
"I expect the JumboSwitch platform will provide communications for NIPCO's core utility business and additional services, like T1, to customers for many years to come," he said.
Driven by the need to deliver hydropower from the Missouri River dams, Northwest Iowa Power Cooperative was formed in 1949 by the rural electric cooperatives. NIPCO and its member cooperatives serve the electric needs of homes, farms, communities, business and industry in 13 counties of western Iowa.