Northwest Iowa Power Cooperative (NIPCO) owns and operates about 860 miles of transmission lines and 550 miles of fiber optic cable. A generation and transmission cooperative that provides power to seven rural electric cooperatives and a group of 14 independent municipal utilities, it controls a 6,500 square mile electrical grid that stretches two counties wide from the near northwest corner of the state of Iowa to just east of Council Bluffs, Iowa on the south.
The fiber optic cable network provides communications to monitor and control the electric system and provides dark fiber and bandwidth to telecommunications businesses such as independent phone companies and long distance carriers. NIPCO’s subsidiary, NIPCO Development Corporation, is also starting to sell fiber capacity and other services, such as DSO, T1, DS3 and SONET, to resale carriers between Sioux Falls, South Dakota and Omaha, Nebraska.
Recognizing a need to add more resources to its core telecommunications business, NIPCO recently implemented a 435 mile self-healing Ethernet ring using its existing dark fiber capacity and long distance switches.
“We could see a growing need to incorporate the flexibility of an Ethernet network into our telecommunications side of the business,” said Dennis Hill, Vice President of Telecommunications Services. “We had been using T1 lines and Net-to-Net boxes (point to point) for Ethernet service, but demand for bandwidth in a multi-point service configuration had increased during the past year.”
The plans, according to Dennis, called for the Ethernet network to be used for a variety of data and video, and also as a backup network for disaster recovery. “Specifically, we anticipated using IP PTZ (pan, tilt & zoom) cameras, Internet access and file server access across its guest network, MV90 meter reading and spurs of Ethernet spread spectrum microwave for telecommunication locations off of the fiber ring. “We also have a SCADA (Supervisory Control and Data Acquisition) system and the disaster recovery issue is critical to us if we should not be able to inhabit our current power dispatching center. Because of that, we felt we would like to have some portability to the servers by moving them anywhere on our ring and being able to plug into the gigabit Ethernet VLAN ports and still be connected to all points in the system like always.”
SONET vs. Ethernet
Several alternatives were discussed during the planning process, including weighing the benefits of SONET vs. Ethernet.
“We also have available OC48 and OC3 rings and while they would have served us very well for our purposes by installing Ethernet cards in the equipment, they did not provide us the additional bandwidth needed that could result in system latency,” said Dennis. “Had we would have taken the SONET route, we would also be concerned about the major increase in the cost of the project by needing to purchase OC192 optical cards to upgrade from the OC48 or OC3 bandwidth limitations. Everything has its place. SONET works very well for most of our uses; however, to afford us the flexibility that we wanted in this project, the Ethernet was more suited.” Once the decision was made to go with Ethernet, NIPCO conducted an extensive product/vendor search and opted to go with Model TC3820 Redundant Ring Gigabit Managed Ethernet Switches from TC Communications in Irvine, California.
Long Distance Capability
According to Keith Schiltz, Senior Telecommunications Technician, the ability of TC’s Ethernet Switches to handle long distances was especially important in the decision-making process. The longest NIPCO network ring segment between switches is about 40 miles (64km) and the shortest segment is 8.75 miles (14.4km).
“With many miles between regeneration sites, the higher power far-reach lasers were very critical to us,” he said. “We calculated and measured our losses between nodes (Ethernet switches) in the optical network and after path losses were determined, we were able to decide what our link budget would be. We used that as an aid to help us decide what we needed for lasers.”
Although NIPCO opted to use four different high-powered laser optical interfaces (1300nm & 1550nm) on the switches, matching the appropriate interfaces with the required distances, it decided to keep the optics for each switch all the same. For example, even though one side of the link might require 80km optics and the other only 20km., the switch would be fitted with the higher-powered laser. A TC3820’s 80km interface will “talk” to the 20km interface, however the distance is limited to the lower-powered laser. “We looked at mixing far-reach lasers with lower power lasers in one switch but decided that for change out and maintenance purposes we needed to stay consistent with standard lasers on both sides of the switch,” Keith explained.
Keith noted that NIPCO also evaluated prospective Switches on other issues including cost, ease of use, features such as VLAN and self-healing ring capability.
“Obviously, cost was an issue in picking a vendor” explained Keith. “Ease of installation from out of the box also helped to reduce labor costs. TC’s set-up system is basic and very straight forward, so little programming knowledge was needed to set up the VLAN networking as opposed to SONET-based equipment.”
Because NIPCO required maximum reliability, it was imperative that the switches support a “self-healing ring” (often called “redundant ring”) network topology. Each TC3820 Ethernet Switch has dual fiber optic ports. If a fiber cable or device failure occurs, the data path automatically switches over to the secondary fiber optic path to maintain ring network integrity. Switchover times are typically less than 38 msec.
“Network wise, we had several issues with the amount of IP-based equipment that needed to be on the corporate network,” Keith explained. “As a result, we created VLANs for different areas of responsibility such as meter reading, SCADA, security camera, SONET network management, guest network, etc.”
The installation went smoothly, according to Keith. “The system has run well with no infant mortality of the components since we installed the equipment.”
Keith did mention that they encountered one minor VLAN programming issue. “After we ran the equipment for a while and became more acquainted, we discovered that a programming problem resulting in the VLAN’s being tagged or untagged would cause the LANs to be busy with GigE network management duties. We reprogrammed the nodes and it took care of our problems.”
Dennis also mentioned that NIPCO has implemented its own custom device server for handling legacy asynchronous devices.
“There are many asynchronous devices in our system using RS-232 ports. We have been developing and using device servers to convert these legacy devices to Ethernet. Some device servers we have designed and built by using Lantronix or Tibbo self-contained device servers. They are no bigger than the Ethernet connector itself and they provide all of the necessary hardware and software to provide a very compact device server.
“One of the unknown issues is how the device server or, for that matter, the VLAN Ethernet interfaces will hold up to ground rise potential developed during faults in the substations.”
Another network issue that NIPCO is tracking closely is the amount of available bandwidth and effects of heavy traffic.
“We were concerned initially about the full-time use of security cameras on the GigE but, after testing, we were satisfied with the video quality and latency issues. As we continue to load the system down, we will be more aware of its effects and can make a better judgment about loading of the ring and latency issues that may or may not develop.
“SCADA polling is very inefficient use of bandwidth and we feel that as the system grows we may find the heavy bandwidth user may see some packet delay but we do not anticipate it to be a large determent to the user. Our SCADA system can always put up with some variances (within reason) for timing due to latency because our protocol is very short bursts of data and not long blocks.”
Given NIPCO’s robust growth over the past few years, it is no surprise that Dennis is already looking ahead to the future. “We installed 17 units (TC3820) to our fiber ring network and left open the option to add more to the network in the future,” he said. “One example would be to create sub-tended rings to reduce the number of miles in a ring.”