News

Energy Department Plugs In 100-GB Ethernet Network

• By Henry Kenyon
• 11/17/2011

With the launch of a national 100 gigabit/sec Ethernet network the Energy Department has just increased the capacity of its scientific data networks. Researchers can now share and move large chunks of data in real time without any latency.

Big science requires big data pipes to move all that information around. This is especially true for the DOE, which maintains the Energy Sciences Network (ESnet) that links its national laboratories and international science partners.

Managed by the Lawrence Berkeley National Laboratory, the Ethernet system was developed under the Advanced Networking Initiative (ANI) and funded with $62 million from the federal Recovery and Reinvestment Act, said Steve Cotter, the ESnet department head at Berkeley Lab.

The additional bandwidth was necessary because the DOE’s existing 10 gigabit/sec network was no longer sufficient to handle the data flow. Cotter noted that since 1990, traffic on ESnet has grown by a factor of 10 roughly every 40 months. “We’re in the age of observation in science with instruments collecting more data than ever before,” he said.

When funds were approved for the project in 2009, DOE began looking at commercial firms developing 100-gigabit systems. However, this was at the height of the economic downturn, which had caused many firms to delay or put off work in high-speed networking technology, Cotter said. This situation was cause for concern. “The DOE was concerned about losing its scientific competitiveness,” he said.

Scientific productivity is based on the ability to move and share data quickly. Because industry was delaying the deployment of 100-gigabit Ethernet technology, it took some persuasion from DOE, in the form of ANI, to move the industry along more quickly than it would have by creating a new market, Cotter said. The department worked with a number of commercial firms, such as Level 3 Communications and LGS Innovations, to push their routing and optical networking technologies to new levels.

The first part of the new network was activated this month and connects three major DOE computing facilities — the National Energy Research Scientific Computing Center in Berkeley, Calif.; the Argonne Leadership Computing Facility near Chicago; the Oak Ridge Leadership Computing Facility in Oak Ridge, Tenn. — and the MAN LAN international exchange point in New York City.

By the end of 2012, ESnet will deploy 100-gigabit connections across the entire DOE network to support scientists across the country. According to DOE, the network is the fastest of its kind in the world.

Another facet of the net Ethernet network is that it was designed from the ground up to measure the energy consumption of all devices connected to it. This will provide scientists with real-time data about the total energy costs associated with data networking, Cotter said.

Networking technology is optimized for speed, but at a cost of energy efficiency. IT systems use a significant amount of the world’s energy, Cotter said. Data collected from the network will be used for IT energy efficiency research. “We need to get a handle on that,” he said.