100 Gigabit Ethernet

Posted on May 5, 2009 in Knowledge Base

In Computer Networking100 Gigabit Ethernet (or 100GbE) and 40 Gigabit Ethernet (or 40GbE) refers to various technologies for transmitting Ethernet frames at a rates of 100 or 40 gigabits per second (100 to 40 Gbit/s), first defined by the IEEE 802.3ba-2010 standard.

Another variant, 802.3bg, was added in March 2011. There is an active task force 802.3bj working on a four lane backplane and copper 100 Gbit/s standard, and also the 802.3bm task force working on a standard for lower cost 100 Gbit/s optical physical interfaces.

Standards

The IEEE 802.3 working group is concerned with the maintenance and extension of the Ethernet data communications standard. Additions to the 802.3 standard are performed by task forces which are designated by one or two letters. For example the 802.3z task force drafted the original gigabit Ethernet standard.

802.3ba is the designation given to the higher speed Ethernet task force which completed its work to modify the 802.3 standard to support speeds higher than 10 Gbit/s in 2010.

The speeds chosen by 802.3ba were 40 and 100 Gbit/s to support both end-point and link aggregation needs. This was the first time two different Ethernet speeds were specified in a single standard. The decision to include both speeds came from pressure to support the 40 Gbit/s rate for local server applications and the 100 Gbit/s rate for internet backbones. The standard was announced in July 2007 and was ratified on June 17, 2010.

The 40/100 Gigabit Ethernet standards encompass a number of differentEthernet physical layer (PHY) specifications. A networking device may support different PHY types by means of pluggable modules. Optical Modules are not standardized by any official standards body but are in multi-source agreements (MSAs). One agreement that supports 40 and 100 Gigabit Ethernet is the C Form-Factor Pluggable (CFP) MSA which was adopted for distances of 100+ meters. QSFP and CXP connector modules support shorter distances.

The standard supports only full-duplex operation. Other electrical objectives include:

  • Preserve the 802.3 / Ethernet frame format utilizing the 802.3 MAC
  • Preserve minimum and maximum FrameSize of current 802.3 standard
  • Support a bit error ratio (BER) better than or equal to 10−12 at the MAC/PLS service interface
  • Provide appropriate support for OTN
  • Support MAC data rates of 40 and 100 Gbit/s
  • Provide Physical Layer specifications (PHY) for operation over single-mode Optical Fiber (SMF), laser optimized Multi-Mode Optical Fiber (MMF) OM3 and OM4, copper cable assembly, and backplane.

The following nomenclature was used for the physical layers:

Physical layer 40 Gigabit Ethernet 100 Gigabit Ethernet
Backplane 40GBASE-KR4 100GBASE-KP4
Improved Backplane 100GBASE-KR4
Copper cable 40GBASE-CR4 100GBASE-CR10
100 m over OM3 MMF 40GBASE-SR4 100GBASE-SR10
125 m over OM4 MMF
10 km over SMF 40GBASE-LR4 100GBASE-LR4
40 km over SMF 100GBASE-ER4
2 km over SMF, serial 40GBASE-FR

The 100 m laser optimized Multi-Mode Fiber (OM3) objective was met by parallel ribbon cable with 850 nm wavelength 10GBASE-SR like optics (40GBASE-SR4 and 100GBASE-SR10). The backplane objective with 4 lanes of 10GBASE-KR type PHYs (40GBASE-KR4). The copper cable objective is met with 4 or 10 differential lanes using SFF-8642 and SFF-8436 connectors. The 10 and 40 km 100 Gbit/s objectives with four wavelengths (around 1310 nm) of 25 Gbit/s optics (100GBASE-LR4 and 100GBASE-ER4) and the 10 km 40 Gbit/s objective with four wavelengths (around 1310 nm) of 10 Gbit/s optics (40GBASE-LR4).

In January 2010 another IEEE project authorization started a task force to define a 40 Gbit/s serial Single-Mode Optical Fiber standard (40GBASE-FR). This was approved as standard 802.3bg in March 2011. It used 1550 nm optics, had a reach of 2 km and was capable of receiving 1550 nm and 1310 nm wavelengths of light. The capability to receive 1310 nm light allows it to inter-operate with a longer reach 1310 nm PHY should one ever be developed. 1550 nm was chosen as the wavelength for 802.3bg transmission to make it compatible with existing test equipment and infrastructure.

In December 2010, a 10×10 Multi Source Agreement (10×10 MSA) began to define an optical Physical Medium Dependent (PMD) sublayer and establish compatible sources of low-cost, low-power, pluggable optical transceivers based on 10 optical lanes at 10 gigabits/second each. The 10×10 MSA was intended as a lower cost alternative to 100GBASE-LR4 for applications which do not require a link length longer than 2 km. It was intended for use with standard single mode G.652.C/D type low water peak cable with ten wavelengths ranging from 1523 to 1595 nm. The founding members were Google, Brocade Communications, JDSU and Santur. Other member companies of the 10×10 MSA included MRV, Enablence, Cyoptics, AFOP, OPLINK, Hitachi Cable America, AMS-IX, EXFO, Huawei, Kotura, Facebook and Effdon when the 2 km specification was announced in March 2011. The 10X10 MSA modules were intended to be the same size as the C Form-factor Pluggable specifications.

There are currently two projects in 802.3 underway to specify additional PHYs. The 802.3bj task force is working to produce 100 Gbit/s 4x25G PHYs for backplane and twin-ax cable (100GBASE-KR4, 100GBASE-KP4 and 100GBASE-CR4). The 802.3bm task force is working to produce lower cost optical PHYs. The detailed objectives for these projects can be found on the 802.3 website.

100G Port Types

100GBASE-CR10

100GBASE-CR10 (“copper”) is a port type for twin-ax copper cable. Its Physical Coding Sublayer 64b/66b PCS is defined in IEEE 802.3 Clause 82 and its Physical Medium Dependent PMD in Clause 85. It uses ten lanes of twin-ax cable delivering serialized data at a rate of 10.3125 Gbit/s per lane.

100GBASE-CR4

100GBASE-CR4 (“copper”) is a port type for twin-ax copper cable. Its Physical Coding Sublayer 64b/66b PCS is defined in IEEE 802.3 Clause 82 and its Physical Medium Dependent PMD in Clause 92 of 802.3bj. It uses four lanes of twin-ax cable delivering serialized data at a rate of 25.78125 Gbit/s per lane.

100GBASE-SR10

100GBASE-SR10 (“short range”) is a port type for multi-mode fiber and uses 850 nm lasers. Its Physical Coding Sublayer 64b/66b PCS is defined in IEEE 802.3 Clause 82 and its Physical Medium Dependent PMD in Clause 86. It uses ten lanes of multi-mode fiber delivering serialized data at a rate of 10.3125 Gbit/s per lane.

100GBASE-SR4

100GBASE-SR4 (“short range”) is a port type for multi-mode fiber being defined in P802.3bm and uses 850 nm lasers. Its Physical Coding Sublayer 64b/66b PCS is defined in IEEE 802.3 Clause 82 and its Physical Medium Dependent PMD in Clause 95. It uses four lanes of multi-mode fiber delivering serialized RS-FEC encoded data at a rate of 25.78125 Gbit/s per lane.

100GBASE-LR4

100GBASE-LR4 (“long range”) is a port type for Single-Mode Fiber and uses four lasers using four wavelengths around 1300 nm. Its Physical Coding Sublayer 64b/66b PCS is defined in IEEE 802.3 Clause 82 and its Physical Medium Dependent PMD in Clause 88. Each wavelength carries data at a rate of 25.78125 Gbit/s.

100GBASE-ER4

100GBASE-ER4 (“extended range”) is a port type for single-mode fiber and uses four lasers using four wavelengths around 1300 nm. Its Physical Coding Sublayer 64b/66b PCS is defined in IEEE 802.3 Clause 82 and itsPhysical Medium Dependent PMD in Clause 88. Each wavelength carries data at a rate of 25.78125 Gbit/s.

100GBASE-KR4

100GBASE-KR4 is a port type for backplanes. Its Physical Coding Sublayer 64b/66b PCS is defined in IEEE 802.3 Clause 82 and its Physical Medium Dependent PMD in Clause 93 of 802.3bj. It delivers Reed Solomon encoded serialized data at a rate of 25.78125 Gbit/s per lane over four lanes of up to one meter of backplane. The Reed Solomon forward error correction is defined in Clause 91.

100GBASE-KP4

100GBASE-KR4 is a port type for backplanes. Its Physical Coding Sublayer 64b/66b PCS is defined in IEEE 802.3 Clause 82 and its Physical Medium Dependent PMD in Clause 94. The data is further encoded by the Reed Solomon forward error correction is defined in Clause 91 and the four level amplitude modulation is defined in Clause 94 of 802.3bj. 100GBASE-KP4 uses more power than 100GBASE-KR4 but is designed to work on lower cost and legacy backplanes.

40G Port Types

40GBASE-CR4

100GBASE-CR4 (“copper”) is a port type for twin-ax copper cable. Its Physical Coding Sublayer 64b/66b PCS is defined in IEEE 802.3 Clause 82 and its Physical Medium Dependent PMD in Clause 85. It uses four lanes of twin-ax cable delivering serialized data at a rate of 10.3125 Gbit/s per lane.

40GBASE-KR4

100GBASE-KR4 is a port type for backplanes. Its Physical Coding Sublayer 64b/66b PCS is defined in IEEE 802.3 Clause 82 and its Physical Medium Dependent PMD in Clause 84. It uses four lanes of backplane delivering serialized data at a rate of 10.3125 Gbit/s per lane.

40GBASE-SR4

40GBASE-SR4 (“short range”) is a port type for multi-mode fiber and uses 850 nm lasers. Its Physical Coding Sublayer 64b/66b PCS is defined in IEEE 802.3 Clause 82 and its Physical Medium Dependent PMD in Clause 86. It uses four lanes of multi-mode fiber delivering serialized data at a rate of 10.3125 Gbit/s per lane.

40GBASE-LR4

40GBASE-LR4 (“long range”) is a port type for single-mode fiber and uses 1300 nm lasers. Its Physical Coding Sublayer 64b/66b PCS is defined in IEEE 802.3 Clause 82 and its Physical Medium Dependent PMD in Clause 87. It uses four wavelengths delivering serialized data at a rate of 10.3125 Gbit/s per wavelength.

40GBASE-ER4

40GBASE-ER4 (“extended range”) is a port type for single-mode fiber being defined in P802.3bm and uses 1300 nm lasers. Its Physical Coding Sublayer 64b/66b PCS is defined in IEEE 802.3 Clause 82 and its Physical Medium Dependent PMD in Clause 87. It uses four wavelengths delivering serialized data at a rate of 10.3125 Gbit/s per wavelength.

40GBASE-FR

40GBASE-FR is a port type for single-mode fiber. Its Physical Coding Sublayer 64b/66b PCS is defined in IEEE 802.3 Clause 82 and its Physical Medium Dependent PMD in Clause 89. It uses 1550 nm optics, has a reach of 2 km and is capable of receiving 1550 nm and 1310 nm wavelengths of light. The capability to receive 1310 nm light allows it to inter-operate with a longer reach 1310 nm PHY should one ever be developed. 1550 nm was chosen as the wavelength transmission to make it compatible with existing test equipment and infrastructure.

Chip-to-Chip/Chip-to-Module Interfaces

CAUI

CAUI is a 100 Gbit/s 10 lane electrical interface defined in 802.3ba.

CAUI-4

CAUI-4 is a 100 Gbit/s 4 lane electrical interface being defined in 802.3bm.

Connectors

QSFP+

The QSFP+ connector is specified for use with the 40GBASE-CR4 PHY, see Figure 85–20 in the 802.3 spec.

MPO

The 40GBASE-SR4 and 100GBASE-SR10 PHYs use the Multiple-Fiber Push-On/Pull-off (MPO) connector, see subclause 86.10.3.3 of the 802.3 spec.

100G Optical Module standards

The CFP Multi-Source Agreement (MSA) defines hot-pluggable Optical Transceiver form factors to enable 40Gb/s and 100Gb/s applications.

CFP and CFP2 modules use the 10-lane CAUI electrical interface. CFP4 will use the CAUI-4 electrical interface.

There are also CXP and HD module standards.

Products

Backplane

NetLogic Microsystems announced backplane modules in October 2010.

Copper cables

Quellan announced a test board in 2009.

Multimode Fiber

In 2009, Mellanox and Reflex Photonics announced modules based on the CFP agreement.

Single Mode Fiber

Finisar, Sumitomo Electric Industries, and OpNext all demonstrated singlemode 40 or 100 Gbit/s Ethernet modules based on the C Form-factor Pluggable agreement at the European Conference and Exhibition on Optical Communication in 2009.

Compatibility

Optical fiber IEEE 802.3ba implementations were not compatible with the numerous 40 Gbit/s and 100 Gbit/s line rate transport systems because they had different optical layer and modulation formats. In particular, existing 40 Gbit/s transport solutions that used dense wavelength-division multiplexing to pack four 10 Gbit/s signals into one optical medium were not compatible with the IEEE 802.3ba standard, which used either coarse WDM in 1310 nm wavelength region with four 25 Gbit/s or four 10 Gbit/s channels, or parallel optics with four or ten Optical Fibers per direction.

Test and measurement

Ixia developed Physical Coding Sublayer Lanes and demonstrated a working 100GbE link through a test setup atNXTcomm in June 2008. Ixia announced test equipment in November 2008.

Discovery Semiconductors introduced optoelectronics converters for 100 Gbit/s testing of the 10 km and 40 km Ethernet standards in February 2009.

JDS Uniphase introduced test and measurement products for 40 and 100 Gbit/s Ethernet in August 2009.

Spirent Communications introduced test and measurement products in September 2009.

EXFO demonstrated interoperability in January 2010.

Xena Networks demonstrated test equipment at the Technical University of Denmark in January 2011.

Commercial trials and deployments

Unlike the “race to 10Gbps” that was driven by the imminent needs to address growth pains of the Internet in late 1990s, customer interest in 100 Gbit/s technologies was mostly driven by economic factors. Among those, the common reasons to adopt the higher speeds were:

  • to reduce the number of optical wavelengths (“lambdas”) used and the need to light new fiber
  • to utilize bandwidth more efficiently than 10 Gbit/s link aggregate groups
  • to provide cheaper wholesale, internet peering and data center interconnect connectivity
  • to skip the relatively expensive 40 Gbit/s technology and move directly from 10 Gbit/s to 100 Gbit/s

Considering that 100GbE technology is natively compatible with Optical Transport Network (OTN) hierarchy and there is no separate adaptation for SONET/SDH and Ethernet networks, it was widely believed that 100GbE technology adoption will be driven by products in all network layers, from transport systems to edge routers and datacenter switches. Nevertheless, in 2011 components for 100GE networks were expensive and most vendors entering this market relied on internal R&D projects and extensive cooperation with other companies.

Optical transport systems

Optical signal transmission over a nonlinear medium is principally an analog design problem. As such, it has evolved slower than digital circuit lithography (which generally progressed in step with Moore’s law.) This explains why 10 Gbit/s transport systems existed since the mid-1990s, while the first forays into 100 Gbit/s transmission happened about 15 years later – a 10x speed increase over 15 years is far slower than the 2x speed per 1.5 years typically cited for Moore’s law. Nevertheless, by August 2011 at least five firms (Ciena, Alcatel-Lucent, MRV, ADVA Optical and Huawei) made customer announcements for 100 Gbit/s transport systems– with varying degrees of capabilities. Although vendors claimed that 100 Gbit/s lightpaths could use existing analog optical infrastructure, in practice deployment of new, high-speed technology was tightly controlled and extensive interoperability tests were required before moving them into service.

Products

Designing routers or switches supporting 100 Gbit/s interfaces is difficult. One reason is the need to process a 100 Gbit/s stream of packets at line rate without reordering within IP/MPLS microflows. As of 2011, most components in the 100 Gbit/s packet processing path (PHY chips, NPUs, memories) were not readily available off-the-shelf or require extensive qualification and co-design. Another problem is related to the low-output production of 100 Gbit/s optical components, which were also not easily available – especially in pluggable, long-reach or tunable laser flavors.

Alcatel-Lucent

In November 2007 Alcatel-Lucent held the first field trial of 100 Gbit/s optical transmission. Completed over a live, in-service 504-km portion of the Verizon network, it connected the Florida cities of Tampa and Miami. 100GbE interfaces for the 7450 ESS/7750 SR service routing platform were first announced in June 2009, with field trials with Verizon, T-Systems and Portugal Telecom following in June–September 2010. In September 2009 Alcatel-Lucent combined the 100G capabilities of its IP routing and optical transport portfolio in an integrated solution called Converged Backbone Transformation.

In June 2011, Alcatel-Lucent announced a packet processing architecture called FP3, advertised for 400 Gbit/s rates. In May 2012, Alcatel-Lucent announced a router based on the FP3.

Arista

Arista Networks announced its 7500E switch with up to 96 100GbE ports in April 2013.

Brocade

In September 2010, Brocade Communications Systems announced their first 100GbE products based on the former Foundry Networks hardware (MLXe). In June 2011, the new product went live at AMS-IX traffic exchange point in Amsterdam.

Cisco

Cisco Systems and Comcast announced their 100GbE trials in June 2008, however it is doubtful this transmission could approach 100 Gbit/s speeds when using a 40 Gbit/s per slot CRS-1 platform for packet processing. Cisco’s first deployment of 100GbE at AT&T and Comcast occurred in April 2011. Later in the same year, Cisco tested the 100GbE interface between CRS-3 and a new generation of their ASR9K edge router.

Extreme Networks

Extreme Networks announced its first 100GbE product on November 13, 2012, a four-port 100GbE module for the BlackDiamond X8 core switch.

Huawei

In October 2008, the Chinese vendor Huawei presented their first 100GbE interface for their NE5000e router. In September 2009, Huawei also demonstrated an end-to-end 100&Gbit/s link. It was mentioned that Huawei’s products had the self-developed NPU “Solar 2.0 PFE2A” onboard and was using pluggable optics in CFP form-factor. In a mid-2010 product brief, the NE5000e linecards were given commercial name LPUF-100 and credited with using two Solar-2.0 NPUs per 100GbE port in opposite (ingress/egress) configuration. Nevertheless, in October 2010, the company referenced shipments of NE5000e to Russian cell operator “Megafon” as “40Gbps/slot” solution, with “scalability up to” 100Gbit/s.

In April 2011, Huawei announced that the NE5000e was updated to carry 2x100GbE interfaces per slot using LPU-200 linecards. In a related solution brief, Huawei reported 120 thousand Solar 1.0 integrated circuits shipped to customers, but no Solar 2.0 numbers were given. Following the August 2011 trial in Russia, Huawei reported paying 100 Gbit/ DWDM customers, but no 100GbE shipments on NE5000e.

Juniper

Juniper Networks announced 100GbE for its T-series routers in June 2009. The 1x100GbE option followed in Nov 2010, when a joint press release with academic backbone network Internet2 marked the first production 100GbE interfaces going live in real network. Later in the same year, Juniper demonstrated 100GbE operation between core (T-series) and edge (MX 3D) routers. Juniper, in March 2011, announced first shipments of 100GbE interfaces to a major North American service provider (Verizon). In April 2011, Juniper deployed a 100GbE system to the UK network operator JANET. In July 2011, Juniper announced 100GbE with Australian ISP iiNet on their T1600 routing platform.

In March 2012, Juniper Networks started shipping the MPC3E line card for the MX router, a 10GbE CFP MIC, and a 100GbE LR4 CFP optics. In Spring 2013, Juniper Networks announced the availability of the MPC4E line card for the MX router that includes 2 100GnE CFP slots and 8 10GnE SFP+ interfaces.

Dell

Dell’s Force10 switches support 40 Gbit/s interfaces. These 40 Gbit/s fiber-optical interfaces using QSFP+ Transceivers can be found on the Z9000 distributed core switches, S4810 and S4820 as well as the blade-switches MXL and the IO-Aggregator. The Dell PowerConnect 8100 series switches also offer 40 Gbit/s QSFP+ interfaces.

Chelsio

In June 2013, Chelsio Communications, announced 40 Gbit/s Ethernet network adapters based on the fifth generation of its Terminator architecture.