How to Choose Fiber Optics for Mellanox ConnectX Ethernet Adapter Cards?

Mellanox ConnectX Ethernet adapter card provides high-performance networking technologies by utilizing IBTA RoCE technology, delivering efficient RDMA services and scaling in ConnectX-3, 4, 5, 6 EN 10/25/40/50/56/100/200GbE connection. The Mellanox ConnectX EN Ethernet card supports a full suite of software drivers like Microsoft Windows (including Windows 10), Linux distributions, VMware and Citrix XenServer. ConnectX Ethernet adapter card comes in several types for 10/25/40/50/56/100/200GbE network, such as ConnectX-3 EN, ConnectX-3 Pro EN, ConnectX-3 Pro EN 10GBASE-T, ConnectX-4, ConnectX-4 Lx EN, ConnectX-5 EN, ConnectX-6 EN adapter cards. In this article, we present an in-depth network-level performance evaluation of the Mellanox ConnectX Ethernet adapter cards and the supported fiber optic cabling.

Mellanox ConnectX Ethernet Adapter Card

Network Interface Card (NIC), also known as Network Interface Controller, Network Adapter, LAN Adapter or Physical Network Interface. All Mellanox 10/25/40/50/100/200 Gigabit Ethernet adapters deliver high-bandwidth and industry-leading Ethernet connectivity in enterprise data centers, high-performance computing, and embedded environments. From ConnectX-2 to ConnectX-6, Mellanox keep upgrading their Ethernet adapter cards to meet customers requirement.

ConnectX-6 EN—200Gb/s Adapter Card

ConnectX-6 EN 200Gb/s Adapter Card, launched last year, was the world-first 200Gb/s Ethernet network adapter card for Ethernet connectivity, sub-600 ns latency and 200 million messages per second. ConnectX-6 EN, seen in the image, provides two ports of 200Gb/s for Ethernet connectivity. As the first adapter to deliver 200Gb/s throughput, ConnectX-6 is the perfect solution to provide machine learning applications with the levels of performance and scalability that they require.

ConnectX-5 EN—Adapter Supporting 100Gb/s Ethernet

ConnectX-5 just as the image shows, supports two ports of 100Gb/s Ethernet connectivity, sub-700 nanosecond latency, and very high message rate, plus PCIe switch and NVMe over Fabric offloads, providing the highest performance and most flexible solution for the most demanding applications and markets.

Table 1 presents the detailed information about Mellanox ConnectX-5 EN Ethernet Adapter Cards

Ordering Part No.	Description	Speed	Ports	Connectors	Dimensions w/o Bracket
MCX515A-CCAT	100GbE single-port QSFP28, PCIe3.0 x16, tall bracket, ROHS R6	100GbE	1	QSFP28	14.2cm x 6.9cm (Low Profile)
MCX516A-CCAT	100GbE dual-port QSFP28, PCIe3.0 x16, tall bracket, ROHS R6	100GbE	2	QSFP28	14.2cm x 6.9cm (Low Profile)
MCX516A-CDAT	100GbE dual-port QSFP28, PCIe4.0 x16, tall bracket, ROHS R6	100GbE	2	QSFP28	14.2cm x 6.9cm (Low Profile)

ConnectX-4 EN—Adapter Supporting 10G/40G/100Gb/s

ConnectX-4 adapter cards have three different types: ConnectX-4 Lx EN Programmable Adapter Card, ConnectX-4 EN, ConnectX-4 Lx EN. ConnectX-4 Lx EN Programmable Adapter Card is listed in the right image. ConnectX-4 EN network is the single or dual port 100 Gigabit Ethernet adapter cards. ConnectX-4 Lx enables data centers to migrate from 10G to 25G and from 40G to 50G speeds at similar power consumption, cost, and infrastructure needs. If you are into this NIC card, you'd better look through their difference.

ConnectX-3 Pro EN—10/40/56 Gigabit Ethernet Network Interface Cards

ConnectX-3 Gigabit Ethernet interface card is one of the mostly used NIC cards nowadays. This kind of NIC card can support 10/40/56Gb/s Ethernet connectivity with hardware offload engines. The long-term goal at Mellanox ConnectX Gigabit Ethernet adapter card is to allow customers to wire once and switch protocols on the server and switch as required by workloads. Mellanox can presumably charge a premium for such capability, and ConnectX-3 silicon allows Mellanox to create fixed adapters and switches at specific speeds to target specific customer needs and lower price points, too. The image on the right display the ConnectX-3 Pro Programmable Adapter Cards.

For detailed information about ConnectX-3 Pro EN adapter cards, please see the Table 2.

Ordering Part No.	Description	Dimensions w/o Bracket
ConnectX-3 Pro EN Adapter Cards
MCX311A-XCAT	Single 10GbE SFP+	10.2cm x 5.4cm
MCX312A-XCBT	Dual 10GbE SFP+	14.2cm x 6.9cm
MCX313A-BCBT	Single 40/56GbE QSFP	14.2cm x 5.2cm
MCX314A-BCBT	Dual 40/56GbE QSFP	14.2cm x 6.9cm

ConnectX-2—Supporting 10G Ethernet Switches

Mellanox ConnectX-2 EN cards are the older cards for Ethernet only. That means that they will not work in Infiniband networks. In fact, there are still a few examples of OSes without ConnectX-2 support. Prime examples are FreeBSD 9.3 based FreeNAS and NAS4Free versions which did not have built-in support for the cards. The bottom line is that in terms of compatibility, one does need to verify these cards will work.

Many customers do use these in their Windows 10 home workstation for a 10Gb SFP+ back-haul network. Speeds are reliable and excellent. Nowadays, Mellanox does not supply this card, but ebay and Amazon do provide ConnectX-2 cards under $19.

How to Choose the Cables and Optical Modules for ConnectX-3 Ethernet Adapter Cards?

According to Mellanox, ConnectX-3 10G/40G/56G Ethernet adapter cards are interoperable with 10/40 Gb Ethernet switches. Passive copper cables with ESD protection are supported in this NIC card. ConnectX-3 cards can support up to 56Gb/s that can accommodate several fiber cables and optical transceiver types. Take the MCX313A-BCBT as an example, it has single 40GbE QSFP+ port. QSFP+ optical transceivers and DAC cables are feasible in this Ethernet Adapter Card. For the 10G SFP+ to 40G QSFP+ migration, Mellanox QSA (QSFP+ to SFP+) modules are required.

A previous article QSFP+ to SFP+ Adapter (QSA) Module Vs. QSFP+ to SFP+ Breakout Cable provides the detailed information about how to use the QSA module.

DAC Twinax or 10GBASE-T Copper Cable For ConnectX-2 NIC Cards

Mellanox ConnectX EN Gigabit Ethernet Cards can support up to 200Gbps data rate. Each ConnectX adapter card, as mentioned above, supports optical transceivers and modules applied for 10G/25G/40G/100G Ethernet network. ConnectX-2 EN Ethernet card, for example, can be used for 10GbE networks. Except the expensive SFP+ optical transceivers, SFP+ DAC Twinax and 10GBASE-T copper cables are the commonly used  transmission media for 10G data center inter-rack and TOR switching. So how to choose between them?

DAC Twinax Cable

DAC SFP+ Twinax cables integrates SFP+ modules and Twinax cables on the same conduit. It uses the SFP+ optical transceiver modules as connectors (not the real transceiver) on both ends so as to provide a cost-effective and low-power consumption solution for data center interconnection. 10G DAC Twinax cables can be divided into two types: active DAC twinax and passive DAC cables. For distances >5m and <10m, pickup the active version. Otherwise passive direct attach copper cables should be fine.

Fiberstore is one of the best vendors for DAC and other fiber optics. Their packaging and after-sale support for even the smallest orders is excellent. For connecting two ConnectX-2 cards together, you can get one of the SFP+ copper cables.

Ethernet Network Cables

Ethernet Network Cables like Cat5, Cat5e, Cat6 do look alike from the outside and they all use the same connectors as well. However, they have significant differences on the inside. One simple method is that you can find out the type of cable you have by looking at the text printed on the side of the cable. There are other cables possible (like Cat7 and Cat8 cables) but Cat5, Cat5e, Cat6 cables are the most common now.

• CAT5

Category 5 cabling is an older type of network cabling compared with Cat5e and Cat6 cables. Cat5 cables support transfer speeds of 10/100 Mbit/s (Fast Ethernet). Nowadays, many installers treat it as old and obsolete and try not to use it. For those who buy cat5 cables,  they tend to use in older devices such as an older router or switch.

• CAT5e

Category 5 enhanced (Cat5e) cables currently are the most commonly used Ethernet network cables owing to its enhanced transmission data rate and reduced crosstalk. Cat5e cabling supports 1000 Mbit/s and cust down on external and internal crosstalk. It basically means that Cat5e is better at keeping signals on different circuits or channels from interfering with each other.

• CAT6

Category 6 cabling (Cat6) supports 10 Gbit/s speeds with additional crosstalk improvements. If you're purchasing a new cable for future proofing,  it should really be CAT6 or above. That doesn't imply Cat6 cables protect your network from the future or anything, it just means it will keep it up to date for longer when the next products comes along.

Note : Your network speed depends on  the slowest part of your setup. If you want to deploy gigabit ports, gigabit routers, switches,  then please make sure you cabling also supports those speeds as well.

 

Summary

Mellanox is the world-class vendor for selling multi-protocol chips and adapter cards for servers. It provides a wide range of high-speed interconnection solutions including Gigabit Ethernet cards and InfiniBand adapter cards. The above mentioned ConnectX Gigabit Ethernet cards can not be used in InfiniBand networks. ConnectX Mellanox Gigabit Ethernet cards are the cost-effective devices for your network applications. If you do have the Mellanox NIC cards, you can buy the cost-effective DAC and optical transceiver modules from FS.COM to further reduce the total cost of your budget.