Switch-Assisted Load Balancing w/ Fault Tolerance

Question

While studying for Cisco and Microsoft certifications I read a lot about NIC Teaming, Etherchannels and similar stuff.

I know you can do static teaming in MS Windows Server or creating a port-channel in Cisco or use Dynamic protocols such as LACP and PAgP. But they all offer load-balancing using source or destination MAC address or IP address, a combination XORed together or similar things, right? And it seems there's no way to increase your actual throughput if you have a long long file transfer from one server to another (ex. having 4 GigE bonded together to increase it to 4 Gbps).

So I wonder if it is true? Is there a way to use all the bandwidth of bonded interfaces? I heard from few sources that there's something like SLB (Switch-Assisted Load Balancing w/ Fault Tolerance) which supports full load balancing and allows to use full bandwidth (like 4 Gbps from my example between two servers in one transaction).

I've heard so many contradictory things about that, and I am totally confused. Help me guys!

Answer 1

At layer 2, all load balancing is, at best, done by an XOR or hash of the source and destination MAC, and if you're lucky, it may even read into layer 3 and hash that data too.

At layer 3, however, where we're basically talking about multiple gateways (so, effectively, two physical links with a unique next-hop across each) you can max out the bandwidth across the links IF you're prepared to do per-packet balancing.

Before I go on, per-packet balancing is generally a bad thing due to the fact that it can result in out-of-order packet delivery, this can be especially terrible with TCP connections, but that of course comes down to the implementation and most modern stacks can tolerate this relatively well.

In order to do per-packet balancing, obviously one requirement is that the source and destination IP addresses are not at all on-link to the devices that have the multiple paths since they need to be routed in order for balancing to be possible. Redundancy can be achieved via a routing protocol such as BGP, OSPF, ISIS, RIP, or alternatively, BFD or simple link-state detection.

Finally, there is of course a transport layer solution - protocols like SCTP support connecting to multiple endpoints, and TCP already has drafts in the making that will add options to do similar things. Or... you can just make your application open multiple sockets.

Answer 2

First, let's be realistic about your system performance; you have to ensure that your system can consistently spool the bandwidth you're asking about from disk to the NICs in question (see bottom of answer)...

As long as you're willing to use FTP for your file transfers, you can use a client that supports segmenting the file via FTP REST command for downloads, and a server that supports the FTP COMB command for segmented uploads. By using FTP clients that support segmenting, you can open multiple FTP sessions per file transfer and take advantage of the Layer4 LACP hashing supported by many switch vendors... regardless of whether you're bonding 1GE or 10GE links, segmented transfers accelerate your performance across LACP.

CuteFTP for Windows supports REST transfers and Filezilla supports the COMB command. These are just two of several options you could use.

---

System limits

I don't know what kind of disk you have, but often the single-system limit is disk I/O... there are several points of interest here... from the controller's perspective...

• SATA (revision 1) max xfer: 1.2Gbps after 8b/10b overhead
• SATA (revision 2) max xfer: 2.4Gbps after 8b/10b overhead
• SATA (revision 3) max xfer: 4.8Gbps after 8b/10b overhead
• 4G Fibre Channel max xfer: 3.56Gbps

Another possible limit is the disk speed / spindles / arms... unless you're using an SSD.

Finally you need NICs with sufficient capacity to the motherboard... check how many lanes are on your PCIe NIC. I'm assuming you're on a LAN and that packet loss / congestion won't be an issue. You might want to turn on jumbos if you find that CPU is a limiting factor for your transfers.