I just went through hell in a handbasket trying to get 802.3ad Link Aggregation set up on a Centos 5.2 Xen box. Setting up link aggregation itself isn’t that bad – http://wiki.centos.org/TipsAndTricks/BondingInterfaces for a simple guide (after your managed switch is configd) – but what ever I did, I was unable to get both interfaces simultaneously active.
About the only useful debugging info I got was that the MAC was in use. I was puzzled because as far as I know, link agg takes over the primary MAC and sets that up for both NICs. Furthermore, the same exact hardware was working great on Fedora 10.
bonding: bond0: Warning: the permanent HWaddr of eth0 - [MAC ADDR]- is still in use by bond0. Set the HWaddr of eth0 to a different address to avoid conflicts. bonding: bond0: releasing active interface eth0 bonding: bond0: making interface eth1 the new active one. bonding: bond0: Removing slave eth1 bonding: bond0: releasing active interface eth1 ADDRCONF(NETDEV_UP): bond0: link is not ready bonding: bond0: Adding slave eth0.
Luckily, I stumbled across this bug report. If you scroll down to the last comment, this appears to be a Xen specific issue. By default Xen tries to set its bridge up on eth0, and I assume this prevents the kernel bonding driver from taking over the NIC. By opening up /etc/xen/xend-config.sxp and adding:
(network-script 'network-bridge netdev=bond0')
Xen will bridge to the bond0 interface, and everything will work as expected.
Another trick I had to do was add a start delay to the networking scripts. This is useful if your hardware is crap (cough Broadcom), you need a dhcp lease and it fails, or you are running STP, link aggr., etc. On Fedora, RHEL, and derivitives this is accomplished by adding the NETWORKDELAY directive to /etc/sysconfig/network:
NETWORKING=yes NETWORKDELAY=31
If you need more granularity, you can set delays to specific adapters in the /etc/sysconfig/ifcfg-{x} files with the LINKDELAY directive.
Just a couple of hard lesssons from the trenches, hopefully this will save someone else some time.
No related posts.
I have been intending to set up link bonding with CentOS 5. and Xen for ages. I’m glad you went there first! Do you really get double the throughput as one would expect? And do you get redundancy so if a link fails everything still works?
At my shop we are currently using a proprietary iSCSI target just because it supports MC/S. This iSCSI target is a pain in the butt to deal with and the support stinks. I think the best way to do the same thing is at layer 2 and I have been advocating using the free software iSCSI target (IET) and channel bonding.
Hi Tracy,
I did some basic throughput tests with iperf and tbench. I am using 802.3ad and layer 2+3+4 hashing on both the switch and server, basically the best possible setup. I saw up to 1.83 gbit throughput between two machines set up like this with multiple threads. It also works flawlessly if I yank a cord. Double the throughput, probably not. I was satisfied with the performance (disks are the bottleneck now), so I stopped there. So nothing scientific, but I guess “yes” to both your questions.
I need to do another post – these instructions work for the Xen 3.0 in Centos but if you try and use Xen 3.3 you will get into a lot of trouble, so stay tuned :).
Also, the target I’m looking forward to the most is SCST(http://scst.sourceforge.net/). It kernel level so it should deliver maximum possible performance, but more interestingly it has drivers for qlogic fibre channel and infiniband. You can get old fibre channel gear for almost nothing, so this would be a really nice way to get enterprise class storage on the cheap – FC performance, multipathing, and SAN boot.
There is currently talk on LKML to get SCST merged, so it could be a while before this arrives in the like of RHEL.
Coupled with Btrfs, this may be really compelling in the future though.
Pingback: Xen 3.3 in RHEL/CentOS 5 and more Link Aggregation Fun | Kev009.com
I am using the SCST fibrechannel target at the moment and performance is quite disappointing at only 100MB/s when we should be getting 400MB/s. I am still investigating whether it is something particular to my setup. As for Xen 3.3: I too abandoned most of their network startup scripts from the very beginning and now create my own bridges etc. I first ran into this two years ago. I use AoE instead of iSCSI (faster, simpler) and do diskless CPU servers (check out http://xenaoe.org which is my project site, still needs lots of info but it’s a start) and ran into problems like whenever xen would go to put an interface on a bridge the interface would temporarily be unavailable which means the root fs would go away which blows everything up. So when I PXE boot my cpu servers I do all of the bridge setup in the initrd. Which reminds me, I still need to put my initrd code up on the website…been saying that for months!
Interesting, I played with SCST and saw performance near that, but that is pretty good considering the disks and since I only have 2Gbit FC equipment. Hopefully if it gets pushed up stream your performance problems will be ironed out. I don’t think you can get a better diskless setup than FC because the computer and OS will think it is just a SCSI drive (though this is the same as iSCSI with proper HBAs and such).
On my Xen and VMWare systems (2 total at the moment), I have a simple SCSI RAID-1 for OS and swap, and am storing the VMs on NFS. The NFS filer and VM systems are all using 2 channel 802.3ad Ethernet. Performance is pretty good surprisingly, and the filer’s disks are the weak point at the moment.
For your netboots are you using PXE or Etherboot? I have a bunch of machines that will be diskless nodes running as a Torque grid. NFS root seems to be a bit of a pain, at least getting a “clean” environment. I’ve tried the Redhat system-config-netboot, but it misses obvious things like removing the networking scripts. Do you know if there is anything like a netboot distro of sorts? I feel like I am reinventing the wheel every time I have to fix something :-/.
I will take a look into AoE as well.