bonding.txt

Linux Kernel 2.6.9 for OMAP1710

		from the ARP. When the ARP Reply arrives from the client,
		its hw address is retrieved and the bonding driver
		initiates an ARP reply to this client assigning it to one
		of the slaves in the bond. A problematic outcome of using
		ARP negotiation for balancing is that each time that an ARP
		request is broadcasted it uses the hw address of the
		bond. Hence, clients learn the hw address of the bond and
		the balancing of receive traffic collapses to the current
		salve. This is handled by sending updates (ARP Replies) to
		all the clients with their assigned hw address such that
		the traffic is redistributed. Receive traffic is also
		redistributed when a new slave is added to the bond and
		when an inactive slave is re-activated. The receive load is
		distributed sequentially (round robin) among the group of
		highest speed slaves in the bond.

		When a link is reconnected or a new slave joins the bond
		the receive traffic is redistributed among all active
		slaves in the bond by intiating ARP Replies with the
		selected mac address to each of the clients. The updelay
		modeprobe parameter must be set to a value equal or greater
		than the switch's forwarding delay so that the ARP Replies
		sent to the clients will not be blocked by the switch.

		Prerequisites:

		1. Ethtool support in the base drivers for retrieving the
		speed of each slave.

		2. Base driver support for setting the hw address of a
		device also when it is open. This is required so that there
		will always be one slave in the team using the bond hw
		address (the curr_active_slave) while having a unique hw
		address for each slave in the bond. If the curr_active_slave
		fails it's hw address is swapped with the new curr_active_slave
		that was chosen.

primary

        A string (eth0, eth2, etc) to equate to a primary device. If this
        value is entered, and the device is on-line, it will be used first
        as the output media. Only when this device is off-line, will
        alternate devices be used. Otherwise, once a failover is detected
        and a new default output is chosen, it will remain the output media
        until it too fails. This is useful when one slave was preferred
        over another, i.e. when one slave is 1000Mbps and another is
        100Mbps. If the 1000Mbps slave fails and is later restored, it may
        be preferred the faster slave gracefully become the active slave -
        without deliberately failing the 100Mbps slave. Specifying a
        primary is only valid in active-backup mode.

updelay

        Specifies the delay time in milli-seconds to enable a link after a
        link up status has been detected. This should be a multiple of miimon
        value, otherwise the value will be rounded. The default value is 0.

use_carrier

        Specifies whether or not miimon should use MII or ETHTOOL
        ioctls vs. netif_carrier_ok() to determine the link status.
        The MII or ETHTOOL ioctls are less efficient and utilize a
        deprecated calling sequence within the kernel.  The
        netif_carrier_ok() relies on the device driver to maintain its
        state with netif_carrier_on/off; at this writing, most, but
        not all, device drivers support this facility.

        If bonding insists that the link is up when it should not be,
        it may be that your network device driver does not support
        netif_carrier_on/off.  This is because the default state for
        netif_carrier is "carrier on." In this case, disabling
        use_carrier will cause bonding to revert to the MII / ETHTOOL
        ioctl method to determine the link state.

        A value of 1 enables the use of netif_carrier_ok(), a value of
        0 will use the deprecated MII / ETHTOOL ioctls.  The default
        value is 1.


Configuring Multiple Bonds
==========================

If several bonding interfaces are required, either specify the max_bonds
parameter (described above), or load the driver multiple times.  Using
the max_bonds parameter is less complicated, but has the limitation that
all bonding instances created will have the same options.  Loading the
driver multiple times allows each instance of the driver to have differing
options.

For example, to configure two bonding interfaces, one with mii link
monitoring performed every 100 milliseconds, and one with ARP link
monitoring performed every 200 milliseconds, the /etc/conf.modules should
resemble the following:

alias bond0 bonding
alias bond1 bonding

options bond0 miimon=100
options bond1 -o bonding1 arp_interval=200 arp_ip_target=10.0.0.1

Configuring Multiple ARP Targets
================================

While ARP monitoring can be done with just one target, it can be useful
in a High Availability setup to have several targets to monitor. In the
case of just one target,  the target itself may go down or have a problem
making it unresponsive to ARP requests. Having an additional target (or
several) increases the reliability of the ARP monitoring.

Multiple ARP targets must be seperated by commas as follows:

# example options for ARP monitoring with three targets
alias bond0 bonding
options bond0 arp_interval=60 arp_ip_target=192.168.0.1,192.168.0.3,192.168.0.9

For just a single target the options would resemble:

# example options for ARP monitoring with one target
alias bond0 bonding
options bond0 arp_interval=60 arp_ip_target=192.168.0.100

Potential Problems When Using ARP Monitor
=========================================

1. Driver support

The ARP monitor relies on the network device driver to maintain two
statistics: the last receive time (dev->last_rx), and the last
transmit time (dev->trans_start).  If the network device driver does
not update one or both of these, then the typical result will be that,
upon startup, all links in the bond will immediately be declared down,
and remain that way.  A network monitoring tool (tcpdump, e.g.) will
show ARP requests and replies being sent and received on the bonding
device.

The possible resolutions for this are to (a) fix the device driver, or
(b) discontinue the ARP monitor (using miimon as an alternative, for
example).

2. Adventures in Routing

When bonding is set up with the ARP monitor, it is important that the
slave devices not have routes that supercede routes of the master (or,
generally, not have routes at all).  For example, suppose the bonding
device bond0 has two slaves, eth0 and eth1, and the routing table is
as follows:

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
10.0.0.0        0.0.0.0         255.255.0.0     U        40 0          0 eth0
10.0.0.0        0.0.0.0         255.255.0.0     U        40 0          0 eth1
10.0.0.0        0.0.0.0         255.255.0.0     U        40 0          0 bond0
127.0.0.0       0.0.0.0         255.0.0.0       U        40 0          0 lo

In this case, the ARP monitor (and ARP itself) may become confused,
because ARP requests will be sent on one interface (bond0), but the
corresponding reply will arrive on a different interface (eth0).  This
reply looks to ARP as an unsolicited ARP reply (because ARP matches
replies on an interface basis), and is discarded.  This will likely
still update the receive/transmit times in the driver, but will lose
packets.

The resolution here is simply to insure that slaves do not have routes
of their own, and if for some reason they must, those routes do not
supercede routes of their master.  This should generally be the case,
but unusual configurations or errant manual or automatic static route
additions may cause trouble.

Switch Configuration
====================

While the switch does not need to be configured when the active-backup,
balance-tlb or balance-alb policies (mode=1,5,6) are used, it does need to
be configured for the round-robin, XOR, broadcast, or 802.3ad policies
(mode=0,2,3,4).


Verifying Bond Configuration
============================

1) Bonding information files
----------------------------
The bonding driver information files reside in the /proc/net/bonding directory.

Sample contents of /proc/net/bonding/bond0 after the driver is loaded with
parameters of mode=0 and miimon=1000 is shown below.

        Bonding Mode: load balancing (round-robin)
        Currently Active Slave: eth0
        MII Status: up
        MII Polling Interval (ms): 1000
        Up Delay (ms): 0
        Down Delay (ms): 0

        Slave Interface: eth1
        MII Status: up
        Link Failure Count: 1

        Slave Interface: eth0
        MII Status: up
        Link Failure Count: 1

2) Network verification
-----------------------
The network configuration can be verified using the ifconfig command. In
the example below, the bond0 interface is the master (MASTER) while eth0 and
eth1 are slaves (SLAVE). Notice all slaves of bond0 have the same MAC address
(HWaddr) as bond0 for all modes except TLB and ALB that require a unique MAC
address for each slave.

[root]# /sbin/ifconfig
bond0     Link encap:Ethernet  HWaddr 00:C0:F0:1F:37:B4
          inet addr:XXX.XXX.XXX.YYY  Bcast:XXX.XXX.XXX.255  Mask:255.255.252.0
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:7224794 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3286647 errors:1 dropped:0 overruns:1 carrier:0
          collisions:0 txqueuelen:0

eth0      Link encap:Ethernet  HWaddr 00:C0:F0:1F:37:B4
          inet addr:XXX.XXX.XXX.YYY  Bcast:XXX.XXX.XXX.255  Mask:255.255.252.0
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:3573025 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1643167 errors:1 dropped:0 overruns:1 carrier:0
          collisions:0 txqueuelen:100
          Interrupt:10 Base address:0x1080

eth1      Link encap:Ethernet  HWaddr 00:C0:F0:1F:37:B4
          inet addr:XXX.XXX.XXX.YYY  Bcast:XXX.XXX.XXX.255  Mask:255.255.252.0
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:3651769 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1643480 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100
          Interrupt:9 Base address:0x1400


Frequently Asked Questions
==========================

1.  Is it SMP safe?

	Yes. The old 2.0.xx channel bonding patch was not SMP safe.
	The new driver was designed to be SMP safe from the start.

2.  What type of cards will work with it?

	Any Ethernet type cards (you can even mix cards - a Intel
	EtherExpress PRO/100 and a 3com 3c905b, for example).
	You can even bond together Gigabit Ethernet cards!

3.  How many bonding devices can I have?

	There is no limit.

4.  How many slaves can a bonding device have?

	Limited by the number of network interfaces Linux supports and/or the
	number of network cards you can place in your system.

5.  What happens when a slave link dies?

	If your ethernet cards support MII or ETHTOOL link status monitoring
        and the MII monitoring has been enabled in the driver (see description
        of module parameters), there will be no adverse consequences. This
        release of the bonding driver knows how to get the MII information and
	enables or disables its slaves according to their link status.
	See section on High Availability for additional information.

	For ethernet cards not supporting MII status, the arp_interval and
        arp_ip_target parameters must be specified for bonding to work
        correctly. If packets have not been sent or received during the
        specified arp_interval duration, an ARP request is sent to the
        targets to generate send and receive traffic. If after this
        interval, either the successful send and/or receive count has not
        incremented, the next slave in the sequence will become the active
        slave.

	If neither mii_monitor and arp_interval is configured, the bonding
	driver will not handle this situation very well. The driver will
	continue to send packets but some packets will be lost. Retransmits
	will cause serious degradation of performance (in the case when one
	of two slave links fails, 50% packets will be lost, which is a serious
	problem for both TCP and UDP).

6.  Can bonding be used for High Availability?

        Yes, if you use MII monitoring and ALL your cards support MII link
        status reporting. See section on High Availability for more
        information.

7.  Which switches/systems does it work with?

	In round-robin and XOR mode, it works with systems that support
	trunking:

	* Many Cisco switches and routers (look for EtherChannel support).
	* SunTrunking software.
	* Alteon AceDirector switches / WebOS (use Trunks).
	* BayStack Switches (trunks must be explicitly configured). Stackable
	  models (450) can define trunks between ports on different physical
	  units.
	* Linux bonding, of course !

	In 802.3ad mode, it works with with systems that support IEEE 802.3ad
	Dynamic Link Aggregation:

	* Extreme networks Summit 7i (look for link-aggregation).
	* Many Cisco switches and routers (look for LACP support; this may
	  require an upgrade to your IOS software; LACP support was added
	  by Cisco in late 2002).
	* Foundry Big Iron 4000

        In active-backup, balance-tlb and balance-alb modes, it should work
        with any Layer-II switch.


8.  Where does a bonding device get its MAC address from?

	If not explicitly configured with ifconfig, the MAC address of the
	bonding device is taken from its first slave device. This MAC address
	is then passed to all following slaves and remains persistent (even if
	the the first slave is removed) until the bonding device is brought
	down or reconfigured.

	If you wish to change the MAC address, you can set it with ifconfig:

	  # ifconfig bond0 hw ether 00:11:22:33:44:55

	The MAC address can be also changed by bringing down/up the device
	and then changing its slaves (or their order):

	  # ifconfig bond0 down ; modprobe -r bonding