bonding.txt

Linux Kernel 2.6.9 for OMAP1710

                   Linux Ethernet Bonding Driver mini-howto

Initial release : Thomas Davis <tadavis at lbl.gov>
Corrections, HA extensions : 2000/10/03-15 :
  - Willy Tarreau <willy at meta-x.org>
  - Constantine Gavrilov <const-g at xpert.com>
  - Chad N. Tindel <ctindel at ieee dot org>
  - Janice Girouard <girouard at us dot ibm dot com>
  - Jay Vosburgh <fubar at us dot ibm dot com>

Note :
------
The bonding driver originally came from Donald Becker's beowulf patches for
kernel 2.0. It has changed quite a bit since, and the original tools from
extreme-linux and beowulf sites will not work with this version of the driver.

For new versions of the driver, patches for older kernels and the updated
userspace tools, please follow the links at the end of this file.


Table of Contents
=================

Installation
Bond Configuration
Module Parameters
Configuring Multiple Bonds
Switch Configuration
Verifying Bond Configuration
Frequently Asked Questions
High Availability
Promiscuous Sniffing notes
8021q VLAN support
Limitations
Resources and Links


Installation
============

1) Build kernel with the bonding driver
---------------------------------------
For the latest version of the bonding driver, use kernel 2.4.12 or above
(otherwise you will need to apply a patch).

Configure kernel with `make menuconfig/xconfig/config', and select "Bonding
driver support" in the "Network device support" section. It is recommended
to configure the driver as module since it is currently the only way to
pass parameters to the driver and configure more than one bonding device.

Build and install the new kernel and modules.

2) Get and install the userspace tools
--------------------------------------
This version of the bonding driver requires updated ifenslave program. The
original one from extreme-linux and beowulf will not work. Kernels 2.4.12
and above include the updated version of ifenslave.c in
Documentation/networking directory. For older kernels, please follow the
links at the end of this file.

IMPORTANT!!!  If you are running on Redhat 7.1 or greater, you need
to be careful because /usr/include/linux is no longer a symbolic link
to /usr/src/linux/include/linux.  If you build ifenslave while this is
true, ifenslave will appear to succeed but your bond won't work.  The purpose
of the -I option on the ifenslave compile line is to make sure it uses
/usr/src/linux/include/linux/if_bonding.h instead of the version from
/usr/include/linux.

To install ifenslave.c, do:
    # gcc -Wall -Wstrict-prototypes -O -I/usr/src/linux/include ifenslave.c -o ifenslave
    # cp ifenslave /sbin/ifenslave


Bond Configuration
==================

You will need to add at least the following line to /etc/modprobe.conf
so the bonding driver will automatically load when the bond0 interface is
configured. Refer to the modprobe.conf manual page for specific modprobe.conf
syntax details. The Module Parameters section of this document describes each
bonding driver parameter.

	alias bond0 bonding

Use standard distribution techniques to define the bond0 network interface. For
example, on modern Red Hat distributions, create an ifcfg-bond0 file in
the /etc/sysconfig/network-scripts directory that resembles the following:

DEVICE=bond0
IPADDR=192.168.1.1
NETMASK=255.255.255.0
NETWORK=192.168.1.0
BROADCAST=192.168.1.255
ONBOOT=yes
BOOTPROTO=none
USERCTL=no

(use appropriate values for your network above)

All interfaces that are part of a bond should have SLAVE and MASTER
definitions. For example, in the case of Red Hat, if you wish to make eth0 and
eth1 a part of the bonding interface bond0, their config files (ifcfg-eth0 and
ifcfg-eth1) should resemble the following:

DEVICE=eth0
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none

Use DEVICE=eth1 in the ifcfg-eth1 config file. If you configure a second
bonding interface (bond1), use MASTER=bond1 in the config file to make the
network interface be a slave of bond1.

Restart the networking subsystem or just bring up the bonding device if your
administration tools allow it. Otherwise, reboot. On Red Hat distros you can
issue `ifup bond0' or `/etc/rc.d/init.d/network restart'.

If the administration tools of your distribution do not support
master/slave notation in configuring network interfaces, you will need to
manually configure the bonding device with the following commands:

    # /sbin/ifconfig bond0 192.168.1.1 netmask 255.255.255.0 \
      broadcast 192.168.1.255 up

    # /sbin/ifenslave bond0 eth0
    # /sbin/ifenslave bond0 eth1

(use appropriate values for your network above)

You can then create a script containing these commands and place it in the
appropriate rc directory.

If you specifically need all network drivers loaded before the bonding driver,
adding the following line to modprobe.conf will cause the network driver for
eth0 and eth1 to be loaded before the bonding driver.

install bond0 /sbin/modprobe -a eth0 eth1 && /sbin/modprobe bonding

Be careful not to reference bond0 itself at the end of the line, or modprobe
will die in an endless recursive loop.

If running SNMP agents, the bonding driver should be loaded before any network
drivers participating in a bond. This requirement is due to the the interface
index (ipAdEntIfIndex) being associated to the first interface found with a
given IP address. That is, there is only one ipAdEntIfIndex for each IP
address. For example, if eth0 and eth1 are slaves of bond0 and the driver for
eth0 is loaded before the bonding driver, the interface for the IP address
will be associated with the eth0 interface. This configuration is shown below,
the IP address 192.168.1.1 has an interface index of 2 which indexes to eth0
in the ifDescr table (ifDescr.2).

     interfaces.ifTable.ifEntry.ifDescr.1 = lo
     interfaces.ifTable.ifEntry.ifDescr.2 = eth0
     interfaces.ifTable.ifEntry.ifDescr.3 = eth1
     interfaces.ifTable.ifEntry.ifDescr.4 = eth2
     interfaces.ifTable.ifEntry.ifDescr.5 = eth3
     interfaces.ifTable.ifEntry.ifDescr.6 = bond0
     ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.10.10.10 = 5
     ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.192.168.1.1 = 2
     ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.74.20.94 = 4
     ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.127.0.0.1 = 1

This problem is avoided by loading the bonding driver before any network
drivers participating in a bond. Below is an example of loading the bonding
driver first, the IP address 192.168.1.1 is correctly associated with
ifDescr.2.

     interfaces.ifTable.ifEntry.ifDescr.1 = lo
     interfaces.ifTable.ifEntry.ifDescr.2 = bond0
     interfaces.ifTable.ifEntry.ifDescr.3 = eth0
     interfaces.ifTable.ifEntry.ifDescr.4 = eth1
     interfaces.ifTable.ifEntry.ifDescr.5 = eth2
     interfaces.ifTable.ifEntry.ifDescr.6 = eth3
     ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.10.10.10 = 6
     ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.192.168.1.1 = 2
     ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.74.20.94 = 5
     ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.127.0.0.1 = 1

While some distributions may not report the interface name in ifDescr,
the association between the IP address and IfIndex remains and SNMP
functions such as Interface_Scan_Next will report that association.


Module Parameters
=================

Optional parameters for the bonding driver can be supplied as command line
arguments to the insmod command. Typically, these parameters are specified in
the file /etc/modprobe.conf (see the manual page for modprobe.conf). The
available bonding driver parameters are listed below. If a parameter is not
specified the default value is used. When initially configuring a bond, it
is recommended "tail -f /var/log/messages" be run in a separate window to
watch for bonding driver error messages.

It is critical that either the miimon or arp_interval and arp_ip_target
parameters be specified, otherwise serious network degradation will occur
during link failures.

arp_interval

        Specifies the ARP monitoring frequency in milli-seconds.
        If ARP monitoring is used in a load-balancing mode (mode 0 or 2), the
        switch should be configured in a mode that evenly distributes packets
        across all links - such as round-robin. If the switch is configured to
        distribute the packets in an XOR fashion, all replies from the ARP
        targets will be received on the same link which could cause the other
        team members to fail. ARP monitoring should not be used in conjunction
        with miimon. A value of 0 disables ARP monitoring. The default value
        is 0.

arp_ip_target

	Specifies the ip addresses to use when arp_interval is > 0. These
	are the targets of the ARP request sent to determine the health of
	the link to the targets. Specify these values in ddd.ddd.ddd.ddd
	format. Multiple ip adresses must be seperated by a comma. At least
	one ip address needs to be given for ARP monitoring to work. The
	maximum number of targets that can be specified is set at 16.

downdelay

        Specifies the delay time in milli-seconds to disable a link after a
        link failure has been detected. This should be a multiple of miimon
        value, otherwise the value will be rounded. The default value is 0.

lacp_rate

        Option specifying the rate in which we'll ask our link partner to
	transmit LACPDU packets in 802.3ad mode.  Possible values are:

	slow or 0
		Request partner to transmit LACPDUs every 30 seconds (default)

	fast or 1
		Request partner to transmit LACPDUs every 1 second

max_bonds

	Specifies the number of bonding devices to create for this
	instance of the bonding driver.  E.g., if max_bonds is 3, and
	the bonding driver is not already loaded, then bond0, bond1
	and bond2 will be created.  The default value is 1.

miimon

        Specifies the frequency in milli-seconds that MII link monitoring
        will occur. A value of zero disables MII link monitoring. A value
        of 100 is a good starting point. See High Availability section for
        additional information. The default value is 0.

mode

	Specifies one of the bonding policies. The default is
	round-robin (balance-rr).  Possible values are (you can use
	either the text or numeric option):

	balance-rr or 0

		Round-robin policy: Transmit in a sequential order
		from the first available slave through the last. This
		mode provides load balancing and fault tolerance.

	active-backup or 1

		Active-backup policy: Only one slave in the bond is
		active. A different slave becomes active if, and only
		if, the active slave fails. The bond's MAC address is
		externally visible on only one port (network adapter)
		to avoid confusing the switch.  This mode provides
		fault tolerance.

	balance-xor or 2

		XOR policy: Transmit based on [(source MAC address
		XOR'd with destination MAC address) modula slave
		count]. This selects the same slave for each
		destination MAC address. This mode provides load
		balancing and fault tolerance.

	broadcast or 3

		Broadcast policy: transmits everything on all slave
		interfaces. This mode provides fault tolerance.

	802.3ad or 4

		IEEE 802.3ad Dynamic link aggregation. Creates aggregation
		groups that share the same speed and duplex settings.
		Transmits and receives on all slaves in the active
		aggregator.

		Pre-requisites:

		1. Ethtool support in the base drivers for retrieving the
		speed and duplex of each slave.

		2. A switch that supports IEEE 802.3ad Dynamic link
		aggregation.

	balance-tlb or 5

		Adaptive transmit load balancing: channel bonding that does
		not require any special switch support. The outgoing
		traffic is distributed according to the current load
		(computed relative to the speed) on each slave. Incoming
		traffic is received by the current slave. If the receiving
		slave fails, another slave takes over the MAC address of
		the failed receiving slave.

		Prerequisite:

		Ethtool support in the base drivers for retrieving the
		speed of each slave.

	balance-alb or 6

		Adaptive load balancing: includes balance-tlb + receive
		load balancing (rlb) for IPV4 traffic and does not require
		any special switch support. The receive load balancing is
		achieved by ARP negotiation. The bonding driver intercepts
		the ARP Replies sent by the server on their way out and
		overwrites the src hw address with the unique hw address of
		one of the slaves in the bond such that different clients
		use different hw addresses for the server.

		Receive traffic from connections created by the server is
		also balanced. When the server sends an ARP Request the
		bonding driver copies and saves the client's IP information