bonding.txt

Linux Kernel 2.6.9 for OMAP1710

	  # ifconfig bond0 .... up
	  # ifenslave bond0 eth...

	This method will automatically take the address from the next slave
	that will be added.

	To restore your slaves' MAC addresses, you need to detach them
	from the bond (`ifenslave -d bond0 eth0'). The bonding driver will then
	restore the MAC addresses that the slaves had before they were enslaved.

9.  Which transmit polices can be used?

	Round-robin, based on the order of enslaving, the output device
	is selected base on the next available slave. Regardless of
	the source and/or destination of the packet.

	Active-backup policy that ensures that one and only one device will
	transmit at any given moment. Active-backup policy is useful for
	implementing high availability solutions using two hubs (see
	section on High Availability).

	XOR, based on (src hw addr XOR dst hw addr) % slave count. This
	policy selects the same slave for each destination hw address.

	Broadcast policy transmits everything on all slave interfaces.

	802.3ad, based on XOR but distributes traffic among all interfaces
	in the active aggregator.

	Transmit load balancing (balance-tlb) balances the traffic
	according to the current load on each slave. The balancing is
	clients based and the least loaded slave is selected for each new
	client. The load of each slave is calculated relative to its speed
	and enables load balancing in mixed speed teams.

	Adaptive load balancing (balance-alb) uses the Transmit load
	balancing for the transmit load. The receive load is balanced only
	among the group of highest speed active slaves in the bond. The
	load is distributed with round-robin i.e. next available slave in
	the high speed group of active slaves.

High Availability
=================

To implement high availability using the bonding driver, the driver needs to be
compiled as a module, because currently it is the only way to pass parameters
to the driver. This may change in the future.

High availability is achieved by using MII or ETHTOOL status reporting. You
need to verify that all your interfaces support MII or ETHTOOL link status
reporting.  On Linux kernel 2.2.17, all the 100 Mbps capable drivers and
yellowfin gigabit driver support MII. To determine if ETHTOOL link reporting
is available for interface eth0, type "ethtool eth0" and the "Link detected:"
line should contain the correct link status. If your system has an interface
that does not support MII or ETHTOOL status reporting, a failure of its link
will not be detected! A message indicating MII and ETHTOOL is not supported by
a network driver is logged when the bonding driver is loaded with a non-zero
miimon value.

The bonding driver can regularly check all its slaves links using the ETHTOOL
IOCTL (ETHTOOL_GLINK command) or by checking the MII status registers. The
check interval is specified by the module argument "miimon" (MII monitoring).
It takes an integer that represents the checking time in milliseconds. It
should not come to close to (1000/HZ) (10 milli-seconds on i386) because it
may then reduce the system interactivity. A value of 100 seems to be a good
starting point. It means that a dead link will be detected at most 100
milli-seconds after it goes down.

Example:

   # modprobe bonding miimon=100

Or, put the following line in /etc/modprobe.conf:

   options bond0 miimon=100

There are currently two policies for high availability. They are dependent on
whether:

   a) hosts are connected to a single host or switch that support trunking

   b) hosts are connected to several different switches or a single switch that
      does not support trunking


1) High Availability on a single switch or host - load balancing
----------------------------------------------------------------
It is the easiest to set up and to understand. Simply configure the
remote equipment (host or switch) to aggregate traffic over several
ports (Trunk, EtherChannel, etc.) and configure the bonding interfaces.
If the module has been loaded with the proper MII option, it will work
automatically. You can then try to remove and restore different links
and see in your logs what the driver detects. When testing, you may
encounter problems on some buggy switches that disable the trunk for a
long time if all ports in a trunk go down. This is not Linux, but really
the switch (reboot it to ensure).

Example 1 : host to host at twice the speed

          +----------+                          +----------+
          |          |eth0                  eth0|          |
          | Host A   +--------------------------+  Host B  |
          |          +--------------------------+          |
          |          |eth1                  eth1|          |
          +----------+                          +----------+

  On each host :
     # modprobe bonding miimon=100
     # ifconfig bond0 addr
     # ifenslave bond0 eth0 eth1

Example 2 : host to switch at twice the speed

          +----------+                          +----------+
          |          |eth0                 port1|          |
          | Host A   +--------------------------+  switch  |
          |          +--------------------------+          |
          |          |eth1                 port2|          |
          +----------+                          +----------+

  On host A :                             On the switch :
     # modprobe bonding miimon=100           # set up a trunk on port1
     # ifconfig bond0 addr                     and port2
     # ifenslave bond0 eth0 eth1


2) High Availability on two or more switches (or a single switch without
   trunking support)
---------------------------------------------------------------------------
This mode is more problematic because it relies on the fact that there
are multiple ports and the host's MAC address should be visible on one
port only to avoid confusing the switches.

If you need to know which interface is the active one, and which ones are
backup, use ifconfig. All backup interfaces have the NOARP flag set.

To use this mode, pass "mode=1" to the module at load time :

    # modprobe bonding miimon=100 mode=active-backup

	or:

    # modprobe bonding miimon=100 mode=1

Or, put in your /etc/modprobe.conf :

    options bond0 miimon=100 mode=active-backup

Example 1: Using multiple host and multiple switches to build a "no single
point of failure" solution.


                |                                     |
                |port3                           port3|
          +-----+----+                          +-----+----+
          |          |port7       ISL      port7|          |
          | switch A +--------------------------+ switch B |
          |          +--------------------------+          |
          |          |port8                port8|          |
          +----++----+                          +-----++---+
          port2||port1                           port1||port2
               ||             +-------+               ||
               |+-------------+ host1 +---------------+|
               |         eth0 +-------+ eth1           |
               |                                       |
               |              +-------+                |
               +--------------+ host2 +----------------+
                         eth0 +-------+ eth1

In this configuration, there is an ISL - Inter Switch Link (could be a trunk),
several servers (host1, host2 ...) attached to both switches each, and one or
more ports to the outside world (port3...). One and only one slave on each host
is active at a time, while all links are still monitored (the system can
detect a failure of active and backup links).

Each time a host changes its active interface, it sticks to the new one until
it goes down. In this example, the hosts are negligibly affected by the
expiration time of the switches' forwarding tables.

If host1 and host2 have the same functionality and are used in load balancing
by another external mechanism, it is good to have host1's active interface
connected to one switch and host2's to the other. Such system will survive
a failure of a single host, cable, or switch. The worst thing that may happen
in the case of a switch failure is that half of the hosts will be temporarily
unreachable until the other switch expires its tables.

Example 2: Using multiple ethernet cards connected to a switch to configure
           NIC failover (switch is not required to support trunking).


          +----------+                          +----------+
          |          |eth0                 port1|          |
          | Host A   +--------------------------+  switch  |
          |          +--------------------------+          |
          |          |eth1                 port2|          |
          +----------+                          +----------+

  On host A :                                 On the switch :
     # modprobe bonding miimon=100 mode=1     # (optional) minimize the time
     # ifconfig bond0 addr                    # for table expiration
     # ifenslave bond0 eth0 eth1

Each time the host changes its active interface, it sticks to the new one until
it goes down. In this example, the host is strongly affected by the expiration
time of the switch forwarding table.


3) Adapting to your switches' timing
------------------------------------
If your switches take a long time to go into backup mode, it may be
desirable not to activate a backup interface immediately after a link goes
down. It is possible to delay the moment at which a link will be
completely disabled by passing the module parameter "downdelay" (in
milliseconds, must be a multiple of miimon).

When a switch reboots, it is possible that its ports report "link up" status
before they become usable. This could fool a bond device by causing it to
use some ports that are not ready yet. It is possible to delay the moment at
which an active link will be reused by passing the module parameter "updelay"
(in milliseconds, must be a multiple of miimon).

A similar situation can occur when a host re-negotiates a lost link with the
switch (a case of cable replacement).

A special case is when a bonding interface has lost all slave links. Then the
driver will immediately reuse the first link that goes up, even if updelay
parameter was specified. (If there are slave interfaces in the "updelay" state,
the interface that first went into that state will be immediately reused.) This
allows to reduce down-time if the value of updelay has been overestimated.

Examples :

    # modprobe bonding miimon=100 mode=1 downdelay=2000 updelay=5000
    # modprobe bonding miimon=100 mode=balance-rr downdelay=0 updelay=5000


Promiscuous Sniffing notes
==========================

If you wish to bond channels together for a network sniffing
application --- you wish to run tcpdump, or ethereal, or an IDS like
snort, with its input aggregated from multiple interfaces using the
bonding driver --- then you need to handle the Promiscuous interface
setting by hand. Specifically, when you "ifconfing bond0 up" you
must add the promisc flag there; it will be propagated down to the
slave interfaces at ifenslave time; a full example might look like:

   ifconfig bond0 promisc up
   for if in eth1 eth2 ...;do
       ifconfig $if up
       ifenslave bond0 $if
   done
   snort ... -i bond0 ...

Ifenslave also wants to propagate addresses from interface to
interface, appropriately for its design functions in HA and channel
capacity aggregating; but it works fine for unnumbered interfaces;
just ignore all the warnings it emits.


8021q VLAN support
==================

It is possible to configure VLAN devices over a bond interface using the 8021q
driver. However, only packets coming from the 8021q driver and passing through
bonding will be tagged by default. Self generated packets, like bonding's
learning packets or ARP packets generated by either ALB mode or the ARP
monitor mechanism, are tagged internally by bonding itself. As a result,
bonding has to "learn" what VLAN IDs are configured on top of it, and it uses
those IDs to tag self generated packets.

For simplicity reasons, and to support the use of adapters that can do VLAN
hardware acceleration offloding, the bonding interface declares itself as
fully hardware offloaing capable, it gets the add_vid/kill_vid notifications
to gather the necessary information, and it propagates those actions to the
slaves.
In case of mixed adapter types, hardware accelerated tagged packets that should
go through an adapter that is not offloading capable are "un-accelerated" by the
bonding driver so the VLAN tag sits in the regular location.

VLAN interfaces *must* be added on top of a bonding interface only after
enslaving at least one slave. This is because until the first slave is added the
bonding interface has a HW address of 00:00:00:00:00:00, which will be copied by
the VLAN interface when it is created.

Notice that a problem would occur if all slaves are released from a bond that
still has VLAN interfaces on top of it. When later coming to add new slaves, the
bonding interface would get a HW address from the first slave, which might not
match that of the VLAN interfaces. It is recommended that either all VLANs are
removed and then re-added, or to manually set the bonding interface's HW
address so it matches the VLAN's. (Note: changing a VLAN interface's HW address
would set the underlying device -- i.e. the bonding interface -- to promiscouos
mode, which might not be what you want).


Limitations
===========
The main limitations are :
  - only the link status is monitored. If the switch on the other side is
    partially down (e.g. doesn't forward anymore, but the link is OK), the link
    won't be disabled. Another way to check for a dead link could be to count
    incoming frames on a heavily loaded host. This is not applicable to small
    servers, but may be useful when the front switches send multicast
    information on their links (e.g. VRRP), or even health-check the servers.
    Use the arp_interval/arp_ip_target parameters to count incoming/outgoing
    frames.



Resources and Links
===================

Current development on this driver is posted to:
 - http://www.sourceforge.net/projects/bonding/

Donald Becker's Ethernet Drivers and diag programs may be found at :
 - http://www.scyld.com/network/

You will also find a lot of information regarding Ethernet, NWay, MII, etc. at
www.scyld.com.

Patches for 2.2 kernels are at Willy Tarreau's site :
 - http://wtarreau.free.fr/pub/bonding/
 - http://www-miaif.lip6.fr/~tarreau/pub/bonding/

To get latest informations about Linux Kernel development, please consult
the Linux Kernel Mailing List Archives at :
   http://www.ussg.iu.edu/hypermail/linux/kernel/

-- END --