bonding.txt

《嵌入式系统设计与实例开发实验教材二源码》Linux内核移植与编译实验

8.  Where does a bonding device get its MAC address from?

	If not explicitly configured with ifconfig, the MAC address of the
	bonding device is taken from its first slave device. This MAC address
	is then passed to all following slaves and remains persistent (even if
	the the first slave is removed) until the bonding device is brought
	down or reconfigured.
	
	If you wish to change the MAC address, you can set it with ifconfig:

	  # ifconfig bond0 ha ether 00:11:22:33:44:55

	The MAC address can be also changed by bringing down/up the device
	and then changing its slaves (or their order):
	
	  # ifconfig bond0 down ; modprobe -r bonding
	  # ifconfig bond0 .... up
	  # ifenslave bond0 eth...

	This method will automatically take the address from the next slave
	that will be added.
	
	To restore your slaves' MAC addresses, you need to detach them
	from the bond (`ifenslave -d bond0 eth0'), set them down
	(`ifconfig eth0 down'), unload the drivers (`rmmod 3c59x', for
	example) and reload them to get the MAC addresses from their
	eeproms. If the driver is shared by several devices, you need
	to turn them all down. Another solution is to look for the MAC
	address at boot time (dmesg or tail /var/log/messages) and to
	reset it by hand with ifconfig :

	  # ifconfig eth0 down
	  # ifconfig eth0 hw ether 00:20:40:60:80:A0

9.  Which transmit polices can be used?

	Round robin, based on the order of enslaving, the output device
	is selected base on the next available slave.  Regardless of
	the source and/or destination of the packet.

	XOR, based on (src hw addr XOR dst hw addr) % slave cnt.  This
	selects the same slave for each destination hw address.

	Active-backup policy that ensures that one and only one device will
	transmit at any given moment. Active-backup policy is useful for
	implementing high availability solutions using two hubs (see
	section on HA).

High availability
=================

To implement high availability using the bonding driver, you need to
compile the driver as module because currently it is the only way to pass
parameters to the driver. This may change in the future.

High availability is achieved by using MII status reporting. You need to
verify that all your interfaces support MII link status reporting. On Linux
kernel 2.2.17, all the 100 Mbps capable drivers and yellowfin gigabit driver
support it. If your system has an interface that does not support MII status
reporting, a failure of its link will not be detected!

The bonding driver can regularly check all its slaves links by checking the
MII status registers. The check interval is specified by the module argument
"miimon" (MII monitoring). It takes an integer that represents the
checking time in milliseconds. It should not come to close to (1000/HZ)
(10 ms on i386) because it may then reduce the system interactivity. 100 ms
seems to be a good value. It means that a dead link will be detected at most
100 ms after it goes down.

Example:

   # modprobe bonding miimon=100

Or, put in your /etc/modules.conf :

   alias bond0 bonding
   options bond0 miimon=100

There are currently two policies for high availability, depending on whether
a) hosts are connected to a single host or switch that support trunking
b) hosts are connected to several different switches or a single switch that
   does not support trunking.

1) HA on a single switch or host - load balancing
-------------------------------------------------
It is the easiest to set up and to understand. Simply configure the
remote equipment (host or switch) to aggregate traffic over several
ports (Trunk, EtherChannel, etc.) and configure the bonding interfaces.
If the module has been loaded with the proper MII option, it will work
automatically. You can then try to remove and restore different links
and see in your logs what the driver detects. When testing, you may
encounter problems on some buggy switches that disable the trunk for a
long time if all ports in a trunk go down. This is not Linux, but really
the switch (reboot it to ensure).

Example 1 : host to host at double speed

          +----------+                          +----------+
          |          |eth0                  eth0|          |
          | Host A   +--------------------------+  Host B  |
          |          +--------------------------+          |
          |          |eth1                  eth1|          |
          +----------+                          +----------+

  On each host :
     # modprobe bonding miimon=100
     # ifconfig bond0 addr
     # ifenslave bond0 eth0 eth1

Example 2 : host to switch at double speed

          +----------+                          +----------+
          |          |eth0                 port1|          |
          | Host A   +--------------------------+  switch  |
          |          +--------------------------+          |
          |          |eth1                 port2|          |
          +----------+                          +----------+

  On host A :                             On the switch :
     # modprobe bonding miimon=100           # set up a trunk on port1
     # ifconfig bond0 addr                     and port2
     # ifenslave bond0 eth0 eth1

2) HA on two or more switches (or a single switch without trunking support)
---------------------------------------------------------------------------
This mode is more problematic because it relies on the fact that there
are multiple ports and the host's MAC address should be visible on one
port only to avoid confusing the switches.

If you need to know which interface is the active one, and which ones are
backup, use ifconfig. All backup interfaces have the NOARP flag set.

To use this mode, pass "mode=1" to the module at load time :

    # modprobe bonding miimon=100 mode=1

Or, put in your /etc/modules.conf :

    alias bond0 bonding
    options bond0 miimon=100 mode=1

Example 1: Using multiple host and multiple switches to build a "no single
point of failure" solution.


                |                                     |
                |port3                           port3|
          +-----+----+                          +-----+----+
          |          |port7       ISL      port7|          |
          | switch A +--------------------------+ switch B |
          |          +--------------------------+          |
          |          |port8                port8|          |
          +----++----+                          +-----++---+
          port2||port1                           port1||port2
               ||             +-------+               ||
               |+-------------+ host1 +---------------+|
               |         eth0 +-------+ eth1           |
               |                                       |
               |              +-------+                |
               +--------------+ host2 +----------------+
                         eth0 +-------+ eth1

In this configuration, there are an ISL - Inter Switch Link (could be a trunk),
several servers (host1, host2 ...) attached to both switches each, and one or
more ports to the outside world (port3...). One an only one slave on each host
is active at a time, while all links are still monitored (the system can
detect a failure of active and backup links).

Each time a host changes its active interface, it sticks to the new one until
it goes down. In this example, the hosts are not too much affected by the
expiration time of the switches' forwarding tables.

If host1 and host2 have the same functionality and are used in load balancing
by another external mechanism, it is good to have host1's active interface
connected to one switch and host2's to the other. Such system will survive
a failure of a single host, cable, or switch. The worst thing that may happen
in the case of a switch failure is that half of the hosts will be temporarily
unreachable until the other switch expires its tables. 

Example 2: Using multiple ethernet cards connected to a switch to configure
           NIC failover (switch is not required to support trunking). 


          +----------+                          +----------+
          |          |eth0                 port1|          |
          | Host A   +--------------------------+  switch  |
          |          +--------------------------+          |
          |          |eth1                 port2|          |
          +----------+                          +----------+

  On host A :                                 On the switch :
     # modprobe bonding miimon=100 mode=1     # (optional) minimize the time
     # ifconfig bond0 addr                    # for table expiration
     # ifenslave bond0 eth0 eth1

Each time the host changes its active interface, it sticks to the new one until
it goes down. In this example, the host is strongly affected by the expiration
time of the switch forwarding table.

3) Adapting to your switches' timing
------------------------------------
If your switches take a long time to go into backup mode, it may be
desirable not to activate a backup interface immediately after a link goes
down. It is possible to delay the moment at which a link will be
completely disabled by passing the module parameter "downdelay" (in
milliseconds, must be a multiple of miimon).

When a switch reboots, it is possible that its ports report "link up" status
before they become usable. This could fool a bond device by causing it to
use some ports that are not ready yet. It is possible to delay the moment at
which an active link will be reused by passing the module parameter "updelay"
(in milliseconds, must be a multiple of miimon).

A similar situation can occur when a host re-negotiates a lost link with the
switch (a case of cable replacement).

A special case is when a bonding interface has lost all slave links. Then the
driver will immediately reuse the first link that goes up, even if updelay
parameter was specified. (If there are slave interfaces in the "updelay" state,
the interface that first went into that state will be immediately reused.) This
allows to reduce down-time if the value of updelay has been overestimated.

Examples :

    # modprobe bonding miimon=100 mode=1 downdelay=2000 updelay=5000
    # modprobe bonding miimon=100 mode=0 downdelay=0 updelay=5000

4) Limitations
--------------
The main limitations are :
  - only the link status is monitored. If the switch on the other side is
    partially down (e.g. doesn't forward anymore, but the link is OK), the link
    won't be disabled. Another way to check for a dead link could be to count
    incoming frames on a heavily loaded host. This is not applicable to small
    servers, but may be useful when the front switches send multicast
    information on their links (e.g. VRRP), or even health-check the servers.
    Use the arp_interval/arp_ip_target parameters to count incoming/outgoing
    frames.  

Resources and links
===================

Current developement on this driver is posted to:
 - http://www.sourceforge.net/projects/bonding/

Donald Becker's Ethernet Drivers and diag programs may be found at :
 - http://www.scyld.com/network/

You will also find a lot of information regarding Ethernet, NWay, MII, etc. at
www.scyld.com.

For new versions of the driver, patches for older kernels and the updated
userspace tools, take a look at Willy Tarreau's site :
 - http://wtarreau.free.fr/pub/bonding/
 - http://www-miaif.lip6.fr/willy/pub/bonding/

To get latest informations about Linux Kernel development, please consult
the Linux Kernel Mailing List Archives at :
   http://boudicca.tux.org/hypermail/linux-kernel/latest/

-- END --