IB Bonding
===============================================================================

1. Introduction
2. How to work with ib-bond
3. How to work with interface configuration scripts
3.1 Configuration with initscripts support
3.1.1 Writing network scripts under Redhat-AS4 (Update 4, 5, 6 or 7)
3.1.2 Writing network scripts under Redhhat-EL5
3.2 Configuration with sysconfig support
3.2.1 Writing network scripts under SLES-10
3.3 Configuring Ethernet slaves

1. Introduction
-------------------------------------------------------------------------------
ib-bonding is a High Availability solution for IPoIB interfaces. It is based
on the Linux Ethernet Bonding Driver and was adopted to work with IPoIB.
However, the support for for IPoIB interfaces is only for the active-backup
mode, other modes should not be used.
ib-bonding package contains a bonding driver and a utility called ib-bond to
manage and control the driver operation.

2. How to work with ib-bond
-------------------------------------------------------------------------------

* Creating a bonding network interface
 --bond-name: sets the name of the bonding network interface. Default is bond0
 --bond-ip <IP/[MASK]>: sets the IP address of bond0. If MASK is not given it
   is set to 255.255.255.0 (24 bits). Note that MASK should be the number of 1
   bits in the netmask.
 --slaves: a comma separated list of slave ib devices. If not given ib0 and
   ib1 will be used as slaves. Child interfaces are allowed.
 --miimon: the MII monitoring interval in mSec. Default is 100
* Deleting a bonding network interface
 --stop: unenslave slaves and delete a specific bonding network interface (use with --bond-name)
 --stop-all: unenslave slaves and delete all bonding network interfaces
* Querying a bonding network interface
 --status: show the status of a specific bonding network interface (use with --bond-name)
 --status-all: show the status of all bonding network interfaces
 
Examples:

* To bring up bond0 with ib0 and ib2 as slaves (assumes 2 HCAs)
	ib-bond --bond-ip 192.186.10.100 --slaves ib0,ib2
* To bring up bond1 with ib0.f1f1 1and ib1.f1f1 as slaves with non default
  netmask
	ib-bond --bond-name bond1 --bond-ip 192.186.10.100/25 --slaves ib0.f1f1,ib1.f1f1
* To query the status of bond1
	ib-bond --bond-name bond1 --status
* To query the status of all bonding interfaces
	ib-bond --status-all
* To stop bond1 
	ib-bond --bond-name bond1 --stop
* To stop all bonding interfaces
	ib-bond --stop-all

3. How to work with interface configuration scripts
-------------------------------------------------------------------------------
Using ib-bond to configure interfaces doesn't save the configuration anywhere,
so whenever the master or one of the slaves is destroyed the configuration
should be restored by running ib-bond again (e.g. after system reboot).
It is possible to avoid that if you create an interface configuration script for
the ibX and bondX interfaces. To do that, you should use the standard syntax to
create the bonding configuration (depending on your OS).

3.1 Configuration with initscripts support
------------------------------------------
Note: This feature is available only for Redhat-AS4 (Update 4, Update 5,
Update 6 or Update 7) and for Redhat-EL5 and above.

3.1.1 Writing network scripts under Redhat-AS4 (Update 4, 5, 6 or 7)
-----------------------------------------------------------------
* In the master (bond) interface script add the line:
TYPE=Bonding
MTU=<according to the slave's MTU>

Exmaple: for bond0 (master) the file is named /etc/sysconfig/network-scripts/ifcfg-bond0
with the following text in the file:

DEVICE=bond0
IPADDR=192.168.1.1
NETMASK=255.255.255.0
NETWORK=192.168.1.0
BROADCAST=192.168.1.255
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
TYPE=Bonding
MTU=65520

Note: 65520 is a valid mtu value only if all IPoIB slaves operate in connected
mode and are configured with the same value. For IPoIB slaves that work in
datagram modee, use MTU=2044. If you don't set correct mtu or don't set mtu at
all (and letting it to be set to the default value), performance of the
interface might decrease.

* In the slave (ib) interface script put the following lines:
SLAVE=yes
MASTER=<bond name>
TYPE=InfiniBand
PRIMARY=<yes|no>

Example: the script for ib0 (slave) would be named /etc/sysconfig/network-scripts/ifcfg-ib0
with the following text in the file:

DEVICE=ib0
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none
TYPE=InfiniBand
PRIMARY=yes

Note: If the slave interface is not primary then the line PRIMARY= is not
required and can be omitted.

After the configuration is saved, restart the network service by running:
/etc/init.d/network restart

3.1.2 Writing network scripts under Redhhat-EL5
-----------------------------------------------
Follow the instructions in 3.1.1 (Writing network scripts under Redhat-AS4)
with the following changes:
* In the bondX (master) script - the line TYPE=Bonding is not needed.
* In the bondX (master) script - you may add to the configuration more options
with the following line
BONDING_OPTS=" primary=ib0 updelay=0 downdelay=0"
* in the ibX (slave) script - the line TYPE=InfiniBand necessary when using 
  bonding over devices configured with partitions ( p_key)
Example:
  ifcfg-ibX.8003 and ifcfg-ibY.8003 must include TYPE=InfiniBand line in 
  their configuration files, when using as slaves for bondX device
* in /etc/modprobe.conf add the following lines
alias bond0 bonding
options bond0 miimon=100 mode=1 max_bonds=1

If you want more than one bonding interface, name them bond1, bond2... and 
just add the necessary lines in /etc/modprobe.conf and change max_bonds=1 to
max_bonds=N where N=number_of_bonding_interfaces

Note: restarting OFED doesn't keep the bonding configuration via initscripts.
You have to restart the network service in order to recreate the bonding
interface.

3.2 Configuration with sysconfig support
----------------------------------------
Note: This feature is available only for SLES-10 and above.

3.2.1 Writing network scripts under SLES-10 
-----------------------------------------------
* In the master (bond) interface script add the lins:

BONDING_MASTER=yes
BONDING_MODULE_OPTS="mode=active-backup miimon=<value>"
BONDING_SLAVE0=slave0
BONDING_SLAVE1=slave1
MTU=<according to the slave's MTU>

Exmaple: for bond0 (master) the file is named /etc/sysconfig/network/ifcfg-bond0
with the following text in the file:

BOOTPROTO="static"
BROADCAST="10.0.2.255"
IPADDR="10.0.2.10"
NETMASK="255.255.0.0"
NETWORK="10.0.2.0"
REMOTE_IPADDR=""
STARTMODE="onboot"
BONDING_MASTER="yes"
BONDING_MODULE_OPTS="mode=active-backup miimon=100 primary=ib0 updelay=0 downdelay=0"
BONDING_SLAVE0=ib0
BONDING_SLAVE1=ib1
MTU=65520

Note: 65520 is a valid mtu value only if all IPoIB slaves operate in connected
mode and are configured with the same value. For IPoIB slaves that work in
datagram modee, use MTU=2044. If you don't set correct mtu or don't set mtu at
all (and letting it to be set to the default value), performance of the
interface might decrease.

Note: primary, downdelay and updelay is an optional bonding interface
configuration. You may choose to use them, change them or delete them from the
configuration script (by editing the line that starts with BONDING_OPTS)

* The slave (ib) interace script should look like this:

BOOTPROTO='none'
STARTMODE='off'
PRE_DOWN_SCRIPT=/etc/sysconfig/network/unenslave.sh

After the configuration is saved, restart the network service by running:
/etc/init.d/network restart

3.3 Configuring Ethernet slaves
-------------------------------
It is not possible to have a mix of Ethernt slaves and IPoIB slaves under the
same bonding master. It is possible however that a bonding master of Ethernet
slaves and a bonding master of IPoIB slaves will co-exist in one machne.
To configure Ethernet slaves under a bonding master use the following
instructios (depending  on the OS)

* Under Redhat-AS4

Use the same instructions as for IPoIB slaves with the following exceptions

- In the master configuration file add the line
SLAVEDEV=1
- In the slave configuration file leave the line
TYPE=InfiniBand
- For Ethernet, it is possible to set parameters of the bonding module in /etc/modprobe.conf
with the following line for example
options bonding miimon=100 mode=1 primary=eth0
Note that alias names for the bonding module (such as bond0) may not work.

* Under Redhat-AS5

No special instructions are required.

* Under SLES10

When using both type of bonding under, it is neccessary to update the
MANDATORY_DEVICES  environment variable in /etc/sysconfig/network/config with the names
of the InfiniBand devices ( ib0, ib1, etc. ). Otherwise, bonding devices will be created 
before InfiniBand devices at boot time.

Note: If there is more than one Ethernet NIC installed then there might be a
race for the interface name eth0, eth1 etc. This may lead to unexpected
relation between logical and physical devices which may lead to wrong bonding
configuration. This issue may be solved by binding a logical device name (e.g.
eth0) to a physical (hardware) device by specifying the MAC address in the
ethN configuration file.
