Infiniband HOWTO: IP over Infiniband (IPoIB)

Next Previous Contents

## 5. IP over Infiniband (IPoIB)

The OFED stack allows you to run TCP/IP over your infiniband network, allowing you to run non-infiniband aware applications across your network. Several native infiniband applications also use IPoIB for host resolution (eg Lustre and SDP).

## 5.1 List the network devices

Check that the IBoIP modules is loaded.

> `#modprobe ib_ipoib`

You will now have an "ib" network interface for each of your infiniband cards.

> `

>     #ifconfig -a

>     <snip>
>     ib0       Link encap:UNSPEC  HWaddr 80-06-00-48-FE-80-00-00-00-00-00-00-00-00-00-00
>               BROADCAST MULTICAST  MTU:2044  Metric:1
>               RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>               TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>               collisions:0 txqueuelen:256
>               RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

>     ib1       Link encap:UNSPEC  HWaddr 80-06-00-49-FE-80-00-00-00-00-00-00-00-00-00-00
>               BROADCAST MULTICAST  MTU:2044  Metric:1
>               RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>               TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>               collisions:0 txqueuelen:256
>               RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
>     <snip>

> `

## 5.2 IP Configuration

You can now configure the ib network devices using /etc/network/interfaces.

> `

>     auto ib0
>     iface ib0 inet static
>       address 172.31.128.50
>       netmask 255.255.240.0
>       broadcast 172.31.143.255

> `

Bring the network device up, as normal.

> `ifup ib0`

## 5.3 Connected vs Unconnected Mode

IPoIB can run over two infiniband transports, Unreliable Datagram (UD) mode or Connected mode (CM). The difference between these two modes are described in:

    RFC4392 - IP over InfiniBand (IPoIB) Architecture
    RFC4391 - Transmission of IP over InfiniBand (IPoIB) (UD mode)
    RFC4755 - IP over InfiniBand: Connected Mode

ADDME: Pro/cons of these two methods?

You can switch between these two mode at runtime with:

> `

>      echo datagram > /sys/class/net/ibX/mode
>      echo connected > /sys/class/net/ibX/mode

> `

The default is datagram (UD) mode. If you with to use CM then you can add a script to /etc/network/interfaces/if-up.d to automatically set CM mode on your interfaces when they are configured.

## 5.4 TCP tuning

In order to obtain maximum IPoIB throughput you may need to tweak the MTU and various kernel TCP buffer and window settings. See the details in the ipoib_release_notes.txt document in the ofed-docs package.

## 5.5 ARP and dual ported cards

If you have a dual ported card with both ports on the same IB subnet, but different IP subnets, you will need to tweak the ARP settings for the IPoIB interfaces. See ipoib_release_notes.txt in the ofed-docs package for a full discussion of this issue.

> `

>        sysctl -w net.ipv4.conf.ib0.arp_ignore=1
>        sysctl -w net.ipv4.conf.ib1.arp_ignore=1

> `

Next Previous Contents