Infiniband HOWTO: SDP

Next Previous Contents

## 7. SDP

Sockets Direct Protocol (SDP) is a network protocol which provides an RDMA accelerated alternative to TCP over infiniband networks. OFED provides an LD_PRELOADable library (libsdp.so) which allows programs which use TCP to use the more efficient SDP protocol instead. The use of an LD_PRELOADable libary means that the switch in protocol is transparent, and does not require the application to be recompiled.

## 7.1 Configuration

SDP used IPoIB for address resolution, so you must configure IPoIB before using SDP.

You should also ensure the ib_sdp kernel module is installed.

    modprobe ib_sdp

You can use libsdp in two ways; you can either manually LD_PRELOAD the library whilst invoking your application, or create a config file which specifies which applications will use SDP.

To manually LD_PRELOAD a library, simply set the LD_PRELOAD variable before invoking your application.

    LD_PRELOAD=libsdp.so ./path/to/your/application ...

If you which to choose which programs will use SDP you can edit /etc/sdp.conf and specify which programs, ports and addresses are eligible for use.

## 7.2 Example Using SDP with Netpipe

The following example shows how to use libsdp to make the TCP benchmarking application, netpipe, use SDP rather than TCP. NodeA is the server and NodeB is the client. IPoIB is configured on both nodes, and NodeA's IPoIB address is 10.0.0.1

Install netpipe on both nodes.

    aptitude install netpipe-tcp

First, run the netpipe benchmark over TCP in order to obtain a baseline number.

> `

>     nodeA# NPtcp
>     nodeB# NPtcp -h 10.0.0.1
>     Send and receive buffers are 16384 and 87380 bytes
>     (A bug in Linux doubles the requested buffer sizes)
>     Now starting the main loop
>       0:       1 bytes   2778 times -->      0.22 Mbps in      34.04 usec
>       1:       2 bytes   2937 times -->      0.45 Mbps in      33.65 usec
>       2:       3 bytes   2971 times -->      0.69 Mbps in      33.41 usec
>     <snip>
>     121: 8388605 bytes      3 times -->   2951.89 Mbps in   21680.99 usec
>     122: 8388608 bytes      3 times -->   3008.08 Mbps in   21276.00 usec
>     123: 8388611 bytes      3 times -->   2941.76 Mbps in   21755.66 usec

> `

Now repeat the test, but force netpipe to use SDP rather than TCP.

> `

>     nodeA# LD_PRELOAD=libsdp.so NPtcp
>     nodeB# LD_PRELOAD=libsdp.so  NPtcp -h 10.0.0.1
>     Send and receive buffers are 16384 and 87380 bytes
>     (A bug in Linux doubles the requested buffer sizes)
>     Now starting the main loop
>       0:       1 bytes   9765 times -->      1.45 Mbps in       5.28 usec
>       1:       2 bytes  18946 times -->      2.80 Mbps in       5.46 usec
>       2:       3 bytes  18323 times -->      4.06 Mbps in       5.63 usec
>     <snip>
>     121: 8388605 bytes      5 times -->   7665.51 Mbps in    8349.08 usec
>     122: 8388608 bytes      5 times -->   7668.62 Mbps in    8345.70 usec
>     123: 8388611 bytes      5 times -->   7629.04 Mbps in    8389.00 usec

> `

You should see a significant increase in performance when using SDP.

Next Previous Contents