Infiniband HOWTO: SRP

Next Previous Contents

## 8. SRP

SRP (SCSI Remote protocol or SCSI RDMA protocol) is a protocol that allows the use of SCSI devices across infiniband. If you have infiniband storage, use can use SRP to acess the devices.

## 8.1 Configuration

Ensure that your infiniband storage is presented to the host in question. Check your storage controller documentation. Ensure that the ib_srp kernel module is loaded and that the srptools package is installed.

> `modprobe ib_srp`

## 8.2 SRP daemon configuration

srp_daemon is responsible for discovering and connecting to SRP targets. The default configuration shipped with srp_daemon is to ignore all presented devices; this is a failsafe to prevent devices from being mounted by accident on the wrong hosts.

The srp_daemon config file /etc/srp_daemon.conf has a simply syntax, and is described in the srp_daemon(1) manpage. Each line in this file is a rule which can be either to allow connection or to disallow connection according to the first character in the line (a or d accordingly) and ID of the storage device.

### Determine the IDs of presented devices

You can determine the IDs of SRP devices presented to your hosts by running the ibsrpdm -c command.

> `

>     # ibsrpdm -c
>     id_ext=50001ff10005052a,ioc_guid=50001ff10005052a,dgid=fe8000000000000050001ff10005052a,pkey=ffff,service_id=2a050500f11f0050

> `

### Configure srp_deamon to connect to the devices

Once we have the IDs of the devices, we can add them to /etc/srp_daemon.conf. You can also specify other srp related options for the target, such as max_cmd_per_lun and Max_sect. These are storage specific; check your vendor documentation for reccomended values.

> `

>     # This rule allows connection to our target
>     a id_ext=50001ff10005052a,ioc_guid=50001ff10005052a,max_cmd_per_lun=32,max_sect=65535
>     # This rule disallows everything else
>     d

> `

Restart the srp_daemon and the storage target should now become visible; check the kernel log to see if the disk has been detected.

    /etc/init.d/srptools restart

In the example kernel log output the disk has been descovered as scsi device sdb.

> `

>     scsi 3:0:0:1: Direct-Access     IBM      DCS9900          5.03 PQ: 0 ANSI: 5
>     sd 3:0:0:1: [sdb] 1953458176 4096-byte hardware sectors (8001365 MB)
>     sd 3:0:0:1: [sdb] Write Protect is off
>     sd 3:0:0:1: [sdb] Mode Sense: 97 00 10 08
>     sd 3:0:0:1: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA
>     sd 3:0:0:1: [sdb] 1953458176 4096-byte hardware sectors (8001365 MB)
>     sd 3:0:0:1: [sdb] Write Protect is off
>     sd 3:0:0:1: [sdb] Mode Sense: 97 00 10 08
>     sd 3:0:0:1: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA
>      sdb:<6>scsi4 : SRP.T10:50001FF10005052A
>      unknown partition table
>     sd 3:0:0:1: [sdb] Attached SCSI disk
>     sd 3:0:0:1: Attached scsi generic sg5 type 0

> `

## 8.3 Multipathing, LVM and formatting

The newly detected SRP device can be treated as an other scsi device. If you have multiple infiniband adapters you can use multipath-tools on top of the SRP devices to protects against a network failure. If you are not using multipathed IO you can simply format the device as normal.

Next Previous Contents