Procedure for installing Qlogic SAN cards in Suse 10 & 11

[Nov 10, 2011] Adding and Removing SAN Disks from SUSE Device Manager

January 23, 2009 | Novell User Communities

Remove a disk

• Run the multipath -ll command, note the UUID (the big hex number), LUN and sdX device of the disk, eg in the example below it is LUN 2:, and they are /dev/sdf and /dev/sde - you will need this info for the procedure. Just confirm this is in fact the one you want to remove - cross-check the LUN and size of the volume on the SAN before proceeding...

        3600a0b80000fb6e50000000e487b02f5 dm-10 IBM,1742
        [size=1.6T][features=1 queue_if_no_path][hwhandler=1 rdac]
        \_ round-robin 0 [prio=6][active]
        \_ 1:0:0:2 sdf 8:80 [active][ready]
        \_ round-robin 0 [prio=1][enabled]
        \_ 0:0:0:2 sde 8:64 [active][ghost]

  Note - the dm-XX is not permanent and may change when you've added or removed disks, so don't rely on old info - check each time.
• Also ccat /proc/scsi/scsi to see match up the kernel SCSI/acronym> devices with the SCSI IDs and LUNs of the SAN disks.

• First you need to remove the disk from the volume group.
  
  • If the disk is in use, either delete what is on it (if there is a logical volume limited to that disk), or use pvmove. (this of course assumes you have sufficient space to move everything off the original disk)

    NB - with pvmove on SUSE10 sp2, there is a bug where if you are moving from a bigger disk to smaller disk(s), it may complain there isn't enough space. Just move as many extents as are on the first smaller disk, the you can move the rest on to the second, eg: pvmove /dev/dm-1:0-20000 /dev/dm-2.

  • Once stuff is deleted/removed: vgreduce VolGroup dev/dm-XX and pvremove /dev/dm-XX.
• Using the disk ID for the next command (the dm-xx isn't recognised), from multipath -ll:
  dmsetup remove 3600a0b80000f7b270000000b47b15c26. (of course you need to use your own disk ID)
• Now, finally, you remove the SCSI devices from the kernel:

      echo 1 > /sys/block/sdX/device/delete
      echo 1 > /sys/block/sdY/device/delete	  

You should now have all traces removed, you can run multipath -ll and cat /proc/scsi/scsi to cross check. You can now remove the mapping from the SAN and delete the logical volume if required.

[Nov 10, 2011] How to setup - use multipathing on SLES

SLES 9 information -- outdated

The boot scripts will only detect MPIO devices if the modules for the respective controllers are loaded at boot time. To achieve this, simply add the needed driver module to the variable INITRD_MODULES within the file /etc/sysconfig/kernel.

Example:

Your system contains a RAID controller that is accessed by the cciss driver and you are using ReiserFS as a filesystem. The MPIO devices will be connected to a Qlogic controller accessed by the driver qla2xxx, which is not yet configured to be used on this system. The mentioned entry within /etc/sysconfig/kernel will then probably look like this:

INITRD_MODULES="cciss reiserfs"

Using an editor, you would now change this entry:

INITRD_MODULES="cciss reiserfs qla2xxx"

When you have applied this change, you will need to recreate the INITRD on your system to reflect it. Simply run this command:

mkinitrd

When you are using GRUB as a bootmanager, you do not use to make any further changes. Upon the next reboot the needed driver will be loaded within the INITRD. If you are using LILO as bootmanager, please remember to run it once to update the boot record.

Configuring multipath-tools

If your system is one of those listed above, no further configuration should be required.

You might otherwise have to create /etc/multipath.conf (see the examples under /usr/share/doc/packages/multipath-tools/) and add an appropriate devices entry for your storage subsystem.

One particularly interesting option in the /etc/multipath-tools.conf file is the "polling_interval" which defines the frequency of the path checking that can be configured.

Alternatively, you might choose to blacklist certain devices which you do not want multipath-tools to scan.

You can then run:

multipath -v2 -d

to perform a 'dry-run' with this configuration. This will only scan the devices and print what the setup would look like.

The output will look similar to:

3600601607cf30e00184589a37a31d911
[size=127 GB][features="0"][hwhandler="1 emc"]
\_ round-robin 0 [first]
  \_ 1:0:1:2 sdav 66:240  [ready ]
  \_ 0:0:1:2 sdr  65:16   [ready ]
\_ round-robin 0
  \_ 1:0:0:2 sdag 66:0    [ready ]
  \_ 0:0:0:2 sdc  8:32    [ready ]

showing you the name of the MPIO device, its size, the features and hardware handlers involved, as well as the (in this case, two) priority groups (PG). For each PG, it shows whether it is the first (highest priority) one, the scheduling policy used to balance IO within the group, and the paths contained within the PG. For each path, its physical address (host:bus:target:lun), device nodename and major:minor number is shown, and of course whether the path is currently active or not.

Paths are grouped into priority groups; there's always just one priority group in active use. To model an active/active configuration, all paths end up in the same group; to model active/passive, the paths which should not be active in parallel will be placed in several distinct priority groups. This normally happens completely automatically on device discovery.

Enabling the MPIO components

Now run

/etc/init.d/boot.multipath start
/etc/init.d/multipathd start

as user root. The multipath devices should now show up automatically under /dev/disk/by-name/; the default naming will be the WWN of the Logical Unit, which you can override via /etc/multipath.conf to suit your tastes.

Run

insserv boot.multipath multipathd

to integrate the multipath setup into the boot sequence.

From now on all access to the devices should go through the MPIO layer.

Querying MPIO status

To query the current MPIO status, run

multipath -l

This will output the current status of the multipath maps in a format similar to the command already explained above:

3600601607cf30e00184589a37a31d911
[size=127 GB][features="0"][hwhandler="1 emc"]
\_ round-robin 0 [active][first]
  \_ 1:0:1:2 sdav 66:240  [ready ][active]
  \_ 0:0:1:2 sdr  65:16   [ready ][active]
\_ round-robin 0 [enabled]
  \_ 1:0:0:2 sdag 66:0    [ready ][active]
  \_ 0:0:0:2 sdc  8:32    [ready ][active]

However, it includes additional information about which priority group is active, disabled or enabled, as well as for each path whether it is currently active or not.

Tuning the fail-over with specific HBAs

HBA timeouts are typically setup for non-MPIO environments, where longer timeouts make sense - as the only alternative would be to error out the IO and propagate the error to the application. However, with MPIO, some faults (like cable failures) should be propagated upwards as fast as possible so that the MPIO layer can quickly take action and redirect the IO to another, healthy path.

For the QLogic 2xxx family of HBAs, the following setting in /etc/modprobe.conf.local is thus recommended:

options qla2xxx qlport_down_retry=1 ql2xfailover=0 ql2xretrycount=5

Managing IO in error situations

In certain scenarios, where the driver, the HBA or the fabric experiences spurious errors,it is advisable that DM MPIO is configured to queue all IO in case of errors leading loss of all paths, and never propagate errors upwards.

This can be achieved by setting

defaults {
		default_features "1 queue_if_no_path"
}

in /etc/multipath.conf.

As this will lead to IO being queued forever, unless a path is reinstated, make sure that multipathd is running and works for your scenario. Otherwise, IO might be stalled forever on the affected MPIO device, until reboot or until you manually issue a

dmsetup message 3600601607cf30e00184589a37a31d911 0 fail_if_no_path				

(substituting the correct map name), which will immediately cause all queued IO to fail. You can reactivate the queue if no path feature by issueing

dmsetup message 3600601607cf30e00184589a37a31d911 0 queue_if_no_path

You can also use these two commands to switch between both modes for testing, before committing the command to your /etc/multipath.conf.

4. Using the MPIO devices

• Using the whole MPIO devices directly

  If you want to use the whole LUs directly (if for example you're using the SAN features to partition your storage), you can simply use the /dev/disk/by-name/xxx names directly for mkfs, /etc/fstab, your application, etc.
  

• Using LVM2 on top of the MPIO devices

  To make LVM2 recognize the MPIO devices as possible Physical Volumes (PVs), you will have to modify /etc/lvm/lvm.conf. You will also want to modify it so that it does not scan and use the physical paths, but only accesses your MPIO storage via the MPIO layer.

  Thus, change the "filter" entry in lvm.conf as follows and add the types extension to make LVM2 recognize them:

  filter = [ "a|/dev/disk/by-name/.*|", "r|.*|" ]
  types = [ "device-mapper", 1 ] 

  This will allow LVM2 to only scan the by-name paths and reject everything else. (If you are also using LVM2 on non-MPIO devices, you will of course need to make the necessary adjustments to suit your setup.)

  You can then use pvcreate and the other LVM2 commands as usual on the /dev/disk/by-name/ path.
  

• Partitions on top of MPIO devices

  It is not currently possible to partition the MPIO devices themselves. However, if the underlying physical device is partitioned, the MPIO device will reflect those partitions and the MPIO layer will provide /dev/disk/by-name/>name<p1 ... pN devices so you can access the partitions through the MPIO layer.

  So you will have to partition the devices prior to enabling MPIO; if you change the partitioning in the running system, MPIO will not automatically detect this and reflect the changes; you will have to reinit MPIO, which in a running system, with active access to the devices, will likely imply a reboot.

  Thus, using the LUNs directly or via LVM2 is recommended.