Use Linux as a SAN Provider

At one-tenth the cost of the typical commercial appliance, Linux can deliver storage with speed and redundancy. Make the move toward a full-featured iSCSI SAN solution with what you already have in your server room.

Multipathing

Another feature often run with iSCSI is multipathing. This allows
Linux to use multiple networks at once to access the iSCSI target. It
usually is run on separate physical networks, so in the event that
one fails, the other still will be up and the initiator will
not experience loss of a volume or a system crash. Multipathing can
be set up in two ways, either active/passive or active/active.
Active/active generally is the preferred way, as it can be set up not
only for redundancy, but also for load balancing. Like Fibre Channel,
multipath assigns World Wide Identifiers (WWIDs) to devices. These are
guaranteed to be unique and unchanging. When one of the paths is
removed, the other one continues to function. The initiator may
experience slower response time, but it will continue to function.
Re-integrating the second path allows the system to return to its
normal state.

RAID

When working with local disks, people often turn to Linux's software RAID
or LVM systems to provide redundancy, growth and snapshotting.
Because SAN volumes show up as block devices, it is possible to use
these tools on them as well. Use them with care though. Setting up
RAID 5 across three iSCSI volumes causes a great deal of network traffic
and almost never gives you the results you're expecting. Although, if
you have enough bandwidth available and you aren't doing many writes,
a RAID 1 setup across multiple iSCSI volumes may not be
completely out of the question. If one of these volumes drops,
rebuilding may be an expensive process. Be careful about how much
bandwidth you allocate to rebuilding the array if you're in a production
environment. Note that this could be used at the same time as
multipathing in order to increase your bandwidth.

To set up RAID 1 over iSCSI, first load the RAID 1 module:

modprobe raid1

After partitioning your first disk, /dev/sdb, copy the partition table
to your second disk, /dev/sdc. Remember to set the partition type to
Linux RAID autodetect:

sfdisk -d /dev/sdb | sfdisk /dev/sdc

Assuming you set up only one partition, use the mdadm command to create
the RAID group:

mdadm --create /dev/md0 --level=1 --raid-disks=2 /dev/sdb1 /dev/sdc1

After that, cat the /etc/mdstat file to watch the state of the
synchronization of the iSCSI volumes. This also is a good time to
measure your network throughput to see if it will stand up under
production conditions.

Conclusion

Running a SAN on Linux is an excellent way to bring up a shared
environment in a reasonable amount of time using commodity parts.
Spending a few thousand dollars to create a multiterabyte array is a
small budget when many commercial arrays easily can extend into the
tens to hundreds of thousands of dollars. In addition, you gain
flexibility. Linux allows you to manipulate the underlying
technologies in ways most of the commercial arrays do not. If you're
looking for a more-polished solution, the Openfiler Project provides
a nice layout and GUI to navigate. It's worth noting that many
commercial solutions run a Linux kernel under their shell, so unless
you specifically need features or support that isn't available with
standard Linux tools, there's little reason to look to commercial
vendors for a SAN solution.

Michael Nugent has spent a good deal of his time designing large-scale
solutions to fit into tiny budgets, leveraging Linux to fulfill
roles that typically would be filled by large commercial appliances.
Recently, Michael has been working to design large, private clouds for
SaaS environments in the financial industry. When not building
systems, he likes sailing, scuba diving and hanging out with his cat,
MIDI. Michael can be reached at michael@michaelnugent.org.