Because for the Linux I/O scheduler to work, nr_requests needs to
be at least twice as big as the scsi queue depth.

For all other scsi drivers, the scsi queue depth is somewhere between
0 and 63. Most are between 1 and 8.

Default nr_requests is 128, so this problem exists only with the
3ware driver/controller that has a queue depth of 254 ..

It's more complicated than that though when you have more than
one scsi device attached to the 3ware controller (multiple raid
arrays or JBODs defined), since the total queue depth of the
controller is 254. In that case one scsi device can starve
others on the same controller, so you want to tune down the
queue depth per device .. e.g. with 8 JBODs set queue_depth
per device to 32, set nr_requests to 128.

Perhaps the initial queue_depth per device should be set to
254 / tw_dev->tw_num_units, that would be optimal.
Something like

I think such a change should be submitted through the people
at 3ware, though.

> Is modifying nr_requests allowed?

Well we need to do the same things that ll_rw_blk::queue_requests_store()
does, only we don't need to worry about locking or existing queue
contents since the queue has been instantiated but the scsi device
is not active yet.

I do notice now however, that between 2.6.4 and 2.6.9-rc1
blk_queue_congestion_threshold() has been added which we should
probably call after adjusting nr_requests. Unfortunately it's
a static function in ll_rw_blk.c ..