> We have Dell powerconnect 2724 Gigabit switches to connect the nodes
> in our cluster.
> As you said, may be the speed of PCI bus is a bottleneck.
> How can check it in practical?

Are all your gige nics on the same bus?

You might want to try running multiple copies of TCP pt2pt benchmarks
simultaneously on your machine to see what kind of performance you
get. E.g., run 4 copies of netperf on node A talking to 4
corresponding copies of netper on node B. Do you get full bandwidth
out of all 4 copies?