Seems like it is not a problem of the driver.
I run a benchmark with hdparm and it seems like the tool is just
requesting the data in 4k blocks. Is this possible?
With normal file operations I can see packets with 16 sectors.
Ed Cashin schrieb:
> On Fri May 22 07:19:45 EDT 2009, philip.kirchhoff@... wrote:
>> Hello Ed,
>>
>> I was now able to install the driver. But still the number of requested
>> sectors is not higher than 8.
>> Do you have any other idea?
>
> Which driver did you install and which is running? You can tell by
> examining the output of aoe-version.
>
> If it's version 71, and you only see 4 KiB payloads (you can see the
> payload size in the output of today's aoe-stat), then the driver is
> probably choosing to use a smaller size because either the local
> interface is something like 4200, or else the target's advertised
> buffer count is eight.
>
> Those are the two inputs to the decision the aoe driver uses to
> determine payload size.
>

On Fri May 22 07:19:45 EDT 2009, philip.kirchhoff@... wrote:
>
> Hello Ed,
>
> I was now able to install the driver. But still the number of requested
> sectors is not higher than 8.
> Do you have any other idea?
Which driver did you install and which is running? You can tell by
examining the output of aoe-version.
If it's version 71, and you only see 4 KiB payloads (you can see the
payload size in the output of today's aoe-stat), then the driver is
probably choosing to use a smaller size because either the local
interface is something like 4200, or else the target's advertised
buffer count is eight.
Those are the two inputs to the decision the aoe driver uses to
determine payload size.
--
Ed

Hello Ed,
I was now able to install the driver. But still the number of requested
sectors is not higher than 8.
Do you have any other idea?
Best regards
Philip
Ed Cashin schrieb:
> Hi. Did you mean to take the conversation off the mailing list?
>
> Anyway, the most common reason for an invalid module error at load
> time is some mis-match between the kernel sources you're using and
> the kernel you're running.
>
> If you check dmesg in such a case, the kernel will usually tell
> you exactly what doesn't match. Usually it's a detail like the
> compiler version, SMP support in the kernel, etc.
>
> You might have already seen it, but we have an Ubuntu page on
> the CORAID support site.
>
> http://support.coraid.com/support/linux/distros/ubuntu.html
>
> Please let me know (or support@...) if that information
> needs updating.
>

On Tue, May 19, 2009 at 6:20 PM, Ed Cashin <ecashin@...> wrote:
> I don't understand why you mention "vbladed" and "detach" together.
> The vbladed is a wrapper for vblade that launches it as a daemon.
>
> Your aoe-flush command cannot work without a /dev/etherd/flush
> device file in place. Check the Linux Support Page at CORAID
> for distro-specific tips, where you'll find RHEL/CentOS tips
> that should help udev create the device nodes.
Ed, Can you please send me the exact URL for this. I have everything working
on fedora ( means ls -l /dev/etherd was working fine with each entry
corrected ) but on CENTOS it is not working correctly.....
I mean there is no entry for /dev/etherd/flush or anything except following
one:
[root@... aoetools-30]#
[root@... aoetools-30]# ls -l /dev/etherd/
total 0
brw-r----- 1 root disk 152, 48 May 21 16:34 e0.3
[root@... aoetools-30]#
[root@... aoetools-30]#
Please help by suggesting how to get udev customize for this .
Krishna
>
>
> --
> Ed
>

I've been using something like this with a small patch to vblade (see
attached diff).
To sum it up, there is an added option of '-p' to specify a file to
write the pid and ppid to.
Another part is the vblade-skeleton used as an init script in /etc/
init.d/vblade-resourcename. Just copy it to a new file and update the
variables ( probably needs an update to work on redhat style systems
using chkconfig ).
Example... so to start and stop a vblade with resource name e1.0:
/etc/init.d/vblade-e1.0 start
/etc/init.d/vblade-e1.0 stop
or for SIGKILL
/etc/init.d/vblade-e1.0 stop-force
Hope thats useful.
--
Matth Ingersoll

On Wed, 2009-05-20 at 11:57 +0530, er krishna wrote:
> >>>> What about a new dameon, if we create vbladectl which will keep
> track of all the exported device based on interface & path
> ( /dev/loop0 or eth0 ) release the attached resource or add new
> resource ? I think we can have a look for this implemenatation.
>
> Ed, does it seems valuable or funny. A patch will be welcome if I try
> for this implementation. Just suggest.
What I was going to do was add some more options to vblade, i.e. -d /
--daemon. vbladed would then become a symbolic link to vblade, and if
vblade is run as vbladed, --daemon is automatically assumed, everything
just goes to syslog.
The other change I will finish and submit is a simple ini style
configuration file, something like this:
[daemon]
loglevel = info
; %S is always expanded to shelf, %s always expands to slot
lockfile = /var/run/vblade/blade-%S-%s.pid
[/dev/volgroup/foo]
shelf = xx
slot = xx
eth = xx
directio = true / false
on_boot = true
[/home/foo/file.img]
...
...
...
I don't think a wire protocol (as vbladectl would suggest) is really
needed, all each vbladed process has to do is lock
via /var/run/vblade/{shelf}.{slot} , which would contain its process ID.
Similarly, vbladectl can just parse the configuration file.. i.e
vbladectl stop /dev/volgroup/foo
vbladectl restart /dev/volgroup/foo
vbladectl show would have the task of validating whatever is
in /var/run/vblade, and could easily correlate each export to the pid of
the blade handling it.
I wasn't going to get any more complicated than that, as that alone is
enough for anyone to write custom rc scripts / tools.
Really, in fact, all of that can be accomplished without touching vblade
itself. It just seemed a little cleaner to just modify it vs making a
bunch of wrappers to work around it.
This approach would also keep from breaking existing use of vblade /
vbladed.
Cheers,
--Tim

Dear Tim,
Thanks for the info and valuable suggestions.
On Wed, May 20, 2009 at 10:18 AM, Tim Post <echo@...> wrote:
> Hi,
>
> On Wed, 2009-05-20 at 09:40 +0530, er krishna wrote:
>
> >
> > Stop vblade/vbladed on the storage server.
> >
> > How ? Currently I am doing ps-eaf | grep vblade ; & then I kill all
> > the process; is there any other way ?
>
> You could try the command pidof vblade. This will print all vblade PID's
> to stdout.
>
> so kill `pidof vblade` may be easier for you, however it is not
> discriminate. In other words, you may want to stop just one vblade, not
> all of them. Currently there is no easy way to keep track of which PID
> is exporting what.
>
> I have some patches that I'll send later this week that 'properizes'
> vblade for use as a daemon. It then becomes easier to manage. The
> changes are kind of invasive, so I'm guessing they can just live in
> contrib/.
>>>> What about a new dameon, if we create vbladectl which will keep track
of all the exported device based on interface & path ( /dev/loop0 or eth0 )
release the attached resource or add new resource ? I think we can have a
look for this implemenatation.
Ed, does it seems valuable or funny. A patch will be welcome if I try for
this implementation. Just suggest.
>
>
>
> > ok, so aoe-flush is necessary; Currently I am doing rmmod aoe ;
> > modprobe aoe ; aoe-stat & results are as per exptation. But aoe-flush
> > seems better way.
>
> Yes.
>
> > 1) On server ps -eaf | grep vblade; find all the processes & kill
> > those by kill -9.
>
> Don't use -9, its not needed. Let vblade / vbladed handle the signals as
> they are designed to do (i.e. clean up tasks, etc).
>
> > 2) Run ssh command & login on each machine ( w/o passwd ) ; then
> > execute aoe-flush command on each node on the network.
>
> Yes, that is the simplest way. Puppet as others have mentioned would
> also do the job.
>
> Cheers,
> --Tim
>
>
>
>

Hi,
On Wed, 2009-05-20 at 09:40 +0530, er krishna wrote:
>
> Stop vblade/vbladed on the storage server.
>
> How ? Currently I am doing ps-eaf | grep vblade ; & then I kill all
> the process; is there any other way ?
You could try the command pidof vblade. This will print all vblade PID's
to stdout.
so kill `pidof vblade` may be easier for you, however it is not
discriminate. In other words, you may want to stop just one vblade, not
all of them. Currently there is no easy way to keep track of which PID
is exporting what.
I have some patches that I'll send later this week that 'properizes'
vblade for use as a daemon. It then becomes easier to manage. The
changes are kind of invasive, so I'm guessing they can just live in
contrib/.
> ok, so aoe-flush is necessary; Currently I am doing rmmod aoe ;
> modprobe aoe ; aoe-stat & results are as per exptation. But aoe-flush
> seems better way.
Yes.
> 1) On server ps -eaf | grep vblade; find all the processes & kill
> those by kill -9.
Don't use -9, its not needed. Let vblade / vbladed handle the signals as
they are designed to do (i.e. clean up tasks, etc).
> 2) Run ssh command & login on each machine ( w/o passwd ) ; then
> execute aoe-flush command on each node on the network.
Yes, that is the simplest way. Puppet as others have mentioned would
also do the job.
Cheers,
--Tim

Dear Sam,
Thanks for the mail. I am doing this only.
On Tue, May 19, 2009 at 8:06 PM, Sam Hopkins <sah@...> wrote:
> > Stop vblade/vbladed on the storage server.
>
> man kill
>
> There is no way to stop vblade with vblade. I suspect you're
> wanting something akin to the init.d scripts to start/stop
> a program.
Yes i was looking something like that.
> vblade doesn't do that.
vbladectl should be an utility which should do this. I will try to make one;
( Actually for specific work which I am trying its necessary to do this )
> Use the kill command
> to stop the process.
>
> Sam
>
>

On Tue, May 19, 2009 at 6:40 PM, Tim Post <echo@...> wrote:
> Hi,
>
> On Tue, 2009-05-19 at 18:30 +0530, er krishna wrote:
> > Dear ED,
> >
> > Thanks for reply and sorry for all the unclear mails from myside.
> > Actaully my requiremnt is to stop vblade service from server side ( or
> > to remove the exported device ), Please tell weather its possible with
> > vblade and if it is then how ? Any specific command or url will be
> > most welcome.
> >
> >
>
> Stop vblade/vbladed on the storage server.
How ? Currently I am doing ps-eaf | grep vblade ; & then I kill all the
process; is there any other way ?
>
>
> Run aoe-flush on the nodes accessing the storage.
ok, so aoe-flush is necessary; Currently I am doing rmmod aoe ; modprobe aoe
; aoe-stat & results are as per exptation. But aoe-flush seems better way.
>
> This can easily be wrapped in automation scripts, as facilitated by
> things like SSH.
Yes, I exactly want to do this from server end only. I can do script. Please
correct following steps if there is any better way.
1) On server ps -eaf | grep vblade; find all the processes & kill those by
kill -9.
2) Run ssh command & login on each machine ( w/o passwd ) ; then execute
aoe-flush command on each node on the network.
>
>
> Cheers,
> --Tim
>
>

> Stop vblade/vbladed on the storage server.
man kill
There is no way to stop vblade with vblade. I suspect you're
wanting something akin to the init.d scripts to start/stop
a program. vblade doesn't do that. Use the kill command
to stop the process.
Sam

Hi,
On Tue, 2009-05-19 at 18:30 +0530, er krishna wrote:
> Dear ED,
>
> Thanks for reply and sorry for all the unclear mails from myside.
> Actaully my requiremnt is to stop vblade service from server side ( or
> to remove the exported device ), Please tell weather its possible with
> vblade and if it is then how ? Any specific command or url will be
> most welcome.
>
>
Stop vblade/vbladed on the storage server.
Run aoe-flush on the nodes accessing the storage.
This can easily be wrapped in automation scripts, as facilitated by
things like SSH.
Cheers,
--Tim

Dear ED,
Thanks for reply and sorry for all the unclear mails from myside. Actaully
my requiremnt is to stop vblade service from server side ( or to remove the
exported device ), Please tell weather its possible with vblade and if it is
then how ? Any specific command or url will be most welcome.
On Tue, May 19, 2009 at 6:20 PM, Ed Cashin <ecashin@...> wrote:
> I don't understand why you mention "vbladed" and "detach" together.
> The vbladed is a wrapper for vblade that launches it as a daemon.
daemonize
Perhaps, I have misundrstood vblade man page, sorry for it.
The vbladed script can be used to daemonize the vblade process, detaching
it from your terminal and sending its output to the system logs.
>
> Your aoe-flush command cannot work without a /dev/etherd/flush
> device file in place. Check the Linux Support Page at CORAID
> for distro-specific tips, where you'll find RHEL/CentOS tips
> that should help udev create the device nodes.
Thanks for the suggestion. I am trying this.
>
>
> --
> Ed
Best Regards,
Krishna

Oh. And as for the vblade process itself, just kill it like any other
userland process.
(And like any userland process, you'd send a SIGTERM, not a SIGKILL,
so that the process has a chance to do any necessary cleanup on exit.)
--
Ed

I don't understand why you mention "vbladed" and "detach" together.
The vbladed is a wrapper for vblade that launches it as a daemon.
Your aoe-flush command cannot work without a /dev/etherd/flush
device file in place. Check the Linux Support Page at CORAID
for distro-specific tips, where you'll find RHEL/CentOS tips
that should help udev create the device nodes.
--
Ed

Dear All,
Can anybody tell me how to detach exported block device through
vblade/vbladed command. For ex: ./vblade 0 0 eth0 /dev/hda6 is the command
to export the block device over network. What is the exact command to
remove/detach this block device from network.
What I did to do this, I killed all the processes after doing ps -eaf | grep
vblade, it worked but didn't remove entry from client side. Still aoe-stat &
ls -l /dev/etherd shows an entry of corresponding e0.0 exported device,
which is not exported now. Can Anybody help me here, It seems I am missing
something.
Thanks & Best Regards,
Krishna

Hello,
I have a problem with the Linux (Ubuntu 2.6.29) AoE driver. Although my
NIC supports MTU 9000 and my AoE server sends sector count value of 16,
there are always only 8 transfers with 8 sectors requested. What could
be the reason?
Thanks,
Philip

Hello Mark,
Am Freitag, den 08.05.2009, 14:21 +0100 schrieb Mark Sutton:
> On 20 Apr 2009, at 11:12, er krishna wrote:
>
> > I am trying to setup NFS failover and its recovery via GFS (using
> > CMAN) tool. Of course, I have created a gfs file system over my
> > logical volume ( basically its on exported block devices ). I have
> > three nodes centos, centos1 and centos2. I am attaching my
> > cluster.conf file for further reference. Everything seems fine, but
> > I am not able to migrate my NFS services when I poweroff my first
> > node. Anybody has any idea about it ?
> >
>
> A quick glance at your cluster.conf suggests you're not telling cman
> how many votes to require for quorum. I'm not sure what the behaviour
> of cman is in these conditions but it might well be setting
> expected_votes to the total number of nodes in your cluster ie 3. So
> if one node went down then the cluster would lose quorum and any
> cluster operations including failover will fail.
>
> <cman/>
>
> For a 3 node cluster I guess expected_votes would be best at 2, so
> maybe try changing your cman line to this:
>
> <cman expected_votes="2" />
cman does the right thing by default so there is no need to specify it.
In this case it also wouldn't make any difference at all:
quorum = floor( expected_votes / 2 ) + 1
2 in booth cases.
> And that might fix it... Of course this thread is a little old now,
> you might have already fixed the problem...
>
> Hope this helps,
>
> Mark
Marc

On 20 Apr 2009, at 11:12, er krishna wrote:
> I am trying to setup NFS failover and its recovery via GFS (using
> CMAN) tool. Of course, I have created a gfs file system over my
> logical volume ( basically its on exported block devices ). I have
> three nodes centos, centos1 and centos2. I am attaching my
> cluster.conf file for further reference. Everything seems fine, but
> I am not able to migrate my NFS services when I poweroff my first
> node. Anybody has any idea about it ?
>
A quick glance at your cluster.conf suggests you're not telling cman
how many votes to require for quorum. I'm not sure what the behaviour
of cman is in these conditions but it might well be setting
expected_votes to the total number of nodes in your cluster ie 3. So
if one node went down then the cluster would lose quorum and any
cluster operations including failover will fail.
<cman/>
For a 3 node cluster I guess expected_votes would be best at 2, so
maybe try changing your cman line to this:
<cman expected_votes="2" />
And that might fix it... Of course this thread is a little old now,
you might have already fixed the problem...
Hope this helps,
Mark

Hi,
I try to setup a virtualized test environment to experiment with
heartbeat2. I've written an aoe resource agent that performs (among
other things) the monitoring of the aoe targets. It does so by pinging
targets listed by aoe-stat.
Two virtual macnines run the cluster services, and the host is providing
the aoe targets (with vblade) through a tap interface that's also
connected to one of the interfaces of the guests.
My problem is that aoeping doesn't seem to work through the tap
interface. I can see the config request appearing on the tap in the
host, and, according to strace, vblade sends back the reply using
write(). However, I can't see the reply actually going out on the tap.
I also tried this locally, with a vblade running on a tap and pinging,
but got the same results. On real wire, with two physical hosts, it
works fine.
Apart from that, the vblades appear to function properly.
Do you have any idea what the problem is?
Thanks,
--
Szabolcs