I am having trouble mounting a directory read/write from one machine on
two other machines. I can mount directories read-only with no problmes.
I can mount at least one directory read/write from one machine to
another.

I've found the problem. It is an arrogant implementation of an ambiguous
specification and a very poor selection of error codes.

The ambiguous term is 'reexport'. If you read the documentation of NFS,
it is stated plainly that you can NOT import a file system using NFS
and export it again using NFS. (I.E. the exporting of an nfs mounted
file system Marc is referring to.) For clarity I will call this
import/reexport.

The problem I have is that one part of a local file system is being
exported R/W and a subset of that file system is being exported R/O.
For clarity I will call this activity double export.

Import/reexport can cause extra network traffic. Arguably, it can be
useful at bridge and routing points, but the protocol expressly forbids
it.

Double export is another matter. While I have not dug up the RFC,
several otherwise competent books do not mention this as being a
problem. It can be mildly confusing since you may be able to change
a file using one path name and have the changes appear magically
when accessed using another path name. The access permissions may also
be different using the different names. Cache entries may be stale as
a result of this kind of thing, however the problem is no worse than
the problems associated with any local changes. Other than this
confusion, there is no insuperable reason for forbidding double
exports.

However, the Linux implementation does exactly that. What makes this
confusing is that it labels the attempt with an 'invalid argument'
error. This error is usually reserved for badly formed arguments,
with missing components, bad pointers or syntax errors. The error code
that should be returned should mean 'request refused' since
the refusal to export the sub-tree is a policy decision. However
this is not the worst part of the problem. It might be possible to
work around the problem by ordering the entries in the /etc/exports
file from less restrictive to more restrictive, much as the ordering
of ACLs can be used for precise control of file permissions. However
the /etc/exports parsing code arbitrarily reorders the requests,
making this impossible.

I got around this problem by only specifying the most liberal access
on the server and used the client side specifications to restrict the
access modes. This solution has some very obvious security problems
associated with it. I can do this because I can trust all the people
who use my sub-net, but I would have a very serious problem if I had
to meet a serious security audit. (NFS has a poor security reputation
precisely because of problems like this.)

Max

10-02-2007, 02:55 PM

unix

Re: Problem mounting r/w (r/o works)

Now I see what is happening a bit better. This all has to do with
FSID's (file system ID's). During the mount operation, the NFS client
asks mountd on the server for the filehandle for an exported fileystem.
The mountd then returns a FSID for this filesystem, which is a unique
handle for every mounted filesystem on the sever. So, when you ask for
the handle for /export/home, let's say you get back 0x12345 as the
filehandle. now, according to the exports file, /export/home is rw.

You also have /export/home/allmine exported as ro (which is technically
illegal, because you have already exported the filesystem with the
handle 0x12345). Depending on the server, either it ignores the new
entry, or it replaces it. The client then sends a mount request to the
server for /export/home/allmine. The mountd on the server looks at the
request, and it finds the one (and only) entry for that particular
filesystem (with the unique id of 0x12345) and returns that entry.
The client then sees that he got back the entry for the parent, but that
you asked for the child, so he simply mounts the child based on the
parents permissions.

My explanation may be a bit 'fuzzy', but I hope that this helps.

Marc

Max T.E. Woodbury wrote:[color=blue]
> "Marc D. Behr" wrote:
>[color=green]
>>so are you trying to export an nfs mounted filesystem again via nfs
>>(sure looks that way).[/color]
>
>
> I've found the problem. It is an arrogant implementation of an ambiguous
> specification and a very poor selection of error codes.
>
> The ambiguous term is 'reexport'. If you read the documentation of NFS,
> it is stated plainly that you can NOT import a file system using NFS
> and export it again using NFS. (I.E. the exporting of an nfs mounted
> file system Marc is referring to.) For clarity I will call this
> import/reexport.
>
> The problem I have is that one part of a local file system is being
> exported R/W and a subset of that file system is being exported R/O.
> For clarity I will call this activity double export.
>
> Import/reexport can cause extra network traffic. Arguably, it can be
> useful at bridge and routing points, but the protocol expressly forbids
> it.
>
> Double export is another matter. While I have not dug up the RFC,
> several otherwise competent books do not mention this as being a
> problem. It can be mildly confusing since you may be able to change
> a file using one path name and have the changes appear magically
> when accessed using another path name. The access permissions may also
> be different using the different names. Cache entries may be stale as
> a result of this kind of thing, however the problem is no worse than
> the problems associated with any local changes. Other than this
> confusion, there is no insuperable reason for forbidding double
> exports.
>
> However, the Linux implementation does exactly that. What makes this
> confusing is that it labels the attempt with an 'invalid argument'
> error. This error is usually reserved for badly formed arguments,
> with missing components, bad pointers or syntax errors. The error code
> that should be returned should mean 'request refused' since
> the refusal to export the sub-tree is a policy decision. However
> this is not the worst part of the problem. It might be possible to
> work around the problem by ordering the entries in the /etc/exports
> file from less restrictive to more restrictive, much as the ordering
> of ACLs can be used for precise control of file permissions. However
> the /etc/exports parsing code arbitrarily reorders the requests,
> making this impossible.
>
> I got around this problem by only specifying the most liberal access
> on the server and used the client side specifications to restrict the
> access modes. This solution has some very obvious security problems
> associated with it. I can do this because I can trust all the people
> who use my sub-net, but I would have a very serious problem if I had
> to meet a serious security audit. (NFS has a poor security reputation
> precisely because of problems like this.)
>
> Max[/color]

10-02-2007, 02:55 PM

unix

Re: Problem mounting r/w (r/o works)

"Marc D. Behr" wrote:[color=blue]
>
> Now I see what is happening a bit better. This all has to do with
> FSID's (file system ID's). During the mount operation, the NFS client
> asks mountd on the server for the filehandle for an exported fileystem.
> The mountd then returns a FSID for this filesystem, which is a unique
> handle for every mounted filesystem on the sever. So, when you ask for
> the handle for /export/home, let's say you get back 0x12345 as the
> filehandle. now, according to the exports file, /export/home is rw.
>
> You also have /export/home/allmine exported as ro (which is technically
> illegal, because you have already exported the filesystem with the
> handle 0x12345).[/color]

1) Why is it illegal? Besides it ends up doing the equivalent of
exporting /export/home/allmine and refusing to export /export/home.
Not exporting /export/home/allmine is my work-around.

2) In the real case /home was the root of the file system, not
/home/pub but /home/pub gets exported and /home does not.

3) There are various 'export' points involved. /var/lib/nfs/etab,
/var/lib/nfs/xtab and the kernel tables which can be checked using
/proc/fs/nfs/exports in addition to /etc/exports. The export of
/pub (a.k.a /home/pub) and /home both make it into /var/lib/nfs/etab
but only /pub makes it into /var/lib/nfs/xtab and
/proc/fs/nfs/exports. The entries in the various files appear in
different orders.
[color=blue]
> Depending on the server, either it ignores the new entry, or it
> replaces it.[/color]

In fact neither happens. On Linux it returns a bogus 'Invalid Argument'
error code. However that is a bit of a nit. I will NOT quibble that it
in effect ignores the request.
[color=blue]
> The client then sends a mount request to the
> server for /export/home/allmine. The mountd on the server looks at
> the request, and it finds the one (and only) entry for that
> particular filesystem (with the unique id of 0x12345) and returns
> that entry. The client then sees that he got back the entry for the
> parent, but that you asked for the child, so he simply mounts the
> child based on the parents permissions.[/color]

Once I was able to figure out what was going on, this is almost what is
going on. In fact that may be exactly what is going on since I have not
tried to write to the 'ro' directory. However, I would not be surprised
if the client refused to forward a write request since as far as it is
concerned the fs is 'ro'.
[color=blue]
> My explanation may be a bit 'fuzzy', but I hope that this helps.[/color]

I can handle the fuzz. What I am having trouble with is the lousy
error messages and the fact the the server reorders my export requests.
Subtract EITHER one of those and my debug time would have gone way down.