At every introduction to a new office environment, I spend (waste?) a certain
period of time to connect to network shares through CIFS/SMB. This was not
different at my new work place either. After spending almost two days to
troubleshoot the problem, I figured out that the problem was due to the
missing /sbin/request-key binary provided by keyutils package in Xubuntu
14.04 LTS. So how did I really get it solved?

Defining the Problem

It is really sick that cifs kernel module which is responsible for loading
CIFS/SMB filesystems returns EINVAL (that is, error(22): Invalid argument)
for an entire family of errors and makes it impossible for the user to spot
the problem. This was also the case for me.

So given that the UNC is correct, the local mount point does exist and
accessible, and mount options are valid, invalid argument error does not
provide any context for the problem.

Further searching on the internet leaded me to LinuxCIFS
troubleshooting
at Samba Wiki. There it is told that I can enable debug messages for cifs
module by echoing a non-zero value to /proc/fs/cifs/cifsFYI, hence I did so:

$ echo 1 | sudo tee /proc/fs/cifs/cifsFYI

And tried to mount the network share again. And observed the following snippet
in the dmesg output:

So dns_resolve_server_name_to_ip could not resolve path for some reason.

DNS at the Kernel-Level

So how does kernel really resolve domain names to IP addresses? After reading
mount.cifs(8) a couple of times, I
saw See Also section telling that
Documentation/filesystems/cifs.txt
in the linux kernel source tree may contain additional options and
information. I checked that too and spotted the following lines:

DFS support allows transparent redirection to shares in an MS-DFS name
space. In addition, DFS support for target shares which are specified as UNC
names which begin with host names (rather than IP addresses) requires a user
space helper (such as cifs.upcall) to be present in order to translate
host names to ip address, and the user space helper must also be configured
in the file /etc/request-key.conf. Samba, Windows servers and many NAS
appliances support DFS as a way of constructing a global name space to ease
network configuration and improve reliability.

To use cifs Kerberos and DFS support, the Linux keyutils package should be
installed and something like the following lines should be added to the
/etc/request-key.conf file:

The DNS resolver module provides a way for kernel services to make DNS
queries by way of requesting a key of key type dns_resolver. These
queries are upcalled to userspace through /sbin/request-key.

The long story short, CIFS module employs DNS resolver module, which
internally upcalls the queries to userspace through /sbin/request-key. I got
further curious and checked the request-key(8):

This program is invoked by the kernel when the kernel is asked for a key
that it doesn’t have immediately available. The kernel creates a partially
set up key and then calls out to this program to instantiate it. It is not
intended to be called directly.

The Solution

Everything was pointing that something was wrong with my /sbin/request-key.
Ooops! The binary was not there at all. apt-file search /sbin/request-key
told that the binary was provided by keyutils package, which was missing in
my system. After installing keyutils, I did not observe any DNS resolver
issues in the dmesg logs and mount.cifs worked without a problem.