DBM User Authentication

This week, we explain how to store user authentication
information in DBM files for faster access when you have
thousands of users.

The feature on User
Authentication shows how to restrict pages to
selected people. We showed how to use the htpasswd
program to create the necessary .htpasswd files, and how to
create group files to provide more control over the users. We
also said that .htpasswd files and group files like this are
not very efficient when a large number of users are involved.
This is because these are plain text files and for every
request in the authenticated area Apache has to read through
the file looking for the user. A much faster way to store the
user information is to use files in DBM format. This
article explains how to create and manage DBM format user
authentication files.

What is DBM?

DBM files are a simple and relatively standard method of
storing information for quick retrieval. Each item of
information stored in a DBM file consists of two parts: a
key and a value. If you know the key you can
access the value very quickly. The DBM file maintains an
'index' of the keys, each of which points to where the value
is stored within the file, and the index is usually arranged
such that values can be accessed with the minimum number of
file system accesses even for very large numbers of keys.

In practice, on many systems a DBM 'file' is actually stored
in two files on the disk. If, for example, a DBM file called
'users' is created, it will actually be stored in files
called users.pag and users.dir. If you ever
need to rename or delete a DBM from the command line,
remember to change both the files, keeping the
extensions (.pag and .dir) the same. Some newer versions of
DBM only create one file.

Provided the key is known in advance DBM format files are a
very efficient way of accessing information associated with
that key. For web user authentication, the key will be the
username, and the value will store their (encrypted)
password. Looking up usernames and their passwords in a DBM
file will be more efficient than using a plain text file when
more than a few users are involved. This will be particularly
important for sites with lots of users (say, over 10,000) or
where there are lots of accesses to authenticated pages.

Preparing Apache for DBM Files

If you want to use DBM format files with Apache, you will
need to make sure it is compiled with DBM support. By
default, Apache cannot use DBM files for user authentication,
so the optional DBM authentication module needs to be
included. Note that this is included in addition to
the normal user authentication module (which uses plain text
files, as explained in the previous article). It is possible
to have support for multiple file formats compiled into
Apache at the same time.

To add the DBM authentication module, edit your Configuration
file in the Apache src directory. Remove the comment from the
line which currently says

# Module dbm_auth_module mod_auth_dbm.o

To remove the comment, delete the # and space character at
the right-hand end of the line. Now update the Apache
configuration by running ./Configure, then re-make
the executable with make.

However, before compiling you might also need to tell Apache
where to find the DBM functions. On some systems this is
automatic. On others you will need to add the text
-lndbm or -ldbm to the EXTRA_LIBS line in
the Configuration file. (Apache 1.2 will attempt to do this
automatically if needed, but you might still need to
configure it manually in some cases). If you are not sure
what your system requires, try leaving it blank and
compiling. If at the end of the compilation you see errors
about functions such as _dbm_fetch() not being found, try
each of these choices in turn. (Remember to re-run
./Configure after changing Configuration). If you
still cannot get it to compile, you might have a system where
the DBM library is installed in a non-standard directory, or
where the there is no DBM library available. You could either
contact you system administrator, or download and compile
your own copy of the DBM libraries (a good choice might be
GDBM:
read about it or download it).

Creating A DBM Users File

For standard (htpasswd) user authentication password files,
the program htpasswd is used to add new users and set their
passwords. To create and manage DBM format user files another
program from the Apache support directory is used. The
program is called dbmmanage and is written in perl (so
you will need perl on your system, and it will need to have
been compiled with support for the same DBM library you
compiled into Apache. If you have only just installed DBM on
your system you will might need to re-compile perl to build
in DBM support).

This program can be used to create a new DBM file, add users
and passwords to it, change passwords, or delete users. To
start by creating a new DBM file and adding a user to it, run
the command:

dbmmanage /usr/local/etc/httpd/usersdbm adduser martin hamster

The creates the DBM file /usr/local/etc/httpd/usersdbm (which
might actually consist of /usr/local/etc/httpd/usersdbm.dir
and /usr/local/etc/httpd/usersdbm.pag), if it does not
already exist. It then adds the user 'martin' with password
'hamster'. This command can be used with other usernames and
passwords to add more users, or with an existing username to
change that user's password. A user can be deleted from the
password file with

dbmmanage /usr/local/etc/httpd/usersdbm delete martin

You can get a list of all the users in the DBM file with

dbmmanage /usr/local/etc/httpd/usersdbm view

Restricting a Directory

Now you have a DBM user authentication file with some users
in it, you are ready to create an authenticated area. You can
restrict a directory either using a <Directory> section
in access.conf or by using a .htaccess file. The feature on
user authentication explained how you can set up a basic
.htaccess file, using this example:

To use DBM files, the only change is to replace the directive
AuthUserFile line with

AuthDBMUserFile /usr/local/etc/httpd/usersdbm

This single change tells Apache that the user file is now in
a DBM format, rather than plain text. All the rest of the
user authentication setup remains the same (so the
authentication type is still Basic, and the syntax of
require is the same as before).

Using Groups

Each user can be in one or more "groups", and you can
restrict access to people just in a specified group. This
makes it possible to manage all your users on your site in a
single database, and customise the areas that each can
access. The use of DBM files for storing group information is
particularly efficient because you can use the same file to
store both password and group information.

The dbmmanage command can be used to set group
information for users. For example, to add the user "martin"
to the group "staff", you would use

dbmmanage /usr/local/etc/httpd/users adduser martin hamster staff

You put a user into multiple groups but listing them,
separated by commas. For example,

Note that dbmmanage has to be told the password as well, and
there is no way to set or change group information for a user
without knowing their password. This means in practice that
dbmmanage is not suitable for managing users in groups, and
you will have to write your own management scripts. Some help
writing perl to manage DBM files is given later in this
article.

After creating a user and group file containing details of
which users are in which groups, you can restrict access by
these groups. For example, to restrict access to an area to
only people in the group staff, you could use:

Custom Management of DBM Files

The supplied dbmmanage script to manage DBM files is adequate
for basic editing, but cannot handle advanced use, such as
managing group information. It is also command line driven,
while a Web interface might be a better choice in many
situations. To do either of these things you will have to
write programs to manage DBM files yourself. Using perl this
is not too difficult.

As a simple example, say you have an existing .htpasswd file
and you want to convert it to a DBM file, putting all the
users in a specific group. We will introduce the concepts
here, and there is a link below to the completed program for
you to download. It will be written in Perl which is quick to
write and easy to customise, although the principles of DBM
use are the same whatever language is used.

The basic way to look in a DBM file is given here. DBM files
are opened in Perl as 'hashed arrays'. The "key" is the user
name, and the value is the encrypted password and optionally
group information. A simple script to lookup all the keys and
values in a DBM is:

Note that if the given DBM file does not exist, it will be
created. This script will work with both perl 4 and perl 5
(although Perl 5 users might prefer to use the new tie
facility instead of dbmopen). To lookup a known key you would
use:

Now we can write a script to convert a htpasswd file into a
DBM database, optionally putting each user into one or more
groups. The script is htpasswd2dbm.pl,
and is used like this:

cd /usr/local/etc/httpd
htpasswd2dbm.pl -htpasswd users usersdbm

The -htpasswd option specifies the htpasswd file to be read,
the the final argument is the DBM file to create (or add to).
To set a group, use the -group argument. For example, to put
all the users from this file into the groups admin and staff,
use

htpasswd2dbm.pl -htpasswd users -group admin,staff usersdbm

The program will add users to an existing DBM database, so it
can be used to merge multiple htpasswd files. If you give
users from different files different groups, you will be able
to set up access restrictions on a group-by-group basis, and
manage all your users in one database. Note that if there is
already a user with the same username in the DBM file it will
be overwritten by the new information.

Group information stored in a DBM file as part of the value.
If no group information is stored, the key associated with a
username just consists of the encrypted password. To store
group information, the encrypted password is followed by a
colon, then a list of groups that the user is in, each
separated by a comma. So a typical key might look like this:

E7yT67YGht65:admin,staff

A program written in perl can easily extract the group
information, for example:

It is also possible to store additional information in the
DBM file, by following the groups list with a colon. Apache
will ignore any data after a colon following the groups list,
so it could be used, for example, to store the real name and
contact details for the user, and an expiry date. This could
be stored in the DBM like this: