Both the custom and agent interfaces can use various authentication methods, such as a local database, or Active Directory/LDAP. It is also possible to use external authentication (HTTPBasicAuth) in which case OTRS does not take responsibility for authentication any more, but instead relies on an Apache environment variable to provide the username. The is the way forward if you want to use SAML or federated authentication, but there are some issues with.
The biggest issue is that is not possible to provision accounts in OTRS before users have logged in. This is because there is no way of knowing a user's details until they have authenticated. To overcome this I wrote a new customer authentication module for OTRS that creates customer accounts on the fly (auto-provisioning).
At the moment we have no use case yet for auto-provisioning agents. This is left as a future exercise, one idea is to auto-provisioning agents based on the value of a specific SAML attribute.
The standard HTTPBasicAuth can be used for the agent interface.
Below is the recipe for getting OTRS to work with federated authentication using Ubuntu 14.04, OTRS 3.3.8 and mod_auth_mellon 0.7. If you manage to implement it on another combination of software, please let me know.

Prerequisites
Before you start, make sure you have these bits in place:

An attribute that can be used as username in OTRS (for example eduPersonPrincipalName). Attributes for first name, last name, and e-mail are optional but highly recommended as the service would be pretty useless without these. In this case we assume that 'givenName', 'sn', and 'mail' can be used.

The user name of the to-be administrator account. So, if you choose eduPersonPrincipalName as the attribute for username, you need to know your own value (for instance 'dvisser@surfnet.nl').

OTRS
Go to https://www.otrs.com/try/, scroll to Source, and pick the latest version of OTRS Help Desk.
Follow the instructions at http://otrs.github.io/doc/manual/admin/ ... index.html, do a standard install and make sure everything works.
Pay special attention to the phrase "Please install OTRS from source, and do not use the OTRS packages that Debian/Ubuntu provides."
It will require some fiddling to get all the Perl modules sorted, I suggest to use the Ubuntu packaged modules as much as possible.

The docs all seem to assume that you'd want to run OTRS inside a subdirectory (https://example.com/otrs), but we want it to be the root of our vhost (https://otrs.example.com), in which case this configuration is a little bit different, see below (you should have the HTTPS stuff already configured, probably in /etc/apache2/mods-enabled/ssl.conf):

The site is now configured so that the bare URL will go to the customer interface. This makes the most sense because typically customers will have less clue about where to go.
The agent interface is where you should log in to with the default root@localhost account.
Once you're in, you should create a new agent with full permissions, and make sure the username is your eduPersonPrincipalName.

As you can see, the attribute eduPersonPrincipalName is being used as the username. This is the attribute that should always be sent by the IdP. In other words, you have to make that authentication doesn't go through when there is an eduPersonPrincipalName. mod_auth_mellon populate the REMOTE_USER environment variable with the value of this SAML attribute.
Icon
Keep in mind that eduPersonPrincipalName might work well in a controller environment, such a a national identity federation, where the attribute is always present, validated, etc. It might not be so smart if you use a SAML proxy, in which case you could end up with different users with the same eduPersonPrincipalName.

Configuration
When creating a new customer, we also need data for several other fields: first name, last name, e-mail, and customID. Since mod_auth_mellon copies all the SAML attributes into Apache environment variables anyway, we can use them. The module uses the standard SAML attributes (prefixed with MELLON_ because that's their environment variable name) wherever possible.

You can override these values in your configuration (System/Config.pm) , which at the minimum looks like this:

# Customer Auth
$Self->{'Customer::AuthModule'} = 'Kernel::System::CustomerAuth::HTTPBasicAuthMellon';
# Because auto-provisioned users will all have the same e-mail address
$Self->{CustomerUser}->{CustomerUserEmailUniqCheck} = 0;
$Self->{'CustomerPanelLoginURL'} = 'https://otrs.example.com/mellon/login?ReturnTo=/customer.pl';
$Self->{'CustomerPanelLogoutURL'} = 'https://otrs.example.com/mellon/logout?ReturnTo=http://www.terena.org';
# Uncomment to override the environment vars to be used
#$Self->{'Customer::AuthModule::HTTPBasicAuthMellon::UsernameEnvVar'} = 'MELLON_eduPersonPrincipalName';
#$Self->{'Customer::AuthModule::HTTPBasicAuthMellon::MailEnvVar'} = 'MELLON_mail';
#$Self->{'Customer::AuthModule::HTTPBasicAuthMellon::FirstNameEnvVar'} = 'MELLON_givenName';
#$Self->{'Customer::AuthModule::HTTPBasicAuthMellon::LastNameEnvVar'} = 'MELLON_sn';
#$Self->{'Customer::AuthModule::HTTPBasicAuthMellon::CustomerIDEnvVar'} = 'MELLON_otrs_customer_id';
# Agents are NOT auto-provisioned. The will have to be created manually.
# To find their username, they could first log in as a customer, so that you can see their username
# in the Customer User Manager overview.
$Self->{'AuthModule'} = 'Kernel::System::Auth::HTTPBasicAuth';
$Self->{'LoginURL'} = 'https://otrs.example.com/mellon/login?ReturnTo=/index.pl';
$Self->{'LogoutURL'} = 'https://otrs.example.com/mellon/logout?ReturnTo=http://www.terena.org';

At this point, you should be able to log in to the site as an admin with your new account.
If you log in to the customer page, your account will be automatically created.

Known limitationsCustomers cannot edit their details. Once customers are auto-provisioned, they cannot edit their name, e-mail, or any other values. This might be an issue in your environment. Only agents can edit this.

Unknown agents get stuck in an authentication loop. When agents that don't have an account try to log in, they get stuck in an endless authentication loop. Another agent should first create a new agent account.
If the username isn't know yet, then the new agent could first log in to the customer interface. The existing agent can then see what username was used, and use this to create an agent account.