I opened the Linux Domain Identity, Authentication, and Policy Guide, tried to figure out which options would be useful and… I froze when I reached the section Determining which CA configuration to use. I had literally no idea what the documentation meant by “Server with an integrated IdM CA” or “Server without a CA“. But I had to choose something to start with.
In this blog post, I will explain what this choice really means, and what are the consequences of picking one over the other.

Basic requirement: HTTP and LDAP server certificates

First of all, FreeIPA is composed of many services accessed through the network, among which a LDAP server and an HTTP server. These 2 services can be accessed through a standard port (in clear) or through a SSL port, meaning that they both need a server certificate.

The HTTP and LDAP server certificates are needed during the installation, because the installer will put them in the right NSS database and configure their nickname and location for the HTTP and LDAP servers to find them. There are multiple ways to obtain server certificates, but one needs to understand first the basic notions around Public Key Infrastructure (PKI).

I will use a comparison with the delivery of a passport: in order to have a passport issued with your name, you need to provide official documents (for instance a birth certificate and a photo) to the government agency that will validate the documents, make sure that you are who you claim to be, and then issue the passport. The server certificate can be compared to the passport, that will later prove your identity to whoever trusts the government agency, and the government agency can be compared to the Certificate Authority.

So in order to obtain server certificates, it is possible to:

request certificates to an official Certificate Authority. Many commercial or non-profit companies provide this type of service (Verisign, Let’sEncrypt, GoDaddy etc…)

request certificates to a home-made Certificate Authority. It is possible to create a home-made self-signed Certificate Authority with tools such as certutil or openssl. The main difference with the previous method is that people are less likely to trust your home-made CA (it’s like asking them to accept a passport that was issued by a newly founded country not recognized yet by the rest of the world). Self-signed here means that the Certificate Authority passport is delivered by… the Certificate Authority itself!

install your own Certificate Authority with FreeIPA, that will sign the certificates needed by the HTTP and LDAP server.

The last option corresponds to a “Server with an integrated IdM CA” and has many advantages over the first options:

certificates have a limited lifetime and need to be renewed before they expire (otherwise the HTTP/LDAP servers stop working). The renewal process is time-consuming but also can be forgotten if the sysadmin does not carefully track the dates, leaving the deployment in a state where some services become unavailable.
When FreeIPA is installed with an embedded Certificate Authority, FreeIPA automatically monitors the expiry dates of the certificates and triggers a renewal a few weeks before expiration, ensuring service continuity. This is the first advantage of the embedded CA, but not the only one.

FreeIPA with an embedded CA is also able to deliver certificates for the users, the hosts or the services managed by FreeIPA. Various certificates profiles can be defined, populating specific fields in each type of certificate (for instance extensions with the OCSP responder URL…)
For more information on the embedded Certificate Authority, you can refer to Dogtag Certificate System documentation (the embedded CA in FreeIPA is a tailored version of Dogtag).

At this point, if you decide to install FreeIPA with an embedded Certificate Authority, there are 2 possible choices. The embedded CA can either be:

a self-signed CA: the Certificate Authority is created “from scratch” without the need for any external authority. It is the root CA, meaning that its own certificate was not delivered by anyone else but signed by itself.

a CA subordinate to an external CA. This means that FreeIPA CA certificate was signed by another CA, a sort of parent CA.

Corresponding installation options

CA-less installation

As said above, we need one certificate for the HTTP server and one for the LDAP server. They have to be provided to ipa-server-install or ipa-replica-install with the options:

Installation with an embedded self-signed CA

FreeIPA CA is created during FreeIPA installation, and generates the HTTP and LDAP certificates. There is no need to provide any cert file! No options!

Installation with an externally-signed embedded CA

The installation is a 2-step process. In the first step, ipa-server-install must be called with –external-ca and generates a CSR file (Certificate Signing Request). This CSR file needs to be sent to the external CA that will perform a bunch of validations to authenticate the recipient of the certificate and issue a certificate for FreeIPA Certificate Authority.

In the second step, ipa-server-install is called with –external-cert-file to provide the certificate obtained from the external CA. The installer then configures FreeIPA certificate authority as a subCA of the external CA, and FreeIPA CA can issue the HTTP and LDAP server certificates.

What if…

I installed FreeIPA without any embedded CA but I change my mind?

FreeIPA allows to install an embedded CA at a later time, using ipa-ca-install. The tool provides the same options as ipa-server-install: you can either install a self-signed CA or an externally signed CA.

Important: installing an embedded CA with ipa-ca-install does not replace the HTTP and LDAP server certificates. If they were initially delivered by an external CA, they will not be automatically renewed.

I installed FreeIPA with a self-signed CA but I’d rather have an externally-signed CA?

FreeIPA allows to switch from self-signed CA to externally-signed CA using ipa-cacert-manage renew –external-ca. This is a 2-step process similar to ipa-server-install –external-ca, where the 1st step produces a CSR that needs to be supplied to an external CA. The external CA then issues a CA cert that is provided back to ipa-cacert-manage renew through the –external-cert-file option.

I installed FreeIPA with an externally signed CA but I’d rather have a self-signed CA?

FreeIPA allows to switch from externally signed CA to self-signed CA using ipa-cacert-manage renew –self-signed.

I want to replace HTTP and LDAP certificates with certificates signed by a third-party Certificate Authority?

FreeIPA provides the ipa-server-certinstall tool that will replace the current HTTP or LDAP certificates with the certs provided in the file.Important: ipa-server-certinstall can be called to install externally signed certificates for HTTP and LDAP even if FreeIPA is installed with an embedded CA. In this case, FreeIPA CA still has the capability to issue certificates for users, hosts or services. The only difference is that HTTP and LDAP certificates are not issued by IPA CA.

Other FreeIPA commands related to certificates

When an embedded CA is installed, its certificate must be present in various files or NSS databases on all the FreeIPA hosts (master, replicas and clients) so that any FreeIPA machine trusts the certificates delivered by the embedded CA.

In addition to that, the HTTP and LDAP server certificates can be issued either by IPA CA or by an external CA, and the issuer can even be changed over time. In the external CA case, this means that the external CA needs to be trusted by all the FreeIPA machines for the IPA commands to work (the CLI communicates with the HTTP server using the https port, and this requires to trust the CA that issued the HTTP server certificate). Yet another CA certificate to add to files and databases on all the FreeIPA machines…

To ease this configuration, the tool ipa-certupdate is able to retrieve the CA certificates stored in LDAP (the embedded FreeIPA CA or the external CA certs), and install them in all the relevant files and NSS databases. It needs to be called when the CA cert is manually renewed or when a new external CA cert is added.

ipa-cacert-manage install is used to add a new external CA certificate in the LDAP store. It does not replace FreeIPA embedded CA but rather declares another certificate authority as trusted. This is useful when the HTTP and LDAP server certificates need to be replaced by certs signed by a new CA, not yet known by FreeIPA. After calling ipa-cacert-manage install (that puts the new CA in LDAP store), you need to call ipa-certupdate on all FreeIPA machines (to get the CA from the LDAP store and put it in the local NSS databases).

ipa cert-request is used to request new certificates for users, hosts or services. The certificate is signed by FreeIPA embedded CA (meaning that this command is available only when an embedded CA is configured).

Conclusion

By now you should be able to pick a deployment option and understand the differences between CA-less or with embedded CA, and self-signed CA or externally-signed CA.

You should also be aware that your choice is not definitive and that it is possible to install a CA at a later time or change your certificate chain from self-signed to externally-signed and vice-versa.

]]>https://floblanc.wordpress.com/2017/12/05/demystifying-the-certificate-authority-component-in-freeipa/feed/3floblancTroubleshooting FreeIPA: pki-tomcatd fails to starthttps://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcatd-fails-to-start/
https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcatd-fails-to-start/#commentsMon, 11 Sep 2017 15:05:53 +0000http://floblanc.wordpress.com/?p=1504Continue reading Troubleshooting FreeIPA: pki-tomcatd fails to start→]]>When performing the upgrade of FreeIPA, you may encounter an issue with pki-tomcatd failing to start. At first this issue looks related to the upgrade, but often reveals a latent problem and gets detected only because the upgrade triggers a restart of pki-tomcatd.

So how to troubleshoot this type of issue?

Upgrade logs

The upgrade is using /var/log/ipaupgrade.log and may contain a lot of useful information. In this specific case, I could see:

[...] DEBUG The ipa-server-upgrade command failed,
exception: ScriptError: CA did not start in 300.0s
[...] ERROR CA did not start in 300.0s
[...] ERROR The ipa-server-upgrade command failed. See
/var/log/ipaupgrade.log for more information

CA debug logs

The first step is to understand why pki-tomcatd refuses to start. This process is launched inside Tomcat and corresponds to the CA component of FreeIPA. It is logging into /var/log/pki/pki-tomcat/ca/debug:

Communication between pki-tomcatd and the LDAP server

We can see that pki-tomcatd is trying to open a LDAP connection through SSL but fails to authenticate. Within FreeIPA, pki-tomcat is storing data in the 389-ds LDAP server and needs to communicate through LDAP with this server.

The configuration of this communication channel can be read in /etc/pki/pki-tomcat/ca/CS.cfg:

The connection is using port 636 (SSL port) with SSL Client authentication (authtype=SslClientAuth). This means that pki-tomcatd provides a user certificate to the LDAP server, and the LDAP server maps a user to this certificate in order to authenticate the communications.

Note: Authtype can either be SslClientAuth or BasicAuth (authentication with username and password).

In this case, the SSL client authentication is done with the certificate named ‘subsystemCert cert-pki-ca‘ that is stored in /etc/pki/pki-tomcat/alias. So what could be causing the authentication to fail? We need to check that the certificate is available in /etc/pki/pki-tomcat/alias and that pki-tomcat is able to use the associated private key, and that the LDAP server is able to map this certificate to a user.

Check the subsystemCert cert-pki-ca

The first step consists in making sure that this certificate is present in /etc/pki/pki-tomcat/alias:

This means that when the LDAP server receives an authentication request with a certificate issued by the CA CN=Certificate Authority,O=IPADOMAIN.COM, it will look for users that contain a seeAlso attribute equal to the subject of the certificate, and the user entry must contain the certificate in the usercertificate attribute (verifycert: on).

With a default config, the ‘subsystemCert cert-pki-ca‘ is mapped to the user uid=pkidbuser,ou=people,o=ipaca. So let’s compare the user entry and the certificate:

The certificate in the userCertificate attribute is different from the one in the NSS database! This can also be seen by comparing the serial number with the value from the ldap entry (in description: 2;<serial>;<issuer>;<subject>):

This explains why pki-tomcat could not authenticate to the LDAP server. The fix consists in updating the LDAP entry with the right certificate (and do not forget to update the description attribute with the right serial number!):

But we still do not know the root cause for the inconsistency between the NSS database /etc/pki/pki-tomcat/alias and the user entry for uid=pkidbuser. This could be described in another blog post (for the impatients, the automatic renewal of the certificate failed to update the LDAP server entry…)

]]>https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcatd-fails-to-start/feed/1floblancTroubleshooting: mapping between a SmartCard certificate and an IdM userhttps://floblanc.wordpress.com/2017/06/02/troubleshooting-mapping-between-a-smartcard-certificate-and-an-idm-user/
https://floblanc.wordpress.com/2017/06/02/troubleshooting-mapping-between-a-smartcard-certificate-and-an-idm-user/#commentsFri, 02 Jun 2017 14:59:32 +0000http://floblanc.wordpress.com/?p=1387Continue reading Troubleshooting: mapping between a SmartCard certificate and an IdM user→]]>Authentication with a SmartCard may fail when the SmartCard certificate is not linked to any IdM user, or to a user different from the one specified on the console.

In order to find which user is associated to a given SmartCard certificate, you can run the following command:

If the result is not what you were expecting, you need first to check how certificates are mapped to users.

By default, a certificate is associated to a user when the user entry contains the full certificate in its usercertificate attribute. But this behavior can be modified by defining certificate mapping rules:

Once this is done, you may need to clear sssd cache to force SSSD to reload the entries before retrying ipa certmap-match:

ipaclient$ sudo sss_cache -E

Flexible mapping with certificate identity mapping rule

When the mapping is based on certificate mapping rules, the same tool ipa certmap-match can be used to check which user entry is associated to a certificate. When the result is not what you expect, you can enable sssd domain logs by adding the following in /etc/sssd/sssd.conf on IdM master:

[domain/ipadomain.com]
...
debug_level = 9

then restart sssd with

root@ipaserver$ systemctl restart sssd

The logs will be located in /var/log/sssd/sssd_ipadomain.log.

Check that the certificate identity mapping rules are properly loaded

When sssd is restarted, it reads the mapping rules and should print the following in /var/log/sssd/sssd_ipadomain.log:

Check that the matching rule corresponds to the certificate

When SSSD tries to associate the certificate to a user, it starts by finding which rule can be applied based on the matching rule (for instance rulesmartcard applies to all certificates issued by CN=Smart Card CA,O=EXAMPLE.ORG because its matching rule is <ISSUER>CN=Smart Card CA,O=EXAMPLE.ORG).

If no matching rule applies to the certificate, SSSD will not be able to associate the certificate with a user, and will display the following in /var/log/sssd/sssd_ipadomain.log:

In this case, you need to create or modify an identity mapping rule, so that the match rule applies to your certificate. See sss-certmap man page for the supported syntax of the –matchrule option of ipa certmaprule-add command.

Check that the expected certificate identity mapping rule is used

When SSSD tries to find the user associated to the certificate, you will see the following logs in /var/log/sssd/sssd_ipadomain.log:

]]>https://floblanc.wordpress.com/2017/06/02/troubleshooting-mapping-between-a-smartcard-certificate-and-an-idm-user/feed/4floblancTroubleshooting: authentication to the system console or Gnome Desktop Manager of an IdM host with a SmartCardhttps://floblanc.wordpress.com/2017/06/02/troubleshooting-authentication-to-the-system-console-or-gnome-desktop-manager-of-an-idm-host-with-a-smartcard/
https://floblanc.wordpress.com/2017/06/02/troubleshooting-authentication-to-the-system-console-or-gnome-desktop-manager-of-an-idm-host-with-a-smartcard/#commentsFri, 02 Jun 2017 13:55:23 +0000http://floblanc.wordpress.com/?p=1330Continue reading Troubleshooting: authentication to the system console or Gnome Desktop Manager of an IdM host with a SmartCard→]]>IdM allows to authenticate to an IdM enrolled-host by providing a SmartCard certificate instead of a username/password. The below steps are based on system console authentication but the process is similar when using Gnome desktop login authentication.

When the authentication fails, the issue usually comes from a wrong configuration of the IdM system for SmartCard, or of PKINIT.

Configuration of the IdM host for SmartCard authentication

If the console does not even prompt for the SmartCard PIN, chances are high that the system was not properly configured for SmartCard authentication.

SSSD configuration for smart card

Check that /etc/sssd.conf contains

[pam]
pam_cert_auth = True

If you need to update the file, do not forget to restart sssd with

root@ipaclient$ systemctl restart sssd

SmartCard CA must be trusted

Check that the SmartCard CA is trusted in the /etc/pki/nssdb database:

Configuration for PKINIT

If the console prompts for the SmartCard PIN but displays

ipaclient login: demosc1
Pin for PIV Card:
Login incorrect

it is possible that the authentication fails trying to acquire a Kerberos ticket with PKINIT. In this case, login with username/password to the IdM host and try to manually perform kinit in order to get more information:

The file defined in pkinit_anchors must exist, be readable and contain the certificate of the CA which signed the SmartCard certificate. If it is not the case, run the following commands on any IPA server:

]]>https://floblanc.wordpress.com/2017/06/02/troubleshooting-authentication-to-the-system-console-or-gnome-desktop-manager-of-an-idm-host-with-a-smartcard/feed/1floblancTroubleshooting: ssh to an IdM host with a SmartCardhttps://floblanc.wordpress.com/2017/06/02/troubleshooting-ssh-to-an-idm-host-with-a-smartcard/
https://floblanc.wordpress.com/2017/06/02/troubleshooting-ssh-to-an-idm-host-with-a-smartcard/#commentsFri, 02 Jun 2017 12:59:19 +0000http://floblanc.wordpress.com/?p=1260Continue reading Troubleshooting: ssh to an IdM host with a SmartCard→]]>IdM allows to perform ssh from a non-enrolled host into an IdM enrolled host, using Smart Card authentication instead of ssh authorized keys. The ssh command would be the following to log as demosc1 into the host ipaclient.ipadomain.com:

Configuration of the remote (IdM) host

When IdM is properly configured, ssh will prompt for the SmartCard PIN and authenticate the user. If there is an issue with the certificate, ssh will revert to another authentication type (private keys or username/password).

In this case, enable debug logs for ssh authentication on IdM host. Edit /etc/sssd/sssd.conf and add the following line in [ssh] section:

[ssh]
debug_level = 9

then restart sssd using

root@ipaclient$ systemctl restart sssd

The logs will be located on the IdM host in /var/log/sssd/sssd_ssh.log.

]]>https://floblanc.wordpress.com/2017/06/02/troubleshooting-ssh-to-an-idm-host-with-a-smartcard/feed/1floblancFreeIPA: troubleshooting SmartCard authenticationhttps://floblanc.wordpress.com/2017/06/02/freeipa-troubleshooting-smartcard-authentication/
https://floblanc.wordpress.com/2017/06/02/freeipa-troubleshooting-smartcard-authentication/#respondFri, 02 Jun 2017 12:12:58 +0000http://floblanc.wordpress.com/?p=1150Continue reading FreeIPA: troubleshooting SmartCard authentication→]]>RHEL 7.4 beta is now available, delivering a new version of IPA which contains the support for Smart Card authentication. This feature allows to use a certificate contained in a SmartCard in order to login to IdM WebUI, to ssh to an IdM-enrolled host, or to login to the console or Gnome Desktop Manager of an IdM-enrolled host.

This feature is really powerful but may also seem difficult to troubleshoot. I will explain where to look for additional information when authentication fails, and how to fix the most common issues.

The information is split into posts specific to each authentication method:

]]>https://floblanc.wordpress.com/2017/06/02/freeipa-troubleshooting-smartcard-authentication/feed/0floblancUsing Certmonger to track certificateshttps://floblanc.wordpress.com/2016/12/06/using-certmonger-to-track-certificates/
https://floblanc.wordpress.com/2016/12/06/using-certmonger-to-track-certificates/#commentsTue, 06 Dec 2016 13:17:07 +0000http://floblanc.wordpress.com/?p=709Continue reading Using Certmonger to track certificates→]]>When FreeIPA is installed with an integrated IdM CA, it is using certmonger to track and renew its certificates. But what does this exactly mean?

When the certificates are reaching their expiration date, certmonger detects that it needs to renew them and takes care of the renewal (request a renewed certificate, then install the new certificate at the right location and finally restart the service so that it picks up the new certificate). It means that the system administrator does not need to bother anymore with renewals!

Well… When everything works well it is really a great functionality. But sometimes a small problem can prevent the renewal and FreeIPA ends up with expired certificates and HTTP or LDAP services refusing to start. In this case, it is really difficult to understand what has gone wrong, and how to fix the issue.

In this post, I will explain what is happening behind the scene with certmonger, so that you understand where to look for if you need to troubleshoot.

Certmonger concepts

Certmonger daemon and CLI

Certmonger provides 2 main components:

the certmonger daemon that is the “engine” tracking the list of certificates and launching renewal commands

the command-line interface: getcert, that allows to send commands to the certmonger daemon (for instance request a new certificate, list the tracked certificates, start or stop tracking a certificate, renew a certificate…)

Certificate Authority

Certmonger provides a generic interface allowing to communicate with various certificate systems, such as Dogtag, FreeIPA… A simple definition for Certificate System would be a software solution able to deliver certificates. This allows to use the same certmonger command independently of the Certificate System that will actually handle the request. The getcert command just reads the additional argument -c to know with which Certificate authority to interface.

Then certmonger needs to know how to interface with each type of Certificate System. This is done by defining Certificate Authorities that can be listed with:

Each section starting with ‘CA’ defines a type of Certificate Authority that certmonger knows to handle. The output of the command also shows a helper-location, which is the command that certmonger will call to discuss with the Certificate Authority. For instance:

Certificate renewal

When a certification is near its expiration date, certmonger daemon will automatically issue a renewal command using the CA helper, obtain a renewed certificate and replace the previous cert with the new one.

It is also possible to manually renew a certificate in advance by using the command getcert resubmit -i <id>, where <id> is the Request ID displayed by getcert list for the targetted certificate. This command will renew the certificate using the right helper command.

Start/Stop tracking a certificate

The commands getcert start-tracking and getcert stop-tracking enable or disable the monitoring of a certificate. It is important to understand that they do not manipulate the certificate (stop-tracking does not delete it or remove it from the NSS database) but simply add/remove the certificate to/from the list of monitored certificates.

Pre and post-save commands

When a certificate is tracked by certmonger, it can be useful to define pre-save and post-save commands that certmonger will call during the renewal process. For instance:

Output of getcert list

It is possible to check the status for each certificate using getcert list:

when the certificate is still valid, the status should be MONITORING.

when the certificate is near its expiration date, certmonger will request its renewal and the status will change from MONITORING to SUBMITTING and finally back to MONITORING (you may also see intermediate status PRE_SAVE_CERT and POST_SAVE_CERT).

When the renewal fails, getcert list will also show an error message. It will help determine which phase failed, and from there you will need to check the logs specific to the CA helper or to the pre-save or post-save commands.

In the next post, I will detail the errors that can arise with the helpers used with FreeIPA.

]]>https://floblanc.wordpress.com/2016/12/06/using-certmonger-to-track-certificates/feed/5floblancUsing a Dogtag instance as external CA for FreeIPA installationhttps://floblanc.wordpress.com/2016/09/02/using-a-dogtag-instance-as-external-ca-for-free-ipa-installation/
https://floblanc.wordpress.com/2016/09/02/using-a-dogtag-instance-as-external-ca-for-free-ipa-installation/#commentsFri, 02 Sep 2016 12:29:41 +0000http://floblanc.wordpress.com/?p=674Continue reading Using a Dogtag instance as external CA for FreeIPA installation→]]>A FreeIPA user recently had issues installing FreeIPA with an external CA. He was using Dogtag certificate system as external CA and FreeIPA installation was failing, complaining about the certificate provided by Dogtag.

So I decided to try the same deployment and share my findings in this post.

A little background…

FreeIPA server can be configured to act as a Certificate Authority inside FreeIPA IDM domain. It will then be able to create the certificates used by the LDAP server, the Apache server used for the Web GUI or the users and hosts.

This CA can be set-up in different ways:

The CA is a root CA, meaning that its certificate is self-signed

or the CA is subordinate to an external, 3rd-party CA, meaning that its certificate is signed by the 3rd party CA.

There are a wide range of products that can be used as 3rd-party CAs, among which Dogtag certificate system. In this blog post, I will explain how Dogtag can provide the certificate for IPA CA.

Instructions

The following instructions apply to Fedora 24. They will:

run the 1st step of ipa-server-install to generate a CSR

submit the CSR to Dogtag and have Dogtag issue a certificate for FreeIPA server

run the 2nd step of ipa-server-install with the certificate obtained in step 2.

Generation of the certificate using Dogtag

We then need to copy this CSR on the Dogtag instance and submit the CSR, approve it and export the certificate.

The submission is an important step as it allows to specify a profile. Basically, if we pick caCACert profile, we signal our intent to use the produced certificate as a Certificate Authority in our FreeIPA deployment, and the resulting certificate will contain the required extensions: