Of course the first thing to do in order to be able to replication our DIT is to have another CentOS machine. So go ahead and install it on a seperate computer. We will continue with our example two machines : alice and bob. Alice is the current OpenLDAP server while bob was the client. At the end of this document, bob will be the second OpenLDAP server. Which in OpenLDAP syncrepl parlance, we have these entities :

provider : alice.company.com (a.k.a. master server)

consumer : bob.company.com (a.k.a. replica server)

Another important thing to do is to read the OpenLDAP replication chapter in the administrator's guide. The following exerpt is of particular interest :

Syncrepl supports both pull-based and push-based synchronization. In its basic refreshOnly synchronization mode, the provider uses pull-based synchronization where the consumer servers need not be tracked and no history information is maintained. The information required for the provider to process periodic polling requests is contained in the synchronization cookie of the request itself. To optimize the pull-based synchronization, syncrepl utilizes the present phase of the LDAP Sync protocol as well as its delete phase, instead of falling back on frequent full reloads. To further optimize the pull-based synchronization, the provider can maintain a per-scope session log as a history store. In its refreshAndPersist mode of synchronization, the provider uses a push-based synchronization. The provider keeps track of the consumer servers that have requested a persistent search and sends them necessary updates as the provider replication content gets modified.

Replication is handeled by an OpenLDAP overlay. Check the slapo-syncprov(5) man page for the provider overlay information. With that in mind, we will setup a refreshAndPersist replication using the delta-syncrepl replication scheme. Note that, as the official documentation says :

As you can see, you can let your imagination go wild using Syncrepl and slapd-ldap(8) tailoring your replication to fit your specific network topology.

Provider Configuration

Setting up delta-syncrepl requires configuration changes on both the master (i.e. provider) and replica (i.e. consumer) servers. We will start by configuring the provider machine (i.e. alice.company.com) and then continue to the consumer machine (i.e. bob.company.com).

Provider Module Configuration

To configure a module on the provider, we first need to check if we have one? Since we have a Kerberos realm and SASL GSSAPI authentication setup, let's use this to simplify our queries. Note that you can always use the cn=admin,dc=company,dc=com RootDN to perform all the tasks in this blog post, but the queries are longer to write.

The above query did not return the olcModulePath object. So we need to create it. But what is our module path? The cn=module documentation shows us that module names end in « .la ». With that info, a simple rpm query will show us where they're stored on the filesystem.

rpm -ql openldap-servers | grep '\.la$'

/usr/lib/openldap/accesslog.la

/usr/lib/openldap/auditlog.la

/usr/lib/openldap/collect.la

/usr/lib/openldap/constraint.la

/usr/lib/openldap/dds.la

/usr/lib/openldap/deref.la

/usr/lib/openldap/dyngroup.la

/usr/lib/openldap/dynlist.la

/usr/lib/openldap/memberof.la

/usr/lib/openldap/pcache.la

/usr/lib/openldap/ppolicy.la

/usr/lib/openldap/refint.la

/usr/lib/openldap/retcode.la

/usr/lib/openldap/rwm.la

/usr/lib/openldap/seqmod.la

/usr/lib/openldap/smbk5pwd.la

/usr/lib/openldap/sssvlv.la

/usr/lib/openldap/syncprov.la

/usr/lib/openldap/translucent.la

/usr/lib/openldap/unique.la

/usr/lib/openldap/valsort.la

Ok, so we know our olcModulePath is /usr/lib/openldap. We also know that we have quite a bunch of different modules available. Let's write another LDIF file to set the olcModulePath, but also which modules we want to load. Since we need both the accesslog and the syncprov overlays, we might as well load them right?!

This user needs a real password. Make sure to record this password and keep it safe.

ldappasswd -xZ -S cn=replication,dc=company,dc=com

Note that in this example our replication user is named just « replication ». If you plan on having more than one replicated server, then ideally choose a unique name for each of the replicated server. This way you can audit which machine replicates and when. You can also decide which part of the DIT gets replicated to which machine. But that's another story.

Provider Limits and ACLs to the Replication User

We now need to give access via ACLs and some limits to the new user. Let's do it in two steps starting with the limits (or lack of actually :)

Next step is to give read access to the replication user. Before we do this, it's usually a good idea to double-check our current ACL list.

ldapsearch -LLLb cn=config olcAccess

We can now create another LDIF file to update our ACLs. The idea here is to have the same sets of ACLs on both the provider and the consumer. Except that on the consumer, we don't need any ACLs with regards to the replication user (i.e. the one defined in olcSyncRepl:).

This is of course crucial. Do not continue with this blog post until you can access the entire DIT with the replication DN. Once it works, we need to setup our consumer OpenLDAP machine.

Consumer Configuration

Consumer OpenLDAP Installation

Install another CentOS 6 machine and connect to it.

ssh bob.company.com

Make sure that ntpd(8) is running and that both alice and bob's clocks are synchronized. All of your servers' clocks must be tightly synchronized using either NTP(See http://www.ntp.org/ for info on NTP), an atomic clock or some other reliable time reference.

Configure an empty OpenLDAP server. I've already covered how to do this in a previous blog post, but we now need to change this a little. So we can't really follow the other post. Let's do it all over again then. So the first thing we must do is upgrade the sudo package to have the latest sudo schema. I've explained how to achieve this goal in another blog post. But I'll show here just what we need which starts by installing the latest (as of this writing) sudo package from the sudo website. Note that if your machine is 32 bit, then the package would be sudo-1.8.4-5.el6.i386.rpm.

Configure a system-wide LDAP client configuration. This is to simplifiy our life and reduce typing later on. Don't worry about the TLS configurations for now. We will configure them later, but it doesn't hurt to have them in the file at the moment. Don't forget that in this blog post, the OpenLDAP server's FQDN is bob.company.com and notalice.company.com as we used to. This is important because here we don't want to query nor modify the provider (i.e. alice.company.com), but only the consumer (i.e. bob.company.com).

Create the OpenLDAP kerberos keytab, the the host's principal key and the autofsclient key while we're in the kadmin dialogue. IMPORTANT : notice how we place the host/ and autofsclient/ principals in the /etc/krb5.keytab while the ldap/ principal gets stored in /etc/openldap/krb5.keytab.

Fix permissions on the new Kerberos keytab. The goal here is to let the ldap group be able to read the ldap/ principal stored in the keytab.

sudo chown root:ldap /etc/openldap/krb5.keytab

sudo chmod 640 /etc/openldap/krb5.keytab

Restart the slapd(8) daemon.

sudo /etc/init.d/slapd restart

Test the SASL GSSAPI authentication.

kdestroykinit -p drobilla/admin@COMPANY.COMldapwhoami

Consumer DIT Initial Load

One last step before we start the replication is to load the DIT from our provider into our consumer. It's not required, but if your DIT is large, this will save quite a lot of time. Do to this, follow these steps :

Connect to the provider.

ssh alice.company.com

Create a new LDIF file with the entire DIT of the provider.

sudo slapcat | tee -a /tmp/provider.slapcat.ldif

Transfer the LDIF file over to the consumer.

scp /tmp/provider.slapcat.ldif bob.company.com:/tmp

Remove the temporary file. After all, it's the entire DIT that's in there in clear text...

Load the provider LDIF file into the consumer's database. Notice the « -w » switch to slapadd(8C) which will write syncrepl context information. Once all entries are added, the contextCSN will be updated with the greatest CSN in the database. That's pretty handy in our case :)

sudo slapadd -l /tmp/provider.slapcat.ldif -w

Again, remove the temporary file. After all, it's the entire DIT that's in there in clear text...

sudo rm /tmp/provider.slapcat.ldif

Start the consumer daemon.

sudo /etc/init.d/slapd startCheck to see if the entire DIT is now on the consumer machine?ldapsearch -xZWD cn=admin,dc=company,dc=com -b dc=company,dc=com -H ldap://bob.company.comCheck to see if the SASL GSSAPI user has the proper ACLs to this DIT on the consumer?
kinit -p drobilla/admin@COMPANY.COMldapsearch -ZLLLb dc=company,dc=com -H ldap://bob.company.com

Replication Test

We now have a TLS and SASL GSSAPI enabled OpenLDAP server bob.company.com configured as the consumer of our provider alice.company.com machine. Let's see if it works? Do test this, connect to the provider and change something. In this example, we will change our test.user's shell.

Connect to the provider server.

ssh alice.company.comCheck the current value for loginShell.ldapsearch -LLLZb cn=test.user,ou=users,dc=company,dc=com loginShell

loginShell: /bin/bash

Change the loginShell value to /bin/sh.

ldapmodify <<-EOF

dn: cn=test.user,ou=users,dc=company,dc=com

changetype: modify

replace: loginShell

loginShell: /bin/sh

EOF

Keep an eye on the slapd.log file. You should now see these lines on the consumer :

Consumer Kerberos Slave Setup

The whole point of replicating our DIT is to have two copies of it should the provider fails. But our DIT also supports our Kerberos infrastructure. We must thus make sure that our consumer can also act as a Kerberos slave should our provier -and Kerberos master- fails.

To enable the consumer machine to become a Kerberos slave, we must of course install the required packages and we already did that (see above). We must then send several provider's Kerberos files over to our consumer. Those files are the Kerberos private key and the Kerberos stash keyfile.

Make sure the krb5kdc daemon is running when the machine boots. Note that we do not run the kadmin daemon on the slave KDC.

sudo chkconfig krb5kdc on

Start the krb5kdc daemon.

sudo /etc/init.d/krb5kdc start

Consumer Backup

Don't forget to backup the consumer now that it's working. See this blog post on how to do just that.

Check Replication Status

Sometimes the log files are not clear in order to determine if the replication is finished or not. To check this, simply query both the provider and the consumer servers for the contextSCN. If the values are the same, then replication is finished.

As you can see, this requires quite a few changes. So you should probably script these changes. I usually setup an admin server that runs Apache with configuration files on it. Clients can thus simply wget them. Easy. Or you can use Puppet. Even better!

With the new client configuration, try shutting down the LDAP and Kerberos services on alice.company.com and see if the clients can still work by using bob.company.com.

Troubleshooting

olcLogLevel

If you're not sure about the syncrepl engine, then enable logging for this.

syncrepl_message_to_entry

It means that you're missing a schema in on the consumer server. Of course the rid=000 can be different on your server. It's the replication ID configured in the olcSyncrepl: config of the consumer server. Compare the schemas on both machines and fix the consumer so that it has exactly the same schemas as the provider. So, the first thing you must do is check the schemas on the provider :

ssh alice.company.com

ldapsearch -ZLLLb cn=schema,cn=config dn

Then connect to the consumer and check the schemas there :

ssh bob.company.com

ldapsearch -ZLLLb cn=schema,cn=config dn

If the consumer is missing some schemas present on the provider, then add those missing schemas to the consumer and try the replication again.

But when you look into the /var/lib/ldap/accesslog directory, there is a DB_CONFIG file and the permissions are good.

What's the problem?

Well, it's simply that the HDB has the wrong « objectClass: olcDdbConfig ». It should be « objectClass: olcHdbConfig ». Notice the small, but very critical, difference?

That's it!

That means we finished our initial goals :

Install OpenLDAP 2.4.

Configure Transport Layer Security (TLS).

Manage users and groups in OpenLDAP.

Configure pam_ldap to authenticate users via OpenLDAP.

Use OpenLDAP as sudo's configuration repository.

Use OpenLDAP as automount map repository for autofs.

Use OpenLDAP as NFS netgroup repository again for autofs.

Use OpenLDAP as the Kerberos principal repository.

Setup OpenLDAP backup and recovery.

Setup OpenLDAP replication.

The next set of goals are coming. I want to enable Referential Integrity in the LDAP DIT (i.e. when you delete a user it is also deleted from the various groups he's part of). I'm also interested in pulling information, such as users, passwords and grous from Active Directory servers and thus remove Samba.

HTH, Let me know if I made a mistake as this post was not easy to put together.

Now, here's a rundown of what I *think* might happen for those ideas. Recall that I haven't tested any of them.

a) DNS CNAME.

Create a CNAME in your DNS, say ldap.company.com. which points in a round-robin fashion (search the web for « bind dns round robin configuration » and see what I mean). It would work. But the various problems I can see are :

1. What if one LDAP server is down? The DNS resolver on the client will have cached the IP of your CNAME and will thus always retry that IP. Even if your timeouts in your client's ldap.conf file are low, the DNS will always return the same IP for the duration of the DNS TTL. This could cause quite a bunch of frustration.

2. TLS. How will you configure TLS when the IP of your CNAME changes? Not only that, but the hostname of your various LDAP servers will like as not be the same. So certificate verification will result in an error on the client side. That means you have to configure clients not to check certificates, which reduces your overall site's security.

b) Very short timeout values.

This could probably work. It doesn't break TLS and if a server is down, the clients will see it. But it would mean testing and testing to get the right time out values.

c) Configure LVS in front of OpenLDAP servers.

See LVS if you don't know that great software yet! http://www.linuxvirtualserver.org/I've successfully deployed LVS for Oracle database servers and Tomcat application servers. But in both cases, neither were using TLS. Again, LVS would be great to test. But the TLS issue would still nag me. At least if an OpenLDAP server is down, then the LVS cluster would not send it's IP to the clients.

d) Use hardware load balancers.

Well, one would need hardware load balancers to test this. I don't. And I haven't yet seen an Open Source hardware load balancer :) Maybe the Elfiq link load balancers would cut it? See http://www.elfiq.com/ for more info.

Thank you for providing this how-to guide for OpenLDAP on CentOS 6.2, it has been very helpful.

I have been tasked with the duty of setting up a replication of my employers current Red Hat 6 OpenLDAP server,and have been following your guide, but I have run into a bit of a brick wall.

For testing purposes, I have been working on a VM installation of CentOS 6, and have been mostly successful, until I reached the "Provider Overlays On Primary Database" section in the last bullet point.

Someone had mentioned before that you forgot to include a ldapmodify command for that section, but he apparently didn't follow up with the exact location of the missing command.

As mentioned it is in the "Provider Overlays On Primary Database" section, where you stated:

"Do do this objective, [ You said do do ;-) ] we must of course write another LDIF file in which we will a) setup new indexes to our primary database, b) add the syncprov overlay and c) add the accesslog overlay."

You then listed the vi command "vi ~/ldap/overlay.primary.ldif"

And then stated:

"We then should have new overlays on the primary database."

It's after the vi command that you forgot to include the ldapmodify command.

I'm not running a Kerberos server either, so I have been using the simple bind method with my Root DN. I attempted to add the "overlay.primary.ldif" this way, but the ldif you have posted appears to have a syntax error in it, that I tried to correct.

The commented out dc=caprion,dc=com part, I presume was suppossed to be dc=company,dc=com, referring to the main database.

But in the next section, you forgot to comment out the first line:"Add the accesslo overlay on the dc=company,dc=com database.", which I did add a comment to my ldif file, so the two lines are commented out.# Add the accesslo overlay on the dc=company,dc=com database.# scan the accesslog DB every day, and purge entries older than 7 days

I then ran this command to add the ldif file:ldapmodify -xWD cn=Manager,dc=company,dc=com -af ~/ldap/overlay.primary.ldif - NOTE: I'm using Manager instead of admin.

This appeared to add the ldif file, but didn't appear to add the syncprov or accesslog sections of the overlay.primary.ldif file.

When I run the search command it returns only the following:ldapsearch -xLLLWD cn=Manager,dc=company,dc=com -b olcDatabase={1}bdb,cn=config dnEnter LDAP Password: dn: olcDatabase={1}bdb,cn=config

Was the command I used to add the ldif incorrect, or is there something missing in the ldif file?

I'm not real familar with the ldap commands, but I used what I thought was correct.

Make sure you installed the "overlay.primary.ldif" to the right OpenLDAP server. In other words, this LDIF file is for the primary OpenLDAP machine and NOT for the slave one. To be totally sure of which machine you're working on, use the -H flag to any ldap commands. In your case, try this (replace master.company.com by your own master server of course) :

If not, then we have to investigate further. If you see those lines, then you may have some ACL issues preventing you from seeing this. But since you're working as the RootDN, this should not be a problem as the RootDN always has access to the data.

You could also make sure that your database is {1} and not {2} or something else. To list all the olcDatabase configured in your server, try this :

DavidI followed this to setup between Debian slapd 2.4.23-7.2 and CentOS openldap-servers-2.4.23-26.el6_3.2.i686 both using cn=config method. All was perfect until the actual replication. The initial sync of the cn=admin,dc=example,dc=com conflicts with the core schema - that says for a simpleSecurityObject a MUST for the userPassword. You can create and LDIF with a MAY for the initial sync ( when slave DB empty ) then reset to MUST after. The specific olcObjectClasses is ( don't forget to include all of them from /etc/openldap/schema/core.ldifolcObjectClasses: ( 0.9.2342.19200300.100.4.19 NAME 'simpleSecurityObject' DESC 'RFC1274: simple security object' SUP top AUXILIARY MAY userPassword )This only is required for whole initial sync from provider.

BTW - the write up is perfect otherwise. It may just have been my stuff but you may want to just put this in as a now

I'm racking my brain here trying to get past applying the limits.ldif file to the master. I did enough of the Kerberos portion so as to enable users to change their LDAP passwords but didn't proceed any further-- not sure if that's part of the problem or not-- it doesn't seem like it should interfere. Anyway, without a KDC up and running I tried applying the limits.ldif like so:

I looked around on Google and there was mention of the ldif file needing white space between dn entries, which yours has. I even tried splitting them into separate ldif files and no luck with either. It seems to me like it doesn't even understand what olcLimits even is? Or perhaps it's choking on one of its attributes? Any help would be greatly appreciated-- I've absolutely LOVED this series of tutorials and can't thank you enough!

Well it's a perfect day for egg on my face... Did some googling around and wound up here: http://support.apple.com/kb/TS4462?viewlocale=en_US and I thought "What the Hell, why not throw a changeType: modify into each of the entries in the limits.ldif file" and it worked! So I used the following command:

Some pretty cool stuff! I got replication working but the wheels fall off when it comes to the Kerberos replication and I have it narrowed down it being an ACL problem. Would you be so kind as to post an expected ACL dump of both alice and bob when replication is all done? That would definitely help me understand this stuff better. Again, great work!

Unfortunately, my OpenLDAP machines are located in a corporation for which I don't work for anymore. So I'll have to rebuild my entire setup from scratch! How funny can that be hey? But I'm supposed to go there next week for a brain dump to the person taking my old position. I'll try to do an ACL dump on both master and slave.

Ugh that sucks! But rebuilding can help catch things and refine them further. I've always kept in the back of my mind just how one could use puppet to augment some of this but haven't gotten to that point of tinkering with it yet. I'll keep sloggin' away and see if I can figure it out. Staying tuned.

I have case which i build many master server to replicate to one slave ldap server.Master server have dynamic ip.So not possible for me to pointing ldap provider from ldap slave server.So i want to configure provider to pointing their ldap server is.

Do you have a Kerberos realm setup? If so, then make sure you do a « kinit -p user/admin@YOUR.REALM » before you try the « ldapmodify command.

If you don't have Kerberos, then try to authenticate to your OpenLDAP server using a different credential. Like the Admin user with -WD cn=admin,dc=example,dc=org or with the super-user with something like « sudo ldapmodify -Y EXTERNAL -H ldapi:/// -f module.ldif » instead.