Right now I'm just testing SSL with a self-signed certificate, but one
potentially serious issue has come up. When ever I restart OpenLDAP I
get the prompt "Enter PEM pass phrase". This means that the OpenLDAP
server can auto-start on reboot. Is there anyway to bypass that? In
apache, for example, SSLPassPhraseDialog has an option to execute a
program, and I use that option to supply the pass phrase. Is there any
kind of equivalent in OpenLDAP.

You should decrypt the SSL key so it doesn't require a password to read it.
But this is really a question about how to use OpenSSL, and not related to
OpenLDAP itself, so fairly off topic here.