I am writing a an instant messenger program for my Networks class. I plan to use a block cipher to encrypt the contents of the messages for transit over the internet. During my kernel config I saw options for the linux kernel api in userspace. Can some one explain to me or point me to documentation on how I can call the kernel crypto api function from a C or C ++ program?

The Kernel crypto API is mainly for things like disk-encryption that happen inside the kernel (modules). Don't know for sure, but I'd expect any "userspace" options related to the crypto API are targeted toward similar applications like FUSE modules. For a pure userspace application it's better to use a userspace library like openssl for encryption for several reasons (portability, performance, compatibility, debugging options, ...).

A more common use of those kernel routines is, say, VPN. The key-negotiation is handled by software but the actual encryption and decryption of packets (using a conventional cipher ...) is done in the kernel.

If you want to do crypto in a userland program, use a well-established library to do it ... gpg, openssl, and so-on. The kernel routines have nothing to do with this although they do implement some of the same ciphers.