If you want to enable the C/R thread then you need to specify it. Try
adding '--enable-ft-thread' to you Open MPI configure in addition to
'--enable-mpi-threads'. The C/R thread should help your problem below.

Also it looks like you are specifying the wrong BLCR path. Above you
said that it was installed in '/root/MS' but you are passing '/usr/
local/lib'.

Have you confirmed that you can successfully checkpoint/restart a non-
MPI program on this system with BLCR?

> # make
> # make install
>
> then i added the following to the .bash_profile under home
> directory( i went to home directory by doing cd ~)
>
> /sbin/insmod
> /usr/local/lib/blcr/2.6.23.1-42.fc8/blcr_imports.ko
> /sbin/insmod /usr/local/lib/blcr/2.6.23.1-42.fc8/blcr.ko

I suspect that this is related to your application. Have you tried to
checkpoint/restart a simple example program, something that has a core
loop like (Note the MPI_Barrier is necessary if you are not using the
C/R thread since we need to call into the Open MPI library to check
for a checkpoint):
---------
for(i = 0; i < 100; i++) {
MPI_Barrier(MPI_COMM_WORLD);
printf("Counting %d\n", i);
sleep(1);
}
----------

This error message indicates that Open MPI's daemons could not
communicate with the application processes. The daemons send SIGUSR1
to the process to initiate the handshake (you can change this signal
with -mca opal_cr_signal). If your application does not respond to the
daemon within a time bound (default 20 sec, though you can change it
with -mca snapc_full_max_wait_time) then this error is printed, and
the checkpoint is aborted.--------------------