If OMPI is spinning consuming 100% of your CPU, it usually means that
some MPI function call is polling waiting for completion. Given the
pattern you are seeing, I'm wondering if some Open MPI collective call
is not finishing until you re-enter the MPI progression engine.

Specifically, is your pattern like this:

- some MPI collective function
- enter a long period of computation involving no MPI calls
- call another MPI function

If so, you could well be getting bitten by what is known as an "early
completion" optimization in the Open MPI v1.2 series that allows us to
lower our latency slightly in some cases. In OMPI v1.2.6, we added an
MCA parameter to disable this behavior: set then
pml_ob1_use_early_completion MCA parameter to 0 and try your app again.

This parameter is unnecessary in the [upcoming] v1.3 series; we
changed how completions are done such that this should not be an issue.