RSpec is failing but CircleCI reports my tests have passed

CircleCI uses the exit code of each test command to determine success or
failure. A combination of bugs in Ruby and RSpec can make RSpec return an
exit code of 0 (indicating success) even when the specs have failed, tricking
us into reporting success incorrectly.

The gory details

RSpec runs in an at_exit block. This makes it easy to run with
either ruby or rspec.

But this also makes RSpec susceptible to some bugs and/or incorrect
at_exit blocks resetting its exit code.
The major causes are:

A bug in Ruby’s at_exit code
where exceptions thrown during at_exit cause the exit status to be reset
to 0, even if the exception is handled inside the at_exit
block. This bug has been difficult to truly squash and has appeared in
several Ruby versions
over time.

at_exit blocks added by the code under test which don’t
propagate the exit status properly.

How to fix

You should be able to reproduce the problem either locally or via SSH into a
build machine:
bundle exec rspec some-failing-test followed by echo
$? to show the exit code.

The first thing to do is to upgrade your Ruby patchset and/or your RSpec
version to make sure you have the fixes for the above bugs.

After that, with RSpec, the most common source of incorrect exit statuses is
at_exit blocks which either call exit and fail to
propagate the previous exit status, or throw exceptions. Do you have any
at_exit code that might be breaking?

You can use this pattern to preserve non-zero exit codes if you do need to
call exit from your at_exit blocks: