It looks like rake in $PATH is not the same as rake in the Gemfile. This means I have to prepend every rake invocation with bundle exec - which also means that the command isn't counted as having been run.