Chef tracks metadata about nodes in its database. This includes
operational facts about the node (uptime, memory, etc.), and
chef-related things like when the node last checked in. It also
includes intentional data such as what run list should be applied to
the node.

Periodically, a node polls its server for updates. What happens is:

node checks in with server

node gets current metadata from server, including its run list of
recipes and roles

node performs actions as per the run list

node saves its metadata back to the server, including the run list
it just applied

All well and good, except that step three can potentially be long
running. There’s plenty of time for an administrator to change the
node’s desired run list (or other intentional metadata) using the
knife tool or the web interface. But now, when the node’s run
completes, it saves its old state back to the server, over-writing
whatever updates an administrator applied while it was running. And
you won’t know unless you look.

This is unfortunate.

There’s a bug that more or less describes this in the project’s
tracker. It was raised quite recently, so hopefully someone from the
Chef team will take a look at it soon. There’s also a thread on
the Chef mailing list.