It must be something specific to my environment that triggers it because this is exactly the same command that is run as part of make check and it succeeds in the gitbuilder. If I remove the "paxos propose interval = 0.01" the problem does not show.

Associated revisions

mon: MDSMonitor: wait for osdmon to be writable when requesting proposal

Otherwise we may end up requesting the osdmon to propose while it ismid-proposal. We can't simply return EAGAIN to the user either becausethen we would have to expect the user to be able to successfully racewith the whole cluster in finding a window in which 'mds fs new' commandwould succeed -- which is not a realistic expectation. Having thecommand to osdmon()->wait_for_writable() guarantees that the commandwill be added to a queue and that we will, eventually, tend to it.

mon: MDSMonitor: wait for osdmon to be writable when requesting proposal

Otherwise we may end up requesting the osdmon to propose while it ismid-proposal. We can't simply return EAGAIN to the user either becausethen we would have to expect the user to be able to successfully racewith the whole cluster in finding a window in which 'mds fs new' commandwould succeed -- which is not a realistic expectation. Having thecommand to osdmon()->wait_for_writable() guarantees that the commandwill be added to a queue and that we will, eventually, tend to it.

Loic, try this patch with the same conditions in which you triggered it: c84a13ae87eed5550bafda394d983a8e843cc08c

that patch has been sitting on wip-9321.giant and is intended to fix the behavior that is causing the daemon to assert (MDSMonitor needs to wait on osdmon to be writeable before requesting it to propose).

Hopefully you're able to reproduce it reliably enough to make sure the patch fixes it.