In an attempt to get some control over our alerting we have put our watcher definitions in a GitHub repo and put together a build script so Travis deploys updates to the watchers automatically. This seems to work only only half-way in the sense that

the new watchers are deployed to our elasticsearch instance on elastic.co, and

the GUI says that the watchers are run, however

the watchers never actually fire

The only way we've found to "turn them on" is to manually log in to elasticsearch/kibana, and manually save each watcher.

This is not really the behaviour we expected, so we suspect there's something we are missing.