StreamRouter interval should be configurable
#2391

Comments

edited

Hello,

With hundreds of streams and dozens of graylog nodes, mongo load starts to be a (big) problem.
Obviously, this is mainly due to the hits done by StreamRouter every second for getting fresh rules.
As you can see on my Ganglia graph, switching this interval from 1 second to 1 minute saves me a lot of Mongo CPU.

I'm aware that this is introducing a minute of lag between stream creation and real routing and so probably need an explicit configuration directive in server.conf

Previously every StreamRouter instance was polling for stream changes
every second. Now it's listening on the server event bus for updates.
Stream related REST endpoints are now posting events when updating
stream configuration.
Fixes#2391

…2496)
* Switch stream router from polling to subscription for updates
Previously every StreamRouter instance was polling for stream changes
every second. Now it's listening on the server event bus for updates.
Stream related REST endpoints are now posting events when updating
stream configuration.
* Switch extractor filter from polling to subscription for updates
Before this, every ExtractorFilter was using a cache which expired after
1 second for the extractors. So it was basically polling the extractors
every second.
* Switch static fields filter from polling to subscription for updates
Before this, every StaticFieldFilter instance was using a cache which
expired after 1 second. It was basically polling the static fields
configuration every second.
* Switch rules filter from polling to subscription for updates
Before this, every RulesFilter instance was using a cache which expired
after 1 second so it was basically polling for new rules every second.
Fixes#2391