The company I work for is currently investigating the deployment of an centralized automation system (like Salt or Puppet) for our servers (all Ubuntu/FreeBSD). We will probably go along with Salt, but I think it is irrelevant to my question.

My quesiton: Is there a good way for monitoring machines for local changes not included in the automation system?

For example: for a quick fix, someone started a service or modified a configuration file on a given machine. Is there a way to check for such things using Salt/Puppet/whatever? Or do I need to use external programs like AIDE for that?

The configuration state files are written in yaml/jinja2 and can be extended in python. It's a huge thing for me, since I won't have to learn ruby or the puppet manifest language. Also, Salt can both manage configuration and execute remote commands. So no need for another tool in case I need to run something on multiple machines (for example restart nfs clients).
–
Wiesław HerrJan 18 '13 at 16:24

Aide will use aide.conf to figure out exactly what files/directories to monitor and how. It would be possible to write up a Puppet template that could set up aide.conf to not look at directores/files that are created (and can be verified) with rpm.

On the flip side, you could just list the file you want to monitor...

In order to re-initialize the Aide DB on package updates, aide --init needs to be run whenever yum is run via Puppet (then the newly created DB needs to be moved into place). Aide takes so very long to run this initialization that this is untenable (for us). It might be fine in your case.

Use nice to get aide to not be a resource hog. A nice factor of -19 was suggested by www.howtoforge.com.

Gives you a nice colourized report telling you what is in compliance with your defined states without actually applying the changes. That gives you what you want in that it shows you where the state of your systems deviates from what the automation system wants it to be. There are good reasons for that to be so, perhaps you are learning salt or have a delicate legacy system and don't want to go all out with an automation system. The more states you add to salt the better the better but it means you can grow organically as you learn.