New unMenu Package: MRTG... any interest?

62 posts in this topic Last Reply June 4, 2012

Recommended Posts

Folks, I just discovered unRAID recently and am setting up a box. I was fiddling around with unRAID packages while waiting for my files to copy and noticed that there didn't seem to be any traffic plots. So as a learning experience I got MRTG and SNMP working on my unRAID 5 beta 14. Everything seems to work, but it's all very much a work in progress. (it's a beta package on a beta unRAID release!) If you haven't heard of MRTG, it's a traffic grapher (http://oss.oetiker.ch/mrtg/). They way I've gotten it working is it:

3. defaults to setting up a ro snmp community public (can be overridden)

4. creates a base mrtg.cfg file in /boot/config if there isn't one there (so you can customize it and not have it disappear)

5. creates a /tmp/mrtg directory where the log/png files are stored (these will disappear upon reboot). The directory is user definable so, in theory, you could stick it on a cache drive or even on the array and have a full history saved. MRTG's log files don't grow past a certain size, so no worries about running out of ram.

6. adds a 5 minute recurring cron job to update the mrtg charts

7. piggybacks on emhttp (the unRAID GUI) to create a simply/crappy website at http://tower/mrtg/index.html'>http://tower/mrtg/index.html (for unRAID 5.0b14) or http://tower/log/mrtg/index.html (for unRAID 4.7) which shows all the interfaces' daily charts and allows for drilling into the week/month/year data (make sure you include the index.html!)

Things to keep in mind:

- MRTG will give warnings the first 2 intervals as it has to create log files/image files (so if you have root email forwarded, you will see it) no longer an issue

- not sure how to get the index.php index.html to show up as the default page under http://tower/mrtg using emhttp

- probably some bugs in the unMenu conf file (it's my first one, so bear with me)

- all the dependencies other than perl take up 7.7MB of tgz files (and all but mrtg are Slackware packages)

There's no reason why you couldn't set up a bunch of other machines' interfaces (routers, other machines, etc) manually in the mrtg.cfg file and have them all tracked. Also, SNMP allows for disk space/load average tracking as well as eth interface tracking so you could theoretically do disk space/load average charts with MRTG as well.

Anyway, if anyone ends up using it, let me know if you notice any issues and if you have any suggestions for the conf file.

Cheers,

Adam

Update 2012-01-09:

I've setup a 4.7 unRAID USB and this now works there as well. The URL is unfortunately different between 4.7 and 5.0b14

A minor issue will be if a person enters a data path that contains a space or special characters the shell interprets, but it is hard to protect against bad data entered in the configuration fields. You've done well in giving them examples of what is expected.

For a first time unMENU package, you figured out the format pretty well. There is possibly one more dependency if this package is installed on unRAID 4.X. Those versions do not have "php" installed. They will probably not work as your index.php will not be interpreted.

Joe L.

Share this post

Link to post

Thanks for the feedback! I changed the cron line to your suggestion (I hadn't considered that). I also removed the PHP dependency as I was only using it for the index.php. I'm now just using a one-line perl script to generate an index.html file (so the mrtg page is now http://tower/mrtg/index.html). I also ended up running mrtg twice as part of the install so no cron warnings are sent out.

I haven't tested it on 4.7 yet as I have only one USB memory stick (which is blacklisted) and I'm waiting for a better one to show up :-) One thing I'm concerned about is I checked the version of Slackware for 5.0 and it was 13.1, so that is the version of tgz files I used in my conf file and where I compiled the mrtg tgz. I'm guessing 4.7 is using Slackware 12.x, would that cause a problem?

Also, although it installs just fine manually, it doesn't seem to want to auto install. If I run /boot/packages/mrtg-2.17.3-i486-0.tgz.auto_install after a boot, it works fine. I suspect it's getting auto installed before perl, but I can't see the messages on the console because it scrolls by too fast (and I can't find a message log file). Is there any way to have the mrtg package auto install get scheduled for after the perl package install (besides adding a line to the 'go' script)?

Sorry to bug you with these questions.

Thanks,

Adam

Share this post

Link to post

I suspect it's getting auto installed before perl, but I can't see the messages on the console because it scrolls by too fast (and I can't find a message log file). Is there any way to have the mrtg package auto install get scheduled for after the perl package install (besides adding a line to the 'go' script)?

Sorry to bug you with these questions.

Thanks,

Adam

The .auto_install scripts are invoked in lexical order, by the one line added to the bottom of the "go" script.

To change the order, you'll need to modify that line, or... rename the .auto_install file (which is named after your package file) to start with a letter > "p"

or... since you are pretty good with scripts, modify the package manager awk script to respect a new tag for PACKAGE_DEPENDENCY.

or, add a line like this towards the top of your installation lines in your package:

Share this post

Link to post

I do this a lot in my unMenu packages. Take a look at my SABnzb, SickBeard, Couchpotato, Transmission packages for ideas. On first install of the package it checkes to see if the required components are installed, if not it directs them to the correct URL to install that package. Rinse and repeat until all dependencies have been satisfied.

Share this post

Link to post

I fiddled with it some more and used prostuff1's suggestion, now it works and stays working upon reboot. Cleaned up some and added comments to the conf file and posted my sample mrtg.cfg (that does all the extra things), but for now the base package install just sets up ethernet interface monitoring.

So, for my use, it is fine. I'll see if anyone else is interested before I put more effort into this package. I've got another one I'm looking into trying to get working.

Share this post

Link to post

I was looking for something that give an overall picture of what's going on with my server, especially CPU load.. I'm running a HP N40L microserver and it would be good to evaluate how it's coping with running PS3MS during encoding tasks.

I'm running 4.7 at the moment, and installed Perl, and it appeared to install fine, but can't access tower/mrtg/index.html

I did get an error, that's all that appeared out of place.

ln: creating symbolic link `/usr/local/emhttp/mrtg': No such file or directory

I didn't edit the Configuration Variables, and haven't performed a reboot.

Share this post

Link to post

I'm running 4.7 and have it working. I also couldn't get the web interface running, but I have problems with other apps using emhttp, so I don't know if I have the same problem as Lacehim. I changed the cfg so it writes to a folder inside my lighttpd destination and that works for me. I prefer using lighttpd, anyhoo.

A couple of questions:

1. After I change the config, how do I restart mrtg or get it to reload the cfg so it writes a new index.html?

2. How do I add /mnt/cache as a disk target (or whatever it's called) -- I need to monitor that, rather than disk1/disk2

Share this post

Link to post

I'm running 4.7 at the moment, and installed Perl, and it appeared to install fine, but can't access tower/mrtg/index.html

I did get an error, that's all that appeared out of place.

ln: creating symbolic link `/usr/local/emhttp/mrtg': No such file or directory

Thanks! Ah, it looks like the emhttp directory is different in 4.7 (I only have 5.0 installed right now), I've got to find another USB stick to put 4.7 on so I can see what the differences on. Was hoping my "real" stick would have come in by now from newegg. I'll see if I can dig up one from somewhere and let you know what I find. It looks like other than the different web directory, it should work fine (neilt0 was able to get it working using lighttpd).

Share this post

Link to post

I'm running 4.7 and have it working. I also couldn't get the web interface running, but I have problems with other apps using emhttp, so I don't know if I have the same problem as Lacehim. I changed the cfg so it writes to a folder inside my lighttpd destination and that works for me. I prefer using lighttpd, anyhoo.

A couple of questions:

1. After I change the config, how do I restart mrtg or get it to reload the cfg so it writes a new index.html?

2. How do I add /mnt/cache as a disk target (or whatever it's called) -- I need to monitor that, rather than disk1/disk2

Thanks again!

Yup, you've got the same issue as Lacehim, just a different path to the main web code in 4.7 I was piggybacking on emhttp only because it was there. As long as it works for you under lighttpd, that's all that matters :-)

To answer your questions:

1. when you make changes to mrtg.cfg, make sure you update both files (fiddle with the one in /tmp/mrtg or wherever, but save it to the one in /boot/config when you're happy). I was hoping to just use the one in /boot/config all of the time, but mrtg has some weird problem (it thinks there are stale lock files in the /boot/config directory when there aren't). mrtg will see the changes the next time it runs via cron, although if you add a new chart, it will whine the first two times (missing log files, etc). I find that running it manually twice works better (instead of getting cron mails).

env LANG=C mrtg /path/to/mrtg.cfg

to update the mrtg landing page, run this

indexmaker --output=/path/to/index.html /path/to/mrtg.cfg

2. Hmm, maybe I should break out the snmpd.conf file as well, that's where you need to make the changes to do other disks. As you've already made changes to the mrtg-unmenu-package.conf file, have another look in there and look for this bit:

Change the red number(s) to be the one which signifies the offset you put your /mnt/cache line in earlier (if it was the first disk line, that's 1... second would be 2, etc). Change the bolds to match for the legends and titles. If you only want the one disk, set both numbers to the same thing.

Edit: also should mention, not sure how the cache drive works, but for regular disks in the array, mrtg throws an error when the array isn't running.

Hope that helps, and let me know if you have any more questions.

Share this post

Link to post

I'll give this a go - I know of the program but have never put in the effort to find out how it works. Am I correct in assuming I need to install the mrtg slackware package (wherever it is) and then use the .conf file provided to enable it within unMenu?

Share this post

Link to post

I'll give this a go - I know of the program but have never put in the effort to find out how it works. Am I correct in assuming I need to install the mrtg slackware package (wherever it is) and then use the .conf file provided to enable it within unMenu?

As far as I know, there is no Slackware mrtg package. My conf file installs one I compiled myself.

Share this post

Link to post

plowna, just put the .conf in the unmenu folder, and install from packages in unmenu.

If your missing perl it will let you know to install it.

AdamT, I ran "which emhttp" on my tower, and it says emhttp is in /usr/local/sbin/emhttp so I've modified the .conf file with the additional /sbin folder. I'll have to reboot and reinstall it I guess but it's the only reference to emhttp I can see in the .conf file.

Share this post

Link to post

Not sure why it is not working for me. I installed php before I installed the config file. It appears to install correctly and I get a cfg file in my config folder. When I go to http://tower2/mrtg I am asked for my password which would be normal but then i get a blank page.

Any ideas where i should look?

thanks

running beta14

Share this post

Link to post

AdamT, I ran "which emhttp" on my tower, and it says emhttp is in /usr/local/sbin/emhttp so I've modified the .conf file with the additional /sbin folder. I'll have to reboot and reinstall it I guess but it's the only reference to emhttp I can see in the .conf file.

Actually, you don't want to do that... the /usr/local/sbin/emhttp is the binary for emhttp. I've got this fixed in my latest release, I'll post a longer explanation separately.

Share this post

Link to post

It looks like emhttp is very, very picky. I have to actually include the index.html in the path, so for you it would be http://tower2/mrtg/index.html. Let me know if that works, in the latest version I put a comment in the description about the need to have the index.html in there.

added default support for /mnt/disk1 - /mnt/disk20 & /mnt/cache. the cache drive should be 1 drive 'number' higher than your last disk. If you want to double check, run this command from the command line after installing this new package:

snmpwalk -v 1 -c public localhost dskPath

this shows all the disks SNMP knows about (the blue depends on if you overrode the default values in the package configuration).

I broke out the snmpd.conf file, so if anyone wants to make changes to the SNMP service, the file to change is /boot/config/snmpd.conf. make sure you do a

/etc/rc.d/rc.snmpd restart

after you make changes to snmpd.conf

there is only one copy needed of the mrtg.cfg file now. turns out mrtg can't put a lockfile in a FAT32 partition (i.e. /boot), so I added a flag to have it put somewhere else. only thing this means is all you have to do is update /boot/config/mrtg.cfg with any changes. for advanced users, the flag I added was

Share this post

Link to post

Hmm, I did some digging and it's possible to show bytes read/written to each disk (here's a page on how to do it http://oss.oetiker.ch/mrtg-trac/wiki/Net-SNMP). I'm not sure that shares will have this tracking because it talks about devices. I guess it comes down to how unRAID treats shares under the covers. Unfortunately, the Slackware SNMP package doesn't seem to have support for disk i/o installed, so I can't see if it would work for shares as well. It looks like SNMP needs to be compiled with '--with-mib-modules=ucd-snmp/diskio' but Slackware's isn't:

Let me see if I can re-compile Slackware's SNMP. If there are too many dependancies, I won't be able to do it (I don't actually have a Slackware dev box setup, I'm compiling stuff on unRAID which doesn't have a lot of packages installed).

Share this post

Link to post

Let me see if I can re-compile Slackware's SNMP. If there are too many dependancies, I won't be able to do it (I don't actually have a Slackware dev box setup, I'm compiling stuff on unRAID which doesn't have a lot of packages installed).

Ok, I was able to re-compile net-snmp with the required option and I can get stats by both disk (physical drive) and array (drive + parity). That means with this you could track how much was read/written to each disk drive individually or to each "disk share". Unfortunately, it doesn't see user shares at all. I suspect that is because they are really done in software "above" the level that SNMP is monitoring. However, if there is some file somewhere in unRAID that has this information, we can add it to the charts that way.

I haven't included this new net-snmp in the mrtg package yet. is anyone interested in by disk read/writes?