Overview

CentOS7’RHEL7 – I needed to log dropped packets form IPtables to a separate file using rsyslog, I like my logs in separate files, and then rotate them. I read several online guides and most worked but ended up with logs going to a separate file just fine, but they were still going to /var/log/messages, I did not require the double logging, I did set “& ~” in rsyslog.conf but it just didn’t work, after a bit of experimenting the following worked great.

iptables

I only needed to log dropped packets coming in, so I added the following to iptables, they are appended after the last “INPUT” lines, the format of the text “Dropped: ” is important, as it changes the way it’s filtered and displayed.

Create a file for the logs

touch /var/log/iptables-drop

rsyslog changes

I created a new conf file under /etc/rsyslog.d/ called iptables-drop.conf, if you have quite a few conf here you may want to give the file a numerical ID as well. Also check that the rsyslog.conf file has the line $IncludeConfig /etc/rsyslog.d/*.conf uncommented, it usually is by default though.

I then added to iptables-drop.conf, note, that many of the guides use “& ~” but this is deprecated and will tell you so in logs, so use “& stop“.

:msg, contains, "Dropped: " -/var/log/iptables-drop
& stop

Restart services

Now restart rsyslogd and iptables and you should be good to go. You can check by using tail and sending some rogue traffic

Notes

I first added the lines in iptables-drop.conf to the main rsyslog.conf, but it didn’t seem to work well, could be an ordering issue, but the correct ways is to use rsyslog.d so that is the way to go.

I also tried the often recommended “:msg, startswith …. ” but that did not work well for me, changing “startswith” to “contains” got it all working great.

There is another easier way of doing all of this as well, if you don’t require a separate log you can use journald to do much of the heavy lifting, all you need to do is add the lines to iptables and you are good to go, no rsyslog configuration required at all. To view them just use the following.

Overview

I had Tenable Security Center running on a RHEL6 VM which scans around 1000 hosts several times a week. REHL6 is getting a bit long-in-the-tooth and also there was a need to upgrade to the latest and greatest version of SC and RHEL, those being at the time SecurityCenter-5.8.0-el7.x86_64 and RHEL 7.6.

In theory, this should work with other Linux distros as well, there will be some subtle changes, but conceptually you should be good to go.

The user of SC wanted to keep all the custom dashboards, plugins and scan data, so this meant that the entire SC configuration and associated database needed to be migrated, there was only a 24hr window to do this as the scans are used almost daily.

Plan (Simplified)

Stand-up and new RHEL7 VM with a temporary IP and name that is different to the original SC, I named it SC-new

Stop all SC services on the current SC (named SC-old) and use rsync over SSH to copy everything over

Once data has been copied, power off SC-old

Change the identity of SC-new to be the same as SC-old, so, same IP host name etc – this is crucial for the migration to work, the host name is tied to the license keys.

Install the same version of SC as it had on the old one, once this is done and configured SC should work just like the old one

Next, upgrade the SC to the new version

Finally patch the latest version

Test

Assumptions

I’m assuming you have full access to both hosts and have the firewall etc all configured, have stood-up a new RHEL7 host, and have some Linux command line experience. I have successfully migrated using this process but use at your own risk, if you have never done anything like this before I would seek professional advice before going any further.

I also had the luxury of taking VM snapshots for the entire process, so could revert back if anything what wrong.

Environment

VMs on VSphere, 12 cores 8GB RAM

Separate 500GB logical volume for /opt – which is the location SC installs

Migrating from RHEL6.10 to RHEL7.6

Security Center from SecurityCenter-5.6.2.1-es7.x86_64 to SecurityCenter-5.8.0-el7.x86_64

Tools used

Ensure you have rsync installed on both hosts. I used rsync to transfer data. The advantage of using rsync is that if your transfer gets interrupted you can resume it, also you can transfer the bulk of the data, say in the evening, and then just copy over the changes another day. For me, the entire sc directory was 200GB in size and took around 1hr to transfer, a few days later, when I ran another sync it only had to copy 6GB of data over so took minutes, this is a huge time saver.

On the new host for SC, check you have the following installed; zip, unzip and java-11-openjdk.x86_64, at this point you DO NOT INSTALL any Security Center products.

Prep

Ensure new host has been stood-up, with a different name and IP, is ready to accept transfer, has zip, unzip and java-11-openjdk.x86_64 installed

yum install zip unzip java-11-openjdk.x86_64

You have configured enough disk for the transfer

Inform users there maybe some downtime

The process

Current hosts – SC-old

Stop all Security Center services before rsyncing the data over

service SecurityCenter stop

Check it has stopped, there should no output

ps -fu tns

On sc-old – copy/rsync the entire contents of /opt/sc over to the new host, it may take several hours so plan accordingly.

rsync -avr --stats --progress -e ssh /opt/sc user1@SC-new:/opt/

You are now done with the old Security Center, so power if off, if all goes well, you never need this again.

New host – SC-new

Check that you now have a /opt/sc directory and all looks good, currently, as Security Center is not installed, the directory and contents are owned by the user used to copy the data over, we need to fix this later, the user and group will eventually be tns.

pwd
/opt
ls -ltr
drwxr-x--- 16 user1 users 4096 Jan 23 12:20 sc

Now we are going to change the identity of the new host to be the same as the old, this is crucial, without doing it will complain about licensing and fail to work.

hostnamectl set-hostname sc-old

Now edit /etc/host

vi /etc/hosts
10.1.2.3 SC-old SC-old.mydomain

Change IP by editing ifcfg-xxx file, or by what ever method you are used to.

vi /etc/sysconfig/network-scripts/ifcfg-ens192

For me, this was all I had to change, but YMMV, so check.

Reboot for the identity change to kick-in.

Now install the same version of Security Center that was on the old on, for me it was SecurityCenter-5.6.2.1-es7.x86.

You can see it checked and found that the tns user was not there, don’t worry, it creates this for you during the re-install, a long with the new systemd type service start-up instead of the old initd.

Now cd into /opt, when installing I noticed that it didn’t recursively set all the permissions up, to fix do

chown -R tns:tns sc

At this point you are ready to test and see if you old version of SC works on the new OS, before upgrading to the new version of SC.

I started the SecurityCenter service, it started but there were a few errors, so I rebooted and checked

Next I got the SC user to login to the admin web GUI and runs some scans and jobs etc to test that the old version was working before upgrading to the new version of SC, after a day of use they reported it was working great, so now we can upgrade and patch.

audisp, rsyslogd and journald

After using rsyslog to send logs onto a central log collector, I noticed that all the audit logs are still going to ”/var/log/messages” as well as ”/var/log/audit”, not ideal, the only solution that worked, out of the many I tested, is detailed below, the audit logs still go to /var/log/audit (separate LV of course), /var/log/messages doesn’t fill up with audit logs, and all the logs get forwarded to a central log collector. The following configs below seem to work in general, but you may have to tweak them for your application, specifically the rate and burst limiting for journald.

And now journald.conf, the default is ”RateLimitInterval=30s” and ”RateLimitBurst=1000”, I had to change this as I was seeing dropped journalctl messages – ”journal: Suppressed 10361 messages from”. I had to exclude ”/var/lib/rsyslog” in audit.rules as it was creating thousands of messages, i.e. a logging loop. It is possible that rsyslog can drop messages as well, they will show up as ”imjournal: begin to drop messages due to rate-limiting”, to rate-limit rsyslog you make changes the rsyslog.conf file.

RateLimitInterval=15sRateLimitBurst=3000

If you need to rate-limit rsyslog to can add the following, however, I’ve not tested this.

Old style syslog format

# File to store the position in the journal$IMJournalStateFile imjournal.state
$imjournalRatelimitInterval 300
$imjournalRatelimitBurst 30000

I had a need to know when a host needed rebooting after yum-cron had updated the kernel, rather than logging into each host I decided I could make use of ”swatch”, in my case, it watches the ”yum.conf” file for the string ”Installed: kernel”, if the string appears in the log it sends an email out.

I have not found away for a single daemon instance to watch more than a single log file, there are some examples online showing this configuration, but when I tested it only ever worked on a single file being watched. After further reading and research this is the expected behavior.

Configuring swatch to monitor more than a single log file source

This is pretty much the same as the above, but I split things up a little, it makes it easier to manage. I’ve assumed you’ve done the initial install, also note that the file paths have changed in the service start-up files to reflect having to run more than a single instance of swatch.

Adding vmware-tools to RHEL6 or greater is easy, assuming you have the EPEL repo enabled all you do is:

yum install open-vm-tools.x86_64

That’s it your done.

Things are a tad more complicated for lower version of RHEL, for example RHEL5, some of this is down to the way VMWare have changed the way they support vmware-tools on Linux, they scrapped having the vm-tools.rpm the linux.iso which you use to push out from the vSpehre client, they now advise you to use your disto’s repos.

Add this and adjust for your release/architecture, tip, DO NOT USE the ''latest'' repo, it has caused issues, always go for a point/named release
[vmware-tools]
name=VMware Tools
#baseurl=http://packages.vmware.com/tools/esx/5.1latest/rhel6/x86_64 # DO NOT USE
#baseurl=http://packages.vmware.com/tools/esx/5.1latest/rhel6/i386
#baseurl=http://packages.vmware.com/tools/esx/5.1latest/rhel5/i386
#baseurl=http://packages.vmware.com/tools/esx/5.1latest/rhel5/x86_64
baseurl=http://packages.vmware.com/tools/esx/5.5u2/rhel5/x86_64 <====== THIS WORKS, NOTE THE PATH
#baseurl=http://packages.vmware.com/tools/esx/4.0latest/rhel6/x86_64
#baseurl=http://packages.vmware.com/tools/esx/4.0latest/rhel5/i686
#baseurl=http://packages.vmware.com/tools/esx/4.0latest/rhel6/i686
enabled=1
gpgcheck=1

Microwave went faulty a few days ago, with the following symptoms, play close attention to the symptoms, as this was leading me to think it was a cap going out, and nothing that serious as the oven was still heating. There are quite a few articles out there on the web where people have replaced transformers, megatron’s, controller boards etc (each costing anywhere from $25 – $100 each), when all they needed to replace was 10 cent cap.

Garbled buzzing/beeping sound when the door opened, or any cooking function was selected

The buzzing/beeping sound changes when any of the buttons are pressed, it seemed unwanted electrical signals were getting to the piezo speaker/buzzer

The oven light flickered in time with the buzzing sound

Turntable not turning – I recently replaced the turntable motor so I knew it was ok

It would still heat food even though it was doing all of the above

So, disconnect power, then take the oven off the wall, remove the outer case, then remove the circuit board that hosts the display and control panel.

A guy on appliancepartspros.com forum said cap C1 50v 220uF goes out, so I take a look at that one and the others nearby, they all look ok. I decided to remove C1 and the other 3 that make up a block of 4, C2 50v 47uF, C3 16v 470uF and C4 2200uF 16v. I tested C1 with my multimeter and sure enough it’s dud, just for good measure I also replaced the others in that block of 4, even though they tested ok.

I then decided to test the other caps next to this block, this column of caps are all the same physical size, unlike the first block of 4 which have various sizes – they are C5 100uF 10v, C6 22uF 16v, C14 22uF 16v, C15 4.7uF 50v. Glad I did, as found that C5 was also dud.

Resistor R51 39k ohm also looked as if it had got hot, which is possible as it works along with the relay which has been chattering away, so I replaced that as well, it did test ok though.