b.Select the “Software Development Workstation” option then unselect groups virtualization, virtualization-client and virtualization-platform or let the installer finish then use “yum groupremove virtualization virtualization-client virtualization-platform” once you’ve booted into the OS the first time. By selecting the “Software Development Workstation” as your install type, it will pretty much install every package you’ll need to do the kernel compile.

2.Now do a yum update and reboot the box so you’re using the new kernel.

3.Install some remaining dependencies for this full kernel build:

a.yum install asciidoc

b.yum install newt-devel

c. yum install xmlto (thanks Gilbert!)

4.Commence kernel build prep.

a.Find out which kernel you’re currently using w/ “uname -r” and write it down.

b.Create user for building kernel & SCST by running “useradd mockbuild” go ahead and change from user root to that user “su – mockbuild” for the next step (no, mockbuild doesn’t need a password so don’t give it one!)

c.While as the mockbuild user, execute “mkdir -p ~/rpmbuild/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS}” which creates the directory structures for the build and finally “echo ‘%_topdir %(echo $HOME)/rpmbuild’ > ~/.rpmmacros” which creates your .rpmmacros file w/ the build directory location. (NOTE: spacing is very important on this command. Just copy & paste to be sure you typed it correctly.)

e.While in your homedir and as the mockbuild user, execute “rpmbuild -bp –target=$(uname -m) ./rpmbuild/SPECS/kernel.spec” (that’s 2 hyphens in front of target! WP is mangling the command) so it will uncompress the kernel sources and move them to your ./rpmbuild/BUILD folder. (NOTE: While it’s running, open up another console and as root run “rngd -r /dev/urandom” as the certificates will drain out the random number generator and kind of hang the rpmbuild.)

5.Download SCST from svn & patch kernel in BUILD directory.

a.We need to grab a copy of SCST using svn. There’s a number of branches but we want the latest stable. So run “svn co https://svn.code.sf.net/p/scst/svn/branches/2.2.x scst” while the mockbuild user and in /home/mockbuild directory. This will
be the latest stable version of SCST which is 2.2.1.

b.Now that our user mockbuild has a copy of scst in his home directory, go ahead and cd into the unpacked kernel so we can patch it. Should be “cd ~/rpmbuild/BUILD/kernel-2.6.32-358.14.1.el6/linux-2.6.32-358.14.1.el6.x86_64/”.

a.Check your .config files (defualt one inside your ~rpmbuild/BUILD/kernel-*/linux-*/.config to your current kernel config in /boot/config-2.6.32-*) They shoud pretty much match. This means you can skip “make oldconfig” which would prompt you for stuff missing from your provided .config file with what this kernel you’re about to compile has as of now (meaning, if you’re compiling a newer kernel, your .config file is missing some stuff so it will prompt you for defaults and update your older .config file!)

b.Change the kernel name by editing the Makefile. We need to separate this kernel from the others by editing the EXTRAVERSION varible. Change it to “=-scst” version versus being just empty or blank.

c.Now run “make menuconfig” and change the following settings for SCST:

i.Select “Network support”->”Networking options” make sure “TCP/IP networking” is checked (should be) and check “TCP/IP zero-copy transfer completion notification” underneath so SCST will be notified when a transfer of data across the nic has finished. This way, it can hurry up & send the next block. Should speed things up.

iii.Verify that “Enable the block layer”->”IO Schedulers”->”CFQ I/O scheduler” and “Default I/O scheduler” set to CFQ are both set. Again, they should be.

iv.Lastly, set “Processor type an features”->”Preemption Model” is set to “No Forced Preemption” versus Desktop mode.

v.Once done, save your config so we can compile this kernel.d.Once that’s done, we can compile it. Type “time make -j8 all” as the mockbuild user. This makes the kernel & modules while timing it and using all of your CPU’s cores (for me, I’ve got 8 cores so -j8!)

e.Now we need to install our new kernel. Do a “make modules_install” as root which will copy the modules over to /lib/modules/<kernel name> directory (you did call your kernel something different by setting the EXTRAVERSION variable right?) After it’s done, do a “make install” as root as well which will update the /boot directory & grub. Now update grub.conf so it will default to our new kernel upon boot up.

f.Once that’s done, copy over the current .config file in the /home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-358.14.1.el6/linux-2.6.32-358.14.1.el6.x86_64 directory to the /boot directory and name it such that it matches your other config files in that directory (such as config-2.6.32-scst.)

7.Reboot your system so we can use the new kernel and compile scst off the new kernel source files.

8.Now we’ll compile SCST.

a.Go ahead and change the various Makefiles at once to release version by issuing “make debug2release” in the main scst directory. This will remove the debugging stuff. Do this as root. Also, since CentOS 6 now uses sysfs and not proc, we don’t have to do a “make enable_proc” like for CentOS 5.

a.Need to go ahead and tell our system to start it up automagically on bootup so execute “/usr/lib/lsb/install_initd scst” as root user.

b.Go ahead and either reboot to test it or “service scst start” to fire it up. (NOTE: We haven’t done a config file yet so it’s running w/ nothing up!)

c.Next, we need a simple config. Since I’m using actual RAID volumes as disks in SCST, I’ve gone w/ BLOCKIO devices and list them as /dev/sdX devices. You may want to use FILEIO devices instead or even a different device handler all together (scst_disk instead of scst_vdisk.) Here’s a couple of different examples:

i.dev_disk example which uses scst_disk:

HANDLER dev_disk {

DEVICE 3:0:0:0

}

TARGET_DRIVER iscsi {

enabled 1

TARGET iqn.2013-07.com.example.san1:iscsi-tgt1 {

enabled 1

LUN 0 3:0:0:0

}

}

ii.vdisk_blockio example which uses scst_vdisk:

HANDLER vdisk_blockio {

DEVICE disk01 {

filename /dev/sdb1

}

DEVICE disk02 {

filename /dev/sdc1

}

}

TARGET_DRIVER iscsi {

enabled 1

TARGET iqn.2012-09.com.example.san1:iscsi-tgt1 {

enabled 1

LUN 0 disk01

LUN 1 disk02

}

}

iii.vdisk_fileio example which also uses scst_vdisk:

HANDLER vdisk_fileio {

DEVICE mydisk01 {

filename /home/testdisk1

nv_cache 1

}

}

TARGET_DRIVER iscsi {

enabled 1

TARGET iqn.2013-07.net.spiderbird.san2:iscsi-tgt1 {

enabled 1

LUN 0 mydisk01

}

}

10.Finalizing your install:

a.Now that we’re done, it’s best to configure yum not to update your kernel w/ a newer one which would break all of your hard work. Modify your /etc/yum.conf file and add “exclude=kernel*” to it w/o the double quotes. This will help keep your system somewhat up-to-date w/o breaking SCST & your custom kernel.

11.Troubleshooting:

a.Keep in mind, if you’re going to use multiple SCST SANs in your backend, you need to make sure all of your exposed LUNs have different t10_dev_id values else your host will get confused and assume it’s seeing different paths to similar items. Also, using setup_id is supposed to help prevent this but evidently doesn’t work in version 2.2.x. You can skate around this issue by using different names for your defined DEVICEs (instead of using disk01 try using SAN1disk01.)

b.Pass through devices work as expected except for when that pass through device is a simple SATA drive. Your ESXi 5+ server still won’t put a VMFS partition on there because it isn’t reliable storage. Best to stick w/ RAID volumes in pass through mode instead.

c.I forget C….

Here’s another really good write up on installing SCST into OEL 6.5 from scratch by Gilbert:

Like most geeks I’ve got lots of files and a need for spacious network shares. However, as soon as you go over a 2TB partition, you’ll encounter problems w/ how to correctly partition a drive or RAID volume using the newer 4096 sector spec called simply Advanced Format. I initially stumbled upon a series of articles written by Roderick W. Smith over on IBM’s DeveloperWorks when searching how to properly check partition alignment under Linux for SSDs but quickly had an “Uh Oh” moment when I realized SSDs weren’t my only problem and I had in fact been flying blind on larger than 2TB partitions under Linux. Thankfully I didn’t have systems in production where this was a problem but I was literally building a system at work where this could bite me in the rear!

To bottom line it for you, if you’re using software RAID or a true HW RAID setup, you need to stripe your file system across the partition in accordance to the number of discs involved, type of RAID and the stripe size in KBs. My example below is strictly regarding Areca ARC RAID cards and might not be applicable to your setup.

I’m using an ARC-1680ix-24 w/ 4GB of ram on board. I’ve got 24 – 500GB HDs with one volume using all HDs in a RAID 6 raidset. So, for me the math is such when using EXT4 filesystem:

Recently had a question at work about the maximum number of recipients Sendmail would allow thru our in-house server. I wasn’t sure but knew I hadn’t personally configured a hard limit. Did some quick searching and stumbled upon confMAX_RCPT_PER_MESSAGE as the answer. I haven’t set it to anything in our config but didn’t know what the default limit was in Sendmail so I did some digging. Stumbled upon the following links of value (to me at least!)

I’m currently working on building my own iSCSI SAN w/ SCST & a modified Linux kernel. Things are going pretty well but I needed to also bond out the 5 Gb nics so my VMHosts can maximize their connections. In Linux, you bond out an interface by creating a new network interface and telling the physical nics to register w/ that virtual network interface. There’s also 7 different types of bonding modes but the most common one is mode 6 or balance-ALB (active load balancing.)

If you’re using CentOS or RHEL like I am you cd to /etc/sysconfig/network-scripts/ and vi ifcfg-bond0.

Once you’ve got vi open, you dump the following in (this is an example only. Please change your IPADDR, NETMASK and NETWORK parameters as needed.)

And oh ya, don’t forget to modify your modprobe.conf file to specify the loading of the bond kernel module and what options it will needed @ run-time.

alias bond0 bonding
options bond0 mode=6 miimon=100

Couple of useful links. There’s tons of good info on bonding w/ Linux. Keep in mind, some distros don’t want you to modify the individual interface config files so use their administrator control panel first if possible.

I’m a real big fan of nVidia’s Linux drivers for their GPUs and have installed them numerous times with CentOS 5. I just recently installed them under Red hat 6 (RHEL 6) and happened to notice a warning by the nVidia driver that Nouveau was already installed and needed to be removed first. Further investigation reveals that Fedora & the latest version of RHEL 6 are including Nouveau at the initial ramdisk so not only do you need to create your own modprobe.conf blacklist file but you also need to add a line on the end of your grub.conf file to tell the ramdisk not to load Nouveau at boot up.

First, creating a modprobe blacklist file to prevent the kernel from loading the Nouveau kernel module. You could easily add blacklist nouveau to any ole modprobe.conf file but you never know when your linux distribution may overwrite your modified file w/ the next update. You’re better off to just create your own modprobe blacklist file and dropping it inside /etc/modprobe.d/ folder. I went ahead and followed some directions I stumbled upon on the interwebs and created a file called /etc/modprobe.d/disable-nouveau.conf.

I then added the following to that file…

blacklist nouveau options nouveau modeset=0

Once that was done, I then needed to modify my /etc/grub.conffile so the initial ramdisk would stop loading Nouveau too. To do this I added rdblacklist=nouveauto my kernel vmlinuz line like such…

I’m in the process of building up a SAN from scratch using SCST & CentOS. I’ve played around w/ kernels in the past but hadn’t done it in a while. When downloading 2.6.36.2 from Kernel.org recently I couldn’t get a bootable kernel no matter what I did w/ CentOS (and yes, I did load the CentOS .config file into menuconfig before compiling my kernel.)

Come to find out, because CentOS is using a modified 2.6.18 kernel that little .config file is very out dated. As of kernel 2.6.31 a new kernel parameter is available called “enable deprecated sysfs features to support old userspace tools” which is under the General setup area. Once I enabled that option, I was good to go. It has something to do with why my SAS RAID card wasn’t getting picked up upon boot up. Everything else after that was a piece of cake!

You could have also just added this to your .config file. (NOTE: Hats off to Vanecka!)CONFIG_SYSFS_DEPRECATED_V2=y

Moving lots of data:I do all sorts of networking related stuff and one of the few things I do often are server migrations. So eventually, there comes a time for me to copy a large amount of data from one server to another while maintaining the file permissions and structure. That’s where tar & ssh come into play. If I’m moving a users’ home directory from one server to another I usually use tar & ssh together via a pipe and let it copy off the files perfectly.

tar zcvf - jsmith | ssh "destinations IP" "cd /home/; tar zpxvf -"

Within seconds of the command starting off, you’ll be asked for the password for your ssh session. Then, the data will get flung across the network to the destination folder. Works great!

Verifying Apache Configuration:When adding additional Virtual Hosts to your Apache config file it’s nice to make sure they’ve been picked up correctly (or! that you’ve modified the right httpd.conf file!) A simple way of letting Apache tell you what it’s going to use is via the -t option.

httpd -t -D DUMP_VHOSTS

It parses the running configuration and dumps the virtual hosts a nice readable format. You can also use “DUMP_MODULES” to see a list of the modules it will use which can be useful for whether or not that database module is going to be loaded or not.

MySQL database dump:It never fails, when doing a migration you’re going to need to dump your databases from a MySQL DB into another DB on your newer system. If they’re the same version however you can run mysqldump w/ a couple command-line switches and then import them on the new server.

mysqldump -u root -p --databases "your DB here" > dbfilename.sql

After you’ve moved your database file to the new server you can then import it in via:

mysql -u root -p -D "your DB here" < /path/to/dbfilename.sql

Keep in mind, we’re assuming you’ve already created the database inside your new MySQL server and will assign it proper permissions so your end user can connect to it!

This is the second part of my FTPES, vsftpd & CentOS article. In the first part I walked you thru how I built a certificate, sent the CSR off to my CA and finally, modified the vsftpd.conf file. In this part I’ll show you how to test the service via command-line so you can actually see the certificate and how to punch a hole thru your FW because the data part of your FTP session is now encrypted.

First, let’s talk about verifying the configuration. One of the obvious things you can do is open up a command prompt session and attempt to log into your ftp server. Once connected, does it accept an anonymous connection? Was that what you wanted? Also, was the plain jane ftp command allowing you to log in or did you get a 331 error, “Non-anonymous sessions must use encryption.” If you’ve configured it correctly it shouldn’t allow you to login at all. So how do you test it using encryption? By using openssl w/ s_client of course! Use…

openssl s_client -starttls ftp -connect yourserver.example.com:21

This will allow you to not only log into your server using encryption via command-line but also verify that you’ve got the proper certificate & certificate chain installed. I personally tried connecting w/ out the starttls option but wasn’t successful (instead I get a “SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol” error message. I beleive this is because I’m using vsftpd in explicit mode not impicit mode.) The openssl s_client command can also be used for a number of other encrypted services for debugging certificates which is extremely helpful!

So you’ve verified that FTPES does work on the local LAN but doesn’t seem to work thru your FW. This is where passive ftp comes into play because your FTP traffic is now encrypted and your FW can’t do an inspection to determine which port is going to be used next and open it up for you ahead of time. Most FWs come with some sort of FTP inspect feature but we just killed it because that data is totally encrypted now and it’s unable to sniff the FTP traffic and sense which port needs to be opened. You won’t have to turn on passive FTP because it’s on by default with vsftpd. You should specify the port range and I also turned on pasv_address too:

Once done, this will force your vsftpd server to use a port range of 63000 to 65535 for data connections. Alot of the commands I used above can be found w/in the man page for vsftpd.conf (along with their default values!)

Now you just need to open up that same range on your FW device (NOTE: Each FW is going to be different!) On my Cisco Adaptive Security appliance (ASA) I’m defining an object-group first for the port range like such:

Just be sure to use your public IP address for ‘YOUR_PUBLIC_IP’ in the above ACE. Now when it comes to ACL rules there’s tons of different ways to allow traffic so I won’t go into much more detail here other than how I punched a hole thru my FW to allow the encrypted vsftpd data channel traffic through.

That’s about it folks. Remember, if you’re having problems with your configuration, break it down into simple pieces and troubleshoot it that way versus trying to eat the entire problem at once!

So this past weekend I went to visit my Aunt & Uncle in Harrisonburg, VA. While there, my Aunt tells me she can’t send email thru my server which I was fairly certain was once again, “user error.” Come to find out. It was a misconfiguration on my part! Ah ha!

Long story short, I had reconfigured my email server about two months ago and added a DNSBL (zen.spamhaus.org) to my sendmail features and it was blocking her from relaying email thru my sendmail server. I simply had to add…

FEATURE(`delay_checks')

… to my sendmail.mc file and restart the service. (NOTE: On CentOS 5.0 if you restart the sendmail service it will check your sendmail.mc & sendmail.cf files and do a ‘make -C /etc/mail’ for you if the sendmail.mc is newer.)

Now, Sendmail will wait on the DNSBL and give the user a chance to authenticate first which will allow them to then bypass the DNSBL. Seems her residential comcast IP Address was added to zen.spamhaus.org which is commonly done to blocks of residential user IP address spaces to prevent spam.

Ask anyone that knows me and they’ll tell you I love Linux. In particular, I love Red Hat’s flavor of Linux. I’ve still got my first Linux book “The Complete Red Hat 5.2 Installation Guide” and while I don’t consult it first for all problems I do remember thumbing thru its pages when trying to configure those tulip.o drivers and how to change my X Windows config file settings (NOTE: I still to this day get tripped up on configuring X Windows!) So that having been said, it’s no wonder I’m using CentOS and have been for years since Red Hat has stopped releasing a free desktop variety for users to use (I know they’ve got Fedora but that product lines’ release cycle is too aggressive for my tastes.)

So that having been said, I love CentOS. I’ve been using it at home for a while now and whenever possible do my best to stick w/ the stock rpms so updates go smoothly. I’ve recently been trying to get vsftpd daemon to use encryption while still jailing my end users in their home directories and here’s what I found out. Vsftpd doesn’t natively do sftp but it does do ftpes pretty well. So I’m going to walk thru generating a private key, making a certificate signing request, adding the public key from your third party certificate authority (CA) to their certificate bundle to form a chained public key and finally, configuring vsftpd to do ftpes.

First, I logged into the box and grabbed superuser.

su -

Then changed directory to the location of the certificates.

cd /etc/pki/tls/certs/

Next, I ran openssl to generate my private key w/o a passphrase! (NOTE: If you decide to use a passphrase a number of services won’t auto start on a power failure which means you’ll need to run home, start the service manually again and enter in the passphrase so the service can use the certificate. Some services can be configured to manually put the passphrase in for you but that’s lots of effort.)

openssl genrsa -out myserver.mydomain.private.key 2048

Had I wanted to generate a private key w/ a passphrase it would have been…

openssl genrsa -des3 -out myserver.mydomain.private.key 2048

You’ll notice at the end of the command I put 2048. That stands for how many bits this certificate is generated with. A number of CAs will require now a days that you use a 2048 bit key versus the old standby of 1024 bit ( the CA I use GoDaddy is one of them.)

The very next thing we need to do is lock down your private key. This is used to encode encrypted communications to the end user who will now use your public key to decrypt the data. So let’s make it readable only by root.

chmod 600 myserver.mydomain.private.key

Once that’s done, it’s time to generate our certificate signing request (CSR) that needs to be passed to your CA so they can generate a public certificate for you.

openssl req -new -key myserver.mydomain.private.key -out myserver.csr

At this point you’ll be prompted for answers to a series of questions which will be placed in your CSR. The big thing you need to get right is your Common Name (CN) question. Make sure the CN is the server name that matches up w/ your DNS entry for this server. Folks are going to use DNS to access your server so the certificate needs to match up with what they are going to use to hit your server across the internet. It’s interesting to note, that w/ GoDaddy, pretty much all of the other fields you fill in get thrown out in the generated public certificate they send back to you so don’t get too freaked out with putting your location information in there. Again, this is with GoDaddy. Other CAs may publish that information w/ the public certificate.

After you’ve submitted your CSR to your CA. You’ll have to go thru the whole domain verification process which can be its’ own little tap dance. I know when our company had their domain names register w/ someone else besides GoDaddy it took 24 to 48 hours for whois verification and whatnot. Once we moved our domain names to GoDaddy the verification process took less that 5 minutes. Coincidence? Business strategy? I don’t know but I’m sure glad we moved our domains over to GoDaddy.

Okay, so you’ve finally gotten your certificate from your CA. Yay! Now comes the fun of lumping that certificate w/ the included CA bundle they sent you and configuring your vsftpd daemon.

First, concatenating the two files together into a combined public certificate file. Why? Cause vsftpd doesn’t have an option for the certificate chain file your CA gave you w/ your public certificate.

cat myserver.crt CA_bundle.crt > myserver.mydomain.public.key

This will put both files together into a combined file which will have all certificates. Now we can configure vsftpd and be done.

Open up the vsftpd.conf file w/ vi.

vi /etc/vsftpd/vsftpd.conf

When looking at your vsftpd.conf file I started at the beginning and slowly worked my way thru turning off features I knew I didn’t need. One of the first was anonymous_enabled. It defaults to enabled unless you uncomment the line and tell it NO. Most everything else I left as is. It’s interesting to note, that the SSL features you’re looking for in the config file aren’t present so you’ll need to add them manually to the file. Here’s what I’ve put in mine. Most of these commands by the way are covered in your man page for vsftpd.conf so take a look there for a more detailed explanation.