Archives for: November 2012

11/25/12

The backstory is that I've been working in a primarily Sun shop, and one of the things we've been doing is running Solaris 10 on large boxes, such as T2000's, T5120's, M4000's and cutting them up with Solaris zones/containers with the global native into a management vlan and tagging appropriate vlans for the zones, and the zones have their own default route specification so all has been great.

ipf on the global so the zones can't tamper with their own firewalls, and on some of the 'zone' servers using:

/usr/sbin/ndd -set /dev/ip ip_restrict_interzone_loopback 1

The zones remain isolated from each other. Or it avoids problems of short circuiting (asymetric routing + wrong IP)....because some of the zones are behind the F5.

For years, we've tossed around the idea of introducing FreeBSD into our datacenter, and finally one of our customers decided that while they really like the containers/zones and ZFS, the cost of replacing their aging Sun server would be better done by replacing it with a FreeBSD server. While they own the hardware, and it resides at their facility...we provide the system administration support. But, this opened the door to having FreeBSD on our work site. I was in the process of replacing my aging Sun Ultra 20 with an Optiplex 990...which originally I was looking at install Ubuntu on, but instead I went with FreeBSD 9.0 (though the effort in getting the Desktop working on it, and recreating my Sun desktop/work environment on it...made me question if that was really the right way to go. But, I got it working.) And, it helps me try some of the things before doing them on the customer's server (which is headless in a closet [well, they have a monitor & keyboard for console access], so all the desktop stuff was for my benefit...it paved the way to me getting a working FreeBSD desktop at home .... though I may end up with a different system for my main desktop at home and have the FreeBSD machine go headless with my other FreeBSD servers....unless there's some way to easily share between the two....switching doesn't qualify.)

So, in needing to deploy some new internal services (such as monitoring) and not really wanting to go through the major process of find all the bits and pieces and creating packages under our CM system for Solaris. Its quite the pain building each and every perl module as separate CM packages, instead of having some system that automatically builds and installs (or makes packages) for you...ala ports or CPAN. I've done package install requests that start out as install one package, and end up building 100 or so packages instead.

I had contemplated sneaking Ubuntu in since I run the same monitoring servers on an Ubuntu server at home, but the work to incorporate Ubuntu into our configuration management infrastructure got sidelined by FreeBSD. And, there's no decision on whether Ubuntu will come into play (though the high cost of RedHat licenses to just get patches...for systems that are rarely patched....is making Ubuntu look attractive.)

Anyways things led to me starting work on pxe boot installing Proliant DL380s with FreeBSD 9.0 and creating 'jail' servers to work like our 'zone' servers.

11/05/12

For a long time, I've been running a 6 drive RAID 10 array of Hitachi 5K3000 2TB drives in Orac for backuppc. This configuration got me at 5.4TB array, and somewhat better performance than when I tried a RAID6 configuration. But, eventually, I kept running out of space and the price of harddrives went up so expanding the array over time didn't happen as I had hoped. Being RAID10, the options were other concat another array, either 2 in RAID1 or 4 in RAID10 or 4 as 2 RAID1....using volume manager. Or maybe see if RAID10 would deal with having all 6 drives upgraded to 3TB, though hadn't considered the transition of 512 to 4k and how it would cope with that.

Before the failure of one of the 1.5TB drives in the above mentioned RAID1 set, I had 4 ST31500341AS in a RAID5 on old-Zen. It had been done in under RR622, under Windows, and NTFS partitions, etc. I had tried copying the data at various times, not really having anywhere that would hold the data elsewhere...but wanting to get it over to FreeBSD for recovery. While I got the rr622 driver working, and it saw that I had a single array (rather than the native driver that would see the 4 individual disks.) I couldn't get access to the data. Though it had worked when I was previously playing around with Xen (had tried copying it then....to a 2 1TB RAID0 set, but then one of the 1TB drives died....so I lost the copy, I had then replaced it with a 2TB RAID1 set....using an ST2000DL003 and an ST2000DM003, the DL being a 5900RPM drive and having a 5 year warranty...while the DM drive is a 7200RPM drive, but with only a 1 year warranty. And, turns out the 1 year is generous.

At work, I had built my FreeBSD desktop using a pair of the ST1000DM003 drives...and 3 drive failures later....it is now a pair of ST2000DL003 drives. Yeah...I was having trouble with the array, and apparently using XFS was a mistake too...because I thought it was recovering, but instead it was slowly eating the data. When I had nuked the RR622 RAID5 array, and had switched to using it as JBOD and create a RAIDZ set under FreeBSD...I found that there was nothing to copy back from the RAID1 array. D'Oh!

Though I had also copied the Microsoft WindowsImageBackup files, to see if I could mount the VHD file under VirtualBox to help in recovery. I largely had the data in bits and pieces elsewhere, it was the environment I was wanting to recreate...and Oops!Backup didn't back up that part anyways (the data I was mainly trying to migrate). The image mounted, and I could see it...but soon after Windows would try to fix it and then it would disappear....kind of like what it did on February 15th to make the original Zen go away. No idea what kind of disk rotting the Intel Matrix RAID had been doing, when it had to initialize the array again every time after a Windows crash. I've had Ubuntu crashes, but the RAID arrays remained stable...usually. While Windows & Intel RST....it was pretty much every time. I'm sure it was slowly corrupting things overtime to where things wouldn't recover, though it choose to do that after an automatic reboot for Windows updates...and the day before I left for my first Gallifrey One made things even more annoying.

Anyway with another 1.5TB drive freed up, I contemplated adding it to the RAIDZ I had made of the 4 1.5TB drives, keeping it as a hot spare, or just use it by itself -- living dangerously. I ended up with the latter for some temporary data. Because in my mind I was starting to lean to what happened next.

Now instead of subjecting some poor random forum to a long rambling thought, I will try to consolidate those things into this blog where they can be more easily ignored profess to be collected thoughts from my mind.