2013-05-31

I have written
some posts
about moving our Business Intelligence solution The Data warehouse to new hardware. We also upgrade the software and reshuffle some components, actually its like building a new system. The ETL server is now virtualized from Mandriva 2010 Linux to Mageia 2 Linux as we could not get tightVNC to work in Mageia 3! It seems like the history repeats itself from the Mandrake days, every second version has issues. Anyway Mageia 2 runs nicely so far, touch wood. When Mandrake, Mandriva, Mageia works these distros are second to none. So the ‘brain’ of the data warehouse runs on Mageia these days. With no problem at all I compiled
PHP 5.5 RC2
together with the SAP rfc communications modules
SAPRFC
and
SAPNWRFC
. So we soon run a serious large scale production, (some 20,000 jobs a month), on Mageia 2 and PHP 5.5 RC2 - I love it. Now I have to come up with a way to upgrade MySQL from 5.5 to 5.6, we have a
tricky Mysql setup
, but I will find a way to do it.
(In Mageia I run MariaDB Client against MySQL 5.5 at first I didn’t even notice).

This whole project revives the original spirit of the data warehouse project, new innovative and low budget. Year 2000 I set out on a journey to prove you can create high quality IT systems on a shoestring budget. I think I done it again. This time it’s way more expensive, my bosses forced me to spend much more money than I had in mind, how often does that happen? Still we have a very modest budget considering the application.

2013-05-16

We copied the 400GB Database with Rsync. It took some 7 hours to migrate, but after that it was just to fire up mySQL in the new server and change DNS addresses to point to the new server. In the old server we set up a MySQL proxy to reroute inbound traffic to the new server just in case. The transition was extremely smooth not a single problem until a mail arrived

WTF, this was not something I expected by upgrading MySQL some patch levels of the same release. After we have rummaged the building we found an old derelict Dell server in the basement. It turned out to be an old Mandrake 8.1 Linux running on the machine for years, and those who knew of the server setup had long since left. We (wisely) decided not to touch this server, instead we started up a copy of the database in version 5.5.9 and rerouted the obsolete application(s) to this DB instance. Next we will virtualize a copy of this Linux system and then upgrade it to more modern Linux probably a Mageia Linux.

2013-05-03

I have written some posts about design and build your own
BI infrastructure
. Now when I’m getting a new assignment, we (my bosses :-) decided I had to replace
my hardware
with more professional irons.

The new Data Warehouse

I chose two servers from Dell, o
ne PowerEdge
R720
and one
R720XD
, both equipped with one Xeon ES-2609, 32GB 1600MHz RAM and 12TB RAID 6 SATA/Nearline SAS disk space (
I like oceans of space
).

I will use the
R720 as a physical MySQL database server
, and the R720XD will contain all other (virtual) servers. Going virtual is an architectural change I planned to do for a long time, but I never found the time. I rather would have used my custom build hardware, but the guys who decide decided otherwise. ‘Since you jump ship, we do not dare to use
your old computers
. We want proper servers.’ I do not know how the new infrastructure will perform but I got a hunch it will perform better than my now old irons. I will also go from 100MB network connections to gigabit, this will definitely speed up communications.