As of May I’m very busy architecting & implementing cluster for Java Enterprise Edition on comodity hardware (mainly x86_32 based) for my engineering work – to obtain BEng title. Our subject is:
“Web service based on scalable and highly available J2EE application cluster”. We have team consisting of 4 persons in which I’m responsible for all kind of systems/hardware scaling/clusters/load balancing/databases/networking/tunning everything . What kind of portal we are creating is to be decided by developers (it will likely be some kind of Web 2.0 portal).
Rest of the team is dedicated to J2EE programming. We are mainly playing with technology.
Currently rock-solid base core cluster architecture looks like this:

We are utilizing:

Load balancers: Linux Virtual Servers with DirectRouting on CentOS5 (configured as a part of Redhat Cluster Suite)

SNMPv2(LVS,OSes,JBOSS,Oracle) to monitor everything with single (selfwritten) Java application which graphs everything in realtime.

As this is basic configuration with database as an single point of failure, in Septemer I’m going to setup DataGuard for Oracle. Also I’m testing more advanced scale up. Currently I’m in process of setting up Solaris Cluster with Oracle RAC 10gR2 implemented on iSCSI storage provided by third node based on Solaris Nevada with iSCSI target to test Transparent Application Failover. I’ve been scratching my head over this one for awhile now. Yeah, it is real hardcore… more over that’s not the end of the story – Disaster Recovery with some other interesting bits of technology is going to be implemented later on… all on x86_32 comodity hardware Also we are going to put C-JDBC(Sequoia project) under stress…

There is some kind of incompatibility between Linux 2.6 NFSv4 server nad Solaris 10 (U3) NFSv4 client. On installed Solaris you can put some variables into /etc/default/nfs and it should work, but when you are trying to bootstrap from Linux NFS server using Jumpstart you have to search for another solution:

1) Build a new miniroot image with /etc/default/nfs altered?
2) Simpler… alter Linux NFS server to provide only eg. only NFSv2 service
This can be achieved by recompiling kernel without NFSv4 or by much more cleaner solution – disabling NFSv4 services on runtime.

For more info consider reading man pages for rpc.nfsd and rpc.mountd. Internally those switches write “+2 -3 -4″ to /proc/fs/nfsd/versions. Versions file can be only modified after stopping [nfsd] kernel service ( you’ll get EBUSY errno while trying to change it with nfsd lanuched ).

After adding following snippet to OpenLDAP’s slapd.conf file we are preventing anyone from viewing user password(including Solaris LDAP proxy bind, excluding logging in user and admin/Manager of slapd):
access to attrs=userPassword,shadowLastChange
by dn="cn=admin,dc=lab1" write
by anonymous auth
by self write
by * read

“Sun StorageTek Availability Suite, or AVS for short, is an OpenSolaris Community project that provides two filter drivers; Remote Mirror Copy & Point in Time Copy, a filter-driver framework, and an extensive collection of supporting software and utilities.

The Remote Mirror Copy and Point in Time Copy software allows volumes and/or their snapshots, to be replicated between physically separated servers in real time, or by point-in-time, over virtually unlimited distances. Replicated volumes can be used for tape and disk backup, off-host data processing, disaster recovery solutions, content distribution, and numerous other volume based processing tasks.”

Today I configured the following scenario:

Both avs1 and avs2 nodes are running OpenSolaris Nevada build 65. It works great! Each of the nodes is running ZFS mirror on two disks. Also the AVS bitmaps should be RAID protected (for example using SVM). After the DR switch:

AVS was commercial project, but Sun decided to release it for free as a Open Source project, so enjoy!

Some time ago I’ve written proof-of-concept Solaris loadable kernel module to demonstrate sending packets from kernel space. You can see proof-of-concept MPEG movie here. Similar modules have been floating on the net for Linux for years, but there wasn’t any for Solaris. The plan was to write backdooring LKM with networking abilities possibly with some advanced hiding features like controling Balrog from DNS server – Balrog had to simulate DNS client making requests to /etc/resolv.conf’s proxy DNS servers ( the idea was to fool firewall/IDS/IPS systems which allow DNS traffic from servers ). Due to lack of time I had to abort the project - only bits of code responsible for sending and reciving have been written, even without in-kernel DNS library. On the movie you can see sending data on UDP port 53 after module initialisation. It was real hackery to get things done simply because orginal Solaris 10 kernel didn’t have API for accessing kernel-side of sockets ( fortunately source code from OpenSolaris helped me a lot ;] )… The resuling C code of Balrog is so ugly that I’m not going even to release it, however today I’ve noticed new OpenSolaris project named kernel-sockets so maybe it’s time for a small rewrite ?

Linux target is running Debian/4.0, 2.6.18 kernel and iSCSI target version 0.4.14 – I wish it was Solaris box, but my very old home SCSI controllers aren’t supported by Solaris ( DELL MegaRAID 428 – PERC2 and InitIO ) – however there are some drivers but for Solaris 2.7-2.8, but after small war with them I must say that I failed…. even after playing hardcore stuff in /etc/driver_aliases