Contingency Planning

I want a help to implement a contingency plannning, for Jira and Jenkins (running on tomcat), and SVN over Apache? Where do I start my job, where is the best choice and lower price choice? Thank you very much if you can help.

I'm in a similar situation and have been thinking about stuff like this. Please don't take anything I say as advice because I haven't implemented it. I'm just looking for discussion.

It seems there are two approaches we can take: High-Availability, and Fall-over. My definition of these terms is something like:

High-Availability is redundant servers with mirrored data. Users can access any server so if one goes down one of the others pick up the load automatically or nearly so.

Fail-Over also has redundant servers but lack the automatic part. Users can only access the active server and when it goes down the backup is put into service with a (nearly) up to date copy of the necessary data.

I use svn and mysql. I'm not sure what JIRA and Jenkins use to store their data but I suspect they use similar to one of those, a database or file based architecture.

What I'm thinking of is to set up a duplicate server and use rsync to backup svn (as we're doing now) and use the built in mysql mirroring for the db. Then when the primary server has a problem the machines get swapped. I'm not sure if that's a DNS change which can take hours to propagate or an IP address change.

Then when the problem is corrected that system becomes the backup.

The problem is that the rsync job probably will run once an hour or so so there may be data lost during that time and it will be hard if not impossible to recover it.

As I said, I haven't implemented this yet so I'm not sure of what I'm missing. I am interested in the discussion.

Joe

It's not what your program can do, it's what your users do with the program.

Joao Bosco Jares
Ranch Hand

Joined: May 13, 2011
Posts: 44

posted Nov 25, 2013 09:19:38

0

Hi Joe Areeda, you are right, no mistery here. I developed three diagrams much simple as possible to solve this issue with minimal resource too. I am sure that approach can be better with more resources (load balancers, machines, heartbeat, etc..) but for my requirements I am starting simple as possible.
Please tell me your doubts and review with you can. Best regards. Thanks a lot.

My Spanish is weak but I think I understand you diagrams well enough to comment.

What worries me is the load balancing and multiple virtual hosts seem to be relying on file replication to allow clients to use multiple servers to access the data simultaneously.

If the passive servers are read-only this will work. But the bottom diagram with Apache acting as a load balancer to work with 2 copies of the SVN repository is asking for trouble, in my humble opinion. Imagine two people trying to update the same file at the same time on different systems. Without some cross system file locking (which doesn't exist as far as I know) the repositories will be out of sync. I admit I don't know if Subversion has some way to deal with a problem like this.

In my scenario only one of those servers would be active at any one time because of this exact problem.

I'm not 100% certain, but I think SVN has good enough file locking that concurrent access by multiple users via filesytem access (as opposed to something like the svn server) is safe. That is, of course, as long as you use the standard svn utilities and not some brute-force code of your own. Subversion itself was designed for atomic version control.

If 2 people attempt to commit the same file at the same time, there should be a conflict which would require resolution. That is true whether updating via svn client program, apache http or the svn server. So for that particular product, you'd be safe on a single repository. But I'm not so sanguine when rsyncing, especially if both versions are being updated. A better solution is to keep one repository as the active one, mirror to the other copy and switch only when the primary goes down.

Incidentally, I use drbd to mirror my repositories and that's pretty much how drbd works except at the block device level and not at the file level. I make actual (svndump) backups daily and I don't worry TOO much about lost data in the interim, since I also have a backup of sorts in the original desktop working copy I did the commit from (and in the case of IDEs such as Eclipse, also the daily local history).

Last time I installed a Jira server, I was using Oracle as its backing database, although that's not the only DBMS you can use. That particular system was on a SAN with RAID disks and tape backups. Mirroring databases is generally fairly tricky and should be done with database-specific tools and not simply rsyncing, as there's a chance of file corruption otherwise.

I'm not 100% certain, but I think SVN has good enough file locking that concurrent access by multiple users via filesytem access (as opposed to something like the svn server) is safe. That is, of course, as long as you use the standard svn utilities and not some brute-force code of your own. Subversion itself was designed for atomic version control.

If 2 people attempt to commit the same file at the same time, there should be a conflict which would require resolution. That is true whether updating via svn client program, apache http or the svn server. So for that particular product, you'd be safe on a single repository. But I'm not so sanguine when rsyncing, especially if both versions are being updated. A better solution is to keep one repository as the active one, mirror to the other copy and switch only when the primary goes down.

Incidentally, I use drbd to mirror my repositories and that's pretty much how drbd works except at the block device level and not at the file level. I make actual (svndump) backups daily and I don't worry TOO much about lost data in the interim, since I also have a backup of sorts in the original desktop working copy I did the commit from (and in the case of IDEs such as Eclipse, also the daily local history).

Last time I installed a Jira server, I was using Oracle as its backing database, although that's not the only DBMS you can use. That particular system was on a SAN with RAID disks and tape backups. Mirroring databases is generally fairly tricky and should be done with database-specific tools and not simply rsyncing, as there's a chance of file corruption otherwise.

kkkkk yeah is Brazilian Portuguese So, my solution is exactly as was described by Tim. I wrote some texts to expose the solution beyond the diagrams, but I am with Joe that I must adjust the diagrams to explain better about the active and passive AHS modes, and minimize the miss understanding about the two filesystem concurrency, that does not exist. So when server breaks is necessary a svn restore on the last dump.