We currently have a multiple server setup with two main mailstores and a single archive server. We have 4000 users, and use the standard zimbra backup disk-disk for the mailstores. For now we have no plans on backing that data up any other way. for DR , we may just rely on the archive. We have seperate ldap and proxy server that gets backed up using zimbra backup, then the backup directory gets moved to tape - since the dataset is so small.

The archive server catches pretty much everything. I want to move all of this data to tape for offsite storage. Using the standard zimbra backup is not practical for this - since we use Tivoli Storage Manager, and it will have to back up the entire archive backup every night, practically. (Multiple TB of data) The easiest way would be to just back up to tape the actual stores, and mysql data.

What would be the best way of backing this up? Should I use the FOSS method of LVM snapshots - or since I have seperate partitions can I just do a mysql dump for metadata, backup static config files, and then just use tsm to backup /opt/archiveX while zimbra is running?

I was smart, and created my huge stores /opt/archiveX with LVM leaving some free space on the VG in case i want to use a snapshot for backups.

Thanks.

02-15-2011, 07:18 AM

bdial

i don't see how you would use the archive server for DR. Doesn't that have *all* e-mail? so if your production setup ever died how would you restore accounts to the state they were in?

Have you looked at doing grouped backups for the archive server? so it would only full backup a few accounts per night, and incremental the others. That way it never has to actually backup Multiple TB at one time.

02-15-2011, 07:34 AM

i2ambler

I would prefer to use the actual zimbra stores themselves as DR, but its just not practical without using some sort of san to san replication. Unforunately I have no power over quota - so every user seems to have well over a gig of email. And some users up to 10+ gigs. (!!) I have mulled over the idea of only backing up to tape the big wigs - and everyone else will just have to live with losing their data if we have to go to DR. Not a very good solution, but, I cant think of any other way to do it..

Either way, I need to get a backup of the archive to tape for compliance - its just a question of how I do it.. if I just snatch the /archiveX directories all I will get every night with TSM is just the changes.. never more than that. TSM never gets a full backup other than the first time you use it... so its great for a situation like this, but not so great for a situation like trying to backup the /backup directory since each backup is put into its own dated directory and changes every night.

02-15-2011, 06:45 PM

LMStone

We decided not to add separate servers for Zimbra Archive & Discovery mailboxes and from the backup reports the archive mailboxes are backed up just like any other mailbox nightly.

You could run a Full backup say, just monthly, and then run only incrementals in between. Alternatively, to use less disk space, run a Full backup every day and destroy the previous day's disk backup after copying it to tape (you'll just need to delete the redo logs periodically since only incremental backups move them into the backup directories). Recall Zimbra changed the default backup options a few revs back so that unless you add the "--noZip" option to the backup crontab, backups no longer use hard links, making each backup independent and capable of being moved offsite, or at least off server.

Also, I too concur that relying just on the A&D server's mail store for D/R is suboptimal at best!

Hope that helps,
Mark

02-17-2011, 11:38 AM

i2ambler

Backing up the zimbra archive server with zimbra's backup would take an enormous amount of space - and since each and every night all files would change the backup to tape would take over 24 hours, I would think. If we have multiple TB of archive doing a full backup each night is just not practical nor is it really needed - since I wont ever have to recover an individual archive mailbox like I would a users normal mailbox. TSM, however, can just grab the daily changes each night - which may only be a few gigs. *IF* I use the FOSS method, or a modified FOSS method.

I am trying to think of a practical way of DR and backup/recovery.. Like I mentioned - maybe using the FOSS way of backing things up would work? (using lvm snapshot, then swooping in with TSM to grab the snapshot)

OR - would backing up the message store using TSM and then just doing a mysql dump work without having to do lvm snapshots? would Zimbra be recoverable with that backup? If I dont get the /index directory - will the indexes be recreated? TSM wont try to copy anything that is 'open' and in use.. that may be the way I will go with DR for both archive and regular mail stores.. and just use the zimbra disk to disk backups for individual mailbox restores..

I created the mailstore VG on the archive server with enough space for snapshots, but the other mailstores do not have space for snapshots. I could recreate them. I was hoping to not have to do lvm snapshots and/or shutting down zimbra for the entire time backups are occuring. I would much prefer using the mailbox stores and not the archive for DR....

I hope this makes sense... I have to imagine other uses have run into this issue before..

02-17-2011, 05:40 PM

LMStone

Quote:

Originally Posted by i2ambler

Backing up the zimbra archive server with zimbra's backup would take an enormous amount of space - and since each and every night all files would change the backup to tape would take over 24 hours, I would think.

If you used the "--noZip" option in the zimbra backup cron tab then the use of hard links will result in the only increase in each day's archive server backups to be about the size of that day's inbound and outbound messages. Is Tivoli able to handle hard links? If so, Tivoli incremental backups of /opt/zimbra/backup should be fairly quick.

Quote:

Originally Posted by i2ambler

If we have multiple TB of archive doing a full backup each night is just not practical nor is it really needed - since I wont ever have to recover an individual archive mailbox like I would a users normal mailbox.

Are you really sure about that?

I've never seen a situation where the client was willing to make the spend to have A&D but was fine with losing the A&D store in the event of a hardware failure.

If storage is really scarce and you are running A&D for every mailbox, what about dropping all backups of regular mailboxes and just run zmbackup against the A&D mailboxes?

If the whole farm died, you would restore all the A&D mailboxes to a new server farm, create the regular mailboxes, and share out each user's A&D mailbox read-only to each user's empty mailbox.

Indeed, you could do that now and really crank down the quotas on the regular mailboxes to claw back space, since users would have read-only access to their own quota-less A&D mailbox.

It's not the use case originally contemplated for A&D, but...

Hope that helps,
Mark

02-22-2011, 12:38 PM

i2ambler

[QUOTE=LMStone;209984]If you used the "--noZip" option in the zimbra backup cron tab then the use of hard links will result in the only increase in each day's archive server backups to be about the size of that day's inbound and outbound messages. Is Tivoli able to handle hard links? If so, Tivoli incremental backups of /opt/zimbra/backup should be fairly quick.

Is there an issue with 6.0.7 and the --noZip option? I havent bothered to upgrade since we were awaiting 7.0 I am only using about a week's worth of backups on the zimbra server by using the zmbackup -del 8d option to save hard drive space. TSM will handle and backup hard links.. Can you point me to a location that describes more info on the --noZip option and how it works with hardlinks? What will happen if I implement --noZip now, after I have been running backups for a while? I would love to be able to use TSM to backup our regular stores without having to use any silly LVM magic.

We arent OK with losing the archive data.. we just wont have to go in and recover and individual mailbox - we may have to restore the entire archive.. but not an individual mailbox - since users dont have read-write access. I will, however, have to recover individual non-archive mailboxes... I have already had to do that. We also, already, share out the read-only archives to people who seem to want to have access to ridiculous sums of email.

That is sort of where this whole conversation started.. For me to use the A&D box as a DR.. if the farm dies - the A&D server can be restored, users recreated, and the users could have access to their unorganized email. HOWEVER, I can easily use the regular stores as DR if the --nozip will indeed work to minimise backup time with TSM. The other item, I dont want to have to use zimbra backup on the archive: is there a way to backup the data without having to snapshot the LVM? I have seperate luns for the mail storage and the mysql data.. Its not a big deal - since I already created the archive volume groups with space for snapshots. I just hate to double(or probably quadruple) the harddrive space needed for the archive by implementing the zimbra disk-to-disk backup. My boss just looks at me funny when I say we basically would have 4 copies of the same mail.. on the SAN alone!

Thanks for the great advice!

02-22-2011, 12:50 PM

LMStone

[QUOTE=i2ambler;210417]

Quote:

Originally Posted by LMStone

If you used the "--noZip" option in the zimbra backup cron tab then the use of hard links will result in the only increase in each day's archive server backups to be about the size of that day's inbound and outbound messages. Is Tivoli able to handle hard links? If so, Tivoli incremental backups of /opt/zimbra/backup should be fairly quick.

Is there an issue with 6.0.7 and the --noZip option? I havent bothered to upgrade since we were awaiting 7.0

Thanks for the great advice!

It used to be that the "--noZip" option was the default behavior for zmbackup. But somewhere around 6.0.7-ish or so (I don't know exactly when, sorry!) Zimbra changed the default. That had the effect of removing hard links from backups, making each backup easily portable offsite.

But if you keep your Zimbra backups local, you want the hardlinks to avoid using a ton of disk space every day for the same files.

A number of users started complaining here on the forums; asking why backups all of a sudden started taking up so much disk space. And then our alert system started warning us about low disk space on one of our backup volumes...

So now we read the Release Notes much more carefully! :-)

If it were me, I'd move up to 6.0.10 to get all the security fixes if nothing else. But, we are going to hold off on upgrading to 7 until we get our and our clients' Documents docs converted to Briefcase docs. There are several posts from unhappy users indicating the converter built in to the 7 upgrade installer failed, and they then lost access to their Documents since that feature no longer exists in 7.

I then filed an RFE in bugzilla to break out the Documents-to-Briefcase converter as a standalone tool we can execute on a running ZCS6 system. In that way, if certain Documents fail to convert, we can at least do a manual cut 'n paste before moving to 7.

Glad I could help a little here!

All the best,
Mark

02-23-2011, 12:06 PM

i2ambler

[QUOTE=LMStone;210418]

Quote:

Originally Posted by i2ambler

It used to be that the "--noZip" option was the default behavior for zmbackup. But somewhere around 6.0.7-ish or so (I don't know exactly when, sorry!) Zimbra changed the default. That had the effect of removing hard links from backups, making each backup easily portable offsite.

But if you keep your Zimbra backups local, you want the hardlinks to avoid using a ton of disk space every day for the same files.

A number of users started complaining here on the forums; asking why backups all of a sudden started taking up so much disk space. And then our alert system started warning us about low disk space on one of our backup volumes...

So now we read the Release Notes much more carefully! :-)

If it were me, I'd move up to 6.0.10 to get all the security fixes if nothing else. But, we are going to hold off on upgrading to 7 until we get our and our clients' Documents docs converted to Briefcase docs. There are several posts from unhappy users indicating the converter built in to the 7 upgrade installer failed, and they then lost access to their Documents since that feature no longer exists in 7.

I then filed an RFE in bugzilla to break out the Documents-to-Briefcase converter as a standalone tool we can execute on a running ZCS6 system. In that way, if certain Documents fail to convert, we can at least do a manual cut 'n paste before moving to 7.

Glad I could help a little here!

All the best,
Mark

Thanks.. Do you know how the 'hardlinks' work? I read somewhere that it hardlinks to your actual store? Since my backup directory is a totally different LUN, that will not work at all.

02-23-2011, 12:42 PM

LMStone

I could easily be wrong, but my understanding is that there are no hard links between the message blobs in the Zimbra Store and the message blobs in the backups.

With the "--noZip" option in current ZCS versions, there are hard links between the blobs in each day's backups. So, if user A has a 10MB PowerPoint attachment, and you keep thirty days of backups, backing up that message blob takes a little over 10MB, not 300MB.

Within the message store, there are hard links as well. So, if user A gets a 10MB attachment from user B, and then forwards the email with the attachment to users C and D, the message blob takes a little over 10MB, not 40MB, in the store.

Also, hard links are broken when you move mailboxes in NE to another server. IOW, if you moved users' A, B, C and D's mailboxes from zimbra server 1 to zimbra server 2, the four mailboxes will use at least 30MB more storage on server 2 than they did on server 1 -- a lot more if the users frequently forward large attachments between themselves.