Check Permissions First

If you recently did a server move or similar type of operation, make sure zimbra:zimbra permissions are applied correctly to store directory. You could run the following to double check permissions (as root):

Error - Not Enough Memory To Run zmblobchk

If the "Processing BLOB store" part errors out with Java out of memory, you could try adjusting the variable for it. The default is set to 30%. This would require a mailstore/jetty restart though [zmmailboxdctl restart]. If the mailstore doesn't come back up, check /opt/zimbra/log/zmmailbox.log - you most likely exceeded the threshold for your box. Remember, this is a percentage and a 32bit machine with more than 4GB can cause issues with this setting since it could try to allocate more than it actually can to the thread.

BLOB Issue Script

I've commented out the two delete lines and also the "$msghash &= $fmask;" one.

If you're using HSM, this script isn't smart enough to detect it. It will flag and remove entries that are in the HSM store because it doesn't see them in the primary store.You can modify the output though to use the actual path to the HSM store. I had one case where this was the case and copying the missing blobs to the HSM path worked.

Comment from other support staff member on a case that used this.

Once you've gotten a list of the 'NOT OK' files generated by this tool, you'll want to check the store directories to see if the message blobs exist with an incorrect change number. That's the second number in the filename. If the first number matches the ID it's looking for but the second does not, you can rename the file to the new change number and get the message blob back without losing data. If not, then the blob is completely missing. If that's the case, we should try to figure out what happened to it by going through the mailbox.log files. You should be able to grep for the missing message ID and the user's account ID to find a log entry showing something happening to it. It's possible it was deleted and the deletion was never written to mysql for some reason.

Remember this from above, "you'll want to check the store directories to see if the message blobs exist with an incorrect change number. That's the second number in the filename. If the first number matches the ID it's looking for but the second does not, you can rename the file to the new change number and get the message blob back without losing data."

One can uncomment the 2 lines in the script with delete and run the script again. This will remove the reference to the blob.

To re-index the users mailbox [this can take start, cancel, status]:

zmprov rim user@domainname start

Running the script again shouldn't show "NOT OK" lines.

FYI - I still need to gather more information about this situation and how to guide one to make a decision to delete, re-index, and so forth.

BLOB Script To Copy From Restore

You might need to use these variables with the restore command. I need to find a way for someone to figure out what restore to goto for a particular blob:

-restoreToTime <arg> - Replay the redo logs until the time specified.

-restoreToIncreLabel <arg> - Replay redo logs up to and including this incremental backup.

-restoreToRedoSeq <arg> - Replay up to and including this redo log sequence

-br - Replays the redo logs in backup only, which excludes archived and current redo logs of the system.

-rf - Restores to the full backup only, does not include any incremental backups since that backup

BLOB Script To Copy From Restore (HSM Issue)

One customer ran into an issue that the blob references were really expecting them in the HSM mailstore path.

Example:

Default mailstore for PROD : /opt/zimbra/mail/

HSM mailstore path : /opt/zimbra/mail0 which was another partition mount, compared to /opt/zimbra/mail

So, what this script did was used the output from the blob-check.pl script which has the missing blobs formatted for the PROD path rather than the HSM path. The blob-check.pl script doesn't handle HSM data. The script then finds the blobs in the full backup directory of the user and copies them into the HSM path that the WEBCLIENT is expecting. This whole situation with this customer might of been a fluke. So don't consider these steps as applying to your situation unless you do your research. This case took 9 days to resolve and figure out.

Notice, if you use this script below you really need to review it in details and adjust it for your situation. There's more to adjust beyond the base variables.

#!/bin/bash
# Steps
# Make sure directory path exists for the copy command
# Get mailboxId - zmprov ga USER@DOMAIN | grep zimbraId:
# mkdir /opt/zimbra/mail0/0/[mailboxId #]/msg/{1..15} or how what ever # you need.
# Change USEREMAIL
# Get path for BACKUPFILES and uncomment for use
# Run find command with echo and confirm. Then rerun using the find with cp rather than echo
# Put in user's email for USEREMAIL
USEREMAIL="USER@DOMAIN"
ZIMBRAID=`zmprov ga $USEREMAIL | grep zimbraId: | awk '{print $2}'`
# See below
# ROOTBACKUPPATH="/opt/zimbra/mail0/BACKUP/sessions/full-20080707.004336.789"
#The line below takes to long, see instructions underneath
#BACKUPFILES=`find $ROOTBACKUPPATH -name *$ZIMBRAID*`
# To run manually, which will increase speed of script if you then paste full path in for BACKUPFILES var
# zmprov ga USER@DOMAIN | grep zimbraId: | awk '{print $2}'
# Then place the out in this find command, leave the wildcards on each side of it
# find /opt/zimbra/mail0/BACKUP/sessions/full-20080707.004336.789 -name *OUTPUTHERE*
# This should give you path like the exampe below, paste it between the quotes and uncomment the variable BACKUPFILES
#EXAMPLE FOR BACKUPFILES="/opt/zimbra/mail0/BACKUP/sessions/full-20080707.004336.789/accounts/a07/1b5/a071b5ad-b341-4de4-b6ee-4463f322873b"
# BACKUPFILES="PASTE_IN_PATH_HER"
for BLOBPATH in `blob-check.pl $USEREMAIL | grep "NOT OK" | awk '{print $1}' | sed 's/store/mail0/'`
do
MSGID=`echo $BLOBPATH | awk -F/ '{print $NF}'`
# Run echo first to make sure copy path seems correct, ctrl-c to cancel. And then uncomment the other find command with the cp. Comment the echo one.
find $BACKUPFILES -name *$MSGID* -exec echo {} $BLOBPATH \;
#find $BACKUPFILES -name *$MSGID* -exec cp -uv {} $BLOBPATH \;
done

The Really Bad Situation, Blob References Gone & Mail Still Missing

Method One - zmlmtpinject

Method Two - Use IMAPSYNC

I haven't tested this and would just be guessing, but I believe IMAPSYNC could be used.

From the IMAPSYNC notes, "imapsync is the adequate tool because it reduces the amount of data transferred by not transferring a given message if it is already on both sides. Same headers, same message size and the transfer is done only once. All flags are preserved, unread will stay unread, read will stay read, deleted will stay deleted."

This would involve having imap setup on ZCS and then doing a redirected user restore (which would consume a license).

The "new" account would be restored-userPROD@servername.com . The -rf flags should look only at the backup data in regards to the restore and not catch the "production" references to missing blobs and what not.

You could use the following command to bring the blob back in and regenerate the metadata -- zmmailbox am along -d . The am flag details are : addMessage(am) [opts] {dest-folder-path} {filename-or-dir} [{filename-or-dir} ...] . But the -d flag states : -d/--date <arg> received date (msecs since epoch) . To get the epoch time in seconds using the same receive data, for example :

ls -l --time-style=+%s /opt/zimbra/store/0/1303/msg/0/1059-1726.msg

And then remember the -d requires milliseconds [a thousandth (1/1,000) of a second] so you'll need to convert it. Or, you could just use something like this below. This with the cut would work well for a for loop script against a number of msg files.