Answered by:

Exchange 2010 Corrupt Mailbox and Rules issue.

Question

We appear to be experiencing problems with a corrupt mailstore, we are getting messages like the ones posted below below in the application event log.

We use outlook 2010 final and some rules are not working for any user, the rules that are working it is not showing new mails that go into folders.

At '26/07/2010 09:59:27' the Exchange store database 'Mailbox Database 1095862411' copy on this server detected corruption on the active copy of the database. To help identify the specific failure, consult the Event log on the server for other storage and
"ExchangeStoreDb" events. Service recovery was attempted by failover to another copy, which was unsuccessful in restoring the service. Error: There is only one copy of this mailbox database (Mailbox Database 1095862411). Automatic recovery
is not available.

Answers

You can take the database offline and run a ESEUTIL /P to repair the database, however, realize that depending upon the level of the damage in the EDB it could simply repair the few problem pages OR it could make the database unusable. From what you
are telling me I would

Make a backup of the EDB, LOG set

Create a new Database

Migrate your users over to the new database

Basically get ALL of the users moved or all of them moved that you can do without error

If you get errors on moving some of the mailboxes, then I would A: Export the users information to PST so that you have a backup and then B: try a /P against the database with the remaining users. Once the /P completes you will need to run an ESEUTIL
/D and then remount the database

Have the users check to see if they can access their mailboxes and if it looks like any data is missing

Move the mailboxes to the new EDB and Kill the OLD one

Keep the PST's around in case any of the problem users are missing data

All replies

First off I would check your event logs to see how long this has been going on. This problem is frequently caused by a hardware failure or by improper configuration of the anti-virus product and indicates that the logical database structure has become
corrupted. This may occur for one or more of the following reasons:

Disk caching has not committed transactions to the hard disk and the server has stopped responding (crashed).

Incorrect log files were replayed during a database restoration.

The server has a defective hard disk controller. Note A defective hard disk controller is typically indicated by disk input/output (I/O) errors that are listed in the application and system logs.

Database log files have been removed that were not fully committed to the database. (Typically caused by AV solution misconfiguration)

Because logical database structure corruption is not detected by the backup program, you may not immediately become aware of the problem. It only appears when a user tries to access the page that the data is stored on which is what it sounds like is happening
here.

Resolutions:

Check your event logs and try to figure out how long this has been going on.

Check your backup system, have you been getting successful backups? When was the last known good backup? If its still showing as backing up properly see # 5 above, i.e. the backup process doesn't check the validity of the entire DB so it
may show as 100% good even though the issue exists.

Unless you can find some issue with your anti-virus product ore recently made some upgrade right before the issue started I would make an immediate plan to build a new server and migrate your users to it ASAP before the problem becomes worse.

The only time I have determined not to move off the hardware is when everything was fine until a customer updated the AV solution, added more memory, new driver etc and they can then undo the upgrade that caused the problem. Even then
however, AFTER correcting the issue I would create a new database and migrate the users into it because the existing database does have corruption in it.

You can also look at using ESEUTIL /P to correct the errors, however BEWARE that this is a drastic action that could very well decimate the database and make it completely unusable and should only be undertaken AFTER MAKING A COPY, and ONLY if its
your last ditch effort and ONLY after exporting the data from the EDB to PST so that you have an alternative recovery path.

If you want a robust GUI driven way to export mailbox data from the Production Exchange server OR a way to repair the offline EDB and then extract the data to PST MSG or XML you should check out Lucid8's DigiScope
http://www.lucid8.com/product/digiscope.asp

You can take the database offline and run a ESEUTIL /P to repair the database, however, realize that depending upon the level of the damage in the EDB it could simply repair the few problem pages OR it could make the database unusable. From what you
are telling me I would

Make a backup of the EDB, LOG set

Create a new Database

Migrate your users over to the new database

Basically get ALL of the users moved or all of them moved that you can do without error

If you get errors on moving some of the mailboxes, then I would A: Export the users information to PST so that you have a backup and then B: try a /P against the database with the remaining users. Once the /P completes you will need to run an ESEUTIL
/D and then remount the database

Have the users check to see if they can access their mailboxes and if it looks like any data is missing

Move the mailboxes to the new EDB and Kill the OLD one

Keep the PST's around in case any of the problem users are missing data

Microsoft is conducting an online survey to understand your opinion of the Technet Web site. If you choose to participate, the online survey will be presented to you when you leave the Technet Web site.