Thursday, June 7, 2012

Since its introduction in SQL 2005 SP1, mirroring has become a common method to provide high availability for critical databases. With any HA strategy, proper monitoring is a must to ensure everything is working as planned with minimal data loss and disruption incurred in the event of a failover.

SQL Server provides several pertinent performance counters, as well as the Database Mirroring Monitor, to provide a base level of visibility into mirroring. Unfortunately there are known issues with alerting on mirroring thresholds as listed in Microsoft Connect here.

The issue is marked as ”Closed as Won’t Fix”, which leaves us with the need for a workaround.

The discovery of this led me to write this post. Not only will I provide scripts to configure a workaround, but I’ll show you how you can gain additional visibility and automation by combining this solution with SQL Sentry’s comprehensive monitoring and alerting capabilities.

While still available, mirroring will likely be used less in SQL Server 2012. The new Always On technology provides many enhancements over other HA options, and I’ll likely write a post about that in the future. See Aaron Bertrand’s (b|t) blog post on TechNet for an overview.

That said, I don’t suspect everyone will immediately upgrade to this new technology, so mirroring will be around for a while.

After some testing I quickly realized the solution wasn’t working as the alerts were database specific. Changing the "Database name:" field in the alert definition to <All Databases> in the alert properties fixed the issue. This may be sufficient if you only have one database mirrored on this server, but not if you have more than one since the alerts, and really the underlying error messages themselves, don’t provide for a database name.

I considered several options for a workaround. Ultimately, I wanted to make use of what native resources were reliable and practical as I didn’t want to reinvent the wheel. I know many people use a lot of the native functionality as covered in the MSDN article, Monitoring Mirroring Status, which discusses the dbmmonitor system stored procedures.

The Solution

So, my solution was to generate new error messages to replace the four native mirror performance errors. They are listed in the Technet article with Error numbers, 32040, 32042, 32043, and 32044. The below script creates new ones where I’ve replaced the leading 3 with a 5. The error messages are essentially the same as the original, except I’ve added a parameter for a database name. If you’ve already created your own custom error messages with those numbers, you’ll want to modify the script accordingly.

-- Create custom mirroring error messages to replace native ones so they can be database specific.

-- Error numbers are same as native, except leading 3 is replaced with a 5.

-- Severity level is 10, but error numbers and severity levels can be modified to suit your environment.

-- Created by: Steven Wright, SQL Sentry

USE master

-- Check to see if there is already an error with this ID in sysmessages

Now that we’ve created these enhanced error messages, we need to ensure that they are the ones raised when your mirroring performance thresholds are exceeded. I’m referring to the standard warning thresholds you can set natively in SQL Server as described in MSDN.

Normally these errors are raised after a check by sp_dbmmonitorupdate. If you’ve used sp_dbmmonitoraddmonitoring to set things up, you should have a job on your server called ‘Database Mirroring Monitor Job’ that runs the update proc at scheduled intervals.

Below is a script to create a replacement proc called sp_Sentry_dbmmonitorupdate that is otherwise identical, but raises our newly created errors and passes a database name to give us more complete details. Simply replace the call to the native proc in the job’s step with a call to this new one in msdb. If you call it with a null parameter for database_name, it will cycle through each mirrored database, passing the appropriate name when raising the error.

USE [msdb]

GO

/****** This is a modified version of sys.dbmmonitorupdate which raises customized errors for mirroring to include the dbname.

Written by Steven Wright, SQL Sentry ******/

SET ANSI_NULLS ON

GO

SET QUOTED_IDENTIFIER ON

GO

CREATEprocedure [dbo].[sp_Sentry_dbmmonitorupdate]

(

@database_name sysname = null-- if null update all mirrored databases

)

as

begin

set nocount on

if (is_srvrolemember(N'sysadmin') <> 1 )

begin

raiserror(21089, 16, 1)

return 1

end

if ( db_name() != N'msdb' )

begin

raiserror(32045, 16, 1, N'msdb..sp_Sentry_dbmmonitorupdate')

return 1

end

declare @retcode int

if object_id ( N'msdb.dbo.dbm_monitor_data', N'U' ) isnull

begin

createtable msdb.dbo.dbm_monitor_data ( -- go through the code to see if there SHOULD be nulls.

Finally we need to create our SQL Agent Alerts that are fired when these errors are raised. SQL Sentry Event Manager can monitor these alerts. This allows us to automate responses, such as sending an email, or take other more advanced actions that SQL Sentry provides. We can then create a custom view that allows us to see a chronological, Outlook-style calendar picture of this activity alongside any other events like high impact queries that may trigger our alerts.

In addition to the four performance thresholds covered by the errors we’ve created, I’ve added some additional alerts for monitoring status changes such as mirror failovers as well as monitoring for automatic page repair. Automatic page repair is a great feature that was introduced with SQL 2008 and provides for automatic repair of corrupt pages using the partner database since it is an exact copy.

Paul Randal (b|t) has more details about automatic page repair in his blog.

So here is the script to create the alerts. Note I tried to keep the format as similar to that used in the original TechNet article for consistency.

We’ve now successfully created our workaround and should be getting properly detailed alerts based on those conditions. Remember to run these scripts on both your principal and mirror server for complete coverage.

Increased Visibility and Notification with SQL Sentry

Now that we’ve set up reliable detection of our mirroring thresholds, let’s take a look at the increased visibility and automation that can be provided with SQL Sentry Event Manager.

SQL Sentry Event Manager can monitor many SQL related events and show this activity in an Outlook-style calendar view. Among the events that can be monitored and shown are SQL Agent Alerts like the ones we just created.

In our Log Shipping and Replication whitepaper, we show you how to create Custom Event Views for providing visibility into log shipping and replication activity. The same functionality can be applied to mirroring and the mirroring alerts we created. In fact, if you are monitoring your mirrored servers with Event Manager and Performance Advisor, known as the SQL Sentry Power Suite, you can add any Top SQL statements to the view. This allows you to see historically, the exact times high impact queries were executing alongside any mirroring alerts that were likely generated due to the extra load as shown in the example screenshot.

Notice we get our status alerts too. In the above test I ran queries to generate enough activity against the mirrored database to trigger some of our performance alerts. I can see and be notified of mirror failovers the same way as seen below.

With both scenarios I get confirmation from both sides of the mirror because I’ve set up my alerts on both the principal instance and the mirror instance. The same visibility and notification applies to my automatic page repair alerts.

Automation

Monitoring your mirrors with Event Manager can allow you to add automation to your environment as well. The most common action would be to get emails or pages when these alerts are raised. That is basic functionality in Event Manager without the need for database mail. You can get a quick review on setting up notifications in the SQL Sentry User Guide.

Another example might involve an application that requires access to multiple mirrored databases on the same server. Should one failover, you want them all to failover together. SQL Sentry’s Execute SQL actions allow you to do just that. Simply enter the appropriate commands in the Execute SQL action for this alert, and SQL Sentry can automate the failover of the other mirror.

Director of Client Services, Jason Hall (b|t), has a blog post detailing the use of Execute Actions in SQL Sentry.

New in SQL Sentry Version 7

SQL Sentry version 7 has added even more visibility into mirroring. First you may have noticed a new node under your SQL Server instances in the Navigator pane labeled Databases. Double clicking on a database provides details including mirroring information.

Additionally, you may have noticed new metrics in the Performance Advisor dashboard. In the upper right, we have provided a chart for backup throughput. In version 7, should we detect at least one mirrored database on the instance, we’ll show Send and Redo Queue size as well. This provides both real time and historical views into the size of either queue.

When you combine this with the calendar visibility for these events and alerting on all these mirroring thresholds and status events, you get comprehensive coverage and can rest assured you’ll know whenever your mirrored databases experience any issues. If you haven’t already, download a free evaluation copy of the complete SQL Sentry Suite at http://www.sqlsentry.net/download-trial/trial.asp. And if you have any questions, don’t hesitate to contact me or the rest of the team by emailing support at sqlsentry.net.