SQL Server Service Broker demystified

Introduction

Service Broker was introduced in SQL Server 2005 and can be easily hailed as one of the best new features added to SQL Server. With Service Broker a database developer can create powerful asynchronous applications within a server or between different servers. The range of such applications may vary from simple workload queuing to complex cross-server auditing. Moreover, being part of the database engine, Service Broker enjoys qualities like transactional consistency, reliability, and security. In a nutshell, It is a powerful tool for a DBA and not that difficult to learn.

I will start with the basic terminology and components that make up a service broker application (and in the process we will also have a look at the code snippets to create those components).

Synchronous vs. Asynchronous Messaging

What makes Service Broker so powerful is the asynchronous messaging system that allows developers to develop loosely coupled components that work independently to perform some task. While in a synchronous messaging the interacting components wait on each other till the message is received (that is, the sender will not continue until the receiver has received the message), in an asynchronous messaging system the sender and receiver keep doing their work independently. The advantage of asynchronous communication (and thus of the service broker) is that the sender and receiver can overlap their computation because they do not wait for each other.

Transactional consistency

In the service broker infrastructure, the delivery of messages between two parties( called endpoints as you will later see) is transactional. That is, if a transaction rollbacks, all service broker operations within that transaction will also roll back.

The heart of service broker architecture is a dialog, which according to Technet, is reliable, persistent, bi-directional, ordered exchange of messages between two endpoints. An endpoint is the sender or receiver of messages. So, let's start with going through the components that make up Service Brokers.

Enabling Service Broker

First of all, we need to enable Service Broker on our database. To check whether Service Broker is enabled on your database you need to query the sys.databases catalog.

SELECT is_broker_enabled
FROM sys.databases
WHERE name = ‘Your_Database’
GO

To enable Service Broker on your database, we use the alter database command,

Be careful while running this command as WITH ROLLBACK IMMEDIATE will disconnect all the users.

Message Types

A message type is the definition of the format of the messages that will be part of the dialog between the two endpoints. Message type performs any validation required on the contents of the message before sending the message. If the message is to be sent to another database (on same or different server) the message type should exist on both the databases.

Permission for creating a message type defaults to members of ddl_admin or db_owner database roles and the sysadmin server role. The arguments and their definitions are detailed below:

message_type_name: Is the name of the message type to be created. The message_type_name can be upto 128 characters. By convention, the name is a URL of the form //<hostname/pathname/name> but using a URL format is not required.

AUTHORIZATION owner_name: This defines who will be the owner of the message. When the current user is sa or dbo, it can be the name of any valid user. In all other cases, it must be either the current user, the name of the user who the current user has impersonate permission for or the name of the role the current user belongs to. When this clause is omitted the owner is the user who executed the statement.

VALIDATION: This specifies how the message body for messages of this type are validated by service broker. When this clause is omitted the validation defaults to none. The possible values are :

NONE: No validation is performed. The message body may contain data or it may be NULL.

EMPTY: The message body must be NULL.

WELL_FORMED_XML: The message body must contain well-formed XML. When this clause is specified, the receiver loads the message into XML parser to ensure that it can be parsed. If the message fails this validation, it is discarded and an error message is sent to the sender.

VALID_XML WITH SCHEMA COLLECTION schema_collection_name: The message body must contain XML that complies with a schema in the specified collection. schema_collection_name must be the name of an existing XML schema collection.

Let’s have an example of creating a message type:

CREATE MESSAGE TYPE My_First_Msg
VALIDATION = WELL_FORMED_XML
GO

You can run the following query to view the message types in the database where you have created the message them:

SELECT * FROM sys.service_message_types

Although Service Broker does not require that both sides of the conversation use the same validation, it is better to have it this way to help troubleshooting.

If you have high message volume, each message passing through the XML parser can adversely affect the performance. So, you can avoid using XML validations in such case.

Contracts

Service Broker contracts are database objects which define which message type or types will be used in any conversation. Service Broker contracts define two different service roles: the initiator and the target. The initiator of the conversation starts the conversation by sending a message to the target. The contract that the conversation uses defines which service role can send messages of what message type.

To create a contract we use the Create Contract command. The syntax is:

AUTHORIZATION: Has exactly the same meaning as in creating a message type

message_type_name: is the name of the message that is to be included as part of the contract.

SENT BY: specifies which endpoint can send the particular type of message.

INITIATOR: specifies that only initiator can send the messages of the specified message type.

TARGET: specifies that only target can send the messages of the specified message type.

ANY: specifies that the messages of the specified type can be sent by either initiator or target.

[DEFAULT]: specifies that this contract supports messages of the default message type. By default, all databases contain a message type named DEFAULT.

Let’s continue creating these objects to be used in our messaging application,

--Either the initiator or the target may send the same message type
CREATE CONTRACT My_First_Contract
(My_First_Msg SENT BY ANY)
GO

A contract does not specify the ordering of messages. Service Broker requires the first message of a conversation to be sent by the initiator. After the first message, there are no ordering requirements in the conversation dialog.

More than one contract can use the same message type(s).

The message type and direction cannot be changed once a contract is defined. If you have to change the message type, you must drop and recreate the contract.

Queue

As I mentioned earlier that Service Broker is an asynchronous messaging infrastructure. Lets dive a bit deeper into this now. In asynchronous messaging, you send a message and start doing something else, without caring a bit about the receiver. The processing of the sent message depends solely on the receiver’s mood. He might process it now or later without casting any effect on what you are doing on your end. But, the message needs to be stored somewhere between the time you sent it and the time it will be processed. This magical place where your messages vanish just as you send them is called the queue.

You can imagine the queue as a table (actually SQL Server does implement queue via a hidden table in your database). You cannot directly manipulate ( insert, update or delete) this table though. To see the messages in the queue SQL Server provides you with a read-only view on this table which you can query. When a sender sends a message, it is put to the sender queue. The transport layer moves that message reliably to the destination queue. The receiver queue can now pull the message whenever it feels like. The message from the sender queue is deleted only after it is successfully moved to the receiver queue. So, there is no question of any message being lost in transit.

There is one temporary queue on every SQL Server instance, called the transmission_queue. When a message is sent over a network, Service Broker puts the message in this queue, on the initiator server. Then, the Service Broker sends the message over the network and marks it as waiting for acknowledgement in the transmission_queue. When the message is received in the target queue, Service Broker sends an acknowledgement back to the initiator. When this acknowledgement is received, the message is deleted from the transmission_queue. This process is called the dialog. Note that you will only see messages in transmission_queue when there is some message in transit. You can query the transmission_queue using the following select statement:

STATUS: specifies whether the queue is available or unavailable for use after creation. It is a good practice to set the queue off so the application cannot use it. Once the setup is done completely you can turn it on using the alter queue statement.

RETENTION: When OFF (default) the message is deleted from the queue as soon as the transaction that receives the message commits. When set to ON, the messages remain in the queue until the conversation ends. While it may seem harmless to keep it ON, doing so increases the size of the queue and may affect performance of the system as a whole. As a guideline, you should set it ON only when you have a business need to keep the messages around after processing

ACTIVATION: I will deal with this is the next section as it deserves detailed discussion

POISON_MESSAGE_HANDLING: specifies whether poison message handling is enabled for the queue. The default is ON. If set to ON, the queue will be disabled after five consecutive transaction rollbacks. Setting it to OFF allows to have a custom poison message handling system in place.

ON { filegroup | [ DEFAULT ] } : specifies the SQL Server filegroup where the queue will be created. If the volume of messages is going to be high in your application, it might be better to keep the queue on a separate filegroup.

--We will be creating a queue each for the initiator and the target
CREATE QUEUE MY_First_Queue_Init
CREATE QUEUE My_First_Queue_Target
GO

Activation

Service Broker Activation is the mechanism that enables the applications to scale dynamically based on the message traffic. Activation uses Service Broker to start an application when there is work for the program to do.

There are two types of activation: internal and external. Internal activation works with SQL Server stored procedures and the procedure is directly activated. External activation works with SQL Server event that indicates that the program should start another queue reader.

Service Broker activation works in two steps. First, Service Broker determines whether activation is necessary. Then, service Broker determines whether activation occurs.

Let’s get back to the create queue syntax and have a look at the activation clause and related arguments:

STATUS: If this is ON, Service Broker activates the stored procedure specified in the PROCEDURE_NAME argument. You can use this argument to stop activation of the procedure by setting this argument to OFF. The default value is ON for this argument.

PROCEDURE_NAME = <procedure>: Specifies the name of the procedure you want to be executed when the message arrives at the receiver queue. This value must be a SQL Server identifier.

MAX_QUEUE_READERS: specifies the maximum number of instances of the procedure that the Service Broker starts at the same time. Its value can have any value between 0 and 32,767. While you may feel everything right to set a high value to this parameter but you need to have a more careful analysis of your system. While a high value will definitely make your Service Broker application very quick to respond, but it would also add a lot of stress on your servers. So, give some time and thought to balance your application requirements and your server health.

EXECUTE AS: specifies the account under which the procedure will be run at the time of activation.

Service

Let’s think of our system as a postal service. To send a letter you would write an address on the envelope. In Service Broker world, this address is the service. A target service represents an address that accepts requests for the tasks identified by the contracts that the service specifies. An initiating service represents a return address for a conversation with a target service. Each service uses a queue to store messages. Messages sent to the service are delivered to the queue.

So as we have the basic infrastructure in place for our SQL Server postal services, lets get ready to send and receive a couple of messages. Before that just a brief description of what conversations are in Service Brokers. A conversation is the SQL Server primitive over which messages are sent. A dialog is just a special type of conversation where we have a two-way conversation between two parties (future versions of SQL Server will also support monologs).

To send messages you need to start a dialog. A dialog defines the exchange of messages between two services and specifies the contract that will be used for the messages in the dialog. Once created, you will get a “dialog handle”, a unique guid identifier for the initiator. This identifier will be used whenever the initiator wants to send any message or wants to receive any message from the target.

In order to start conversation between the services, we need to start a dialog.

So we are all set to start talking with our friend sitting on the target service. To send a message, we need few things though. We need a message type to validate what we are sending, we need our side of the conversation handle and of course, we need a message. As I feel that we have all of those ingredients of our recipe with us, we better start talking!

Now that we have sent a message, let’s see if this actually works. To check whether the target received our message we need to check the target queue.

SELECT *
FROM My_First_Queue_Target

So now you should be looking at a single row of data. The various columns summarize the properties of the conversation the two queues are having.

Receiving a message

Now that we have sent a message to the target queue and the message has safely reached its destination, we need to receive it too( think of it as a post delivered to your mailbox by the postal services, you need to pick it up from the mailbox finally). Note that once you pick the message up from the target queue, it will be removed from there (known as one-point delivery). We will be extracting the type of the message, conversation handle and the message body in the following code:

When we are done with sending or receiving messages, we should always end the conversation. Else the dialog will remain active and keep listening for new messages. Idle conversations use space on the database and can create problems if it is a busy database.