Performance should always be considered while developing a migration or integration. Generally speaking, the faster you can move the data the better. However, you must take into account the limitations of the platform in order to achieve the highest throughput. In the case of CRM, that means calling the Organization Service as efficiently as possible.

This post isn’t a comprehensive list of all the things you should consider while tuning the performance of your migration/integration. Instead, this post highlights a light-weight approach for running requests that you can use and build upon to enhance performance.

For this post we will be using the ExecuteMultipleRequest from CRM for batching and C# tasks for multi-threading. The ExecuteMultipleRequest does have the limitation of being throttled in CRM Online, so you should not combine ExecuteMultipleRequests with multi-threading if you are targeting CRM Online as you will receive concurrent fault errors.

In our sample we are going to create a simple console application that calls the Organization Service using different combinations of threading and batching. In order to run this sample you will need access to an instance of CRM 2013+. Make sure that you target a dev or sandbox environment. Also, it should be noted that this sample does not handle errors or write logs of any kind.

In order to take advantage of the multi-threading piece you will need to update your console application’s app.config and increase the allowed concurrent web connections. Add this section inside the <configuration> node –

/// <summary>
/// Sets up and runs requests in multi-threaded mode
/// </summary>
/// <param name=”orgService”>Organization Service instance</param>
/// <param name=”requests”>Requests to run</param>
/// <param name=”useExecuteMultiple”>Specify whether to use ExecuteMultipleRequest to batch requests</param>
public static void MultiThreadRequests(IOrganizationService orgService, List<OrganizationRequest> requests, bool useExecuteMultiple)
{
// Split all requests into a multi-dimensional list – this is a list of lists containing OrganizationRequest items
// We want to hand off a list of requests to each thread so they can take care of larger chunks
List<List<OrganizationRequest>> splitRequestLists = SplitRequestList(requests);

// Create a list of function calls – these functions will be used by Tasks in RunTasks()
// We use bool so we specify if the list was processed
List<Func<bool>> requestActions = new List<Func<bool>>();

if (requests.Count > 0)
{
// Go through all the requests from the list
// The requests will be executed once enough requests fill up the ExecuteMultipleRequest or we’ve run out of requests
foreach (OrganizationRequest request in requests)
{
executeMultipleRequest.Requests.Add(request);

requestCount++;

// Have we reached our batch size or did we already hit the total count?
// Only fire if there are actually requests
if ((executeMultipleRequest.Requests.Count == BATCH_SIZE || requestCount == requests.Count)
&& executeMultipleRequest.Requests.Count > 0)
{
// Run all requests
ExecuteMultipleResponse response = orgService.Execute(executeMultipleRequest) as ExecuteMultipleResponse;

// This response will be faulted if there is at least one error
if (response.IsFaulted)
{
// TODO: Handle errors
}

/// <summary>
/// Splits up a large list of OrganizationRequest items into lists that will be used for batching
/// </summary>
/// <param name=”requests”>Requests to run</param>
/// <returns>List of Lists of OrganizationRequest items</returns>
public static List<List<OrganizationRequest>> SplitRequestList(List<OrganizationRequest> requests)
{
List<List<OrganizationRequest>> splitRequestList = new List<List<OrganizationRequest>>();

// We always round up to the next whole number to ensure all requests are accounted for
int listCount = (int)Math.Ceiling((double)requests.Count / LIST_BATCH_SIZE);

// Fill empty lists with requests up to the threshold
foreach (OrganizationRequest request in requests)
{
// Increment the listIndex if the list is full
if (splitRequestList[currentListIndex].Count == LIST_BATCH_SIZE)
{
currentListIndex++;
}

splitRequestList[currentListIndex].Add(request);
}

return splitRequestList;
}
}
}

You will want to format the document after pasting in the code for readability. Also, you will notice that the Organization Service used here is the Xrm.Client version. Make sure to change the SERVER_URL, USER_NAME, and PASSWORD constants before running the application.

Running the application will create and delete 1000 accounts using a combination of multi-threading and batching. At the end of each operation it will output the total time it took for each operation.

The beginning of the operation is always the same – a list of requests will be created that are then fed into the other methods. Depending on the parameters for batching/multi-threading, the list will be cut up into smaller lists and then distributed amongst multiple threads. These requests will then be batched depending on if the ExecuteMultipleRequest is being used.

You will notice that the result times improve when multi-threading and batching are added to the equation. However, you will eventually hit a max throughput that will be dependent on various factors including – the amount of fields per record, network speed, CRM Online vs On-Premise, machine resources (RAM/CPU), etc.

In general, you will see a noticeable increase in performance with just a few changes to how you call the Organization Service. However, you will still want to make sure you understand the implications of how the addition of batching and multi-threading affects your data. For example, if you are importing accounts, where the Parent Account field is filled out, you will need to do it two passes with creates first followed by updates.

For CRM On-Premise the best option is probably to use a combination of batching and multi-threading. For CRM Online you will either want to use batching on a single thread, making sure that the operation runs at a time when it’s unlikely for another ExecuteMultipleRequest to run at the same time, or do multi-threading without batching.

Related

1 Comment

Sean
on 01/05/2016 at 8:12 pm

Hi, a couple of tips
1. Organization Service proxy is *not* thread-safe, however you can cache and re-use IServiceManagement (but you’ll have to create a new service proxy per thread, and using the same IServiceManagement will help speed that up – you can also keep that proxy around on the same thread. It just won’t work on two threads at the same time).