Tuesday, 22 April 2014

There are several tutorials on the Web explaining how to integrate SSIS with Dynamics CRM using the script component. All of them however show you only the basic setup, where records from a data source are processed 1 by 1 when executing CRM commands (e.g. creating CRM records). In this post I would like to show you have to leverage the ExecuteMultipleRequest class from CRM SDK to create bulk operations for records from the SSIS data source.

Tutorial scenario

At first we will create a simple database with 1 table that stores user names

Then we will create an SSIS project

Next, we will add our db table as data source, so SSIS can read information about users

Then, we will add a script component that creates contacts in CRM for each user from the table

When the project is created add a Data Flow task to your main package:

Data Source

Double click your Dat Flow task to open it

Double click "Source Assitance" from the toolbox

On the first screen of the wizard select "SQL Server" as source type and select "New..."

On second screen provide you SQL server name and authentication details and select your database

A new block will be added to you Data Flow, representing your DB table. It has an error icon on, cause we haven't selected the table yet. Also, you will see a new connection manager representing you DB connection:

Double click the new block, from the dropdown select the Contacts table we created and hit OK. The error icon should disappear

Script component

Drag and drop the Script Component from the toolbox to you Data Flow area

Create a connection (arrow) from your data source to your script:

Double click your script componet to open it

Go to "Input Columns" tab and select all columns

Go to "Inputs and Outputs" tab and rename "Input 0" to "ContactInput"

1-by-1 import

Now that we have basic components setup let's write some code! In this step we will create a basic code for importing Contacts into CRM. I'm assuming you have basic knowledge of CRM SDK, therefore the CRM specific code will not be explained in details.

Open the script component created in the previous steps and click "Edit Script...". A new instance of Visual Studio will open with a new, auto-generated script project. By default the main.cs file will be opened - this is the only file you need to modify. However, before modyfing the code you need to add references to following libraries:

Microsoft.Sdk.Crm.Proxy

Microsoft.Xrm.Client

Microsoft.Xrm.Sdk

Microsoft.Runtime.Serialization

Now we are ready to write the code. Let's start by creating a connection to you CRM organization. This will be created in the existing PreExecute() method like this:

Obviously the code above requires some null-checks, error handling etc but in general that's all you need to do in order to import your contacts into CRM. If you close the VS instance with the script project it will be automatically saved and built.

You can now hit F5 in the original VS window to perform the actual migration.

Bulk import

In the basic setup described above there is 1 CRM call for each record passed to the script component. Calling web services over the network may be a very time consuming operation. CRM team is aware of that and that is why they introduced the ExecuteMultipleRequest class, which basically allows you to create a set of CRM requests on the client side and send them all at once in a single web service call. In response you will receive an instance of the RetrieveMultipleResponse class, allowing you to process response for each single request.

Let's modify the script code to leverage the power of the ExecuteMultipleRequest class. To do that overwrite the ContactInput_ProcessInput method. The default method implementation can be found in the ComponentWrapper.cs file and it as simple as this:

As you can see by default it calls the ContactInput_ProcessInputRow method that we implemented in the previous step for each record from the source. We need to modify it, so it creates a batch of CRM requests and then send it to CRM at once:

Execution time comparison

As you can see the code for sending requests in batches is a bit longer (but still quite simple I believe) so you may be tempted to go with the simpler version. If you don't care about performance too much (little data, no time limitations) then it might be the way to go for you. However, it's always better to know your options and take a conscious decision. SSIS packages usually process large amount of data, which often takes a lot of time. If you add additional step performing CRM operations via CRM SDK (i.e. via CRM web services) you may be sure this will affect significantly the execution time.

I've measured the execution time for both methods. Importing 1000 contacts into CRM took:

1-by-1 - 2:22s

Bulk import - 0:44s

In my simple scenario bulk import was 3x faster than 1-by-1. The more data you send to CRM the bigger the difference may be.

7 comments:

Hi Filips, I have requirement where in I pull say for example 1 million Revenue record from upstream system and push the raw data in to staging and from staging push 75000 revenue records in to CRM 2013 online. I have various filter criteria in CRM like Area,fiscal period. Once I apply that filters in CRM and click retrieve. it should just fetch the revenue nos and not the actual records. Method you have suggested will that support this option please? ofcourse performance is the main part since we retrieving huge no of data. Please suggest for CRM online