In this article we will discuss another obvious performance issues with SharePoint Solutions involving large volume of data transactions surfacing SharePoint Custom Pages.

This could become more prominent if we have strict governance in place and we are not allowed to make use of advanced server side options (Custom Web Service End Point, MTA Enabled Modules etc.).

In one of the recent assignment I came across a similar scenario where I need to crawl data from an external Web Service end Point and surface data on SharePoint Pages. Since the anticipated data volume was huge and traditional caching approaches like Cookies wont’ work due to size limitations.

In pursuit of the solution I have gone through the “HTML5 Web Storage APIs” that allows you to setup an In-Browser Transactional Database System called “IndexedDB”.

Here is a quick introduction of IndexedDB for details I must recommend you to visit IndexedDB

“IndexedDB is a transactional database system, like an SQL-based RDBMS. However, unlike SQL-based RDBMSes, which use fixed-column tables, IndexedDB is a JavaScript-based object-oriented database. IndexedDB lets you store and retrieve objects that are indexed with a key; any objects supported by the structured clone algorithm can be stored. Operations performed using IndexedDB are done asynchronously, so as not to block applications.”

I also want to thanks to “Raymond Camden” for his detailed research on Storage Limits for IndexedDB and believe you must refer this link to understand the limits carefully before getting into concrete implementations.

Now let’s try to understand the implementation details by using following diagram:

Solution Architecture Diagram & Explanation

In this solution the SharePoint Page will try to look for the required data in Local Indexed DB created to support this page. If data is not found in local database, page will issue the request for data from SharePoint List.

Since we are dealing with “100,000” Items present in SharePoint List, I made use of “REST API + OData Continuation” data access strategy to overcome SharePoint List Threshold Limits. This mechanism will access only 100 List Items at a time and it is safe to extend this limit up to 2000 items per fetch.

Each fetch will a JSON Object that will be persisted into Indexed DB as individual record. I opt this strategy to reduce the page load time. If the number of items are not much you can add each item as separate record.

Every subsequent data call will be automatically diverted to the local database as primary source.

Additionally we can add “Auto Refresh Modules” to keep the local database fresh with SharePoint List Changes and sync the changes with Indexed DB “Asynchronously”.

Ideally speaking for a complete solution “Auto Refresh Modules” are must to have.

So this all about execution summary for this solution.

Now let’s have look at implementation details as follows-

I have created a SharePoint List with two columns and “100,000” Items added to it as shown below.

Demo

This list will be acting as data source for the page. In actual scenarios this source could be a Web Service End Point which can provide voluminous data on demand.

Before getting into code let’s see how this Page will behave on execution. Demonstrating the page in action will be helpful later when we get a deep dive in code.

If we run the page we will see this page took about “3 minutes” to get execution completed.

The first execution cycle will include the following actions:

Initialize IndexedDB Database

Query SharePoint List

Add REST API Response to IndexedDB

Load page with data from IndexedDB

Since we are adding data to the store asynchronously, overall application will remain functional even it is taking 3 minutes to complete.

Following screen shot showing data adding to IndexedDB asynchronously

We can also review the Indexed DB initialized as the part of this request using “Developer Tools or F12 Key” with in the browser as shown below-

We can explore each item in the each of the JSON Object as shown below-

Now refresh the page to see the execution again and we can see roughly “1 second” to complete the page request.

The subsequent execution cycle will include the following actions:

Query IndexedDB for data

Load page with data from IndexedDB

So we can see how we can trim the execution path by using a well-defined strategy.

Code Analysis

Let’s do the code analysis to understand the concrete implementation.

In Step 1 we are enclosing some of the literals as variables and will refer theses variables later in the code

In Step 2 we are checking if respective Indexed Database is initialized already or not and if not Initialize the Database. In this demos let’s call this database as “Products”

In Step 3“onsuccess” event handler will get executed and database object will get stored in a global variable “SharePointOptimization.sharePointStore”. This variable will be acting as start point for all the operations on the database in future.

In Step 6 we are making use of OData Continuation Techniques to overcome SharePoint List Threshold restrictions.

In this step we also call “AddDataToStore” function that will add SharePoint List Items coming as JSON Object to the Local Indexed Database. It is important to recall that in this demo I am storing 1 JSON Object as 1 record in database and each object contains information for 100 List Items.

In Step 7 we are adding JSON Objects to IndexedDB. In order to do that we need to perform following operations-

Initialize Transaction with Read Write Operation Permissions

Get Handle on “Products” Database inside IndexedDB Data Stores

Call asynchronous “add” method to add JSON Object to “Products” Store

In Step 8 we are calling “QuerySharePoint” function to query data from SharePoint List in case data is not available in Local Database.

Steps 9, 10, 11 explains about “ReadSPStore” function where we will read the data from Local Data Store (IndexedDB)

In Step 9 following operations are performed-

Initialize Transaction with Read Operation Permissions

Get Handle on “Products” Database inside IndexedDB Data Stores

Call asynchronous “count” method to get total number of JSON Object available in “Products” Store

If success read the record from IndexedDB and add to the local array variable

Call “continue” function as long as there are items left in local store

Once all data is read and save to the local array pass this array to “RenderUI” function to render this data on the interface as required

In Step 12 we can plug any UI engine to produce more intuitive UI as applicable, for the sake of this demo I am writing out the Count of Store records * 100 (since each record contains 100 Items) to show the total number of items stored in the local store.

Steps 13, 14, 15 show you a helper function to check if local store contains required data or not. It helps to decide if we need to read data from Local Store or SharePoint List

“GetProductCount” function is quite similar to the “ReadSPStore” function except it perform a lesser number of operations

In Step 16 we will initialize Local SharePoint Store by calling “InitializeSharePointStore” function

In Step 17 we can see some of the UI elements to build a basic UI for this demo

Point of caution

Before implementing this mechanism make sure you have identified all the compatibility issues around this corner.

I would recommend you to refer the following site every now and then to make sure you are using features supported by the targeted browsers.

In this demo we will discuss the caching mechanism for SharePoint Objects which are more complex in nature than the primitive data types.

I have selected “SharePoint List Items Collection” object for this demo since while working with SharePoint it is often that we need to query the SharePoint Lists using CSOM or REST API, which either ways returns a SharePoint List Items Collection Object that we can hold in the local cache and can boost Read/Write Operations in a more efficient manner.

So let’s starts with a new SharePoint App Project

Choose “App for SharePoint” Project Template

Specify SharePoint Site URL (On Premise or Online)

Select hosting model as “SharePoint-Hosted”

Here I am using SharePoint Online Site to go with this demonstration

Enter SharePoint Online Account Credentials

Let Visual Studio configure the project for you

For the sake of this demo I have created a list called Products which has got few items added as shown below:

I have added HTML code to default.aspx page that will display following elements to the UI

Result Panel: This is a div that will display the output of the operations performed

Get List Button: This will retrieve the data from SharePoint and add it to the Cache

Step2: Verifying if the current browser supports the HTML5 Local Storage API

Step 3: Verifying if the current browser supports the HTML5 Local Storage API by utilizing Feature Detection capabilities of Modernizr.js Framework. For more details on Modernizr.js you can visit https://modernizr.com/

Step 4: This function first checks if Local Storage API is supported in the current browser and if it is supported then clear the specific key by specify the key name or all the keys from local cache by calling Clear() methods that you can see as commented in this function.

Step 5: This is the helper method that checks if current browser supports the HTML5 Local Storage API traditionally by querying JavaScript “window” object

Step 6: Same as Step 2 & 3

Step 7: Instantiate the local storage object

Step 8: Try to reading SharePoint Items Collection from Local Cache by specify Cache Key after converting it into string array

Step 9: If Items Collection is not present in Local Cache then we are calling getProductListItemsCollection(). This function will fetch the items from Products List and save it to the local cache for further use. We will see the implementation details of this function in Steps 12,13,14 & 15 down the line

Step 10: If the items collection is available in the local cache then it will be fetched from local cache based on the cache key

Step 11: Calling explodeJSONObject() which is a helper function that reads the item values out of the JSON Object that is passed as input parameter. We will see the implementation details of this function in Step 16.

Step 12: Getting Products List and query items using CAML Query

Step 13: Converting Lists Items Collection into a valid JSON Object by calling getJSONObject() method. We will see the implementation details of getJSONObject() method in Step 17 down below

Step 14: Once we received the valid JSON Object we can save it to Local Storage Cache by converting it back to String Array.

Step 15: Calling explodeJSONObject() which is a helper function that reads the item values out of the JSON Object that is passed as input parameter. We will see the implementation details in Step 16.

Step 16: explodeJSONObject() method takes JSON form of SharePoint List Items Collection and display it to the Result Panel by treating each row in the List Item Collection as Object with each column as its properties. Since we are fetching Item ID and Product Title Columns in the SharePoint List Items Collection so we can use it as [“RowObject”.ID] and [“RowObject”.Title] to display it in the result panel.

SharePoint Pages are intended to act as data presentation layer to the end Users for the data stored in SharePoint List. In the rich interactive Client Centric Applications it is expected to have quick response to different User Actions like Read Data from or Save Data to SharePoint List.

Any delays due to read/write operations to SharePoint can negatively impact the User experience and could lead to denial of acceptance on developed features.

Using HTML5 Local Storage API, Web Applications can store data locally in user’s browser. Local Storage is more secure and large data can be stored without impacting overall Web Application Performance.

This new API allows large amount of data (approx. 5 MB) to be cached locally which is far larger than cookies which were commonly used for caching data.

Local Storage is maintained per domain & protocol (Origin of Request) and all Web Pages of same origin can use the same Cache.

In this article we will see HTML5 Local Storage API in action and utilize it to cache SharePoint Properties of primitive data type like String, Integer, Boolean and in one of the upcoming articles in the same series I will explain the caching of some of the complex SharePoint Objects.

Let’s kick start this demo with a new SharePoint App Project in Visual Studio 2013

Choose “App for SharePoint” Project Template

Specify SharePoint Site URL (On Premise or Online). Here I am using SharePoint Online Site to go with this demonstration

Select hosting model as “SharePoint-Hosted”

Enter SharePoint Online Account Credentials

Wait till Visual Studio configure the project for you

I have added HTML code to default.aspx page that will display following elements to the UI

Result Panel: This is a div that will display the output of the operations performed

Get Web Title: This HTML Button will retrieve the Title of the Host Web and add it to the Cache on click event

Clear Cache: This HTML Button will clear the local cache whenever needed on click event

Step2: This function first checks if Local Storage API is supported in the current browser and if it is supported then clear the specific key by specify the key name or all the keys from local cache by calling Clear() methods that you can see as commented in this function.

Step 3: This is the helper method that checks if current browser supports the HTML5 Local Storage API traditionally by querying JavaScript “window” object

Step 4: This method is a helper function that helps to retrieve Query String Parameters out of the requested URL

Step 5: Verifying if the current browser supports the HTML5 Local Storage API

Step 6: Verifying if the current browser supports the HTML5 Local Storage API by utilizing Feature Detection capabilities of Modernizr.js Framework. For more details on Modernizr.js you can visit https://modernizr.com/

Step 7: Instantiate the local storage object

Step 8: Reading Web Title value out of Local Storage Cache by specifying the key

Step 9: If the Web Title exists in the Local Cache then the value is retrieve out of it without any need of issuing call to SharePoint else we call another helper method “getWebTitle()” that will issue the request to SharePoint and get the Web Title, display it to result Panel and add the value to the Local Cache and make it available to server any subsequent request.

Step 10: In this method we are making CSOM call to SharePoint in usual way, once the call gets succeed the Web Title will get displayed in the Result Panel and also adding the value to local storage by calling setItem() method. This value will be available to serve from Cache in response to any future request for Web Title.

With this we are done with the code.

Before we start execution we need to allow Read Permission to the App on Site Collection

Launch AppManifest.xml

Go to “Permissions” Tab

Select “Site Collection” from Scope Dropdown

Select “Read” from Permission Dropdown

Run the App

Enter the SharePoint Online Credential when prompted

Trust the App when prompted

When App lunches we can see the UI based on the HTML we have added earlier to the default.aspx page

We can see Result Panel where we will see the results of the operations and “Get Web Title”, “Clear Cache” buttons as well

Now when we click on “Get Web Title” button we can see two actions performed & logged in the Result Panel as follows-

Since the Web Title is not available in the Local Cache, it is retrieved from SharePoint and saved back to the local cache

Displaying Web Title in the Result Panel

If we investigate the Client – Server Request cycle using Fiddler we can see that there is actually a request issued to SharePoint and the response received from the server in form of requested data.

When we click “Get Web Title” button again we will see following two actions performed & logged in the Result Panel

Since the Web Title is available in the Local Cache this time , so no call has been issued to SharePoint for data and Web Title has been retrieved from the Local Cache

Displaying Web Title in the Result Panel retrieved out of Local Cache.

If we look into Fiddler again we can see that no request has been issued to SharePoint since we got the requested data available in Local Cache.

That’s it for this article.

In the upcoming article on Client Caching I will explain the caching scenarios related to bit more complex SharePoint Object like “SPListItemCollection” which is quite common return type when we query data from SharePoint Lists and deserve special attention as long as the caching is concerned.