Dealing with rate limited services

In Running GmailApp in parallel we found that throwing additional parallel executors at a rate limited service doesn't help much - it will only let you do so much at the same time. However Running things in parallel using HTML service is a sequential scheduler as well as a parallel orchestrator. So we can split the job up and run sequential sets of parallel threads to eventually get the job done. A head and tail janitor process carry forward the work that has already been done, or needs to be done to a reduction process ready for the next set. In this way we get through all the work, as long as we keep each thread to below 6 minutes.

As usual we'll create profiles and some new .map executors for this.

The profiles

The first task is to get all the threads matching the search text in your email.

function logEmailsProfileSets() {

var profile = [];

// get the matching threads

profile.push( [

{

"name": "GET THREADS",

"functionName": "getTheThreads",

"skip": false,

"debug":false,

"options": {

"searchText": "The Excel Liberation forum has moved to a Google+ community"

}

}

]);

Next, a reduction to bring all the results together. As usual we'll use the common function reduceTheResults(), that we've used in all the examples for that. The reason for a reduction is that we might later split the getTheThreads() function into parallel execution tasks, and we'd need to combine the results.

// next reduce the messages to one

var profileReduction = [];

profileReduction.push({

"name": "reduction",

"functionName":"reduceTheResults",

"options":{

}

});

Now we need to split the threads into some number of sets, and within each set, a number of threads to be worked on in parallel. First the head janitor function. His role is to carry forward any work that has already been done in previous sets.

// get and process all the messages

var SETS = 4;

var CHUNKS = 2;

// a set is a collection of threads that can run together, including a head janitor and tail janitor function to mop up

for (var j=0; j < SETS ; j++) {

var profileSet = [];

var janitor = {

"name": "HEAD JANITOR:" + j,

"functionName":"janitor",

"skip":false,

"debug":false,

"options": {

"setIndex": j,

"sets":SETS,

"janitor": "head"

}

};

profileSet.push(janitor);

Now the threads within each set

// now the threads

for (var i =0; i <CHUNKS;i++ ) {

profileSet.push ( {

"name": "MESSAGES:" + i,

"functionName":"getMessageSets",

"skip":false,

"debug":false,

"options": {

"setIndex": j,

"sets":SETS,

"threads":CHUNKS,

"index": i

}

});

}

And then the tail janitor - his job will be to carry forward the work that needs to be done in subsequent sets

// now the tail janitor

var janitor = cUseful.clone(janitor);

janitor.name = "TAIL JANITOR:" + j;

janitor.options.janitor = "tail";

profileSet.push(janitor);

// add this set to the list

profile.push(profileSet);

We need a reduction between each set.

// add this set to the list, then do a reduction

profile.push(profileSet , profileReduction);

}

Now we need a special reduction that will blow out cases where we have multiple recipients into separate records

profile.push([{

"name": "reduction",

"functionName":"reduceTheEmailToResults",

"debug":true,

"options":{

}

}]);

Finally we'll need to log the results

// finally log the results

profile.push( [{

"name": "LOG",

"functionName": "logTheResults",

"skip": false,

"debug": false,

"options": {

"driver": "cDriverSheet",

"clear": true,

"parameters": {

"siloid": "emails",

"dbid": "1yTQFdN_O2nFb9obm7AHCTmPKpf5cwAd78uNQJiCcjPk",

"peanut": "bruce"

}

}

}]);

return profile;

}

The only change we need to make now is to call this function to set up the run profile

function showSidebar() {

// kicking off the sidebar executes the orchestration

libSidebar('asyncService',ADDONNAME, logEmailsProfileSets() );

}

The executors

Now we need to write the couple of new functions mentioned in these profiles that will be called by the htmlservice

and this one will be run in parallel, processing chunks of the total messages. It differs from the one in Running GmailApp in parallel, since we now have the concept of 'sets'. We're also postponing the duplication of records with multiple recipients in the to: list until the final reduction, rather than doing it here.

/**

* do a chunk of message processing - this one is expecting to do it in sets

* @param {object} options describes what to do

* @param {object} reduceResults this would contain results from a previous stage if present

Authorization

GmailApp needs authorization, so its worth running a small test first to force an authorization dialog, as well as to test your executor functions. That can easily be done by emulating a couple of execution steps in a simple function like this.