Making data request with synchronous, asynchronous, batch, and auto mode

Making data request with synchronous, asynchronous, batch, and auto mode

You can make the following request types with Qlik Web Connectors:

Synchronous

Asynchronous

Batch

Auto

For/next loops are often used to run a large number of API requests with Qlik Web Connectors. These modes let you extract the maximum performance from Qlik Web Connectors and minimize load time of the QlikView or Qlik Sense application.

For example, if you run a large number of text messages through one of the available sentiment analysis connectors, like Repustate or Sentiment140 it can result in a long load time.

In the script example above, each message is processed sequentially. This means that the sentiment score for the next message cannot be calculated until the current request has completed. If each request takes 250 ms, then it will take 40 minutes to score the sentiment score of 10,000 Tweets.

Many APIs allow multiple requests to be made in parallel from the same client so we can use different request types to improve the performance of the Qlik Web Connectors.

These requests are handled by separate processes on the server or cloud, allowing more requests to be processed per second. The Asynchronous mode allows us to take advantage of this feature.

Some APIs also allow multiple requests to be combined into a single API call, for example sending 1000 text messages in a single request to be processed for sentiment. This is referred to as sending requests in a batch and it is the Batch mode that allows us to take advantage of this feature.

Synchronous requests

Synchronous requests are the default mechanism and all tables of all connectors support this. A request to Qlik Web Connectors results in one or more sequential requests to the underlying API and this is the script you receive when copying the standard generated script from Qlik Web Connectors.

For simple for/next loops, It is possible to simplify the script by using a temporary table.

Create a temporary table named Params that contains the same parameters that are passed in each iteration of the for/next loop above.

There are several important changes to the new script:

An additional column named rowKey, which is set to the Tweet ID, so we can link it back into the data model when the results come back.

The Params table is saved as a CSV file. It is important that the user account that is running the Qlik Web Connectors has read access to this file.

Note: The parameters in the Params table should not be URL encoded.

Warning: There are certain parameters you cannot put in the Params table. In particular, the table or format parameters cannot be used. These should be in the final Qlik Web Connectors request. For a complete list of parameters, see Special parameters

The script makes a request to Qlik Web Connectors but instead of passing the usual parameters, it passes the path of the previously created CSV file using the processParamsSync parameter. We also read back the rowKey and alias ID back to the Tweet ID.

You can apply this method to almost any Qlik Web Connectors request, and it is the basis of the Asynchronous, Batch and Auto modes.

Note: You can use rowKey2 and rowKey3 parameters with processParamsSync.

Asynchronous requests

To make asynchronous requests, you simply change the parameter name processParamsSync into processParamsAsync:

Note: You can use rowKey2 and rowKey3 parameters with processParamsAuto.

Note: Auto request mode is not recommended because only a few connectors explicitly support asynchronous mode, so Qlik Web Connectors would fall back on the slowest synchronous mode. We recommended that you look at each table you are using and decide whether to use synchronous, asynchronous or batch mode.

Using URLs instead of file paths

When using a processParamsxxx parameter, as well as passing a file location to it, you can also pass a URL beginning with http or https.

Special parameters

Certain parameters can be used by any Qlik Web Connectors connector. These parameters should not be placed in the ParamsCSV file but specified in the request URL for the table.

table

The name of the table requested. Should only be included in the final request.

format

Tells Qlik Web Connectors what format to return the data in. As all the data for a given request must be returned in the same format this cannot appear in the Params file. In almost all cases this should be set to qvx.

appID

An identifier for your application passed to Qlik Web Connectors allowing it to be logged alongside any errors which may occur.

noCache

Causes Qlik Web Connectors to ignore the cache.

requestStamp

Allows you to specify an id for a request which you can then use to access follow on data from another table related to the request.

Feedback

Did this information help you?

Thanks for letting us know. Is there anything you'd like to tell us about this topic?