HTTP Client

The HTTP Client processor sends requests to an
HTTP resource URL and writes the results to a field in the record. You can use the HTTP
Client processor to perform a range of standard requests or you can use an expression to
determine the request for each record.

When you configure the HTTP Client, you define the resource URL, header attributes, and
method to use. For some methods, you can specify the request body and default content
type.

You can configure the processor to include response header fields in the record as a set
of record header attributes or as a map in a record field.

You can also configure the timeout, request transfer encoding, maximum number of parallel
requests, and authentication type. You can optionally use an HTTP proxy and configure
SSL/TLS properties.

You can also configure the processor to use the OAuth 2 protocol to connect to an HTTP
service.

HTTP Method

You can use the
following methods with the HTTP Client processor:

GET

PUT

POST

DELETE

HEAD

Expression - An expression that evaluates to one of the other methods.

Expression Method

The Expression method allows you to write an
expression that evaluates to a standard HTTP method. Use the Expression method to
generate a workflow. For example, you can use an expression that performs a lookup (GET)
or passes data to the server (PUT) based on the data in a field.

Parallel Requests

The HTTP Client processor sends multiple requests at a time. To preserve record order,
the processor waits until all requests for the entire batch are completed before processing
the next batch.

You can specify the maximum number of parallel requests. Default is 1. Increasing the number of
parallel requests can improve performance but increases the load on the server. Network
latency can also significantly impact the performance of this processor.

Response Header Fields

The HTTP Client processor can include response header field information in records. You can
write the data to a record as follows:

Record header attributes

The processor can write data in response header fields to corresponding
record header attributes.

When writing to record header attributes, you define a prefix for the
attributes to avoid name conflicts. The processor then appends the response
header field name to the prefix.

For example, if you define "info-" as the prefix, then data in the
content-encoding response header field is written to the
"info-content-encoding" record header attribute.

The processor can also write the response header fields to a field in the
record. The processor writes the response header fields to the record field
as a map of key-value pairs where the key is the response header field name.

When writing to a field, you specify the field to use. If the field includes
data, the processor overwrites the data. If the field does not exist, the
processor creates the field.

OAuth 2 Authorization

You can configure the HTTP Client processor to use the OAuth 2 protocol to connect to an
HTTP service that uses basic, digest, or universal authentication, OAuth 2 client
credentials, OAuth 2 username and password, or OAuth 2 JSON Web Tokens (JWT).

The OAuth 2 protocol
authorizes third-party access to HTTP service resources without sharing credentials. The
HTTP Client processor uses credentials to request an access token from the service. The
service returns the token to the processor, and then the processor includes the token in
a header in each request to the resource URL.

The credentials that you enter to request an access token
depend on the credentials grant type required by the HTTP service. You can define
the following OAuth 2 credentials grant types for HTTP Client:

Client credentials grant

HTTP Client sends its own credentials - the client ID and
client secret or the basic, digest, or universal
authentication credentials - to the HTTP service. For
example, use the client credentials grant to process data
from the Twitter API or from the Microsoft Azure Active
Directory (Azure AD) API.

HTTP Client sends the credentials for the resource owner -
the resource owner username and password - to the HTTP
service. Or, you can use this grant type to migrate
existing clients using basic, digest, or universal
authentication to OAuth 2 by converting the stored
credentials to an access token.

On the HTTP tab, set Authentication
Type to None, and then select
Use OAuth 2.

On the OAuth 2 tab, select JSON Web
Tokens for the grant type.

In the Token URL property, enter the following URL used
to request the access token:

https://www.googleapis.com/oauth2/v4/token

Select the following algorithm to sign the JWT: RSASSA-PKCS-v1_5
using SHA-256.

Enter the Base64 encoded key used to sign the JWT.

To access the key, download the JSON key file when you generate the Google
credentials. Locate the "private_key" field in the file, which contains a
string version of the key. Copy the string into the JWT Signing
Key property, and then replace all "\n" literals with new
lines.

You
can include the expression language in the JWT claims. For example, in the
sample claim above, both the "exp" (expiration time) claim and the "iat"
(issued at) claim include Data Collector time
functions to set the expiration time and the issue time.

Tip: Google access tokens expire after 60 minutes. As a result, set
the expiration time claim to be slightly less than 60 minutes so that HTTP
Client can request a new token within the time limit.

The following image shows the OAuth 2
tab configured for Google service accounts:

Data Formats

The HTTP Client processor writes the server response to the specified field in the
record.

The processor treats the response as the
specified data format and processes data differently based on the data format:

JSON

Writes a parsed JSON response to the specified field.

When an object exceeds the specified maximum object length, the processor
processes the object based on the error handling configured for the stage.

Text

Writes the response as a string to the specified field.

When a line exceeds the specified maximum line length, the processor
truncates the line. The processor adds a boolean field named Truncated to
indicate if the line was truncated.

XML

Writes the XML response to the specified field.

When specified, the processor uses the delimiter element to nest additional
elements in the field.

When a record exceeds the specified maximum record length, the processor
skips the record and continues processing with the next record. It sends the
skipped record to the pipeline for error handling.

Configuring HTTP Client Processor

Configure an HTTP Client processor to perform requests against a resource
URL.

In the Properties panel, on the General tab, configure the
following properties:

General Property

Description

Name

Stage name.

Description

Optional description.

Required Fields

Fields that must include data for the record to be passed
into the stage.

Tip: You might
include fields that the stage uses.

Records
that do not include all required fields are processed
based on the error handling configured for the
pipeline.

Preconditions

Conditions that must evaluate to TRUE to allow a record
to enter the stage for processing. Click
Add to create additional
preconditions.

Records that do not meet all preconditions
are processed based on the error handling configured for
the stage.

On Record Error

Error record handling for the stage:

Discard - Discards the record.

Send to Error - Sends the record to the pipeline for
error handling.

Stop Pipeline - Stops the pipeline. Not valid for
cluster pipelines.

On the HTTP tab, configure the following properties:

HTTP Property

Description

Output Field

Field to use for the response. You can use a new or
existing field.

Header Output Location

Location to write response header field information.

Header Attribute Prefix

Prefix to use when writing response header field
information to record header attributes.

Header Output Field

Field to use when writing response header field
information to a field in the record.