The name of a file containing the definitions for how columns
should be processed by filters

serverPort

number

3112

The network port to use when communicating with the network client

timeout

number

10

Timeout in seconds before treating the network client as failed
when waiting to send or receive content.

The network filter operates by sending field data, as defined in the
corresponding filter configuration file, out to a network server that
processes the information and sends it back to be re-introduced in place
of the original field data. This can be used to translate and reformat
information during the replication scheme.

The filter operation works as follows:

All filtered data will be sent to a single network server, at the
configured port.

A single network server can be used to provide multiple
transformations.

The protocol used by the network filter must be followed to
effectively process the information. A failure in the network server
or communication will cause the replicator to raise an error and
replication to go OFFLINE.

The network server must be running before the replicator is started.
If the network server cannot be found, replication will go
OFFLINE.

10.4.25.1. Network Client Configuration

The format of the configuration file defines the translation operation
to be requested from the network client, in addition to the schema,
table and column name. The format for the file is JSON, with the
top-level hash defining the operation, and an array of field selections
for each field that should be processed accordingly. For example:

10.4.25.2. Network Filter Protocol

The network filter protocol has been designed to be both lightweight and
binary data compatible, as it is designed to work with data that may be
heavily encoded, binary, or compressed in nature.

The protocol operates through a combined JSON and optional binary
payload structure that communicates the information. The JSON defines
the communication type and metadata, while the binary payload contains
the raw or translated information.

The filter communicates with the network server using the following
packet types:

prepare

The prepare message is called
when the filter goes online, and is designed to initialize the
connection to the network server and confirm the supported filter
types and operation. The format of the connection message is:

The size of the payload; a value of -1 indicates that there is
no payload.

The format of the response should be a JSON object and payload with
the list of supported filter types in the payload section. The
payload immediately follows the JSON, with the size of the list
defined within the payload
field of the returned JSON object:

The message type; when acknowledging the original prepare
request it should be
acknowledge.

return

The return value. A value of 0 (zero) indicates no faults. Any
true value indicates there was an issue.

payload

The length of the appended payload information in bytes. This is
used by the filter to identify how much additional data to read
after the JSON object has been read.

The payload should be a comma-separated list of the supported
transformation types within the network server.

filter

The filter message type is
sent by Tungsten Replicator for each value from the replication stream
that needs to be filtered and translated in some way. The format of
the request is a JSON object with a trailing block of data, the
payload, that contains the information to be filtered. For example:

The row of the source information from the THL that is being
filtered.

schema

The schema of the source information from the THL that is being
filtered.

table

The table of the source information from the THL that is being
filtered.

column

The column of the source information from the THL that is being
filtered.

seqno

The sequence number of the event from the THL that is being
filtered.

fragments

The number of fragments in the THL that is being filtered.

fragment

The fragment number within the THL that is being filtered. The
fragments may be sent individually and sequentially to the
network server, so they may need to be retrieved, merged, and
reconstituted depending on the nature of the source data and the
filter being applied.

transformation

The transformation to be performed on the supplied payload data.
A single network server can support multiple transformations, so
this information is provided to perform the corrupt operation.
The actual transformation to be performed is taken from the JSON
configuration file for the filter.

payload

The length, in bytes, of the payload data that will immediately
follow the JSON filter request..

The payload that immediately follows the JSON block is the data from
the column that should be processed by the network filter.

The response package should contain a copy of the supplied
information from the requested filter, with the
payload size updated to the
size of the returned information, the message type changed to
filtered, and the payload
containing the translated data. For example:

10.4.25.3. Sample Network Client

The following sample network server script is written in Perl, and is
designed to translated packed hex strings (two-hex characters per byte)
from their hex representation into their character representation.