Table of Contents

Overview

Buffer plugins are used by output plugins. For example, out_s3 uses buf_file by default to store incoming stream temporally before transmitting to S3.

Buffer plugins are, as you can tell by the name, pluggable. So you can choose a suitable backend based on your system requirements.

How Buffer Works

A buffer is essentially a set of “chunks”. A chunk is a collection of events concatenated into a single blob. Each chunk is managed one by one in the form of files (buf_file) or continuous memory blocks (buf_memory).

The Lifecycle of Chunks

You can think of a chunk as a cargo box. A buffer plugin uses a chunk as a lightweight container, and fills it with events incoming from input sources. If a chunk becomes full, then it gets “shipped” to the destination.

Internally, a buffer plugin has two separated places to store its chunks: “stage” where chunks get filled with events, and “queue” where chunks wait before the transportation. Every newly-created chunk starts from stage, then proceeds to queue in time (and subsequently gets transferred to the destination).

Control Retry Behaviour

A chunk can fail to be written out to the destination for a number of reasons. The network can go down, or the traffic volumes can exceed the capacity of the destination node. To handle such common failures gracefully, buffer plugins are equipped with a built-in retry mechanism.

How exponential backoff works

By default, Fluentd increases the wait interval exponentially for each retry attempt. For example, assuming that the initial wait interval is set to 1 second and the exponential factor is 2, each attempt occurs at the following time points:

Note that, in practice, Fluentd tweaks this algorithm in a few aspects:

Wait intervals are randomized by default. That is, Fluentd diversifies the wait interval by multiplying by a randomly-chosen number between 0.875 and 1.125. You can turn off this behaviour by setting retry_randomize to false.

Wait intervals can be capped to a certain limit. For example, if you set retry_max_interval to 5 seconds in the example above, the 4th retry will wait for 5 seconds, instead of 8 seconds.

If you want to disable the exponential backoff, set the retry_type option to “periodic”.

Handling successive failures

By default, Fluentd will break a retry loop (except a successful write) on the following conditions:

The number of retries exceeds retry_max_times (default: none)

The seconds elapsed since the first retry exceeds retry_timeout (default: 72h)

In these events, all chunks in the output queue are discarded. If you want to avoid this, you can enable retry_forever to make Fluentd retry indefinitely.

Configuration Example

Below is a full configuration example which covers all the parameters controlling retry bahaviours.