Kubernetes logs analysis with Elassandra, Fluent-Bit and Kibana

No Single Point Of Failure

Elasticsearch is by design a sharded master-slave architecture.

The Elasticsearch master node manages the mapping changes and only the primary shards takes write operations. The replica shards are read-only and can be promoted to primary shards by the master node in case of failure or maintenance. By relying on Apache Cassandra, Elassandra is master-less and has no single point of write. All nodes can process the search requests, request a mapping update, and depending on the CCassandra replication factor, take write operations.

As a result, Elassandra is pretty simple to manage with Kubernetes and maintenance operations (rolling restart for upgardes) can be done without any downtimes. Scaling up or down also becomes a lot more simple, no need to re-index the existing Elasticsearch indices, only the moved data is re-indexed in Elasticsearch. Elassandra also natively supports cross datacenter replication.

Elassandra, Fluentbit and Kibana on Kuberenetes

To demonstrate the ability of Elassandra to work propely on Kubernetes, we have used Fluent-Bit, a fast and lightweight log processor and forwarder that support the elasticsearch backend, but the same configuration could also use Fluentd or Filebeat.

To deploy our EFK stack on Kuberenetes, we have used the HELM charts as follows. First one needs to deploy an Elassandra with a 3 nodes cluster with SSD storage (here on Azure Kubenetes Service) from the Strapdata HELM repository.

Insert only mode

In Elassandra, elasticsearch _source document are stored in a Cassandra table as rows. When inserting into Elassandra via the the elasticsearch index API, empty fields are populated with null by default in the Cassandra storage layer in order to overwrite any existing documents. It generates some Cassandra tombstones, but when inserting some immutable log records, it is useless, and in this case, the index.index_insert_only settings optimize the Cassandra storage.

Drop on delete index

In Elassandra, Cassandra is basically the data storage layer and by default, removing an index does not remove the underlying Cassandra table and keyspace. You can alter it with the index.drop_on_delete_index setting, removing the last index on a table will remove the underlying Cassandra table. Thus, purging Elasticsearch indices will also drop underlying keyspaces automatically.

Replication

In Elassandra, data replication is no longer done by Elasticsearch but by Cassandra itself. Then, the underlying keyspace replication map controls the location (in which cassandra datacenters) and the number of copies. The replication setting in the Elasticsearch template defines the Cassandra replication map, here on DC1 with 2 copies.

Table options

The table_options settings define the Cassandra table options used at creation time. Here, as the log records are immutable, we have choosen the Time Window Compactio Strategy designed for the time series data. You could also set here the Cassandra default TTL or the compression settings (LZ4 by default).

Search strategy

In Elassandra, the coordinator node distributes the sub-queries to the available nodes according the search strategy defined by a search_strategy_class. By default, the PrimaryFirstSearchStrategy sends a sub-query to all nodes in the Cassandra datacenter. Here, we use the RandomSearchStrategy that computes the smallest set of nodes required to get a consistent result in the datacenter. For example, with a replication factor of 2 in a 6 nodes cluster, this strategy will only request 3 nodes, reducing the global load of the clusters.

Elassandra mapping

Because the Elasticsearch fields are multi-valued, Elassandra stores each field of type X in a Cassandra list. If you look at the generated SSTables with the sstabledump utility, you will be able to find Cassandra rows generated by fluent-bit:

As you can see, due to the Cassandra collections (list, set and map), useless overheads are created. Elassandra extends the Elasticsearch mapping with new attributes, and the “cql_collection”:”singleton” attribute allows to explicitly map the single-valued fields to the corresponding Cassandra type. By adding this, to the single-valued fields in our fluent-bit index template, the Cassandra rows becomes lighter as shown bellow:

As a result, the Cassandra storage for logs is optimized and becomes a lot more efficient than storing the JSON _source document in a Lucene stored field.

Elassandra storage optimization

For each log records, Fluentbit adds some Kubernetes metadata, and for a docker container, these metadata are always the same. With the Cassandra wide row storage, we can store these metadata only once and for many log records, thus reducing the disk size of SSTables. The following figure illustrates the Cassandra storage of Elasticsearch documents:

To achieve this, an Elasticsearch pipeline transforms the original JSON documents to add a new unique timestamp stored as a timeuuid (a Type 1 UUID), and build a document _id in the form of a Cassandra compound key with the docker container id as the partition key, and the new timeuuid as a clustering key.

After cleaning the existing index and redeploying the new elasticsearch pipeline and template, the SSTables contains one wide row per container and per day. The log storage is now optimal and Kuberetes metadata is only stored once per day for each docker container. Even timestamp fields (es_time and time) can be removed because some time is freed up for the Cassandra clustering key, a timeuuid column seen as a date by elasticsearch.

Thanks to the Cassandra multi-master architecture, write operations were made possible on at least one node while one was restarting, event through the elasticsearch API ! While a node was missing, the Elasticsearch indices became yellow, indicating there was a missing node, but the search was still consistent.

After the node restarted, the Elasticsearch indices switched back to green and the Cassandra nodes 0 and 2 sent hint handoff to node 1 to recover from the missed log records.

Conclusion

Elassandra presents an efficient new opportunity with Kuberentes for many user-cases. As demonstrated, with the right configuration, you can use the Elasticsearch pipeline processors to change and optimize the Cassandra storage, even if you don’t index at all in Elasticsearch (just say index=no in your Elasticsearch mapping, and you will only get the data in the underlying Cassandra table). And furthermore, you can maintain your Elassandra clusters with no downtime when rolling upgrades or easily scale up/down Elassandra (no need to re-index all).

Finally, Elassandra will support the cross-kuberentes-cluster replication in the multi-cloud environments, but that’s another story!