Log Insight: Server vs. Forwarder

Based on some recent Log Insight conversations, I realized there is some confusion around what the difference is between a Log Insight server and a Log Insight forwarder. In this post, I would like to clear up the confusion.

Log Insight Server

First, let’s start with the easy part — what is a Log Insight server? A Log Insight server is the OVA that VMware ships providing you a management product that can be used to ingest data as well as query data. The server is the central part of Log Insight and what was originally released with Log Insight 1.0. Since the release of Log Insight 1.0 two other “components” have been released in addition to the server: Log Insight agents (both Windows and Linux) and Event Forwarding (Log Insight forwarder).

Log Insight Forwarder

In Log Insight 2.5 Event Forwarding was introduced. This feature made it possible to forward events ingested by a Log Insight server to another logging product — either via the ingestion API if the remote destination was another Log Insight server or via syslog for other remote destinations. This means that a Log Insight server — starting with Log Insight 2.5 — is also a “forwarder” or at least contains the capability to serve as a forwarder.

Log Forwarder

So what is a (log) forwarder? Just like it sounds, a forwarder provides the ability to forward logs to a remote destination. Now you might be wondering why you would need a (log) forwarder. Here are a few reasons:

Log Insight Instance

I am about to use another term: “Log Insight Instance” that I would like to define. When I say Log Insight instance, I am referring to a Log Insight server — either standalone or cluster. I use the term instance because it is a centrally controlled and managed Log Insight configuration. You will see why this distinction matters next.

Server vs. Forwarder

OK, now you know what a Log Insight server is, when the Event Forwarding feature was introduced and that it is part of every Log Insight server, and the reasons to or benefits of using forwarding. Now for the critical question: what is the difference between a Log Insight server and a Log Insight forwarder?
A server can be, but does not need to be, a forwarder. A forwarder is always a server. When you hear the term “Log Insight forwarder” you should think a completely different Log Insight instance. The best way to think of it is that a Log Insight forwarder is a log aggregator, but is not the central location of all logs within an environment. It is common to deploy a Log Insight forwarder at every datacenter which forwards its logs to a central Log Insight instance in a single datacenter or two different datacenters for when DR is required.
Now you might be wondering what if I have a central Log Insight instance that also forwards? As a specific example, let’s say Log Insight is the central log aggregator in my environment, but the security team has a different SIEM tool they use for log analysis today and cannot switch. In this case, the Log Insight instance is a server, which also forwards a subset of events to a remote destination — it would not be a Log Insight forwarder as forwarding is not its primary responsibility.
So to be crystal clear, when you hear “Log Insight Forwarder” you so think of a dedicated Log Insight instance whose primary job is to forward events to a remote destination. A Log Insight forwarder should not normally be used for query.

Unsupported

The reason for this post is to clear up the difference between a Log Insight server and a Log Insight forwarder and also to ensure you do not end up with an unsupported configuration. Note that a geo-cluster — a single Log Insight cluster with nodes in different datacenters — is not supported. So if you have say three datacenters and you wish to put forwarders in two you would not build a 3-node cluster in the primary datacenter — a master and two worker nodes — and then deploy and add two additional workers nodes — one in each datacenter — to deploy forwarders. Instead, you would have a 3-node cluster in your primary datacenter and you would deploy two standalone and completely separate Log Insight server instances in the two other datacenters and configure them to forward to the primary datacenter. The net result will be three Log Insight instances which are managed completely separately.

Related

4 Comments

Hywel Burris

Hi Steve,
Thanks for that. So my understanding of this is we now have a highly available solution for a single site, which is great. However, to get a highly available solution for two sites we’d need to deploy two master nodes to have two independent systems and syslog from both sites forwarded to both instances. Is this still the best solution (as with previous versions) for maximum uptime with no loss of any log data on a failure to one site?
Thanks

Hey Hywel — Thanks for the comment! Yes, if you want active-active DR then what you proposed is the best solution. Note you mentioned system between datacenters (i.e. over a WAN) — the best practice would be to deploy Log Insight forwarders in every datacenter and send events to two upstream Log Insight instances as forwarders use Log Insight’s ingestion API which features many benefits including compression.

Great thanks Steve. You have however got me thinking. We were going to use F5 load balancers, in the core network, to split the syslog streams and forward to both LI servers. This does however have a negative symptom should we lose one of the LI hosts or a piece of network inbetween, as it will become “out of sync”. Will using LI Forwarders record upto what point it has forwarded logs to the offline LI host and catch up when it comes back online? (obviously assuming there is enough space to buffer the logs)