All data over batched files stored on Amazon S3, Splunk or ELK consumes the received files remotely for ingestion.

Sensor ---> LCC (All Streams) ---> Amazon S3 ---> ( Splunk | ELK )

Bulk events are uploaded to Amazon S3 for archival while alerts and auditing events are sent in real-time to Splunk via Syslog.
This has the added benefit of reducing Splunk license cost while keeping the raw events available for analysis at a cheaper cost.

Splunk provides you with a simple web interface to view and search the data.
It has a paying enterprise version and a free tier.

Below are manual steps to using Splunk with LimaCharlie data. But you can also use
this installation script to install and configure a free
version on a Debian/Ubuntu server automatically.

Because the LimaCharlie.io cloud needs to be able to reach your Splunk instance at all times to upload data, we recommend
you create a virtual machine at a cloud provider like DigitalOcean, Amazon AWS or Google Cloud.

Splunk is the visualization tool, but there are many ways you can use to get the data to Splunk. We will use SFTP as it
is fairly simple and safe.

Back in limacharlie.io, in your organization view, create a new Output.

Give it a name, select the "sftp" module and select the stram you would like to send.

Set the "username" that you used to setup the SFTP service.

Set either the "password" field or the "secret_key" field depending on which one you chose when setting up SFTP.

In "dest_host", input the public IP address of the virtual machine you created.

Set the "dir" value to "/uploads/".

Click "Create".

After a minute, the data should start getting written to the /var/sftp/uploads directory on the server and Splunk should ingest it.

In Splunk, doing a query for "sourcetype=limacharlie" should result in your data.

If you are using the free version of Splunk, note that user management is not included. The suggested method to make
access to your virtual machine safe is to use an SSH tunnel. This will turn a local port into the remote Splunk port
over a secure connection. A sample SSH tunnel command looks like this:

If you have your own visualization stack, or you just need the data archived, you can upload
directly to Amazon S3. This way you don't need any infrastructure.

If the is_indexing option is enabled, data uploaded to S3 will be in a specific format enabling some indexed queries.
LC data files begin with a d while special manifest files (indicating
which data files contain which sensors' data) begin with an m. Otherwise (not is_indexing) data is uploaded
as flat files with a UUID name.

The is_compression flag, if on, will compress each file as GZIP when uploaded.

It is recommended you enable is_compression.

Log in to AWS console and go to the IAM service.

Click on "Users" from the menu.

Click "Add User", give it a name and select "Programmatic access".

Click "Next permissions", then "Next review", you will see a warning about no access, ignore it and click "Create User".

Take note of the "Access key", "Secret access key" and ARN name for the user (starts with "arn:").

Go to the S3 service.

Click "Create Bucket", enter a name and select a region.

Click "Next" until you get to the permissions page.

Select "Bucket policy" and input the policy in sample below:
where you replace the "<>" with the ARN name of the user you created and the "<>" with the
name of the bucket you just created.

Click "Save".

Click the "Permissions" tab for your bucket.

Back in limacharlie.io, in your organization view, create a new Output.

Give it a name, select the "s3" module and select the stream you would like to send.

If you have your own visualization stack, or you just need the data archived, you can upload
directly to Google Cloud Storage (GCS). This way you don't need any infrastructure.

If the is_indexing option is enabled, data uploaded to GCS will be in a specific format enablingsome indexed queries.
LC data files begin with a d while special manifest files (indicating
which data files contain which sensors' data) begin with an m. Otherwise (not is_indexing) data is uploaded
as flat files with a UUID name.

The is_compression flag, if on, will compress each file as GZIP when uploaded.

It is also possible to stream an output over HTTPS. This interface allows you to stream smaller dataset
like investigations or specific sensors or detections. This stream can be achieved via HTTP only without
any additional software layer, although the Python API makes
this task easier using the Spout object.

This feature is heavily used by the Web Interface's Live view of a sensor.

This feature is activated like this:

Issuing an HTTP POST to https://stream.limacharlie.io/<OID> where <OID> is the organization ID you would like to
stream from. As additional data in the POST, specify the following parameters:

api_key: this is the secret API key as provided to you in limacharlie.io.

type: this is the stream type you would like to create, one of event, detect, audit, deployment or log.

cat: optional, specifies the detection type to filter on.

tag: optional, specifies the sensor tags to filter on.

inv_id: optional, specifies the investigation ID to filter on.

The response from this POST will be a stream of data.
The format of this data will be newline-seperated JSON much like all other Outputs.

Note that this method of getting data requires you to have a fast enough connection to receive the data as the buffering
done on the side of stream.limacharlie.io is very minimal. If you are not fast enough, data will be dropped and you will
be notified of this by special events in the stream like this: {"__trace":"dropped", "n":5} where n is the number of
that were dropped. If no data is present in the stream (like rare detections), you will also receive a {"__trace":"keepalive"}
message aproximately every minute to indicate the stream is still alive.

Using this ouput, every element will be sent over HTTP(S) to a webserver of your choice via a POST.

The JSON data will be found in the data parameter of the application/x-www-form-urlencoded encoded POST.

An HTTP header name Lc-Signature will contain an HMAC signature of the contents. This HMAC is computed from the string
value of the data parameter and the secret_key set when creating the Output, using SHA256 as the hashing algorithm.

The validity of the signature can be checked manually or using the Webhook objects of the Python API or
the JavaScript API.

For example, here is a sample Google Cloud Function that can receive a webhook: