AWS Cloudtrail and AWS Config micro-services are vital to companies as they migrate to or begin the process of using Amazon Web Services (AWS). Both services allow the management and monitoring of security, audit, or configuration domains within the AWS Cloud. The Splunk Add-on for Amazon Web Services allows AWS customers the capability to utilize the many features of Splunk software and adds the availability of premium apps such as Splunk Enterprise Security and Splunk IT Service Intelligence.

In this post, we will look at two scenarios where multiple AWS accounts are configured to log Cloudtrail and Config into a consolidated S3 bucket to be ingested into your Splunk system. The first scenario is where the traditional Splunk Add-on resides on a Splunk Heavy Forwarder. The second scenario involves using the Splunk HEC input with the AWS Lambda service.

The following two scenarios were chosen because they are the most common acceptable solutions to the Multiple AWS Accounts logging AWS Cloudtrail and AWS Config use case. There may be other methods that will work better for a given customer and these two are not the only solutions.

Scenario 1

This logging method involves a Splunk heavy forwarder where the AWS add-on is installed and configured. Splunk recommendations such as using S3 buckets, SQS queues and potentially SNS should be utilized where possible. The option to invoke the assumeRole feature can make administration easier than entering and managing the AWS accounts with secret keys individually. The basic steps and flow for this scenario are documented within the Splunk documentation and there are videos on YouTube on this subject as well.

Scenario 2

Assumptions:

In this scenario, it is assumed that Splunk HEC is configured for use in the Splunk environment, AWS Lambda service is enabled within the environment, and AWS VPC’s and policies have been configured to allow traffic between Lambda and the VPC where the Splunk HEC Endpoint is configured. In this environment, there is a desire to move from pulling logs to pushing logs.

Flow Steps:

Cloudtrail or Config is assigned to log in a particular S3 bucket

S3 bucket sends notification of a new object to a Lambda function which will process the notification and pull the data mentioned in the notification

In the case of Cloudtrail or Config data, the payload is compressed, the Lambda function decompresses the payload and converts it to JSON format. After the conversion, the data is sent via an HTTP Post to the Splunk HEC endpoint.

In the event of an error, the payload is not discarded but placed into a designated SQS Queue (Dead Letter Queue)

Reduction in the number of AWS modular inputs on a given HF and therefore their management

Caveats:

Consolidated logging to a central logging account will require custom policies and permissions on S3 buckets and possible IAM roles that will allow the assumeRole feature

Lamba is NOT a free service and there will be an additional cost associated with every time the function runs. The cost will depend on the memory allocated and the timeout for the function. Check your AWS contract or contact them for better pricing information

Basic knowledge of Python, Javascript or other languages allowable by AWS Lambda are highly beneficial when modifying or creating a blueprint (code) for Lambda function execution

Summary

Cloud computing services have experienced a growth curve over the past few years. The need for security, audit and configuration domain insight is an important component that will help organizations as they migrate or add assets to their cloud environment.

Splunk’s AWS integration is a solution that can greatly help with logging AWS Cloudtrail and AWS Config services so that those insights are available easily and when needed. The complexity of the AWS environment may include the logging of multiple accounts into one Splunk environment.

In the use case of one (1) Splunk environment with multiple AWS accounts there are at least 2 options available to ensure that data is indexed and available for analysis. Option 1 involves the traditional use of a Splunk heavy forwarder while Option 2 does not require a Splunk heavy forwarder and uses Splunk HTTP Event Collector (HEC).

Both options are very robust, however, Option 2 has the capability to mitigate risks and is easily suited to growth in the number of AWS accounts and in the amount of data being indexed.