AWS S3 Source

Amazon Simple Storage Service (Amazon S3) provides a simple web services interface that can be used to store and retrieve any amount of data from anywhere on the web. Add an Amazon S3 Source to upload messages to Sumo Logic.

One Amazon S3 Source can collect data from just a single S3 bucket. However, you can configure multiple S3 Sources to collect from one S3 bucket. For example, you could use one S3 Source to collect one particular data type, and then configure another S3 Source to collect another data type.

NOTE: Amazon S3 sources have some limitations. First, we only support log files that do NOT change after they are uploaded to S3. Support is not provided if your logging approach relies on updating files stored in an S3 bucket. Second, you cannot use Live Tail with Amazon S3 sources.

Configure an Amazon S3 Source

Add an AWS Source for the S3 Source to Sumo Logic. See below for details.

AWS Sources

When you create an Amazon S3 source, you associate it with a hosted collector. Before creating the source, identify the hosted collector you want to use, or create a new hosted collector, as necessary. For instructions, see Configure a Hosted Collector.

Amazon S3 sources have some limitations. First, we only support log files that do NOT change after they are uploaded to S3. Support is not provided if your logging approach relies on updating files stored in an S3 bucket. Second, you cannot use Live Tail with Amazon S3 sources.

On the Collectors page, click Add Source next to a hostedcollector, either an existing hosted collector, or one you have created for this purpose.

Select your AWS Source type.

Enter a name to display for the new Source. Description is optional.

For Bucket Name, enter the exact name of your organization's S3 bucket.
Be sure to double-check the name as it appears in AWS, for example:

For Path Expression, enter the string that matches the S3 objects you'd like to collect. A wildcard (*) can be used in this string. (Do NOT use a leading forward slash. See About Amazon Path Expressions for details). Remember that recursive path expressions use a single wild card.

Collection should begin. Select a collection start time from the menu, or select All Time to collect all logs.
You won't be able to modify the Collection should begin option after you create the source. You can send an UPDATE request or create or modify cutoffTimestamp in your Source JSON file instead.

For Source Category, enter any string to tag the output collected from this Source. (Category metadata is stored in a searchable field called _sourceCategory.)

For Scan Interval, use the default of 5 minutes. Alternately, enter the frequency Sumo Logic will scan your S3 bucket for new data. To learn more about Scan Interval considerations, see About setting the S3 Scan Interval.

Set any of the following under Advanced:

Enable Timestamp Parsing. This option is selected by default. If it's deselected, no timestamp information is parsed at all.

Time Zone. There are two options for Time Zone. You can use the time zone present in your log files, and then choose an option in case time zone information is missing from a log message. Or, you can have Sumo Logic completely disregard any time zone information present in logs by forcing a time zone. It's very important to have the proper time zone set, no matter which option you choose. If the time zone of logs can't be determined, Sumo Logic assigns logs UTC; if the rest of your logs are from another time zone your search results will be affected.

Infer Boundaries. Enable when you want Sumo Logic to automatically attempt to determine which lines belong to the same message.
If you deselect the Infer Boundaries option, you will need to enter a regular expression in the Boundary Regex field to use for detecting the entire first line of multi-line messages.

Boundary Regex. You can specify the boundary between messages using a regular expression. Enter a regular expression for the full first line of every multi-line message in your log files. For an example, see the Define Boundary Regex topic.

Recommended articles

Sumo Logic is the industry’s leading secure, cloud-native, machine data analytics service, delivering real-time, continuous intelligence across the entire application lifecycle and stack. More than 1,000 customers around the globe rely on Sumo Logic for the analytics and insights to build, run and secure their modern applications and cloud infrastructures.