IMPORTANT: Only administrators can create Field Extraction Rules (FERs). Analysts may view FERs, run searches against them, and use the field that are extracted by them.

A Field Extraction Rule uses these components:

Rule Name. Describes the rule.

Scope. Specifies the subset of data to use when fields are parsed. For example, you might include the sourceCategory associated with the data you'd like to parse from, or perhaps the sourceHost. Think of the Scope as the first portion of an ad hoc search, before the first pipe ( | ). You'll use the Scope to run a search against the rule.

Parse Expression. Defines the fields you'd like to parse. Choose one or more fields to parse. Because fields are associated with the Rule Name, you can parse one particular field into as many rules as you'd like. For example, to parse a single field, the definition could look like this:

Templates. Parse Templates are provided for common applications such as Apache, AWS, and Microsoft IIS logs. Instead of creating a parse expression, you can select a template from the list, preview it, and then click to apply it. The template will overwrite any existing parse expression.

Limitations

There is a limit of 50 Field Extraction Rules and 200 fields. This includes the default fields defined by Sumo Logic (about 16). The 200-field limit is per account, and deleting rules does not create more space.

Field Extraction Rules are limited to a maximum of 16k (16,384) characters.

Because fields are parsed at the time of data ingestion, Field Extraction Rules only apply to data moving forward. If you want to parse data ingested before the creation of your FER, you can either parse your data in your query, or create Scheduled Views to extract fields for your historical data.

Best practices for designing Rules

Include the most accurate keywords to identify the subset of data from which you want to extract data. Lock down the scope as tightly as possible to make sure it's extracting just the data you want, nothing more. Using a broader scope means that Sumo Logic will inspect more data for the fields you'd like to parse, which may mean that fields are extracted when you don't actually need them.

Create multiple, specific rules. Instead of constructing complicated rules, create multiple rules with basic scope, then search on more than one (rules are additive). The OR and AND commands are supported, just as in any search. For example, you could use one rule to parse Apache log response codes, and then use another rule to parse response time. When used together, you can get all of the information you may need.

Don't extract fields you don't need. Extract the minimum number of fields that should all be present in logs. Every field you include in the scope shows up in every search, so including extra fields means you'll see more results than you may need. It's better to create more rules that extract the fields that are most commonly used. First, look at common data sources and see what's most frequently extracted. Then, think about what you most frequently parse from those sources, then create rules to automatically extract those fields.

Create multiple parse nodrop statements in an FER for a field name to match distinct log patterns. The different parse statements will effectively function like an OR statement since only one will match the log message and return the field value.

Test the scope before creating the rule. Make sure that you can extract fields from all messages you need to be returned in search results. Test them by running a potential rule as a search.

Make sure all fields appear in the Scope you define. When Field Extraction is applied to data, all fields must be present to have any fields indexed; even if one field isn't found in a message, that message is dropped from the results. In other words, it's all or nothing. For multiple sets of fields that are somewhat independent, make two rules.

Re-use field names in multiple FERs if scope is distinct and separate and not matching same messages. To save space and allow for more FERs within your 200 field limit, you can re-use the field names as long as they are used in non-overlapping FERs.

Avoid targeting the same field name in the same message with multiple FERs. When more than one FER targets the same sourceCategory/message with the same field name, one of the rules will NOT apply. The rule applied to the specific field name is randomly selected. Don't use the same field names in multiple FERs that target the same messages.

Supported parsing and search operators

The following operators can be used as part of the Parse Expression in a Field Extraction rule:

parse regex

parse anchor

parse nodrop

csv

double

fields

json

keyvalue

num

Search operators are only used on fields that are extracted using parse expressions.

Unsupported parsing options

The following parsing options are not supported in a Field Extraction Rule:

parse multi

parse regex multi

csv auto

json auto

keyvalue auto

Creating a new Field Extraction Rule

Field Extraction Rules are created and managed using the Field page in the Sumo Logic Web Application. Admins can create their own rules, and delete rules created by other admins.

Recommended articles

Sumo Logic is the industry’s leading secure, cloud-native, machine data analytics service, delivering real-time, continuous intelligence across the entire application lifecycle and stack. More than 1,000 customers around the globe rely on Sumo Logic for the analytics and insights to build, run and secure their modern applications and cloud infrastructures.