Best practices for data search

By incorporating these ideas when searching data in Devo, you can avoid common problems and improve the effectiveness of the application when working with data tables. In case you need additional help, please contact the Devo support team.

Browser recommendations

When dealing with massive amounts of data, you need to consider the browser memory limitations. For optimal performance:

Periods of time when data is not loaded. These are represented as 'holes' in the chart over the table.

Total loaded rows

The number of rows loaded in all the queries open in your session.

Total rows limit

Limit of total rows displayed.

Estimated memory usage

Current memory consumption.

Limit the number of running queries to optimize performance by minimizing memory use.

If you need to have multiple queries open, use a second incognito mode to have multiple sessions.

By default, data tables show the data from the last 24 hours. Before starting to search or applying any filters, narrow your search selecting a smaller time range. For example, if you want to see the events coming in real-time, select Last hour.

The following are some examples and recommended actions when working on a data table.

Even if you want to search in a broader range of time, apply the filters and operations before selecting a previous date in the From field.

✓ Define the query and then select Last 30 days.

X Avoid modifying the time range selectingLast 30 days before defining the query.

Apply all the filters before grouping. This reduces the memory and computation required for the search activity.

✓ Run a filter prior to grouping data, as in this example where the country US is applied before grouping.

XAvoid grouping data before applying a filter, as shown here. This causes many more rows to be part of the grouping, which is memory-intensive and can severely impact browser performance.

When grouping data, use fields that have a low variability/cardinality, i.e. not many unique values. You can see the distinct values clicking the column header and selecting the arrow at the left.

✓In this example, the grouping is done using the country column (13 distinct values):

X Avoid grouping by numeric fields with a large number of distinct values, like the responseLength column in this example that has 2001 distinct values. Instead, execute an aggregation to sum, average, or min/max, etc. in order to understand the values.

When the data table that you are working with has a large number of columns, hide the columns you don't need.

✓ You can do this when selecting the table in the Finder. Find out how.

✓ You can also do this in the query window using the Search column layout tool. Learn more.

Summary

Time scope

Use time scope to limit overall volume of data.

Filtering

Use filters to limit overall density of data

Use filters to limit potential variations in results. This helps to cut down on the possible number of unique combinations that the grouping will have to produce, and makes any resulting visualization more practical.

Minimize number of columns included to cut down on the possible number of unique combinations that the grouping will have to produce.

Grouping

Never group by columns with a great number of distinct values, like the timestamp column (maximum number of distinct values).

Minimize number of columns included to cut down on the possible number of unique combinations that the grouping will have to produce.

Use the smallest time segmentation that make sense for the query when grouping data. Larger settings use more memory to complete the grouping and any subsequent aggregation.

Applications and dashboards

Use applications and dashboards to push grouping and aggregation calculate to be 100% server side.