Data Engineering

The Data Engineering tools enable new data fields and columns to be constructed from the base data in order to aid analysis and understanding of the data. Each of the available tools is described below.

For some tools, such as the Aggregate tool, you will have the choice whether to create permanent or in line columns. Permanent columns are added to the database and displayed in the Data tab for easy access. In line columns enable you to quickly create files and columns which although not permanent, can be saved and reused in the form of tool templates and documents. This method is recommended for columns you are creating 'on the fly' as it avoids creating permanent entries in the database.

The Data Engineering tools are listed on the Tools tab under the 'Data Engineering' category, and can be created as standalone documents from the General tab, by clicking New > Engineering and then selecting the required tool.

NOTE: please consult with MG2 support before creating engineered fields that you need to remain permanent in the OI database. In most cases, these permanent fields need to be created by MG2 support in the back-end to remain permanent. You can, however create a temporary engineered field using the tools below if you're testing "what if" scenarios.

Standard Engineering Tools

Aggregate: Allows records at a lower database level to be aggregated (rolled up) at a higher one. This functionality is typically used to rank customers and products according to criteria from lower or foreign database tables.

Days Between: Allows a selection to be made based on a calculation of the number of days

from or to a particular event, using an existing date column in the database.

Decode: can be used to make the names of tables and fields more meaningful by adding descriptive labels to encoded column data. It also allows the user to assign a value to nulls, correct mistakes in labels, and add filters to existing columns to create new subsets of records.

Parametric Groups: Can be used to create columns whose values are based on queries. The query results are saved in ‘bins’, which are essentially groups of records that share common criteria as specified in the source query.

Banding Tools

Auto Numeric Band: Creates a new column comprising groups of numerical data by copying an existing numeric column and remapping the values into more meaningful ranges.

Numeric Band: Similar to the Auto Numeric Band, but is more flexible as the user can split a numeric column’s values range into uneven bands, which are not necessarily contiguous.

Specialist Tools:

Copy Values To: Used in conjunction with Rank by X to copy a specified field up to a Primary table to create columns like “Most recent transaction” on the customer table or “Highest Earner” on the household table.

Expression: This tool allows advanced users to create new results by manipulating existing data using a range of Engine tools and standard operators.

First or Last: Enables you to extract the first, last, or a specific value from a transaction table.

Prioritize List: Can be used to rank text columns according to a defined criteria. For example, to rank job titles before performing an N per X operation to enable you to select the most prestigious job titles per household.

Rank by X: Ranking is the process whereby for each subset of records associated with the same key in a table, a sequence number is assigned according to the relative value of another field in the table (the X).