Not a view not a table, meet the Materialized view

According to Wikipedia, a SQL View is the result set of a stored query on the data. Let’s say you have a lot of different tables that you are constantly requesting, using always the same joins, filters and aggregations. With a view, you could simplify access to those datasets while providing more meaning to the end user. It avoids repeating the same complex queries and eases schema evolution.

For example, an application needs access to a products dataset with the product owner and the total number of order for each product. Such queries would need to join the User and Order tables with the Product table. A view would mask the complexity of the schema to the end users by only providing one table with custom and dedicated ACLs.

However such views in Hive used to be virtual and implied huge and slow queries. Instead, you could create an intermediate table to store the results of your query, but such operations require changing your access patterns and has the challenge of making sure the data in the table stays fresh.

We can identify four main types of optimization:

Change data’s physical properties (distribute, sort).

Filter or partition rows.

Denormalization.

Preaggregation.

The goal of Materialized views (MV) is to improve the speed of queries while requiring zero maintenance operations.

The main features are:

Storing the result of a query just like a table (the storage can be in Hive or Druid).

The definition of the MV is used to rewrite query and requires no change in your previous patterns.

The freshness of the data is ensured by the system.

A simple insert in the table is very efficient since it does not require rebuilding the view.

Examples

Let’s see a few examples in order to illustrate the rewriting capabilities.

Denormalization and filtering

Denormalization is the operation of grouping two or more tables into one bigger table. Basically it removes the need of a heavy JOIN operation.

Here are two tables describing employees and departments:

empId

empName

deptID

empSalary

empHireDate

0

Frodo

10

20000

2018-03-03

1

Sam

10

22300

2016-11-11

2

Gimli

20

42300

2016-02-13

3

Galadriel

30

50000

2015-05-28

4

Legolas

30

72000

2015-05-01

And

deptId

deptName

10

Accounting

20

Workers

30

HR

To get all employees recruited in HR during the first semester of 2016 we would use a request looking like this:

Enabling Materialized view rewriting

In order to enable query rewriting using Materialized views this global property is needed: SET hive.materializedview.rewriting=true;.

The user can then select which view is enabled for rewriting: ALTER MATERIALIZED VIEW mv ENABLE|DISABLE REWRITE.

Query rewriting is enable by default for all materialized views.

Rebuilding a Materialized view

Once the MV has been created, it is populated with the data present in the tables. But after data has been modified in the source table, it is not automatically reflected in the MV. Hence, The MV needs to be rebuilt using the command:

ALTER MATERIALIZED VIEW mv REBUILD;

The optimizer will always attempt an incremental rebuild instead of a full one. An incremental rebuild can only be made when new data has been inserted.

In the case of DELETE or UPDATE in the table, a full rebuild will be applied.

Stale data

Data is considered stale if it is the value from the most recent version committed to the original data source. When a MV has not yet been synchronized, the view is considered stale. The system is aware of the status of the MV and will only use the data combined with the fresh data in the original table.

Using the parameter hive.materializedview.rewriting.time.window we can override the default behaviour of ignoring stale data. This parameter is a time window after which outdated materialized views become invalid for automatic query rewriting.

Conclusion

This new feature of Hive looks very promising. Its ability to integrate in nearly any workflow with little to no impact makes it very interesting in a wide variety of use cases. The need for a full rebuild after an update is still a very limiting factor but improvements are on the roadmap.