Details

These engines are integrated into Spark as either a Spark SQL data source (full integration) or raw data source (partial integration).

The data source com.sap.spark.engines.disk is used by the disk engine. SQL statements issued on the disk engine are fully integrated into Spark SQL. Disk engine tables therefore behave in exactly the same way as Spark SQL tables.

Step 2: Running 3_Data_on_Disk

The first engine to look at is the Disk Engine. Switch to Zeppelin notebook 3_Data_on_Disk.

First create a disk engine table. Disk engine tables need a partition function and a derived partition scheme. This is what you do in the first two paragraphs.

Create a second disk engine table and verify tables created.

Continue by adding a new paragraph to run a simple cross-engine query.