Importing Data Locker to Big Query

Introduction

One of the major benefits of transferring data from Data Locker to GCS is that you can import data to Google Big Query. You don't need to rely on your organization's developers to retrieve data from Data Locker. With a simple UI configuration you can transfer the data straight into Google Big Query.

Google Big Query can easily handle data from Data Locker thanks to the standard structure of the CSV files in Data Locker. Once you import data into Google Big Query you can start analyzing it right away.

Prerequisites

Before you can configure GCS to pull data from Data Locker, the following is required:

Your Data Locker Home Folder

Configuring a Transfer Job in GCS

The first step is to configure a transfer job in GCS. Here, you configure GCS to pull data from Data Locker on a daily basis. Once the data gets into GCS, you can import it to Big Query.

Open your GCS Transfer Console

Click on Transfer in the left-hand side menu

Click on Create transfer

Under Select Sourcechoose Amazon S3 bucket

In the Amazon S3 bucket field put af-ext-reports

In the Transfer files with these prefixes field put the following path: your-home-folder/data-locker-hourly/

Specify your access key and secret in the relevant fields

Under Select Destination specify the GCS storage bucket

Under Configure Transfer choose when to run the transfer job

Importing Data from GCS to Big Query

Once you have data from Data Locker imported into GCS, you can transfer it to Big Query. The files are zipped in .gz format but Big Query knows how to unzip the files. Big Query also handles the standard CSV format of files from Data Locker for seamless table creation. Once you can create a table, you can go ahead and analyze your data quickly and easily.