BigQuery is revolutionizing the way Big Data is maintained and processed. Its “server-less” architecture is not just enterprise friendly but also developer friendly as it takes care of all the hardware configuration and scalability matrices on its own.

As a developer you just need to know the process of data extraction and that is all. All the performance related hard-work have been taken care by Google’s BigQuery.

Click on select project drop-down and New Project. Take a look at the below snapshot.

Upon clicking the New Project button, a new page will open where you will be asked to enter project details and create new project.

Create New GCP Project

Once your project is created, navigate back to the Credentials Page and look for Manage service Account Link at the bottom. Your application should use Service Account to interact with different Google Services.

Before moving forward, let me give you a quick overview of Service Account and its significance to better understand the tutorial.

Understanding Service Account

The first thing I tell everyone: Do not confuse Service Account with User Account.

Service accounts are not user specific, in-fact, these are special type of Google Accounts that belongs to your application instead of a User. Service account is a way to access Google Services with proper authentication and authorization.

Google assume identity of a Service account while calling Google APIs. Service accounts are made with the sole purpose to eliminate User from directly accessing the resources.

How does Service Account maintain authentication and authorization?

Service account have special keys that are used to authenticate themselves to google while accessing google services.

When you create Service Account, you decide:

Which resources must be accessible by the service account?

What permissions does the Service Account need in order to access a specific resource?

Where will the code that assumes the identity of the service account be running: on Google Cloud Platform or on-premises?

Use below flowchart to figure out how to configure service account:

How to Create Service Account

Login to your Google Cloud Platform dashboard. Navigate to IAM & admin link on the left side bar and click on Service Accounts link.

After selecting project you will see all the listed accounts in the list format. And on the top navbar there is an option to create new service account. Click on Create New Service Account and provide the necessary information.

Make sure to choose the BigQuery role under Project role option. Checkout the image below:

Service Account Creation Page Google cloud Platform

Do not forget to check the furnish new private key checkbox. This will provide you file for download which we will be using later to access data from BigQuery tables.

Click save and your Service account will be created.

Note: Download the private key and save it on your local machine.

Great, you are done with all the boring service account creation part. Now, let’s jump into the development part. Here, you will be fetching the data from the BigQuery tables in JSON format.

Environment Setup

Set the path to the Private Key file in your System Environment. Follow following steps:

Right Click on PC in windows explore and click on properties.

Click on Advanced System Settings

Advanced System Settings

Click on Environment Variables button and create new environment variable with the key GOOGLE_APPLICATION_CREDENTIALS.

Environment VariablesGOOGLE_APPLICATION_CREDENTIALS

You are done setting up the environment and now can make calls to Google BigQuery.

JAVA Project Setup for BigQuery Data Access Using REST API(s)

In this project, you will query the Hacker News comments from the BigQuery database. This dataset is uploaded to BigQuery publicly and is accessible by everyone. To give you the overview, below is the query that you will be firing on the Bigquery database:

Query result for Hacker News comments on 3.41 GB dataset

You will first need to setup JAVA project and include all the required dependencies to make HTTP calls and retrieve data from the BigQuery database. For this project you will be using the pom.xml file provided by Google in their sample projects. You can copy and paste the file contents from the following link: https://github.com/vslala/BigQueryRestSample/blob/master/pom.xml. The main dependencies included in the pom.xml file are:

Google Cloud BigQuery Library (1.35.0)

Google API Client (1.23.0)

Google HTTP Client (1.23.0)

Google OAuth Client (1.23.0)

You can explore pom.xml file later. Once you have created a project and imported all the dependencies into your classpath, then its time for some action 😉

Project Structure

For this project, I would suggest you to use simple structure and try out the application.

I assume you have already created a project with all the required dependencies.

LabelsSample.java

This class is the bootstrap class for the project. It contains the main() method that gives call to the BigQueryDemo.implicit() method.

Take a look at the class:

/*
* Copyright 2016 Google Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.example.bigquery;
import java.io.IOException;
/** Sample demonstrating labeling a BigQuery dataset or table. */
public class LabelsSample {
public static void main(String[] args) throws IOException, InterruptedException {
BigQueryDemo.implicit();
}
}

Article By: Varun Shrivastava

Varun Shrivastava is an innovative full stack web developer. He is a passionate blogger and loves to write about philosophy, programming, tech and relationships. He works as a Software Developer. This is his den, you can get in touch with him here anytime you want.

Website Search

Free Ebook Download ($7.06 Value)

Get your free guide on How To Backup your Wordpress site for Free. It's free twice :). Grab it now!

Varun Shrivastava is an innovative full stack web developer. He is a passionate blogger and loves to write about philosophy, programming, tech and relationships. He works as a Software Developer. This is his den, you can get in touch with him here anytime you want.

Subscribe to Our Newsletters

Get your free guide on How To Backup your Wordpress site for Free. It's free twice :). Grab it now!