Archives

Introduction to the Video Intelligence API

Google Cloud Video Intelligence API makes videos searchable, and discoverable, by extracting metadata with an easy to use REST API. You can now search every moment of every video file in your catalog and find every occurrence as well as its significance. It quickly annotates videos stored in Google Cloud Storage, and helps you identify key nouns entities of your video, and when they occur within the video. Separate signal from noise, by retrieving relevant information at the video, shot or per frame.

Select a sample video from the list, for example: Volleyball Court, (which is a video made at the Google Mountain View office). Notice the labels.

Select another sample video from the list: Google Work.

Click on the Shots tab. Notice all the keywords detected from the video, which are being renewed per video shot!

Click on the API tab. Notice how the JSON response would look like.

Now, lets try it with one of my own videos, which I’ve uploaded as a public available video in Cloud Storage: gs://leeboonstra-videos/mov_bbb.mp4

Write metadata on video upload

Machine Learning for videos, can be super interesting, in case you want to implement it within your own systems. Let’s say you host a lot of videos on your website. Instead of manually writing meta per video; you could create an ETL job, (for example through Cloud Functions), which listens to the upload event of Google Cloud Storage, runs the Video Intelligence API, and writes the metadata in a database.

This will download the key on your local machine. Create a folder on your machine called: cloudfunctions-videoapi, and move the file over.

Create storage buckets

When you write the JavaScript code for the cloud function, you will need to upload it somewhere. Under the hood, GCP will create a container with a Node environment for you, so you can execute your function. You can upload function code, in a bucket of the Google Storage. Let’s create a bucket which contains function code.

You can create the bucket via the Cloud Console (menu > Storage > Create Bucket), or from the command-line, on your local machine (if you have the gcloud sdk installed), or from your online terminal in the Cloud Console:

gsutil mb -p [PROJECT_ID] gs://[BUCKET_NAME]

Create two buckets, with the name of your [project id] + -functions-src, and one [project-id]-videos. (This way, your bucket will be unique.)

After you’ve created the videos bucket, you will need to make this bucket public, so your video’s get gs:// public urls. You can do this by clicking on the menu button of the bucket (the button with the 3 vertical dots). Click Add Item:

User - allAuthenticatedUsers - Reader

Click Save.

Client Library

Since the client node js library is still in Alpha at the time of writing, we will download the alpha version and host it locally. Once the library gets publically available, you can ignore the next step, and instead link to the online version of the client library. (See: https://www.npmjs.com/package/google-cloud)

TimeOuts

The way how the video api works, is that it first will read in the video to the memory. The machine learning under the hood, is similar to Google’s Cloud Vision API, which can detect what’s on an image. But for the Video Intelligence API this works per frame.
Cloud functions can timeout. You will need to specify a timeout in seconds. By default it’s set to 60s. A number between 1 and 540 (9min) is required. A video with a long duration, will likely make that the cloud function will timeout. So becareful.
You can either setup, the timeout in the Cloud Functions / <myfunction / EDIT menu. Or pass it in directly from the command-line:

Testing the function

Go to Storage, and click on your videos bucket.
Click Upload Files, upload a mp4 file (such as: http://www.divx.com/en/devices/profiles/video ) to the bucket. You will need to make the video public available. Once you’ve done, it will give you a public link.

Once, it’s uploaded, we can check the logs, to see if all the tags are logged. Every time you use console.log() in your Cloud Function code, you will write data to your logs. You can access these logs, by opening your Cloud Console > Logging

Conclusion

By now, you managed to test the Video Intelligence API with your own JavaScript code, within a cloud function based on file upload in a bucket. The power of this cloud function, is that you could easily build an application around this, which makes use of microservices (cloud functions).

It wouldn’t be much work to create an interface (for example with Sencha Ext JS, or just with plain HTML, CSS and JavaScript), which shows a list of video’s and the tags.

I’m logging the tags in the logs of Stack Driver. But instead I could save it in the data store. I could create another cloud function, based on an HTTP trigger which loads the data of the datastore and displays it in the front-end list.

Another idea could be, to pass in the results of the Video Intelligence API into another Machine Learning API, such a translate. To translate the keywords to a different language, before saving it in the database.

TIP: In case you don’t want English meta data, it’s also possible to put the Translate API right after the Video Intelligence API call!