Configure Hive connectivity

By default, Hive saves data for multiple file formats as either binary files or as a set of text files delimited with special characters. Splunk Analytics for Hadoop currently supports 4 Hive (v0.12) file format types: Textfile, RCfile, ORC files and Sequencefile.

Splunk Analytics for Hadoop supports different file formats via its preprocessor framework, providing a data preprocessor called HiveSplitGenerator. This data processor lets Splunk Analytics for Hadoop access and process data stored or used by Hive.

The easiest way to configure Splunk Analytics for Hadoop to connect to Hive tables is to edit indexes.conf to:

Provide Splunk Analytics for Hadoop with the metastore URI.

Specify that Splunk Analytics for Hadoop use the HiveSplitGenerator to read the Hive data.

If you don't want Splunk Analytics for Hadoop to access your metastore server, you can manually configure it to access raw data files that make up your Hive tables. See "Configure Splunk Analytics for Hadoop to read your Hive tables without connecting to Metastore" in this topic.

Splunk Analytics for Hadoop currently supports the following versions of Hive:

Ensure your Hadoop and Hive versions are compatible

When you set up your Hadoop data provider, make sure it uses a compatible version of Hive. Hadoop 2.x or below requires versions of Hive that are 2.x or below. Hadoop 3.x requires versions of HIve that are at least 3.x. If you configure a Hadoop cluster that is version 3.x with a Hive instance that is version 2.x or lower, you will run into connectivity issues when you try to save a file in a Hive file format.

Configure Hive connectivity with a metastore

To configure Hive connectivity, you provide the vix.hive.metastore.uris.

Splunk Analytics for Hadoop uses the information in the provided Metastore server to read the table information, including column names, types, data location and format, thus allowing it to process the search request.

Here's an example of a configured provider stanza that properly enables Hive connectivity. Note that a table contains one or more files, and that each virtual index could have multiple input paths, one for each table.

In the rare case that the split logic of the Hadoop InputFormat implementation of your table is different from that of Hadoop's FileInputFormat, the HiveSplitGenerator split logic does not work. Instead, you must implement a custom SplitGenerator and use it to replace the default SplitGenerator. See Configure Splunk Analytics for Hadoop to use a custom file format for more information.

Configure Splunk Analytics for Hadoop to use a custom file format

To use a custom file format, you edit your provider stanza to add a .jar file that contains your custom classes as follows:

vix.splunk.jars

Note that if you don't specify a InputFormat class, files are treated as text files and broken into records by new-line character.

If you are unable or do not wish to expose your Metastore server, you can configure Hive connectivity by specifying additional configuration items. For Splunk Analytics for Hadoop, the minimum required information is:

columnnames

columntypes

Other information is required if you specify it when you create the table (for example if your tables specify InputFormat instead of Hive, you must tell Splunk Analytics for Hadoop.)

Create a stanza in indexes.conf that provides Splunk Analytics for Hadoop with the list of column names and types of your Hive table(s). These column names become the field names you see when running reports in Splunk Analytics for Hadoop:

Partitioning table data

When using the Hive Metastore, Splunk Analytics for Hadoop automatically analyzes the tables, preserving partition keys and values, and, based on your search criteria, pruning any unwanted partitions. This can help speed up searches.

When not using a Metastore, you can update your [virtual-index] stanza to tell Splunk Analytics for Hadoop about the partitions using key values as part of the file path. For example, the following configuration

Enter your email address, and someone from the documentation team will respond to you:

Send me a copy of this feedback

Please provide your comments here. Ask a question or make a suggestion.

Feedback submitted, thanks!

You must be logged into splunk.com in order to post comments.
Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic.
If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk,
consider posting a question to Splunkbase Answers.

0
out of 1000 Characters

Your Comment Has Been Posted Above

We use our own and third-party cookies to provide you with a great online experience. We also use these cookies to improve our products and services, support our marketing campaigns, and advertise to you on our website and other websites. Some cookies may continue to collect information after you have left our website.
Learn more (including how to update your settings) here »