Details

Description

When select a column family, the only way to figure out all columns inside the column family is to do a scan. Since HBase does not have an API to get column names only, we have to provide a temporary workaround. We can add an option to do full table scan, and we scan small amount of data to figure out the column patterns by default. This short cut make assumption that data are in repeated pattern, scanning one row should be sufficient to find all the columns. The same principle applies in scanning for unique row names.

Ari Rabkin
added a comment - 28/Dec/10 22:18 I think that assumption is usually a safe one – particularly if the scan only covers, say, the most recent 20 rows.
What about explicitly keeping a separate metadata table, either in HBase or in Zookeeper?

A separate metadata table to keep track of the columns is a better implementation than this jira implies. There are two missing pieces for the recommendation to work. We need a workflow scheduler to run periodically, and the mapreduce job to scan for extra columns with specified table name and time range.

For the first one, workflow scheduler system, we can use Oozie workflow scheduler. For mapreduce which scan for columns, and update the meta data table, I filed CHUKWA-570.

Eric Yang
added a comment - 29/Dec/10 03:11 A separate metadata table to keep track of the columns is a better implementation than this jira implies. There are two missing pieces for the recommendation to work. We need a workflow scheduler to run periodically, and the mapreduce job to scan for extra columns with specified table name and time range.
For the first one, workflow scheduler system, we can use Oozie workflow scheduler. For mapreduce which scan for columns, and update the meta data table, I filed CHUKWA-570 .