Indexing Is Removed since 3.0
There are alternate options which might work similarily to indexing:
- Materialized views with automatic rewriting can result in very similar results. Hive 2.3.0 adds support for materialzed views.
- Using columnar file formats (Parquet, ORC) – they can do selective scanning; they may even skip entire files/blocks.
Indexing has been removed in version 3.0 (HIVE-18448).
Overview of Hive Indexes
The goal of Hive indexing is to improve the speed of query lookup on certain columns of a table. Without an index, queries with predicates like 'WHERE tab1.col1 = 10' load the entire table or partition and process all the rows. But if an index exists for col1, then only a portion of the file needs to be loaded and processed.
The improvement in query speed that an index can provide comes at the cost of additional processing to create the index and disk space to store the index.
Hive indexing was added in version 0.7.0, and bitmap indexing was added in version 0.8.0.
Documentation and examples of how to use Hive indexes can be found here:
- Indexed Hive – overview and examples by Prafulla Tekawade and Nikhil Deshpande, October 2010
- Tutorial: SQL-like join and index with MapReduce using Hadoop and Hive – blog by Ashish Garg, April 2012
Configuration Parameters for Hive Indexes
The Configuration Properties document describes parameters that configure Hive indexes.
This section gives some indexing examples adapted from the Hive test suite.
In Hive 0.12.0 and earlier releases, the index name is case-sensitive for CREATE INDEX and DROP INDEX statements. However, ALTER INDEX requires an index name that was created with lowercase letters (see HIVE-2752). This bug is fixed in Hive 0.13.0 by making index names case-insensitive for all HiveQL statements. For releases prior to 0.13.0, the best practice is to use lowercase letters for all index names.
Create/build, show, and drop index:
Create then build, show formatted (with column names), and drop index:
Create bitmap index, build, show, and drop:
Create index in a new table:
Create index stored as RCFile:
Create index stored as text file:
Create index with index properties:
Create index with table properties:
Drop index if exists:
Rebuild index on a partition: