...
- Infers the schema of the Hive table from the Avro schema.
- Reads all Avro files within a table against a specified schema, taking advantage of Avro's backwards compatibility abilities
- Supports arbitrarily nested schemas.
- Translates all Avro data types into equivalent Hive types. Most types map exactly, but some Avro types don't exist in Hive and are automatically converted by the AvroSerde.
- Understands compressed Avro files.
Transparently converts the Avro idiom of handling nullable types as Union\[T, null\] into just T and returns null when appropriate.Wiki Markup - Writes any Hive table to Avro files.
- Has worked reliably against our most convoluted Avro schemas in our ETL process.
...
Avro type | Becomes Hive type | Note | ||
---|---|---|---|---|
null | void | |||
boolean | boolean | |||
int | int | |||
long | bigint | |||
float | float | |||
double | double | |||
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="e1d464c1-fcf5-4912-800d-36e7fc939ccc"><ac:plain-text-body><![CDATA[ | bytes | binary | Bytes are converted to Array[smallint] prior to Hive 0.12.0 | ]]></ac:plain-text-body></ac:structured-macro> |
string | string | |||
record | struct | |||
map | map | |||
list | array | |||
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="657caca3-31cd-4fba-9372-3b30523b8417"><ac:plain-text-body><![CDATA[ | union | union | Unions of [T, null] transparently convert to nullable T, other types translate directly to Hive's unions of those types. However, unions were introduced in Hive 7 and are not currently able to be used in where/group-by statements. They are essentially look-at-only. Because the AvroSerde transparently converts [T,null], to nullable T, this limitation only applies to unions of multiple types or unions not of a single type and null. | ]]></ac:plain-text-body></ac:structured-macro> |
enum | string | Hive has no concept of enums | ||
bytes | binary | Bytes are converted to Array[smallint] prior to Hive 0.12.0 | ||
string | string | |||
record | struct | |||
map | map | |||
list | array | |||
union | union | Unions of [T, null] transparently convert to nullable T, other types translate directly to Hive's unions of those types. However, unions were introduced in Hive 7 and are not currently able to be used in where/group-by statements. They are essentially look-at-only. Because the AvroSerde transparently converts [T,null], to nullable T, this limitation only applies to unions of multiple types or unions not of a single type and null. | ||
enum | string | Hive has no concept of enums <ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="fe71efaa-c55e-4f4d-83ed-7ab627c55847"><ac:plain-text-body><![CDATA[ | ||
fixed | binary | Fixeds are converted to Array[smallint] prior to Hive 0.12.0 ]]></ac:plain-text-body></ac:structured-macro> |
Creating Avro-backed Hive tables
...