Page History

...

Due to discrepancies of Avro and Pig data models, AvroStorage has:

• Wiki MarkupLimited support for "record": we do not support recursively defined record because the number of fields in such records is data dependent. For instance, _\{"type":"record","name":"LinkedListElem", "fields":\[{"name":"data","type":"int"\},\{"name":"next", "type":\["null","LinkedListElem"\]\}\]\}_;
• Wiki MarkupLimited support for "union": we only accept nullable union like \ ["null", "some-type"\].

For simplicity, we also make the following assumption:

...

Users can choose not to provide any parameter to AvroStorage and Avro schema of output data is derived from its Pig schema. This may result in undesirable schemas due to discrepancies of Pig and Avro data models or problems of Pig itself:

...

• The derived Avro schema will wrap each (nested) field with a nullable union because Pig allows NULL values for every type and Avro doesn't. For instance, when you read in Avro data of schema _"boolean"_ and store it using AvroStorage(), you will get _\["null","boolean"\]_.

• The derived Avro schema may contain unwanted tuple wrappers because: 1) Pig only generates tuples; 2) items of Pig bags can only be tuples. AvroStorage can automatically get rid of such wrappers, but sometimes you still see them as in example B.

...

• field<n> notnull
  This indicates the n th field (and its nested fields) in the output tuple is notnull.

• data pathunmigrated-wiki-markup
• +field<n> def:name+
  Users can provide predefined schemas in Avro files using option +\--data path+, where _path_ points to a directory of Avro files or a single Avro file. This is used together with field parameter +field<n> def:name+. AvroStorage internally constructs two maps: map\[typeName\]=>schema and map\[fieldName\]=>schema and users can specify which schema to use by providing corresponding _name_. This option is useful when users want to do simple processing of input data (like filtering and projection) and store it using predefined schemas in input. Please refer to example C for more details.

• field<n> str
  Users can directly specify the schema of field n where str is a string representation of Avro schema. The usage of this option is similar to schema str except that the schema is only applied to the n th field.

...

The other maps from field names to schema as:

The other maps from field names to schema as:

...

This documentation was originally written by Lin Guo, and appeared at http://linkedin.jira.com/wiki/display/HTOOLS/AvroStorage+-+Pig+support+for+Avro+data^{Image Removed}