Definition

A set of records in tabular format (a table) ingested in `Hudi` - represents data internal to `Hudi` as opposed to external data, un-managed by `Hudi`.

Design decisions

  1. each def~table has a single `parquet` file and one or more def~timelines (with `delta file`s / `log file`s)
  2. external data is ingested in `Hudi` by one or more def~commits

Related concepts

  1. file format
  2. def~commit
  3. def~query-type
  4. def~query-scan-mode

Status (draft)


Definition

A set of records in tabular format (a table) ingested in `Hudi` - represents data internal to `Hudi` as opposed to external data, un-managed by `Hudi`.

Design decisions

  1. each def~table has a single `parquet` file and one or more def~timelines (with `delta file`s / `log file`s)
  2. external data is ingested in `Hudi` by one or more def~commits

Related concepts

  1. file format
  2. def~commit
  3. def~query-type
  4. def~query-scan-mode

Status (draft)


  • No labels

1 Comment

  1. SemanticBeeng

    #todo clarify how `data schema` is defined and used when external datasets are registered & ingested in `Hudi`.