...
| Code Block | ||||
|---|---|---|---|---|
| ||||
{
"type": "record",
"namespace": "org.apache.hudi.common",
"name": "ErrorRecord",
"fields": [
{
"name": "uid",
"type": "string"
},
{
"name": "ts",
"type": "string"
},
{
"name": "schema",
"type": ["null", "string"],
"default": null
},
{
"name": "record",
"type": ["null", "string"],
"default": null
},
{
"name": "message",
"type": ["null", "string"],
"default": null
},
{
"name": "context",
"type": ["null", {"type": "map", "values": "string"}],
"default": null
}
} |
- `uid`: uuid for the error record
- `ts`: creation unix timestamp for the error record
- `schema`: original schema for the record if any
- `record`: original serialized record in json if any
- `message`: additional message or any string like error stacktrace to be attached
- `context`: kv pairs for any related context info like commitTime, tableName, partitionpath, recordKey, etc
Errors table
Users can choose to use local configure, based on their preferences, error tables as local or a global one, depending on their preferencesglobal ones.
Local error table
Users may apply `hoodie.write.error.table.global=false` (default) to Hudi writers such that the failed By default, if error table is enabled, it will be a local error table. Failed records will be written to a local Hudi table alongside with the target tableoriginal Hudi table with a suffix (configurable) like `_errors`.
Global error table
Users may apply To write to a global error table, users can configure `hoodie.write.error.table.base.path=<some file system path>` and `hoodie.write.error.table.global=true` to Hudi writers such that the failed records will be written to a global Hudi tablename=foobar`. If either of these 2 configs were set, error table is set to global mode and `hoodie.write.error.table.suffix` will be omitted.
Configurations
| key | default | |||
|---|---|---|---|---|
| hoodie.write.error.table.enabled | set to true to activate error table handling feature | false | ||
| hoodie.write.error.table.suffix | suffix for local error table name, stored alongside the target table. If the Hudi table is "foo", errored records will be saved to "foo_errors" at the same base dir as configured via `hoodie.base.path` | "_errors" | ||
| hoodie.write.error.table.global | set to true to use global errors table | falsename | error table name | "hoodie_errors" |
| hoodie.write.error.table.global.base.path | base path for global error table | same as `hoodie.base.path` |
...