Problem Statement

The pinot schema timespec looks like this:

...

1. PRIMARY: The primary date time column. This will be the date time column which keeps the milliseconds value. This will be used as the default time column, in references by pinot code (e.g. retention manager)
2. SECONDARY: The date time columns which are not the primary columns with milliseconds value. These can be date time columns in other granularity, put in by applications for their specific use cases.
3. DERIVED: The date time columns which are derived, say using other columns, generated via rollups, etc.

Examples

“dateTimeFieldSpec”:
{

  “name” : “Date”,
  “dataType” : “LONG”,
  “format” : “1:HOURS:EPOCH”,
  “granularity” : “1:HOURS”,
  “dateTimeType” : "PRIMARY"

}

“dateTimeFieldSpec”:
{

  “name” : “Date”,
  “dataType” : “LONG”,
  “format” : “1:MILLISECONDS:EPOCH”,
  “granularity” : “5:MINUTES”,
  “dateTimeType” : "PRIMARY"

}

“dateTimeFieldSpec”:
{

  “name” : “Date”,
  “dataType” : “LONG”,
  “format” : “1:DAYS:SIMPLE_DATE_FORMAT:yyyyMMdd”,
  “granularity” : “1:DAYS”,
  “dateTimeType” : "SECONDARY"

}

Migration

Once this change is pushed in, we will migrate all our clients to start populating the new DateTimeFieldSpec, along with the TimeSpec.

...