Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Enable the new resource. The way of enabling it is to change the setting "RESOURCE_ENABLED" from false to true in ZNode CONFIGS/RESOURCE/leadControllerResource in ZK.
Here is the value of leadControllerResource before enabling the resource:

{
 "id" : "leadControllerResource",
 "simpleFields" :{ "RESOURCE_ENABLED" : "false" },
 "mapFields" : {},
 "listFields" : {}
}

Once the setting "RESOURCE_ENABLED" turned enabled, all All the dual-mode controllers will be immediately registered as masters/slaves in the new resourcePeriodic tasks and real-time segment completion will immediately be distributed. The following criteria must be met in order to test the robustness of this feature before we move on to the next step. It could take days, weeks or months depending on the installation:

  1. All LLC and HLC tables have completed at least one segment and started new ones.
  2. All tables are accounted for in all the periodic tasks (no table is ignored).
  3. At least one round of rolling restart of pinot controllers is done, and criteria 1 and 2 are verified after the restart.
  4. If any of these criteria goes wrong, disable the lead controller resource and everything comes back to the original state. 

If you want to keep Helix controller and Pinot controller running in the same hardware, you can stop at this step. If you want to have these two controllers run in separate hardware, please follow the following steps below.

Step 2

After verifying everything working fine, we can add 1 or more Helix-only controllers to the cluster, so that they can be the candidates of the Helix cluster leadership.

...

Get the leaders for all the tables in the cluster 

GET /leader/tables 

lead controller resource enabledlead controller resource disabled
{
  "leadControllerResourceEnabled": true,
  "leadControllerEntryMap": {
    "leadControllerResource_0": {
      "tableNames": [
        "testTable1_OFFLINE"
      ],
      "leadControllerId": "Controller_172.25.124.150_9000"
    },
    ...
    "leadControllerResource_23": {
      "tableNames": ["testTable2_REALTIME"],
      "leadControllerId": "Controller_172.25.124.150_9008"
    }
  }
}
{
  "leadControllerResourceEnabled": false,
  "leadControllerEntryMap
": { "leadControllerResource_0": { "tableNames": [ "testTable_OFFLINE" ], "leadControllerId": "172.25.124.150_9000" }, ...
"
leadControllerResource_23"
: {
"tableNames": ["testTable2_REALTIME"], "leadControllerId": "172.25.124.150_9000"
}
}
}

Given a table name, return whether lead controller resource is enabled, the partition id and lead controller instance id 

GET /leader/tables/{tableName} 

lead controller resource enabledlead controller resource disabled
{
  "leadControllerResourceEnabled": true,
  "leadControllerEntryMap": {
    "leadControllerResource_7": {
      "tableNames": [
        "testTable_OFFLINE"
      ],
      "leadControllerId": "Controller_172.25.124.150_9000"
    }
  }
}
{
  "leadControllerResourceEnabled": false,
  "leadControllerEntryMap": {
"leadControllerResource_7": { "tableNames": [ "testTable_OFFLINE" ], "leadControllerId": "172.25.124.150_9000" }
}
}


Test Plans and Schedule

Once the final plan has been adjusted and finalized, we can do the following steps. 

...