Motivation

  1. Currently segment admin rest APIs are mixed with table admin rest APIs, and the documents for the APIs do not match the actual behavior.
  2. Segment toggle (enable/disable) should not be allowed as LLC real-time segment manager and rebalancer won't work properly with segments manually turned OFFLINE in ideal state.
  3. Segment deletion should not be allowed via GET request.
  4. Support batch segment deletion (right now we only support single delete/delete all).

New APIs

The following requests can take a query parameter "type" (OFFLINE or REALTIME) for table type. The request will be performed to tables that match the table name and type.
E.g. "foobar_OFFLINE" matches: ("foobar_OFFLINE", null), ("foobar_OFFLINE", OFFLINE), ("foobar", null), ("foobar", OFFLINE);
"foobar_OFFLINE" does not match: ("foo", null), ("foobar_REALTIME", null), ("foobar_REALTIME", OFFLINE), ("foobar_OFFLINE", REALTIME).

Requests with optional "type":

  • GET /segments/{tableName}
  • GET /segments/{tableName}/servers
  • POST /segments/{tableName}/reload

Requests with mandatory "type":

  • POST /segments/{tableName}/delete
  • DELETE /segments/{tableName}

The following response example are from the table of HybridClusterIntergationTest.

GET /segments/{tableName}

  • Get the name of all segments

  • Example response: 

    GET http://localhost:18998/segments/mytable
    [
  {
    "OFFLINE": [
      "mytable_16071_16101_0 %",
      "mytable_16102_16129_1 %",
      "mytable_16130_16159_2 %",
      "mytable_16160_16189_3 %",
      "mytable_16190_16220_4 %",
      "mytable_16221_16250_5 %",
      "mytable_16251_16281_6 %",
      "mytable_16282_16312_7 %"
    ]
  },
  {
    "REALTIME": [
      "mytable_REALTIME_1573511783058_0__0__1573511783108",
      "mytable_REALTIME_1573511783058_0__0__1573511786593",
      "mytable_REALTIME_1573511783058_0__0__1573511788515"
    ]
  }
]

GET /segments/{tableName}/servers

  • Get the map from server to segments hosted by the server

  • Example response:

    GET http://localhost:18998/segments/mytable/servers
    [
  {
    "tableName": "mytable_OFFLINE",
    "serverToSegmentsMap": {
      "Server_localhost_8098": [
        "mytable_16071_16101_0 %",
        "mytable_16102_16129_1 %",
        "mytable_16130_16159_2 %",
        "mytable_16160_16189_3 %",
        "mytable_16190_16220_4 %",
        "mytable_16221_16250_5 %",
        "mytable_16251_16281_6 %",
        "mytable_16282_16312_7 %"
      ]
    }
  },
  {
    "tableName": "mytable_REALTIME",
    "serverToSegmentsMap": {
      "Server_localhost_8099": [
        "mytable_REALTIME_1573511783058_0__0__1573511783108",
        "mytable_REALTIME_1573511783058_0__0__1573511786593",
        "mytable_REALTIME_1573511783058_0__0__1573511788515"
      ]
    }
  }
]

GET /segments/{tableName}/crc

  • Get the map from segment to CRC of the segment (OFFLINE table only)

  • Example response:

    GET http://localhost:18998/segments/mytable/crc
    {
  "mytable_16071_16101_0 %": "2588864916",
  "mytable_16102_16129_1 %": "2849879226",
  "mytable_16130_16159_2 %": "1461787915",
  "mytable_16160_16189_3 %": "267087240",
  "mytable_16190_16220_4 %": "14495027",
  "mytable_16221_16250_5 %": "1782846277",
  "mytable_16251_16281_6 %": "3738545065",
  "mytable_16282_16312_7 %": "3706869006"
}

GET /segments/{tableName}/{segmentName}/metadata

  • Get the metadata for a segment

  • Example response:

    GET http://localhost:18998/segments/mytable/mytable_16071_16101_0%20%25/metadata
    [
  {
    "tableName": "mytable_OFFLINE",
    "segmentMetadata": {
      "segment.offline.download.url": "http://localhost:18998/segments/mytable/mytable_16071_16101_0+%25",
      "segment.time.unit": "DAYS",
      "segment.start.time": "16071",
      "segment.end.time": "16101",
      "segment.total.docs": "9746",
      "segment.table.name": "mytable",
      "segment.creation.time": "1573511773203",
      "segment.name": "mytable_16071_16101_0 %",
      "segment.index.version": "v1",
      "segment.offline.refresh.time": "-9223372036854775808",
      "custom.map": null,
      "segment.type": "OFFLINE",
      "segment.offline.push.time": "1573511783497",
      "segment.crc": "2588864916"
    }
  }
]

POST /segments/{tableName}/{segmentName}/reload

  • Reload a segment

POST /segments/{tableName}/reload

  • Reload all segments

POST /segments/{tableName}/delete

  • Batch delete segments in the payload of segment names of json list format

DELETE /segments/{tableName}/{segmentName}

  • Delete a segment

DELETE /segments/{tableName}

  • Delete all segments

Deprecated APIs

Deprecated APIs are still supported for backward-compatible, except for the un-intended behavior of segment toggle (enable/disable/drop).

Here is the map from deprecated APIs to new APIs. The response format for APIs might now be identical to the old ones, so when migrating old APIs to new APIs, adjust the response parsing code accordingly.

  • GET /tables/{tableName}/segments → GET /segments/{tableName}/servers
  • GET /tables/{tableName}/segments/metadata → GET /segments/{tableName}/servers
  • GET /tables/{tableName}/segments/crc → GET /segments/{tableName}/crc
  • GET /tables/{tableName}/segments/{segmentName} → GET /segments/{tableName}/{segmentName}/metadata
  • GET /tables/{tableName}/segments/{segmentName}/metadata → GET /segments/{tableName}/{segmentName}/metadata
  • GET /tables/{tableName}/segments/{segmentName}/reload → POST /segments/{tableName}/{segmentName}/reload
  • POST /tables/{tableName}/segments/{segmentName}/reload → POST /segments/{tableName}/{segmentName}/reload
  • GET /tables/{tableName}/segments/reload → POST /segments/{tableName}/reload
  • POST /tables/{tableName}/segments/reload → POST /segments/{tableName}/reload

The deprecated APIs are subject to be removed in the next release.

Other Segment Level APIs (Unchanged)

  • GET /segments/{tableName}/{segmentName}: download a segment
  • POST /segments: upload a segment
  • POST /v2/segments: upload a segment

Backward-incompatible Changes

Segment toggle (including enable/disable/drop) support is removed.

  • No labels