Versions Compared

Old Version 5

changes.mady.by.user lincoln lee

Saved on

compared with

New Version 6

changes.mady.by.user ron liu

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Modifying this option may affect the refresh mode of the background refresh jobs of all registered dynamic tables in the Catalog, so please modify it carefully. If you really need to modify the job refresh mode, please prioritize to realize it by modifying dynamic table freshness.


Gateway Rest API

v3/dynamic-tables/refresh

Verb: POST

Response code: 200 OK

Trigger a refresh operation of dynamic tables.

Request

body

{

  "tables": [],

  "scheduleTime": "",

  "configuration" {}

}

Response

body

{

  jobId: "",

  clusterInfo: {}

}

This API is used for workflow scheduler to trigger a refresh operation of dynamic table aumatically.#TODO#

Proposed Design

Integrating the aforementioned context, Dynamic Table is a new table type introduced for Flink SQL that enables users to build and manage data pipelines, it is automatically refreshed at specified freshness intervals. This means that the dynamic table needs to be bound to a transformation SQL statement, and based on the defined data freshness requirements, a background data refresh job will be created to automatically update the data.

           Image Added

So dynamic table is composed of three parts:

...

  • SQL CLI: A user interaction interface responsible for submitting dynamic table statements. 
  • SQL Gateway: In charge of parsing dynamic table statements, creating tables, submitting jobs to flink cluster. Additionally, the workflow scheduler will interact with SQL Gateway to submit batch workflows.
  • Dynamic Table Manager: As an embedded manager in SQL Gateway that interacts with Workflow Scheduler through Restful APIs. It is responsible for deducing the data refresh mode for dynamic table, deriving streaming job runtime parameters, CRON expressions for batch scheduling jobs, creating batch-scheduled jobs to scheduler, and more. 
  • Catalog: Storage for dynamic table metadata, which includes schema, options, freshness, definition query, refresh job information, etc. 
  • Pluggable Workflow Scheduler: Interacts bidirectionally with SQL Gateway through Restful APIs, responsible for creating batch-scheduled jobs and triggering their execution. It supports open-source workflow schedulers like Airflow, DolphinScheduler through a pluggable mechanism. 
  • Flink Cluster: Runs Flink Jobs, including both streaming and batch jobs.

#TODO-add-arch-img#Image Added

As illustrated in the architecture diagram above, the workflow for executing a Dynamic Table create statement is as follows:

...