architecture diagram
...
sequence diagram
...
core module sequence diagram
the communication messages between core and dispatcher
| RPC | Desc | Request | Response |
| submitSql | The core module submit record sql to dispatcher | SubmitRequest { String recordSql; Enum engine; // Spark,Hive,Presto,etc. String owner; Integer maxRetryCount; } | SubmitResponse{ Integer code; String jobId; Enum errorCode; Exception ex; } |
| getJobStatus | The core module get the status of job from the dispatcher by jobid | JobStatusRequest { String jobId; } | JobStatusResponse{ Integer code; Enum jobStatus; Enum errorCode; Exception ex; } |
| getMetricResult | The core module will get the result of recordSql from the dispatcher by jobid | MetricRequest { String jobId; } | MetricResponse{ Integer code; Double metric; Enum errorCode; Exception ex; } |
| validateSQL | The core module submit record sql to dispatcher for validating the syntax of record sql | ValidateSQLRequest { String recordSql; Enum engine; // Spark,Hive,Presto,etc. } | ValidateSQLResponse{ Integer code; Enum errorCode; Exception ex; } |
| Code | 200 | The dispatcher accept the request and process success | |
| 400 | The request is rejected by the dispatcher, because it's a bad request | ||
| 500 | The dispatcher accept the request but process failed | ||
| ErrorCode | 0 | recordSql syntax error | |
| 1 | internal error, dispatcher self is crashed | ||
| 2 | external error, target engine is crashed when dispatcher call,etc | ||
| JobStatus | 0 | ACCEPTED | |
| 1 | RUNNING | ||
| 2 | SUCCESS | ||
| 3 | FAILED |
the communication messages between the master node of core and the worker node of core
| RPC | Desc | Request | Response | Resp Desc |
| registDQWorkerNode | When DQ Worker node start, it will regist it self to master then the master can submit tasks to the worker node | String hostName | int code | 200: |
...
| the master returns 200 and then it will assigned tasks to the node other:regist failed, and the woker will retry regularly until the master returns 200 | ||||
| reportDQWorkNodeStatus | The worker node will report self status regularly, including the id list of running tasks, waitting tasks, success tasks, failed tasks, etc. | List<Integer> runningIdList List<Integer> waittingIdList List<Integer> successIdList List<Integer> failedIdList | int code | 200: the master returns 200 and then it will assign tasks to the node |
| submitDQTask | The master can submit tasks to the worker node | int instanceId | int code | 200: the worker accepts the task and then the task's status can be queried or be killed in this node 5XX: the task was rejected by the node because the error happened in the node server 4XX: the task was rejected by the node because the request is error, like find no task info by the instanceId, etc. |
| stopDQTask | The master can stop tasks in the worker node | int instanceId | int code | 200: the worker process the request success, 5XX: the request was rejected by the node because the error happened in the node server 4XX: the request was rejected by the node because the request is error, like find no task info in the node by the instanceId, etc. |
| querySingleDQTask | The master can DQ Task status from the worker node | int instanceId | int code int status | 200: the worker process the request success, then the value of status will descripe task's status 0: init 1: waitting 2: recording 3: evaluating 4: alerting 5: success 6: failed 5XX: the request was rejected by the node because the error happened in the node server 4XX: the request was rejected by the node because the request is error, like find no task info in the node by the instanceId, etc. |
| nodeHeartBeat | The heart beat message will be used to confirm the woker is alive, and then the task can be assigned to the node | long timestamp | int code | 200: the master accepts it and updates the worker info success |