architecture diagram
sequence diagram
core module sequence diagram
the communication messages between core and dispatcher
RPC | Desc | Request | Response |
submitSql | The core module submit record sql to dispatcher | SubmitRequest { String recordSql; Enum engine; // Spark,Hive,Presto,etc. String owner; Integer maxRetryCount; } | SubmitResponse{ Integer code; String jobId; Enum errorCode; Exception ex; } |
getJobStatus | The core module get the status of job from the dispatcher by jobid | JobStatusRequest { String jobId; } | JobStatusResponse{ Integer code; Enum jobStatus; Enum errorCode; Exception ex; } |
getMetricResult | The core module will get the result of recordSql from the dispatcher by jobid | MetricRequest { String jobId; } | MetricResponse{ Integer code; Double metric; Enum errorCode; Exception ex; } |
validateSQL | The core module submit record sql to dispatcher for validating the syntax of record sql | ValidateSQLRequest { String recordSql; Enum engine; // Spark,Hive,Presto,etc. } | ValidateSQLResponse{ Integer code; Enum errorCode; Exception ex; } |
Code | 200 | The dispatcher accept the request and process success | |
400 | The request is rejected by the dispatcher, because it's a bad request | ||
500 | The dispatcher accept the request but process failed | ||
ErrorCode | 0 | recordSql syntax error | |
1 | internal error, dispatcher self is crashed | ||
2 | external error, target engine is crashed when dispatcher call,etc | ||
JobStatus | 0 | ACCEPTED | |
1 | RUNNING | ||
2 | SUCCESS | ||
3 | FAILED |
the communication messages between the master node of core and the worker node of core
RPC | Desc | Request | Response | Resp Desc |
registDQWorkerNode | When DQ Worker node start, it will regist it self to master then the master can submit tasks to the worker node | String hostName | int code | 200: the master returns 200 and then it will assigned tasks to the node other:regist failed, and the woker will retry regularly until the master returns 200 |
reportDQWorkNodeStatus | The worker node will report self status regularly, including the id list of running tasks, waitting tasks, success tasks, failed tasks, etc. | List<Integer> runningIdList List<Integer> waittingIdList List<Integer> successIdList List<Integer> failedIdList | int code | 200: the master returns 200 and then it will assign tasks to the node |
submitDQTask | The master can submit tasks to the worker node | int instanceId | int code | 200: the worker accepts the task and then the task's status can be queried or be killed in this node 5XX: the task was rejected by the node because the error happened in the node server 4XX: the task was rejected by the node because the request is error, like find no task info by the instanceId, etc. |
stopDQTask | The master can stop tasks in the worker node | int instanceId | int code | 200: the worker process the request success, 5XX: the request was rejected by the node because the error happened in the node server 4XX: the request was rejected by the node because the request is error, like find no task info in the node by the instanceId, etc. |
querySingleDQTask | The master can DQ Task status from the worker node | int instanceId | int code int status | 200: the worker process the request success, then the value of status will descripe task's status 0: init 1: waitting 2: recording 3: evaluating 4: alerting 5: success 6: failed 5XX: the request was rejected by the node because the error happened in the node server 4XX: the request was rejected by the node because the request is error, like find no task info in the node by the instanceId, etc. |
nodeHeartBeat | The heart beat message will be used to confirm the woker is alive, and then the task can be assigned to the node | long timestamp | int code | 200: the master accepts it and updates the worker info success |
7 Comments
William Guo
in the sequence diagram, we need to let our end users to configure timeout.
Eugene Liu
Hi, Warden Wang , I give two comments on your RPC interface definition.
Warden Wang
Thanks for your suggestions.
Anonymous
Anonymous
Anonymous
Anonymous