...
ExecutionJobVertexInitializedEvent: This event is responsible for recording the initialization information of ExecutionJobVertex, its content contains the decided parallelism of this job vertex, and its input information. This event will be triggered and written out when a an execution job vertex is initialized.
ExecutionVertexFinishedEvent：This event is responsible for recording the information of finished task. ExecutionVertexFinishedEvent: This event is responsible for recording the information of finished task. Our goal is that all finished tasks don’t need to re-run, so the simple idea is to trigger an event when a task is finished. The content of this event contains:
The state of the finished task/ExecutionVertex, including IO metrics, accumulators, etc. These contents can be easily obtained from ExecutionGraph.
- If the job vertex which this task belongs to has operator coordinators, the states of the operator coordinators also need to be recorded.
...
interface ShuffleMaster<T extends ShuffleDescriptor> extends AutoCloseable { //… other methods /** * Get all partitions and their metrics, the metrics mainly includes the meta information of partition(partition bytes, etc). * @param jobId ID of the target job * @return All partitions belongs to the target job and their metrics */ Collection<PartitionWithMetrics> getAllPartitionWithMetrics(JobID jobId); interface PartitionWithMetrics { ShuffleMetrics getPartitionMetrics(); ShuffleDescriptor getPartition(); } interface ShuffleMetrics { ResultPartitionBytes getPartitionBytes(); } } |
Compatibility, DeprecationDeprecation, and Migration Planand Migration Plan
In the first version, the job recovery will be an optional optimization which the user has to activate explicitly by setting the config option execution.batch.job-recovery.enabled: true. This entails that Flink's default behavior won't change.
...