Motivation:

  • Gobblin job config stores lot of config for source and target systems for connection, users, password, etc...
  • These info is duplicated at job level and can really be helpful if managed with persistence on database (like MySQL) to easy and flexible management and control at single place.

Proposed Change: 

  • Add "gobblin_sync_systems" table that holds all source and target system informations
  • use token/variables to describe the source/target properties in job config
  • runtime job config resolving should be updated accordingly.


Design:

MySQL DDL: ( V1_0_7__sync_systems.sql )


gobblin_sync_systems
CREATE TABLE `gobblin_sync_system`
(
    `id`             varchar(255) NOT NULL,
    `name`           varchar(64)  NOT NULL,
    `type`           varchar(64)  NOT NULL DEFAULT '',
    `datacenter`     varchar(64)           DEFAULT NULL,
    `region`         varchar(64)           DEFAULT NULL,
    `zone`           varchar(64)           DEFAULT NULL,
    `users`          varchar(1024)         DEFAULT NULL,
    `on_hold`        tinyint(1)   NOT NULL DEFAULT '0',
    `deprecated`     tinyint(1)   NOT NULL DEFAULT '0',
    `source_support` tinyint(1)   NOT NULL DEFAULT '0',
    `target_support` tinyint(1)   NOT NULL DEFAULT '0',
    `configs`        text,
    `created_date`   timestamp    NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_date`   timestamp    NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    PRIMARY KEY (`id`)
) ENGINE = InnoDB
  DEFAULT CHARSET = utf8;


CREATE TABLE `gobblin_sync_system_maintenance`
(
    `id`          int(11)      NOT NULL AUTO_INCREMENT,

    `sync_system` varchar(255) NOT NULL DEFAULT '',
    `type`        varchar(255) NOT NULL,
    `description` varchar(1024)         DEFAULT NULL,
    `start_time`  datetime              DEFAULT NULL,
    `end_time`    datetime              DEFAULT NULL,
    PRIMARY KEY (`id`)
) ENGINE = InnoDB
  AUTO_INCREMENT = 1
  DEFAULT CHARSET = utf8;

APIs:

more description <TBD>

/syncsysm

/syncsystem/maintenance

/syncsystem/maintenance/next
  • No labels