Motivation:
- Gobblin job config stores lot of config for source and target systems for connection, users, password, etc...
- These info is duplicated at job level and can really be helpful if managed with persistence on database (like MySQL) to easy and flexible management and control at single place.
Proposed Change:
- Add "gobblin_sync_systems" table that holds all source and target system informations
- use token/variables to describe the source/target properties in job config
- runtime job config resolving should be updated accordingly.
Design:
MySQL DDL: ( V1_0_7__sync_systems.sql )
Code Block |
---|
language | sql |
---|
firstline | 1 |
---|
title | gobblin_sync_systems |
---|
linenumbers | true |
---|
|
CREATE TABLE `gobblin_sync_systems`system`
(
`id` varchar(255) NOT NULL,
`name` varchar(64) NOT NULL,
`type` varchar(64) NOT NULL DEFAULT '',
`datacenter` varchar(64) DEFAULT NULL,
`region` varchar(64) DEFAULT NULL,
`zone` varchar(64) DEFAULT NULL,
`users` varchar(1024) DEFAULT NULL,
`on_hold` tinyint(1) NOT NULL DEFAULT '0',
`deprecated` tinyint(1) NOT NULL DEFAULT '0',
`source_support` tinyint(1) NOT NULL DEFAULT '0',
`target_support` tinyint(1) NOT NULL DEFAULT '0',
`config` `configs` text,
`created_date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`updated_date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`id`)
) ENGINE = InnoDB
DEFAULT CHARSET = utf8;
CREATE TABLE `gobblin_sync_systemssystem_maintenance`
(
`id` int(11) varchar(255) NOT NULL DEFAULT '',AUTO_INCREMENT,
`sync_system_id`system` varchar(255) NOT NULL DEFAULT '',
`type` varchar(255) NOT NULL,
`description` varchar(1024) DEFAULT NULL,
`start_time` datetime DEFAULT NULL,
`end_time` datetime DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE = InnoDB
AUTO_INCREMENT = 1
DEFAULT CHARSET = utf8; |
APIs:
more description <TBD>
Code Block |
---|
/syncsysm
/syncsystem/maintenance
/syncsystem/maintenance/next
|