- Motivation:
- current way of managing job config via files makes it hard to manage job configurations, persisting it in MySQL would make it easier to manage and also provide various CRUD API interfaces to mange the job configuration.
- Proposed Change:
- create MySQL table `gobblin_job_config`
- create REST endpoints for CRUD operations
- add functionality to detect change in any job config and schedule or trigger job accordingly ( similar to current mechanism based on file change )
- New or Changed Public Interfaces:
- POST /
...
- job
- GET /
...
- job/{jobName}
- DELETE /
...
- job/{jobName}
- Migration Plan and Compatibility:
- since this is new functionality, in addition to the current file based job config, user can select either one to manage the job.
- Rejected Alternatives:
- prefer to use
...
- REST.li
...
- (instead of JDBI) as it is built in framework for other APIs
Design:
MySQL tabletables:
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
CREATE TABLE `gobblin_job_store` ( `job_id` int(11) NOT NULL AUTO_INCREMENT, `job_name` varchar(255) NOT NULL, `job_name`description` varchar(255) DEFAULT NULL, `job_is_disabled` tinyint(1) DEFAULT '0', `created_date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, `job_config` json DEFAULT NULL`updated_date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, PRIMARY KEY (`job_name`), UNIQUE KEY `job_id_index` (`job_id`) USING BTREE ) ENGINE=InnoDB AUTO_INCREMENT=2 DEFAULT CHARSET=utf8; CREATE TABLE `gobblin_job_properties_new ` ( `job_name` varchar(255) NOT NULL, `property_key` varchar(255) NOT NULL, `property_value` mediumtext, `created_date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, `modified`updated_date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, PRIMARY KEY (`job_id`name`,`property_key`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; |
- Create New job example
| Code Block | ||
|---|---|---|
| ||
POST http://localhost:9090/job
{
"jobName": "PullFromWikipedia",
"jobDescription": "A getting started example for Gobblin",
"jobDisabled": false,
"jobProperties": {
"job.group": "Wikipedia",
"source.class": "org.apache.gobblin.example.wikipedia.WikipediaSource",
"test_remove": "test_remove_value",
"something": "asdsad"
},
"_jobPropertiesToRemove": [
"something"
]
}
|
2. GET existing job
| Code Block | ||
|---|---|---|
| ||
GET http://localhost:9090/job/PullFromWikipedia
{
"jobName": "PullFromWikipedia",
"jobId": 14,
"jobProperties": {
"source.class": "org.apache.gobblin.example.wikipedia.WikipediaSource",
"test_remove": "test_remove_value",
"job.group": "Wikipedia",
"something": "asdsad"
},
"jobDisabled": false,
"jobDescription": "A getting started example for Gobblin - pull some pages from wikipedia",
"created_date": 1575937161000,
"updated_date": 1575937161000
} |