...
- Motivation:
- Gobblin has about 400k java lines of code spread across ~76 different modules. Its very difficult to navigate the code, specially for new beginners. the proposal is to re-organise the modules into proper parent categories.
- Proposed Change:
- collapse all modules under following parent modules
New Modules New Purpose Notes Gobblin-core
All gobbline core components required for the platform to run end to end Gobblin-API
All REST APIs that operates on platform Gobblin-connectors
All extractor, convertors, writers and publisher classes for different source and target platforms This can be new repo all together for all kinds of gobblin plugins. Gobblin-plugins
All Gobblin orchastrator or supportive components like Oozie, Azkaban, etc... Gobblin-docs
All documents Gobblin-utils
All additional or supportive utilities like crypto, encryptions, etc... Gobblin-deployments
All gobblin deployment modes, aws, yarn, MR, etc...
- collapse all modules under following parent modules
- New or Changed Public Interfaces: there should not be have any impact on compatibility.
- Migration Plan and Compatibility: since the packaging structure ( bin, conf, lib ) is not changing, there should not be requiring migration plan. ( May only the full package.class_name if its used as utility)
- Rejected Alternatives: N/A ( there is no alternative to arranging the modules)
...
| Parent Module | Sub-Module | Old Purpose | New Parent Module | new purpose||||
|---|---|---|---|---|---|---|---|
| gobblin-admin | |||||||
| gobblin-api | for all platform API, REST and Gobblin-service APIs|||||||
| gobblin-audit | |||||||
| gobblin-aws | Gobblin-deployment-modes | ||||||
| gobblin-binary-management | |||||||
| gobblin-cluster | |||||||
| gobblin-compaction | gobblin-utils | ||||||
| gobblin-config-management | Gobblin-core | ||||||
| gobblin-config-client | |||||||
| gobblin-config-core | gobblin-connectors | New Repo for all source, writers, publishers or even convertors | |||||
| gobblin-core | Gobblin-core | ||||||
| gobblin-core-base | Gobblin-core | ||||||
| gobblin-data-management | ?? | ||||||
| gobblin-distribution | |||||||
| gobblin-docker | Gobblin-deployment-modes | ||||||
| gobblin-base | |||||||
| gobblin-distributions | |||||||
| gobblin-standalone | |||||||
| gobblin-wikipedia | |||||||
| gobblin-docs | |||||||
| gobblin-example | |||||||
| gobblin-hive-registration | Gobblin-connectors | gobblin-integration-test-log-dir | Gobblin-test | gobblin-integration-test-work-dir | Gobblin-test | ||
| gobblin-metastore | Gobblin-core | ||||||
| gobblin-metrics-libs | gobblin-metrics | ||||||
| gobblin-metrics | |||||||
| gobblin-metrics-base | |||||||
| gobblin-modulesgobblin-connectors | separate this out under newgobblin-connectors | repo||||||
| gobblin-avro-json | |||||||
| gobblin-azkaban | Gobblin-plugins | ||||||
| gobblin-azure-datalake | |||||||
| gobblin-codecs | gobblin-utils | ||||||
| gobblin-compliance | gobblin-utils | ||||||
| gobblin-couchbase | |||||||
| gobblin-crypto | gobblin-utils | ||||||
| gobblin-crypto-provider | gobblin-utils | ||||||
| gobblin-elasticsearch | |||||||
| gobblin-elasticsearch-deps | |||||||
| gobblin-eventhub | |||||||
| gobblin-grok | |||||||
| gobblin-helix | |||||||
| gobblin-http | |||||||
| gobblin-kafka-08 | |||||||
| gobblin-kafka-09 | |||||||
| gobblin-kafka-common | |||||||
| gobblin-metadata | gobblin-metrics | ||||||
| gobblin-metrics-graphite | |||||||
| gobblin-metrics-hadoop | |||||||
| gobblin-metrics-influxdb | |||||||
| gobblin-orc-dep | |||||||
| gobblin-parquet | |||||||
| gobblin-service-kafka | |||||||
| gobblin-sql | |||||||
| gobblin-zuora | |||||||
| google-ingestion | |||||||
| gobblin-oozie | Gobblin-plugins | ||||||
| gobblin-rest-service | Gobblin-core | ||||||
| gobblin-rest-api | |||||||
| gobblin-rest-client | |||||||
| gobblin-rest-server | |||||||
| gobblin-restli | Gobblin-core / Gobblin-service ? | ||||||
| gobblin-flow-config-service | |||||||
| gobblin-restli-utils | |||||||
| gobblin-throttling-service | |||||||
| gobblin-runtime | Gobblin-core | ||||||
| gobblin-runtime-hadoop | Gobblin-connectors | ||||||
| gobblin-salesforce | gobblin-connectors | ||||||
| gobblin-service | Gobblin-deployment-modes | ||||||
| gobblin-test | Gobblin-test | ||||||
| gobblin-test-harness | Gobblin-test | ||||||
| gobblin-test-utils | Gobblin-utils | ||||||
| gobblin-tunnel | |||||||
| gobblin-utils | |||||||
| gobblin-yarn | Gobblin-deployment-modes | ||||||
Gobblin-deployment-modes | for all aws, yarn, MR, deployment modes | modules
Phase-2: code refactor to support module arrangement
...