Phase-1: Module arrangements
- Motivation:
- Gobblin has about 400k java lines of code spread across ~76 different modules. Its very difficult to navigate the code, specially for new beginners. the proposal is to re-organise the modules into proper parent categories.
- Proposed Change:
- collapse all modules under following parent modules
New Modules New Purpose Notes Gobblin-core
All gobbline core components required for the platform to run end to end Gobblin-API
All REST APIs that operates on platform Gobblin-connectors
All extractor, convertors, writers and publisher classes for different source and target platforms This can be new repo all together for all kinds of gobblin plugins. Gobblin-plugins
All Gobblin orchastrator or supportive components like Oozie, Azkaban, etc... Gobblin-docs
All documents Gobblin-utils
All additional or supportive utilities like crypto, encryptions, etc... Gobblin-deployments
All gobblin deployment modes, aws, yarn, MR, etc...
- collapse all modules under following parent modules
- New or Changed Public Interfaces: there should not be have any impact on compatibility.
- Migration Plan and Compatibility: since the packaging structure ( bin, conf, lib ) is not changing, there should not be requiring migration plan. ( May only the full package.class_name if its used as utility)
- Rejected Alternatives: N/A ( there is no alternative to arranging the modules)
Note: bold are parent modules that we should keep, every other module should be rollable under the parent ones.
Parent Module | Sub-Module | Old Purpose | New Parent Module |
---|---|---|---|
gobblin-admin | |||
gobblin-api | |||
gobblin-audit | |||
gobblin-aws | Gobblin-deployment-modes | ||
gobblin-binary-management | |||
gobblin-cluster | |||
gobblin-compaction | gobblin-utils | ||
gobblin-config-management | Gobblin-core | ||
gobblin-config-client | |||
gobblin-config-core | |||
gobblin-core | Gobblin-core | ||
gobblin-core-base | Gobblin-core | ||
gobblin-data-management | ?? | ||
gobblin-distribution | |||
gobblin-docker | Gobblin-deployment-modes | ||
gobblin-base | |||
gobblin-distributions | |||
gobblin-standalone | |||
gobblin-wikipedia | |||
gobblin-docs | |||
gobblin-example | |||
gobblin-hive-registration | Gobblin-connectors | ||
gobblin-metastore | Gobblin-core | ||
gobblin-metrics-libs | gobblin-metrics | ||
gobblin-metrics | |||
gobblin-metrics-base | |||
gobblin-modules | gobblin-connectors | ||
gobblin-avro-json | |||
gobblin-azkaban | Gobblin-plugins | ||
gobblin-azure-datalake | |||
gobblin-codecs | gobblin-utils | ||
gobblin-compliance | gobblin-utils | ||
gobblin-couchbase | |||
gobblin-crypto | gobblin-utils | ||
gobblin-crypto-provider | gobblin-utils | ||
gobblin-elasticsearch | |||
gobblin-elasticsearch-deps | |||
gobblin-eventhub | |||
gobblin-grok | |||
gobblin-helix | |||
gobblin-http | |||
gobblin-kafka-08 | |||
gobblin-kafka-09 | |||
gobblin-kafka-common | |||
gobblin-metadata | gobblin-metrics | ||
gobblin-metrics-graphite | |||
gobblin-metrics-hadoop | |||
gobblin-metrics-influxdb | |||
gobblin-orc-dep | |||
gobblin-parquet | |||
gobblin-service-kafka | |||
gobblin-sql | |||
gobblin-zuora | |||
google-ingestion | |||
gobblin-oozie | Gobblin-plugins | ||
gobblin-rest-service | Gobblin-core | ||
gobblin-rest-api | |||
gobblin-rest-client | |||
gobblin-rest-server | |||
gobblin-restli | Gobblin-core / Gobblin-service ? | ||
gobblin-flow-config-service | |||
gobblin-restli-utils | |||
gobblin-throttling-service | |||
gobblin-runtime | Gobblin-core | ||
gobblin-runtime-hadoop | Gobblin-connectors | ||
gobblin-salesforce | gobblin-connectors | ||
gobblin-service | Gobblin-deployment-modes | ||
gobblin-test | Gobblin-test | ||
gobblin-test-harness | Gobblin-test | ||
gobblin-test-utils | Gobblin-utils | ||
gobblin-tunnel | |||
gobblin-utils | |||
gobblin-yarn | Gobblin-deployment-modes | ||
Gobblin-deployment-modes |
Phase-2: code refactor to support module arrangement
- Motivation:
- while re-arranging modules, once figured out based on above table, we may have to refactor some code to better categorize within the module.
- Proposed Change:
- TBD
- New or Changed Public Interfaces: there should not be have any impact on compatibility.
- Migration Plan and Compatibility: TBD
- Rejected Alternatives: TBD