...
- Motivation:
- Gobblin has about 400k java lines of code spread across ~76 different modules. Its very difficult to navigate the code, specially for new beginners. the proposal is to re-organise the modules into proper parent categories.
- Proposed Change:
- collapse all modules under following parent modules
- Gobblin-core
- Gobblin-API
- Gobblin-connectors
- Gobblin-plugins
- Gobblin-docs
- Gobblin-utilitiesutils
- Gobblin-deployments
- collapse all modules under following parent modules
- New or Changed Public Interfaces: there should not be have any impact on compatibility.
- Migration Plan and Compatibility: since the packaging structure ( bin, conf, lib ) is not changing, there should not be requiring migration plan. ( May only the full package.class_name if its used as utility)
- Rejected Alternatives: N/A ( there is no alternative to arranging the modules)
...
Note: bold are parent modules that we should keep, every other module should be rollable under the parent ones.
Dir/ Parent Module (parent) | Dir/ Module (sub) | Purpose | new purpose | collapsible underSub-Module | Old Purpose | New Parent Module | new purpose | |
---|---|---|---|---|---|---|---|---|
gobblin-admin | ||||||||
gobblin-api | for all platform API, REST and Gobblin-service APIs | |||||||
gobblin-audit | ||||||||
gobblin-aws | Gobblin-deployment-modes | |||||||
gobblin-binary-management | ||||||||
gobblin-cluster | ||||||||
gobblin-compaction | gobblin-utils | |||||||
gobblin-config-management | Gobblin-core | |||||||
gobblin-config-client | ||||||||
gobblin-config-core | ||||||||
gobblin-connectors | New Repo for all source, writers, publishers or even convertors | |||||||
new separate repo for gobblin-connectorscore | gobblinGobblin-core | |||||||
gobblin-core-base | Gobblin-core | |||||||
gobblin-data-management | ?? | |||||||
gobblin-distribution | ||||||||
gobblin-docker | Gobblin-deployment-modes | |||||||
gobblin-base | ||||||||
gobblin-distributions | ||||||||
gobblin-standalone | ||||||||
gobblin-wikipedia | ||||||||
gobblin-docs | ||||||||
gobblin-example | ||||||||
gobblin-hive-registration | Gobblin-connectors | |||||||
gobblin-integration-test-log-dir | Gobblin-test | |||||||
gobblin-integration-test-work-dir | Gobblin-test | |||||||
gobblin-metastore | Gobblin-core | |||||||
gobblin-metrics-libs | gobblin-metrics | |||||||
gobblin-metrics | ||||||||
gobblin-metrics-base | ||||||||
gobblin-modules | gobblin-connectors | separate this out under new gobblin-connectors repo | gobblin-connectors||||||
gobblin-avro-json | ||||||||
gobblin-azkaban | Gobblin-plugins | |||||||
gobblin-azure-datalake | ||||||||
gobblin-codecs | gobblin-utils | |||||||
gobblin-compliance | gobblin-utils | |||||||
gobblin-couchbase | ||||||||
gobblin-crypto | gobblin-utils | |||||||
gobblin-crypto-provider | gobblin-utils | |||||||
gobblin-elasticsearch | ||||||||
gobblin-elasticsearch-deps | ||||||||
gobblin-eventhub | ||||||||
gobblin-grok | ||||||||
gobblin-helix | ||||||||
gobblin-http | ||||||||
gobblin-kafka-08 | ||||||||
gobblin-kafka-09 | ||||||||
gobblin-kafka-common | ||||||||
gobblin-metadata | gobblin-metrics | |||||||
gobblin-metrics-graphite | ||||||||
gobblin-metrics-hadoop | ||||||||
gobblin-metrics-influxdb | ||||||||
gobblin-orc-dep | ||||||||
gobblin-parquet | ||||||||
gobblin-service-kafka | ||||||||
gobblin-sql | ||||||||
gobblin-zuora | ||||||||
google-ingestion | ||||||||
gobblin-oozie | Gobblin-plugins | |||||||
gobblin-rest-service | Gobblin-core | |||||||
gobblin-rest-api | ||||||||
gobblin-rest-client | ||||||||
gobblin-rest-server | ||||||||
gobblin-restli | Gobblin-core / Gobblin-service ? | |||||||
gobblin-flow-config-service | ||||||||
gobblin-restli-utils | ||||||||
gobblin-throttling-service | ||||||||
gobblin-runtime | Gobblin-core | |||||||
gobblin-runtime-hadoop | Gobblin-connectors | |||||||
gobblin-salesforce | gobblin-connectors | |||||||
gobblin-service | Gobblin-deployment-modes | |||||||
gobblin-test | Gobblin-test | |||||||
gobblin-test-harness | Gobblin-test | |||||||
gobblin-test-utils | Gobblin-utils | |||||||
gobblin-tunnel | ||||||||
gobblin-utilityutils | ||||||||
gobblin-yarn | Gobblin-deployment-modes | |||||||
Gobblin-deployment-modes | for all aws, yarn, MR, deployment modes modules |
Phase-2: code refactor to support module arrangement
...