...
Phase-1: Module arrangements
- Motivation:
- Gobblin has about 400k java lines of code spread across ~76 different modules. Its very difficult to navigate the code, specially for new beginners. the proposal is to re-organise the modules into proper parent categories.
- Proposed Change:
- collapse all modules under following parent modules
- Gobblin-core
- Gobblin-API
- Gobblin-connectors
- Gobblin-docs
- Gobblin-utilities
- Gobblin-deployments
- collapse all modules under following parent modules
- New or Changed Public Interfaces: there should not be have any impact on compatibility.
- Migration Plan and Compatibility: since the packaging structure ( bin, conf, lib ) is not changing, there should not be requiring migration plan. ( May only the full package.class_name if its used as utility)
- Rejected Alternatives: N/A ( there is no alternative to arranging the modules)
Note: bold are parent modules that we should keep, every other module should be rollable under the parent ones.
| Dir/ Module (parent) | Dir/ Module (sub) | Purpose | new purpose | collapsible under |
|---|---|---|---|---|
| gobblin-admin | ||||
| gobblin-api | for all platform API, REST and Gobblin-service APIs | |||
| gobblin-audit | ||||
| gobblin-aws | ||||
| gobblin-binary-management | ||||
| gobblin-cluster | ||||
| gobblin-compaction | ||||
| gobblin-config-management | ||||
| gobblin-config-client | ||||
| gobblin-config-core | ||||
| gobblin-connectors | all source, writers, publishers or even convertors | new separate repo for gobblin-connectors | ||
| gobblin-core | ||||
| gobblin-core-base | ||||
| gobblin-data-management | ||||
| gobblin-distribution | ||||
| gobblin-docker | ||||
| gobblin-base | ||||
| gobblin-distributions | ||||
| gobblin-standalone | ||||
| gobblin-wikipedia | ||||
| gobblin-docs | ||||
| gobblin-example | ||||
| gobblin-hive-registration | ||||
| gobblin-integration-test-log-dir | ||||
| gobblin-integration-test-work-dir | ||||
| gobblin-metastore | ||||
| gobblin-metrics-libs | ||||
| gobblin-metrics | ||||
| gobblin-metrics-base | ||||
| gobblin-modules | separate this out under new gobblin-connectors repo | gobblin-connectors | ||
| gobblin-avro-json | ||||
| gobblin-azkaban | ||||
| gobblin-azure-datalake | ||||
| gobblin-codecs | ||||
| gobblin-compliance | ||||
| gobblin-couchbase | ||||
| gobblin-crypto | ||||
| gobblin-crypto-provider | ||||
| gobblin-elasticsearch | ||||
| gobblin-elasticsearch-deps | ||||
| gobblin-eventhub | ||||
| gobblin-grok | ||||
| gobblin-helix | ||||
| gobblin-http | ||||
| gobblin-kafka-08 | ||||
| gobblin-kafka-09 | ||||
| gobblin-kafka-common | ||||
| gobblin-metadata | ||||
| gobblin-metrics-graphite | ||||
| gobblin-metrics-hadoop | ||||
| gobblin-metrics-influxdb | ||||
| gobblin-orc-dep | ||||
| gobblin-parquet | ||||
| gobblin-service-kafka | ||||
| gobblin-sql | ||||
| gobblin-zuora | ||||
| google-ingestion | ||||
| gobblin-oozie | ||||
| gobblin-rest-service | ||||
| gobblin-rest-api | ||||
| gobblin-rest-client | ||||
| gobblin-rest-server | ||||
| gobblin-restli | ||||
| gobblin-flow-config-service | ||||
| gobblin-restli-utils | ||||
| gobblin-throttling-service | ||||
| gobblin-runtime | ||||
| gobblin-runtime-hadoop | ||||
| gobblin-salesforce | gobblin-connectors | |||
| gobblin-service | Gobblin-deployment-modes | |||
| gobblin-test | ||||
| gobblin-test-harness | ||||
| gobblin-test-utils | ||||
| gobblin-tunnel | ||||
| gobblin-utility | ||||
| gobblin-yarn | Gobblin-deployment-modes | |||
Gobblin-deployment-modes | for all aws, yarn, MR, deployment modes modules |
...
Phase-2: code refactor to support module arrangement
- Motivation:
- while re-arranging modules, once figured out based on above table, we may have to refactor some code to better categorize within the module.
- Proposed Change:
- TBD
- New or Changed Public Interfaces: there should not be have any impact on compatibility.
- Migration Plan and Compatibility: TBD
- Rejected Alternatives: TBD