The ultimate goal here is to merge the Storm and JStorm code bases and communities. The current proposal is to do a feature freeze on storm-core while this work happens, but everything under external should still be open for development.
I (Bobby) envision this merger happening in two phases major, but we may blur the lines between these two as development happens.
Phase 1. Migrate clojure code to java using JStorm code as a reference.
| Namespace | Dependency | JIRA | Notes | JStorm Similar Code |
|---|---|---|---|---|
| backtype.storm.LocalCluster | backtype.storm.config backtype.storm.testing | https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/LocalCluster.java | ||
| backtype.storm.LocalDRPC | backtype.storm.config backtype.storm.daemon.drpc backtype.storm.util | https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/LocalDRPC.java | ||
| backtype.storm.MockAutoCred | backtype.storm.config backtype.storm.testing | Mock implementation of INimbusCredentialPlugin,IAutoCredentials and ICredentialsRenewer for testing only. | NONE | |
| backtype.storm.clojure | backtype.storm.thrift backtype.storm.util | NONE | Clojure Storm API (Needs to be kept for backwards compatability, possibly in separate maven package) | NONE |
| backtype.storm.clojure-test | backtype.storm.clojure backtype.storm.config backtype.storm.daemon.common backtype.storm.testing backtype.storm.thrift | NONE | Tests for Storm Clojure API | NONE |
| backtype.storm.cluster | backtype.storm.config backtype.storm.converter backtype.storm.daemon.common backtype.storm.log backtype.storm.util backtype.storm.zookeeper | current state of the cluster (Some of this moves to java as a part of heartbeat server) | https://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/cluster | |
| backtype.storm.cluster-test | backtype.storm.cluster backtype.storm.config backtype.storm.log backtype.storm.testing backtype.storm.thrift backtype.storm.util backtype.storm.zookeeper | Tests for backtype.storm.cluster | NONE? | |
| backtype.storm.command.activate | backtype.storm.log backtype.storm.thrift | Activate a topology | https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/command/activate.java | |
| backtype.storm.command.config-value | backtype.storm.config backtype.storm.log | Read a config value | https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/command/config_value.java | |
| backtype.storm.command.deactivate | backtype.storm.log backtype.storm.thrift | deactivate a topology | https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/command/deactivate.java | |
| backtype.storm.command.dev-zookeeper | backtype.storm.config backtype.storm.util backtype.storm.zookeeper | Launch a development zookeeper instance | NONE | |
| backtype.storm.command.get-errors | backtype.storm.daemon.common backtype.storm.daemon.nimbus backtype.storm.log backtype.storm.thrift backtype.storm.util | get recent errors from a topology | NONE | |
| backtype.storm.command.healthcheck | backtype.storm.config backtype.storm.log | Check to see if this node appears to be healthy or not. | NONE | |
| backtype.storm.command.kill-topology | backtype.storm.config backtype.storm.log backtype.storm.thrift | Kill a running topology | https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/command/kill_topology.java | |
| backtype.storm.command.kill-workers | backtype.storm.config backtype.storm.daemon.common backtype.storm.daemon.supervisor backtype.storm.util | Kill all running worker processes on a node. | NONE | |
| backtype.storm.command.list | backtype.storm.log backtype.storm.thrift | list running topologies | https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/command/list.java (Implementation is quire different) | |
| backtype.storm.command.monitor | backtype.storm.thrift | Monitor a running topology | https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/command/metrics_monitor.java (similar concept but implementation is very different) | |
backtype.storm.command.rebalance | backtype.storm.config backtype.storm.log backtype.storm.thrift | rebalance a running topology | https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/command/rebalance.java (many different features from what apache storm currently supports) | |
| backtype.storm.command.set-log-level | backtype.storm.log backtype.storm.thrift | set log levels for a running topology | NONE | |
| backtype.storm.command.shell-submission | backtype.storm.config backtype.storm.log backtype.storm.thrift backtype.storm.util backtype.storm.zookeeper | NONE??? | Run clojure code passed in on the command line. we need to decide if we still want to support this or not. And if not remove it. | NONE |
| backtype.storm.command.upload-credentials | backtype.storm.log backtype.storm.util | Upload new credentials to a running topolgy | NONE | |
| backtype.storm.config | backtype.storm.log backtype.storm.util | Some utils like functions (Part that makes storm config values into clojure constants needs to stay here and cannot be ported) | some in https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/utils/JStormUtils.java | |
| backtype.storm.converter | backtype.storm.daemon.common backtype.storm.log backtype.storm.stats backtype.storm.util | NONE? | Convert between thrift and clojure types. When porting we should just use the Thrift type whenever possible. | NONE |
| backtype.storm.daemon.acker | backtype.storm.config backtype.storm.log backtype.storm.util | Track acks for a tuple tree | https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/task/acker/Acker.java | |
| backtype.storm.daemon.builtin-metrics | backtype.storm.stats | Built-in metrics | Very Different | |
| backtype.storm.daemon.common | backtype.storm.config backtype.storm.daemon.acker backtype.storm.log backtype.storm.thrift backtype.storm.util | Common utils shared by the daemons (Some things should just use the Thrift object) | https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/cluster/Common.java | |
| backtype.storm.daemon.drpc | backtype.storm.config backtype.storm.daemon.common backtype.storm.log backtype.storm.ui.helpers backtype.storm.util | DRPC server with HTTP and thrift support | (But missing HTTP support) | |
| backtype.storm.daemon.executor | backtype.storm.cluster backtype.storm.config backtype.storm.daemon.builtin-metrics backtype.storm.daemon.common backtype.storm.daemon.task backtype.storm.disruptor backtype.storm.log backtype.storm.stats backtype.storm.thrift backtype.storm.timer backtype.storm.tuple backtype.storm.util | Thread that runs tasks.
Also includes grouping code. | https://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/task
(But has a lot more with it too) | |
| backtype.storm.daemon.logviewer | backtype.storm.config backtype.storm.daemon.common backtype.storm.daemon.supervisor backtype.storm.log backtype.storm.timer backtype.storm.ui.helpers backtype.storm.util | Display and manage log files. | UI Supports this so there is probably a supervisor API for this. But not really the same. | |
| backtype.storm.daemon.nimbus | backtype.storm.cluster backtype.storm.config backtype.storm.converter backtype.storm.daemon.common backtype.storm.local-state backtype.storm.log backtype.storm.scheduler.DefaultScheduler backtype.storm.stats backtype.storm.timer backtype.storm.tuple backtype.storm.util backtype.storm.zookeeper | Its nimbus | https://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/daemon/nimbus | |
| backtype.storm.daemon.supervisor | backtype.storm.cluster backtype.storm.command.healthcheck backtype.storm.config backtype.storm.daemon.common backtype.storm.daemon.worker backtype.storm.event backtype.storm.local-state backtype.storm.log backtype.storm.process-simulator backtype.storm.timer backtype.storm.util | launch and monitor workers | https://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/daemon/supervisor | |
| backtype.storm.daemon.task | backtype.storm.config backtype.storm.daemon.builtin-metrics backtype.storm.daemon.common backtype.storm.log backtype.storm.stats backtype.storm.thrift backtype.storm.util | helper functions for task data and sending tuples | (does not have both executors and tasks) | |
| backtype.storm.daemon.worker | backtype.storm.cluster backtype.storm.config backtype.storm.daemon.common backtype.storm.daemon.executor backtype.storm.disruptor backtype.storm.local-state backtype.storm.log backtype.storm.messaging.loader backtype.storm.timer backtype.storm.util | The worker process with multiple executors and tasks | https://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/daemon/worker | |
| backtype.storm.disruptor | backtype.storm.log backtype.storm.util | NONE? | wrapper around the disruptor queue. Might need some abstract base classes if it makes since. | |
| backtype.storm.drpc-test | backtype.storm.clojure backtype.storm.config backtype.storm.daemon.common backtype.storm.daemon.drpc backtype.storm.testing | Test DPRC | NONE | |
| backtype.storm.event | backtype.storm.log backtype.storm.util | NONE? | Thread for processing events. Java has built in things to deal with this, do we need to create another one? | |
| backtype.storm.fields-test | Test java backtype.storm.tuple.Fields class | NONE | ||
| backtype.storm.grouping-test | backtype.storm.clojure backtype.storm.config backtype.storm.daemon.common backtype.storm.daemon.executor backtype.storm.log backtype.storm.testing backtype.storm.thrift | Test groupings | NONE | |
| backtype.storm.integration-test | backtype.storm.clojure backtype.storm.config backtype.storm.daemon.common backtype.storm.testing backtype.storm.thrift backtype.storm.util | Run some storm topologies |
Not exactly the same but close | |
backtype.storm.local-state | backtype.storm.log backtype.storm.util | Wrapper around LocalState, with some helper functions for converting between storm and thrift | but not totally | |
| backtype.storm.local-state-test | backtype.storm.testing | test for LocalState class | ||
| backtype.storm.log | NONE | Just use slf4j | ||
| backtype.storm.logviewer-test | backtype.storm.config backtype.storm.daemon.logviewer backtype.storm.daemon.supervisor backtype.storm.ui.helpers backtype.storm.util | test for LogViewer | NONE | |
| backtype.storm.messaging-test | backtype.storm.config backtype.storm.daemon.common backtype.storm.testing backtype.storm.thrift | test for local messaging transport | NONE? | |
| backtype.storm.messaging.loader | backtype.storm.messaging.local | NONE? | Just a small bit of glue code for calling transfer-local-fn when messaging batch arrives | NONE |
| backtype.storm.messaging.local | NONE | just a wrapper into backtype.storm.messaging.local.Context | NONE | |
| backtype.storm.messaging.netty-integration-test | backtype.storm.config backtype.storm.testing backtype.storm.thrift backtype.storm.util | Integration tests for netty messaging layer | ||
| backtype.storm.messaging.netty-unit-test | backtype.storm.config backtype.storm.daemon.worker backtype.storm.log backtype.storm.testing backtype.storm.util | Unit tests for netty messaging layer | ||
| backtype.storm.metric.testing | Creates a FakeMetriConsumer class, that records datapoints | NONE | ||
| backtype.storm.metrics-test | backtype.storm.clojure backtype.storm.config backtype.storm.daemon.common backtype.storm.metric.testing backtype.storm.testing backtype.storm.thrift | Test for the metrics system | NONE | |
| backtype.storm.multilang-test | backtype.storm.config backtype.storm.daemon.common backtype.storm.testing backtype.storm.thrift | Runs a few topolgies using the multi-lang feature. These should be moved to be a part of the multi-lang jars, and fixed to actually test something. | NONE | |
| backtype.storm.nimbus-test | backtype.storm.MockAutoCred backtype.storm.cluster backtype.storm.config backtype.storm.converter backtype.storm.daemon.common backtype.storm.daemon.nimbus backtype.storm.log backtype.storm.stats backtype.storm.testing backtype.storm.thrift backtype.storm.timer backtype.storm.util backtype.storm.zookeeper | test that nimbus works | NONE? | |
| backtype.storm.process-simulator | backtype.storm.log backtype.storm.util | Simulate processes in local mode | ??? | |
| backtype.storm.scheduler-test | backtype.storm.config backtype.storm.daemon.nimbus backtype.storm.scheduler.EvenScheduler backtype.storm.testing | Test even scheduler | ??? | |
| backtype.storm.scheduler.DefaultScheduler | backtype.storm.config backtype.storm.scheduler.EvenScheduler backtype.storm.util | test DefaultScheduler, is this different from even scheduler??? | ||
| backtype.storm.scheduler.EvenScheduler | backtype.storm.config backtype.storm.log backtype.storm.util | try to schedule processes evenly across the cluster. |
maybe | |
| backtype.storm.scheduler.IsolationScheduler | backtype.storm.config backtype.storm.log backtype.storm.scheduler.DefaultScheduler backtype.storm.util | Schedule Topologies in an isolated pool of machines, else schedule evenly. | NONE? | |
| backtype.storm.scheduler.multitenant-scheduler-test | backtype.storm.config backtype.storm.daemon.nimbus backtype.storm.log backtype.storm.testing | Test Multi-tenant scheduler | NONE | |
| backtype.storm.scheduler.resource-aware-scheduler-test | backtype.storm.config backtype.storm.daemon.nimbus backtype.storm.testing backtype.storm.thrift backtype.storm.util | Test RAS | NONE | |
| backtype.storm.security.auth.AuthUtils-test | Test AuthUtils | NONE | ||
| backtype.storm.security.auth.DefaultHttpCredentialsPlugin-test | Test DefaultHttpCredentailsPlugin | NONE | ||
| backtype.storm.security.auth.ReqContext-test | Test RequestContext | NONE | ||
| backtype.storm.security.auth.SaslTransportPlugin-test | Test SaslTransportPlugin | NONE | ||
| backtype.storm.security.auth.ThriftClient-test | backtype.storm.config backtype.storm.util | Test secure ThriftClient | NONE | |
| backtype.storm.security.auth.ThriftServer-test | backtype.storm.config | Test secure ThriftServer | NONE | |
| backtype.storm.security.auth.auth-test | backtype.storm.config backtype.storm.daemon.common backtype.storm.daemon.nimbus backtype.storm.testing backtype.storm.util | Test Authentication and Authorization | NONE | |
| backtype.storm.security.auth.authorizer.DRPCSimpleACLAuthorizer-test | backtype.storm.config backtype.storm.util | Test DRPC Authorization | NONE | |
| backtype.storm.security.auth.auto-login-module-test | backtype.storm.util | Test AutoTGTKrb5LoginModule | NONE | |
| backtype.storm.security.auth.drpc-auth-test | backtype.storm.config backtype.storm.daemon.common backtype.storm.daemon.drpc backtype.storm.log backtype.storm.testing backtype.storm.util | Test DRPC Authorization | NONE | |
| backtype.storm.security.auth.nimbus-auth-test | backtype.storm.cluster backtype.storm.config backtype.storm.daemon.common backtype.storm.daemon.nimbus backtype.storm.log backtype.storm.testing backtype.storm.util backtype.storm.zookeeper | Test Nimbus Authorization | NONE | |
| backtype.storm.security.serialization.BlowfishTupleSerializer-test | backtype.storm.util | Test BlowfishSerializer | NONE | |
| backtype.storm.serialization-test | backtype.storm.config backtype.storm.util | Test KryoValuesSerializer | NONE | |
| backtype.storm.serialization.SerializationFactory-test | backtype.storm.config | Test SerializationFactory | NONE? | |
| backtype.storm.stats | backtype.storm.log backtype.storm.util | clojure methods for getting built-in statistics | NONE? | |
| backtype.storm.submitter-test | backtype.storm.config backtype.storm.testing | Test ZookeeperAuthentication payload generation that is a part of StormSubmitter | NONE | |
| backtype.storm.subtopology-test | backtype.storm.daemon.common backtype.storm.testing | NONE | Just remove commented out | NONE |
backtype.storm.supervisor-test | backtype.storm.cluster backtype.storm.config backtype.storm.daemon.common backtype.storm.daemon.supervisor backtype.storm.daemon.worker backtype.storm.testing backtype.storm.thrift backtype.storm.timer backtype.storm.util | Test the supervisor | NONE? | |
| backtype.storm.testing | backtype.storm.cluster backtype.storm.config backtype.storm.daemon.acker backtype.storm.daemon.common backtype.storm.daemon.executor backtype.storm.daemon.nimbus backtype.storm.daemon.supervisor backtype.storm.daemon.worker backtype.storm.local-state backtype.storm.log backtype.storm.messaging.loader backtype.storm.process-simulator backtype.storm.thrift backtype.storm.util backtype.storm.zookeeper | lots of helper functions/macros for running tests | NONE? | |
| backtype.storm.testing4j | backtype.storm.LocalCluster backtype.storm.log backtype.storm.testing backtype.storm.util | backtype.storm.Testing class for use from java | NONE | |
| backtype.storm.testing4j-test | backtype.storm.clojure backtype.storm.config backtype.storm.integration-test backtype.storm.testing backtype.storm.thrift backtype.storm.util | tests for the Testing4j class | NONE | |
| backtype.storm.thrift | backtype.storm.config backtype.storm.log backtype.storm.util backtype.storm.zookeeper | helper methods for manipulating thrift objects | NONE | |
| backtype.storm.tick-tuple-test | backtype.storm.clojure backtype.storm.config backtype.storm.daemon.common backtype.storm.testing backtype.storm.thrift | Run topology with system ticks | NONE? | |
| backtype.storm.timer | backtype.storm.log backtype.storm.util | Timer like class that uses simulated time | NONE (SimulatedTime does not appear to be used by their tests) | |
| backtype.storm.transactional-test | backtype.storm.clojure backtype.storm.config backtype.storm.daemon.common backtype.storm.testing backtype.storm.util | Test transactional topologies | Just one test | |
| backtype.storm.tuple | backtype.storm.tuple-test backtype.storm.testing | small helper for getting hash code of a list | NONE? | |
| backtype.storm.ui.core | backtype.storm.config backtype.storm.converter backtype.storm.daemon.common backtype.storm.log backtype.storm.stats backtype.storm.thrift backtype.storm.tuple backtype.storm.ui.helpers backtype.storm.util backtype.storm.zookeeper | User Interface | Completely Different | |
| backtype.storm.ui.helpers | backtype.storm.config backtype.storm.log backtype.storm.util | Helper functions for the UI/HTTP | Completely Different | |
| backtype.storm.util | backtype.storm.log | util functions | Possibly https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/utils/JStormUtils.java | |
| backtype.storm.utils-test | backtype.storm.config backtype.storm.util | test some of utils | Possibly https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/test/java/com/alibaba/jstorm/utils/JStormUtilsTest.java | |
| backtype.storm.utils.ZookeeperServerCnxnFactory-test | Test ZookeeperServerCnxnFactory | NONE | ||
| backtype.storm.versioned-store-test | backtype.storm.testing | Test VersionedStore | NONE | |
| backtype.storm.worker-test | backtype.storm.daemon.common backtype.storm.daemon.worker backtype.storm.log backtype.storm.testing backtype.storm.util | Test Worker | NONE | |
| backtype.storm.zookeeper | backtype.storm.config backtype.storm.log backtype.storm.util | Test Zookeeper | NONE | |
| storm.trident.integration-test | backtype.storm.testing backtype.storm.util storm.trident.testing | Run trident topolgies | NONE? | |
| storm.trident.state-test | backtype.storm.config backtype.storm.testing backtype.storm.util storm.trident.testing | Test some of trident state | ||
| storm.trident.testing | backtype.storm.LocalDRPC backtype.storm.testing backtype.storm.util | helper functions for testing trident | ||
| storm.trident.tuple-test | backtype.storm.testing backtype.storm.util storm.trident.testing | test trident tuple |
Phase 2. Port JStorm features to the new java based storm-core.