The ultimate goal here is to merge the Storm and JStorm code bases and communities. The current proposal is to do a feature freeze on storm-core while this work happens, but everything under external should still be open for development.
I (Bobby) envision this merger happening in two phases major, but we may blur the lines between these two as development happens.
Phase 1. Migrate clojure code to java using JStorm code as a reference
- STORM-1225Getting issue details... STATUS
Namespace | Dependency | JIRA | Notes | JStorm Similar Code |
---|---|---|---|---|
backtype.storm.LocalCluster | backtype.storm.config backtype.storm.testing | - STORM-1282Getting issue details... STATUS | https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/LocalCluster.java | |
backtype.storm.LocalDRPC | backtype.storm.config backtype.storm.daemon.drpc backtype.storm.util | - STORM-1274Getting issue details... STATUS | https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/LocalDRPC.java | |
backtype.storm.MockAutoCred | backtype.storm.config backtype.storm.testing | - STORM-1283Getting issue details... STATUS | Mock implementation of INimbusCredentialPlugin,IAutoCredentials and ICredentialsRenewer for testing only. | NONE |
backtype.storm.clojure | backtype.storm.thrift backtype.storm.util | - STORM-1259Getting issue details... STATUS | Clojure Storm API (Needs to be kept for backwards compatibility, possibly in separate maven package) | NONE |
backtype.storm.clojure-test | backtype.storm.clojure backtype.storm.config backtype.storm.daemon.common backtype.storm.testing backtype.storm.thrift | Tests for Storm Clojure API | NONE | |
backtype.storm.cluster | backtype.storm.config backtype.storm.converter backtype.storm.daemon.common
backtype.storm.util backtype.storm.zookeeper | - STORM-1273Getting issue details... STATUS | current state of the cluster (Some of this moves to java as a part of heartbeat server) | https://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/cluster |
backtype.storm.cluster-test | backtype.storm.cluster backtype.storm.config
backtype.storm.testing backtype.storm.thrift backtype.storm.util backtype.storm.zookeeper | - STORM-1284Getting issue details... STATUS | Tests for backtype.storm.cluster | NONE? |
backtype.storm.command.activate |
backtype.storm.thrift | - STORM-1260Getting issue details... STATUS | Activate a topology | https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/command/activate.java |
backtype.storm.command.config-value | backtype.storm.config
| - STORM-1242Getting issue details... STATUS | Read a config value | https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/command/config_value.java |
backtype.storm.command.deactivate |
backtype.storm.thrift | - STORM-1261Getting issue details... STATUS | deactivate a topology | https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/command/deactivate.java |
backtype.storm.command.dev-zookeeper | backtype.storm.config backtype.storm.util backtype.storm.zookeeper | - STORM-1262Getting issue details... STATUS | Launch a development zookeeper instance | NONE |
backtype.storm.command.get-errors | backtype.storm.daemon.common backtype.storm.daemon.nimbus
backtype.storm.thrift backtype.storm.util | - STORM-1285Getting issue details... STATUS | get recent errors from a topology | NONE |
backtype.storm.command.healthcheck | backtype.storm.config
| - STORM-1243Getting issue details... STATUS | Check to see if this node appears to be healthy or not. | NONE |
backtype.storm.command.kill-topology | backtype.storm.config
backtype.storm.thrift | - STORM-1263Getting issue details... STATUS | Kill a running topology | https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/command/kill_topology.java |
backtype.storm.command.kill-workers | backtype.storm.config backtype.storm.daemon.common backtype.storm.daemon.supervisor backtype.storm.util | - STORM-1286Getting issue details... STATUS | Kill all running worker processes on a node. | NONE |
backtype.storm.command.list |
backtype.storm.thrift | - STORM-1264Getting issue details... STATUS | list running topologies | https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/command/list.java (Implementation is quire different) |
backtype.storm.command.monitor | backtype.storm.thrift | - STORM-1265Getting issue details... STATUS | Monitor a running topology | https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/command/metrics_monitor.java (similar concept but implementation is very different) |
backtype.storm.command.rebalance | backtype.storm.config
backtype.storm.thrift | - STORM-1266Getting issue details... STATUS | rebalance a running topology | https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/command/rebalance.java (many different features from what apache storm currently supports) |
backtype.storm.command.set-log-level |
backtype.storm.thrift | - STORM-1267Getting issue details... STATUS | set log levels for a running topology | NONE |
backtype.storm.command.shell-submission | backtype.storm.config
backtype.storm.thrift backtype.storm.util backtype.storm.zookeeper | - STORM-1318Getting issue details... STATUS | Run clojure code passed in on the command line. we need to decide if we still want to support this or not. And if not remove it. | NONE |
backtype.storm.command.upload-credentials |
backtype.storm.util | - STORM-1244Getting issue details... STATUS | Upload new credentials to a running topolgy | NONE |
backtype.storm.config |
backtype.storm.util | - STORM-1227Getting issue details... STATUS | Some utils like functions (Part that makes storm config values into clojure constants needs to stay here and cannot be ported) | some in https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/utils/JStormUtils.java |
backtype.storm.converter | backtype.storm.daemon.common
backtype.storm.stats backtype.storm.util | NONE | Convert between thrift and clojure types. When porting we should just use the Thrift type whenever possible. | NONE |
backtype.storm.daemon.acker | backtype.storm.config
backtype.storm.util | - STORM-1245Getting issue details... STATUS | Track acks for a tuple tree | https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/task/acker/Acker.java |
backtype.storm.daemon.builtin-metrics | backtype.storm.stats | - STORM-1268Getting issue details... STATUS | Built-in metrics | Very Different |
backtype.storm.daemon.common | backtype.storm.config backtype.storm.daemon.acker
backtype.storm.thrift backtype.storm.util | - STORM-1269Getting issue details... STATUS | Common utils shared by the daemons (Some things should just use the Thrift object) | https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/cluster/Common.java |
backtype.storm.daemon.drpc | backtype.storm.config backtype.storm.daemon.common
backtype.storm.ui.helpers backtype.storm.util | - STORM-1270Getting issue details... STATUS | DRPC server with HTTP and thrift support | (But missing HTTP support) |
backtype.storm.daemon.executor | backtype.storm.cluster backtype.storm.config backtype.storm.daemon.builtin-metrics backtype.storm.daemon.common backtype.storm.daemon.task backtype.storm.disruptor
backtype.storm.stats backtype.storm.thrift backtype.storm.timer backtype.storm.tuple backtype.storm.util | - STORM-1277Getting issue details... STATUS | Thread that runs tasks.
Also includes grouping code. | https://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/task
(But has a lot more with it too) |
backtype.storm.daemon.logviewer | backtype.storm.config backtype.storm.daemon.common backtype.storm.daemon.supervisor
backtype.storm.timer backtype.storm.ui.helpers backtype.storm.util | - STORM-1280Getting issue details... STATUS | Display and manage log files. | UI Supports this so there is probably a supervisor API for this. But not really the same. |
backtype.storm.daemon.nimbus | backtype.storm.cluster backtype.storm.config backtype.storm.converter backtype.storm.daemon.common backtype.storm.local-state
backtype.storm.scheduler.DefaultScheduler backtype.storm.stats backtype.storm.timer backtype.storm.tuple backtype.storm.util backtype.storm.zookeeper | - STORM-1276Getting issue details... STATUS | Its nimbus | https://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/daemon/nimbus |
backtype.storm.daemon.supervisor | backtype.storm.cluster backtype.storm.command.healthcheck backtype.storm.config backtype.storm.daemon.common backtype.storm.daemon.worker backtype.storm.event backtype.storm.local-state
backtype.storm.process-simulator backtype.storm.timer backtype.storm.util | - STORM-1279Getting issue details... STATUS | launch and monitor workers | https://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/daemon/supervisor |
backtype.storm.daemon.task | backtype.storm.config backtype.storm.daemon.builtin-metrics backtype.storm.daemon.common
backtype.storm.stats backtype.storm.thrift backtype.storm.util | - STORM-1271Getting issue details... STATUS | helper functions for task data and sending tuples | (does not have both executors and tasks) |
backtype.storm.daemon.worker | backtype.storm.cluster backtype.storm.config backtype.storm.daemon.common backtype.storm.daemon.executor backtype.storm.disruptor backtype.storm.local-state
backtype.storm.messaging.loader backtype.storm.timer backtype.storm.util | - STORM-1278Getting issue details... STATUS | The worker process with multiple executors and tasks | https://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/daemon/worker |
backtype.storm.disruptor | backtype.storm.log backtype.storm.util | - STORM-1272Getting issue details... STATUS | wrapper around the disruptor queue. Might need some abstract base classes if it makes since. | |
backtype.storm.drpc-test |
backtype.storm.config backtype.storm.daemon.common backtype.storm.daemon.drpc backtype.storm.testing | - STORM-1287Getting issue details... STATUS | Test DPRC | NONE |
backtype.storm.event |
backtype.storm.util | NONE | Thread for processing events. Java has built in things to deal with this, do we need to create another one? | https://github.com/alibaba/jstorm/tree/master/jstorm-core/src/main/java/com/alibaba/jstorm/event |
backtype.storm.fields-test | - STORM-1228Getting issue details... STATUS | Test java backtype.storm.tuple.Fields class | NONE | |
backtype.storm.grouping-test |
backtype.storm.config backtype.storm.daemon.common backtype.storm.daemon.executor
backtype.storm.testing backtype.storm.thrift | - STORM-1288Getting issue details... STATUS | Test groupings | NONE |
backtype.storm.integration-test |
backtype.storm.config backtype.storm.daemon.common backtype.storm.testing backtype.storm.thrift backtype.storm.util | - STORM-1289Getting issue details... STATUS | Run some storm topologies |
Not exactly the same but close |
backtype.storm.local-state |
backtype.storm.util | - STORM-1246Getting issue details... STATUS | Wrapper around LocalState, with some helper functions for converting between storm and thrift | but not totally |
backtype.storm.local-state-test | backtype.storm.testing | - STORM-1290Getting issue details... STATUS | test for LocalState class | |
backtype.storm.log | NONE | Just use slf4j | ||
backtype.storm.logviewer-test | backtype.storm.config backtype.storm.daemon.logviewer backtype.storm.daemon.supervisor backtype.storm.ui.helpers backtype.storm.util | - STORM-1291Getting issue details... STATUS | test for LogViewer | NONE |
backtype.storm.messaging-test | backtype.storm.config backtype.storm.daemon.common backtype.storm.testing backtype.storm.thrift | - STORM-1292Getting issue details... STATUS | test for local messaging transport | NONE? |
backtype.storm.messaging.loader | backtype.storm.messaging.local | - STORM-1248Getting issue details... STATUS | Just a small bit of glue code for calling transfer-local-fn when messaging batch arrives | NONE |
backtype.storm.messaging.local | - STORM-1247Getting issue details... STATUS | just a wrapper into backtype.storm.messaging.local.Context | NONE | |
backtype.storm.messaging.netty-integration-test | backtype.storm.config backtype.storm.testing backtype.storm.thrift backtype.storm.util | - STORM-1293Getting issue details... STATUS | Integration tests for netty messaging layer | |
backtype.storm.messaging.netty-unit-test | backtype.storm.config backtype.storm.daemon.worker
backtype.storm.testing backtype.storm.util | - STORM-1294Getting issue details... STATUS | Unit tests for netty messaging layer | |
backtype.storm.metric.testing | - STORM-1229Getting issue details... STATUS | Creates a FakeMetriConsumer class, that records datapoints | NONE | |
backtype.storm.metrics-test |
backtype.storm.config backtype.storm.daemon.common backtype.storm.metric.testing backtype.storm.testing backtype.storm.thrift | - STORM-1295Getting issue details... STATUS | Test for the metrics system | NONE |
backtype.storm.multilang-test | backtype.storm.config backtype.storm.daemon.common backtype.storm.testing backtype.storm.thrift | - STORM-1296Getting issue details... STATUS | Runs a few topolgies using the multi-lang feature. These should be moved to be a part of the multi-lang jars, and fixed to actually test something. | NONE |
backtype.storm.nimbus-test | backtype.storm.MockAutoCred backtype.storm.cluster backtype.storm.config backtype.storm.converter backtype.storm.daemon.common backtype.storm.daemon.nimbus
backtype.storm.stats backtype.storm.testing backtype.storm.thrift backtype.storm.timer backtype.storm.util backtype.storm.zookeeper | - STORM-1297Getting issue details... STATUS | test that nimbus works | NONE? |
backtype.storm.process-simulator |
backtype.storm.util | - STORM-1230Getting issue details... STATUS | Simulate processes in local mode | ??? |
backtype.storm.scheduler-test | backtype.storm.config backtype.storm.daemon.nimbus backtype.storm.scheduler.EvenScheduler backtype.storm.testing | - STORM-1298Getting issue details... STATUS
| Test even scheduler | ??? |
backtype.storm.scheduler.DefaultScheduler | backtype.storm.config backtype.storm.scheduler.EvenScheduler backtype.storm.util | - STORM-1232Getting issue details... STATUS | test DefaultScheduler, is this different from even scheduler??? | |
backtype.storm.scheduler.EvenScheduler | backtype.storm.config
backtype.storm.util | - STORM-1231Getting issue details... STATUS | try to schedule processes evenly across the cluster. |
maybe |
backtype.storm.scheduler.IsolationScheduler | backtype.storm.config
backtype.storm.scheduler.DefaultScheduler backtype.storm.util | - STORM-1239Getting issue details... STATUS | Schedule Topologies in an isolated pool of machines, else schedule evenly. | NONE? |
backtype.storm.scheduler.multitenant-scheduler-test | backtype.storm.config backtype.storm.daemon.nimbus
backtype.storm.testing | - STORM-1299Getting issue details... STATUS | Test Multi-tenant scheduler | NONE |
backtype.storm.scheduler.resource-aware-scheduler-test | backtype.storm.config backtype.storm.daemon.nimbus backtype.storm.testing backtype.storm.thrift backtype.storm.util | - STORM-1300Getting issue details... STATUS | Test RAS | NONE |
backtype.storm.security.auth.AuthUtils-test | - STORM-1233Getting issue details... STATUS | Test AuthUtils | NONE | |
backtype.storm.security.auth.DefaultHttpCredentialsPlugin-test | - STORM-1234Getting issue details... STATUS | Test DefaultHttpCredentailsPlugin | NONE | |
backtype.storm.security.auth.ReqContext-test | - STORM-1235Getting issue details... STATUS | Test RequestContext | NONE | |
backtype.storm.security.auth.SaslTransportPlugin-test | - STORM-1236Getting issue details... STATUS | Test SaslTransportPlugin | NONE | |
backtype.storm.security.auth.ThriftClient-test | backtype.storm.config backtype.storm.util | - STORM-1237Getting issue details... STATUS | Test secure ThriftClient | NONE |
backtype.storm.security.auth.ThriftServer-test | backtype.storm.config | - STORM-1238Getting issue details... STATUS | Test secure ThriftServer | NONE |
backtype.storm.security.auth.auth-test | backtype.storm.config backtype.storm.daemon.common backtype.storm.daemon.nimbus backtype.storm.testing backtype.storm.util | - STORM-1301Getting issue details... STATUS | Test Authentication and Authorization | NONE |
backtype.storm.security.auth.authorizer.DRPCSimpleACLAuthorizer-test | backtype.storm.config backtype.storm.util | - STORM-1240Getting issue details... STATUS | Test DRPC Authorization | NONE |
backtype.storm.security.auth.auto-login-module-test | backtype.storm.util | - STORM-1241Getting issue details... STATUS | Test AutoTGTKrb5LoginModule | NONE |
backtype.storm.security.auth.drpc-auth-test | backtype.storm.config backtype.storm.daemon.common backtype.storm.daemon.drpc
backtype.storm.testing backtype.storm.util | - STORM-1302Getting issue details... STATUS | Test DRPC Authorization | NONE |
backtype.storm.security.auth.nimbus-auth-test | backtype.storm.cluster backtype.storm.config backtype.storm.daemon.common backtype.storm.daemon.nimbus
backtype.storm.testing backtype.storm.util backtype.storm.zookeeper | - STORM-1303Getting issue details... STATUS | Test Nimbus Authorization | NONE |
backtype.storm.security.serialization.BlowfishTupleSerializer-test | backtype.storm.util | - STORM-1249Getting issue details... STATUS | Test BlowfishSerializer | NONE |
backtype.storm.serialization-test | backtype.storm.config backtype.storm.util | - STORM-1250Getting issue details... STATUS | Test KryoValuesSerializer | NONE |
backtype.storm.serialization.SerializationFactory-test | backtype.storm.config | - STORM-1251Getting issue details... STATUS | Test SerializationFactory | NONE? |
backtype.storm.stats |
backtype.storm.util | - STORM-1252Getting issue details... STATUS | clojure methods for getting built-in statistics | NONE? |
backtype.storm.submitter-test | backtype.storm.config backtype.storm.testing | - STORM-1304Getting issue details... STATUS | Test ZookeeperAuthentication payload generation that is a part of StormSubmitter | NONE |
backtype.storm.subtopology-test | backtype.storm.daemon.common backtype.storm.testing | NONE | Just remove commented out | NONE |
backtype.storm.supervisor-test | backtype.storm.cluster backtype.storm.config backtype.storm.daemon.common backtype.storm.daemon.supervisor backtype.storm.daemon.worker backtype.storm.testing backtype.storm.thrift backtype.storm.timer backtype.storm.util | - STORM-1305Getting issue details... STATUS | Test the supervisor | NONE? |
backtype.storm.testing | backtype.storm.cluster backtype.storm.config backtype.storm.daemon.acker backtype.storm.daemon.common backtype.storm.daemon.executor backtype.storm.daemon.nimbus backtype.storm.daemon.supervisor backtype.storm.daemon.worker backtype.storm.local-state
backtype.storm.messaging.loader backtype.storm.process-simulator backtype.storm.thrift backtype.storm.util backtype.storm.zookeeper | - STORM-1281Getting issue details... STATUS | lots of helper functions/macros for running tests | NONE? |
backtype.storm.testing4j | backtype.storm.LocalCluster
backtype.storm.testing backtype.storm.util | - STORM-1306Getting issue details... STATUS | backtype.storm.Testing class for use from java | NONE |
backtype.storm.testing4j-test |
backtype.storm.config backtype.storm.integration-test backtype.storm.testing backtype.storm.thrift backtype.storm.util | - STORM-1307Getting issue details... STATUS | tests for the Testing4j class | NONE |
backtype.storm.thrift | backtype.storm.config
backtype.storm.util backtype.storm.zookeeper | - STORM-1258Getting issue details... STATUS | helper methods for manipulating thrift objects | NONE |
backtype.storm.tick-tuple-test |
backtype.storm.config backtype.storm.daemon.common backtype.storm.testing backtype.storm.thrift | - STORM-1308Getting issue details... STATUS | Run topology with system ticks | NONE? |
backtype.storm.timer |
backtype.storm.util | - STORM-1253Getting issue details... STATUS | Timer like class that uses simulated time | https://github.com/alibaba/jstorm/blob/master/jstorm-core/src/main/java/com/alibaba/jstorm/callback/AsyncLoopThread.java |
backtype.storm.transactional-test |
backtype.storm.config backtype.storm.daemon.common backtype.storm.testing backtype.storm.util | - STORM-1309Getting issue details... STATUS | Test transactional topologies | Just one test |
backtype.storm.tuple |
| - STORM-1275Getting issue details... STATUS | small helper for getting hash code of a list | NONE? |
backtype.storm.tuple-test | backtype.storm.testing | - STORM-1310Getting issue details... STATUS | ||
backtype.storm.ui.core | backtype.storm.config backtype.storm.converter backtype.storm.daemon.common
backtype.storm.stats backtype.storm.thrift backtype.storm.tuple backtype.storm.ui.helpers backtype.storm.util backtype.storm.zookeeper | - STORM-1311Getting issue details... STATUS | User Interface | Completely Different |
backtype.storm.ui.helpers | backtype.storm.config
backtype.storm.util | - STORM-1254Getting issue details... STATUS | Helper functions for the UI/HTTP | Completely Different |
backtype.storm.util | - STORM-1226Getting issue details... STATUS | util functions | Possibly https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/utils/JStormUtils.java | |
backtype.storm.utils-test | backtype.storm.config backtype.storm.util | - STORM-1255Getting issue details... STATUS | test some of utils | Possibly https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/test/java/com/alibaba/jstorm/utils/JStormUtilsTest.java |
backtype.storm.utils.ZookeeperServerCnxnFactory-test | - STORM-1256Getting issue details... STATUS | Test ZookeeperServerCnxnFactory | NONE | |
backtype.storm.versioned-store-test | backtype.storm.testing | - STORM-1313Getting issue details... STATUS | Test VersionedStore | NONE |
backtype.storm.worker-test | backtype.storm.daemon.common backtype.storm.daemon.worker
backtype.storm.testing backtype.storm.util | - STORM-1312Getting issue details... STATUS | Test Worker | NONE |
backtype.storm.zookeeper | backtype.storm.config
backtype.storm.util | - STORM-1257Getting issue details... STATUS | Test Zookeeper | NONE |
storm.trident.integration-test | backtype.storm.testing backtype.storm.util storm.trident.testing | - STORM-1315Getting issue details... STATUS | Run trident topolgies | NONE? |
storm.trident.state-test | backtype.storm.config backtype.storm.testing backtype.storm.util storm.trident.testing | - STORM-1316Getting issue details... STATUS | Test some of trident state | |
storm.trident.testing | backtype.storm.LocalDRPC backtype.storm.testing backtype.storm.util | - STORM-1314Getting issue details... STATUS | helper functions for testing trident | |
storm.trident.tuple-test | backtype.storm.testing backtype.storm.util storm.trident.testing | - STORM-1317Getting issue details... STATUS | test trident tuple |
The big things that change when moving from clojure is that we lose ring and hiccup for building the UI/REST services. We need to pick some alternatives for these. For the REST APIs Jersey is the most wide spread choice. hiccup is used in logviewer currently and we should look at replacing it either with static HTML + REST, like for the regular UI, or just return HTML we build ourselves and HTML escaping on parameters.
Phase 2. Port JStorm features to the new java based storm-core.
- STORM-1319Getting issue details... STATUS
The following is an evaluation of major feature differences between storm and JStorm