...
I (Bobby) envision this merger happening in two phases major, but we may blur the lines between these two as development happens.
Phase 1. Migrate clojure code to java using JStorm code as a reference
...
| Jira | ||||||
|---|---|---|---|---|---|---|
|
...
| Namespace | Dependency | JIRA | Notes | JStorm Similar Code | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| backtype.storm.LocalCluster | backtype.storm.config backtype.storm.testing |
| https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/LocalCluster.java | |||||||||||||||||||||
| backtype.storm.LocalDRPC | backtype.storm.config backtype.storm.daemon.drpc backtype.storm.util |
| https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/LocalDRPC.java | |||||||||||||||||||||
| backtype.storm.MockAutoCred | backtype.storm.config backtype.storm.testing |
| Mock implementation of INimbusCredentialPlugin,IAutoCredentials and ICredentialsRenewer for testing only. | NONE | ||||||||||||||||||||
| backtype.storm.clojure | backtype.storm.thrift backtype.storm.util | NONE |
| Clojure Storm API (Needs to Clojure Storm API (Needs to be kept for backwards compatabilitycompatibility, possibly in separate maven package) | NONE | |||||||||||||||||||
| backtype.storm.clojure-test | backtype.storm.clojure backtype.storm.config backtype.storm.daemon.common backtype.storm.testing backtype.storm.thrift |
| Tests for Storm Clojure API | NONE | ||||||||||||||||||||
| backtype.storm.cluster | backtype.storm.config backtype.storm.converter backtype.storm.daemon.common
backtype.storm.util backtype.storm.zookeeper |
| current state of the cluster (Some of this moves to java as a part of heartbeat server) | https://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/cluster | ||||||||||||||||||||
| backtype.storm.cluster-test | backtype.storm.cluster backtype.storm.config
backtype.storm.testing backtype.storm.thrift backtype.storm.util backtype.storm.zookeeper |
| Tests for backtype.storm.cluster | NONE? |
| Tests for backtype.storm.cluster | NONE? | |||||||||||||||||
| backtype.storm.backtype.storm.command.activate |
backtype.storm.thrift |
| Activate a topology | https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/command/activate.java | ||||||||||||||||||||
| backtype.storm.command.config-value | backtype.storm.config
|
| Read a config value | https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/command/config_value.java | ||||||||||||||||||||
| backtype.storm.command.deactivate |
backtype.storm.thrift |
| deactivate a topology |
| deactivate a topology | https://https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/command/deactivate.java | ||||||||||||||||||
| backtype.storm.command.dev-zookeeper | backtype.storm.config backtype.storm.util backtype.storm.zookeeper |
| Launch a development zookeeper instance | NONE | ||||||||||||||||||||
| backtype.storm.command.get-errors | backtype.storm.daemon.common backtype.storm.daemon.nimbus
backtype.storm.thrift backtype.storm.util |
| get recent errors from a topology | NONE | ||||||||||||||||||||
| backtype.storm.command.healthcheck | backtype.storm.config
| Check to see if this node appears to be healthy or not. | NONE |
| Check to see if this node appears to be healthy or not. | NONE | ||||||||||||||||||
| backtype.storm.command.kill-topology | backtype.storm.config
backtype.storm.thrift |
| Kill a running topology | https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/command/kill_topology.java | ||||||||||||||||||||
| backtype.storm.command.kill-workers | backtype.storm.config backtype.storm.daemon.common backtype.storm.daemon.supervisor backtype.storm.util |
| Kill all running worker processes on a node. | NONE | ||||||||||||||||||||
| backtype.storm.command.list |
backtype.storm.thrift |
| list running topologies |
| list running topologies | httpshttps://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/command/list.java (Implementation is quire different) | ||||||||||||||||||
| backtype.storm.command.monitor | backtype.storm.thrift |
| Monitor a running topology | https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/command/metrics_monitor.java (similar concept but implementation is very different) | ||||||||||||||||||||
backtype.storm.command.rebalance | backtype.storm.config
backtype.storm.thrift |
| rebalance a running topology | https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/command/rebalance.java (many different features from what apache storm currently supports) | ||||||||||||||||||||
| backtype.storm.command.set-log-level |
backtype.storm.thrift |
| set log levels for a running topology | NONE |
| set log levels for a running topology | NONE | |||||||||||||||||
| backtype.backtype.storm.command.shell-submission | backtype.storm.config
backtype.storm.thrift backtype.storm.util backtype.storm.zookeeper |
| Run clojure code passed in on the command line. we need to decide if we still want to support this or not. And if not remove it. | NONE | ||||||||||||||||||||
| backtype.storm.command.upload-credentials |
backtype.storm.util |
| Upload new credentials to a running topolgy | NONE | ||||||||||||||||||||
| backtype.storm.config |
backtype.storm.util |
| Some utils like functions (Part that makes storm config values into clojure constants needs to stay here and cannot be ported) | some in https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/utils/JStormUtils.java | ||||||||||||||||||||
| backtype.storm.converter | backtype.storm.daemon.common
backtype.storm.stats backtype.storm.util | NONE? | Convert between thrift and clojure types. When porting we should just use the Thrift type whenever possible. | NONE | ||||||||||||||||||||
| backtype.storm.daemon.acker | backtype.storm.config
backtype.storm.util | Track acks for a tuple tree |
| Track acks for a tuple tree | httpshttps://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/task/acker/Acker.java | |||||||||||||||||||
| backtype.storm.daemon.builtin-metrics | backtype.storm.stats |
| Built-in metrics | Very Different | ||||||||||||||||||||
| backtype.storm.daemon.common | backtype.storm.config backtype.storm.daemon.acker
backtype.storm.thrift backtype.storm.util |
| Common utils shared by the daemons (Some things should just use the Thrift object) | https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/cluster/Common.java | ||||||||||||||||||||
| backtype.storm.daemon.drpc | backtype.storm.config backtype.storm.daemon.common
backtype.storm.ui.helpers backtype.storm.util |
| DRPC server with HTTP and thrift support |
| DRPC server with HTTP and thrift support | (But missing HTTP support) | ||||||||||||||||||
| backtype.storm.daemon.executor | backtype.storm.cluster backtype.storm.config backtype.storm.daemon.builtin-metrics backtype.storm.daemon.common backtype.storm.daemon.task backtype.storm.disruptor
backtype.storm.stats backtype.storm.thrift backtype.storm.timer backtype.storm.tuple backtype.storm.util |
| Thread that runs tasks.
Also includes grouping code. | https://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/task
(But has a lot more with it too) | ||||||||||||||||||||
| backtype.storm.daemon.logviewer | backtype.storm.config backtype.storm.daemon.common backtype.storm.daemon.supervisor
backtype.storm.timer backtype.storm.ui.helpers backtype.storm.util |
| Display and manage log files. | UI Supports this so there is probably a supervisor API for this. But not really the same. and | ||||||||||||||||||||
| backtype.storm.daemon.backtype.storm.daemon.nimbus | backtype.storm.cluster backtype.storm.config backtype.storm.converter backtype.storm.daemon.common backtype.storm.local-state
backtype.storm.scheduler.DefaultScheduler backtype.storm.stats backtype.storm.timer backtype.storm.tuple backtype.storm.util backtype.storm.zookeeper | Its nimbus |
| Its nimbus | https://github.https://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/daemon/nimbus | |||||||||||||||||||
| backtype.storm.daemon.supervisor | backtype.storm.cluster backtype.storm.command.healthcheck backtype.storm.config backtype.storm.daemon.common backtype.storm.daemon.worker backtype.storm.event backtype.storm.local-state
backtype.storm.process-simulator backtype.storm.timer backtype.storm.util |
| launch and monitor workers | https://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/daemon/supervisor | ||||||||||||||||||||
| backtype.storm.daemon.task | backtype.storm.config backtype.storm.daemon.builtin-metrics backtype.storm.daemon.common
backtype.storm.stats backtype.storm.thrift backtype.storm.util |
| helper functions for task data and sending tuples | (does not have both executors and tasks) | ||||||||||||||||||||
| backtype.storm.daemon.worker | backtype.storm.cluster backtype.storm.config backtype.storm.daemon.common backtype.storm.daemon.executor backtype.storm.disruptor backtype.storm.local-state
backtype.storm.messaging.loader backtype.storm.timer backtype.storm.util |
| The worker process with multiple executors and tasks |
| The worker process with multiple executors and tasks | https://https://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/daemon/worker | ||||||||||||||||||
| backtype.storm.disruptor | backtype.storm.log backtype.storm.util NONE? |
| wrapper around the disruptor queue. Might need some abstract base classes if it makes since. | |||||||||||||||||||||
| backtype.storm.drpc-test |
backtype.storm.config backtype.storm.daemon.common backtype.storm.daemon.drpc backtype.storm.testing |
| Test DPRC | NONE | ||||||||||||||||||||
| backtype.storm.event |
backtype.storm.util | NONE? | Thread for processing events. Java has built in things to deal with this, do we need to create another one? | https://github.com/alibaba/jstorm/tree/master/jstorm-core/src/main/java/com/alibaba/jstorm/event | ||||||||||||||||||||
| backtype.storm backtype.storm.fields-test |
| Test java backtype.storm.tuple.Fields class | NONE | |||||||||||||||||||||
| backtype.storm.grouping-test |
backtype.storm.config backtype.storm.daemon.common backtype.storm.daemon.executor
backtype.storm.testing backtype.storm.thrift |
| Test groupings | NONE | ||||||||||||||||||||
| backtype.storm.integration-test |
backtype.storm.config backtype.storm.daemon.common backtype.storm.testing backtype.storm.thrift backtype.storm.util |
| Run some storm topologies |
| Run some storm topologies |
Not exactly the same but close | ||||||||||||||||||
backtype.storm.local-state |
backtype.storm.util |
| Wrapper around LocalState, with some helper functions for converting between storm and thrift | but not totally | ||||||||||||||||||||
| backtype.storm.local-state-test | backtype.storm.testing |
| test for LocalState class | |||||||||||||||||||||
| backtype.storm.log | NONE | Just use slf4j | ||||||||||||||||||||||
| backtype.storm.logviewer-test | backtype.storm.config backtype.storm.daemon.logviewer backtype.storm.daemon.supervisor backtype.storm.ui.helpers backtype.storm.util |
| test for LogViewer | NONE |
| test for LogViewer | NONE | |||||||||||||||||
| backtype.storm.messaging-test | backtype.storm.messaging-test | backtype.storm.config backtype.storm.daemon.common backtype.storm.testing backtype.storm.thrift |
| test for local messaging transport | NONE? | |||||||||||||||||||
| backtype.storm.messaging.loader | backtype.storm.messaging.localNONE? |
| Just a small bit of glue code for calling transfer-local-fn when messaging batch arrives | NONE | ||||||||||||||||||||
| backtype.storm.messaging.local | NONE |
| just a wrapper into backtype.just a wrapper into backtype.storm.messaging.local.Context | NONE | ||||||||||||||||||||
| backtype.storm.messaging.netty-integration-test | backtype.storm.config backtype.storm.testing backtype.storm.thrift backtype.storm.util |
| Integration tests for netty messaging layer | |||||||||||||||||||||
| backtype.storm.messaging.netty-unit-test | backtype.storm.config backtype.storm.daemon.worker
backtype.storm.testing backtype.storm.util |
| Unit tests for netty messaging layer | |||||||||||||||||||||
| backtype.storm.metric.testing |
| Creates a FakeMetriConsumer class, that records datapoints | NONE |
| Creates a FakeMetriConsumer class, that records datapoints | NONE | ||||||||||||||||||
| backtype.storm backtype.storm.metrics-test |
backtype.storm.config backtype.storm.daemon.common backtype.storm.metric.testing backtype.storm.testing backtype.storm.thrift |
| Test for the metrics system | NONE | ||||||||||||||||||||
| backtype.storm.multilang-test | backtype.storm.config backtype.storm.daemon.common backtype.storm.testing backtype.storm.thrift |
| Runs a few topolgies using the multi-lang feature. These should be moved to be a part of the multi-lang jars, and fixed to actually test something. | NONE | ||||||||||||||||||||
| backtype.storm.nimbus-test | backtype.storm.MockAutoCred backtype.storm.cluster backtype.storm.config backtype.storm.converter backtype.storm.daemon.common backtype.storm.daemon.nimbus
backtype.storm.stats backtype.storm.testing backtype.storm.thrift backtype.storm.timer backtype.storm.util backtype.storm.zookeeper |
| test that nimbus works | NONE? |
| test that nimbus works | NONE? | |||||||||||||||||
| backtype. backtype.storm.process-simulator |
backtype.storm.util |
| Simulate processes in local mode | ??? | ||||||||||||||||||||
| backtype.storm.scheduler-test | backtype.storm.config backtype.storm.daemon.nimbus backtype.storm.scheduler.EvenScheduler backtype.storm.testing |
| Test even scheduler | ??? | ||||||||||||||||||||
| backtype.storm.scheduler.DefaultScheduler | backtype.storm.config backtype.storm.scheduler.EvenScheduler backtype.storm.util | test DefaultScheduler, is this different from even scheduler??? |
| test DefaultScheduler, is this different from even scheduler??? | backtype.storm.scheduler.EvenScheduler | backtype.storm.config
backtype.storm.util | try to schedule processes evenly across the cluster. | https://github.com/apachealibaba/stormjstorm/blob/jstorm-importmaster/jstorm-core/src/main/java/com/alibaba/jstorm/schedule/default_assign/DefaultTopologyScheduler.java
maybebut not totally | ||||||||||||||||
| backtype.storm.scheduler.IsolationSchedulerEvenScheduler | backtype.storm.config
backtype.storm.scheduler.DefaultScheduler backtype.storm.util | Schedule Topologies in an isolated pool of machines, else schedule evenly. | util |
| try to schedule processes evenly across the cluster. |
maybe NONE? | ||||||||||||||||||
| backtype.storm.scheduler.multitenant-scheduler-testIsolationSchedulerzookeeper | backtype.storm.config
backtype.storm.logscheduler.DefaultScheduler backtype.storm.testing | Test Multi-tenant scheduler | util |
| Schedule Topologies in an isolated pool of machines, else schedule evenly. | yes,the scheduler is integrated into defaultScheduler.NONE | ||||||||||||||||||
| backtype.storm.scheduler.resourcemultitenant-aware-scheduler-test | backtype.storm.config backtype.storm.daemon.nimbus
backtype.storm.thrift backtype.storm.util | Test RAS | NONE | backtype.storm.security.auth.AuthUtils-test | testing |
| Test Multi-tenant schedulerTest AuthUtils | NONE | ||||||||||||||||
| backtype.storm.security.auth.DefaultHttpCredentialsPluginscheduler.resource-aware-scheduler-test | Test DefaultHttpCredentailsPlugin | NONE | backtype.storm.config backtype.storm.daemon.nimbus backtype.storm.testing backtype.storm.thrift backtype.storm.util |
| Test RAS | backtype.storm.security.auth.ReqContext-test | Test RequestContext | NONE | ||||||||||||||||
| backtype.storm.security.auth.SaslTransportPluginAuthUtils-test |
| Test SaslTransportPlugin |
| Test AuthUtils | NONE | |||||||||||||||||||
| backtype.storm.security.auth.ThriftClientDefaultHttpCredentialsPlugin-test | backtype.storm.config backtype.storm.util | Test secure ThriftClient | NONE |
| Test DefaultHttpCredentailsPlugin | backtype.storm.security.auth.ThriftServer-test | backtype.storm.config | Test secure ThriftServer | NONE | |||||||||||||||
| backtype.storm.security.auth.authReqContext-test | backtype.storm.config backtype.storm.daemon.common backtype.storm.daemon.nimbus backtype.storm.testing backtype.storm.util |
| Test RequestContextTest Authentication and Authorization | NONE | ||||||||||||||||||||
| backtype.storm.security.auth.authorizer.DRPCSimpleACLAuthorizerSaslTransportPlugin-test | backtype.storm.config backtype.storm.util |
| Test SaslTransportPluginTest DRPC Authorization | NONE | ||||||||||||||||||||
| backtype.storm.security.auth.auto-loginThriftClient-module-test | backtype.storm. util | config backtype.storm.util |
| Test secure ThriftClientTest AutoTGTKrb5LoginModule | NONE | |||||||||||||||||||
| backtype.storm.security.auth.drpcThriftServer-auth-test | backtype.storm.config backtype.storm.daemon.common backtype.storm.daemon.drpc
backtype.storm.testing backtype.storm.util |
| Test secure ThriftServerTest DRPC Authorization | NONE | ||||||||||||||||||||
| backtype.storm.security.auth.nimbus-auth-test | backtype.storm.clusterconfig backtype.storm.configbacktype.storm.daemon.common backtype.storm.daemon.nimbus backtype.storm.testing backtype.storm.util backtype.storm.zookeeper | Test Nimbus Authorization | NONE | backtype.storm.security.serialization.BlowfishTupleSerializer-test | backtype.storm.util |
| Test Authentication and AuthorizationTest BlowfishSerializer | NONE | ||||||||||||||||
| backtype.storm.serializationsecurity.auth.authorizer.DRPCSimpleACLAuthorizer-test | backtype.storm.config backtype.storm.util | Test KryoValuesSerializer | NONE | backtype.storm.serialization.SerializationFactory-test | backtype.storm.config | Test SerializationFactory |
| Test DRPC Authorization | NONENONE? | |||||||||||||||
| backtype.storm.statsbacktypesecurity.storm.logauth.auto-login-module-test | backtype.storm.util | clojure methods for getting built-in statistics |
| Test AutoTGTKrb5LoginModule | NONENONE? | |||||||||||||||||||
| backtype.storm.submitter.security.auth.drpc-auth-test | backtype.storm.config backtype.storm.daemon.testing | Test ZookeeperAuthentication payload generation that is a part of StormSubmitter | NONE | common backtype backtype.storm. subtopology-testdaemon.drpc
backtype.daemonstorm.commontesting backtype.storm.testing | NONE | util |
| Test DRPC AuthorizationJust remove commented out | NONE | |||||||||||||||
| backtype backtype.storm.supervisor.security.auth.nimbus-auth-test | backtype.storm.cluster backtype.storm.config backtype.storm.daemon.common backtype.storm.daemon.supervisornimbus
backtype.storm.testing backtype.storm.thriftutil backtype.storm.timer backtype.storm.util | Test the supervisor | NONE? | backtype.storm.testing | backtype.storm.cluster backtype.storm.config backtype.storm.daemon.acker backtype.storm.daemon.common backtype.storm.daemon.executor backtype.storm.daemon.nimbus backtype.storm.daemon.supervisor backtype.storm.daemon.worker backtype.storm.local-state
backtype.storm.messaging.loader backtype.storm.process-simulator backtype.storm.thrift backtype.storm.util backtype.storm.zookeeper | zookeeper |
| Test Nimbus Authorization | NONE | |||||||||||||||
| backtype.storm.security.serialization.BlowfishTupleSerializer-test | backtype.storm.util |
| Test BlowfishSerializer | NONE | ||||||||||||||||||||
| backtype.storm.serialization-test | backtype.storm.config backtype.storm.util |
| Test KryoValuesSerializer | NONE | ||||||||||||||||||||
| backtype.storm.serialization.SerializationFactory-test | backtype.storm.config |
| Test SerializationFactory | NONE? | ||||||||||||||||||||
| backtype.storm.stats |
backtype.storm.util |
| clojure methods for getting built-in statisticslots of helper functions/macros for running tests | NONE? | ||||||||||||||||||||
| backtype.storm.testing4jsubmitter-test | backtype.storm.LocalClusterconfig backtype.storm.logbacktype.storm.testingtesting |
| Test ZookeeperAuthentication payload generation that is a part of StormSubmitter | NONE | ||||||||||||||||||||
| backtype.storm.subtopology-test | backtype.storm.daemon.utilcommon | backtype.storm. Testing class for use from javatesting | NONE | Just remove commented out | NONE | |||||||||||||||||||
| backtype backtype.storm. testing4jsupervisor-test | backtype.storm.clojurecluster backtype.storm.config backtype.storm.daemon.common backtype.storm.daemon.supervisor backtype.storm.daemon.worker.integration-test backtype.storm.testing backtype.storm.thrift backtype.storm.util | tests for the Testing4j class | timer backtype.storm.util |
| Test the supervisor | NONE?NONE | ||||||||||||||||||
| backtype.storm.thrifttesting | backtype.storm.configcluster backtype.storm.logconfig backtype.storm.daemon.utilacker backtype.storm.zookeeper | helper methods for manipulating thrift objects | NONE | daemon.common backtype.storm.daemon.executor backtype.storm.tick-tuple-testbacktype.storm.daemon.clojurenimbus backtype.storm.daemon.configsupervisor backtype.storm.daemon.commonworker backtype.storm.testinglocal-state
| Run topology with system ticks | NONE? |
backtype.storm.messaging.loader backtype.storm.timerbacktype.storm.logprocess-simulator backtype.storm.util | Timer like class that uses simulated time | NONE (SimulatedTime does not appear to be used by their tests) | thrift backtype backtype.storm. transactional-testutil backtype.storm.clojurezookeeper |
| lots of helper functions/macros for running tests | NONE? | |||||||||||
| backtype.storm.testing4j | backtype.storm.configLocalCluster
backtype.storm.testing backtype.storm.util | Test transactional topologies | Just one test |
| backtype.storm.Testing class for use from java | NONE | ||||||||||||||||||
| backtype.storm.testing4j | backtype.storm.tuple | backtype.storm.tuple-test | NONE?
| small helper for getting hash code of a list |
backtype.storm. ui.coreconfig backtype.storm.configintegration-test backtype.storm.convertertesting backtype.storm.daemon.commonthrift backtype.storm.log backtype.storm.stats util |
| tests for the Testing4j class | NONE | ||||||||||||||||
| backtypebacktype.storm.thrift | backtype.storm.tupleconfig
backtype.storm.util backtype.storm.zookeeper |
| helper methods for manipulating thrift objects | NONE | ||||||||||||||||||||
| backtype.storm.tick-tuple-test | User Interface | Completely Different | backtype.storm.ui.helpers |
backtype.storm.logconfig backtype.storm.daemon.util | Helper functions for the UI/HTTP | Completely Different | common backtype.storm. utiltesting backtype.storm. logthrift |
| Run topology with system ticks | NONE? | ||||||||||||||
| backtype.storm.timer |
backtype.storm.util |
| Timer like class that uses simulated time | util functions | Possibly https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/utils/JStormUtils.java | backtype.storm.utils-test | backtype.storm.config backtype.storm.util | test some of utils | Possibly https://github.com/apachealibaba/stormjstorm/blob/jstorm-importmaster/jstorm-core/src/testmain/java/com/alibaba/jstorm/utilscallback/JStormUtilsTestAsyncLoopThread.java | |||||||||||||||
| backtype backtype.storm.utils.ZookeeperServerCnxnFactorytransactional-test | Test ZookeeperServerCnxnFactory | NONE | backtype
backtype.storm. testing | Test VersionedStore | NONE | config backtype.storm. worker-testbacktype.storm.daemon.common backtype.storm.daemon.workertesting backtype.storm.log backtype.storm.testing backtype.storm.util | Test Worker | NONE | ||||||||||||||||
| backtype.storm.zookeeper | backtype.storm.config
backtype.storm.util | Test Zookeeper | NONE | |||||||||||||||||||||
util |
| Test transactional topologies | Just one test | |||||||||||||||||||||
| backtype.storm.tuple |
|
| small helper for getting hash code of a list | NONE? | ||||||||||||||||||||
| backtype.storm.tuplestorm.trident.integration-test | backtype.storm.testing backtype.storm.util storm.trident.testing | Run trident topolgies | NONE? |
| storm.trident.state-test | |||||||||||||||||||
| backtype.storm.configui.core | backtype.storm.testingconfig backtype.storm.utilconverter storm.trident.testing | Test some of trident state | backtype.storm. tridentdaemon. testingcommon
backtype.storm.testingstats backtype.storm.util | helper functions for testing trident | thrift backtype.storm.tuple storm.trident.tuple-testbacktype.storm.ui.testinghelpers backtype.storm.util backtype.storm.trident.testing | test trident tuple |
The big things that change when moving from clojure is that we lose ring and hiccup for building the UI/REST services. We need to pick some alternatives for these. For the REST APIs Jersey is the most wide spread choice. hiccup is used in logviewer currently and we should look at replacing it either with static HTML + REST, like for the regular UI, or just return HTML we build ourselves and HTML escaping on parameters.
Phase 2. Port JStorm features to the new java based storm-core.
The following is an evaluation of major feature differences between storm and JStorm
zookeeper |
| User Interface | Completely Different | |||||||||
| backtype.storm.ui.helpers | backtype.storm.config
backtype.storm.util |
| Helper functions for the UI/HTTP | Completely Different | ||||||||
| backtype.storm.util |
| util functions | Possibly https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/utils/JStormUtils.java | |||||||||
| backtype.storm.utils-test | backtype.storm.config backtype.storm.util |
| test some of utils | Possibly https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/test/java/com/alibaba/jstorm/utils/JStormUtilsTest.java | ||||||||
| backtype.storm.utils.ZookeeperServerCnxnFactory-test |
| Test ZookeeperServerCnxnFactory | NONE | |||||||||
| backtype.storm.versioned-store-test | backtype.storm.testing |
| Test VersionedStore | NONE | ||||||||
| backtype.storm.worker-test | backtype.storm.daemon.common backtype.storm.daemon.worker
backtype.storm.testing backtype.storm.util |
| Test Worker | NONE | ||||||||
| backtype.storm.zookeeper | backtype.storm.config
backtype.storm.util |
| Test Zookeeper | https://github.com/alibaba/jstorm/blob/master/jstorm-core/src/main/java/com/alibaba/jstorm/zk/Zookeeper.java | ||||||||
| storm.trident.integration-test | backtype.storm.testing backtype.storm.util storm.trident.testing |
| Run trident topolgies | NONE? | ||||||||
| storm.trident.state-test | backtype.storm.config backtype.storm.testing backtype.storm.util storm.trident.testing |
| Test some of trident state | |||||||||
| storm.trident.testing | backtype.storm.LocalDRPC backtype.storm.testing backtype.storm.util |
| helper functions for testing trident | |||||||||
| storm.trident.tuple-test | backtype.storm.testing backtype.storm.util storm.trident.testing |
| test trident tuple |
The big things that change when moving from clojure is that we lose ring and hiccup for building the UI/REST services. We need to pick some alternatives for these. For the REST APIs Jersey is the most wide spread choice. hiccup is used in logviewer currently and we should look at replacing it either with static HTML + REST, like for the regular UI, or just return HTML we build ourselves and HTML escaping on parameters.
Phase 2. Port JStorm features to the new java based storm-core.
| Jira | ||||||
|---|---|---|---|---|---|---|
|
The following is an evaluation of major feature differences between storm and JStorm
| Feature | Apache Storm 0.11.0-SNAPSHOT (and expected pull requests) | JStorm-2.1.0/JStorm2.0.4 | Notes | JIRA to port JStorm feature | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Scheduler | Even, Default, Isolation, Multi-tenant, RAS(work in progress) |
| The scheduler interface is pluggable, so we should be able to support both schedulers if needed. |
| ||||||||
| Nimbus HA | Support for a pool of nimbus servers. Once Blobstore is merged in, "leader election" and "state storage" will be separate. | Support for configuring more than one backup nimbus. When the master nimbus is down, the most appropriate spare nimbus (topologies on disk most closely match the records in ZooKeeper) will be chosen to be promoted. | Need to evaluate the strengths and weaknesses of each and decide on updates to storm if any. |
| ||||||||
| Topology Structure | worker 1->* executor 1->* task | worker 1-> task | Need to evaluate if removing executors will add enough benefit to developers/performance that we can drop it from architecture. Probably need resource aware re-balancing or Jstorm rebalancing that can support changing parallelism before this can happen. |
| ||||||||
| Topology Master | None exactly. The Heartbeat Server (Currently under review) reduces load on ZK, but it is not really the same and may be a complement to TopologyMaster | New system bolt "topology master" was added, which is responsible for collecting task heartbeat info of all tasks and reporting the info to nimbus. Besides task heartbeat info, it also can be used to dispatch control messages within the topology. Topology master significantly reduces the amount of read/write to ZooKeeper. Before this change, ZooKeeper was the bottleneck for deploying big clusters and topologies. | Need to evaluate how this impacts storm architecture especially around security. |
| ||||||||
| Backpressure | Bang-Bang controller on Disruptor queue capacity using ZooKeeper for broadcast. |
|
| |||||||||
| Monitoring of task execute thread | Potential Pull Request, but none right now. | Monitors the status of the execute thread of tasks. It is effective to find the slow bolt in a topology, and potentially uncovers deadlock as well. | Yes we should do this |
| ||||||||
| Message processing | Deserialization happens on the netty thread Serialization happens after the send queue when batching is happening. |
| The two sound equivalent now, but we should talk to see if there are other optimizations needed. |
| ||||||||
| Tuple Batching | Batching in DisruptorQueue | Do batch before sending tuple to transfer queue and support for adjusting the batch size dynamically according to samples of actual batch size sent out for past intervals. | Should evaluate both implementations, and see which is better for performance, and possible if we can/should move some of the dynamic batching logic into disruptor. |
| ||||||||
| Grouping | Load aware balancing in shuffle grouping |
| We should look at combining both of these to have shuffle look at both distance and load to decide where to send a tuple. |
| ||||||||
| web-UI | Yes | Different | Does someone know a good UI designer that we can use? I don't really like either of them (Bobby) but that is just me |
| ||||||||
| metric system | IMetric and IMetricConsumer |
| Ideally we should have a way to display most/system metrics in the UI. IMetric is too generic to make this happen, but we cannot completely drop support for it. But perhaps we need to depricate it if the JStorm metrics are much better. |
| ||||||||
| rebalance command | Basic functionality of rebalance | Besides rebalance, scale-out/in by updating the number of workers, ackers, spouts & bolts dynamically without stopping topology. Routing is updated dynamically within upstream components. | dynamic routing with some groupings is difficult to get right when there is state, we need to be sure this is well documented, and might want to disallow it for some groupings without a force flag. |
| ||||||||
| list command | List information of topologies | List information of all topologies, all supervisors, and JStorm version | more info is good, but we want it human readable too. perhaps with a machine readable option too |
| ||||||||
| zktool command | N/A | Supports some ZooKeeper operations, e.g. "list", "read"… | Will need to be evaluated for security |
| ||||||||
| metricsMonitor command | N/A | Allows toggling on/off some metrics which may impact topology performance | Sounds great |
| ||||||||
| restart command | N/A | Restart a topology. Besides restart, this command can also update the topology configuration. | Sounds great |
| ||||||||
| update_topology command | N/A | Update jars and configuration dynamically for a topology, without stopping the topology. | Sounds great, should work will with blob-storm |
| ||||||||
| cgroup | N/A | Supports controlling the upper limit of CPU core usage for a worker using cgroups | Sounds like a good start, will be nice to integrate it with RAS requests too. |
| ||||||||
| logging |
|
| Need to evaluate differences and see what we want long term |
| ||||||||
| Worker classloader (isolation) | N/A - uses shading for most dependencies | The "worker classloader" avoids problem of re-loading classes | This sounds great |
| ||||||||
| Multi-thread of spout | NA | There are two modes of spout, "single-thread" and "multi-thread" in JStorm. The "single-thread" mode is simliar to Storm while the "multi-thread" mode separates the processing of ack/fail and nextTuple to two threads. It means we can stay in nextTuple for a long time without any side effect on ack/fail. | This improves the response and throughput of spout |
| ||||||||
| Feature | Apache Storm 0.11.0-SNAPSHOT (and expected pull requests) | JStorm-2.1.0/JStorm2.0.4 | Notes | JIRA to port JStorm feature | ||||||||
| Security |
| N/A | NONE | |||||||||
| Scheduler | Even, Default, Isolation, Multi-tenant, RAS(work in progress) |
| The scheduler interface is pluggable, so we should be able to support both schedulers if needed. | |||||||||
| Nimbus HA | Support for a pool of nimbus servers. Once Blobstore is merged in, "leader election" and "state storage" will be separate. | Support for configuring more than one backup nimbus. When the master nimbus is down, the most appropriate spare nimbus (topologies on disk most closely match the records in ZooKeeper) will be chosen to be promoted. | Need to evaluate the strengths and weaknesses of each and decide on updates to storm if any. | |||||||||
| Topology Structure | worker 1->* executor 1->* task | worker 1-> task | Need to evaluate if removing executors will add enough benefit to developers/performance that we can drop it from architecture. Probably need resource aware re-balancing or Jstorm rebalancing that can support changing parallelism before this can happen. | |||||||||
| Topology Master | None exactly. The Heartbeat Server (Currently under review) reduces load on ZK, but it is not really the same and may be a complement to TopologyMaster | New system bolt "topology master" was added, which is responsible for collecting task heartbeat info of all tasks and reporting the info to nimbus. Besides task heartbeat info, it also can be used to dispatch control messages within the topology. Topology master significantly reduces the amount of read/write to ZooKeeper. Before this change, ZooKeeper was the bottleneck for deploying big clusters and topologies. | Need to evaluate how this impacts storm architecture especially around security. | |||||||||
| Backpressure | Bang-Bang controller on Disruptor queue capacity using ZooKeeper for broadcast. |
| ||||||||||
| Monitoring of task execute thread | Potential Pull Request, but none right now. | Monitors the status of the execute thread of tasks. It is effective to find the slow bolt in a topology, and potentially uncovers deadlock as well. | Yes we should do this | |||||||||
| Message processing | Deserialization happens on the netty thread Serialization happens after the send queue when batching is happening. |
| The two sound equivalent now, but we should talk to see if there are other optimizations needed. | |||||||||
| Tuple Batching | Batching in DisruptorQueue | Do batch before sending tuple to transfer queue and support for adjusting the batch size dynamically according to samples of actual batch size sent out for past intervals. | Should evaluate both implementations, and see which is better for performance, and possible if we can/should move some of the dynamic batching logic into disruptor. | |||||||||
| Grouping | Load aware balancing in shuffle grouping |
| We should look at combining both of these to have shuffle look at both distance and load to decide where to send a tuple. | |||||||||
| REST API | Yes | N/A | NONE | |||||||||
| web-UI | Yes | Different | Does someone know a good UI designer that we can use? I don't really like either of them (Bobby) but that is just me | |||||||||
| metric system | IMetric and IMetricConsumer |
| Ideally we should have a way to display most/system metrics in the UI. IMetric is too generic to make this happen, but we cannot completely drop support for it. But perhaps we need to depricate it if the JStorm metrics are much better. | |||||||||
| rebalance command | Basic functionality of rebalance | Besides rebalance, scale-out/in by updating the number of workers, ackers, spouts & bolts dynamically without stopping topology. Routing is updated dynamically within upstream components. | dynamic routing with some groupings is difficult to get right when there is state, we need to be sure this is well documented, and might want to disallow it for some groupings without a force flag. | |||||||||
| list command | List information of topologies | List information of all topologies, all supervisors, and JStorm version | more info is good, but we want it human readable too. perhaps with a machine readable option too | |||||||||
| zktool command | N/A | Supports some ZooKeeper operations, e.g. "list", "read"… | Will need to be evaluated for security | |||||||||
| metricsMonitor command | monitor is similar, less metrics and read-only | Allows toggling on/off some metrics which may impact topology performance | Sounds great | |||||||||
| restart command | N/A | Restart a topology. Besides restart, this command can also update the topology configuration. | Sounds great | |||||||||
| update_topology command | N/A | Update jars and configuration dynamically for a topology, without stopping the topology. | Sounds great, should work will with blob-storm | |||||||||
| cgroup | N/A | Supports controlling the upper limit of CPU core usage for a worker using cgroups | Sounds like a good start, will be nice to integrate it with RAS requests too. | |||||||||
| logging |
|
| Need to evaluate differences and see what we want long term | |||||||||
| Worker classloader (isolation) | N/A - uses shading for most dependencies | The "worker classloader" avoids problem of re-loading classes | This sounds great | |