This Confluence has been LDAP enabled, if you are an ASF Committer, please use your LDAP Credentials to login. Any problems file an INFRA jira ticket please.

Page tree
Skip to end of metadata
Go to start of metadata

The ultimate goal here is to merge the Storm and JStorm code bases and communities.  The current proposal is to do a feature freeze on storm-core while this work happens, but everything under external should still be open for development.

 

I (Bobby) envision this merger happening in two phases major, but we may blur the lines between these two as development happens.

 

Phase 1. Migrate clojure code to java using JStorm code as a reference

STORM-1225 - Getting issue details... STATUS

NamespaceDependencyJIRANotesJStorm Similar Code
backtype.storm.LocalCluster

backtype.storm.config

backtype.storm.testing

STORM-1282 - Getting issue details... STATUS  https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/LocalCluster.java
backtype.storm.LocalDRPC

backtype.storm.config

backtype.storm.daemon.drpc

backtype.storm.util

STORM-1274 - Getting issue details... STATUS  https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/LocalDRPC.java
backtype.storm.MockAutoCred

backtype.storm.config

backtype.storm.testing

STORM-1283 - Getting issue details... STATUS Mock implementation of INimbusCredentialPlugin,IAutoCredentials and ICredentialsRenewer for testing only.NONE
backtype.storm.clojure

backtype.storm.thrift

backtype.storm.util

STORM-1259 - Getting issue details... STATUS Clojure Storm API (Needs to be kept for backwards compatibility, possibly in separate maven package)NONE
backtype.storm.clojure-test

backtype.storm.clojure

backtype.storm.config

backtype.storm.daemon.common

backtype.storm.testing

backtype.storm.thrift

STORM-1259 - Getting issue details... STATUS

Tests for Storm Clojure APINONE
backtype.storm.cluster

backtype.storm.config

backtype.storm.converter

backtype.storm.daemon.common

backtype.storm.log

backtype.storm.util

backtype.storm.zookeeper

STORM-1273 - Getting issue details... STATUS current state of the cluster (Some of this moves to java as a part of heartbeat server)https://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/cluster
backtype.storm.cluster-test

backtype.storm.cluster

backtype.storm.config

backtype.storm.log

backtype.storm.testing

backtype.storm.thrift

backtype.storm.util

backtype.storm.zookeeper

STORM-1284 - Getting issue details... STATUS Tests for backtype.storm.clusterNONE?
backtype.storm.command.activate

backtype.storm.log

backtype.storm.thrift

STORM-1260 - Getting issue details... STATUS Activate a topologyhttps://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/command/activate.java
backtype.storm.command.config-value

backtype.storm.config

backtype.storm.log

STORM-1242 - Getting issue details... STATUS Read a config valuehttps://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/command/config_value.java
backtype.storm.command.deactivate

backtype.storm.log

backtype.storm.thrift

STORM-1261 - Getting issue details... STATUS deactivate a topologyhttps://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/command/deactivate.java
backtype.storm.command.dev-zookeeper

backtype.storm.config

backtype.storm.util

backtype.storm.zookeeper

STORM-1262 - Getting issue details... STATUS Launch a development zookeeper instanceNONE
backtype.storm.command.get-errors

backtype.storm.daemon.common

backtype.storm.daemon.nimbus

backtype.storm.log

backtype.storm.thrift

backtype.storm.util

STORM-1285 - Getting issue details... STATUS get recent errors from a topologyNONE
backtype.storm.command.healthcheck

backtype.storm.config

backtype.storm.log

STORM-1243 - Getting issue details... STATUS Check to see if this node appears to be healthy or not.NONE
backtype.storm.command.kill-topology

backtype.storm.config

backtype.storm.log

backtype.storm.thrift

STORM-1263 - Getting issue details... STATUS Kill a running topologyhttps://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/command/kill_topology.java
backtype.storm.command.kill-workers

backtype.storm.config

backtype.storm.daemon.common

backtype.storm.daemon.supervisor

backtype.storm.util

STORM-1286 - Getting issue details... STATUS Kill all running worker processes on a node.NONE
backtype.storm.command.list

backtype.storm.log

backtype.storm.thrift

STORM-1264 - Getting issue details... STATUS list running topologieshttps://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/command/list.java (Implementation is quire different)
backtype.storm.command.monitorbacktype.storm.thrift STORM-1265 - Getting issue details... STATUS Monitor a running topologyhttps://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/command/metrics_monitor.java (similar concept but implementation is very different)

backtype.storm.command.rebalance

backtype.storm.config

backtype.storm.log

backtype.storm.thrift

STORM-1266 - Getting issue details... STATUS rebalance a running topologyhttps://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/command/rebalance.java (many different features from what apache storm currently supports)
backtype.storm.command.set-log-level

backtype.storm.log

backtype.storm.thrift

STORM-1267 - Getting issue details... STATUS set log levels for a running topologyNONE
backtype.storm.command.shell-submission

backtype.storm.config

backtype.storm.log

backtype.storm.thrift

backtype.storm.util

backtype.storm.zookeeper

STORM-1318 - Getting issue details... STATUS Run clojure code passed in on the command line.  we need to decide if we still want to support this or not.  And if not remove it.NONE
backtype.storm.command.upload-credentials

backtype.storm.log

backtype.storm.util

STORM-1244 - Getting issue details... STATUS Upload new credentials to a running topolgyNONE
backtype.storm.config

backtype.storm.log

backtype.storm.util

STORM-1227 - Getting issue details... STATUS Some utils like functions (Part that makes storm config values into clojure constants needs to stay here and cannot be ported)some in https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/utils/JStormUtils.java
backtype.storm.converter

backtype.storm.daemon.common

backtype.storm.log

backtype.storm.stats

backtype.storm.util

NONEConvert between thrift and clojure types.  When porting we should just use the Thrift type whenever possible.NONE
backtype.storm.daemon.acker

backtype.storm.config

backtype.storm.log

backtype.storm.util

STORM-1245 - Getting issue details... STATUS Track acks for a tuple treehttps://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/task/acker/Acker.java
backtype.storm.daemon.builtin-metricsbacktype.storm.stats STORM-1268 - Getting issue details... STATUS Built-in metricsVery Different
backtype.storm.daemon.common

backtype.storm.config

backtype.storm.daemon.acker

backtype.storm.log

backtype.storm.thrift

backtype.storm.util

STORM-1269 - Getting issue details... STATUS Common utils shared by the daemons (Some things should just use the Thrift object)https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/cluster/Common.java
backtype.storm.daemon.drpc

backtype.storm.config

backtype.storm.daemon.common

backtype.storm.log

backtype.storm.ui.helpers

backtype.storm.util

STORM-1270 - Getting issue details... STATUS

DRPC server with HTTP and thrift support

https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/drpc/Drpc.java

(But missing HTTP support)

backtype.storm.daemon.executor

backtype.storm.cluster

backtype.storm.config

backtype.storm.daemon.builtin-metrics

backtype.storm.daemon.common

backtype.storm.daemon.task

backtype.storm.disruptor

backtype.storm.log

backtype.storm.stats

backtype.storm.thrift

backtype.storm.timer

backtype.storm.tuple

backtype.storm.util

STORM-1277 - Getting issue details... STATUS

Thread that runs tasks.

 

Also includes grouping code.

https://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/task

 

(But has a lot more with it too)

backtype.storm.daemon.logviewer

backtype.storm.config

backtype.storm.daemon.common

backtype.storm.daemon.supervisor

backtype.storm.log

backtype.storm.timer

backtype.storm.ui.helpers

backtype.storm.util

STORM-1280 - Getting issue details... STATUS Display and manage log files.

UI Supports this so there is probably a supervisor API for this.  But not really the same.

https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/daemon/supervisor/Httpserver.java

and

https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/utils/HttpserverUtils.java

backtype.storm.daemon.nimbus

backtype.storm.cluster

backtype.storm.config

backtype.storm.converter

backtype.storm.daemon.common

backtype.storm.local-state

backtype.storm.log

backtype.storm.scheduler.DefaultScheduler

backtype.storm.stats

backtype.storm.timer

backtype.storm.tuple

backtype.storm.util

backtype.storm.zookeeper

STORM-1276 - Getting issue details... STATUS Its nimbushttps://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/daemon/nimbus
backtype.storm.daemon.supervisor

backtype.storm.cluster

backtype.storm.command.healthcheck

backtype.storm.config

backtype.storm.daemon.common

backtype.storm.daemon.worker

backtype.storm.event

backtype.storm.local-state

backtype.storm.log

backtype.storm.process-simulator

backtype.storm.timer

backtype.storm.util

STORM-1279 - Getting issue details... STATUS launch and monitor workershttps://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/daemon/supervisor
backtype.storm.daemon.task

backtype.storm.config

backtype.storm.daemon.builtin-metrics

backtype.storm.daemon.common

backtype.storm.log

backtype.storm.stats

backtype.storm.thrift

backtype.storm.util

STORM-1271 - Getting issue details... STATUS helper functions for task data and sending tuples(does not have both executors and tasks)
backtype.storm.daemon.worker

backtype.storm.cluster

backtype.storm.config

backtype.storm.daemon.common

backtype.storm.daemon.executor

backtype.storm.disruptor

backtype.storm.local-state

backtype.storm.log

backtype.storm.messaging.loader

backtype.storm.timer

backtype.storm.util

STORM-1278 - Getting issue details... STATUS The worker process with multiple executors and taskshttps://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/daemon/worker
backtype.storm.disruptor

backtype.storm.log

backtype.storm.util

STORM-1272 - Getting issue details... STATUS

wrapper around the disruptor queue. Might need some abstract base classes if it makes since.

 
backtype.storm.drpc-test

backtype.storm.clojure

backtype.storm.config

backtype.storm.daemon.common

backtype.storm.daemon.drpc

backtype.storm.testing

STORM-1287 - Getting issue details... STATUS Test DPRCNONE
 

backtype.storm.event

backtype.storm.log

backtype.storm.util

NONEThread for processing events.  Java has built in things to deal with this, do we need to create another one?https://github.com/alibaba/jstorm/tree/master/jstorm-core/src/main/java/com/alibaba/jstorm/event
 backtype.storm.fields-test  STORM-1228 - Getting issue details... STATUS Test java backtype.storm.tuple.Fields classNONE
 backtype.storm.grouping-test

backtype.storm.clojure

backtype.storm.config

backtype.storm.daemon.common

backtype.storm.daemon.executor

backtype.storm.log

backtype.storm.testing

backtype.storm.thrift

STORM-1288 - Getting issue details... STATUS Test groupingsNONE
 backtype.storm.integration-test

backtype.storm.clojure

backtype.storm.config

backtype.storm.daemon.common

backtype.storm.testing

backtype.storm.thrift

backtype.storm.util

STORM-1289 - Getting issue details... STATUS Run some storm topologies

https://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/test/java/com/alibaba/jstorm/topology

 

Not exactly the same but close

backtype.storm.local-state

backtype.storm.log

backtype.storm.util

STORM-1246 - Getting issue details... STATUS Wrapper around LocalState, with some helper functions for converting between storm and thrift

https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/backtype/storm/generated/LocalStateData.java

but not totally

 backtype.storm.local-state-test backtype.storm.testing STORM-1290 - Getting issue details... STATUS test for LocalState class 
 backtype.storm.log NONEJust use slf4j 
 backtype.storm.logviewer-test

backtype.storm.config

backtype.storm.daemon.logviewer

backtype.storm.daemon.supervisor

backtype.storm.ui.helpers

backtype.storm.util

STORM-1291 - Getting issue details... STATUS test for LogViewerNONE
 backtype.storm.messaging-test

backtype.storm.config

backtype.storm.daemon.common

backtype.storm.testing

backtype.storm.thrift

STORM-1292 - Getting issue details... STATUS test for local messaging transportNONE?
 backtype.storm.messaging.loader backtype.storm.messaging.local STORM-1248 - Getting issue details... STATUS Just a small bit of glue code for calling transfer-local-fn when messaging batch arrivesNONE
 backtype.storm.messaging.local  STORM-1247 - Getting issue details... STATUS just a wrapper into backtype.storm.messaging.local.ContextNONE
 backtype.storm.messaging.netty-integration-test

backtype.storm.config

backtype.storm.testing

backtype.storm.thrift

backtype.storm.util

STORM-1293 - Getting issue details... STATUS Integration tests for netty messaging layer 
 backtype.storm.messaging.netty-unit-test

backtype.storm.config

backtype.storm.daemon.worker

backtype.storm.log

backtype.storm.testing

backtype.storm.util

STORM-1294 - Getting issue details... STATUS Unit tests for netty messaging layer 
 backtype.storm.metric.testing  STORM-1229 - Getting issue details... STATUS Creates a FakeMetriConsumer class, that records datapointsNONE
 backtype.storm.metrics-test

backtype.storm.clojure

backtype.storm.config

backtype.storm.daemon.common

backtype.storm.metric.testing

backtype.storm.testing

backtype.storm.thrift

STORM-1295 - Getting issue details... STATUS Test for the metrics systemNONE
 backtype.storm.multilang-test

backtype.storm.config

backtype.storm.daemon.common

backtype.storm.testing

backtype.storm.thrift

STORM-1296 - Getting issue details... STATUS Runs a few topolgies using the multi-lang feature.  These should be moved to be a part of the multi-lang jars, and fixed to actually test something.NONE
 backtype.storm.nimbus-test

backtype.storm.MockAutoCred

backtype.storm.cluster

backtype.storm.config

backtype.storm.converter

backtype.storm.daemon.common

backtype.storm.daemon.nimbus

backtype.storm.log

backtype.storm.stats

backtype.storm.testing

backtype.storm.thrift

backtype.storm.timer

backtype.storm.util

backtype.storm.zookeeper

STORM-1297 - Getting issue details... STATUS test that nimbus worksNONE?
 backtype.storm.process-simulator

backtype.storm.log

backtype.storm.util

STORM-1230 - Getting issue details... STATUS Simulate processes in local mode???
 backtype.storm.scheduler-test

backtype.storm.config

backtype.storm.daemon.nimbus

backtype.storm.scheduler.EvenScheduler

backtype.storm.testing

STORM-1298 - Getting issue details... STATUS

 

Test even scheduler???
 backtype.storm.scheduler.DefaultScheduler

backtype.storm.config

backtype.storm.scheduler.EvenScheduler

backtype.storm.util

STORM-1232 - Getting issue details... STATUS test DefaultScheduler, is this different from even scheduler???

https://github.com/alibaba/jstorm/blob/master/jstorm-core/src/main/java/com/alibaba/jstorm/schedule/default_assign/DefaultTopologyScheduler.java

but not totally

 backtype.storm.scheduler.EvenScheduler

backtype.storm.config

backtype.storm.log

backtype.storm.util

STORM-1231 - Getting issue details... STATUS try to schedule processes evenly across the cluster.

https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/schedule/default_assign/DefaultTopologyScheduler.java

 

maybe

 backtype.storm.scheduler.IsolationSchedulerzookeeper

backtype.storm.config

backtype.storm.log

backtype.storm.scheduler.DefaultScheduler

backtype.storm.util

STORM-1239 - Getting issue details... STATUS Schedule Topologies in an isolated pool of machines, else schedule evenly.yes,the scheduler is integrated into defaultScheduler.
 backtype.storm.scheduler.multitenant-scheduler-test

backtype.storm.config

backtype.storm.daemon.nimbus

backtype.storm.log

backtype.storm.testing

STORM-1299 - Getting issue details... STATUS Test Multi-tenant schedulerNONE
 backtype.storm.scheduler.resource-aware-scheduler-test

backtype.storm.config

backtype.storm.daemon.nimbus

backtype.storm.testing

backtype.storm.thrift

backtype.storm.util

STORM-1300 - Getting issue details... STATUS Test RASNONE
 backtype.storm.security.auth.AuthUtils-test  STORM-1233 - Getting issue details... STATUS Test AuthUtilsNONE
 backtype.storm.security.auth.DefaultHttpCredentialsPlugin-test  STORM-1234 - Getting issue details... STATUS Test DefaultHttpCredentailsPluginNONE
 backtype.storm.security.auth.ReqContext-test  STORM-1235 - Getting issue details... STATUS Test RequestContextNONE
 backtype.storm.security.auth.SaslTransportPlugin-test  STORM-1236 - Getting issue details... STATUS Test SaslTransportPluginNONE
 backtype.storm.security.auth.ThriftClient-test

backtype.storm.config

backtype.storm.util

STORM-1237 - Getting issue details... STATUS Test secure ThriftClientNONE
 backtype.storm.security.auth.ThriftServer-testbacktype.storm.config STORM-1238 - Getting issue details... STATUS Test secure ThriftServerNONE
 backtype.storm.security.auth.auth-test

backtype.storm.config

backtype.storm.daemon.common

backtype.storm.daemon.nimbus

backtype.storm.testing

backtype.storm.util

STORM-1301 - Getting issue details... STATUS Test Authentication and AuthorizationNONE
 backtype.storm.security.auth.authorizer.DRPCSimpleACLAuthorizer-test

backtype.storm.config

backtype.storm.util

STORM-1240 - Getting issue details... STATUS Test DRPC AuthorizationNONE
 backtype.storm.security.auth.auto-login-module-testbacktype.storm.util STORM-1241 - Getting issue details... STATUS Test AutoTGTKrb5LoginModuleNONE
 backtype.storm.security.auth.drpc-auth-test

backtype.storm.config

backtype.storm.daemon.common

backtype.storm.daemon.drpc

backtype.storm.log

backtype.storm.testing

backtype.storm.util

STORM-1302 - Getting issue details... STATUS Test DRPC AuthorizationNONE
 backtype.storm.security.auth.nimbus-auth-test

backtype.storm.cluster

backtype.storm.config

backtype.storm.daemon.common

backtype.storm.daemon.nimbus

backtype.storm.log

backtype.storm.testing

backtype.storm.util

backtype.storm.zookeeper

STORM-1303 - Getting issue details... STATUS Test Nimbus AuthorizationNONE
 backtype.storm.security.serialization.BlowfishTupleSerializer-test backtype.storm.util STORM-1249 - Getting issue details... STATUS Test BlowfishSerializerNONE
 backtype.storm.serialization-test

backtype.storm.config

backtype.storm.util

STORM-1250 - Getting issue details... STATUS Test KryoValuesSerializerNONE
 backtype.storm.serialization.SerializationFactory-testbacktype.storm.config STORM-1251 - Getting issue details... STATUS Test SerializationFactoryNONE?
 backtype.storm.stats

backtype.storm.log

backtype.storm.util

STORM-1252 - Getting issue details... STATUS clojure methods for getting built-in statisticsNONE?
 backtype.storm.submitter-test

backtype.storm.config

backtype.storm.testing

STORM-1304 - Getting issue details... STATUS Test ZookeeperAuthentication payload generation that is a part of StormSubmitterNONE
 backtype.storm.subtopology-test

backtype.storm.daemon.common

backtype.storm.testing

NONEJust remove commented outNONE

backtype.storm.supervisor-test

backtype.storm.cluster

backtype.storm.config

backtype.storm.daemon.common

backtype.storm.daemon.supervisor

backtype.storm.daemon.worker

backtype.storm.testing

backtype.storm.thrift

backtype.storm.timer

backtype.storm.util

STORM-1305 - Getting issue details... STATUS Test the supervisorNONE?
 backtype.storm.testing

backtype.storm.cluster

backtype.storm.config

backtype.storm.daemon.acker

backtype.storm.daemon.common

backtype.storm.daemon.executor

backtype.storm.daemon.nimbus

backtype.storm.daemon.supervisor

backtype.storm.daemon.worker

backtype.storm.local-state

backtype.storm.log

backtype.storm.messaging.loader

backtype.storm.process-simulator

backtype.storm.thrift

backtype.storm.util

backtype.storm.zookeeper

STORM-1281 - Getting issue details... STATUS lots of helper functions/macros for running testsNONE?
 backtype.storm.testing4j

backtype.storm.LocalCluster

backtype.storm.log

backtype.storm.testing

backtype.storm.util

STORM-1306 - Getting issue details... STATUS backtype.storm.Testing class for use from javaNONE
 backtype.storm.testing4j-test

backtype.storm.clojure

backtype.storm.config

backtype.storm.integration-test

backtype.storm.testing

backtype.storm.thrift

backtype.storm.util

STORM-1307 - Getting issue details... STATUS tests for the Testing4j classNONE
 backtype.storm.thrift

backtype.storm.config

backtype.storm.log

backtype.storm.util

backtype.storm.zookeeper

STORM-1258 - Getting issue details... STATUS helper methods for manipulating thrift objectsNONE
 backtype.storm.tick-tuple-test

backtype.storm.clojure

backtype.storm.config

backtype.storm.daemon.common

backtype.storm.testing

backtype.storm.thrift

STORM-1308 - Getting issue details... STATUS Run topology with system ticksNONE?
 backtype.storm.timer

backtype.storm.log

backtype.storm.util

STORM-1253 - Getting issue details... STATUS Timer like class that uses simulated timehttps://github.com/alibaba/jstorm/blob/master/jstorm-core/src/main/java/com/alibaba/jstorm/callback/AsyncLoopThread.java
 backtype.storm.transactional-test

backtype.storm.clojure

backtype.storm.config

backtype.storm.daemon.common

backtype.storm.testing

backtype.storm.util

STORM-1309 - Getting issue details... STATUS Test transactional topologiesJust one test
 backtype.storm.tuple

 

 

STORM-1275 - Getting issue details... STATUS small helper for getting hash code of a listNONE?
backtype.storm.tuple-testbacktype.storm.testing STORM-1310 - Getting issue details... STATUS   
backtype.storm.ui.core

backtype.storm.config

backtype.storm.converter

backtype.storm.daemon.common

backtype.storm.log

backtype.storm.stats

backtype.storm.thrift

backtype.storm.tuple

backtype.storm.ui.helpers

backtype.storm.util

backtype.storm.zookeeper

STORM-1311 - Getting issue details... STATUS User InterfaceCompletely Different
backtype.storm.ui.helpers

backtype.storm.config

backtype.storm.log

backtype.storm.util

STORM-1254 - Getting issue details... STATUS Helper functions for the UI/HTTPCompletely Different
backtype.storm.utilbacktype.storm.log STORM-1226 - Getting issue details... STATUS util functionsPossibly https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/utils/JStormUtils.java
backtype.storm.utils-test

backtype.storm.config

backtype.storm.util

STORM-1255 - Getting issue details... STATUS test some of utilsPossibly https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/test/java/com/alibaba/jstorm/utils/JStormUtilsTest.java
backtype.storm.utils.ZookeeperServerCnxnFactory-test  STORM-1256 - Getting issue details... STATUS Test ZookeeperServerCnxnFactoryNONE
backtype.storm.versioned-store-testbacktype.storm.testing STORM-1313 - Getting issue details... STATUS Test VersionedStoreNONE
backtype.storm.worker-test

backtype.storm.daemon.common

backtype.storm.daemon.worker

backtype.storm.log

backtype.storm.testing

backtype.storm.util

STORM-1312 - Getting issue details... STATUS Test WorkerNONE
backtype.storm.zookeeper

backtype.storm.config

backtype.storm.log

backtype.storm.util

STORM-1257 - Getting issue details... STATUS Test Zookeeperhttps://github.com/alibaba/jstorm/blob/master/jstorm-core/src/main/java/com/alibaba/jstorm/zk/Zookeeper.java
storm.trident.integration-test

backtype.storm.testing

backtype.storm.util

storm.trident.testing

STORM-1315 - Getting issue details... STATUS Run trident topolgiesNONE?
storm.trident.state-test

backtype.storm.config

backtype.storm.testing

backtype.storm.util

storm.trident.testing

STORM-1316 - Getting issue details... STATUS Test some of trident state 
storm.trident.testing

backtype.storm.LocalDRPC

backtype.storm.testing

backtype.storm.util

STORM-1314 - Getting issue details... STATUS helper functions for testing trident 
storm.trident.tuple-test

backtype.storm.testing

backtype.storm.util

storm.trident.testing

STORM-1317 - Getting issue details... STATUS test trident tuple 

The big things that change when moving from clojure is that we lose ring and hiccup for building the UI/REST services.  We need to pick some alternatives for these.  For the REST APIs Jersey is the most wide spread choice.  hiccup is used in logviewer currently and we should look at replacing it either with static HTML + REST, like for the regular UI, or just return HTML we build ourselves and HTML escaping on parameters.

Phase 2. Port JStorm features to the new java based storm-core.

STORM-1319 - Getting issue details... STATUS

 

The following is an evaluation of major feature differences between storm and JStorm

FeatureApache Storm 0.11.0-SNAPSHOT (and expected pull requests)JStorm-2.1.0/JStorm2.0.4NotesJIRA to port JStorm feature
SchedulerEven, Default, Isolation, Multi-tenant, RAS(work in progress)
  1. Evenly distribute a component's tasks across the nodes in the cluster.
  2. Balanace the number of tasks in a worker.
  3. Try to assign two tasks which are transferring messages directly into the same worker to reduce network cost.
  4. Support user-defined assignment and using the result of the last assignment. Different solution between Storm and JStorm.
The scheduler interface is pluggable, so we should be able to support both schedulers if needed. STORM-1320 - Getting issue details... STATUS
Nimbus HASupport for a pool of nimbus servers. Once Blobstore is merged in, "leader election" and "state storage" will be separate.Support for configuring more than one backup nimbus. When the master nimbus is down, the most appropriate spare nimbus (topologies on disk most closely match the records in ZooKeeper) will be chosen to be promoted.Need to evaluate the strengths and weaknesses of each and decide on updates to storm if any. STORM-1321 - Getting issue details... STATUS
Topology Structureworker 1->* executor 1->* taskworker 1-> taskNeed to evaluate if removing executors will add enough benefit to developers/performance that we can drop it from architecture.  Probably need resource aware re-balancing or Jstorm rebalancing that can support changing parallelism before this can happen. STORM-1322 - Getting issue details... STATUS
Topology MasterNone exactly. The Heartbeat Server (Currently under review) reduces load on ZK, but it is not really the same and may be a complement to TopologyMasterNew system bolt "topology master" was added, which is responsible for collecting task heartbeat info of all tasks and reporting the info to nimbus. Besides task heartbeat info, it also can be used to dispatch control messages within the topology. Topology master significantly reduces the amount of read/write to ZooKeeper. Before this change, ZooKeeper was the bottleneck for deploying big clusters and topologies.Need to evaluate how this impacts storm architecture especially around security. STORM-1323 - Getting issue details... STATUS
BackpressureBang-Bang controller on Disruptor queue capacity using ZooKeeper for broadcast.
  1. Implement backpressure using "topology master" (TM). TM is responsible for processing the trigger message and sending the flow control request to relevant spouts. "flow control" in JStorm doesn't complete stop the spout from emitting tuples, but instead just slows down the tuple sending.
  2. User can update the configuration of backpressure dynamically without restarting topology, e.g. enable/disable backpressue, high/low watermark, etc.
  STORM-1324 - Getting issue details... STATUS
Monitoring of task execute threadPotential Pull Request, but none right now.Monitors the status of the execute thread of tasks. It is effective to find the slow bolt in a topology, and potentially uncovers deadlock as well.Yes we should do this STORM-1325 - Getting issue details... STATUS
Message processing

Deserialization happens on the netty thread

Serialization happens after the send queue when batching is happening.

  1. Add receiving and transferring queue/thread for each task to make deserialization and serialization asynchronously
  2. Remove receiving and transferring thread on worker level to avoid unnecessary locks and to shorten the message processing phase
The two sound equivalent now, but we should talk to see if there are other optimizations needed. STORM-1326 - Getting issue details... STATUS
Tuple BatchingBatching in DisruptorQueueDo batch before sending tuple to transfer queue and support for adjusting the batch size dynamically according to samples of actual batch size sent out for past intervals.Should evaluate both implementations, and see which is better for performance, and possible if we can/should move some of the dynamic batching logic into disruptor. STORM-1327 - Getting issue details... STATUS
Grouping

Load aware balancing in shuffle grouping

  1. Has a "localfirst" grouping that causes tuples to be sent to the tasks in the same worker by default. But if the load of all local tasks is high, the tuples will be sent out to remote tasks.
  2. Improve localOrShuffle grouping from Storm. In Storm's localOrShuffle grouping the definition of "local" is local within the same worker process. i.e., if there is a bolt that the component can send to in the current worker process it will send the tuples there. If there is not one, it will do round robin between all of the instances of that bolt no matter which hosts they are on. JStorm has extended that so that other workers/JVMs on the same host are considered "local" as well, taking into account the load of the network connections on the local worker.
We should look at combining both of these to have shuffle look at both distance and load to decide where to send a tuple. STORM-1328 - Getting issue details... STATUS
web-UIYesDifferentDoes someone know a good UI designer that we can use?  I don't really like either of them (Bobby) but that is just me STORM-1339 - Getting issue details... STATUS
metric systemIMetric and IMetricConsumer
  1. All levels of metrics, including stream metrics, task metrics, component metrics, topology metrics, even cluster metrics, are sampled & calculated. Some metrics, e.g. ""tuple life cycle"", are very useful for debugging and finding the hotspots of a topology.
  2. Support full metrics data. Previous metric system can only display mean value of meters/histograms, the new metric system can display m1, m5, m15 of meters, and common percentiles of histograms.
  3. Use new metrics windows, the mininum metric window is 1 minute, thus we can see the metrics data every single minute.
  4. Supplies a metric uploader interface, third-party companies can easily build their own metric systems based on the historic metric data.
Ideally we should have a way to display most/system metrics in the UI.  IMetric is too generic to make this happen, but we cannot completely drop support for it.  But perhaps we need to depricate it if the JStorm metrics are much better. STORM-1329 - Getting issue details... STATUS
rebalance commandBasic functionality of rebalanceBesides rebalance, scale-out/in by updating the number of workers, ackers, spouts & bolts dynamically without stopping topology. Routing is updated dynamically within upstream components.dynamic routing with some groupings is difficult to get right when there is state, we need to be sure this is well documented, and might want to disallow it for some groupings without a force flag. STORM-1330 - Getting issue details... STATUS
list commandList information of topologiesList information of all topologies, all supervisors, and JStorm versionmore info is good, but we want it human readable too. perhaps with a machine readable option too STORM-1331 - Getting issue details... STATUS
zktool commandN/ASupports some ZooKeeper operations, e.g. "list", "read"…Will need to be evaluated for security STORM-1332 - Getting issue details... STATUS
metricsMonitor commandN/AAllows toggling on/off some metrics which may impact topology performanceSounds great STORM-1333 - Getting issue details... STATUS
restart commandN/ARestart a topology. Besides restart, this command can also update the topology configuration.Sounds great STORM-1334 - Getting issue details... STATUS
update_topology commandN/AUpdate jars and configuration dynamically for a topology, without stopping the topology.Sounds great, should work will with blob-storm STORM-1335 - Getting issue details... STATUS
cgroupN/ASupports controlling the upper limit of CPU core usage for a worker using cgroupsSounds like a good start, will be nice to integrate it with RAS requests too. STORM-1336 - Getting issue details... STATUS
logging
  1. Supports user-defined log configuration via Log4j 2
  2. Supports dynamic changes to logging of a running topology
  3. Supports log4j and slf4j log APIS
  1. Supports user-defined configuration of log
  2. Supports both logback and log4j
Need to evaluate differences and see what we want long term STORM-1337 - Getting issue details... STATUS
Worker classloader (isolation)N/A - uses shading for most dependenciesThe "worker classloader" avoids problem of re-loading classesThis sounds great STORM-1338 - Getting issue details... STATUS
Multi-thread of spoutNAThere are two modes of spout, "single-thread" and "multi-thread" in JStorm. The "single-thread" mode is simliar to Storm while the "multi-thread" mode separates the processing of ack/fail and nextTuple to two threads. It means we can stay in nextTuple for a long time without any side effect on ack/fail. This improves the response and throughput of spout STORM-1358 - Getting issue details... STATUS

 

 

 

  • No labels