Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Please help us keep this FAQ up-to-date. If there is an answer that you think can be improved, please help improve it. If you look for an answer that isn't here, and later figure it out, please add it. You don't need permission, it's a wiki. (smile)

Table of Contents

Streams

When I commit my process state, what does the Streams library do and how it affects my application's performance?

Streams application's state can be committed either periodically based on the `commit.interval` config, or whenever users call `context.commit` in their application code.

When commit operation is triggered, Streams library will make the following steps:

1. Flush the state store, and write the checkpoint file for the current log end offset of the store.
2. Flush the producer to make sure any batched records has been sent and acked by the brokers.
3. Send a commit offset request to brokers in sync mode to the broker for the input topics.
4*. If exactly_once mode is turned on, there are extra steps needed to make sure the above steps are done atomically.
As one can see, these steps have non-negligible overhead on the throughput, since they need to be done synchronously while no records are being processed at the mean time. On the other hand, the shorter the commit interval less time is needed when streams application is re-bootstrapped. In practice users need to consider tuning commit frequency so that it is not affecting their application performance during normal runs, while still have a fast failover / restart cost.
My application failed to start, with the a rocksDB exception raised as "java.lang.ExceptionInInitializerError.. Unable to load the RocksDB shared libraryjava...". How to resolve this?

...