Versions Compared

Old Version 371

changes.mady.by.user Stanislav Kozlovski

Saved on

compared with

New Version Current

changes.mady.by.user Satish Duggana

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  •  Discussion Recording
  • Notes

    1. Tiered storage upgrade path dicussion:

    • Details need to be documented in the KIP.
    • Current upgrade path plan is based on IBP bump.
    • Enabling of the remote log components may not mean all topics are eligible for tiering at the same time.
    • Should tiered storage be enabled on all brokers before enabling it on any brokers?
    • Is there any replication path dependency for enabling tiered storage?


    2. RLMM persistence format:

    • We agreed to document the persistence format for the materialized state of default RLMM implementation (topic-based).
    • (carry over from earlier discussion) For the file-based design, we don't know yet the % of increase in memory, assuming the majority of segments are in remote storage. It will be useful to document an estimation for this.


    3. Topic deletion lifecycle discussion:

    • Under topic deletion lifecycle, step (4) it would be useful to mention how the RemotePartitionRemover (RPRM) gets the list of segments to be deleted, and whether it has any dependency with the RLMM topic.


    4. Log layer discussion:

    • We discussed the complexities surrounding making code changes to Log layer (Log.scala).
    • Today, the Log class holds attributes and behavior related with local log. In the future, we would have to change the Log layer such that it would also contain the logic for the tiered portion of the log. This addition can pose a maintenance challenge.
    • Some of the existing attributes in the Log layer such as LeaderEpochCache and ProducerStateManager can be related with global view of the log too (i.e. global log is local log + tiered log). It can be therefore useful to think about preparatory refactoring, to see whether we can separate responsibilities related with the local log from the tiered log, and, perhaps provide a global view of the log that combines together both as and when required. The global view of the log could manage the lifecycle of LeaderEpochCache and ProducerStateManager.

    Follow-ups:
    • KIP-405 updates (upgrade path, RLMM file format and topic deletion)
    • Log layer changes

     
    (Notes taken by Kowshik)


...