DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
However, this approach has limitations: offset translation is inherently lossy, especially for lagging consumers. Each partition retains only up to 64 offset-syncs, with the most recent sync always preserved and older syncs spaced exponentially apart. Consequently, recently mirrored records translate accurately, but translation accuracy degrades exponentially as records age increases.
...
…could go up to [63] but 14 entries are enough for 1000000 offsets
Let’s say there is a lagging consumer and its current offset is 998400. The record is mirrored to the target cluster at offset 898400. The closest cached mapping for this source offset is 996900→896900, therefore the translated offset would be 896901 (896900 + 1). During failover, this consumer would reprocess 1499 records (898400 - 896901). This would only grow bigger as the offset gets older as demonstrated by the example.
Also, if a consumer commits a more recent offset 1000100, the MirrorCheckpointConnector is not able to translate that offset, and returns -1. During failover, the consumer would continue from the most recently translated offset which is 1000000, and the consumer would reprocess 100 records.
Each MirrorCheckpointConnector task maintains an offset mapping cache for each of its assigned partitions. For example, after mirroring 1000000 records to a partition, the cache entries and the offset translation might look like this:
...
Source Offset
...
Actual Target offset
...
Closest Cached Mapping
...
Translated Target Offset
(cached target + 1)
...
Re-read Count
...
1000100
...
Not mirrored yet
...
-1
...
1000000
...
900000
...
[0] 1000000→900000
...
900000
...
0
(exact match)
...
999999
...
899999
...
[1] 999900→899900
(gap: 100)
...
899901
...
99
...
999890
...
899890
...
[2] 999700→899700
(gap: 200)
...
899701
...
189
...
999650
...
899650
...
[3] 999300→899300
(gap: 400)
...
899301
...
349
...
999200
...
899200
...
[4] 998500→898500
(gap: 800)
...
898501
...
699
...
998400
...
898400
...
[5] 996900→896900
(gap: 1600)
...
896901
...
1499
...
996800
...
896800
...
[6] 993700→893700
(gap: 3200)
...
893701
...
3099
...
993600
...
893600
...
[7] 987300→887300
(gap: 6400)
...
887301
...
6299
...
987200
...
887200
...
[8] 974500→874500
(gap: 12800)
...
874501
...
12699
...
974400
...
874400
...
[9] 948900→848900
(gap: 25600)
...
848901
...
25499
...
948800
...
848800
...
[10] 897700→797700
(gap: 51200)
...
797701
...
51099
...
897600
...
797600
...
[11] 795300→695300
(gap: 102400)
...
695301
...
102299
...
795200
...
695200
...
[12] 590500→490500
(gap: 204800)
...
490501
...
204699
...
590400
...
490400
...
[13] 180900→80900
(gap: 409600)
...
80901
...
409499
Moreover, the offset-syncs topic records mappings at a fixed internal value defined by by offset.lag.max set to 100 by default. This has several limitations:
...