DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
Report Date: February 4, 2026
Environment: Amazon EKS (ap-south-1)
Flink Version: 2.1.1
Benchmark Duration: ~77 minutes
Patch Used: https://github.com/apache/flink/pull/27187
Author: Samrat Deb
1. Summary
Key Performance Metrics
┌─────────────────────────────────────────────────────────────────────────────────┐
│ CHECKPOINT PERFORMANCE SUMMARY │
├─────────────────────────┬─────────────────┬─────────────────┬───────────────────┤
│ Metric │ Native S3 │ Presto S3 │ Difference │
├─────────────────────────┼─────────────────┼─────────────────┼───────────────────┤
│ Checkpoints Completed │ 78 │ 40 │ Native 1.95x more │
│ Max State Checkpointed │ 15.33 GB │ 15.49 GB │ Similar │
│ Avg Checkpoint Duration │ 48.8s │ 90.1s │ Native 1.85x │
│ P50 Checkpoint Duration │ 52.8s │ 96.8s │ Native 1.83x │
│ P90 Checkpoint Duration │ 72.5s │ 155.0s │ Native 2.14x │
│ P99 Checkpoint Duration │ 76.7s │ 165.3s │ Native 2.15x │
│ Avg Throughput │ ~200 MB/s │ ~92 MB/s │ Native 2.17x │
│ Failed Checkpoints │ 0 │ 0 │ Both 100% │
└─────────────────────────┴─────────────────┴─────────────────┴───────────────────┘
Native S3 (flink-s3-fs-native) comparison to (flink-s3-fs-presto)
- 2x higher checkpoint throughput (200 MB/s vs 90 MB/s)
- 2x faster checkpoint duration at large state sizes
- Same reliability - 100% checkpoint success rate
2. Visual Performance Comparison
2.1 Checkpoint Duration Comparison (at similar state sizes)
CHECKPOINT DURATION AT ~15 GB STATE
════════════════════════════════════
Native S3 │████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░ │ 77s
Presto S3 │█████████████████████████████████████████████████████████████████ │ 165s
└───────────────────────────────────────────────────────────────────┘
0s 30s 60s 90s 120s 150s 180s
Native S3 is 2.14x FASTER at 15 GB state
2.2 Throughput Comparison by State Size
THROUGHPUT (MB/s) BY STATE SIZE
400+ │ ●●
│ ●●
350+ │ ●
│ ●●
300+ │ ●
│ ●
250+ │ ●●
│ ●●●
200+ │ Native S3 ────●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
│ ●●●●●●●
150+ │
│
100+ │ ○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○
│ Presto S3 ────────────────────────────────────────────────────
50+ │
└────────────────────────────────────────────────────────────────────
1GB 3GB 5GB 7GB 9GB 11GB 13GB 15GB
● = Native S3 (avg ~200 MB/s)
○ = Presto S3 (avg ~92 MB/s, caps at ~95 MB/s)
2.3 Checkpoint Duration Over Time
CHECKPOINT DURATION PROGRESSION
Duration
(seconds)
│
180│ ○ ○ ○ ○ ○ ○ ○ ○
│ ○ ○
150│ ○ ○
│ ○ ○
120│ ○ ○
│ ○ ○
90│ ○ ○
│ ○ ○ ● ● ● ● ● ● ● ● ● ● ●
60│ ○ ○ ● ● ● ● ● ● ●
│ ○ ● ● ●
30│ ○ ● ● ● ●
│ ○ ● ● ● ●
0│_○_●_●_________●_______________________________________________
└──────────────────────────────────────────────────────────────
1 5 10 15 20 25 30 35 40 45 50 55 60
Checkpoint Number
● = Native S3 ○ = Presto S3
2.4 State Growth Comparison
STATE SIZE GROWTH OVER TIME
State
(GB)
│
16 │ ● ● ● ● ○ ○ ○ ○
│ ● ○
14 │ ● ● ○
│ ● ● ○
12 │ ● ● ○
│ ● ● ○
10 │ ● ● ○
│ ● ● ○
8 │ ● ● ○
│ ● ● ○
6 │ ● ● ○
│ ● ● ○
4 │ ● ● ○
│● ○
2 │○
│
0 └──────────────────────────────────────────────────────────────
0 10 20 30 40 50 60 70 80 90 (minutes)
● = Native S3 (78 checkpoints) ○ = Presto S3 (40 checkpoints)
Native S3: 78 checkpoints in 77 minutes (faster checkpoint = more state processed)
Presto S3: 40 checkpoints in 77 minutes (slower checkpoint = less state processed)
3. Statistical Analysis
3.1 Duration Percentiles
┌─────────────────────────────────────────────────────────────────────────────────┐
│ CHECKPOINT DURATION PERCENTILES (ms) │
├────────────┬──────────────┬──────────────┬────────────────┬─────────────────────┤
│ Percentile │ Native S3 │ Presto S3 │ Native Faster │ Visual │
├────────────┼──────────────┼──────────────┼────────────────┼─────────────────────┤
│ Min │ 2,051 │ 2,393 │ 1.17x │ █ │
│ P50 │ 52,836 │ 96,771 │ 1.83x │ ████████ │
│ P90 │ 72,487 │ 154,961 │ 2.14x │ ████████████ │
│ P95 │ 73,224 │ 161,200 │ 2.20x │ █████████████ │
│ P99 │ 76,727 │ 165,251 │ 2.15x │ █████████████ │
│ Max │ 76,727 │ 165,251 │ 2.15x │ █████████████ │
│ Average │ 48,831 │ 90,127 │ 1.85x │ ███████ │
└────────────┴──────────────┴──────────────┴────────────────┴─────────────────────┘
3.2 State Size Percentiles
┌─────────────────────────────────────────────────────────────────────────────────┐
│ STATE SIZE PERCENTILES (GB) │
├────────────┬──────────────┬──────────────┬──────────────────────────────────────┤
│ Percentile │ Native S3 │ Presto S3 │ Notes │
├────────────┼──────────────┼──────────────┼──────────────────────────────────────┤
│ Min │ 0.28 GB │ 0.08 GB │ First checkpoint │
│ P50 │ 9.60 GB │ 8.26 GB │ Median checkpoint size │
│ P90 │ 13.54 GB │ 13.49 GB │ Similar at high percentiles │
│ P95 │ 13.95 GB │ 14.08 GB │ Both reached similar max state │
│ P99 │ 14.28 GB │ 14.43 GB │ │
│ Max │ 15.33 GB │ 15.49 GB │ Both scaled to ~15 GB │
│ Average │ 8.95 GB │ 7.71 GB │ │
└────────────┴──────────────┴──────────────┴──────────────────────────────────────┘
3.3 Throughput by State Size Range
┌─────────────────────────────────────────────────────────────────────────────────┐
│ THROUGHPUT BY STATE SIZE (MB/s) │
├───────────────────┬─────────────┬─────────────┬─────────────┬───────────────────┤
│ State Size Range │ Native S3 │ Presto S3 │ Speedup │ Graph │
├───────────────────┼─────────────┼─────────────┼─────────────┼───────────────────┤
│ 0-2 GB │ 362 MB/s │ 79 MB/s │ 4.58x │ ████████████████ │
│ 2-4 GB │ 285 MB/s │ 85 MB/s │ 3.35x │ █████████████ │
│ 4-6 GB │ 173 MB/s │ 84 MB/s │ 2.06x │ ████████ │
│ 6-8 GB │ 165 MB/s │ 86 MB/s │ 1.92x │ ████████ │
│ 8-10 GB │ 180 MB/s │ 91 MB/s │ 1.98x │ ████████ │
│ 10-12 GB │ 193 MB/s │ 93 MB/s │ 2.08x │ ████████ │
│ 12-14 GB │ 198 MB/s │ 93 MB/s │ 2.13x │ █████████ │
│ 14-16 GB │ 203 MB/s │ 94 MB/s │ 2.16x │ █████████ │
└───────────────────┴─────────────┴─────────────┴─────────────┴───────────────────┘
Native S3: Maintains 180-200 MB/s throughput at large state sizes
Presto S3: Caps around 90-95 MB/s regardless of optimizations
4. S3 Bucket Metrics & IOPS Analysis
4.1 Bucket Summary
┌─────────────────────────────────────────────────────────────────────┐
│ S3 BUCKET SUMMARY │
├─────────────────────────────────────────────────────────────────────┤
│ Bucket: flink-benchmark-input-251751825822 │
│ Region: ap-south-1 (Mumbai) │
│ Total Objects: 1,591 │
│ Total Size: 88.60 GB │
└─────────────────────────────────────────────────────────────────────┘
4.2 Checkpoint Storage Breakdown
┌─────────────────────────────────────────────────────────────────────┐
│ CHECKPOINT STORAGE BY JOB │
├───────────────────┬─────────────┬─────────────┬─────────────────────┤
│ Job │ Objects │ Size │ Checkpoints │
├───────────────────┼─────────────┼─────────────┼─────────────────────┤
│ Native S3 │ 551 │ 25.62 GB │ 82 │
│ Presto S3 │ 370 │ 17.83 GB │ 43 │
│ Other (old runs) │ 670 │ 45.15 GB │ - │
├───────────────────┼─────────────┼─────────────┼─────────────────────┤
│ TOTAL │ 1,591 │ 88.60 GB │ - │
└───────────────────┴─────────────┴─────────────┴─────────────────────┘
4.3 IOPS Analysis
┌─────────────────────────────────────────────────────────────────────┐
│ ESTIMATED S3 IOPS │
├───────────────────┬─────────────────────────┬───────────────────────┤
│ Metric │ Native S3 │ Presto S3 │
├───────────────────┼─────────────────────────┼───────────────────────┤
│ Checkpoint Size │ ~15.3 GB │ ~15.5 GB │
│ Duration │ ~77 seconds │ ~165 seconds │
│ Files/Checkpoint │ ~7 files │ ~9 files │
│ Avg File Size │ ~64 MB │ ~64 MB │
│ Multipart Size │ 16 MB │ 16 MB │
│ Parts/File │ ~4 parts │ ~4 parts │
├───────────────────┼─────────────────────────┼───────────────────────┤
│ PUTs/Checkpoint │ ~33 (28 data + 5 meta) │ ~41 (36 data + 5 meta)│
│ Write IOPS │ ~0.43 ops/sec │ ~0.25 ops/sec │
│ Write Throughput │ 198 MB/s │ 94 MB/s │
└───────────────────┴─────────────────────────┴───────────────────────┘
Note: S3 IOPS are low because Flink uses large multipart uploads (16 MB parts),
optimizing for throughput over IOPS - correct for checkpoint workloads.
4.4 Storage Efficiency
┌─────────────────────────────────────────────────────────────────────┐
│ STORAGE EFFICIENCY │
├───────────────────┬─────────────────────────┬───────────────────────┤
│ Metric │ Native S3 │ Presto S3 │
├───────────────────┼─────────────────────────┼───────────────────────┤
│ Total Storage │ 25.62 GB │ 17.83 GB │
│ Total Checkpoints │ 82 │ 43 │
│ Avg Storage/Chkpt │ 312 MB │ 415 MB │
│ Objects/Chkpt │ 6.7 │ 8.6 │
└───────────────────┴─────────────────────────┴───────────────────────┘
Native S3 creates fewer, more compact checkpoint files.
5. Benchmark Setup
5.1 Infrastructure Configuration
┌─────────────────────────────────────────────────────────────────────────────────┐
│ INFRASTRUCTURE │
├──────────────────────────┬──────────────────────────────────────────────────────┤
│ Platform │ Amazon EKS │
│ Region │ ap-south-1 (Mumbai) │
│ S3 Bucket │ flink-benchmark-input-251751825822 │
│ Flink Version │ 2.1.1 │
│ Flink Kubernetes Operator│ Running │
│ State Backend │ RocksDB (full checkpoints) │
│ Checkpoint Mode │ EXACTLY_ONCE │
└──────────────────────────┴──────────────────────────────────────────────────────┘
5.2 Resource Allocation (Identical for Both)
┌────────────────────────────────────────────────────────────────┐
│ RESOURCE ALLOCATION │
├────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────┐ ┌─────────────────────┐ │
│ │ JobManager │ │ TaskManager x2 │ │
│ │ ───────────── │ │ ───────────── │ │
│ │ Memory: 2 GB │ │ Memory: 6 GB │ │
│ │ CPU: 1 core │ │ CPU: 1.5 cores │ │
│ │ Replicas: 1 │ │ Slots: 4 each │ │
│ └─────────────────────┘ └─────────────────────┘ │
│ │
│ Total Parallelism: 8 Total Task Slots: 8 │
│ │
└────────────────────────────────────────────────────────────────┘
5.3 Checkpoint Configuration (Identical for Both)
execution.checkpointing.interval: 60000ms (1 minute)
execution.checkpointing.mode: EXACTLY_ONCE
execution.checkpointing.min-pause: 30000ms
execution.checkpointing.max-concurrent-checkpoints: 1
execution.checkpointing.timeout: 600000ms (10 minutes)
state.backend.type: rocksdb
state.backend.incremental: false (full checkpoints)
state.backend.rocksdb.checkpoint.transfer.thread.num: 2
state.backend.rocksdb.memory.managed: true
5.4 S3 Configuration Comparison
┌─────────────────────────────────────────────────────────────────────────────────┐
│ S3 CONFIGURATION │
├─────────────────────────────────┬───────────────────────────────────────────────┤
│ Native S3 │ Presto S3 │
├─────────────────────────────────┼───────────────────────────────────────────────┤
│ s3.connection.maximum: 100 │ presto.s3.max-connections: 100 │
│ s3.upload.part.size: 16 MB │ presto.s3.multipart.min-part-size: 16 MB │
│ s3.async-client.max-concurrency │ presto.s3.multipart.min-file-size: 16 MB │
│ : 64 │ │
├─────────────────────────────────┼───────────────────────────────────────────────┤
│ AWS SDK v2 with async support │ Presto S3 client (sync I/O). │
│ S3TransferManager for uploads │ Traditional blocking I/O │
└─────────────────────────────────┴───────────────────────────────────────────────┘
6. Workload Description
6.1 LargeStateGeneratorJob
┌─────────────────────────────────────────────────────────────────────────────────┐
│ JOB PARAMETERS │
├────────────────────────┬────────────────────────────────────────────────────────┤
│ --numKeys │ 20,000,000 (20 million unique keys) │
│ --valueSizeKb │ 1 (1 KB per value) │
│ --targetStateTb │ 0.02 (target 20 GB total state) │
│ --recordsPerSecond │ 50,000 (across all parallel instances) │
└────────────────────────┴────────────────────────────────────────────────────────┘
6.2 Data Flow Architecture
┌────────────────────────────────────────────────────────────────────────────────┐
│ │
│ ┌───────────────────┐ ┌───────────────────────┐ ┌─────────────────┐ │
│ │ RecordGenerator │───▶│ LargeStateAccumulator│───▶ │ Metrics Output │ │
│ │ Source │ │ (Stateful) │ │ (Sink) │ │
│ └───────────────────┘ └───────────────────────┘ └─────────────────┘ │
│ │ │ │
│ │ ▼ │
│ │ ┌─────────────────┐ │
│ │ │ RocksDB │ │
│ │ │ State Store │ │
│ │ │ (1KB/key) │ │
│ │ └────────┬────────┘ │
│ │ │ │
│ │ ▼ │
│ │ ┌─────────────────┐ │
│ │ │ Checkpoint to │ │
│ │ │ S3 Bucket │ │
│ │ │ (every 60s) │ │
│ │ └─────────────────┘ │
│ │ │
│ 50K records/sec 20M keys × 1KB = 20GB target state │
│ │
└────────────────────────────────────────────────────────────────────────────────┘
6.3 State Generation Code
/**
* Accumulates 1KB state per key, updating existing state with incoming records.
*/
public static class LargeStateAccumulator extends RichFlatMapFunction<KeyedRecord, StateMetrics> {
private transient ValueState<byte[]> state; // 1KB per key
@Override
public void flatMap(KeyedRecord record, Collector<StateMetrics> out) throws Exception {
byte[] currentState = state.value();
if (currentState == null) {
// Initialize 1KB state for new key
currentState = new byte[valueSizeKb * 1024]; // 1KB
random.nextBytes(currentState);
} else {
// Partial update to trigger checkpoint write
System.arraycopy(record.payload, 0, currentState, updateOffset, copyLength);
}
state.update(currentState); // Triggers RocksDB write
}
}
6.4 Metrics Captured
| Metric | Source | Description |
|---|---|---|
checkpointed_size | Flink REST API | Total bytes written to S3 |
state_size | Flink REST API | Logical state size in bytes |
end_to_end_duration | Flink REST API | Total checkpoint time |
checkpointDuration | JobManager logs | Coordinator-reported duration |
finalizationTime | JobManager logs | Metadata finalization time |
7. Detailed Checkpoint Data
7.1 Native S3 - All 78 Checkpoints
Checkpoints 1-40 (First 40)
| Chk# | State Size | Duration | Throughput | Timestamp |
|---|---|---|---|---|
| 1 | 287 MB | 2,759 ms | 104 MB/s | 18:07:30 |
| 2 | 727 MB | 2,276 ms | 319 MB/s | 18:08:29 |
| 3 | 1.14 GB | 2,513 ms | 467 MB/s | 18:09:30 |
| 4 | 1.57 GB | 4,783 ms | 337 MB/s | 18:10:32 |
| 5 | 1.95 GB | 3,791 ms | 527 MB/s | 18:11:31 |
| 6 | 2.38 GB | 7,366 ms | 331 MB/s | 18:12:34 |
| 7 | 2.81 GB | 7,196 ms | 400 MB/s | 18:13:34 |
| 8 | 3.24 GB | 15,200 ms | 218 MB/s | 18:14:42 |
| 9 | 3.43 GB | 8,220 ms | 427 MB/s | 18:15:35 |
| 10 | 3.85 GB | 25,056 ms | 157 MB/s | 18:16:52 |
| 11 | 4.18 GB | 30,569 ms | 140 MB/s | 18:17:58 |
| 12 | 4.47 GB | 31,017 ms | 148 MB/s | 18:18:59 |
| 13 | 4.64 GB | 25,669 ms | 185 MB/s | 18:19:54 |
| 14 | 4.94 GB | 31,514 ms | 160 MB/s | 18:21:00 |
| 15 | 5.21 GB | 31,751 ms | 168 MB/s | 18:22:02 |
| 16 | 5.51 GB | 33,850 ms | 167 MB/s | 18:23:06 |
| 17 | 5.73 GB | 31,134 ms | 188 MB/s | 18:24:07 |
| 18 | 5.86 GB | 35,451 ms | 169 MB/s | 18:25:12 |
| 19 | 6.13 GB | 37,313 ms | 168 MB/s | 18:26:20 |
| 20 | 6.42 GB | 40,927 ms | 161 MB/s | 18:27:31 |
| 21 | 6.68 GB | 42,390 ms | 161 MB/s | 18:28:43 |
| 22 | 6.82 GB | 39,991 ms | 174 MB/s | 18:29:53 |
| 23 | 6.96 GB | 44,822 ms | 159 MB/s | 18:31:08 |
| 24 | 7.14 GB | 46,227 ms | 158 MB/s | 18:32:24 |
| 25 | 7.36 GB | 48,291 ms | 156 MB/s | 18:33:42 |
| 26 | 7.52 GB | 49,109 ms | 157 MB/s | 18:35:01 |
| 27 | 7.63 GB | 45,938 ms | 170 MB/s | 18:36:17 |
| 28 | 7.78 GB | 48,728 ms | 164 MB/s | 18:37:36 |
| 29 | 8.00 GB | 50,499 ms | 162 MB/s | 18:38:57 |
| 30 | 8.17 GB | 49,087 ms | 170 MB/s | 18:40:16 |
| 31 | 8.30 GB | 51,644 ms | 165 MB/s | 18:41:37 |
| 32 | 8.43 GB | 49,323 ms | 175 MB/s | 18:42:57 |
| 33 | 8.55 GB | 52,644 ms | 166 MB/s | 18:44:19 |
| 34 | 8.75 GB | 48,601 ms | 184 MB/s | 18:45:38 |
| 35 | 8.93 GB | 51,651 ms | 177 MB/s | 18:47:00 |
| 36 | 9.14 GB | 52,817 ms | 177 MB/s | 18:48:22 |
| 37 | 9.24 GB | 54,599 ms | 173 MB/s | 18:49:47 |
| 38 | 9.37 GB | 55,183 ms | 174 MB/s | 18:51:12 |
| 39 | 9.51 GB | 51,105 ms | 190 MB/s | 18:52:33 |
| 40 | 9.69 GB | 51,551 ms | 193 MB/s | 18:53:55 |
Checkpoints 41-78 (Last 38)
| Chk# | State Size | Duration | Throughput | Timestamp |
|---|---|---|---|---|
| 41 | 9.82 GB | 52,874 ms | 190 MB/s | 18:55:18 |
| 42 | 9.95 GB | 53,890 ms | 189 MB/s | 18:56:42 |
| 43 | 10.09 GB | 53,237 ms | 194 MB/s | 18:58:05 |
| 44 | 10.17 GB | 54,014 ms | 193 MB/s | 18:59:29 |
| 45 | 10.27 GB | 53,224 ms | 198 MB/s | 19:00:52 |
| 46 | 10.41 GB | 53,633 ms | 199 MB/s | 19:02:16 |
| 47 | 10.58 GB | 54,361 ms | 199 MB/s | 19:03:40 |
| 48 | 10.73 GB | 57,578 ms | 191 MB/s | 19:05:08 |
| 49 | 10.87 GB | 57,584 ms | 193 MB/s | 19:06:35 |
| 50 | 10.99 GB | 56,541 ms | 199 MB/s | 19:08:02 |
| 51 | 11.10 GB | 59,929 ms | 190 MB/s | 19:09:32 |
| 52 | 11.18 GB | 58,204 ms | 197 MB/s | 19:11:00 |
| 53 | 11.31 GB | 61,798 ms | 187 MB/s | 19:12:32 |
| 54 | 11.47 GB | 59,273 ms | 198 MB/s | 19:14:01 |
| 55 | 11.58 GB | 61,593 ms | 192 MB/s | 19:15:33 |
| 56 | 11.67 GB | 61,095 ms | 196 MB/s | 19:17:04 |
| 57 | 11.82 GB | 61,989 ms | 195 MB/s | 19:18:36 |
| 58 | 11.91 GB | 61,106 ms | 200 MB/s | 19:20:07 |
| 59 | 12.02 GB | 62,594 ms | 197 MB/s | 19:21:39 |
| 60 | 12.16 GB | 63,969 ms | 195 MB/s | 19:23:13 |
| 61 | 12.33 GB | 66,873 ms | 189 MB/s | 19:24:50 |
| 62 | 12.46 GB | 64,757 ms | 197 MB/s | 19:26:25 |
| 63 | 12.59 GB | 67,563 ms | 191 MB/s | 19:28:02 |
| 69 | 13.28 GB | 68,959 ms | 197 MB/s | 19:34:59 |
| 70 | 13.38 GB | 73,109 ms | 187 MB/s | 19:36:42 |
| 71 | 13.53 GB | 72,418 ms | 191 MB/s | 19:38:25 |
| 72 | 13.67 GB | 70,755 ms | 198 MB/s | 19:40:06 |
| 73 | 13.74 GB | 73,105 ms | 193 MB/s | 19:41:50 |
| 74 | 13.82 GB | 70,092 ms | 202 MB/s | 19:43:30 |
| 75 | 13.94 GB | 73,386 ms | 195 MB/s | 19:45:13 |
| 76 | 14.05 GB | 73,215 ms | 196 MB/s | 19:46:57 |
| 77 | 14.16 GB | 74,304 ms | 195 MB/s | 19:48:41 |
| 78 | 14.27 GB | 76,727 ms | 190 MB/s | 19:50:28 |
7.2 Presto S3 - All 40 Checkpoints
| Chk# | State Size | Duration | Throughput | Timestamp |
|---|---|---|---|---|
| 1 | 77 MB | 2,754 ms | 28 MB/s | 18:31:33 |
| 2 | 521 MB | 7,504 ms | 69 MB/s | 18:32:37 |
| 3 | 956 MB | 11,586 ms | 83 MB/s | 18:33:41 |
| 4 | 1.35 GB | 17,030 ms | 81 MB/s | 18:34:47 |
| 5 | 1.72 GB | 21,502 ms | 82 MB/s | 18:35:51 |
| 6 | 2.12 GB | 24,799 ms | 88 MB/s | 18:36:55 |
| 7 | 2.51 GB | 29,610 ms | 87 MB/s | 18:37:59 |
| 8 | 2.89 GB | 37,507 ms | 79 MB/s | 18:39:07 |
| 9 | 3.15 GB | 37,826 ms | 85 MB/s | 18:40:15 |
| 10 | 3.58 GB | 42,293 ms | 87 MB/s | 18:41:27 |
| 11 | 4.02 GB | 46,838 ms | 88 MB/s | 18:42:44 |
| 12 | 4.49 GB | 56,778 ms | 81 MB/s | 18:44:11 |
| 13 | 4.79 GB | 57,738 ms | 85 MB/s | 18:45:39 |
| 14 | 5.31 GB | 64,636 ms | 84 MB/s | 18:47:13 |
| 15 | 5.75 GB | 68,432 ms | 86 MB/s | 18:48:52 |
| 16 | 6.33 GB | 77,623 ms | 84 MB/s | 18:50:39 |
| 17 | 6.71 GB | 78,501 ms | 88 MB/s | 18:52:28 |
| 18 | 7.10 GB | 83,793 ms | 87 MB/s | 18:54:22 |
| 19 | 7.75 GB | 92,572 ms | 86 MB/s | 18:56:24 |
| 20 | 8.12 GB | 95,407 ms | 87 MB/s | 18:58:30 |
| 21 | 8.40 GB | 98,371 ms | 88 MB/s | 19:00:38 |
| 22 | 8.76 GB | 103,421 ms | 87 MB/s | 19:02:52 |
| 23 | 9.13 GB | 109,026 ms | 86 MB/s | 19:05:11 |
| 24 | 9.50 GB | 112,858 ms | 86 MB/s | 19:07:33 |
| 25 | 9.80 GB | 114,512 ms | 88 MB/s | 19:09:58 |
| 26 | 10.11 GB | 117,736 ms | 88 MB/s | 19:12:26 |
| 27 | 10.39 GB | 120,964 ms | 88 MB/s | 19:14:57 |
| 28 | 10.70 GB | 124,532 ms | 88 MB/s | 19:17:31 |
| 29 | 11.05 GB | 125,028 ms | 91 MB/s | 19:20:06 |
| 30 | 11.26 GB | 129,900 ms | 89 MB/s | 19:22:46 |
| 31 | 11.58 GB | 132,906 ms | 89 MB/s | 19:25:29 |
| 32 | 11.90 GB | 138,532 ms | 88 MB/s | 19:28:18 |
| 33 | 12.23 GB | 141,225 ms | 89 MB/s | 19:31:09 |
| 34 | 12.59 GB | 144,931 ms | 89 MB/s | 19:34:04 |
| 35 | 12.88 GB | 148,555 ms | 89 MB/s | 19:37:02 |
| 36 | 13.18 GB | 151,638 ms | 89 MB/s | 19:40:04 |
| 37 | 13.52 GB | 155,462 ms | 89 MB/s | 19:43:09 |
| 38 | 13.86 GB | 159,633 ms | 89 MB/s | 19:46:19 |
| 39 | 14.09 GB | 161,436 ms | 89 MB/s | 19:49:30 |
| 40 | 14.43 GB | 165,414 ms | 89 MB/s | 19:52:46 |
8. Head-to-Head Comparison
8.1 First 40 Checkpoints Comparison
┌─────────────────────────────────────────────────────────────────────────────────┐
│ FIRST 40 CHECKPOINTS COMPARISON │
├─────────────────────────────┬─────────────────┬─────────────────────────────────┤
│ Metric │ Native S3 │ Presto S3 │
├─────────────────────────────┼─────────────────┼─────────────────────────────────┤
│ Time to complete │ 47 minutes │ 81 minutes │
│ Final state size │ 9.69 GB │ 14.43 GB │
│ Total checkpoint time │ 24.8 min │ 59.5 min │
│ Average duration │ 37.2s │ 89.2s │
│ Average throughput │ 192 MB/s │ 85 MB/s │
├─────────────────────────────┼─────────────────┼─────────────────────────────────┤
8.2 At Similar State Size (~14.5 GB)
┌─────────────────────────────────────────────────────────────────────────────────┐
│ COMPARISON AT ~14.5 GB STATE SIZE │
├─────────────────────────────┬─────────────────┬─────────────────────────────────┤
│ Metric │ Native S3 │ Presto S3 │
│ │ (Checkpoint 78) │ (Checkpoint 40) │
├─────────────────────────────┼─────────────────┼─────────────────────────────────┤
│ State Size │ 14.27 GB │ 14.43 GB │
│ Duration │ 76,727 ms │ 165,414 ms │
│ Throughput │ 190 MB/s │ 89 MB/s │
├─────────────────────────────┼─────────────────┼─────────────────────────────────┤
│ Native S3 Advantage │ 2.16x faster │ │
└─────────────────────────────┴─────────────────┴─────────────────────────────────┘
9. Log Evidence
9.1 Native S3 JobManager Logs
Job Initialization:
2026-02-03 18:06:23,724 INFO ClusterEntrypoint - Starting KubernetesApplicationClusterEntrypoint
(Version: 2.1.1, Scala: 2.12)
State Backend:
2026-02-03 18:06:27,735 INFO JobMaster - Loading state backend via factory
org.apache.flink.state.rocksdb.EmbeddedRocksDBStateBackendFactory
2026-02-03 18:06:27,805 INFO StateBackendLoader - State backend loader loads the state backend
as EmbeddedRocksDBStateBackend
Sample Checkpoint Completions:
# Small state - fast
2026-02-03 18:07:30,300 INFO CheckpointCoordinator - Completed checkpoint 1
(301356433 bytes, checkpointDuration=2759 ms)
# Medium state
2026-02-03 18:40:16,158 INFO CheckpointCoordinator - Completed checkpoint 30
(8771678128 bytes, checkpointDuration=49087 ms)
# Large state - still fast!
2026-02-03 19:50:28,xxx INFO CheckpointCoordinator - Completed checkpoint 78
(15327402522 bytes, checkpointDuration=76727 ms)
9.2 Presto S3 JobManager Logs
Sample Checkpoint Completions:
# Small state - similar to native
2026-02-03 18:31:33,079 INFO CheckpointCoordinator - Completed checkpoint 1
(81062497 bytes, checkpointDuration=2754 ms)
# Medium state - slowing down
2026-02-03 18:58:30,274 INFO CheckpointCoordinator - Completed checkpoint 20
(8714842002 bytes, checkpointDuration=95407 ms)
# Large state - very slow
2026-02-03 19:52:46,372 INFO CheckpointCoordinator - Completed checkpoint 40
(15491119926 bytes, checkpointDuration=165414 ms)
9.3 Native S3 TaskManager Warning
WARN S3TransferManager - The provided DefaultS3AsyncClient is not an instance of
S3CrtAsyncClient, and thus multipart upload/download feature is not enabled.
To benefit from maximum throughput, consider using S3AsyncClient.crtBuilder().build()
Note: Even without CRT optimization, Native S3 still outperforms Presto S3 by 2x.
10. Key Findings
10.1 Performance Summary
┌─────────────────────────────────────────────────────────────────────────────────┐
│ KEY FINDINGS │
├─────────────────────────────────────────────────────────────────────────────────┤
│ │
│ 1. Native S3 is 2x FASTER than Presto S3 at checkpoint throughput │
│ │
│ 2. Native S3 maintains ~200 MB/s throughput even at 15+ GB state │
│ │
│ 3. Presto S3 caps at ~90 MB/s regardless of state size │
│ │
│ 4. Native S3 completed 78 checkpoints vs Presto's 40 in same time │
│ │
│ 5. Both achieved 100% checkpoint success rate (no failures) │
│ │
│ 6. Native S3 has room for improvement with CRT client optimization │
│ │
└─────────────────────────────────────────────────────────────────────────────────┘
10.2 Architecture Differences
┌──────────────────────────────────┐ ┌──────────────────────────────────┐
│ NATIVE S3 │ │ PRESTO S3 │
├──────────────────────────────────┤ ├──────────────────────────────────┤
│ │ │ │
│ ┌────────────────────────────┐ │ │ ┌────────────────────────────┐ │
│ │ AWS SDK v2 │ │ │ │ Presto S3 Client │ │
│ │ (Modern, Async-first) │ │ │ │ (Sync, Blocking I/O) │ │
│ └────────────────────────────┘ │ │ └────────────────────────────┘ │
│ │ │ │ │ │
│ ▼ │ │ ▼ │
│ ┌────────────────────────────┐ │ │ ┌────────────────────────────┐ │
│ │ S3TransferManager │ │ │ │ Traditional HTTP │ │
│ │ (Parallel uploads) │ │ │ │ (Sequential uploads) │ │
│ └────────────────────────────┘ │ │ └────────────────────────────┘ │
│ │ │ │ │ │
│ ▼ │ │ ▼ │
│ ┌────────────────────────────┐ │ │ ┌────────────────────────────┐ │
│ │ Non-blocking I/O │ │ │ │ Blocking I/O │ │
│ │ max-concurrency: 64 │ │ │ │ max-connections: 100 │ │
│ └────────────────────────────┘ │ │ └────────────────────────────┘ │
│ │ │ │
│ Throughput: ~200 MB/s │ │ Throughput: ~90 MB/s │
│ │ │ │
└──────────────────────────────────┘ └──────────────────────────────────┘
11. Log Files
| File | Description | Size |
|---|---|---|
| Complete JobManager logs | 135 KB |
| TaskManager 1 logs | 219 KB |
| TaskManager 2 logs | 219 KB |
| REST API checkpoint data | 6.9 KB |
| Complete JobManager logs | 132 KB |
| TaskManager 1 logs | 190 KB |
| TaskManager 2 logs | 190 KB |
| REST API checkpoint data | 6.8 KB |
12. Cluster Configuration for Benchmarking
| Setting | Value |
|---|---|
| Node Group Name | flink-workers |
| Instance Type | m7i-flex.large (2 vCPU, 8GB RAM) |
| Desired Capacity | 5 nodes |
| Min Size | 3 nodes |
| Max Size | 7 nodes |
| Volume Size | 50 GB (gp3) |
| Labels | role: flink-worker |
13. Job Identifiers
| Job | Flink Job ID | Kubernetes Deployment | Checkpoints |
|---|---|---|---|
| Native S3 | 36c995738009b48e9043703b0cf721f8 | native-s3-benchmark | 78 |
| Presto S3 | 51bb9cca583ae9aff3b5544ffec9f921 | presto-s3-benchmark | 40 |