DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
As FLIP-555: Flink Native S3 FileSystem - Apache Flink - Apache Software Foundation established, Flink's dual-connector setup (Hadoop/Presto) adds significant maintenance burden: transitive dependency conflicts, classpath issues, and CVE exposure from Hadoop. Hadoop also lacks features like client-side encryption with custom key providers.
This FLIP is the umbrella for the effort to support Hadoop-less filesystems within Flink that solely rely on cloud SDKs for AWS/Azure/GCS. It defines the common, cloud-agnostic abstractions that all SDK-native filesystem implementations share, placed in flink-core so they are written, tested, and maintained once. It also defines the intent to implement cloud agnostic features like enhanced observability by generalising native implementations (e.g. following up with cloud agnostic version of FLIP-576).
Relationship to Other FLIPs
FLIP | Scope |
|---|---|
This FLIP | Common abstractions in |
AWS S3 SDK-native filesystem ( | |
Filesystem-Plugin Observability ( | |
FLIP-TBD | Native Filesystem-Plugin Observability (cloud agnostic implementation of FLIP-576 to be shared) |
Azure SDK-native filesystem ( | |
GCS SDK-native filesystem ( | |
FLIP-TBD | Client-side encryption extension for SDK-native filesystems |
Future FLIP | Other clouds (e.g. AliCloud OSS SDK-native filesystem) |
Public Interfaces
All new interfaces are annotated @Internal and @Experimental. No existing @Public or @PublicEvolving APIs are modified.
Read Path
ReadContext
Immutable context passed to InputStreamOpener describing the desired read position.
@Internal
@Experimental
@Immutable
public interface ReadContext {
long getPos();
static ReadContext of(long pos);
}
InputStreamOpener
Cloud-agnostic functional interface that opens an InputStream at a given position. The implementation captures cloud-specific state (SDK client, path, bucket) in its closure.
@Internal
@Experimental
@FunctionalInterface
public interface InputStreamOpener {
InputStream open(ReadContext ctx) throws IOException;
}
InputStreamExtension
Extension point for ObjectStorageInputStream. Provides stream opening and lifecycle callbacks. All methods are called with the stream's internal lock held.
@Internal
public interface InputStreamExtension {
interface StreamContext {
long getPos();
long getContentLength();
}
RawAndWrappedInputStreams openStream(StreamContext ctx) throws IOException;
static InputStreamExtension buffering(InputStreamOpener opener, int readBufferSize);
}
RawAndWrappedInputStreams
Value class pairing the raw SDK stream with a wrapped stream (e.g., buffered or decrypting). ObjectStorageInputStream reads from wrapped() and closes sdk() for connection cleanup.
@Internal
public final class RawAndWrappedInputStreams {
public RawAndWrappedInputStreams(InputStream sdk, InputStream wrapped);
public InputStream sdk();
public InputStream wrapped();
}
ObjectStorageInputStream
Thread-safe FSDataInputStream implementation for object storage. Final class, extensible via InputStreamExtension.
@Internal
@ThreadSafe
public final class ObjectStorageInputStream extends FSDataInputStream {
public ObjectStorageInputStream(
long contentLength,
long skipOnSeekThreshold,
InputStreamExtension extension);
}
Key behaviors:
Lazy initialization: the underlying stream is not opened until the first read
Seek optimization: forward seeks within
skipOnSeekThresholduse read-and-discard; larger seeks close and reopen via the extensionThread safety: all public methods are guarded by
ReentrantLock;close()uses non-interruptible lock to guarantee cleanupDelegated lifecycle: lifecycle concerns (e.g., integrity verification on close) are handled by the wrapped stream, not by
ObjectStorageInputStreamitself- Non-owner abort: when close() is called from a thread other than the owner (the thread that constructed the stream), it reads the raw SDK stream reference lock-free and calls abort() on it if it implements Abortable, before acquiring the lock. This unblocks a read in progress on the owner thread promptly. The owner's blocked read propagates an IOException, the lock is released, and close() proceeds normally. Backends that do not implement Abortable (e.g., Azure) fall back to waiting for the lock.
Write Path
WriteContext
Immutable context passed to OutputStreamOpener carrying metadata to attach to the cloud object (e.g., encryption headers).
@Internal
@Experimental
@Immutable
public interface WriteContext {
Map<String, String> getMetadata();
WriteContext EMPTY_WRITE_CONTEXT = Collections::emptyMap;
static WriteContext of(Map<String, String> metadata);
}
OutputStreamOpener
Cloud-agnostic functional interface that creates an OutputStream for a write operation. The implementation captures cloud-specific state in its closure.
@Internal
@Experimental
@FunctionalInterface
public interface OutputStreamOpener {
OutputStream open(WriteContext ctx) throws IOException;
}
ObjectStorageOutputStream
Thread-safe FSDataOutputStream that delegates to a provided OutputStream. Final class.
@Internal
@ThreadSafe
public final class ObjectStorageOutputStream extends FSDataOutputStream {
public ObjectStorageOutputStream(OutputStream delegate, String filePath);
}
Key behaviors:
Position tracking: maintained under lock, incremented on each write
Commit-on-close:
sync()andclose()both flush and close the delegate, making the file visibleThread safety: same
ReentrantLockpattern as the input stream;close()andsync()use non-interruptible lock- Non-owner abort: when close() is called from a non-owner thread (e.g., task cancellation), it calls abort() on the delegate lock-free if the delegate implements Abortable, before acquiring the lock. This aborts an in-progress multipart upload or flush on the owner thread. sync() is always called from the owner thread and is unaffected.
Summary of New Types in flink-core
Class / Interface | Kind | Purpose |
|---|---|---|
| Interface | Read position descriptor |
| Interface | Write metadata descriptor (encryption headers) |
| Functional interface | Opens raw input stream at position |
| Functional interface | Opens raw output stream with metadata |
| Interface | Read stream opening contract |
| Interface | Read-only view of stream state |
| Class (package-private) | Default extension: open + buffer |
| Value class | Pairs raw SDK stream with wrapped stream |
| Final class | Thread-safe |
| Final class | Thread-safe |
| Interface (@Experimental) | Optional capability for SDK streams that support immediate resource release from a non-owner thread; discovered via instanceof |
| Interface (@Experimental) | Optional FS capability for re-encryption and ETag-guarded atomic rename; exposes pathsList, getRichFileStatus, moveVerified; discovered via instanceof |
| Interface (@Experimental) | Extends FileStatus with user blob metadata and ETag; returned by ObjectStorageFileSystem#getRichFileStatus |
ObjectStorageFileSystem and RichFileStatus
Two additional interfaces support re-encryption and ETAg-conditional rename semantics that emerge from advanced use-cases such as key rotation and re-encryption of stored data.
They are placed in flink-core alongside the stream types so that all filesystem plugins share one canonical type identity.
Key behaviors of ObjectStorageFileSystem :
- pathsList: returns a lazy, closeable iterator over all paths under the given prefix. Callers must close the iterator to release any paging cursors or SDK connections.
- getRichFileStatus: performs a single HEAD-equivalent call to fetch file status enriched with user-defined blob metadata and the current ETag. Used to read encryption headers and obtain a stable ETag before a conditional move.
- moveVerified: performs an ETag-guarded rename. The move succeeds only when the source ETag matches srcETag and, if dstETag is non-empty, the destination ETag matches dstETag. An empty dstETag asserts that the destination does not exist. Returns false if any guard fires (no data loss); throws IOException if the storage operation itself failed and the source state is unknown.
Filesystem implementations that support these semantics implement ObjectStorageFileSystem in addition to the standard FileSystem contract.
Callers discover support via instanceof, following the same pattern as Abortable.
Proposed Changes
Module Placement
All new types are placed in flink-core under org.apache.flink.core.fs, alongside the existing FileSystem and FSDataInputStream types. No new module dependencies are introduced.
The design is organized in three explicit layers:
- Layer 1 — Existing Flink Contracts (@Public): FileSystem, FSDataInputStream, FSDataOutputStream, RecoverableWriter. These are unchanged.
- Layer 2 — Shared Object-Storage Mechanics (@Internal / @Experimental): the new types introduced by this FLIP in flink-core. Shared by all provider plugins without additional module dependencies.
- Layer 3 — Provider-Specific SDK Adapters: per-cloud filesystem plugins that implement Layer 1 contracts using Layer 2 mechanics and a provider SDK. Recovery protocols, bulk copy, and storage operations remain provider-specific.
Design Principles
Composition Over Inheritance
Both stream classes are final. Extensible via composition: InputStreamExtension for reads, OutputStreamOpener for writes. This allows stacking multiple extensions without inheritance chains and avoids the fragile-base-class problem.
Thread Safety by Default
Both stream classes use ReentrantLock with a consistent pattern:
Interruptible lock for normal operations (
read,write,seek,getPos)Non-interruptible lock for cleanup operations (
close,sync)
This ensures that task cancellation cannot leave streams in an inconsistent state while still allowing cooperative interruption during normal I/O.
Both stream classes also support non-owner abort: when close() is called from a thread other than the owner, it checks the raw stream reference lock-free and calls abort() on it if the stream implements Abortable, before blocking on the lock. This allows a cancellation thread to unblock a blocked read or write on the owner thread quickly, without waiting for the lock. The Abortable interface is optional — backends that cannot abort (e.g., Azure, GCS) are a no-op on the cancel path and fall back to waiting for the lock.
Exception Preservation
When a filesystem method translates an SDK exception into a Flink exception, the original SDK throwable is always retained as the cause. This ensures that diagnostic information — HTTP status codes, provider error codes, request IDs — is never silently discarded. A later child FLIP may define per-provider classifiers (authentication, throttling, timeout, cancellation, precondition failure); those classifiers wrap, not replace, the SDK cause.
How Cloud Implementations Use These Abstractions
A cloud-specific filesystem would:
Implement
InputStreamOpener: capture the SDK client and blob path, return anInputStreamstarting atctx.getPos()using the cloud SDK's range-read APIImplement
OutputStreamOpener: capture the SDK client and blob path, attachctx.getMetadata()as object metadata/headers, return the SDK's uploadOutputStreamConstruct
ObjectStorageInputStream: pass content length, seek threshold, andInputStreamExtension.buffering(opener, bufferSize)for the default read pathConstruct
ObjectStorageOutputStream: pass the stream from the opener and the file path
Cross-cutting extensions provide a custom InputStreamExtension that wraps streams with additional processing, and inject metadata into WriteContext on the write side.
Forward Compatibility with Advanced Storage Features
Cloud providers expose capabilities beyond basic read/write: server-side copy (Azure Copy Blob, S3 CopyObject), zero-copy rename (Azure HNS), compare-and-swap via conditional writes with ETags, and bulk operations like batch delete. These all operate at the object level, not the stream level.
The abstractions in this FLIP are intentionally scoped to stream I/O. A cloud filesystem is a concrete FileSystem subclass and is free to override rename(), delete(), mkdirs(), and any future FileSystem methods using cloud-native APIs directly, without routing through the stream abstractions.
Compatibility, Deprecation, and Migration Plan
Compatibility
All new types are @Internal. No existing @Public or @PublicEvolving APIs are modified. Existing filesystem plugins are not affected.
Migration
New SDK-native filesystem implementations (Azure, GCS) use these abstractions from the start. flink-s3-fs-native (FLIP-555) can migrate in a follow-up. No configuration changes or user-facing migration steps are required.
Existing Hadoop-Based Implementations
The existing Hadoop-based filesystem plugins (flink-fs-hadoop-shaded, flink-azure-fs-hadoop, flink-s3-fs-hadoop) are annotated @Public and remain fully supported. This FLIP does not deprecate or remove them.
The rollout follows three phases:
Experimental. New SDK-native implementations ship as
@Experimental. Both Hadoop-based and SDK-native plugins are available side by side; users opt in to the new implementations explicitly.Stable / Deprecation. Once the SDK-native implementations are proven in production and reach feature parity, the Hadoop-based plugins are marked
@Deprecated. A deprecation notice in the release notes and documentation guides users to migrate.Removal. The Hadoop-based plugins are removed in a future major version (earliest Flink 3.0), following the standard Flink API deprecation policy.
Until phase 3, users who depend on Hadoop-specific behavior (e.g., custom Hadoop Configuration properties, FileSystem subclass casting) can continue using the existing plugins without changes.
Test Plan
Unit tests for ObjectStorageInputStream and ObjectStorageOutputStream covering core stream contracts: seek behavior, position tracking, lifecycle management, thread safety, and idempotent close.
Unit tests for the extension mechanism, verifying that custom InputStreamExtension implementations receive correct context and can wrap streams transparently.
Integration testing strategy depends on the outcome of the ObjectStorageOperations interface open question. If a common operations interface is adopted, a contract test suite against that interface would cover generic filesystem logic once, with cloud-specific implementations verified separately. Otherwise, each cloud filesystem FLIP defines its own integration tests.
Open Questions
RecoverableWriter
All native FS must support RecoverableWriter, however recovery mechanisms differ per cloud: S3 uses multipart upload with ETag checkpointing (FLIP-555), Azure uses block-list-based recovery, and GCS uses resumable uploads with byte-offset tracking. These may not share enough structure for a useful common interface. This poses open question whether cloud agnostic RecoverableWriter abstraction should be a part of this FLIP?
PathsCopyingFileSystem (Bulk Copy)
Should bulk copy optimization (PathsCopyingFileSystem) be in scope? Currently, only AWS has implementation for this interface.
ObjectStorageOperations Interface
Should this FLIP include a cloud-agnostic ObjectStorageOperations interface (analogous to FLIP-555's S3AccessHelper) that would allow a generalized ObjectStorageFileSystem base? The main benefit is testability: generic filesystem logic could be tested against a single contract rather than per cloud.
Cloud storage APIs diverge in hierarchical namespace support, rename semantics, consistency models, and listing behavior. If the cloud-specific surface is too large, the interface becomes a leaky abstraction. The stream-level abstractions in this FLIP are compatible with but do not require such an interface.
Rejected Alternatives
Inheritance-based design (abstract base class with protected hooks)
An abstract base stream class with protected hooks that cloud implementations override. Rejected because it prevents stacking multiple extensions without deep inheritance chains, introduces fragile base class problems with implicit hook ordering, and forces subclassing over composition.
Separate flink-fs-common module
Placing the abstractions in a new flink-fs-common module was considered. Rejected because the interfaces are lightweight, belong alongside FileSystem / FSDataInputStream, and flink-core already contains the stream base classes. A separate module would add an unnecessary dependency for all filesystem plugins.
