Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

As FLIP-555: Flink Native S3 FileSystem - Apache Flink - Apache Software Foundation established, Flink's dual-connector setup (Hadoop/Presto) adds significant maintenance burden: transitive dependency conflicts, classpath issues, and CVE exposure from Hadoop. Hadoop also lacks features like client-side encryption with custom key providers.

This FLIP is the umbrella for the effort to support Hadoop-less filesystems within Flink that solely rely on cloud SDKs for AWS/Azure/GCS. It defines the common, cloud-agnostic abstractions that all SDK-native filesystem implementations share, placed in flink-core so they are written, tested, and maintained once. It also defines the intent to implement cloud agnostic features like enhanced observability by generalising native implementations (e.g. following up with cloud agnostic version of FLIP-576).

Relationship to Other FLIPs

FLIP

Scope

This FLIP

Common abstractions in flink-core for all SDK-native filesystems

FLIP-555

AWS S3 SDK-native filesystem (flink-s3-fs-native)

FLIP-576

Filesystem-Plugin Observability (flink-s3-fs-native)

FLIP-TBD

Native Filesystem-Plugin Observability (cloud agnostic implementation of FLIP-576 to be shared)

FLIP-598

Azure SDK-native filesystem (flink-azure-fs-native)

FLIP-603: [FileSystems] GCS SDK-native filesystem

GCS SDK-native filesystem (flink-gcs-fs-native)

FLIP-TBD

Client-side encryption extension for SDK-native filesystems

Future FLIP

Other clouds (e.g. AliCloud OSS SDK-native filesystem)

Public Interfaces

All new interfaces are annotated @Internal and @Experimental. No existing @Public or @PublicEvolving APIs are modified.

Read Path

ReadContext

Immutable context passed to InputStreamOpener describing the desired read position.

@Internal
@Experimental
@Immutable
public interface ReadContext {
    long getPos();

    static ReadContext of(long pos);
}
InputStreamOpener opener = ctx -> {
    DataLakeFileClient fileClient = fsClient.getFileClient(path);
    DataLakeFileInputStreamOptions opts = new DataLakeFileInputStreamOptions();
    if (ctx.getPos() > 0) {
        opts.setRange(new FileRange(ctx.getPos()));
    }
    return fileClient.openInputStream(opts).getInputStream();
};

InputStreamOpener

Cloud-agnostic functional interface that opens an InputStream at a given position. The implementation captures cloud-specific state (SDK client, path, bucket) in its closure.

@Internal
@Experimental
@FunctionalInterface
public interface InputStreamOpener {
    InputStream open(ReadContext ctx) throws IOException;
}

InputStreamExtension

Extension point for ObjectStorageInputStream. Provides stream opening and lifecycle callbacks. All methods are called with the stream's internal lock held.

@Internal
public interface InputStreamExtension {

    interface StreamContext {
        long getPos();
        long getContentLength();
    }

    RawAndWrappedInputStreams openStream(StreamContext ctx) throws IOException;

    static InputStreamExtension buffering(InputStreamOpener opener, int readBufferSize);
}


RawAndWrappedInputStreams

Value class pairing the raw SDK stream with a wrapped stream (e.g., buffered or decrypting). ObjectStorageInputStream reads from wrapped() and closes sdk() for connection cleanup.

@Internal
public final class RawAndWrappedInputStreams {
    public RawAndWrappedInputStreams(InputStream sdk, InputStream wrapped);
    public InputStream sdk();
    public InputStream wrapped();
}
Example: buffered read vs. decrypting read
// Buffering: sdk stream wrapped in BufferedInputStream
public RawAndWrappedInputStreams openStream(StreamContext ctx) throws IOException {
    InputStream raw = opener.open(ReadContext.of(ctx.getPos()));
    return new RawAndWrappedInputStreams(raw, new BufferedInputStream(raw, readBufferSize));
}

// CSE: sdk stream wrapped in a decrypting stream
public RawAndWrappedInputStreams openStream(StreamContext ctx) throws IOException {
    InputStream raw = opener.open(ReadContext.of(0L));
    InputStream decrypting = decryptorFactory.apply(new BufferedInputStream(raw, readBufferSize));
    if (ctx.getPos() > 0L) {
        decrypting.skipNBytes(ctx.getPos());
    }
    return new RawAndWrappedInputStreams(raw, decrypting);
}

ObjectStorageInputStream

Thread-safe FSDataInputStream implementation for object storage. Final class, extensible via InputStreamExtension.

@Internal
@ThreadSafe
public final class ObjectStorageInputStream extends FSDataInputStream {
    public ObjectStorageInputStream(
            long contentLength,
            long skipOnSeekThreshold,
            InputStreamExtension extension);
}

Key behaviors:

  • Lazy initialization: the underlying stream is not opened until the first read

  • Seek optimization: forward seeks within skipOnSeekThreshold use read-and-discard; larger seeks close and reopen via the extension

  • Thread safety: all public methods are guarded by ReentrantLock; close() uses non-interruptible lock to guarantee cleanup

  • Delegated lifecycle: lifecycle concerns (e.g., integrity verification on close) are handled by the wrapped stream, not by ObjectStorageInputStream itself

  • Non-owner abort: when close() is called from a thread other than the owner (the thread that constructed the stream), it reads the raw SDK stream reference lock-free and calls abort() on it if it implements Abortable, before acquiring the lock. This unblocks a read in progress on the owner thread promptly. The owner's blocked read propagates an IOException, the lock is released, and close() proceeds normally. Backends that do not implement Abortable (e.g., Azure) fall back to waiting for the lock.


Example: constructing a write stream
// Plain read
return new ObjectStorageInputStream(
        contentLength,
        skipOnSeekThreshold,
        InputStreamExtension.buffering(opener, readBufferSize));

// CSE read
return cseStreamFactory.openEncryptedRead(opener, encryptedReadContext);

Write Path

WriteContext

Immutable context passed to OutputStreamOpener carrying metadata to attach to the cloud object (e.g., encryption headers).

@Internal
@Experimental
@Immutable
public interface WriteContext {
    Map<String, String> getMetadata();

    WriteContext EMPTY_WRITE_CONTEXT = Collections::emptyMap;

    static WriteContext of(Map<String, String> metadata);
}

OutputStreamOpener

Cloud-agnostic functional interface that creates an OutputStream for a write operation. The implementation captures cloud-specific state in its closure.

@Internal
@Experimental
@FunctionalInterface
public interface OutputStreamOpener {
    OutputStream open(WriteContext ctx) throws IOException;
}
Example: Azure Data Lake file upload
OutputStreamOpener opener = ctx -> {
    DataLakeFileClient fileClient = fsClient.getFileClient(path);
    DataLakeFileOutputStreamOptions opts = new DataLakeFileOutputStreamOptions()
            .setMetadata(ctx.getMetadata());
    return fileClient.getOutputStream(opts);
};


ObjectStorageOutputStream

Thread-safe FSDataOutputStream that delegates to a provided OutputStream. Final class.

@Internal
@ThreadSafe
public final class ObjectStorageOutputStream extends FSDataOutputStream {
    public ObjectStorageOutputStream(OutputStream delegate, String filePath);
}


Key behaviors:

  • Position tracking: maintained under lock, incremented on each write

  • Commit-on-close: sync() and close() both flush and close the delegate, making the file visible

  • Thread safety: same ReentrantLock pattern as the input stream; close() and sync() use non-interruptible lock

  • Non-owner abort: when close() is called from a non-owner thread (e.g., task cancellation), it calls abort() on the delegate lock-free if the delegate implements Abortable, before acquiring the lock. This aborts an in-progress multipart upload or flush on the owner thread. sync() is always called from the owner thread and is unaffected.


Example: constructing a write stream
// Plain write
return new ObjectStorageOutputStream(
        opener.open(WriteContext.EMPTY_WRITE_CONTEXT), filePath);

// CSE write
OutputStream cseStream = cseStreamFactory.openEncryptedWrite(
        opener, encryptedWriteContext);
return new ObjectStorageOutputStream(cseStream, filePath);


Class / Interface

Kind

Purpose

ReadContext

Interface

Read position descriptor

WriteContext

Interface

Write metadata descriptor (encryption headers)

InputStreamOpener

Functional interface

Opens raw input stream at position

OutputStreamOpener

Functional interface

Opens raw output stream with metadata

InputStreamExtension

Interface

Read stream opening contract

InputStreamExtension.StreamContext

Interface

Read-only view of stream state

BufferingInputStreamExtension

Class (package-private)

Default extension: open + buffer

RawAndWrappedInputStreams

Value class

Pairs raw SDK stream with wrapped stream

ObjectStorageInputStream

Final class

Thread-safe FSDataInputStream for object storage

ObjectStorageOutputStream

Final class

Thread-safe FSDataOutputStream for object storage

Abortable 

Interface (@Experimental)

Optional capability for SDK streams that support immediate resource release from a non-owner thread; discovered via instanceof

ObjectStorageFileSystem 

Interface (@Experimental)

Optional FS capability for re-encryption and ETag-guarded atomic rename; exposes pathsList, getRichFileStatus, moveVerified; discovered via instanceof

RichFileStatus 

Interface (@Experimental)

Extends FileStatus with user blob metadata and ETag; returned by ObjectStorageFileSystem#getRichFileStatus

ObjectStorageFileSystem and RichFileStatus

Two additional interfaces support re-encryption and ETAg-conditional rename semantics that emerge from advanced use-cases such as key rotation and re-encryption of stored data.

They are placed in flink-core alongside the stream types so that all filesystem plugins share one canonical type identity.


@Internal
@Experimental
public interface ObjectStorageFileSystem {

    /** Lazily enumerates all file paths under {@code prefix}, recursively. Must be closed. */
    CloseableIterator<Path> pathsList(Path prefix) throws IOException;

    /** Fetches file status with user metadata and ETag populated (one HEAD per call). */
    RichFileStatus getRichFileStatus(Path path) throws IOException;

    /**
     * ETag-guarded atomic rename: moves {@code src} → {@code dst} only if ETags match.
     * Empty {@code dstETag} means dst must not exist. Returns {@code false} if any guard fired;
     * throws {@link IOException} if the storage call itself failed (src state undefined).
     */
    boolean moveVerified(Path src, Path dst, String srcETag, String dstETag) throws IOException;
}

/** {@link FileStatus} enriched with cloud-object user metadata and an ETag. */
@Internal
@Experimental
public interface RichFileStatus extends FileStatus {

    /** User-defined blob metadata (e.g. Azure {@code PathProperties.getMetadata()}). */
    Map<String, String> getMetadata();

    /** ETag of the object at fetch time; used for {@link ObjectStorageFileSystem#moveVerified}. */
    String getETag();
}


Key behaviors of ObjectStorageFileSystem :

  • pathsList: returns a lazy, closeable iterator over all paths under the given prefix. Callers must close the iterator to release any paging cursors or SDK connections.
  • getRichFileStatus: performs a single HEAD-equivalent call to fetch file status enriched with user-defined blob metadata and the current ETag. Used to read encryption headers and obtain a stable ETag before a conditional move.
  • moveVerified: performs an ETag-guarded rename. The move succeeds only when the source ETag matches srcETag and, if dstETag is non-empty, the destination ETag matches dstETag. An empty dstETag asserts that the destination does not exist. Returns false if any guard fires (no data loss); throws IOException if the storage operation itself failed and the source state is unknown.

Filesystem implementations that support these semantics implement ObjectStorageFileSystem  in addition to the standard FileSystem  contract.

Callers discover support via instanceof, following the same pattern as Abortable.

Proposed Changes

Module Placement

All new types are placed in flink-core under org.apache.flink.core.fs, alongside the existing FileSystem and FSDataInputStream types. No new module dependencies are introduced.

The design is organized in three explicit layers:


Mermaid source
graph TB
    subgraph L1["Existing Flink Contracts (flink-core, @Public)"]
        FS["FileSystem"]
        IS["FSDataInputStream"]
        OS["FSDataOutputStream"]
        RW["RecoverableWriter"]
    end

    subgraph L2["Shared Object-Storage Mechanics (flink-core, @Internal/@Experimental)"]
        direction LR
        OSIS["ObjectStorageInputStream"]
        OSOS["ObjectStorageOutputStream"]
        ISE["InputStreamExtension"]
        ISO["InputStreamOpener"]
        OSO["OutputStreamOpener"]
        RC["ReadContext"]
        WC["WriteContext"]
        RWS["RawAndWrappedInputStreams"]
        AB["Abortable"]
        OSFS["ObjectStorageFileSystem"]
        RFS["RichFileStatus"]
    end

    subgraph L3["Provider-Specific SDK Adapters"]
        AZ["AzureDataLakeFileSystem\n(flink-azure-fs-native)"]
        S3["NativeS3InputStream\n(flink-s3-fs-native)"]
        GCS["GCS native FS\n(future, FLIP-603)"]
    end

    L1 --> L2
    L2 --> L3


  • Layer 1 — Existing Flink Contracts (@Public): FileSystem, FSDataInputStream, FSDataOutputStream, RecoverableWriter. These are unchanged.
  • Layer 2 — Shared Object-Storage Mechanics (@Internal / @Experimental): the new types introduced by this FLIP in flink-core. Shared by all provider plugins without additional module dependencies.
  • Layer 3 — Provider-Specific SDK Adapters: per-cloud filesystem plugins that implement Layer 1 contracts using Layer 2 mechanics and a provider SDK. Recovery protocols, bulk copy, and storage operations remain provider-specific.


Design Principles

Composition Over Inheritance

Both stream classes are final. Extensible via composition: InputStreamExtension for reads, OutputStreamOpener for writes. This allows stacking multiple extensions without inheritance chains and avoids the fragile-base-class problem.

Thread Safety by Default

Both stream classes use ReentrantLock with a consistent pattern:

  • Interruptible lock for normal operations (read, write, seek, getPos)

  • Non-interruptible lock for cleanup operations (close, sync)

This ensures that task cancellation cannot leave streams in an inconsistent state while still allowing cooperative interruption during normal I/O.

Both stream classes also support non-owner abort: when close() is called from a thread other than the owner, it checks the raw stream reference lock-free and calls abort() on it if the stream implements Abortable, before blocking on the lock. This allows a cancellation thread to unblock a blocked read or write on the owner thread quickly, without waiting for the lock. The Abortable interface is optional — backends that cannot abort (e.g., Azure, GCS) are a no-op on the cancel path and fall back to waiting for the lock.

Exception Preservation

When a filesystem method translates an SDK exception into a Flink exception, the original SDK throwable is always retained as the cause. This ensures that diagnostic information — HTTP status codes, provider error codes, request IDs — is never silently discarded. A later child FLIP may define per-provider classifiers (authentication, throttling, timeout, cancellation, precondition failure); those classifiers wrap, not replace, the SDK cause.

Example of correct exception wrapping:
// Correct: SDK exception is the cause
throw new FileNotFoundException("Path not found: " + path + " - " + e.getMessage(), e);

// Incorrect: SDK cause is dropped (do not do this)
throw new FileNotFoundException("Path not found: " + path);


How Cloud Implementations Use These Abstractions

A cloud-specific filesystem would:

  1. Implement InputStreamOpener: capture the SDK client and blob path, return an InputStream starting at ctx.getPos() using the cloud SDK's range-read API

  2. Implement OutputStreamOpener: capture the SDK client and blob path, attach ctx.getMetadata() as object metadata/headers, return the SDK's upload OutputStream

  3. Construct ObjectStorageInputStream: pass content length, seek threshold, and InputStreamExtension.buffering(opener, bufferSize) for the default read path

  4. Construct ObjectStorageOutputStream: pass the stream from the opener and the file path

Cross-cutting extensions provide a custom InputStreamExtension that wraps streams with additional processing, and inject metadata into WriteContext on the write side.

Forward Compatibility with Advanced Storage Features

Cloud providers expose capabilities beyond basic read/write: server-side copy (Azure Copy Blob, S3 CopyObject), zero-copy rename (Azure HNS), compare-and-swap via conditional writes with ETags, and bulk operations like batch delete. These all operate at the object level, not the stream level.

The abstractions in this FLIP are intentionally scoped to stream I/O. A cloud filesystem is a concrete FileSystem subclass and is free to override rename(), delete(), mkdirs(), and any future FileSystem methods using cloud-native APIs directly, without routing through the stream abstractions.

Compatibility, Deprecation, and Migration Plan

Compatibility

All new types are @Internal. No existing @Public or @PublicEvolving APIs are modified. Existing filesystem plugins are not affected.

Migration

New SDK-native filesystem implementations (Azure, GCS) use these abstractions from the start. flink-s3-fs-native (FLIP-555) can migrate in a follow-up. No configuration changes or user-facing migration steps are required.

Existing Hadoop-Based Implementations

The existing Hadoop-based filesystem plugins (flink-fs-hadoop-shaded, flink-azure-fs-hadoop, flink-s3-fs-hadoop) are annotated @Public and remain fully supported. This FLIP does not deprecate or remove them.

The rollout follows three phases:

  1. Experimental. New SDK-native implementations ship as @Experimental. Both Hadoop-based and SDK-native plugins are available side by side; users opt in to the new implementations explicitly.

  2. Stable / Deprecation. Once the SDK-native implementations are proven in production and reach feature parity, the Hadoop-based plugins are marked @Deprecated. A deprecation notice in the release notes and documentation guides users to migrate.

  3. Removal. The Hadoop-based plugins are removed in a future major version (earliest Flink 3.0), following the standard Flink API deprecation policy.

Until phase 3, users who depend on Hadoop-specific behavior (e.g., custom Hadoop Configuration properties, FileSystem subclass casting) can continue using the existing plugins without changes.

Test Plan

Unit tests for ObjectStorageInputStream and ObjectStorageOutputStream covering core stream contracts: seek behavior, position tracking, lifecycle management, thread safety, and idempotent close.

Unit tests for the extension mechanism, verifying that custom InputStreamExtension implementations receive correct context and can wrap streams transparently.

Integration testing strategy depends on the outcome of the ObjectStorageOperations interface open question. If a common operations interface is adopted, a contract test suite against that interface would cover generic filesystem logic once, with cloud-specific implementations verified separately. Otherwise, each cloud filesystem FLIP defines its own integration tests.

Open Questions

RecoverableWriter

All native FS must support RecoverableWriter, however recovery mechanisms differ per cloud: S3 uses multipart upload with ETag checkpointing (FLIP-555), Azure uses block-list-based recovery, and GCS uses resumable uploads with byte-offset tracking. These may not share enough structure for a useful common interface. This poses open question whether cloud agnostic RecoverableWriter abstraction should be a part of this FLIP?

PathsCopyingFileSystem (Bulk Copy)

Should bulk copy optimization (PathsCopyingFileSystem) be in scope? Currently, only AWS has implementation for this interface.

ObjectStorageOperations Interface

Should this FLIP include a cloud-agnostic ObjectStorageOperations interface (analogous to FLIP-555's S3AccessHelper) that would allow a generalized ObjectStorageFileSystem base? The main benefit is testability: generic filesystem logic could be tested against a single contract rather than per cloud.

Cloud storage APIs diverge in hierarchical namespace support, rename semantics, consistency models, and listing behavior. If the cloud-specific surface is too large, the interface becomes a leaky abstraction. The stream-level abstractions in this FLIP are compatible with but do not require such an interface.

Rejected Alternatives

Inheritance-based design (abstract base class with protected hooks)

An abstract base stream class with protected hooks that cloud implementations override. Rejected because it prevents stacking multiple extensions without deep inheritance chains, introduces fragile base class problems with implicit hook ordering, and forces subclassing over composition.

Placing the abstractions in a new flink-fs-common module was considered. Rejected because the interfaces are lightweight, belong alongside FileSystem / FSDataInputStream, and flink-core already contains the stream base classes. A separate module would add an unnecessary dependency for all filesystem plugins.