DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
Flink’s YARN Application Mode recommends using yarn.provided.lib.dirs with a pre-uploaded Flink distribution on HDFS to make job submission lightweight and reuse cached artifacts across applications.
However, in real deployments this often leads to one (or both) of the following issues:
Operational packaging constraints: Some organizations store shared dependencies as compressed archives (e.g.,
.tgz/.tar.gz) rather than as a directory containing many individual.jarfiles, to reduce small-file overhead and simplify distribution.Classpath length / “argument too long” failures: When many jars are added explicitly to the container classpath (especially combined with multiple
yarn.provided.lib.dirsentries), the generated launch command / environment can exceed OS limits, causing “argument too long” errors.
We want a small, targeted improvement that:
Allows users to point Flink at pre-uploaded, world-readable archive files in HDFS archive files, scoped to jar dependencies (see Public Interfaces).
Ensures jar inclusion into the classpath without enumerating every jar as a separate classpath entry (reducing command length)
Preserves the existing behavior for users who do not opt in.
Public Interfaces
New configuration option
Add:
yarn.provided.lib.archives(List<String>, semicolon-separated)Each entry is a remote archive path (e.g., HDFS) pointing to a pre-uploaded and world-readable archive.
Supported archive types:
.tar.gz,.tgz- Scoped narrowly to jars only. Non-jar resources, flink-dist, and plugins remain the responsibility of yarn.provided.lib.dirs; archives are additive alongside lib.dirs, not a replacement.
Behavioral changes (only when configured)
When yarn.provided.lib.archives is set, Flink will:
Register each archive as a YARN
LocalResourceof type ARCHIVE, which is automatically unarchived by the NodeManager.Add jars from the localized archive to the classpath using a wildcard form, top-level jars only (see limitation in Proposed Changes step 4), to avoid excessively long classpath strings.
No changes for users who do not use the new option.
Proposed Changes
High-level behavior
Read and validate config
Parse
yarn.provided.lib.archivessimilarly toyarn.provided.lib.dirs.Validate each path is remote-accessible from all nodes (same constraint as
yarn.provided.lib.dirs).
Register provided archives as shared local resources
In the YARN upload/localization step, register each archive as:
LocalResourceVisibility.PUBLIC(so it can be reused/cached by NodeManagers across applications),LocalResourceType.ARCHIVE(so it will be unarchived automatically).
Classpath construction using wildcards
Instead of enumerating every jar contained in the archive as a separate classpath entry, add a single wildcard entry:
<archive-file-name>/*
Example (conceptual): if
flink-libs.tgzis localized, addflink-libs.tgz/*to the classpath.This design directly targets the “argument too long” failure mode by keeping classpath entries bounded even when archives contain many jars.
Important limitation (initial scope)
Only jars in the immediate child directory of the unarchived archive are added via the wildcard.
Jars nested deeper (e.g.,
flink-libs.tgz/foo/bar/baz.jar) are not picked up byflink-libs.tgz/*.Users must package archives accordingly (jars at top-level inside the archive).
- Archives must not contain flink-dist, plugin jars, or non-jar resources — those remain governed by yarn.provided.lib.dirs. Jar ordering within the wildcard is unspecified and archive contents must not conflict with each other or with lib.dirs jars (duplicate classes/service providers/logging bindings).
- Validation
- Reject configuration at submission time if:
- yarn.provided.lib.archives is set with no flink-dist source available (no lib.dirs, no local distribution).
- two configured archives resolve to the same localized resource name (mirrors existing getAllFilesInProvidedLibDirs dedup check).
- Reject configuration at submission time if:
Relationship to existing configs
yarn.provided.lib.dirsremains the recommended path for users who already publish directories of jars.yarn.ship-archivesalready exists, but it is for shipping archives per-application submission and is not intended as a “provided shared libs” mechanism that replaces uploading local Flink libs.- flink-dist is never placed inside an archive's wildcard entry and it keeps its current explicit handling via lib.dirs/local distribution. Submission fails fast if no such source is configured (see step 5).
Compatibility, Deprecation, and Migration Plan
Backward compatible: No behavioral change unless
yarn.provided.lib.archivesis set.Migration: Users who currently rely on provided libs but package them as archives can:
Upload
.tgz/.tar.gzto HDFS, ensure world-readable permissions,Set
yarn.provided.lib.archivesto point to those files,Ensure the archive layout places jars at the archive’s top-level directory.
- Archives must contain only jar dependencies. Flink-dist, plugins, and non-jar resources must remain in
yarn.provided.lib.dirs.
No deprecations in this FLIP.
Security Considerations
Archives use the same visibility model as yarn.provided.lib.dirs today: LocalResourceVisibility.PUBLIC, unlike the PRIVATE/APPLICATION visibility used for regular per-application shipped resources, which are not cached or reused across applications. PUBLIC requires world-readable paths, documented but not enforced by Flink. Archive unpacking is delegated entirely to YARN's NodeManager, not done by Flink itself. Deeper YARN-internals security review is out of scope; reviewers with more YARN security background are encouraged to comment.
Test Plan
Unit tests
Validate parsing and remote-path checks for
yarn.provided.lib.archives.Verify archive registration uses
LocalResourceType.ARCHIVE.Verify classpath entries include
<archiveName>/*rather than enumerated jar paths.- Verify submission fails when yarn.provided.lib.archives is set with no flink-dist source available.
- Verify submission fails when two archives resolve to the same localized resource name.
- Verify flink-dist/plugin/non-jar content is never sourced from an archive.
- Jar-ordering/conflict-avoidance within an archive is a documented user constraint (see Proposed Changes, step 4), not independently enforced or tested in code.
Integration / e2e test (YARN)
Submit a YARN application-mode job with:
yarn.provided.lib.archivespointing to an HDFS archive containing multiple jars at top-level
Verify:
Job starts successfully,
Required classes are loadable from jars in the archive,
No “argument too long” error is triggered even with many jars.
Rejected Alternatives
Client-side pre-extraction: extract the archive locally and register individual files instead of using YARN's ARCHIVE type is rejected, defeats NodeManager managed caching and can reintroduce the long classpath problem. Extend wildcard support to yarn.provided.lib.dirs directly instead of adding a new config is rejected, does not address the motivation of shipping a single pre-packaged archive artifact.