This page describes a proposed the Flink Improvement Proposal (FLIP) process for proposing a major change to Flink.To create your own FLIP, click on "Create" on the header and choose "FLIP-Template" other than "Blank page".
Table of Contents
Purpose
The purpose of FLIPs is to have a central place to collect and document planned major enhancements to Apache Flink. While JIRA is still the tool to track tasks, bugs, and progress, the FLIPs give an accessible high level overview of the result of design discussions and proposals. Think of FLIPs as collections of major design documents for user-relevant changes.
...
This means when making this kind of change we need to think through what we are doing as best we can prior to release. And as we go forward we need to stick to our decisions as much as possible. All technical decisions have pros and cons so it is important we capture the thought process that lead to a decision or design to avoid flip-flopping needlessly.
Hopefully we can make these proportional in effort to their magnitude — small changes should just need a couple brief paragraphs, whereas large changes need detailed design discussions.
This process also isn't meant to discourage incompatible changes — proposing an incompatible change is totally legitimate. Sometimes we will have made a mistake and the best path forward is a clean break that cleans things up and gives us a good foundation going forward. Rather this is intended to avoid accidentally introducing half thought-out interfaces and protocols that cause needless heartburn when changed. Likewise the definition of "compatible" is itself squishy: small details like which errors are thrown when are clearly part of the contract but may need to change in some circumstances, likewise performance isn't part of the public contract but dramatic changes may break use cases. So we just need to use good judgement about how big the impact of an incompatibility will be and how big the payoff is.
...
All of the following are public interfaces that people build around:
- DataStream, DataSet, SQL and DataSet Table API, including classes related to that, such as StreamExecutionEnvironment
- Classes marked with the @Public annotation
- On-disk binary formats, such as checkpoints/savepoints
- User-facing scripts/command-line tools, i.e. bin/flink, Yarn scripts, Mesos Kubernetes scripts
- Configuration settings
- Exposed monitoring information
...
Anyone can initiate a FLIP but you shouldn't do it unless you have an intention of getting the work done to implement it (otherwise it is silly).
Process
Here is the process for making a FLIP:
...
Create your Own FLIP
- If you are an Apache Flink committer, create a page which is a child of this one. You can do that by either clicking on "Create" in the header and choose "FLIP-Template" (and not "Blank page") to create your own FLIP. Take the next available FLIP number (see under "FLIP round-up" and give your proposal a descriptive heading. e.g. "FLIP 42: Enable Flink Streaming Jobs to stop gracefully".
- If you don't have the necessary permissions for creating a new page, please create a Google Doc and make that view-only. As a FLIP number, please use FLIP-XXX.
- Post that Google Doc to the mailing list for a discussion thread. When the discussions have been resolved, the contributor ask on the Dev mailing list to a committer/PMC to copy the contents from the Google Doc, and create a FLIP number for them. The contributor can then use that FLIP
to actually have a VOTE thread.
- Post that Google Doc to the mailing list for a discussion thread. When the discussions have been resolved, the contributor ask on the Dev mailing list to a committer/PMC to copy the contents from the Google Doc, and create a FLIP number for them. The contributor can then use that FLIP
Process
Here is the process for making a FLIP:
- Follow the instructions at "Create your Own FLIP" development mailing list.
- Fill in the sections as described above
- Start a [DISCUSS] thread on the Apache mailing list. Please ensure that the subject of the thread is of the format [DISCUSS] FLIP-{your FLIP number} {your FLIP heading} The discussion should happen on the mailing list not on the wiki since the wiki comment system doesn't work well for larger discussions. In the process of the discussion you may update the proposal. You should let people know the changes you are making. You either include the link to the FLIP page on Confluence, or you link to the view-only Google Doc.
- Once the proposal is finalized and there are no more open discussions
- If your FLIP is already in Confluence, proceed to step 5.
- If your FLIP is a Google Doc, please ask on the Dev mailing list to copy the contents from your Google Doc to a FLIP page, and create a FLIP number for you before proceeding to step 5.
- call a [VOTE] to have the proposal adopted. These proposals are more serious than code changes and more serious even than release votes. The criteria for acceptance is lazy majority consensus.
- Please update the FLIP wiki page , and the index below, to reflect the current stage of the FLIP after a vote. This acts as the permanent record indicating the result of the FLIP (e.g., Accepted or Rejected). Also report the result of the FLIP vote to the voting thread on the mailing list so the conclusion is clear.
FLIP round-up
Next FLIP Number: 57
Use this number as the identifier for your FLIP and increment this value.
Adopted/Accepted but unreleased FLIPs
...
FLIP-29: Support map/flatMap/aggregate/flatAggregate on TableAPI
...
...
...
...
...
FLIPs under discussion
...
It's not unusual for a FLIP proposal to take long discussions to be finalized. Below are some general suggestions on driving FLIPs towards consensus. Notice that these are hints rather than rules. Contributors should make pragmatic decisions in accordance with individual situations.
- The progress of a FLIP should not be long blocked on an unresponsive reviewer. A reviewer who blocks a FLIP with dissenting opinions should try to respond to the subsequent replies timely, or at least provide a reasonable estimated time to respond.
- A typical reasonable time to wait for responses is 1 week, but be pragmatic about it. Also, it would be considerate to wait longer during holiday seasons (e.g., Christmas, Chinese New Year, etc.).
- We encourage FLIP proposers to actively reach out to the interested parties (e.g., previous contributors of the relevant part) early. It helps expose and address the potential dissenting opinions early, and also leaves more time for other parties to respond while the proposer works on the FLIP.
- Committers should use their veto rights with care. According to the ASF policy, vetos must be provided with a technical justification showing why the change is bad. They should not be used for simply blocking the process so the voter has more time to catch up.
FLIP round-up
Next FLIP Number: 448
Use this number as the identifier for your FLIP and increment this value.
Under discussion
Page properties report headings Discussion thread sortBy Title cql label = "discussion" and space = currentSpace()
Accepted
Page properties report headings Release,JIRA sortBy Title cql label = "accepted" and space = currentSpace()
Released
Page properties report headings Release,JIRA sortBy Title cql label = "released" and space = currentSpace()
Discarded
Page properties report headings Reason sortBy Title cql label in ("discarded","abandoned","rejected") and space = currentSpace()
...
FLIP-33: Standardize Connector Metrics
...
Implemented and Released FLIPs
...
FLIP-6 - Flink Deployment and Process Model - Standalone, Yarn, Mesos, Kubernetes, etc.
...
...
...
...
Discarded FLIPs
...
FLIP-28: Long-term goal of making flink-table Scala-free
...