Page History

Planned AIPs

The following AIPs have been voted on by the Airflow community and are planned to be included in Airflow 2.0

Candidate AIPs

The following AIPs are under consideration for inclusion in Airflow 2.0

Needs more detail/AIP

This is a list of new features that we would like to see in Airflow 2.0 (that don't yet have AIPs or Jira tickets). These will need AIPs creating and voting on before they make it in to the roadmap.

In no particular order:

• Rework Subdags to be less "bolted-on" and more native to the scheduler
  
  There are all sorts of edge cases around subdags that result from the tasks in the subdag being run by another executor, instead of being handled and scheduled by the core Scheduler. We should make the scheduler "see in" to the Subdags and make it responsible for scheduling tasks. This should make subdags less error prone and more predictable. It may involve replacing/significantly changing the SubDagOperator
  
  
• Move (tested) components out of contrib folder
  https://lists.apache.org/thread.html/c880ef89f8cb4a0240c404f9372615b998c4a4eeca342651927d596c@%3Cdev.airflow.apache.org%3E
  
  
• Filter passwords/sensitive info from logs.
  
  Jenkins does this if the password comes from a connection - it would be good if we could do this too
• Allow Backfill runs to be handled by the scheduler/triggered from UI
  
  It would be nice to not need console access to run airflow backfill, and to have not it not stop if the SSH session is closed.
  
  Lots of details to work out here though around how this would work, where would it show up in UI, priority of tasks, ways of reducing concurrency/load to allow normal tasks to run etc.
• Rationalize HA in Connections
  
  Right now it is possible to create multiple connections with the same ID and some  Connections/hooks will support this and pick a random one from the list. This feature isn't well documented or understood (and the CLI doesn't support it as well as the UI for instance) so we should examine if this makes sense, or if we should support it individually in certain connection types instead.
• Publish "official" production ready docker image
  
  Many people deploy Airflow via docker and currently use puckel/docker-airflow, but it would be nice if we published our own one (mostly just so that we can be more prompt about publishing new releases to docker hub too)
• Make setting up HTTPS connections easier/more expected

Included changes

New webserver that Joy Gao has been working on.


Jira
server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key AIRFLOW-85

Jira
server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key AIRFLOW-1433

• New timezone feature
  

  Jira
  server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key AIRFLOW-288

Remove the legacy import style that's been deprecated since at least 1.8


Jira
server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key AIRFLOW-1922

Jira
server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key AIRFLOW-1942

• Move API out of experimental
• Remove Pickle support
• Update Airflow documentation on airflow.apache.org
  
  

  Jira
  server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key AIRFLOW-1924

• Refactor sensor packages
  https://github.com/apache/incubator-airflow/pull/2875

• 2910

• Front end/"browser" testing
  
  The Airflow UI is non trivial and there have been a number of JS/html bugs that could have been caught by better front-end testing.
  
  It has been suggested to look at Cypress for this over Selenium. What ever we choose we need to pay careful attention to avoid slow or flakey UI tests.

...