Here is a list of features that we have gathered thru hearing from different contributors to the product. Feel free to expand the definitions, elaborate, and ask questions thru this wiki. I tried to stay out of implementation details as much as possible.
This is not in a prioritized order. I will also send out a Survey to gather feedback with regards to the prioritization of these features in the list.
- Modularization of Airflow components
- As in common, scheduler, webservers, operators etc.
- Security improvements and Multi-Tenancy with roles and granular permissions.
- In order to limit access in an organization with multiple teams owning different set of DAGs with various access levels.
- Kerberos support
Putting the password information in a Vault. Currently if you have the key you have access to the whole vault.
Managing connection pools better.
- Defining the difference between a hook and connection.
- UI and CLI level goals
- Remove CLI direct DB access
- Support multiple / HA schedulers
- Related JIRA: - AIRFLOW-85Getting issue details... STATUS
- REST API
- Standardize way to have certain operations programmatically. These operations can include checking the status of a job run, kicking off jobs, marking dependencies.
- Deprecate Charting Application and Data Profiler
- Bolke: there was some feedback on this to leave it in
- Integration Testing Environment
- Many companies that are running Airflow and also contributing to the project are having a dilemma between having new features vs. stability. This dilemma is not limited to jumping onto the new versions of the software that is developed by other members of the community but even developing new features themselves. We need to have an Integration Testing environment for the project to mitigate the risk of destabilizing Airflow installations with the introduction of new features.
- Support for package management via Docker, Kubernetes, ECS etc.
- Stateless Webservers
- Better Log Management and Audibility
- Currently logs sink with dead workers.
- We have the DB triggers right now to track task instance state over time but this functionality should be backed into airflow. e.g. new TIs shouldn’t clobber old ones in the DB, same with dag runs.
- Revamped Dev/Test/Deploy
- Dev and Prod environments aren't in sync. This causes issues in pushing to production.
- Better validation of DAGs
- Better Documentation
- Onboarding documentation
- How the system works?
- Hardening the scheduler.
Documenting how the scheduler operates.
Running a single task continuously.
More visibility into Why do we need to restart scheduler?
Clarifying the contract between the scheduler and the workers(???)
- Backfill has a separate code path. It should be a flavor.
- Better Observability and Ops Management
- Event driven scheduler
First task snowballing and kicking other tasks. This reduces the white space between when task is runnable and makes sure the task is actually is run.
- Revamp the SubDAG operator
So many special cases in the code base.
- It should be handled by the scheduler.
- UI revamp
- Support for large DAGs
- Better performance possibly via assets caching, compression and use of CDN.
- Possibly Reactifying the UI.