This Confluence has been LDAP enabled, if you are an ASF Committer, please use your LDAP Credentials to login. Any problems file an INFRA jira ticket please.

Skip to end of metadata
Go to start of metadata

Here is a list of features that we have gathered thru hearing from different contributors to the product. Feel free to expand the definitions, elaborate, and ask questions thru this wiki. I tried to stay out of implementation details as much as possible.

This is not in a prioritized order. I will also send out a Survey to gather feedback with regards to the prioritization of these features in the list.

 

PackagingSecurityAPITesting
  •  Modularize Airflow components (api, common, scheduler, webserver, operators)

 

Confidentiality

  • Multi-tenancy with roles and permissions. In order to limit access in an organization with multiple teams owning different set of DAGs with various access levels.
  • Full Kerberos support
  • Remove CLI direct DB access
  • Putting the password information in a Vault. Currently if you have the key you have access to the whole vault.

  • UI and CLI level goals
  • Role based access to DAGs

Integrity

  • Manage connection pools better
  • Uniquely attribute task instances to dag runs by using a foreign key (database)

Availability

  • Support multiple schedulers / HA schedulers
  • Stateless webservers

Related JIRA:  AIRFLOW-85 - Getting issue details... STATUS

  • Standardize way to have certain operations programmatically. These operations can include checking the status of a job run, kicking off jobs, marking dependencies.
  • Use SRV record for API auto configuration
  • Make tests more atomic
  • Split between integration tests and unit tests
  • Split tests per file against which they are testing
  • Integration Test Environment. Many companies that are running Airflow and also contributing to the project are having a dilemma between having new features vs. stability. This dilemma is not limited to jumping onto the new versions of the software that is developed by other members of the community but even developing new features themselves. We need to have an Integration Testing environment for the project to mitigate the risk of destabilizing Airflow installations with the introduction of new features.

DocumentationUISchedulerOps
  • Onboarding documentation
  • Runbook
  • How the system works?
  • Support for large DAGs
  • Better performance possibly via assets caching, compression and use of CDN.
  • Possibly Reactifying the UI.
  • Use API instead of server side rendering
  • Select multiple tasks and mark them as success

  • Make the scheduler notify the webserver of new/removed/updated dags
  • Documenting how the scheduler operates.

  • Running a single task continuously.

  • More visibility into Why do we need to restart scheduler? (Or just not need it to)

  • Clarifying the contract between the scheduler and the executors

  • Backfill has a separate code path. It should be a flavor.
  • Event driven scheduler.
    First task snowballing and kicking other tasks. This reduces the white space between when task is runnable and makes sure the task is actually is run.

  • Revamp the SubDAG operator
    • So many special cases in the code base. 

    • It should be handled by the scheduler.
  • Flexible prioritization
  • Turn off backfill/catchup (in the works)
  • Allow "cron-like" behavior (ie. not start_date + interval as first execution)
  • Generate tokens for tasks to obtain their configuration in a secure manner
Executors   
  • Make the LocalExecutor run out of process (or remove it)
  • Add AWS Batch, Slurm, Torque (Community)
  • Add Kafka
   

 

 

 

 

 

  1. Modularization of Airflow components 
    • As in common, scheduler, webservers, operators etc.
  2. Security improvements and Multi-Tenancy with roles and granular permissions.
    • In order to limit access in an organization with multiple teams owning different set of DAGs with various access levels. 
    • Kerberos support
    • Putting the password information in a Vault. Currently if you have the key you have access to the whole vault.

    • Managing connection pools better.

    • Defining the difference between a hook and connection.
    • UI and CLI level goals
    • Remove CLI direct DB access
    • Support multiple / HA schedulers
    • Related JIRA:  AIRFLOW-85 - Getting issue details... STATUS
  3. REST API
    • Standardize way to have certain operations programmatically. These operations can include checking the status of a job run, kicking off jobs, marking dependencies.
  4. Deprecate Charting Application and Data Profiler
    1. Bolke: there was some feedback on this to leave it in
  5. Integration Testing Environment
    • Many companies that are running Airflow and also contributing to the project are having a dilemma between having new features vs. stability. This dilemma is not limited to jumping onto the new versions of the software that is developed by other members of the community but even developing new features themselves. We need to have an Integration Testing environment for the project to mitigate the risk of destabilizing Airflow installations with the introduction of new features.
  6. Containers
    • Support for package management via Docker, Kubernetes, ECS etc. 
  7. Stateless Webservers
  8. Better Log Management and Audibility 
    • Currently logs sink with dead workers.
    • We have the DB triggers right now to track task instance state over time but this functionality should be backed into airflow. e.g. new TIs shouldn’t clobber old ones in the DB, same with dag runs.
  9. Revamped Dev/Test/Deploy
    • Dev and Prod environments aren't in sync. This causes issues in pushing to production.
    • Better validation of DAGs
    • Versioning
  10. Better Documentation
    • Onboarding documentation
    • Runbook
    • How the system works?
  11. Hardening the scheduler.
    • Documenting how the scheduler operates.

    • Running a single task continuously.

    • More visibility into Why do we need to restart scheduler?

    • Clarifying the contract between the scheduler and the workers(???)

    • Backfill has a separate code path. It should be a flavor.
  12. Better Observability and Ops Management
  13. Event driven scheduler 
    • First task snowballing and kicking other tasks. This reduces the white space between when task is runnable and makes sure the task is actually is run.

  14. Revamp the SubDAG operator
    • So many special cases in the code base.

    • It should be handled by the scheduler.
  15. UI revamp
    • Support for large DAGs
    • Better performance possibly via assets caching, compression and use of CDN.
    • Possibly Reactifying the UI.

 

 

 

   

  • No labels