Airflow Dev Calls [ARCHIVED]

Dev Calls Calendar

We have regular calls between Airflow Developers (PMC Members, Committers and community members) to discuss various issues – progress on next releases, issue triage, tracking milestone items, etc.

6 Jan 2021

Attendees

Summary

• Airflow 2.0.1 / Open Issue from 2.0:
  • Milestone: Airflow 2.0.1 Milestone · GitHub
  • Add ‘priority’ labels to all the open issues in 2.0.1 Milestone
  • Release 2.0.1 as soon as the reported bugs are fixed
  • Positive feedback from Scribd – Going PROD with 2.0 in 2 weeks. No major bugs reported by Astronomer, Google / Polidea and Amazon.
• Upgrade Checks
  • Release 1.2.0 soon with a fix for https://github.com/apache/airflow/issues/13293
  • Add a Rule to Check version of Database (MySQL / Postgres / SQLite)
• Providers
  • Elasticsearch and Google (breaking change - 2.0.0) providers would be released soon or at least RCs would be cut next week
• Release Process - Semantic Versioning
  • Changing the version of dependency should not count as breaking change.
  • Ash is working on the doc which will be PRd to Airflow soon
  • Still no unanimous agreement on whether "Major Version should contain new features or just removal of deprecation"
  • Highlight this process to all the committers (chances are some committers are not regularly active). The most important one is that we will strictly follow SEMVER.
• Airflow 2.1 - cadence, candidates and planning
  • Patch releases 2-3 weeks (Might be driven by the bugs and their severity)
  • Minor releases would be released every two months
  • No cadence for Major releases!
  • What happens to the PR that introduces breaking changes??
  • We should not merge that PR until it is a breaking change, instead we should make the PR backwards compatible
  • Or wait until next major release (assign appropriate Github Milestone)

20 Jan 2021

Attendees

Summary

• Airflow 2.0.1:
  • Milestone: Airflow 2.0.1 Milestone · GitHub
  • Target week of Feb 8 for release
• Providers Release
  • Next set of Providers will be released next week once https://github.com/apache/airflow/pull/13767 is merged with appropriate SEMVER version (e.g Google would be a major release while ES would be a patch release)
  • Backport Providers will be released for the ones that have some significant changes after or at the same time as the providers
  • Snowflake released a new version (v2.3.8) and the Snowflake provider which will use this version will be released too as currently all versions for Snowflake providers are YANKED.
    • This will be released after https://github.com/apache/airflow/pull/12188 is merged as it contains some incompatibilities with Azure Blob Storage
• Upgrade Checks
  • Release 1.2.0 soon with a fix for https://github.com/apache/airflow/issues/13293
  • Add a check for importing Operator/Sensor via Plugins if it does not already exist
  • Add docs in 1.10.15 about using upgrade-checks to upgrade to 2
• Helm Chart Release
  • We need to agree on what we call the first version: 0.0.1 vs 2.0.0
  • We need to review the current Helm chart before it is released as a lot of changes have been merged since it was first merged.
  • Current Test Coverage is not that great - ideally, we would like to cover more test cases before it can be released 
  • We hope to release it after 2.0.1 so bugs fixes are in by then and 2.0.1 can become the default version for the Helm Chart

3 Feb 2021

Attendees

Summary

• Airflow 2.0.1:
  • Milestone: Airflow 2.0.1 Milestone · GitHub
  • Will be released on Feb 8
• Providers Release
  • Providers will be released on Feb 8
  • Backport Providers should also be released as along with Providers, assuming vote passes
• Upgrade Checks
  • 1.2.0 will be released soon after Airflow 2.0.1 once the vote passes
• Apache Airflow 1.10.15
  • Milestone: https://github.com/apache/airflow/milestone/22?closed=1
  • CI is fixed - v1-10-stable passing
  • A release date has not been decided yet
• Improve Airflow Observability
  
  • John to create Github Issue around what metrics would be useful to avoid failures early from an Operational point of view. Based on that we can discuss and recommend using existing metrics or add missing metrics.
• Make Doc contributions easier
  • We should have a separate section in CONTRIBUTING.rst for users who want to create PR for only doc-changes.
  • Currently, it is difficult for new contributors to understand how to add cross-links, using sphinx, building docs locally, debugging spelling-check errors.
  • Adding some recipes/templates in the CONTRIBUTING.rst with code to create Cross-references, adding links, spell-check, etc using Sphinx would make it easy.

17 Feb 2021

Attendees

Summary

• Pending Item:
  
  • John Jackson to create Github Issue to improve Airflow's observability story
  • Vikram Koka to create Github Issue/PR to add a guide to help new contributors make doc-only contributions
  • Ash Berlin-Taylor to complete the Draft PR on Release Docs / Version Policy. Jarek Potiuk to follow it up to add version policy for Providers
• Github Issues Stats
  • Vikram presented some stats and charts around our Github Issues
    • Kubernetes was an area that needs attention. 
• Apache Airflow 1.10.15
  • Milestone: https://github.com/apache/airflow/milestone/22?closed=1
  • Tentative Date: 1 Mar 2021
• Helm Chart
  
  • Project Board: https://github.com/apache/airflow/projects/7 to track the work
  • Tentative Release Date: Early April
  • The first release will be 1.0.0

3 March 2021

Attendees

Summary

• Add a policy document around all our Policies / rules in the Github Repo itself
• This should cover any decision we take as PMC / Committers, for example:
• Versioning Policy - Core, Providers and all others
• Committers have a binding vote on AIP Releases
• Lazy Consensus thread will be created on the mailing list for each decision agreed on the Dev Call
• Providers Release Policy
• Batch vs Ad-hoc:
• Release Manager would default to release Providers in Batch
• ad-hoc releases are OK (i.e. if there is a critical bug that needs fixing in a single provider)
• Frequency:
• For Batch release, we will release every month (starting of the month - 1 to 7 most likely)
• Just a note that it generally takes around a week for the vote to pass even though we have 72 hours min period
• Testing
• License and Signature Checks are mandatory (following the ASF rules)
• For Providers, not all changes require strict testing -- you make a judgement based on the changes for a particular provider
• For Airflow core it is different, smoke tests should be carried out and we rely on community testing too (Kaxil to create a separate Mailing list thread to discuss this further since there wasn't 100% agreement)
• Doc-only changes
• When we have doc-only changes for Providers (during batch-release), we should still release a new version. Majority on the Dev call agreed that releasing docs asap is good instead of waiting for the next release with a code-change.
• Feedback on AIPs
• AIP-39 Richer scheduler_interval 
• Foreign keys between TaskInstance and DagRun will be changed from execution_date and dag_id to dagrun_id.
• This and any other changes with columns will need migration that might be slow for deployments with many Dags and large TIs or DagRuns in Metadata DB.
• Kevin Yang from Airbnb said that they have tried something similar but instead of dagrun_id they used DagRun.id (auto-incrementing id for each DagRun) which had a performance impact.
• AIP-38 Modern Web Application is already being Voted on: link
• TaskGroups support in Tree View
• We agreed that this would be good to have since we are planning to remove SubDags for TaskGroups
• A separate Mailing list thread has been created for more details and further discussion: link