Status

StateCompleted
Discussion ThreadAIP-7 Simplified development workflow
JIRA

AIRFLOW-3611 - Getting issue details... STATUS

Created

$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())


Motivation

Currently, the workflow for submitting new contributions is very cumbersome. Just the process of running unit tests is a difficult process, be it that you want to run them locally or through creating your own TravisCI pipeline. Airflow tests depend on many external services and other custom setup, which makes it hard for contributors and committers to work on this codebase. CI builds have also been unreliable, and it is hard to reproduce the causes. Having contributors trying to emulate the build environment every time makes it easier to get to an "it works on my machine" sort of situation.

The goal of this proposal is to outline the work needed to make local testing significantly easier and standardise the best practices to contribute to the Airflow project.


What problem does it solve?

  • It's difficult to reproduce unit and integration tests locally
  • Documentation on running the integration tests is not comprehensive and does not make it easy to run tests

  • Integration tests are difficult to run in a local environment, given their intrinsic coupling of some of them with cloud services. See AIP-4 Automation of System Tests [Deps: AIP-47] for an example of this.

  • The current environment setup on TravisCI (the setup before the tests are run) takes around 4 minutes.
  • Different python versions and backend configuration are not easy to reproduce locally
  • Separate incubator-airflow-ci project is setup with image that is maintained independently from the main Docker image requiring double maintenance. Those images are fairly different.
  • The image needs to be rebuilt manually periodically

Why is this change needed?

  • Being able to reproduce Travis CI environment should help in solving stability issues and decrease time/money used by Travis CI builds
  • It will be easier to onboard new contributors and committers if the environment is easy to setup and use

Already completed (also part of AIP-10 Multi-layered and multi-stage official Airflow CI image )

  • Docker image in https://github.com/apache/airflow-ci is setup. Dockerised CI pipeline is used to run the tests (tick)
  • Setting up docker-compose for container orchestration and configuration (tick).
  • Building the image on DockerHub in incremental way (tick).
  • Using the image in CI environment using DockerHub registry as cache to allow incremental rebuilds (tick)
  • Running a regular cron job with no cache (to test building from the scratch)(tick)
  • Building and incremental rebuilding of the image for local development (tick)

It provides these features 

  • Tox is removed (tick)
  • Build script in CI docker image (tick)
  • Single source of truth for Airflow and CI airflow image (tick)
  • Ability to build/pull/push image automatically (tick)
  • Ability to use different DockerHub user for local development (tick)
  • Changes to sources/PIP packages/Apt packages can be atomic PRs in the same repository (tick)
  • Sizes of images are optimised similar to described to now deprecated Optimizing Docker Image Workflow (tick)

The implementation is based on work done by Polidea (provided by Jarek Potiuk ) scripts that address a few of this pain points https://github.com/PolideaInternal/airflow-breeze-gcp-extension. 

The features of the workflow/environment are described in the documentation in detail.

Here a summary of what's available currently:

  • automated pulling of necessary images (tick)
  • automated rebuild of images when needed (tick)
  • mounting local sources to within local container (tick)
  • easy start of all the dependencies and airflow container (tick)
  • initialising of local virtualenv (for local IDE to work) (tick)
  • choosing which Python version and Backend to use (tick)
  • easy running of commands and tests inside the docker (tick)
  • auto-complete for test names with run-tests command (tick)
  • initialise-database-only-once when running run-tests scripts (tick)
  • catchy name ("breeze") and logo (tick).
  • video showing basic features and usefulness of Breeze (tick)
  • extensive documentation (tick)

Video describing Breeze:

Possible spin-off changes (deferred to later PR):


11 Comments

  1. Jarek Potiuk

    Here is an example (Google Cloud) Related implementation that might be used as inspiration/base for some further work - as explained in the thread: https://lists.apache.org/thread.html/34932447daee4794be4765eff851de9fd68abf63a91ac81f9db1ef7a@%3Cdev.airflow.apache.org%3E

    https://github.com/PolideaInternal/airflow-breeze

  2. Jarek Potiuk

    Work on this AIP is related to AIP-10 Multi-layered and multi-stage official Airflow CI image . I implemented (and I will soon share with wider audience) a proposal of a nice working environment that I ported (removing any GCP-related features) from https://github.com/PolideaInternal/airflow-breeze. You can take a look at https://github.com/apache/airflow/pull/4543/files to see proposal of BREEZE.md documentation that documents the usage and proposal.

  3. Jarek Potiuk

    The current implementation in https://github.com/apache/airflow/pull/4543/files has those features:

    • multi-layered and multi-staging Dockerfile
    • tox is removed
    • build scripts are baked into the image (but both Dockerfile and scripts are part of the airflow repository)
    • separate Docker image per python version
    • image size reduced
    • image optimised for builds
    • CONTRIBUTING.md updated
    • I also added very user friendly './breeze' environment that manages the lifecycle of local environment - it will pull/build images from registry as needed
    • BREEZE.md added with description of the environment

    I am still gathering some stats on build time and sized and I am working on fixing all the tests and simplifying the Dockerfile (I have an idea that I need to try that will remove two stages and wheel packages - I think I can optimise those away looking at build times)

  4. Jarek Potiuk

    Gerardo Curiel  -  I think I have something that I will be ready to show community early next week. That will include a screencast movie where I will show how quickly you can get up and running with the "breeze" environment I am working on. I think it really goes hand-in-hand with the multi-layered Dockerfile work (because it then helps to solve the pain point of having separately built ci-image).

    I would love to be able to take-over the lead here over AIP-7 and update the document here according to my findings, proposal, and structure proposed in the recent "AIP" discussion on devlist. Since you are the original author of the AIP, I wanted to ask you if you are ok with that, I noticed that you have not been very active since it was created, so maybe I can continue the work and lead it? I'd love your feedback in general as well and will be super-happy to take suggestions and comments. Also one thing I have not touched yet was improving the kubernetes part. It needs separate image or as you mentioned running it on GKE cluster - but the latter will be difficult until we get some sponsorship for running a paid  GKE instance. But I would love to involve others at this point and maybe get help from someone on that while I will be working on fixing the remaining failing tests on Travis.

    What do you think Gerardo Curiel?

  5. Jarek Potiuk

    One more thought: I thought maybe (to simplify things and close some things first) maybe the Kubernetes/GKE discussion could be moved to separate AIP? I also work on GCP stuff and actually we are using our own project to run system tests automatically - using Cloud Build and talking to real GCP services. I even have an AIP for that : AIP-4 Automation of System Tests [Deps: AIP-47]  which I am going to take a look at after finishing Docker/Environment. I think once we have foundation with simplified workflow and nicely layered Dockerfile, we could start discussion and try to involve Google and get their interest in maybe donating some regular credits to the community ? I have a number of contacts at Google that I could try to ping and ask for feedback on that matter - but I feel it's much more political issue than technical one and maybe we can separate it out.

  6. Gerardo Curiel

    Jarek Potiuk Great work! And yes, please take the lead on this PR. I was on holidays but now I should become more active. I'll have a proper look at your PR today, but in the meantime, a few suggestions:

    • Small, self-contained PRs are better for the community to review and implement. In the case there is a problem, the changes can be easily reverted and can also help keep the history more concise for others developing PRs and merging changes
    • I agree with keeping the GKE stuff as a separate AIP for simplicity's sake
    • I'm not completely sold on naming the development environment. It becomes another thing for people to learn and differentiate. It would be best to keep it simple and just document it as just the "process to contribute to Airflow". Something as straightforward as https://github.com/kubernetes/community/tree/master/contributors/devel#readme. Sweet name though (smile).


  7. Jarek Potiuk

    Cool. I wil do some more work on documenting it then (smile). I also plan to do a screencast showing in 3-4 minutes the most important features of how breeze works. 

    And for the naming - yeah, I expected that naming might become an issue, I am ok to change it, but I simply love the "breeze" name in "Airflow" context. I think it fits perfectly, it's short and good name for a command line tool as well (especially when you add it to the PATH - there aren't many breeze's out there. I found It is a nice 'soft' touch and keeps your smile on while you use it. Emotions are important and this one is definitely positive.

    Happy to discuss with wider community what they think about the name and see what comes out of the discussion.

  8. Jarek Potiuk

    and yeah - I will make it a separate PR. It will be the whole of breeze script basically as I brought most of it from our development environment. If you can hold off with reviewing it until I split and rearrange it (today most likely) that can make it easier for you to review as I will separate out all the changes related to multi-layered Docker - which should be discussed and merged separately.

  9. Jarek Potiuk

    Gerardo Curiel → I split the PR of multi-layered docker into two commits and I created a Draft PR for that https://github.com/apache/airflow/pull/4932. Feel free to comment/review/try (smile).

  10. Jarek Potiuk

    Gerardo Curiel  -updated the documents to the new format. Please take a look. I will record a screencast tomorrow to show what you can do with Breeze.

  11. Jarek Potiuk

    tomorrow = this weekend (sad)