Date: June 12th 2024

Authors: Elad Kalif, Shahar Epstein

Goal

This doc presents the current state of who is the Airflow user?

followed by open questions for discussions about the Airflow user in Airflow 3.

Intro

Airflow defines a user as someone who interacts with the Airflow application. From Airflow’s point of view, as of version 2.x, the user has the knowledge, skills, and permissions to act on any Airflow components.
For example, if a user wants to set a non-standard scheduling interval for his DAG and the documentation  recommends using Timetables, then we presume that the feature exists and the problem is solved.
However,  from the user's perspective, the problem may not be solved as they might not have the skills needed to register new plugins, or have the skills but not the permissions.
In some organizations, especially large corporates,  the authors of pipelines may not have full access to all of the components of the Airflow application.


Airflow already acknowledges the existence of different actors, as can be seen in the architecture docs. This doc will focus on:

  1. Who are the actors?
  2. What are their qualifications?
  3. What is the actor's scope?

Actors 

An actor can be seen as an Airflow user with a specific role. In smaller deployments, all actors can be single users while in larger deployments there can be many actors with conflicting interests.
The conflicts may be caused by different agenda, roadmaps, etc.

For example, I need to deploy a plugin that requires the assistance of the actor in charge in this area, but they don’t have the time to do it.

Considering the existing architecture of Airflow, we could distinguish between four basic types of actors:

  1. Operational User - Interacts with Airflow’s UI on a DAG/DAG Run specific basis. Normally triggers a run manually via pre-defined form in Airflow's UI.
    This actor may have no technical knowledge and even not be from the data domain realm (for example, product managers).

  2. DAG Author - Creates, modifies, or removes DAGs from the system. May interact with various Airflow components (Connections, Variables, Xcoms, etc..). Within this type, we could recognize two subtypes:
    1. Technical Dag Author - interacts with .py files directly.
    2. Non-technical DAG Author - interacts with an interface on top of .py files (for example, building DAGs via .yaml files).

  3. Deployment Manager - owns the deployment. They have the ability to deploy plugins, install Python packages/providers, and handle the scale of workers, webserver, and schedulers.
    They also decides on which executor to use, configuring the Airflow’s settings, and handling zombie or undead tasks. This actor may also enforce Cluster Policy, and possibly owns the CI/CD process of DAGs’s deployment.

  4. Infrastructure Manager - Owns the serving of the compute/storage/systems to be served for the deployment of Airflow. For example, owns the Kubernetes cluster, DBs available in the company, etc.
    This actor is sometimes external to the Airflow app and isn’t familiar with Airflow.



Actor

Required Skills

Required permissions

Responsibility

Notes

Operational user

Non-technical

UI

run/view existing DAGs


Dag Author

Python Knowledge

UI, API, DAG folder

Deploy DAGs


Deployment Manager

Python, Infrastructure

All components of Airflow

Health and stability of the application


Infrastructure Manager

Provides DB/K8s cluster to run the Airflow application


Sometimes, they’re not even aware of Airflow.

Airflow is just one of the many applications that run on the infrastructure.



It’s important to note that with multi-tenancy, there might be additional sub-actors.

For example, the deployment manager creates a new tenant. Each tenant has an operational user and DAG authors, but also a tenant deployment manager who manages the infrastructure of the tenant.

Airflow 3 - Open Questions

  1. Are we happy with the current model? If not, what changes do we hope to make?
  2. Can we find mechanisms to reduce the conflicts between the different actors?
  3. Should we define and discuss how AIPs affect each one of the actors?
  • No labels

13 Comments

  1. Constance Martineau

    I love this! If we're defining "Infrastructure Manager", I also think it's important to have a category for business stakeholders to represent the "customers" of DAG Authors. Something like this?

    Business Stakeholders - Data Consumers

    • Role: Utilize the outputs generated by Airflow pipelines for making business decisions, generating reports, and other data-driven activities.
    • Interactions: Primarily consume the end results of Airflow DAGs through dashboards, reports, and analytics tools. They rely on accurate and timely data to inform strategic and operational decisions.
    • Technical Knowledge: Generally have a low level of technical expertise in Airflow itself but have a good understanding of the data and its implications for the business.
    • Responsibilities:
      • Defining requirements for data reports and analytics.
      • Requesting new data pipelines or modifications to existing ones to meet business needs.
      • Evaluating the quality and reliability of the data provided.
    • Examples: Business analysts, product managers, executive team members.

  2. Daniel Standish

    there might be a slightly more technical persona than "dag author who touches .py files" e.g. at some organizations you might have people who do most of the work of writing shared operators.

  3. Daniel Standish

    why do we use the word "actor" instead of "persona"?  ("persona" seems more natural to me FWIW)

  4. Daniel Standish

    However,  from the user's perspective, the problem may not be solved as they might not have the skills needed to register new plugins, or have the skills but not the permissions.

    That sounds reasonable, but what are the implications of that?  What's the "so what"?  Sorta wondering what is the goal of articulating these personas, and how it will be useful.  I think the personas that we articulate will depend on the context, on what we're trying to accomplish with them.


    1. Constance Martineau

      It's important to have a shared understanding of attributes and skillsets of the persona you are targeting with a feature so that it is ultimately useful for them.

      Continuing with the Timetables example: This is a feature meant to be used by DAG authors and is underutilized. They were designed to be defined and implemented by deployment managers. There were valid reasons from a security perspective to go this route, but if we don't expect DAG authors to create and install plugins, and we don't expect Deployment Managers to author DAGs, it means the design from a usability perspective wasn't right and the feature should have probably been implemented differently, or perhaps not at all. 

      TLDR: Shared understanding of personas is critical to defining proper requirements

  5. Philippe Lanoe

    What about a viewer role/Actor ? It looks like a basic role to me. For instance a data analyst might want to check what is going on with a pipeline to be able to launch an analysis query in a downstream system.

    1. Elad Kalif

      this is mentioned as operational user

      1. Philippe Lanoe

        It is only mentioned as one of the responsibility/right as an operational user. A viewer user only (external to any job/resources management in Airflow) is a different role/persona/actor as an Operator, which is meant to operate.

        1. Jarek Potiuk

          I think we should clarify that this document is not to define detelailed roles - it does not map to roles of Airflow 1-1, it's not a security document. It's more to define who will be the broad types of users who we want to address in Airflow 3. The Operations user is anyone who wants to use Airflwo via UI to check (and possibly act on) the status of running DAGs. The actor does not Author DAGs , does not deploy Airflow, does not run other workflows that should become part of Airflow DAG - so we name it operator. That covers broad spectrum of view/manage running actions - with the common part that those actors are accessing Airflow via UI to see how things are (and possibly deal with the situation). 

          1. Philippe Lanoe

            I agree, because "An actor can be seen as an Airflow user with a specific role" is ambiguous. Two additional comments:

            • You, along with the description of the Operator mention "Airflow via UI", but if I am not mistaken the interaction is not only limited to UI since the component being interacted with is the webserver, which can also serve the API. (It is one of my inline comment in the table)
            • I agree with the above comment that the description or name is not yet clear for the Operational User. For instance "Normally triggers a run manually" is only one of the (many?) use cases and "normally" depends on the role of the user. I am fine to have one Actor concept to gather all "RBAC Airflow Webserver User roles", but in this case I do not fully get your comment on the deployment manager (team vs organization) thread because if I understood correctly AIP-67 the org deployment manager may cover the scope of team deployment manager, and will create team deployment manager at his/her discretion. In that sense it could be viewed as "details" and could fall under the same higher level concept of a deployment manager.

            1. Jarek Potiuk

              > You, along with the description of the Operator mention "Airflow via UI", but if I am not mistaken the interaction is not only limited to UI since the component being interacted with is the webserver, which can also serve the API. (It is one of my inline comment in the table)

              Yes and No (smile) . For me API is universally usable for generally all actors.

              Of course the UI provides "operation" in the way community decided Airflow UI might be useful, while API allows to extend it via external UI/Integrations with other systems - so from that point of view, it's what you wrote.

              But API is much more than that - it is a cross between "Operation" users,  "Deployment Manager" and "DAG Author" - and any of those actors should be able to use the API. Deployment manager ((n Airflow 3 it will be the only way) can for example extend Airflow via listeners and plugins and use API for that, as well as DAG authors will have to use the API to interact with Airflow if they will want to do more than what "Task SDK" will allow them to do (again in Airflow 3 it will be the only way, you won't be able to use DB for that). I recently had - somewhat heated - discussion about it here - https://github.com/apache/airflow/discussions/40296 explaining that DB access will no longer be possible for user code, and API will have to be used for all that purposes.

              >  I agree with the above comment that the description or name is not yet clear for the Operational User. For instance "Normally triggers a run manually" is only one of the (many?) use cases and "normally" depends on the role of the user. I am fine to have one Actor concept to gather all "RBAC Airflow Webserver User roles", but in this case I do not fully get your comment on the deployment manager (team vs organization) thread because if I understood correctly AIP-67 the org deployment manager may cover the scope of team deployment manager, and will create team deployment manager at his/her discretion. In that sense it could be viewed as "details" and could fall under the same higher level concept of a deployment manager.

              Agree. In this context - possibly there is no need to separate those two.



              1. Philippe Lanoe

                On the API topic: I am not denying that the API can be used by other Actors (each one of them will use a subset of it), just saying that it is also a possibility for the Operational User to use the API. I can imagine in some (crazy?) secure/strict environment these users will only have terminal access to access Airflow Webserver and therefore use the API only.

                It is why I do not understand the "no" part and I think your sentence should be rephrased to "The Operational user is anyone who wants to use Airflow via UI or API to check (and possibly act on) the status of running DAGs". I am not arguing about the second part of the sentence in this this thread, I wrote a comment on the responsibility column.


  6. Jarek Potiuk

    One thing that I would also add here. I think we are focusing a bit on the actor's access (security) and means (UI/API). This document should mostly define what kind  of people are interacting with Airflow, in what way, and what is expected of them. What they should understand, what knowledge and skills they possess and what they want to achieve by interacting with Airflow.

    For example - I think Deployment Managers, should know how to install, setup, deploy and monitor airflow "Application" running, they should be able to inspect all kinds of signals and deployment details, know what Airflow configuration parameters do etc. Whether they use API, UI, or CLI for that is secondary. Similarly - DAG authors, do not have to know how Airflow works and even how it is deployed or configured. They should be able to write/edit and delete DAG files. The "non-python DAG authors" might not even know Python, but they should be able to define and modify DAG structure. Whether they do it via DAG files, DAG factories, or UI (for non-Python DAG authors we might open that possibility) it's also secondary - and the way where and how they get access is mostly a consequence of their needs. Similarly Operation users, should not even need to know how DAGs are defined - are they Python files? YAML? some other ways - they don't care. They want to be able to observe and monitor "pipelines" and be able to interact with them (clear, rerun, backfill) - again, whether they use UI, CLI or API it's secondary.

    I really would love if we focus only on capabilites and needs here and leave out the means that the Actors are interacting with airflow.