Status
Motivation
The Command-Line Interface (CLI) for Airflow includes various types, such as those with direct database access, those operating locally, those running core Airflow components, and those that combine these features. Integrating API functionality into the CLI can provide isolated and secure database access while enabling features like built-in authentication. This integration facilitates the seamless deployment of CLI and API features together, reducing redundancy and simplifying maintenance. Transitioning from direct database access to an API-driven model will enhance the CLI’s capabilities and improve security. A gateway to bridge CLI commands with API interactions will support this shift, requiring new definitions and security protocols to ensure effective and secure communication.
Considerations
What change do you propose to make?
Definitions:
- Core Commands: CLI commands for managing Airflow.
- API Commands: CLI commands designed to interface with the API.
It is proposed to categorize Core Commands into the following groups. Categorizing commands will help identify which ones can be replaced with API Commands. This means an easier transition period if needed.
- Airflow Management Commands: Commands for tasks such as running the webserver, scheduler, internal API server, or operating Airflow in standalone mode.
- Database Access Commands: Commands involving direct database access, such as listing DAGs, and managing tasks, connections, and variables.
- Hybrid Commands: Commands that manage Airflow while also requiring direct database access.
API Commands will support all necessary API interactions, following a consistent naming convention (to be defined), such as:
airflow dags list airflow taskinstances clear dag_id task_id
API commands should be auto-generated as much as possible from configuration files (such as OpenAPI or the chosen API technology). This approach will minimize the number of changes required for new or updated API endpoints, ensuring consistency and reducing manual effort. It would be good to have a tool to auto-generate these commands.
Security Considerations:
- API Commands
- Existing Security Policies: Utilize current security policies for API endpoints, such as permissions required for listing DAGs.
- Core Commands
- Airflow Management Commands: Commands like the webserver and scheduler, which prepare or create environments and do not require prior authorization, should remain unchanged.
- Database Access Commands: Commands that involve direct database access should transition to API Commands, ensuring all database interactions are handled through the API with appropriate authentication and permissions.
- Hybrid Commands: Commands that manage Airflow and also involve direct database access should route database communication through API methods. Examples include loading DagBag and accessing DagRun from commands used by Celery and Kubernetes executors. These commands often operate with daemon options, minimizing instances where both direct database access and management functionalities are needed simultaneously.
The CLI should be secure-by-design. This means any sensitive data used by the CLI should be handled properly, such as being passed through stdin or retrieved from environment variables. Ensuring secure data handling practices will protect against potential security vulnerabilities and maintain user trust.
Command Inventory with Command Type Categorisation
Command | Command Type | Notes |
cheat_sheat_command | Core Command | |
dag-processor | Core Command | |
info | Core Command | |
internal-api | Core Command | |
kerberos | Core Command | |
plugin | Core Command | |
scheduler | Core Command | |
standalone | Core Command | |
triggerer | Core Command | |
version | Core Command | Plan to be a Hybrid Command where it will show the CLI and the Airflow version. CLI version will be able to run locally. The Airflow version should be returned from API, please check Check Command/Subcommand and API Endpoint Mapping. |
webserver | Core Command | |
config | Direct DB Access | Fully REST Integration Needed. Check Command/Subcommand and API Endpoint Mapping. |
connections | Direct DB Access | Fully REST Integration Needed. Check Command/Subcommand and API Endpoint Mapping. |
dags | Direct DB Access | Fully REST Integration Needed. Check Command/Subcommand and API Endpoint Mapping. |
db | Direct DB Access | Fully REST Integration Needed. Check Command/Subcommand and API Endpoint Mapping. |
jobs | Direct DB Access | Fully REST Integration Needed. Check Command/Subcommand and API Endpoint Mapping. |
pools | Direct DB Access | Fully REST Integration Needed. Check Command/Subcommand and API Endpoint Mapping. |
tasks | Direct DB Access | Fully REST Integration Needed. Check Command/Subcommand and API Endpoint Mapping. |
variables | Direct DB Access | Fully REST Integration Needed. Check Command/Subcommand and API Endpoint Mapping. |
rotate-fernet-key | Hybrid Command | Fully REST Integration Needed. Check Mapping for endpoints. |
provider | Hybrid Command | DB connections will be replaced by API flavour. |
aws | Hybrid Command | DB connections will be replaced by API flavour. |
celery | Hybrid Command | Actual CLI command implementation in the core is deprecated. From provider apache-airflow-providers-celery. DB connections will be replaced by API flavour. |
kubernetes | Hybrid Command | Actual CLI command implementation in the core is deprecated. From provider apache-airflow-providers-cncf-kubernetes. DB connections will be replaced by API flavour. |
fab | Hybrid Command | DB connections will be replaced by API flavour. |
Command/Subcommand and API Endpoint Mapping
Command | Subcommand | REST Endpoint Name (with Additional Notes) |
config | get_value | Get a option from configuration |
config | list | Get current configuration |
connections | add | Create a connection |
connections | delete | Delete a connection |
connections | export | Missing: Export connection(s) from a file Additional Note: This could be done via parsing the JSON locally and calling the API endpoints accordingly. I think this would cause too many API requests. I think we can handle this one in a single request via accepting multipart file |
connections | get | Get a connection |
connections | import | Missing: Import connection(s) from a file Additional Note: This could be done via parsing the JSON locally and calling the API endpoints accordingly. I think this would cause too many API requests. I think we can handle this one in a single request via accepting multipart file |
connections | list | List connection |
connections | test | Test a connection |
dags | backfill | Missing: Backfill a dag Additional note: Rely on AIP-78 |
dags | delete | Delete a DAG |
dags | details | Get basic information about a DAG |
dags | list | List DAGs |
dags | list-import-errors | List dag warnings |
dags | list-jobs | Missing: List Jobs |
dags | list-runs | List DAG runs |
dags | next-execution | Get basic information about a DAG |
dags | pause | Update a DAG |
dags | report | Combination of List DAGs and List task instances |
dags | reserialize | Missing: Reserialize a DAG |
dags | show | Missing: Show a DAG (Graph) |
dags | show-dependencies | Missing: Show Dependencies of a DAG |
dags | state | Get a Dag run |
dags | test | Missing: Test a DAG |
dags | trigger | Trigger a new DAG run. |
dags | unpause | Update a DAG |
db | init: deprecated over migrate check check-migrations clean downgrade export-archived migrate reset shell | Missing: We should call all the db methods of `db.py` from endpoints to cover each cases Additional Note: I think we should disable `shell` command since this shouldn't be allowed in airflow CLI context. This should part of user's process which is basically connecting to a database with it's CLI tools/UI tools. |
jobs | check | Missing: List Jobs with Filter (like State, job_type hostname) Additional Note: We should make a decision if we want to support the local part. I think it should use the same approach where API calls for the local deployment. |
pools | list | List pools |
pools | get | Get a pool |
pools | export | Missing: Import pool(s) from a file Additional Note: This could be done via parsing the JSON locally and calling the API endpoints accordingly. I think this would cause too many API requests. I think we can handle this one in a single request via accepting multipart file |
pools | import | Missing: Import pool(s) from a file Additional Note: This could be done via parsing the JSON locally and calling the API endpoints accordingly. I think this would cause too many API requests. I think we can handle this one in a single request via accepting multipart file |
pools | delete | Delete a pool |
pools | set | Update a pool |
tasks | clear | Missing: Clear task instance Additional Note: Rely on AIP-72 Task SDK Updates the state of a task instance endpoint isn't enough to provide wider scope for clearing multiple task instance at the same time such as providing start-date, only-failed etc... |
tasks | failed-deps | Missing: List failed dependencies of a task instance Additional Note: Rely on AIP-72 Task SDK |
tasks | list | List task instances Additional Note: Rely on AIP-72 Task SDK |
tasks | render | Missing: Render a task instance Additional Note: Rely on AIP-72 Task SDK |
tasks | run | Missing: Run a task instance Additional Note: Rely on AIP-72 Task SDK |
tasks | state | Get a task instance Additional Note: Rely on AIP-72 Task SDK |
tasks | states-for-dag-run | List task instances Additional Note: Rely on AIP-72 Task SDK |
tasks | test | Missing: Run a task instance without DB Update Additional Note: Rely on AIP-72 Task SDK |
variables | delete | Delete a variable |
variables | get | Get a variable |
variables | export | Missing: Import variables(s) from a file Additional Note: This could be done via parsing the JSON locally and calling the API endpoints accordingly. I think this would cause too many API requests. I think we can handle this one in a single request via accepting multipart file |
variables | import | Missing: Import variables(s) from a file Additional Note: This could be done via parsing the JSON locally and calling the API endpoints accordingly. I think this would cause too many API requests. I think we can handle this one in a single request via accepting multipart file |
variables | list | List variables |
variables | set | Update a variable |
rotate-fernet-key | rotate-fernet-key | Missing: Rotate fernet key |
providers | * | Missing: List providers with all the informations Additional Note: Current List providers endpoint isn't enough due to it's not returning enough information about providers. |
aws-auth-manager | * | Additional Note: No additional DB call(s) other than loading provider configuration. |
celery | * | Additional Note: No additional DB call(s) other than loading provider configuration. |
kubernetes | cleanup-pods | Additional Note: No additional DB call(s) other than loading provider configuration. |
kubernetes | generate-dag-yaml | List task instances Additional Note: Rely on AIP-72 Task SDK No additional DB call(s) other than loading provider configuration. |
fab | * | Missing: Import Users, Export Users, Import Roles, Export Roles Additional Note: This could be done via parsing the JSON locally and calling the API endpoints accordingly. I think this would cause too many API requests. I think we can handle this one in a single request via accepting multipart file |
version | * | Missing: Get Airflow version Additional Note: This API will only return the Airflow version in the running environment. The CLI version will be locally displayed. |
This mapping doesn't contain any Core Commands due to they will stay the same without any additional changes.
General Note for Hybrid Commands added by providers: `providers_configuration_loaded` calls `ProvidersManager().initialize_providers_configuration()`. This command is only depends on dataset endpoints. When changing the method with API calls, this should apply to all the commands coming from the providers.
Unified Authentication Mechanism for CLI and API Integration
The authentication mechanism should be consistent across all setups, adopting a unified approach for all authentication scenarios. Despite variations in the backend authentication systems, the same authentication mechanism will be used. The only difference is that, instead of redirecting users to the default page, it will redirect them to the CLI.
User Workflow
All users will authenticate through the CLI using a standard command. This command will redirect users to the Webserver Sign-In page, where they will approve the authentication request. Upon approval, the API will return a token to the CLI, completing the authentication process. All subsequent API calls will be made using this token.
Automation Workflow (CI/CD or Automated Processes)
For CLI use in CI/CD pipelines or other automated processes, users will need to acquire the token independently. This can be done through the User Profile, where there will be a button to generate an API token specifically for automated users. Users can set the expiration date for this token to avoid rotating credentials with each run. Additionally, an API should be available to renew the expiration date of the token before it expires. If the token expires, users will need to follow the same procedure to acquire a new token. To support this process, Airflow should include a flag indicating whether a user is an automated (computer) user or an individual (human) user. Automated users should not have access to any webpages in the UI.
Important Notes
- A crucial aspect of this approach is the implementation of an expiration date for tokens obtained through the user workflow. For automated workflows, users should be allowed to set their own expiration dates, and there should be unique permissions for managing tokens for automated users.
- Users will need to enter the API URL once, and it will be saved for future use. After that, they can enter the authentication command without providing the URL. Automated users, however, will need to use a different command from the same command family, which will require additional token information during authentication.
- Individual users cannot generate a token with a long lifespan due to security concerns, as user tokens are typically stored on personal computers. In contrast, tokens for automated processes are generally stored in a secure (secret) backend.
There should be a separation between the actual sign-in endpoint and the CLI sign-in endpoint. This separation will be enforced at the sign-in level. Even if a user is already logged in and can access web pages without signing in again, the CLI sign-in will still require mandatory user validation.
Diagrams
Use Case Diagram: User Flow
Activity Diagram: User Flow
Use Case Diagram: Automation Flow
Activity Diagram: Automation Flow
Release Strategy
Release it as a separate package. This will reduce the number of dependencies and reduce the release maintenance effort. This strategy will be discussed as more broad
Name Suggestion: apache-airflow-cli
Dependencies on other AIPs
Some parts of this AIP rely on other AIP works such as removing the backfill mechanism from CLI or using Task SDK to use some APIs. The scaffolding of the API Commands and authentication mechanisms can start earlier.
Which users are affected by the change?
All the Airflow users that are using the CLI
How are users affected by the change? (e.g. DB upgrade required?)
Users should have to use authentication mechanism for all the API Commands.
For Core Commands,
Command Name | Authentication Needed |
---|---|
Direct Database Access Commands | Yes |
Local Commands | No |
Hybrid Commands | Yes |
Local Commands needs an Airflow Config (CFG) with Database Connection string filled to be able to execute. The other commands (Direct Database Access Commands and Hybrid Commands) won't need an Airflow Config (CFG) to execute.
What is the level of migration effort (manual and automated) needed for the users to adapt to the breaking changes? (especially in context of Airflow 3)
No changes are needed in this context. Users will continue to use Core Commands in the CLI as they do now, while new API Commands will also be available.