AIP-81 Enhanced Security in CLI via Integration of API

Status

Motivation

The Command-Line Interface (CLI) for Airflow includes various types, such as those with direct database access, those operating locally, those running core Airflow components, and those that combine these features. Integrating API functionality into the CLI can provide isolated and secure database access while enabling features like built-in authentication. This integration facilitates the seamless deployment of CLI and API features together, reducing redundancy and simplifying maintenance. Transitioning from direct database access to an API-driven model will enhance the CLI’s capabilities and improve security. A gateway to bridge CLI commands with API interactions will support this shift, requiring new definitions and security protocols to ensure effective and secure communication.

Considerations

What change do you propose to make?

Definitions:

• Core Commands: CLI commands for managing Airflow.
• API Commands: CLI commands designed to interface with the API.

It is proposed to categorize Core Commands into the following groups. Categorizing commands will help identify which ones can be replaced with API Commands. This means an easier transition period if needed.

1. Airflow Management Commands: Commands for tasks such as running the webserver, scheduler, internal API server, or operating Airflow in standalone mode.
2. Database Access Commands: Commands involving direct database access, such as listing DAGs, and managing tasks, connections, and variables.
3. Hybrid Commands: Commands that manage Airflow while also requiring direct database access.

API Commands will support all necessary API interactions, following a consistent naming convention (to be defined), such as:

airflow dags list  
airflow taskinstances clear dag_id task_id 

API commands should be auto-generated as much as possible from configuration files (such as OpenAPI or the chosen API technology). This approach will minimize the number of changes required for new or updated API endpoints, ensuring consistency and reducing manual effort. It would be good to have a tool to auto-generate these commands. 

Security Considerations:

• API Commands
  • Existing Security Policies: Utilize current security policies for API endpoints, such as permissions required for listing DAGs.

• Core Commands

• Airflow Management Commands: Commands like the webserver and scheduler, which prepare or create environments and do not require prior authorization, should remain unchanged.
• Database Access Commands: Commands that involve direct database access should transition to API Commands, ensuring all database interactions are handled through the API with appropriate authentication and permissions.
• Hybrid Commands: Commands that manage Airflow and also involve direct database access should route database communication through API methods. Examples include loading DagBag and accessing DagRun from commands used by Celery and Kubernetes executors. These commands often operate with daemon options, minimizing instances where both direct database access and management functionalities are needed simultaneously.

The CLI should be secure-by-design. This means any sensitive data used by the CLI should be handled properly, such as being passed through stdin or retrieved from environment variables. Ensuring secure data handling practices will protect against potential security vulnerabilities and maintain user trust.

Command Inventory with Command Type Categorisation 

Command/Subcommand and API Endpoint Mapping

This mapping doesn't contain any Core Commands due to they will stay the same without any additional changes.

General Note for Hybrid Commands added by providers: `providers_configuration_loaded` calls `ProvidersManager().initialize_providers_configuration()`. This command is only depends on dataset endpoints. When changing the method with API calls, this should apply to all the commands coming from the providers.

Unified Authentication Mechanism for CLI and API Integration

The authentication mechanism should be consistent across all setups, adopting a unified approach for all authentication scenarios. Despite variations in the backend authentication systems, the same authentication mechanism will be used. The only difference is that, instead of redirecting users to the default page, it will redirect them to the CLI.

User Workflow

All users will authenticate through the CLI using a standard command. This command will redirect users to the Webserver Sign-In page, where they will approve the authentication request. Upon approval, the API will return a token to the CLI, completing the authentication process. All subsequent API calls will be made using this token.

Automation Workflow (CI/CD or Automated Processes)

For CLI use in CI/CD pipelines or other automated processes, users will need to acquire the token independently. This can be done through the User Profile, where there will be a button to generate an API token specifically for automated users. Users can set the expiration date for this token to avoid rotating credentials with each run. Additionally, an API should be available to renew the expiration date of the token before it expires. If the token expires, users will need to follow the same procedure to acquire a new token. To support this process, Airflow should include a flag indicating whether a user is an automated (computer) user or an individual (human) user. Automated users should not have access to any webpages in the UI.

Important Notes

• A crucial aspect of this approach is the implementation of an expiration date for tokens obtained through the user workflow. For automated workflows, users should be allowed to set their own expiration dates, and there should be unique permissions for managing tokens for automated users.
• Users will need to enter the API URL once, and it will be saved for future use. After that, they can enter the authentication command without providing the URL. Automated users, however, will need to use a different command from the same command family, which will require additional token information during authentication.
• Individual users cannot generate a token with a long lifespan due to security concerns, as user tokens are typically stored on personal computers. In contrast, tokens for automated processes are generally stored in a secure (secret) backend.

• There should be a separation between the actual sign-in endpoint and the CLI sign-in endpoint. This separation will be enforced at the sign-in level. Even if a user is already logged in and can access web pages without signing in again, the CLI sign-in will still require mandatory user validation.

Diagrams

Use Case Diagram: User Flow

Activity Diagram: User Flow

Use Case Diagram: Automation Flow

Activity Diagram: Automation Flow

Release Strategy

Release it as a separate package. This will reduce the number of dependencies and reduce the release maintenance effort. This strategy will be discussed as more broad

Name Suggestion: apache-airflow-cli

Dependencies on other AIPs

Some parts of this AIP rely on other AIP works such as removing the backfill mechanism from CLI or using Task SDK to use some APIs. The scaffolding of the API Commands and authentication mechanisms can start earlier.

• AIP-78 Scheduler-managed backfill
• AIP-72 Task Execution Interface aka Task SDK

Which users are affected by the change?

All the Airflow users that are using the CLI

How are users affected by the change? (e.g. DB upgrade required?)

Users should have to use authentication mechanism for all the API Commands.

For Core Commands,

Local Commands needs an Airflow Config (CFG)  with Database Connection string filled to be able to execute. The other commands (Direct Database Access Commands and Hybrid Commands) won't need an Airflow Config (CFG) to execute.

What is the level of migration effort (manual and automated) needed for the users to adapt to the breaking changes? (especially in context of Airflow 3)

No changes are needed in this context. Users will continue to use Core Commands in the CLI as they do now, while new API Commands will also be available.