AIP-91 - Airflow MCP Server (Phase 1 - Read-only actions)

Status

Motivation

Model Context Protocol (MCP) is an open standard, open-source framework that standardizes the way AI models like LLM integrate and share data with external tools, systems and data sources. One could think of it as a "USB-C for AI" - a universal connector that simplifies and standardizes AI integrations. A notable example of an MCP server is GitHub's official implementation, which allows LLMs such as Claude, Copilot, and OpenAI (or "MCP clients") to fetch pull request details, analyze code changes, and generate review summaries.

In the context of Apache Airflow, implementing an official MCP server provides the foundational, secure infrastructure required to expose Airflow's rich operational state to AI reasoning engines. Rather than building fragmented, ad-hoc API wrappers for every new AI feature, MCP provides a single, model-agnostic integration surface.

Key Use Cases Enabled by this AIP:

• AI-Powered IDE Integrations: Developers using MCP-compatible AI code editors (such as Cursor, Windsurf, or VS Code with Copilot) could query live Airflow Dag statuses, trigger states, and task logs directly from their editor while writing or debugging their Dag code.
• An Official Airflow AI Assistant (AIP-101): This is the primary and immediate driver for MCP. AIP-101 proposes an embedded, conversational UI for troubleshooting Airflow, which relies entirely on this MCP server to safely retrieve read-only context (Dags, Dag runs, task instances, logs) while strictly enforcing per-user Role-Based Access Control (RBAC).
• External Enterprise Agents & Chatbots: Organizations can connect their own custom Slack/Teams bots, internal developer portals, or enterprise-wide AI agents directly to the Airflow MCP server to query pipeline statuses without needing to write custom Airflow API integration code.
• Automated Observability: Third-party monitoring and observability platforms can leverage the MCP server to autonomously correlate task failures with system logs, generating automated root-cause analyses for on-call engineers.

Proposed Solution & Architecture

To satisfy the strict security and operational requirements of the Airflow AI Assistant (AIP-101) and other external integrations, the Airflow MCP Server must be built on a highly secure, stateless, and scalable architecture.

Identity Propagation & RBAC Enforcement (The Proxy Pattern)

This is the most critical architectural decision to satisfy AIP-101's requirement that the AI must never exceed the current user’s permissions.

WORK IN PROGRESS

The MCP Server MUST NOT connect directly to the Airflow Metadata Database, nor should it operate using a single, highly privileged "god-mode" service account. Instead, it will utilize Airflow's REST API:

1. Authentication Context: When an MCP Client (e.g., the AIP-101 Assistant backend) establishes a session with the MCP Server or makes a tool invocation request, it MUST pass the authenticated user's context (e.g., a short-lived signed JWT or an active session token).
2. Delegated Execution: When the LLM decides to call a tool (e.g., get_dag_runs), the MCP Server translates this into a standard HTTP request to the Airflow REST API (e.g., GET /api/v1/dags/{dag_id}/dagRuns).
3. Native RBAC Evaluation: The MCP Server attaches the user's token to the authorization header of the REST API request. This delegates all RBAC and permission gating entirely to Airflow's native Auth Manager.
4. Error Handling: If a user asks the LLM to fetch data for a DAG they do not have access to, the Airflow REST API will return a 403 Forbidden. The MCP Server catches this and returns a structured error to the LLM (e.g., "Tool execution failed: User lacks permission to view this DAG"), ensuring mathematical certainty that privilege escalation cannot occur.

Transport & Connectivity

The Model Context Protocol supports multiple transport layers (stdio for local processes, and SSE for remote servers). Because Airflow is a distributed system, the MCP Server MUST support HTTP SSE (Server-Sent Events) as its primary transport layer:

• Standalone Deployment: The MCP Server will run as a standalone service that can be scaled independently of the Airflow Webserver or Scheduler.
• Airflow Configuration: The Airflow instance will configure its connection to the MCP Server via a native Airflow Connection (e.g., conn_type='mcp'). This connection will store the Host (URL of the MCP server) and optionally a Password or Extra configuration for mutual TLS or pre-shared keys (PSK) to secure service-to-service communication.

Tool Registration & The Read-Only Guarantee (Phase 1)

To enforce the Phase 1 invariants of AIP-101, the MCP Server framework MUST guarantee read-only operations at the protocol level.

• Tools exposed to the LLM (e.g., list_dags, get_task_instance, get_task_logs) MUST map exclusively to idempotent, safe HTTP GET methods on the Airflow REST API.
• The MCP server startup sequence will enforce a strict schema validation that rejects any tool definition attempting to use POST, PUT, PATCH, or DELETE methods during Phase 1.

Security Model & Controls

Because the MCP Server exposes Airflow's operational state to non-deterministic external LLMs, it requires stringent, defense-in-depth security controls.

WORK IN PROGRESS

Modifications to Airflow API

We need to ensure that Airflow API is suitable for usage as part of MCP.

Data Minimization & Secret Redaction

Even though the Airflow REST API redacts secrets by default, the MCP Server acts as the final boundary before data is sent to external LLM providers (e.g., OpenAI, Anthropic).

• Payload Masking: The MCP Server MUST pipe all retrieved free-text artifacts (specifically Task Logs and Rendered Templates) through Airflow's native SecretsMasker before serializing the tool response.
• Schema Stripping: The MCP Server MUST use strict Pydantic response models that explicitly drop internal system metadata, server IPs, or debug traces that are not strictly necessary for the LLM to answer user queries.

(Rate Limiting & Payload Bounding)

LLM agents can suffer from "hallucination loops," where they repeatedly call tools in a rapid, infinite loop if they get confused. To protect the core Airflow API from Denial of Service (DoS):

• Hard Pagination: Tool calls that return lists MUST enforce strict server-side pagination limits (e.g., a hard cap of 50 Task Instances per tool call), ignoring LLM requests for larger page sizes.
• Payload Truncation: Large payloads (like Task Logs) MUST be truncated to a configured maximum byte size (e.g., 10KB) to prevent network overflow and to keep responses within the LLM's context window limits. Truncated payloads will include a system string informing the LLM: "[Log truncated. Use the get_log_tail tool to fetch subsequent lines.]"
• Rate Limiting: The MCP Server SHOULD implement token-bucket rate limiting based on the user_id to throttle excessive consecutive tool calls.

Auditability & Traceability

To satisfy security concerns regarding usage of MCP within enterprises, as well as AIP-101's audit requirements, the MCP Server MUST emit structured audit events for every tool invocation.

• Audit Contract: The server will generate a JSON-structured log for each action containing: {"timestamp", "user_id", "mcp_client_id", "tool_name", "parameters_redacted", "latency_ms", "status_code"}.
• Integration: These events MUST be written to standard output for log aggregators, and SHOULD optionally be pushed to Airflow's internal Log table (Audit Logs) via the REST API to ensure a unified audit trail in the Airflow UI.

Considerations

What change do you propose to make?

We propose to implement an official Model Context Protocol (MCP) server for Apache Airflow.

• Standalone Release within the monorepo: The server will be maintained in as part of apache/airflow  monorepo, but will have its own release cycle.
• Standardized Tool Abstraction: The MCP server will expose Airflow state (Dags, Dag Runs, TaskInstances, logs, and configuration) as structured, model-agnostic tools that any MCP-compatible client (like the AIP-101 AI Assistant, Claude Desktop, or custom enterprise agents) can invoke.
• Modifications to Airflow's API to ensure compatibility with MCP usage (RBAC, Data minimization, resource protection, etc.)

What problem does it solve?

Currently, organizations attempting to integrate Large Language Models (LLMs) or AI agents with Apache Airflow are forced to build custom, ad-hoc API wrappers. This lack of standardization introduces several critical challenges:

• Fragmented Integrations: Teams duplicate effort building bespoke API connections for different AI tools and deployment environments.
• Security & Compliance Risks: Custom AI integrations frequently fail to properly enforce Airflow's per-user Role-Based Access Control (RBAC) or consistently redact sensitive connection credentials, creating severe data exfiltration risks.
• Framework Lock-in: Hardcoding integrations to specific AI frameworks (e.g., LangChain, LlamaIndex) limits flexibility and makes it difficult to adopt new models or agent architectures.

By providing an official MCP server, we solve this integration problem at the protocol level. The MCP server acts as a single, secure, audited, and model-agnostic bridge between Airflow's complex internal state and external AI reasoning engines.

With this standardized bridge in place, users can safely and seamlessly interact with Airflow through LLMs, dramatically improving accessibility and operational efficiency. Specifically, it unlocks the ability to:

• Debug task failures in natural language: Easily query task instances and logs to troubleshoot issues, significantly reducing time-to-resolution—especially for non-technical users.
• Analyze cross-Dag dependencies: Gain AI-assisted insights into complex pipeline relationships to improve overall system reliability.
• Optimize performance: Identify opportunities for Dag code optimization and support sparse scheduling to improve compute resource utilization.
• Streamline upgrades: Assist with migration and refactoring planning to ease transitions between Airflow versions or deployment architectures.

Why is it needed?

As Airflow usage scales across organizations, there's a growing need to simplify complex operations and troubleshoot via AI. However, actionable AI-assisted insights require access to live Airflow state.

An official MCP implementation is needed to provide a secure, standardized boundary for this data access. It allows core Airflow to remain unaffected while giving features like the official Airflow AI Assistant (AIP-101) a reliable way to ground its responses in reality. By centralizing the tool definitions in an official MCP server, we ensure that security controls (like data minimization, secret redaction, and RBAC) are consistently applied regardless of which LLM or client is asking for the data.

Are there any downsides to this change?

• CI/CD Overhead: Since the MCP server is maintained as separate distribution, additional effort will be required to develop and maintain related CI pipelines, compatibility matrices, and release processes.
• Complex Testing: Integration tests for complex cases will require an integration with LLMs (e.g., via Airflow's AWS instance) and thorough "red-teaming" to ensure tools cannot be exploited.
• Security Surface Expansion: Running an additional server that exposes Airflow metadata to AI models expands the attack surface. It requires rigorous enforcement of authentication, input validation, and rate-limiting to prevent Denial of Service (DoS) from hyperactive AI agents.

Which users are affected by the change?

The change affects the following roles:

• Operational users - can execute operations on Dags or their results using natural language. MCP-aided LLM capabilities can help bridge technical gaps for non-technical users.
• Dag authors - same as operational users, with the addition of debugging technical issues or leveraging technical insights.
• Deployment Manager - when MCP capabilities are enabled, they should be aware of how to deploy the MCP server.

How are users affected by the change? (e.g. DB upgrade required?)

Installing Airflow's MCP will be opt-in only (i.e., non-breaking for existing deployments).

What is the level of migration effort (manual and automated) needed for the users to adapt to the breaking changes? (especially in context of Airflow 3)

N/A

Other considerations?

N/A

What defines this AIP as "done"?

• A distribution is created under the apache/airflow GitHub monorepo, containing the MCP server.
  • Basic CI - linting, unit tests, release management, and LLM-assisted integration tests (using Aiirflow's AWS instance).
• Implementation of the MCP Server:
  • Implementation of core read-only tools for retrieving Dags, Runs, Tasks, and Logs.
  • Crucial: Implementation of per-user identity propagation and RBAC enforcement. The server must mathematically guarantee a client cannot access data beyond its user's Airflow permissions.
  • Deployment using the official helm chart and via breeze.
• Comprehensive documentation for deployment managers detailing how to configure endpoints, secure the service, and manage external LLM network egress.