AIP-114: Provider-supplied metrics backends via provider.yaml discovery

Status

Motivation

Today, Airflow's metrics backends (StatsD, Datadog, OTel) are hardcoded in core.  Adding a new backend requires changes to airflow-core itself, creating unnecessary coupling.  A CloudWatch metrics backend, for example, logically belongs in the Amazon provider package alongside other AWS integrations, but there is currently no mechanism for a provider to register a metrics backend.

We already solved this problem for executors, auth-managers, and secrets-backends.  Metrics backends are one of the few remaining extension categories that still require core changes to extend.

This AIP proposes extending the existing provider discovery mechanism to support metrics backends, following the identical pattern already established for other extension points.

Considerations

What change do you propose to make?

1. New metrics-backends key in provider.yaml so providers can declare their backend class(es) exactly how they currently declare executors, auth-managers, etc:

metrics-backends:
  - airflow.providers.amazon.aws.metrics.cloudwatch_logger.SafeCloudWatchLogger

1. New _discover_metrics_backends() method in providers_manager.py, following the same pattern as _discover_executors(), _discover_auth_managers(), etc.

2. New config option: [metrics] backend to accept an import path. If set, the backend is loaded with priority over the existing/legacy boolean flags. If unset, existing boolean flags (statsd_on, otel_on) continue to work unchanged.

What problem does it solve?

• Eliminates the requirement to modify airflow-core to add a new metrics backend.
• Allows provider packages to ship metrics backends alongside their other integrations (hooks, operators, executors, etc.).
• As an immediate example, this enables the Amazon provider to ship a native EMF-based CloudWatch metrics backend.

Why is it needed?

Metrics backends are one of the last remaining extension points that force vendor-specific code to live in core. Executors, auth-managers, and secrets-backends have all been decoupled. This creates unnecessary coupling between core release cycles and vendor-specific functionality, and means vendor-specific bug fixes require core releases.

Without this mechanism, there is no way for a provider to register a metrics backend. The only options today are:

1. Add the backend to core (increases coupling, wrong home for vendor-specific code).
2. Ask users to monkey-patch or use unsupported import paths.

This discovery mechanism both decouples provider code from core and opens the door for any provider to ship a metrics backend alongside its other integrations.

Are there any downsides to this change?

Potential for user confusion about backend location:

After this change, some metrics backends will live in core (StatsD, OTel) while new ones live in providers (CloudWatch). This mirrors the current state of executors (LocalExecutor in core, CeleryExecutor in the Celery provider, EcsExecutor in the Amazon provider) and secrets backends (environment variables in core, AWS SSM in the Amazon provider). The boundary is intuitive: generic/protocol implementations live in core, vendor-specific ones live in providers. This is the same split executors and secrets backends already use.

Specifically addressing the SMTP analogy raised in discussion: I believe that SMTP's confusion stems from a partial migration (core SMTP still exists but is deprecated, provider SMTP exists but doesn't fully replace it, and fixes are delayed in both). This proposal explicitly does NOT migrate anything. Existing backends stay in core, fully maintained, and could be moved to their respective providers at a later date. New vendor-specific backends are added in providers. No deprecation or "which one do I use?" ambiguity.

Addressing the Datadog concern: Datadog already has a provider package (apache-airflow-providers-datadog). Today it contains hooks and sensors but not the metrics backend. The Datadog metrics backend could eventually be moved there using this same mechanism, but that is a separate future decision, not included in the scope of this AIP. If/when that happens, it would be a clean migration because the core backend would be removed in the same release the provider backend ships.

Why not migrate all existing backends now? Moving StatsD or OTel to their own provider packages requires creating new providers, and the community's convention is that new providers need someone to commit to stewarding them. This AIP does not commit to that. If someone volunteers to own a statsd or opentelemetry provider in the future, this mechanism makes that migration straightforward; until then, the existing backends remain in core, fully functional and maintained by the community.

Which users are affected by the change?

• Users who want a provider-supplied metrics backend (e.g., CloudWatch): They gain a new capability. They set [metrics] backend in their config.
• Existing users (StatsD, OTel, Datadog): Zero impact. Existing statsd_on, otel_on flags continue to work identically. No migration needed. No behavior change.

How are users affected by the change? (e.g. DB upgrade required?)

No database changes. No DAG changes. No migration required. The change is purely additive: a new optional config key and a new discovery path in providers_manager.

What is the level of migration effort needed for users to adapt?

Zero. This is purely opt-in. Users who do nothing see no change. Users who want a provider-supplied backend set one config key.

Other considerations?

Performance:

Discovery happens once at startup during the existing providers_manager scan. It adds one dict key lookup per installed provider to an already-running loop. The backend is instantiated once via the factory. After initialization, Stats.incr() hits the instantiated backend directly with zero additional per-call overhead. No different from how executors or auth-managers are discovered and loaded today.

Fix delivery (core vs. provider):

• Bugs in the StatsLogger protocol (the interface all backends implement) are fixed in core. All backends benefit.
• Bugs in a specific backend's implementation are fixed where that backend lives. CloudWatch bugs are fixed in the Amazon provider. StatsD bugs are fixed in core. This is identical to how executor bugs work today: a bug in BaseExecutor is fixed in core; a bug in EcsExecutor is fixed in the Amazon provider.
• Users do not need to understand this boundary. They file a bug, and maintainers route it. Same as today.

Conflict resolution: If backend is set, it takes precedence over all boolean flags (statsd_on, otel_on, statsd_datadog_enabled). If unset, existing flag behavior is unchanged.

Backend configuration: Each backend reads its own parameters from Airflow config (same freedom executors and secrets backends have). The discovery mechanism does not dictate how a backend receives its settings.

Relationship to existing patterns:

What defines this AIP as "done"?

1. providers_manager.py discovers and registers metrics backends declared in provider.yaml.
2. The [metrics] backend config option loads and instantiates a provider-supplied backend.
3. At least one provider ships a metrics backend using this mechanism. Amazon CloudWatch Metrics will be implemented as the proof-of-concept.
4. Documentation updated to describe how to configure a provider-supplied metrics backend, including the StatsLogger protocol that third-party backends must implement.
5. Existing statsd_on / otel_on behavior unchanged when backend is not set.