In favor of other scaling architectures we have decided to abandon the KnativeExecutor
Daniel Imberman → I have one question. I think KNative executor is great however there are indeed limits on the very long running tasks. This is particularly important for managed KNative implementations (such s KNative at Google Cloud Platform). My question here is - will that be possible to somehow join both Kubernetes and Knative executor and have a possibility of running some tasks via Kubernetes executor and some via KNative one. Maybe that could be the best solution to make Airflow Kubernetes-Native.
Jarek Potiuk That is actually something we've been actively discussing. One of the queues will be essentially a "KubernetesExecutor" queue. This queue will allow the tasks to be launched as isolated pods rather than via knative for this exact reason.
May I ask what other scaling architectures the KnativeExecutor was abandoned in favor of?
Sarah Johnson CeleryExecutor + KEDA, more robust and easier to set up than Knative. Check Daniel Imberman blog post about it: https://www.astronomer.io/blog/the-keda-autoscaler/ AFAIK KEDA is going to be part of official Airflow Helm chart.
Yes, KEDA support is in https://github.com/apache/airflow/pull/8777
Powered by a free Atlassian Confluence Open Source Project License granted to Apache Software Foundation. Evaluate Confluence today.