Daniel Imberman → I have one question. I think KNative executor is great however there are indeed limits on the very long running tasks. This is particularly important for managed KNative implementations (such s KNative at Google Cloud Platform). My question here is - will that be possible to somehow join both Kubernetes and Knative executor and have a possibility of running some tasks via Kubernetes executor and some via KNative one. Maybe that could be the best solution to make Airflow Kubernetes-Native.
Jarek Potiuk That is actually something we've been actively discussing. One of the queues will be essentially a "KubernetesExecutor" queue. This queue will allow the tasks to be launched as isolated pods rather than via knative for this exact reason.
6 Comments
Jarek Potiuk
Daniel Imberman → I have one question. I think KNative executor is great however there are indeed limits on the very long running tasks. This is particularly important for managed KNative implementations (such s KNative at Google Cloud Platform). My question here is - will that be possible to somehow join both Kubernetes and Knative executor and have a possibility of running some tasks via Kubernetes executor and some via KNative one. Maybe that could be the best solution to make Airflow Kubernetes-Native.
Daniel Imberman
Jarek Potiuk That is actually something we've been actively discussing. One of the queues will be essentially a "KubernetesExecutor" queue. This queue will allow the tasks to be launched as isolated pods rather than via knative for this exact reason.
Sarah Johnson
May I ask what other scaling architectures the KnativeExecutor was abandoned in favor of?
Tomasz Urbaszek
Sarah Johnson CeleryExecutor + KEDA, more robust and easier to set up than Knative. Check Daniel Imberman blog post about it: https://www.astronomer.io/blog/the-keda-autoscaler/ AFAIK KEDA is going to be part of official Airflow Helm chart.
Ash Berlin-Taylor
Yes, KEDA support is in https://github.com/apache/airflow/pull/8777
Sarah Johnson
Thank you