Status

StateAbandoned
Discussion Thread
JIRA


Github PRs


Created

2019-08-16

In favor of other scaling architectures we have decided to abandon the KnativeExecutor


6 Comments

  1. Daniel Imberman → I have one question. I think KNative executor is great however there are indeed limits on the very long running tasks. This is particularly important for managed KNative implementations (such s KNative at Google Cloud Platform). My question here is - will that be possible to somehow join both Kubernetes and Knative executor and have a possibility of running some tasks via Kubernetes executor and some via KNative one. Maybe that could be the best solution to make Airflow Kubernetes-Native. 

  2. Jarek Potiuk That is actually something we've been actively discussing. One of the queues will be essentially a "KubernetesExecutor" queue. This queue will allow the tasks to be launched as isolated pods rather than via knative for this exact reason. 

  3. May I ask what other scaling architectures the KnativeExecutor was abandoned in favor of?

  4. Sarah Johnson CeleryExecutor + KEDA, more robust and easier to set up than Knative. Check Daniel Imberman blog post about it: https://www.astronomer.io/blog/the-keda-autoscaler/ AFAIK KEDA is going to be part of official Airflow Helm chart.