DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
Important Links
Discussion Notes
https://docs.google.com/document/d/1bZsyrG5kjsYd2rJRiN3kR613lO6JPEBd4ItsySneOMw/edit?usp=sharing
Github Umbrella Issue
https://github.com/apache/airflow/issues/64037
Discussion
https://lists.apache.org/thread/9714k5wo2jwz1xwdnh9xnp98hloxms15
Bi-Weekly Interest Group Alignment/Dicuss Call
Helm Release Discussion
Thursday 7 May • 18:00-19:00 CEST (Central European Summer Time)
Link to video call: https://meet.google.com/yjo-ueoq-nak
Complexity Analysis and Rating - Chart Parameters
Parameters from Docs
Based on Helm Chart 1.19.0 release
Common
Parameter | Description | Default | Opinion Jens | Opinion Bugra | Opinion Przemek |
|---|---|---|---|---|---|
| airflow_local_settings file as a string (templated). You can bake an airflow_local_settings.py into your image instead. In that case, set this value to null. |
| keep | keep | Keep |
| Airflow version (Used to make some decisions based on Airflow Version being deployed). |
| keep | keep | Keep |
| Annotations to add to the Api secret. |
| Why is this for flask in AF3? | Description remains from v2 terms. | Move under apiServer section |
| The Flask secret key for Airflow Api to encrypt browser session. |
| Why is this for flask in AF3? | Description remains from v2 terms. | Maybe dedicated section for flask-related configs 🤔
|
| This string (templated) will be mounted into the Airflow API Server as a custom webserver_config.py. You can bake a webserver_config.py in to your image instead or specify a configmap containing the webserver_config.py. |
Examples: apiServerConfig: |- from airflow import configuration as conf # The SQLAlchemy connection string. SQLALCHEMY_DATABASE_URI = conf.get('database', 'SQL_ALCHEMY_CONN') # Flask-WTF flag for CSRF CSRF_ENABLED = True | keep Is example still relevant? | This is still relevant if the user is using providers-fab and adding more fine-grained authentication integration through fab. That also generalised to BaseAuthManager and to providers that implement 3rd-party libraries and tools. I think the description should be updated. | |
| The configmap name containing the webserver_config.py. |
Examples: apiServerConfigConfigMapName: my-api-server-configmap | keep | keep | keep |
| Settings to go into the mounted airflow.cfg |
| keep | keep | keep |
| Enable default user creation. |
| keep | keep | If I remember correctly it is only flask-related, so maybe move to flask dedicated section? |
| Default airflow digest to deploy. Overrides tag. |
| keep | keep | keep |
| Default airflow repository. Overrides all the specific images below. |
| keep | keep | keep |
| Default airflow tag to deploy. |
| keep | keep | keep |
| Airflow executor. |
|
|
| keep |
| The Fernet key used to encrypt passwords (can only be set during install, not upgrade). |
| keep | keep | Move to the new encryption section |
| Annotations to add to the Fernet Key secret. |
| keep | keep | |
| Provide a name to substitute for the full names of resources | "" |
|
| keep |
| Secret key used to encode and decode JWTs to authenticate to public and private APIs (can only be set during install, not upgrade). |
| keep | keep | Move to the new encryption section |
| Annotations to add to the JWT secret. |
| keep | keep | |
| Override the name of the chart | "" |
|
| Change to chartNameOverride |
| Enable PgBouncer. |
| keep | keep | Whole section comment is under pgbouncer section |
| Whether to mount the config secret files under /etc/pgbouncer/ by default. |
| keep | keep | |
| Specify kube scheduler name for Pods. |
| Put this to Kustomize? | No disagreement with Jens. | No strong opinion |
| Use standard naming for all resources using airflow.fullname template | False |
| Deprecate the false value and remove it in future | |
| This string (templated) will be mounted into the Airflow webserver as a custom webserver_config.py. You can bake a webserver_config.py in to your image instead or specify a configmap containing the webserver_config.py. |
Examples: webserverConfig: |- from airflow import configuration as conf # The SQLAlchemy connection string. SQLALCHEMY_DATABASE_URI = conf.get('database', 'SQL_ALCHEMY_CONN') # Flask-WTF flag for CSRF CSRF_ENABLED = True | removed with dropping AF2 support | Drop with Airflow 2
| |
| The configmap name containing the webserver_config.py. |
Examples: webserverConfigConfigMapName: my-webserver-configmap |
| ||
| Annotations to add to the webserver secret. |
|
| ||
| The Flask secret key for Airflow Webserver to encrypt browser session. |
|
| ||
| Specify kube scheduler name for Airflow Celery workers objects and pods created with pod-template-file. |
| Put this to Kustomize? | No disagreement with Jens. | No strong opinion |
Airflow
Parameter | Description | Default | Opinion Jens | Opinion Bugra | Opinion Przemek |
|---|---|---|---|---|---|
| Airflow home directory. Used for mount paths. |
| keep | keep | keep |
| Whether various Airflow components launch jobs. |
| Can this not implicitly be discovered if K8s executor? | It can be automated indeed. We can map to both a feature and a deprecation along with it. | I would leave it as it is. Looking at the current available Kubernetes connections, there is an in_cluster option which basically says "use Kubernetes Service Account which is set for currently running pod e.g. Celery worker". In some cases, despite used Executor, the proper permissions can be or can be not be needed, and if something is not needed, there should be a possibility of removing it (an automatic option would remove it, raising some security concerns). |
| Whether various Airflow components launch pods. |
| Can this not implicitly be discovered if K8s executor? | It can be automated indeed. We can map to both a feature and a deprecation along with it. | |
| The Secret name containing Flask secret_key for the Api. |
| Why is this for flask in AF3? | Description remains from v2 terms. | Maybe move to flask dedicated section 🤔? |
| Git branch |
| Should we transform the gitSync into a bundle definition? | It makes sense. There are multiple ways in various systems of mounting the dags into the components. Making it bundled can help with easier additions on top of it for mounting operations. | Drop in favor of bundle and maybe seperate it to not-dag related section for sync with potential external dags dependencies
|
| Git sync container name. |
|
|
| |
| Name of a Secret containing the repo GIT_SYNC_USERNAME and GIT_SYNC_PASSWORD. |
|
|
| |
| Repository depth. |
|
|
| |
| Configuration for dags empty dir volume. |
|
|
| |
| Enable Git sync. |
|
|
| |
| Environment variables for git sync container. |
Examples: env: - name: GIT_SYNC_TIMEOUT value: '60' env: - name: GIT_SYNC_USERNAME valueFrom: secretKeyRef: key: username name: git-secret |
|
| |
| Extra envFrom ‘items’ that will be added to the definition of Airflow gitSync containers; a string or array are expected (templated). |
Examples: envFrom: |- - secretRef: name: 'proxy-config envFrom: |- - configMapRef: name: 'proxy-config |
|
| |
| Mount additional volumes into git sync container. |
|
|
| |
| When using a ssh private key, the contents of your known_hosts file. |
Examples: knownHosts: |- <host1>,<ip1> <key1> <host2>,<ip2> <key2> knownHosts: <host1>,<ip1> <key1> |
|
| |
| The number of consecutive failures allowed before aborting. |
|
|
| |
| Interval between git sync attempts in Go-style duration string. High values are more likely to cause DAGs to become out of sync between different components. Low values cause more traffic to the remote git repository. |
|
|
| |
| Git revision branch, tag, or hash. |
|
|
| |
| Git repository. |
|
| ||
| Resources on workers git-sync sidecar |
Examples: resources: limits: cpu: 100m memory: 128Mi requests: cpu: 100m memory: 128Mi |
|
| |
| Git revision. |
|
|
| |
| Security context for the gitSync container (deprecated, use securityContexts instead). If not set, the values from securityContext will be used. |
Examples: securityContext: runAsGroup: 0 runAsUser: 50000 |
|
| |
| SSH private key |
|
|
| |
| Name of a Secret containing the repo sshKeySecret. |
|
|
| |
| Subpath within the repo where dags are located. |
|
|
| |
| Git sync container run as user parameter. |
|
|
| |
| Interval between git sync attempts in seconds. High values are more likely to cause DAGs to become out of sync between different components. Low values cause more traffic to the remote git repository. |
|
|
| |
| Where dags volume will be mounted. Works for both persistence and gitSync. If not specified, dags mount path will be set to $AIRFLOW_HOME/dags |
|
|
| keep |
| Access mode of the persistent volume. |
|
|
| keep |
| Annotations for the dag PVC |
|
|
| keep |
| Enable persistent volume for storing dags. |
|
|
| keep |
| The name of an existing PVC to use. |
|
|
| keep |
| Volume size for dags. |
|
|
| keep |
| If using a custom StorageClass, pass name here (templated). |
|
|
| keep |
| Subpath within the PVC where dags are located. |
|
|
| keep |
| Elasticsearch connection configuration. |
Examples: connection: host: '...' pass: '...' port: '...' scheme: https user: '...' |
|
| Drop/move to Kustomize
|
| Host |
|
|
| |
| Password |
|
|
| |
| Port |
|
|
| |
| Scheme |
|
|
| |
| Username |
|
|
| |
| Enable Elasticsearch task logging. |
|
|
| |
| Extra annotations to apply to the elasticsearch secret. |
|
|
| |
| A secret containing the connection string. |
|
|
| |
| Enable |
| What is this for? | SQLAlchemy connection string from K8s secret. Keeping with a better description and maybe renaming to reflect better. | I guess it should be the same as AIRFLOW__DATABASE__SQL_ALCHEMY_CONN so probably drop |
| Enable |
| What is this for? | Enable using K8S secret in env var to pass jwt_secret config | keep |
| Enable |
| What is this for? | Similar to above | keep |
| Enable |
| What is this for? |
All these are the same as above, as setting some config that enables to read secret to pass to the env var. Maybe move them to Kustomize? | keep/move under workers.celery section |
| Enable |
| What is this for? | Drop | |
| Enable |
| What is this for? | keep/move under workers.celery section | |
| Enable |
| What is this for? | keep | |
| Enable |
| What is this for? | Remove with Airflow 2 drop | |
| Enable |
| What is this for? | keep | |
| Enable |
| What is this for? | Drop/move to Kustomize with elasticsearch section | |
| Enable |
| What is this for? | ||
| Enable |
| What is this for? | Drop/move to Kustomize with opensearch section | |
| Enable |
| removed with dropping AF2 support |
| Drop with Airflow 2 |
| Environment variables for all Airflow containers. |
Examples: env: - name: MYENVVAR value: something_fun |
|
| keep |
| Extra env ‘items’ that will be added to the definition of Airflow containers; a string is expected (templated). |
Examples: extraEnv: |- - name: AIRFLOW__CORE__LOAD_EXAMPLES value: True |
|
| keep |
| Extra envFrom ‘items’ that will be added to the definition of Airflow containers; a string is expected (templated). |
Examples: extraEnvFrom: |- - secretRef: name: '{{ .Release.Name }}-airflow-connections' extraEnvFrom: |- - configMapRef: name: '{{ .Release.Name }}-airflow-variables' |
|
| keep |
| The Fernet key secret name. |
|
|
| Move under the new encryption section |
| Group of airflow user. |
|
|
| Probably remove and/or move to Kustomize. It is only used in volume-permissions sidecar in workers, which, looking at the comment, is only needed with the local path provisioner https://github.com/rancher/local-path-provisioner setup. |
| The JWT secret name. |
|
|
| Move under the new encryption section |
| Configuration for logs empty dir volume. |
|
|
| keep |
| Annotations to add to logs PVC |
|
|
| keep |
| Enable persistent volume for storing logs. |
|
|
| keep |
| The name of an existing PVC to use. |
|
|
| keep |
| Volume size for logs. |
|
|
| keep |
| If using a custom StorageClass, pass name here (templated). |
|
|
| keep |
| The subpath of the existing PVC to use. |
|
|
| keep |
| Whether Airflow can launch workers and/or pods in multiple namespaces. If true, it creates |
|
|
| Move under workers.kubernetes section as it is only related to KubernetesExecutor |
| OpenSearch connection configuration. |
Examples: connection: host: '...' pass: '...' port: '...' scheme: https user: '...' |
|
| Drop/move to Kustomize
|
| Host |
|
|
| |
| Password |
|
|
| |
| Port |
|
|
| |
| Scheme |
|
|
| |
| Username |
|
|
| |
| Enable OpenSearch task logging. |
|
|
| |
| A secret containing the connection string. |
|
|
| |
| The content of |
Examples: podTemplate: |- apiVersion: v1 kind: Pod metadata: name: placeholder-name labels: tier: airflow component: worker release: {{ .Release.Name }} spec: priorityClassName: high-priority containers: - name: base ... |
|
| Move to workers.kubernetes as it is only related to KubernetesExecutor |
| Secrets for all Airflow containers. |
Examples: secret: - envName: SecretEnvVar secretKey: somekey secretName: somesecret |
|
| keep |
| User of airflow user. |
|
|
| Probably remove and/or move to Kustomize. It is only used in volume-permissions sidecar in workers, which, looking at the comment, is only needed with the local path provisioner https://github.com/rancher/local-path-provisioner setup. |
| VolumeMounts for all Airflow containers. |
|
|
| keep |
| Volumes for all Airflow containers. |
|
|
| keep |
| The Secret name containing Flask secret_key for the Webserver. |
| removed with dropping AF2 support |
| Drop with Airflow 2 |
Images
Parameter | Description | Default | Opinion Jens | Opinion Bugra | Opinion Przemek |
|---|---|---|---|---|---|
| The airflow image digest. If set, it will override the tag. |
|
|
| Probably more in future: move to per-section config with ongoing decupling of airflow and future multi-team feature |
| The airflow image pull policy. |
|
|
| |
| The airflow image repository. |
|
|
| |
| The airflow image tag. |
|
|
| |
| The flower image pull policy. |
|
|
| Move to Kustomize with the flower section |
| The flower image repository. |
|
|
| |
| The flower image tag. |
|
|
| |
| The gitSync image pull policy. |
|
|
Maybe we can bundle these with the gitSync definitions above. I see from Jed's comment in Slack that if we completely hammer it, bundling can make it easier, but on the other hand, not sure if it would be worth the effort. | Drop with gitSync drop or move as a method to sync external Dag dependencies |
| The gitSync image repository. |
| |||
| The gitSync image tag. |
| Note: Drop support for old versions! | ||
| The time (in seconds) to wait for the DB migrations to complete. |
|
|
| Move under the migrateDatabaseJob section |
| The PgBouncer image pull policy. |
|
|
| Related comment under the PgBouncer section |
| The PgBouncer image repository. |
|
|
| |
| The PgBouncer image tag. |
|
|
| |
| The PgBouncer exporter image pull policy. |
|
|
| |
| The PgBouncer exporter image repository. |
|
|
| |
| The PgBouncer exporter image tag. |
|
|
| |
| The pod_template image pull policy. |
|
|
| Probably more in future: move to per-section config with ongoing decupling of airflow and future multi-team feature |
| The pod_template image repository. If |
|
|
| |
| The pod_template image tag. If |
|
|
| |
| The redis image pull policy. |
|
|
| Opinion under respective sections |
| The redis image repository. |
|
|
| |
| The redis image tag. |
|
|
| |
| The StatsD image pull policy. |
|
|
| |
| The StatsD image repository. |
|
| ||
| The StatsD image tag. |
|
|
| |
| To avoid images with user code for running and waiting for DB migrations set this to |
|
|
| Probably more in future: move to per-section config with ongoing decupling of airflow and future multi-team feature |
Ports
Parameter | Description | Default | Opinion Jens | Opinion Przemek |
|---|---|---|---|---|
| Airflow UI port. |
| keep all | Keep |
| API server port. |
|
| Keep |
| Flower UI port. |
|
| Move with flower to Kustomize |
| PgBouncer port. |
|
| Related comment under the PgBouncer section |
| PgBouncer scrape port. |
|
| |
| Redis port. |
|
| Keep |
| StatsD ingest port. |
|
| Maybe move with statsd to Kustomize |
| StatsD scrape port. |
|
| Maybe move with statsd to Kustomize |
| Triggerer logs port. |
|
| Keep |
| Worker logs port. |
|
| Keep |
Database
Parameter | Description | Default | Opinion Jens | Opinion Bugra | Opinion Przemek |
|---|---|---|---|---|---|
| Annotations to add to the broker url secret. |
|
|
| Move to workers.celery/redis as it is only for CeleryExecutor |
| The name of the database. |
|
|
| Move to the new metastore/database section
|
| The database host. |
|
|
| |
| The user’s password. |
|
|
| |
| The database port. |
|
|
| |
| The database protocol. |
|
|
| |
| Annotations to add to the metadata connection secret. |
|
|
| |
| The database SSL parameter. |
|
|
| |
| The database user. |
|
|
| |
| Metadata connection string secret. |
|
|
| |
| Result backend connection configuration. |
|
|
| Move to workers.celery as it is only for CeleryExecutor
|
| The name of the database. |
|
|
| |
| The database host. |
|
|
| |
| The database password. |
|
|
| |
| The database port. |
|
|
| |
| The database protocol. |
|
|
| |
| The database SSL parameter. |
|
|
| |
| The database user. |
|
|
| |
| Annotations to add to the result backend connection secret. |
|
|
| |
| Result backend connection string secret. |
|
|
| |
| Assign a password to the ‘postgres’ admin user. Otherwise, remote access will be blocked for this user |
| Drop postgres inline support |
This is not even needs a discussion :D | Drop |
| Password for the custom user to create. |
|
| ||
| Password for the ‘postgres’ admin user. |
|
| ||
| Name for a custom user to create |
|
| ||
| Enable PostgreSQL subchart. |
|
| ||
| The PostgreSQL image repository. |
|
| ||
| The PostgreSQL image tag. |
|
|
PgBouncer
Parameter | Description | Default | Opinion Jens | Opinion Bugra | Opinion Przemek |
|---|---|---|---|---|---|
| Specify scheduling constraints for PgBouncer pods. |
|
| I have a comment on this as a whole. While we are planning to hammer PostgreSQL. Why we should also consider dropping support for PgBouncer as well It is just an idea thrown into the wild. Since it is specific to the PostgreSQL environment, maybe our focus can be on making core charts better rather than spending time providing support to another tool similar to PostgreSQL. I am not discussing how useful it is for availability and managing/distributing database connections. My ideation is more on I see no difference with having ProxySQL, which is a similar tool (open source under GNU, GPL-3).
| In addition to Bugra's comment - as I agree on that pgbouncer is just one possible tool which can be used, I think it goes potentially a little deeper in the sense of what we would want this chart to be. Do we want it to be only a baseline (base config and support for Apache Airflow features, nothing more), or do we want to make it more production-grade with a recommended setup (e.g. use pgbouncer) with integration tests, e.g. for upgrade/downgrades, etc. (maybe not like use this or that, but we are testing that kind of setup and it seems to work like with constraints file)? On the whole section itself, I would remove it from the core chart as it is not an Airflow-related component (I had an idea that maybe the core chart should only consist of Airflow-related things, and the rest should be customisation).
|
| Annotations to add to the PgBouncer deployment |
|
| ||
| Args to use for PgBouncer (templated). |
|
| ||
| The name of the file to load user names and passwords from |
|
| ||
| Method of authenticating users |
|
| ||
| Annotations to add to the PgBouncer certificates secret. |
|
| ||
| The allowed ciphers, might be ‘fast’, ‘normal’ or list ciphers separated with ‘:’. |
|
| ||
| Command to use for PgBouncer (templated). |
|
| ||
| Annotations to add to the PgBouncer config secret. |
|
| ||
| The PgBouncer config Secret name. |
|
| ||
| Add additional env vars to pgbouncer container. |
|
| ||
| Launch additional containers into pgbouncer. |
|
| ||
| Add extra general PgBouncer ini configuration: https://www.pgbouncer.org/config.html |
|
| ||
| Add extra metadata database specific PgBouncer ini configuration: https://www.pgbouncer.org/config.html#section-databases |
|
| ||
| Add extra result backend database specific PgBouncer ini configuration: https://www.pgbouncer.org/config.html#section-databases |
|
| ||
| Additional NetworkPolicies as needed. |
|
| ||
| Mount additional volumes into PgBouncer. |
|
| ||
| Mount additional volumes into PgBouncer. |
|
| ||
| Labels to add to the PgBouncer objects and pods. |
|
| ||
| Log successful logins. |
|
| ||
| Log disconnections with reasons. |
|
| ||
| Maximum clients that can connect to PgBouncer (higher = more file descriptors). |
|
| ||
| Metadata pool size. |
|
| ||
| Mount additional volumes into PgBouncer Metrics Exporter. |
|
| ||
| Metrics Exporter liveness probe initial delay |
|
| ||
| Metrics Exporter liveness probe frequency |
|
| ||
| Metrics Exporter liveness probe command timeout |
|
| ||
| Metrics Exporter readiness probe initial delay |
|
| ||
| Metrics Exporter readiness probe frequency |
|
| ||
| Metrics Exporter readiness probe command timeout |
|
| ||
| Resources for the PgBouncer metric exporter. |
Examples: resources: limits: cpu: 100m memory: 128Mi requests: cpu: 100m memory: 128Mi |
| ||
| SSL mode for |
|
| ||
| Annotations to add to the PgBouncer stats secret. |
|
| ||
| Key referencing the PGBouncer Metrics connection URI within an existing Secrets object. Defaults to connection if left null. |
|
| ||
| Name of an existing Secrets object containing PgBouncer Metrics secrets. |
|
| ||
| Select certain nodes for PgBouncer pods. |
|
| ||
| Add annotations for the PgBouncer Pod. |
|
| ||
| Max unavailable pods for PgBouncer. |
|
| ||
| Min available pods for PgBouncer. |
|
| ||
| Enabled PodDistributionBudget. |
|
| ||
| Specify priority for PgBouncer pods. |
|
| ||
| Number of PgBouncer replicas to run in Deployment. |
|
| ||
| Resources for the PgBouncer pods. |
Examples: resources: limits: cpu: 100m memory: 128Mi requests: cpu: 100m memory: 128Mi |
| ||
| Result backend pool size. |
|
| ||
| Specific ClusterIP for the PgBouncer Service. |
|
| ||
| Extra annotations for the PgBouncer Service. |
|
| ||
| Annotations to add to the worker Kubernetes ServiceAccount. |
|
| ||
| Specifies if ServiceAccount’s API credentials should be mounted onto Pods. |
|
| ||
| Specifies whether a ServiceAccount should be created. |
|
| ||
| The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name. |
|
| ||
| Certificate Authority for server side |
|
| ||
| Server Certificate for server side |
|
| ||
| Private key used to authenticate with the server |
|
| ||
| SSL mode for PgBouncer. |
|
| ||
| Specify Tolerations for PgBouncer pods. |
|
| ||
| Specify topology spread constraints for PgBouncer pods. |
|
| ||
| PgBouncer run as user parameter. |
|
| ||
| Increase PgBouncer verbosity. |
|
|
API Server
Parameter | Description | Default | Opinion Jens | Opinion Bugra | Opinion Przemek |
|---|---|---|---|---|---|
| Specify scheduling constraints for API server pods. |
|
|
| keep |
| Allow API server to read k8s pod logs. Useful when you don’t have an external log store. |
|
|
| Maybe it could be automatically set based on the executor value 🤔. Maybe, as I'm not sure about whether it is needed or not, with e.g. remote logging set |
| Annotations to add to the API server deployment |
|
|
| keep |
| Args to use when running the Airflow API server (templated). When running behind a reverse proxy, add –proxy-headers to enable Uvicorn to respect X-Forwarded-Proto, X-Forwarded-For, and X-Forwarded-Port headers. |
Examples: args: - bash - -c - exec airflow api-server --proxy-headers |
|
| keep |
| Command to use when running the Airflow API server (templated). |
|
|
| keep |
| Extra annotations to apply to the API server configmap. |
|
|
| keep |
| Enable Airflow API server deployment. |
|
|
| keep |
| Add additional env vars to API server. When running behind a reverse proxy, set FORWARDED_ALLOW_IPS to specify which IPs are trusted to send X-Forwarded-* headers. Use “*” for trusted environments, or specify proxy IP ranges for production. |
Examples: env: - - name: FORWARDED_ALLOW_IPS value: '*' |
|
| keep |
| Launch additional containers into API server. |
|
|
| keep |
| Add additional init containers into API server. |
|
|
| keep |
| Mount additional volumes into API server. |
|
|
| keep |
| Mount additional volumes into API server. |
|
|
| keep |
| HostAliases for the API server pod. |
Examples: hostAliases: - hostnames: - foo.local ip: 127.0.0.1 hostAliases: - hostnames: - foo.remote ip: 10.1.2.3 |
|
| keep |
| HorizontalPodAutoscalerBehavior configures the scaling behavior of the target. |
| Should we keep both KEDA and HPA? Or move both to a Kustomize?
| My vote would be moving Keda to Kustomize, as it is really custom if we think of core components
| Keep HPA, document Keda in Kustomize
|
| Enable HPA autoscaling for API server |
| |||
| Maximum number of API server replicas created by HPA if HPA is enabled. |
| |||
| Specifications for which to use to calculate the desired replica count. |
| |||
| Minimum number of API server replicas created by HPA if HPA is enabled. |
| |||
| Labels to add to the API server objects and pods. |
|
|
| keep |
| API server Liveness probe failure threshold. |
|
|
| keep |
| API server Liveness probe initial delay. |
|
|
| keep |
| API server Liveness probe period seconds. |
|
|
| keep |
| API server Liveness probe scheme. |
|
|
| keep |
| API server Liveness probe timeout seconds. |
|
|
| keep |
| Peers for API server NetworkPolicyingress. |
|
|
| keep |
| Ports for API server NetworkPolicyingress (if from is set). |
Examples: ports: - port: 8080 |
|
| keep |
| Select certain nodes for API server pods. |
|
|
| keep |
| Annotations to add to the API server pods. |
|
|
| keep |
| Max unavailable pods for API server. |
|
|
| Remove maxUnavailable and minAvailable and set config: {} |
| Min available pods for API server. |
|
|
| |
| Enable pod disruption budget. |
|
|
| keep |
| Specify priority for API server pods. |
|
|
| keep |
| API server Readiness probe failure threshold. |
|
|
| keep |
| API server Readiness probe initial delay. |
|
|
| keep |
| API server Readiness probe period seconds. |
|
|
| keep |
| API server Readiness probe scheme. |
|
|
| keep |
| API server Readiness probe timeout seconds. |
|
|
| keep |
| How many Airflow API server replicas should run. This setting is ignored when HPA (Horizontal Pod Autoscaler) is enabled |
|
|
| keep |
| Resources for API server pods. |
Examples: resources: limits: cpu: 100m memory: 128Mi requests: cpu: 100m memory: 128Mi |
|
| keep |
| Annotations for the API server Service. |
|
|
| keep |
| API server Service loadBalancerIP. |
|
|
| keep |
| API server Service |
Examples: loadBalancerSourceRanges: - 10.123.0.0/16 |
|
| keep |
| Ports for the API server Service. |
Examples: ports: - name: api-server port: 8080 targetPort: api-server ports: - name: only_sidecar port: 9080 targetPort: 8888 |
|
| keep |
| API server Service type. |
|
|
| keep |
| Annotations to add to the API server Kubernetes ServiceAccount. |
|
|
| keep |
| Specifies if ServiceAccount’s API credentials should be mounted onto Pods. |
|
|
| keep |
| Specifies whether a ServiceAccount should be created. |
|
|
| keep |
| The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name. |
|
|
| keep |
| API server Startup probe failure threshold. |
|
|
| keep |
| API server Startup probe initial delay seconds. |
|
|
| keep |
| API server Startup probe period seconds. |
|
|
| keep |
| API server Startup probe scheme. |
|
|
| keep |
| API server Startup probe timeout seconds. |
|
|
| keep |
| Specifies the strategy used to replace old Pods by new ones. |
|
|
| keep |
| Specify Tolerations for API server pods. |
|
|
| keep |
| Enable wait-for-airflow-migrations init container. |
|
|
| Not sure if this should be a container-level flag (I don't have a specific use case in mind when it would make sense to have it only in one component and not in another) |
| Add additional env vars to wait-for-airflow-migrations init container. |
|
|
|
Scheduler
Parameter | Description | Default | Opinion Jens | Opinion Bugra | Opinion Przemek |
|---|---|---|---|---|---|
| Specify scheduling constraints for scheduler pods. |
|
|
| keep |
| Annotations to add to the scheduler deployment |
|
|
| keep |
| Args to use when running the Airflow scheduler (templated). |
|
|
| keep |
| Command to use when running the Airflow scheduler (templated). |
|
|
| keep |
| Enable scheduler |
|
|
| keep |
| Add additional env vars to scheduler. |
|
|
| keep |
| Launch additional containers into scheduler (templated). |
|
|
| keep |
| Add additional init containers into scheduler (templated). |
|
|
| keep |
| Mount additional volumes into scheduler. |
|
|
| keep |
| Mount additional volumes into scheduler. |
|
|
| keep |
| HostAliases for the scheduler pod. |
Examples: hostAliases: - hostnames: - foo.local ip: 127.0.0.1 hostAliases: - hostnames: - foo.remote ip: 10.1.2.3 |
|
| keep |
| Labels to add to the scheduler objects and pods. |
|
|
| keep |
| Command for LivenessProbe | ~ |
|
| keep |
| Minimum consecutive failures for the probe to be considered failed after having succeeded. Minimum value is 1. |
|
|
| keep |
| Number of seconds after the container has started before liveness probes are initiated. |
|
|
| keep |
| How often (in seconds) to perform the probe. Minimum value is 1. |
|
|
| keep |
| Number of seconds after which the probe times out. Minimum value is 1 seconds. |
|
|
| keep |
| Select certain nodes for scheduler pods. |
|
|
| keep |
| Annotations to add to the scheduler pods. |
|
|
| keep |
| Max unavailable pods for scheduler. |
|
|
| Remove maxUnavailable and minAvailable and set config: {} |
| Min available pods for scheduler. |
|
|
| |
| Enable pod disruption budget. |
|
|
| keep |
| Specify priority for scheduler pods. |
|
|
| keep |
| Airflow 2.0 allows users to run multiple schedulers. This feature is only recommended for MySQL 8+ and PostgreSQL |
|
|
| keep |
| Resources for scheduler pods. |
Examples: resources: limits: cpu: 100m memory: 128Mi requests: cpu: 100m memory: 128Mi |
|
| keep |
| This setting tells Kubernetes that its ok to evict when it wants to scale a node down. |
|
|
| keep |
| Security context for the scheduler pod (deprecated, use securityContexts instead). If not set, the values from securityContext will be used. |
Examples: securityContext: fsGroup: 0 runAsGroup: 0 runAsUser: 50000 |
|
| keep |
| Annotations to add to the scheduler Kubernetes ServiceAccount. |
|
|
| keep |
| Specifies if ServiceAccount’s API credentials should be mounted onto Pods. When false, you can use serviceAccountTokenVolume to manually configure service account token volume for pod-launching executors. |
|
|
| keep |
| Specifies whether a ServiceAccount should be created. |
|
|
| keep |
| The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name. |
|
|
| keep |
| Intended audience of the token. Optional - defaults to the identifier of the Kubernetes API server. |
|
|
| Configuration options for other components are limited compared to this one. Maybe we should align the other component options with it? |
| Enable manual service account token volume configuration. |
|
|
| |
| Token expiration time in seconds. |
|
|
| |
| Path where the service account token volume will be mounted. |
|
|
| |
| Name of the service account token volume. |
|
|
| |
| Command for LivenessProbe | ~ |
|
| keep |
| Minimum consecutive failures for the probe to be considered failed after having succeeded. Minimum value is 1. |
|
|
| keep |
| Number of seconds after the container has started before startup probes are initiated. |
|
|
| keep |
| How often (in seconds) to perform the probe. Minimum value is 1. |
|
|
| keep |
| Number of seconds after which the probe times out. Minimum value is 1 seconds. |
|
|
| keep |
| Specifies the strategy used to replace old Pods by new ones when deployed as a Deployment (when not using LocalExecutor and workers.persistence). |
|
|
| keep |
| Grace period for scheduler to finish after SIGTERM is sent from Kubernetes. |
|
|
| keep |
| Specify Tolerations for scheduler pods. |
|
|
| keep |
| Specify topology spread constraints for scheduler pods. |
|
|
| keep |
| Specifies the strategy used to replace old Pods by new ones when deployed as a StatefulSet (when using LocalExecutor and workers.persistence). |
|
|
| keep |
| Enable wait-for-airflow-migrations init container. |
|
|
| Not sure if this should be a container-level flag (I don't have a specific use case in mind when it would make sense to have it only in one component and not in another) |
| Add additional env vars to wait-for-airflow-migrations init container. |
|
|
|
Webserver
Opinion Jens: Full section will be dropped in 1.21 with removal of Airflow 2 support. Big complexity reduction!
Opinion Przemek: Not sure if 1.21, but in general, yes - whole section drop with Airflow 2 support.
Workers
Parameter | Description | Default | Opinion Jens | Opinion Bugra | Opinion Przemek |
|---|---|---|---|---|---|
| Specify scheduling constraints for Airflow Celery worker pods and pods created with pod-template-file. |
| All "workers" options should go to "workers.celery", Deprecations /fallback removed |
| Move proper sections to workers.celery/workers.kubernetes sections and remove all other under the workers section (finish work started in https://github.com/apache/airflow/issues/28880) |
| Annotations to add to the Airflow Celery worker deployment. |
| ditto |
| |
| Args to use when running Airflow Celery workers (templated) (deprecated, use workers.celery.args instead). |
| ditto |
| |
| Args to use when running Airflow Celery workers (templated). |
|
|
| keep |
| Command to use when running Airflow Celery workers (templated). |
|
|
| keep |
| Enable the default worker defined by the workers and workers.celery configurations. |
| Should we have a "default" worker and sets in parallel? Can we not make the set havinf one default to reduce complexity? |
| keep |
| Enable Kerberos init container. |
|
|
| Move to Kustomize with the whole Kerberos stuff |
| Resources on kerberos init container. |
Examples: resources: limits: cpu: 100m memory: 128Mi requests: cpu: 100m memory: 128Mi |
|
| |
| Command for LivenessProbe | ~ |
|
| keep |
| Enable liveness probe for Airflow Celery workers. |
|
|
| keep |
| Minimum consecutive failures for the probe to be considered failed after having succeeded. Minimum value is 1. |
|
|
| keep |
| Number of seconds after the container has started before liveness probes are initiated. |
|
|
| keep |
| How often (in seconds) to perform the probe. Minimum value is 1. |
|
|
| keep |
| Number of seconds after which the probe times out. Minimum value is 1 seconds. |
|
|
| keep |
| Annotations to add to Airflow Celery worker volumes. |
|
|
| keep |
| Enable persistent volumes. |
|
|
| keep |
| Execute init container to chown log directory. This is currently only needed in kind, due to usage of local-path provisioner. |
|
|
| keep |
| Volume size for Airflow Celery worker StatefulSet. |
|
|
| keep |
| If using a custom StorageClass, pass name ref to all StatefulSets here (templated). |
|
|
| keep |
| Specifies the policy for managing pods within the Airflow Celery worker. Only applicable to StatefulSet. |
|
|
| keep |
| Queue name for the worker. |
|
|
| keep |
| Number of Airflow Celery workers. |
|
|
| keep |
| List of worker sets. Each item can overwrite values from the parent workers and workers.celery sections. |
| Here all properties from above and below should go in, with one "default"? |
| |
| Specifies the strategy used to replace old Airflow Celery worker pods by new ones when deployed as a Deployment. |
|
|
| keep |
| Specifies the strategy used to replace old Airflow Celery worker pods by new ones when deployed as a StatefulSet. |
|
|
| keep |
| Command to use when running Airflow Celery workers and using pod-template-file (templated). Use workers.celery.command and/or workers.kubernetes.command to separate value between Celery workers and pod-template-file. |
| ^^^ ditto |
| Move proper sections to workers.celery/workers.kubernetes sections and remove all other under the workers section (finish work started in https://github.com/apache/airflow/issues/28880) |
| Add additional env vars to the Airflow Celery workers and pods created with pod-template-file. |
| ditto |
| |
| Launch additional containers into Airflow Celery workers and pods created with pod-template-file (templated). Note, if used with KubernetesExecutor, you are responsible for signaling sidecars to exit when the main container finishes so Airflow can continue the worker shutdown process! |
| ditto |
| |
| Add additional init containers into Airflow Celery workers and pods created with pod-template-file (templated). |
| ditto |
| |
| Expose additional ports of Airflow Celery worker container. |
| ditto |
| |
| Additional volume mounts attached to the Airflow Celery workers and pods created with pod-template-file. |
| ditto |
| |
| Additional volumes attached to the Airflow Celery workers and pods created with pod-template-file. |
| ditto |
| |
| Specify HostAliases for Airflow Celery worker pods and pods created with pod-template-file. |
Examples: hostAliases: - hostnames: - test.hostname.one ip: 127.0.0.2 hostAliases: - hostnames: - test.hostname.two ip: 127.0.0.3 | ditto |
| |
| HorizontalPodAutoscalerBehavior configures the scaling behavior of the target. |
| Should we have HPA and KEDA? How about moving this to Kustomize? | I really love this idea, as mentioned above, I would move them to Kustomize. | |
| Allow HPA autoscaling (KEDA must be disabled). |
|
|
| |
| Maximum number of Airflow Celery workers created by HPA. |
|
|
| |
| Specifications for which to use to calculate the desired replica count. |
|
|
| |
| Minimum number of Airflow Celery workers created by HPA. |
|
|
| |
| Advanced KEDA configuration. |
|
|
| Move to Kustomize
|
| HorizontalPodAutoscalerConfig specifies horizontal scale config. |
|
|
| |
| HorizontalPodAutoscalerBehavior configures the scaling behavior of the target. |
|
|
| |
| How many seconds KEDA will wait before scaling to zero. |
|
|
| |
| Allow KEDA autoscaling. |
|
|
| |
| Maximum number of Airflow Celery workers created by KEDA. |
|
|
| |
| Minimum number of Airflow Celery workers created by KEDA. |
|
|
| |
| Labels used in matchLabels for namespace in the PgBouncer NetworkPolicy. |
|
|
| |
| How often KEDA polls the airflow DB to report new scale requests to the HPA. |
|
|
| |
| Query to use for KEDA autoscaling. Must return a single integer. |
|
|
| |
| Weather to use PGBouncer to connect to the database or not when it is enabled. This configuration will be ignored if PGBouncer is not enabled. |
|
|
| |
| Enable Kerberos init container. |
| ^^^ ditto | I agree, and Kerberos in Kustomize looks more manageable from both perspectives. | |
| Resources on kerberos init container. |
Examples: resources: limits: cpu: 100m memory: 128Mi requests: cpu: 100m memory: 128Mi | ditto |
| |
| Enable Kerberos sidecar. |
| ditto |
| |
| Resources on kerberos sidecar. |
Examples: resources: limits: cpu: 100m memory: 128Mi requests: cpu: 100m memory: 128Mi | ditto |
| |
| Command to use in pod-template-file (templated). |
|
|
| keep |
| Enable kerberos init container. |
|
|
| Move to Kustomize |
| Resources on kerberos init container. |
Examples: resources: limits: cpu: 100m memory: 128Mi requests: cpu: 100m memory: 128Mi |
|
| |
| Labels to add to the Airflow Celery workers objects and pods created with pod-template-file. |
| ^^^ ditto |
| Move proper sections to workers.celery/workers.kubernetes sections and remove all other under the workers section (finish work started in https://github.com/apache/airflow/issues/28880) |
| Command for LivenessProbe (deprecated, use workers.celery.livenessProbe.command instead) | ~ | ditto |
| |
| Enable liveness probe for Airflow Celery workers (deprecated, use workers.celery.livenessProbe.enabled instead). |
| ditto |
| |
| Minimum consecutive failures for the probe to be considered failed after having succeeded. Minimum value is 1 (deprecated, use workers.celery.livenessProbe.failureThreshold instead). |
| ditto |
| |
| Number of seconds after the container has started before liveness probes are initiated (deprecated, use workers.celery.livenessProbe.initialDelaySeconds instead). |
| ditto |
| |
| How often (in seconds) to perform the probe. Minimum value is 1 (deprecated, use workers.celery.livenessProbe.periodSeconds instead). |
| ditto |
| |
| Number of seconds after which the probe times out. Minimum value is 1 seconds (deprecated, use workers.celery.livenessProbe.timeoutSeconds instead). |
| ditto |
| |
| Select certain nodes for Airflow Celery worker pods and pods created with pod-template-file. |
| ditto |
| |
| Annotations to add to Airflow Celery worker volumes (deprecated, use workers.celery.persistence.annotations instead). |
| ditto |
| |
| Enable persistent volumes (deprecated, use workers.celery.persistence.enabled instead). |
| ditto |
| |
| Execute init container to chown log directory. This is currently only needed in kind, due to usage of local-path provisioner (deprecated, use workers.celery.persistence.fixPermissions instead). |
| ditto |
| Probably remove and/or move to Kustomize. It is only used in volume-permissions sidecar in workers, which, looking at the comment, is only needed with the local path provisioner https://github.com/rancher/local-path-provisioner setup. |
| Volume size for Airflow Celery worker StatefulSet (deprecated, use workers.celery.persistence.size instead). |
| ditto |
| Move proper sections to workers.celery/workers.kubernetes sections and remove all other under the workers section (finish work started in https://github.com/apache/airflow/issues/28880)
|
| If using a custom StorageClass, pass name ref to all StatefulSets here (templated) (deprecated, use workers.celery.persistence.storageClassName instead). |
| ditto |
| |
| Annotations to add to the Airflow Celery workers and pods created with pod-template-file. |
| ditto |
| |
| Max unavailable pods for worker. |
| ditto |
| |
| Min available pods for worker. |
| ditto |
| |
| Enable pod disruption budget. |
| ditto |
| |
| Specifies the policy for managing pods within the Airflow Celery worker (deprecated, use workers.celery.podManagementPolicy instead). Only applicable to StatefulSet. |
| ditto |
| |
| Specify priority for Airflow Celery worker pods and pods created with pod-template-file. |
| ditto |
| |
| Number of Airflow Celery workers (deprecated, use workers.celery.replicas instead). |
| ditto |
| |
| Resource configuration for Airflow Celery workers and pods created with pod-template-file. |
Examples: resources: limits: cpu: 100m memory: 128Mi requests: cpu: 100m memory: 128Mi | ditto |
| |
| Specify runtime for Airflow Celery worker pods and pods created with pod-template-file. |
| ditto |
| |
| This setting tells Kubernetes that it’s ok to evict when it wants to scale a node down. It is used by Airflow Celery workers and pod-template-file. |
| ditto |
| |
| Security context for the Airflow Celery worker pods and pods created with pod-template-file (deprecated, use securityContexts instead). If not set, the values from securityContext will be used. |
Examples: securityContext: fsGroup: 0 runAsGroup: 0 runAsUser: 50000 | ditto |
| |
| Annotations to add to the worker Kubernetes ServiceAccount. |
| ditto |
| |
| Specifies if ServiceAccount’s API credentials should be mounted onto Pods |
| ditto |
| |
| Specifies whether a ServiceAccount should be created. |
| ditto |
| |
| The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name. |
| ditto |
| |
| Specifies the strategy used to replace old Airflow Celery worker pods by new ones when deployed as a Deployment (deprecated, use workers.celery.strategy instead). |
| ditto |
| |
| Grace period for tasks to finish after SIGTERM is sent from Kubernetes. It is used by Airflow Celery workers and pod-template-file. |
| ditto |
| |
| Specify Tolerations for Airflow Celery worker pods and pods created with pod-template-file. |
| ditto |
| |
| Specify topology spread constraints for Airflow Celery worker pods and pods created with pod-template-file. |
| ditto |
| |
| Specifies the strategy used to replace old Airflow Celery worker pods by new ones when deployed as a StatefulSet (deprecated, use workers.celery.updateStrategy instead). |
| ditto |
| |
| Specify additional volume claim template for Airflow Celery workers. |
Examples: volumeClaimTemplates: - accessModes: - ReadWriteOnce name: data-volume-1 resources: requests: storage: 10Gi storageClassName: storage-class-1 volumeClaimTemplates: - accessModes: - ReadWriteOnce name: data-volume-2 resources: requests: storage: 20Gi storageClassName: storage-class-2 | ditto |
| |
| Enable wait-for-airflow-migrations init container. |
| ditto |
| |
| Add additional env vars to wait-for-airflow-migrations init container. |
| ditto |
|
Triggerer
Parameter | Description | Default | Opinion Jens | Opinion Bugra | Opinion Przemek |
|---|---|---|---|---|---|
| Specify scheduling constraints for triggerer pods. |
|
|
| keep |
| Annotations to add to the triggerer deployment |
|
|
| keep |
| Args to use when running the Airflow triggerer (templated). |
|
|
| keep |
| Command to use when running the Airflow triggerer (templated). |
|
|
| keep |
| Enable triggerer |
|
|
| keep |
| Add additional env vars to triggerer. |
|
|
| keep |
| Launch additional containers into triggerer (templated). |
|
|
| keep |
| Add additional init containers into triggerer (templated). |
|
|
| keep |
| Mount additional volumes into triggerer. |
|
|
| keep |
| Mount additional volumes into triggerer. |
|
|
| keep |
| HostAliases for the triggerer pod. |
Examples: hostAliases: - hostnames: - foo.local ip: 127.0.0.1 hostAliases: - hostnames: - foo.remote ip: 10.1.2.3 |
|
| keep |
| Advanced KEDA configuration. |
| Same like above: Should we have KEDA/HPA as Kustomize only? |
I agree moving idea as above | Move to Kustomize |
| HorizontalPodAutoscalerConfig specifies horizontal scale config. |
|
| ||
| HorizontalPodAutoscalerBehavior configures the scaling behavior of the target. |
|
| ||
| How many seconds KEDA will wait before scaling to zero. |
|
| ||
| Allow KEDA autoscaling. |
|
| ||
| Maximum number of triggerers created by KEDA. |
|
| ||
| Minimum number of triggerers created by KEDA. |
|
| ||
| Labels used in matchLabels for namespace in the PgBouncer NetworkPolicy. |
|
| ||
| How often KEDA polls the airflow DB to report new scale requests to the HPA. |
|
| ||
| Query to use for KEDA autoscaling. Must return a single integer. |
|
| ||
| Whether to use PGBouncer to connect to the database or not when it is enabled. This configuration will be ignored if PGBouncer is not enabled. |
|
| ||
| Labels to add to the triggerer objects and pods. |
|
|
| keep |
| Command for LivenessProbe | ~ |
|
| keep |
| Minimum consecutive failures for the probe to be considered failed after having succeeded. Minimum value is 1. |
|
|
| keep |
| Number of seconds after the container has started before liveness probes are initiated. |
|
|
| keep |
| How often (in seconds) to perform the probe. Minimum value is 1. |
|
|
| keep |
| Number of seconds after which the probe times out. Minimum value is 1 seconds. |
|
|
| keep |
| Select certain nodes for triggerer pods. |
|
|
| keep |
| Annotations to add to triggerer volumes. |
|
|
| keep |
| Enable persistent volumes. |
|
|
| keep |
| Execute init container to chown log directory. This is currently only needed in kind, due to usage of local-path provisioner. |
|
|
| Not used - drop |
| Volume size for triggerer StatefulSet. |
|
|
| keep |
| If using a custom StorageClass, pass name ref to all StatefulSets here (templated). |
|
|
| keep |
| Annotations to add to the triggerer pods. |
|
|
| keep |
| Max unavailable pods for triggerer. |
|
|
| Remove maxUnavailable and minAvailable and set config: {} |
| Min available pods for triggerer. |
|
|
| |
| Enable pod disruption budget. |
|
|
| keep |
| Specify priority for triggerer pods. |
|
|
| keep |
| Number of triggerers to run. |
|
|
| keep |
| Resources for triggerer pods. |
Examples: resources: limits: cpu: 100m memory: 128Mi requests: cpu: 100m memory: 128Mi |
|
| keep |
| This setting tells Kubernetes that its ok to evict when it wants to scale a node down. |
|
|
| keep |
| Security context for the triggerer pod (deprecated, use securityContexts instead). If not set, the values from securityContext will be used. |
Examples: securityContext: fsGroup: 0 runAsGroup: 0 runAsUser: 50000 |
|
| Drop |
| Annotations to add to the triggerer Kubernetes ServiceAccount. |
|
|
| keep |
| Specifies if ServiceAccount’s API credentials should be mounted onto Pods. |
|
|
| keep |
| Specifies whether a ServiceAccount should be created. |
|
|
| keep |
| The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name. |
|
|
| keep |
| Specifies the strategy used to replace old Pods by new ones when deployed as a Deployment. |
|
|
| keep |
| Grace period for tasks to finish after SIGTERM is sent from Kubernetes. |
|
|
| keep |
| Specify Tolerations for triggerer pods. |
|
|
| keep |
| Specify topology spread constraints for triggerer pods. |
|
|
| keep |
| Specifies the strategy used to replace old Pods by new ones when deployed as a StatefulSet. |
|
|
| keep |
| Enable wait-for-airflow-migrations init container. |
|
|
| Not sure if this should be a container-level flag (I don't have a specific use case in mind when it would make sense to have it only in one component and not in another) |
| Add additional env vars to wait-for-airflow-migrations init container. |
|
|
|
DagProcessor
Parameter | Description | Default | Opinion Jens | Opinion Bugra | Opinion Przemek |
|---|---|---|---|---|---|
| Specify scheduling constraints for dag processor pods. |
|
|
| keep |
| Annotations to add to the dag processor deployment |
|
|
| keep |
| Args to use when running the Airflow dag processor (templated). |
| Same like with workers: There is this PR to start a deployment per bundle. Should structure be here made similar like with workers? (one default can add more?) | Having the same, not different, structure would be probably easier (in terms of logic) to maintain, but I would wait until we will have on main branch finished work with workers.celery/workers.kubernetes sections to see how logic and complexity will look like without the whole complexity due to backward compatibility | |
| Command to use when running the Airflow dag processor (templated). |
|
|
| keep |
| Define Dag bundles in a structured YAML format. This will be automatically converted to JSON string format for config.dag_processor.dag_bundle_config_list. |
|
|
| keep |
| Keyword arguments for the Dag bundle. |
|
|
| keep |
| Enable standalone dag processor (requires Airflow 2.3.0+). |
|
|
| keep |
| Add additional env vars to dag processor. |
|
|
| keep |
| Launch additional containers into dag processor (templated). |
|
|
| keep |
| Add additional init containers into dag processor (templated). |
|
|
| keep |
| Mount additional volumes into dag processor. |
|
|
| keep |
| Mount additional volumes into dag processor. |
|
|
| keep |
| Labels specific to dag processor objects and pods |
|
|
| keep |
| Command for LivenessProbe | ~ |
|
| keep |
| Minimum consecutive failures for the probe to be considered failed after having succeeded. Minimum value is 1. |
|
|
| keep |
| Number of seconds after the container has started before liveness probes are initiated. |
|
|
| keep |
| How often (in seconds) to perform the probe. Minimum value is 1. |
|
|
| keep |
| Number of seconds after which the probe times out. Minimum value is 1 seconds. |
|
|
| keep |
| Select certain nodes for dag processor pods. |
|
|
| keep |
| Annotations to add to the dag processor pods. |
|
|
| keep |
| Max unavailable pods for dag processor. |
|
|
| Remove maxUnavailable and minAvailable and set config: {} |
| Min available pods for dag processor. |
|
|
| |
| Enable pod disruption budget. |
|
|
| keep |
| Specify priority for dag processor pods. |
|
|
| keep |
| Number of dag processors to run. |
|
|
| keep |
| Resources for dag processor pods. |
Examples: resources: limits: cpu: 100m memory: 128Mi requests: cpu: 100m memory: 128Mi |
|
| keep |
| This setting tells Kubernetes that its ok to evict when it wants to scale a node down. |
|
|
| keep |
| Security context for the dag processor pod (deprecated, use securityContexts instead). If not set, the values from securityContext will be used. |
Examples: securityContext: fsGroup: 0 runAsGroup: 0 runAsUser: 50000 |
|
| keep |
| Annotations to add to the dag processor Kubernetes ServiceAccount. |
|
|
| keep |
| Specifies if ServiceAccount’s API credentials should be mounted onto Pods. |
|
|
| keep |
| Specifies whether a ServiceAccount should be created. |
|
|
| keep |
| The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name. |
|
|
| keep |
| Specifies the strategy used to replace old Pods by new ones when deployed as a Deployment. |
|
|
| keep |
| Grace period for tasks to finish after SIGTERM is sent from Kubernetes. |
|
|
| keep |
| Specify Tolerations for dag processor pods. |
|
|
| keep |
| Specify topology spread constraints for dag processor pods. |
|
|
| keep |
| Enable wait-for-airflow-migrations init container. |
|
|
| Not sure if this should be a container-level flag (I don't have a specific use case in mind when it would make sense to have it only in one component and not in another) |
| Add additional env vars to wait-for-airflow-migrations init container. |
|
|
|
Flower
Parameter | Description | Default | Opinion Jens | Opinion Bugra | Opinion Przemek |
|---|---|---|---|---|---|
| Specify scheduling constraints for Flower pods. |
|
|
| Move to Kustomize |
| Annotations to add to the flower deployment |
|
|
| |
| Args to use when running flower (templated). |
|
|
| |
| Command to use when running flower (templated). |
|
|
| |
| Enable Flower. |
| Is flower a component that should be supported by helm chart? Not complex but could be carved-out into a separate deployment add-on in Kustomize... | I agree to this as well. Let's move them out. It won't add much load but I also mentioned a similar thing for Pgbouncer too. How about approaching all these as the same? | |
| Add additional env vars to flower. |
|
|
| |
| Launch additional containers into the flower pods. |
|
|
| |
| Additional NetworkPolicies as needed (Deprecated - renamed to flower.networkPolicy.ingress.from). |
|
|
| |
| Mount additional volumes into the flower pods. |
|
|
| |
| Mount additional volumes into the flower pods. |
|
|
| |
| Labels to add to the flower objects and pods. |
|
|
| |
| Flower Liveness probe failure threshold. |
|
|
| |
| Flower Liveness probe initial delay. |
|
|
| |
| Flower Liveness probe period seconds. |
|
|
| |
| Flower Liveness probe timeout seconds. |
|
|
| |
| Peers for flower NetworkPolicyingress. |
|
|
| |
| Ports for flower NetworkPolicyingress (if from is set). |
Examples: ports: - port: 5565 |
|
| |
| Select certain nodes for Flower pods. |
|
|
| |
| Password use to access Flower. |
|
|
| |
| Annotations to add to the Flower pods. |
|
|
| |
| Specify priority for Flower pods. |
|
|
| |
| Flower Readiness probe failure threshold. |
|
|
| |
| Flower Readiness probe initial delay. |
|
|
| |
| Flower Readiness probe period seconds. |
|
|
| |
| Flower Readiness probe timeout seconds. |
|
|
| |
| Resources for Flower pods. |
Examples: resources: limits: cpu: 100m memory: 128Mi requests: cpu: 100m memory: 128Mi |
|
| |
| Annotations to add to the flower secret. |
|
|
| |
| A secret containing the user and password pair. |
|
|
| |
| Security context for the flower pod (deprecated, use securityContexts instead). If not set, the values from securityContext will be used. |
Examples: securityContext: fsGroup: 0 runAsGroup: 0 runAsUser: 50000 |
|
| |
| Annotations for the flower Service. |
|
|
| |
| Flower Service loadBalancerIP. |
|
|
| |
| Flower Service |
Examples: loadBalancerSourceRanges: - 10.123.0.0/16 |
|
| |
| Ports for the flower Service. |
Examples: ports: - name: flower-ui port: 8080 targetPort: flower-ui |
|
| |
| Flower Service type. |
|
|
| |
| Annotations to add to the worker Kubernetes ServiceAccount. |
|
|
| |
| Specifies if ServiceAccount’s API credentials should be mounted onto Pods. |
|
|
| |
| Specifies whether a ServiceAccount should be created. |
|
|
| |
| The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name. |
|
|
| |
| Flower Startup probe failure threshold. |
|
|
| |
| Flower Startup probe initial delay seconds. |
|
|
| |
| Flower Startup probe period seconds. |
|
|
| |
| Flower Startup probe timeout seconds. |
|
|
| |
| Specify Tolerations for Flower pods. |
|
|
| |
| Specify topology spread constraints for Flower pods. |
|
|
| |
| Username use to access Flower. |
|
|
|
Redis
Parameter | Description | Default | Opinion Jens | Opinion Bugra | Opinion Przemek |
|---|---|---|---|---|---|
| Direct url to the redis broker (when using an external redis instance) (can only be set during install, not upgrade). |
|
|
| Move to workers.celery/redis as it is only for CeleryExecutor |
| Redis broker URL secret. |
|
|
| |
| Specify scheduling constraints for Redis pods. |
|
|
| From one perspective, it is outside Airflow components (additional thing), from another, it is core for CeleryExecutor setup. I would think of moving it to Kustomize, but it could also stay in the core chart |
| Annotations for the redis. |
|
|
| |
| Configuration for redis empty dir volume. |
|
|
| |
| Enable the Redis provisioned by the chart (you can also use an external Redis instance with data.brokerUrl or data.brokerUrlSecretName). |
|
|
| |
| Labels to add to the redis objects and pods. |
|
|
| |
| Select certain nodes for Redis pods. |
|
|
| |
| If password is set, create secret with it, else generate a new one on install (can only be set during install, not upgrade). |
|
|
| |
| Annotations to add to the redis password secret. |
|
|
| |
| Redis password secret. |
|
|
| |
| Annotations to add to redis volumes. |
|
|
| |
| Enable persistent volumes. |
|
|
| |
| The name of an existing PVC to use. |
|
|
| |
| Volume size for Redis StatefulSet. |
|
|
| |
| If using a custom StorageClass, pass name ref to all StatefulSets here (templated). |
|
|
| |
| Annotations to add to the redis pods. |
|
|
| |
| Specify priority for redis pods. |
|
|
| |
| Resources for the Redis pods |
Examples: resources: limits: cpu: 100m memory: 128Mi requests: cpu: 100m memory: 128Mi |
|
| |
| This setting tells Kubernetes that its ok to evict when it wants to scale a node down. |
|
|
| |
| Security context for the cleanup job pod (deprecated, use securityContexts instead). If not set, the values from securityContext will be used. |
Examples: securityContext: fsGroup: 0 runAsGroup: 0 runAsUser: 50000 |
|
| |
| If using ClusterIP service type, custom IP address can be specified. |
|
|
| |
| If using NodePort service type, custom node port can be specified. |
|
|
| |
| Service type. |
|
|
| |
| Annotations to add to the worker Kubernetes ServiceAccount. |
|
|
| |
| Specifies if ServiceAccount’s API credentials should be mounted onto Pods |
|
|
| |
| Specifies whether a ServiceAccount should be created. |
|
|
| |
| The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name. |
|
|
| |
| Grace period for Redis to exit after SIGTERM is sent from Kubernetes. |
|
|
| |
| Specify Tolerations for Redis pods. |
|
|
| |
| Specify topology spread constraints for Redis pods. |
|
|
| |
| Redis run as user parameter. |
|
|
|
StatsD
Parameter | Description | Default | Opinion Jens | Opinion Bugra | Opinion Przemek |
|---|---|---|---|---|---|
| Specify scheduling constraints for StatsD pods. |
|
|
| Move to Kustomize & document OpenTelemetry setup too |
| Annotations to add to the StatsD deployment. |
|
|
| |
| Args to use when running statsd-exporter (templated). |
|
|
| |
| Maximum number of metric mappings to cache in memory. Higher values improve performance for frequently used metrics but consume more memory. |
|
|
| |
| Time-to-live for cached metric mappings. Determines how long mappings remain in cache before expiring. Set to ‘0s’ to disable expiration. |
|
|
| |
| Cache eviction strategy for metric mappings. lru (Least Recently Used) evicts oldest accessed items, ‘random’ evicts randomly selected items. |
|
|
| |
| Extra annotations to apply to the statsd configmap. |
|
|
| |
| Enable StatsD. |
|
|
| |
| Add additional env vars to statsd container. |
|
|
| |
| Additional mappings for StatsD exporter.If set, will merge default mapping and extra mappings, default mapping has higher priority. So, if you want to change some default mapping, please use overrideMappings |
|
|
| |
| Additional NetworkPolicies as needed. |
|
|
| |
| Labels specific to statsd objects and pods |
|
|
| |
| Select certain nodes for StatsD pods. |
|
|
| |
| Override mappings for StatsD exporter.If set, will ignore setting item in default and extraMappings. So, If you use it, ensure all mapping item contains in it. |
|
|
| |
| Annotations to add to the StatsD pods. |
|
|
| |
| Specify priority for StatsD pods. |
|
|
| |
| Resources for StatsD pods. |
Examples: resources: limits: cpu: 100m memory: 128Mi requests: cpu: 100m memory: 128Mi |
|
| |
| Security context for the StatsD pod (deprecated, use securityContexts instead). |
Examples: securityContext: fsGroup: 0 runAsGroup: 0 runAsUser: 50000 |
|
| |
| Extra annotations for the StatsD Service. |
|
|
| |
| Annotations to add to the StatsD Kubernetes ServiceAccount. |
|
|
| |
| Specifies if ServiceAccount’s API credentials should be mounted onto Pods. |
|
|
| |
| Specifies whether a ServiceAccount should be created. |
|
|
| |
| The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name. |
|
|
| |
| Grace period for statsd to finish after SIGTERM is sent from Kubernetes. |
|
|
| |
| Specify Tolerations for StatsD pods. |
|
|
| |
| Specify topology spread constraints for StatsD pods. |
|
|
| |
| StatsD run as user parameter. |
|
|
|
Jobs
Parameter | Description | Default | Opinion Jens | Opinion Bugra | Opinion Przemek |
|---|---|---|---|---|---|
| Specify scheduling constraints for cleanup pods. |
|
|
| Move it under workers.kubernetes section as it is needed only for KubernetesExecutor |
| Args to use when running the cleanup cronjob (templated). |
|
|
| |
| Command to use when running the cleanup cronjob (templated). |
|
|
| |
| Enable cleanup. |
|
|
| |
| Add additional env vars to cleanup. |
|
|
| |
| Annotations to add to the cleanup cronjob. |
|
|
| |
| labels to add to cleanup pods. |
|
|
| |
| Select certain nodes for cleanup pods. |
|
|
| |
| Annotations to add to cleanup pods. |
|
|
| |
| Specify priority for cleanup pods. |
|
|
| |
| Resources for cleanup pods |
Examples: resources: limits: cpu: 100m memory: 128Mi requests: cpu: 100m memory: 128Mi |
|
| |
| Cleanup schedule (templated). |
|
|
| |
| Security context for the cleanup job pod (deprecated, use securityContexts instead). If not set, the values from securityContext will be used. |
Examples: securityContext: fsGroup: 0 runAsGroup: 0 runAsUser: 50000 |
|
| |
| Annotations to add to the cleanup CronJob Kubernetes ServiceAccount. |
|
|
| |
| Specifies if ServiceAccount’s API credentials should be mounted onto Pods |
|
|
| |
| Specifies whether a ServiceAccount should be created. |
|
|
| |
| The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name. |
|
|
| |
| Specify Tolerations for cleanup pods. |
|
|
| |
| Specify topology spread constraints for cleanup pods. |
|
|
| |
| Specify scheduling constraints for the create user job pod. |
|
|
| Move under the new flask section Agreed on making createUserJob.enabled false by default |
| Annotations to add to the create user job pod. |
|
|
| |
| Specify if you want additional configured env vars applied to this job |
|
|
| |
| Args to use when running create user job (templated). |
|
|
| |
| Command to use when running create user job (templated). |
|
|
| |
| Default user email address. |
|
| ||
| Default user firstname. |
|
|
| |
| Default user lastname. |
|
|
| |
| Default user password. |
|
|
| |
| Default user role. |
|
|
| |
| Default user username. |
|
|
| |
| Whether the create user job should be created. |
| Should we make this Fals eas default not to be insecure by default? | Make sense. The only thing people can expect this is true by default. Maybe we can add docs there to route them to | |
| Add additional env vars to the create user job pod. |
|
|
| |
| Launch additional containers for the create user job pod |
|
|
| |
| Add additional init containers into create user job pod (templated). |
|
|
| |
| Mount additional volumes into create user job |
|
|
| |
| Mount additional volumes into create user job |
|
|
| |
| Annotations to add to the create user job job. |
|
|
| |
| Labels to add to the create user job objects and pods. |
|
|
| |
| Select certain nodes for the create user job pod. |
|
|
| |
| Specify priority for the create user job pod. |
|
|
| |
| Resources for the create user job pod |
Examples: resources: limits: cpu: 100m memory: 128Mi requests: cpu: 100m memory: 128Mi |
|
| |
| Restart policy for the database migration job. | OnFailure |
|
| |
| Security context for the create user job pod (deprecated, use securityContexts instead). If not set, the values from securityContext will be used. |
Examples: securityContext: fsGroup: 0 runAsGroup: 0 runAsUser: 50000 |
|
| |
| Annotations to add to the create user job Kubernetes ServiceAccount. |
|
|
| |
| Specifies if ServiceAccount’s API credentials should be mounted onto Pods. |
|
|
| |
| Specifies whether a ServiceAccount should be created. |
|
|
| |
| The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name. |
|
|
| |
| Specify Tolerations for the create user job pod. |
|
|
| |
| Specify topology spread constraints for the create user job pod. |
|
|
| |
| Limit the lifetime of the job object after it finished execution |
|
|
| |
| Specify if you want to use the default Helm Hook annotations |
|
|
| |
| Specify scheduling constraints for database cleanup pods. |
|
|
| keep |
| Specify if you want additional configured env vars applied to database cleanup job |
|
|
| keep |
| Args to use when running the database cleanup cronjob (templated). |
|
|
| keep |
| Maximum number of rows to delete or archive in a single transaction. |
|
|
| keep |
| Command to use when running the database cleanup cronjob (templated). |
|
|
| keep |
| Enable database cleanup. |
|
|
| keep |
| Add additional env vars to database cleanup. |
|
|
| keep |
| Annotations to add to the database cleanup cronjob. |
|
|
| keep |
| labels to add to database cleanup pods. |
|
|
| keep |
| Select certain nodes for database cleanup pods. |
|
|
| keep |
| Annotations to add to database cleanup pods. |
|
|
| keep |
| Specify priority for database cleanup pods. |
|
|
| keep |
| Resources for database cleanup pods |
Examples: resources: limits: cpu: 100m memory: 128Mi requests: cpu: 100m memory: 128Mi |
|
| keep |
| Number of days to retain records in the metadata database. |
|
|
| keep |
| Database cleanup schedule (templated). |
|
|
| keep |
| Annotations to add to the database cleanup CronJob Kubernetes ServiceAccount. |
|
|
| keep |
| Specifies if ServiceAccount’s API credentials should be mounted onto Pods |
|
|
| keep |
| Specifies whether a ServiceAccount should be created. |
|
|
| keep |
| The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name. |
|
|
| keep |
| Don’t preserve purged records in an archive table. |
|
|
| keep |
| Table names to perform maintenance on. Supported values in: https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#clean |
|
|
| keep |
| Specify Tolerations for database cleanup pods. |
|
|
| keep |
| Specify topology spread constraints for database cleanup pods. |
|
|
| keep |
| Make logging output more verbose. |
|
|
| keep |
| Specify scheduling constraints for the migrate database job pod. |
|
|
| keep |
| Annotations to add to the migrate database job pod. |
|
|
| keep |
| Specify if you want additional configured env vars applied to this job |
|
|
| keep |
| Args to use when running migrate database job (templated). |
|
|
| keep |
| Command to use when running migrate database job (templated). |
|
|
| keep |
| Enable migrate database job. |
|
|
| keep |
| Add additional env vars to migrate database job. |
|
|
| keep |
| Launch additional containers for the migrate database job pod |
|
|
| keep |
| Add additional init containers into migrate database job (templated). |
|
|
| keep |
| Mount additional volumes into migrate database job |
|
|
| keep |
| Mount additional volumes into migrate database job |
|
|
| keep |
| Annotations to add to the migrate database job. |
|
|
| keep |
| Labels to add to the migrate database job objects and pods. |
|
|
| keep |
| Select certain nodes for the migrate database job pod. |
|
|
| keep |
| Specify priority for the migrate database job pod. |
|
|
| keep |
| Resources for the migrate database job pod |
Examples: resources: limits: cpu: 100m memory: 128Mi requests: cpu: 100m memory: 128Mi |
|
| keep |
| Restart policy for the database migration job. | OnFailure |
|
| keep |
| Security context for the migrate database job pod (deprecated, use securityContexts instead). If not set, the values from securityContext will be used. |
Examples: securityContext: fsGroup: 0 runAsGroup: 0 runAsUser: 50000 |
|
| keep |
| Annotations to add to the migrate database job Kubernetes ServiceAccount. |
|
|
| keep |
| Specifies if ServiceAccount’s API credentials should be mounted onto Pods. |
|
|
| keep |
| Specifies whether a ServiceAccount should be created. |
|
|
| keep |
| The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name. |
|
|
| keep |
| Specify Tolerations for the migrate database job pod. |
|
|
| keep |
| Specify topology spread constraints for migrate database job pod. |
|
|
| keep |
| Limit the lifetime of the job object after it finished execution |
|
|
| keep |
| Specify if you want to use the default Helm Hook annotations |
|
|
| Modify the chart deployment logic in a way that this will not be needed in the sense that:
|
Kubernetes
Parameter | Description | Default | Opinion Jens | Opinion Bugra | Opinion Przemek |
|---|---|---|---|---|---|
| Specify scheduling constraints for all pods. |
|
|
| keep |
| Extra annotations to apply to the main Airflow configmap. |
|
|
| keep |
| Extra annotations to apply to all Airflow pods. |
|
|
| keep |
| Container Lifecycle Hooks definition for the API server. If not set, the values from global containerLifecycleHooks will be used. |
Examples: containerLifecycleHooks: postStart: exec: command: - /bin/sh - -c - echo postStart handler > /usr/share/message preStop: exec: command: - /bin/sh - -c - echo preStop handler > /usr/share/message |
|
| keep |
| Number of old ReplicaSets to retain. | ~ |
|
| keep |
| Container security context definition for the API server. |
Examples: container: allowPrivilegeEscalation: false capabilities: drop: - ALL |
|
| keep |
| Pod security context definition for the API server. |
Examples: pod: fsGroup: 0 runAsGroup: 0 runAsUser: 50000 |
|
| keep |
| Specify topology spread constraints for API server pods. |
|
|
| keep |
| Container security context definition for the wait for migrations. |
Examples: container: allowPrivilegeEscalation: false capabilities: drop: - ALL |
|
| Related comment under dagProcessor section next to waitForMigration section |
| Container Lifecycle Hooks definition for the cleanup. If not set, the values from global containerLifecycleHooks will be used. |
Examples: containerLifecycleHooks: postStart: exec: command: - /bin/sh - -c - echo postStart handler > /usr/share/message preStop: exec: command: - /bin/sh - -c - echo preStop handler > /usr/share/message |
|
| keep |
| The failed jobs history limit specifies the number of failed jobs to retain. | ~ |
|
| keep |
| Container security context definition for the cleanup. |
Examples: container: allowPrivilegeEscalation: false capabilities: drop: - ALL |
|
| keep |
| Pod security context definition for the cleanup. |
Examples: pod: fsGroup: 0 runAsGroup: 0 runAsUser: 50000 |
|
| keep |
| The successful jobs history limit specifies the number of finished jobs to retain. | ~ |
|
| keep |
| Default Container Lifecycle Hooks definition. The values in this parameter will be used when containerLifecycleHooks is not defined for specific containers. |
Examples: containerLifecycleHooks: postStart: exec: command: - /bin/sh - -c - echo postStart handler > /usr/share/message preStop: exec: command: - /bin/sh - -c - echo preStop handler > /usr/share/message |
|
| keep |
| Container Lifecycle Hooks definition for the create user job. If not set, the values from global containerLifecycleHooks will be used. |
Examples: containerLifecycleHooks: postStart: exec: command: - /bin/sh - -c - echo postStart handler > /usr/share/message preStop: exec: command: - /bin/sh - -c - echo preStop handler > /usr/share/message |
|
| Related comment under jobs section next to createUserJob section |
| Container security context definition for the create user job. |
Examples: container: allowPrivilegeEscalation: false capabilities: drop: - ALL |
|
| |
| Pod security context definition for the create user job. |
Examples: pod: fsGroup: 0 runAsGroup: 0 runAsUser: 50000 |
|
| |
| Container Lifecycle Hooks definition for the dag processor. If not set, the values from global containerLifecycleHooks will be used. |
Examples: containerLifecycleHooks: postStart: exec: command: - /bin/sh - -c - echo postStart handler > /usr/share/message preStop: exec: command: - /bin/sh - -c - echo preStop handler > /usr/share/message |
|
| keep |
| Container security context definition for the dag processor. |
Examples: container: allowPrivilegeEscalation: false capabilities: drop: - ALL |
|
| keep |
| Pod security context definition for the dag processor. |
Examples: pod: fsGroup: 0 runAsGroup: 0 runAsUser: 50000 |
|
| keep |
| Container security context definition for the wait for migrations. |
Examples: container: allowPrivilegeEscalation: false capabilities: drop: - ALL |
|
| Related comment under dagProcessor section next to waitForMigration section |
| Container Lifecycle Hooks definition for the git sync sidecar. If not set, the values from global containerLifecycleHooks will be used. |
Examples: containerLifecycleHooks: postStart: exec: command: - /bin/sh - -c - echo postStart handler > /usr/share/message preStop: exec: command: - /bin/sh - -c - echo preStop handler > /usr/share/message |
| gitSync related, could be useful to tag if we do bundle or deprecate
| Related comment next to other dags.gitSync fields |
| Container security context definition for the git sync sidecar. |
Examples: container: allowPrivilegeEscalation: false capabilities: drop: - ALL |
| ||
| Container Lifecycle Hooks definition for the database cleanup. If not set, the values from global containerLifecycleHooks will be used. |
Examples: containerLifecycleHooks: postStart: exec: command: - /bin/sh - -c - echo postStart handler > /usr/share/message preStop: exec: command: - /bin/sh - -c - echo preStop handler > /usr/share/message |
|
| keep |
| The failed jobs history limit specifies the number of failed jobs to retain. | 1 |
|
| keep |
| Container security context definition for the database cleanup. |
Examples: container: allowPrivilegeEscalation: false capabilities: drop: - ALL |
|
| keep |
| Pod security context definition for the database cleanup. |
Examples: pod: fsGroup: 0 runAsGroup: 0 runAsUser: 50000 |
|
| keep |
| The successful jobs history limit specifies the number of finished jobs to retain. | 1 |
|
| keep |
| Extra ConfigMaps that will be managed by the chart. |
Examples: extraConfigMaps: '{{ .Release.Name }}-airflow-variables': data: |- AIRFLOW_VAR_HELLO_MESSAGE: 'Hi!' AIRFLOW_VAR_KUBERNETES_NAMESPACE: '{{ .Release.Namespace }}' |
|
| keep |
| Extra secrets that will be managed by the chart. |
Examples: extraSecrets: '{{ .Release.Name }}-airflow-connections': data: |- AIRFLOW_CONN_GCP: 'base64_encoded_gcp_conn_string' AIRFLOW_CONN_AWS: 'base64_encoded_aws_conn_string' stringData: 'AIRFLOW_CONN_OTHER: ''other_conn''' |
|
| keep |
| Container Lifecycle Hooks definition for the network policy. If not set, the values from global containerLifecycleHooks will be used. |
Examples: containerLifecycleHooks: postStart: exec: command: - /bin/sh - -c - echo postStart handler > /usr/share/message preStop: exec: command: - /bin/sh - -c - echo preStop handler > /usr/share/message |
|
Flower settings, above comment related | Related comment under flower section |
| Number of old ReplicaSets to retain. | ~ |
| ||
| Container security context definition for the network policy. |
Examples: container: allowPrivilegeEscalation: false capabilities: drop: - ALL |
| ||
| Pod security context definition for the network policy. |
Examples: pod: fsGroup: 0 runAsGroup: 0 runAsUser: 50000 |
| ||
| List of existing Kubernetes secrets containing Base64 encoded credentials to connect to private registries (will get passed to imagePullSecrets). |
|
|
| keep |
| Add common labels to all objects and pods defined in this chart. |
|
|
| keep |
| Define default/max/min values for pods and containers in namespace. |
|
|
| keep |
| Container Lifecycle Hooks definition for the migrate database job. If not set, the values from global containerLifecycleHooks will be used. |
Examples: containerLifecycleHooks: postStart: exec: command: - /bin/sh - -c - echo postStart handler > /usr/share/message preStop: exec: command: - /bin/sh - -c - echo preStop handler > /usr/share/message |
|
| Related comment under jobs section next to migrateDatabaseJob other fields |
| Container security context definition for the migrate database job. |
Examples: container: allowPrivilegeEscalation: false capabilities: drop: - ALL |
|
| |
| Pod security context definition for the migrate database job. |
Examples: pod: fsGroup: 0 runAsGroup: 0 runAsUser: 50000 |
|
| |
| Enabled network policies. |
|
|
| keep |
| Select certain nodes for all pods. |
|
|
| keep |
| Container Lifecycle Hooks definition for the PgBouncer. If not set, the values from global containerLifecycleHooks will be used. |
Examples: containerLifecycleHooks: postStart: exec: command: - /bin/sh - -c - echo postStart handler > /usr/share/message preStop: exec: command: - /bin/sh - -c - echo preStop handler > /usr/share/message |
|
| Related comment under PgBouncer section |
| Container Lifecycle Hooks definition for the metrics exporter sidecar. If not set, the values from global containerLifecycleHooks will be used. |
Examples: containerLifecycleHooks: postStart: exec: command: - /bin/sh - -c - echo postStart handler > /usr/share/message preStop: exec: command: - /bin/sh - -c - echo preStop handler > /usr/share/message |
|
| |
| Container security context definition for the metrics exporter sidecar. |
Examples: container: allowPrivilegeEscalation: false capabilities: drop: - ALL |
|
| |
| Number of old ReplicaSets to retain. | ~ |
|
| |
| Container security context definition for the PgBouncer. |
Examples: container: allowPrivilegeEscalation: false capabilities: drop: - ALL |
|
| |
| Pod security context definition for the PgBouncer. |
Examples: pod: fsGroup: 0 runAsGroup: 0 runAsUser: 65534 |
|
| |
| Priority Classes created by helm charts |
Examples: priorityClasses: - name: class1 preemptionPolicy: PreemptLowerPriority value: 10000 priorityClasses: - name: class2 preemptionPolicy: Never value: 100000 |
|
| keep |
| Define any ResourceQuotas for namespace. |
|
|
| keep |
| Specifies whether RBAC resources should be created. |
|
|
| keep |
| Specifies whether SCC RoleBinding resource should be created (refer to Production Guide). |
|
|
| keep |
| Container Lifecycle Hooks definition for the redis. If not set, the values from global containerLifecycleHooks will be used. |
Examples: containerLifecycleHooks: postStart: exec: command: - /bin/sh - -c - echo postStart handler > /usr/share/message preStop: exec: command: - /bin/sh - -c - echo preStop handler > /usr/share/message |
|
| Related comment under redis section |
| Container security context definition for the redis. |
Examples: container: allowPrivilegeEscalation: false capabilities: drop: - ALL |
|
| |
| Pod security context definition for the redis. |
Examples: pod: fsGroup: 0 runAsGroup: 0 runAsUser: 999 |
|
| |
| Credentials to connect to a private registry, these will get Base64 encoded and stored in a secret (will get passed to imagePullSecrets) (create manually the credentials secret and add to imagePullSecrets instead). |
Examples: connection: email: '...' host: '...' pass: '...' user: '...' |
|
| keep |
| Email Address |
|
|
| keep |
| Registry Server URL (e.g. https://index.docker.io/v1/ for DockerHub) |
|
|
| keep |
| Password |
|
|
| keep |
| Username |
|
|
| keep |
| Name of the Kubernetes secret containing Base64 encoded credentials to connect to a private registry (will get passed to imagePullSecrets) (Deprecated - renamed to registry.secretNames). |
|
|
| registry.secretNames is not used in chart |
| Global number of old ReplicaSets to retain. Can be overridden by each deployment’s revisionHistoryLimit | ~ |
|
| keep |
| Container Lifecycle Hooks definition for the scheduler. If not set, the values from global containerLifecycleHooks will be used. |
Examples: containerLifecycleHooks: postStart: exec: command: - /bin/sh - -c - echo postStart handler > /usr/share/message preStop: exec: command: - /bin/sh - -c - echo preStop handler > /usr/share/message |
|
| keep |
| Number of old ReplicaSets to retain. | ~ |
|
| keep |
| Container security context definition for the scheduler. |
Examples: container: allowPrivilegeEscalation: false capabilities: drop: - ALL |
|
| keep |
| Pod security context definition for the scheduler. |
Examples: pod: fsGroup: 0 runAsGroup: 0 runAsUser: 50000 |
|
| keep |
| Container security context definition for the wait for migrations. |
Examples: container: allowPrivilegeEscalation: false capabilities: drop: - ALL |
|
| Related comment under dagProcessor section next to waitForMigration section |
| Default pod security context definition (deprecated, use securityContexts instead). The values in this parameter will be used when securityContext is not defined for specific Pods |
Examples: securityContext: fsGroup: 0 runAsGroup: 0 runAsUser: 50000 |
|
| Drop |
| Default container security context definition. The values in this parameter will be used when securityContexts is not defined for specific containers |
Examples: containers: allowPrivilegeEscalation: false |
|
| keep |
| Default pod security context definition. The values in this parameter will be used when securityContexts is not defined for specific Pods. |
Examples: pod: fsGroup: 0 runAsGroup: 0 runAsUser: 50000 |
|
| keep |
| Container Lifecycle Hooks definition for the statsd. If not set, the values from global containerLifecycleHooks will be used. |
Examples: containerLifecycleHooks: postStart: exec: command: - /bin/sh - -c - echo postStart handler > /usr/share/message preStop: exec: command: - /bin/sh - -c - echo preStop handler > /usr/share/message |
|
| Related comment under statsd section |
| Number of old ReplicaSets to retain. | ~ |
|
| |
| Container security context definition for the statsd. |
Examples: container: allowPrivilegeEscalation: false capabilities: drop: - ALL |
|
| |
| Pod security context definition for the statsd. |
Examples: pod: fsGroup: 0 runAsGroup: 0 runAsUser: 50000 |
|
| |
| Specify Tolerations for all pods. |
|
|
| keep |
| Specify topology spread constraints for all pods. |
|
|
| keep |
| Container Lifecycle Hooks definition for the triggerer. If not set, the values from global containerLifecycleHooks will be used. |
Examples: containerLifecycleHooks: postStart: exec: command: - /bin/sh - -c - echo postStart handler > /usr/share/message preStop: exec: command: - /bin/sh - -c - echo preStop handler > /usr/share/message |
|
| keep |
| Number of old ReplicaSets to retain. | ~ |
|
| keep |
| Container security context definition for the triggerer. |
Examples: container: allowPrivilegeEscalation: false capabilities: drop: - ALL |
|
| keep |
| Pod security context definition for the triggerer. |
Examples: pod: fsGroup: 0 runAsGroup: 0 runAsUser: 50000 |
|
| keep |
| Container security context definition for the wait for migrations. |
Examples: container: allowPrivilegeEscalation: false capabilities: drop: - ALL |
|
| Related comment under dagProcessor section next to waitForMigration section |
| Container Lifecycle Hooks definition for the webserver. If not set, the values from global containerLifecycleHooks will be used. |
Examples: containerLifecycleHooks: postStart: exec: command: - /bin/sh - -c - echo postStart handler > /usr/share/message preStop: exec: command: - /bin/sh - -c - echo preStop handler > /usr/share/message |
|
We should either remove these or convert to an API server but I assume just removing is the way to go
| Drop with Airflow 2 |
| Container security context definition for the webserver. |
Examples: container: allowPrivilegeEscalation: false capabilities: drop: - ALL |
| ||
| Pod security context definition for the webserver. |
Examples: pod: fsGroup: 0 runAsGroup: 0 runAsUser: 50000 |
| ||
| Specify topology spread constraints for webserver pods. |
|
| ||
| Container security context definition for the wait for migrations. |
Examples: container: allowPrivilegeEscalation: false capabilities: drop: - ALL |
| ||
| Container Lifecycle Hooks definition for the kerberos init container. If not set, the values from workers.kerberosInitContainer.containerLifecycleHooks will be used. |
Examples: containerLifecycleHooks: postStart: exec: command: - /bin/sh - -c - echo postStart handler > /usr/share/message preStop: exec: command: - /bin/sh - -c - echo preStop handler > /usr/share/message |
| Kerberos, if we move to Kustomize
| Related comment under kerberos section |
| Container security context definition for the kerberos init container. |
Examples: container: allowPrivilegeEscalation: false capabilities: drop: - ALL |
| ||
| Container security context definition for the persistence. |
Examples: container: allowPrivilegeEscalation: false capabilities: drop: - ALL |
|
| keep |
| Container security context definition. |
Examples: container: allowPrivilegeEscalation: false capabilities: drop: - ALL |
|
| keep |
| Pod security context definition. |
Examples: pod: fsGroup: 0 runAsGroup: 0 runAsUser: 50000 |
|
| keep |
| Container Lifecycle Hooks definition for Airflow Celery workers and pods created with pod-template-file. If not set, the values from global containerLifecycleHooks will be used. |
Examples: containerLifecycleHooks: postStart: exec: command: - /bin/sh - -c - echo postStart handler > /usr/share/message preStop: exec: command: - /bin/sh - -c - echo preStop handler > /usr/share/message |
|
| Move to proper workers.celery/workers.kubernetes sections |
| Container Lifecycle Hooks definition for the kerberos init container. If not set, the values from global containerLifecycleHooks will be used. |
Examples: containerLifecycleHooks: postStart: exec: command: - /bin/sh - -c - echo postStart handler > /usr/share/message preStop: exec: command: - /bin/sh - -c - echo preStop handler > /usr/share/message |
|
Kerberos, if we move Kustomize. | Related comment under kerberos section |
| Container security context definition for the kerberos init container. |
Examples: container: allowPrivilegeEscalation: false capabilities: drop: - ALL |
| ||
| Container Lifecycle Hooks definition for the kerberos sidecar. If not set, the values from global containerLifecycleHooks will be used. |
Examples: containerLifecycleHooks: postStart: exec: command: - /bin/sh - -c - echo postStart handler > /usr/share/message preStop: exec: command: - /bin/sh - -c - echo preStop handler > /usr/share/message |
| ||
| Container security context definition for the kerberos sidecar. |
Examples: container: allowPrivilegeEscalation: false capabilities: drop: - ALL |
| ||
| Container Lifecycle Hooks definition for the kerberos init container. If not set, the values from workers.kerberosInitContainer.containerLifecycleHooks will be used. |
Examples: containerLifecycleHooks: postStart: exec: command: - /bin/sh - -c - echo postStart handler > /usr/share/message preStop: exec: command: - /bin/sh - -c - echo preStop handler > /usr/share/message |
| ||
| Container security context definition for the kerberos init container. |
Examples: container: allowPrivilegeEscalation: false capabilities: drop: - ALL |
| ||
| Container security context definition. |
Examples: container: allowPrivilegeEscalation: false capabilities: drop: - ALL |
|
| keep |
| Pod security context definition. |
Examples: pod: fsGroup: 0 runAsGroup: 0 runAsUser: 50000 |
|
| keep |
| Container security context definition for the persistence. |
Examples: container: allowPrivilegeEscalation: false capabilities: drop: - ALL |
|
| Move to proper workers.celery/workers.kubernetes sections |
| Max number of old Airflow Celery workers ReplicaSets to retain. | ~ |
|
| |
| Container security context definition. |
Examples: container: allowPrivilegeEscalation: false capabilities: drop: - ALL |
|
| |
| Pod security context definition. |
Examples: pod: fsGroup: 0 runAsGroup: 0 runAsUser: 50000 |
|
| |
| Container security context definition for the wait-for-airflow-migrations container. |
Examples: container: allowPrivilegeEscalation: false capabilities: drop: - ALL |
|
| Related comment under dagProcessor section next to waitForMigration section |
Ingress
Parameter | Description | Default | Opinion Jens | Opinion Jens | Opinion Przemek |
|---|---|---|---|---|---|
| Annotations for the API server Ingress. |
|
|
| keep |
| Enable API server ingress resource. |
|
|
| keep |
| The hostname for the API server Ingress. (Deprecated - renamed to ingress.apiServer.hosts) |
|
|
| drop |
| The hostnames or hosts configuration for the API server Ingress. |
|
|
| keep |
| The Ingress Class for the API server Ingress. |
|
|
| keep |
| The path for the API server Ingress. |
|
|
| keep |
| The pathType for the API server Ingress (required for Kubernetes 1.19 and above). |
|
|
| keep |
| HTTP paths to add to the API server Ingress before the default path. |
|
|
| keep |
| HTTP paths to add to the API server Ingress after the default path. |
|
|
| keep |
| Enable TLS termination for the API server Ingress. |
|
|
| drop - missing deprecation info |
| The name of a pre-created Secret containing a TLS private key and certificate. |
|
|
| drop - missing deprecation info |
| Enable all ingress resources (deprecated - use ingress.web.enabled and ingress.flower.enabled). |
|
|
| drop |
| Annotations for the flower Ingress. |
|
|
| Move to Kustomize |
| Enable flower ingress resource. |
|
|
| |
| The hostname for the flower Ingress. (Deprecated - renamed to ingress.flower.hosts) |
|
|
| |
| The hostnames or hosts configuration for the flower Ingress. |
|
|
| |
| The Ingress Class for the flower Ingress. |
|
|
| |
| The path for the flower Ingress. |
|
|
| |
| The pathType for the flower Ingress (required for Kubernetes 1.19 and above). |
|
|
| |
| Enable TLS termination for the flower Ingress. |
|
|
| |
| The name of a pre-created Secret containing a TLS private key and certificate. |
|
|
| |
| Annotations for the PgBouncer Ingress. |
| Why should pgbouncer have an ingress? | These are all deployed into the same cluster. So there is no reason to have. It is even a possible open door.
| Related comment under PgBouncer section. Ingress needed for setups where components are separated from each other, looking at different setups (basically, all enabled flags allow for that) |
| Enable PgBouncer ingress resource. |
|
| ||
| The hostname for the PgBouncer Ingress. (Deprecated - renamed to ingress.pgbouncer.hosts) |
|
| ||
| The hostnames or hosts configuration for the PgBouncer Ingress. |
|
| ||
| The Ingress Class for the PgBouncer Ingress. |
|
| ||
| The path for the PgBouncer Ingress. |
|
| ||
| The pathType for the PgBouncer Ingress (required for Kubernetes 1.19 and above). |
|
| ||
| Annotations for the statsd Ingress. |
| Why should statsd have an ingress? | This could make sense if the deployed logging tool is outside of the cluster, I would understand if they need to reach out to statsd
| Move to Kustomize. Ingress needs for setups where Observability tools gather metrics from the endpoint instead of having dedicated agents inside the Kubernetes cluster |
| Enable statsd ingress resource. |
|
| ||
| The hostname for the statsd Ingress. (Deprecated - renamed to ingress.statsd.hosts) |
|
| ||
| The hostnames or hosts configuration for the statsd Ingress. |
|
| ||
| The Ingress Class for the statsd Ingress. |
|
| ||
| The path for the statsd Ingress. |
|
| ||
| The pathType for the statsd Ingress (required for Kubernetes 1.19 and above). |
|
| ||
| Annotations for the web Ingress. |
| Will be removed with deprecation of Airflow 2 |
Yes, API server separatly defined. | Drop with Airflow 2 |
| Enable web ingress resource. |
|
| ||
| The hostname for the web Ingress. (Deprecated - renamed to ingress.web.hosts) |
|
| ||
| The hostnames or hosts configuration for the web Ingress. |
|
| ||
| The Ingress Class for the web Ingress. |
|
| ||
| The path for the web Ingress. |
|
| ||
| The pathType for the web Ingress (required for Kubernetes 1.19 and above). |
|
| ||
| HTTP paths to add to the web Ingress before the default path. |
|
| ||
| HTTP paths to add to the web Ingress after the default path. |
|
| ||
| Enable TLS termination for the web Ingress. |
|
| ||
| The name of a pre-created Secret containing a TLS private key and certificate. |
|
|
Kerberos
Parameter | Description | Default | Opinion Jens | Opinion Bugra | Opinion Przemek |
|---|---|---|---|---|---|
| Name for kerberos credentials cache file. |
|
|
We should move these to Kustomize If the above comments (under Workers section) also agreed? | Move to Kustomize |
| Path to mount shared volume for kerberos credentials cache. |
|
| ||
| Contents of krb5.conf. |
|
| ||
| Path to mount krb5.conf kerberos configuration file. |
|
| ||
| Enable kerberos. |
|
| ||
| Kerberos keytab base64 encoded content. |
|
| ||
| Path to mount the keytab for refreshing credentials in the kerberos sidecar. |
|
| ||
| Principal to use when refreshing kerberos credentials. |
| |||
| How often (in minutes) airflow kerberos will reinitialize the credentials cache. |
|
|