Important Links

Discussion Notes

https://docs.google.com/document/d/1bZsyrG5kjsYd2rJRiN3kR613lO6JPEBd4ItsySneOMw/edit?usp=sharing

Github Umbrella Issue

https://github.com/apache/airflow/issues/64037

Discussion

https://lists.apache.org/thread/9714k5wo2jwz1xwdnh9xnp98hloxms15

Bi-Weekly Interest Group Alignment/Dicuss Call

Helm Release Discussion 

Thursday 7 May • 18:00-19:00 CEST (Central European Summer Time)

Link to video call: https://meet.google.com/yjo-ueoq-nak

Complexity Analysis and Rating - Chart Parameters

Parameters from Docs

Based on Helm Chart 1.19.0 release

Common

Parameter

Description

Default

Opinion Jens

Opinion Bugra

Opinion Przemek

airflowLocalSettings

airflow_local_settings file as a string (templated). You can bake an airflow_local_settings.py into your image instead. In that case, set this value to null.

See values.yaml

keep

keep

Keep

airflowVersion

Airflow version (Used to make some decisions based on Airflow Version being deployed).

3.1.7

keep

keep

Keep

apiSecretAnnotations

Annotations to add to the Api secret.

{}

Why is this for flask in AF3?

Description remains from v2 terms. 

Move under apiServer section

apiSecretKey

The Flask secret key for Airflow Api to encrypt browser session.

~

Why is this for flask in AF3?

Description remains from v2 terms. 

Maybe dedicated section for flask-related configs 🤔


 

apiServer.apiServerConfig

This string (templated) will be mounted into the Airflow API Server as a custom webserver_config.py. You can bake a webserver_config.py in to your image instead or specify a configmap containing the webserver_config.py.

~

Examples:

apiServerConfig: |-
  from airflow import configuration as conf

  # The SQLAlchemy connection string.
  SQLALCHEMY_DATABASE_URI = conf.get('database', 'SQL_ALCHEMY_CONN')

  # Flask-WTF flag for CSRF
  CSRF_ENABLED = True

keep

Is example still relevant?

This is still relevant if the user is using providers-fab and adding more fine-grained authentication integration through fab. That also generalised to BaseAuthManager and to providers that implement 3rd-party libraries and tools. I think the description should be updated.

webserver_config.py migration check mentioned above.

apiServer.apiServerConfigConfigMapName

The configmap name containing the webserver_config.py.

~

Examples:

apiServerConfigConfigMapName: my-api-server-configmap

keep

keep

keep

config

Settings to go into the mounted airflow.cfg

See values.yaml

keep

keep

keep

createUserJob.defaultUser.enabled

Enable default user creation.

True

keep

keep

If I remember correctly it is only flask-related, so maybe move to flask dedicated section?

defaultAirflowDigest

Default airflow digest to deploy. Overrides tag.

~

keepkeepkeep

defaultAirflowRepository

Default airflow repository. Overrides all the specific images below.

apache/airflow

keep

keep

keep

defaultAirflowTag

Default airflow tag to deploy.

3.1.7

keep

keep

keep

executor

Airflow executor.

CeleryExecutor

 

 

keep

fernetKey

The Fernet key used to encrypt passwords (can only be set during install, not upgrade).

~

keep

keep

Move to the new encryption section

fernetKeySecretAnnotations

Annotations to add to the Fernet Key secret.

{}

keep

keep

fullnameOverride

Provide a name to substitute for the full names of resources

""

 

 

keep

jwtSecret

Secret key used to encode and decode JWTs to authenticate to public and private APIs (can only be set during install, not upgrade).

~

keep

keep

Move to the new encryption section

jwtSecretAnnotations

Annotations to add to the JWT secret.

{}

keep

keep

nameOverride

Override the name of the chart

""

 

 

Change to chartNameOverride

pgbouncer.enabled

Enable PgBouncer.

False

keep

keep

Whole section comment is under pgbouncer section 

pgbouncer.mountConfigSecret

Whether to mount the config secret files under /etc/pgbouncer/ by default.

True

keep

keep

schedulerName

Specify kube scheduler name for Pods.

~

Put this to Kustomize?

No disagreement with Jens.No strong opinion

useStandardNaming

Use standard naming for all resources using airflow.fullname template

False

 

 Deprecate the false value and remove it in future

webserver.webserverConfig

This string (templated) will be mounted into the Airflow webserver as a custom webserver_config.py. You can bake a webserver_config.py in to your image instead or specify a configmap containing the webserver_config.py.

~

Examples:

webserverConfig: |-
  from airflow import configuration as conf

  # The SQLAlchemy connection string.
  SQLALCHEMY_DATABASE_URI = conf.get('database', 'SQL_ALCHEMY_CONN')

  # Flask-WTF flag for CSRF
  CSRF_ENABLED = True

removed with dropping AF2 support




Drop with Airflow 2

 

 

webserver.webserverConfigConfigMapName

The configmap name containing the webserver_config.py.

~

Examples:

webserverConfigConfigMapName: my-webserver-configmap

 

webserverSecretAnnotations

Annotations to add to the webserver secret.

{}

 

webserverSecretKey

The Flask secret key for Airflow Webserver to encrypt browser session.

~

 

workers.schedulerName

Specify kube scheduler name for Airflow Celery workers objects and pods created with pod-template-file.

~

Put this to Kustomize?

No disagreement with Jens.

No strong opinion


Airflow

Parameter

Description

Default

Opinion Jens

Opinion Bugra

Opinion Przemek

airflowHome

Airflow home directory. Used for mount paths.

/opt/airflow

keep

keep

keep

allowJobLaunching

Whether various Airflow components launch jobs.

False

Can this not implicitly be discovered if K8s executor?

It can be automated indeed. We can map to both a feature and a deprecation along with it.

I would leave it as it is. Looking at the current available Kubernetes connections, there is an in_cluster option which basically says "use Kubernetes Service Account which is set for currently running pod e.g. Celery worker". In some cases, despite used Executor, the proper permissions can be or can be not be needed, and if something is not needed, there should be a possibility of removing it (an automatic option would remove it, raising some security concerns).

allowPodLaunching

Whether various Airflow components launch pods.

True

Can this not implicitly be discovered if K8s executor?

It can be automated indeed. We can map to both a feature and a deprecation along with it.

apiSecretKeySecretName

The Secret name containing Flask secret_key for the Api.

~

Why is this for flask in AF3?

Description remains from v2 terms. 

Maybe move to flask dedicated section 🤔?

dags.gitSync.branch

Git branch

v2-2-stable

Should we transform the gitSync into a bundle definition?

It makes sense. There are multiple ways in various systems of mounting the dags into the components. Making it bundled can help with easier additions on top of it for mounting operations.

Drop in favor of bundle and maybe seperate it to not-dag related section for sync with potential external dags dependencies


 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

dags.gitSync.containerName

Git sync container name.

git-sync

 

 

dags.gitSync.credentialsSecret

Name of a Secret containing the repo GIT_SYNC_USERNAME and GIT_SYNC_PASSWORD.

~

 

 

dags.gitSync.depth

Repository depth.

1

 

 

dags.gitSync.emptyDirConfig

Configuration for dags empty dir volume.

~

 

 

dags.gitSync.enabled

Enable Git sync.

False

 

 

dags.gitSync.env

Environment variables for git sync container.

[]

Examples:

env:
- name: GIT_SYNC_TIMEOUT
  value: '60'
env:
- name: GIT_SYNC_USERNAME
  valueFrom:
    secretKeyRef:
      key: username
      name: git-secret

 

 

dags.gitSync.envFrom

Extra envFrom ‘items’ that will be added to the definition of Airflow gitSync containers; a string or array are expected (templated).

~

Examples:

envFrom: |-
  - secretRef:
      name: 'proxy-config
envFrom: |-
  - configMapRef:
      name: 'proxy-config

 

 

dags.gitSync.extraVolumeMounts

Mount additional volumes into git sync container.

[]

 

 

dags.gitSync.knownHosts

When using a ssh private key, the contents of your known_hosts file.

~

Examples:

knownHosts: |-
  <host1>,<ip1> <key1>
  <host2>,<ip2> <key2>
knownHosts: <host1>,<ip1> <key1>

 

 

dags.gitSync.maxFailures

The number of consecutive failures allowed before aborting.

0

 

 

dags.gitSync.period

Interval between git sync attempts in Go-style duration string. High values are more likely to cause DAGs to become out of sync between different components. Low values cause more traffic to the remote git repository.

5s

 

 

dags.gitSync.ref

Git revision branch, tag, or hash.

v2-2-stable

 

 

dags.gitSync.repo

Git repository.

https://github.com/apache/airflow.git

 

 

dags.gitSync.resources

Resources on workers git-sync sidecar

{}

Examples:

resources:
  limits:
    cpu: 100m
    memory: 128Mi
  requests:
    cpu: 100m
    memory: 128Mi

 

 

dags.gitSync.rev

Git revision.

HEAD

 

 

dags.gitSync.securityContext

Security context for the gitSync container (deprecated, use securityContexts instead). If not set, the values from securityContext will be used.

{}

Examples:

securityContext:
  runAsGroup: 0
  runAsUser: 50000

 

 

dags.gitSync.sshKey

SSH private key

~

 

 

dags.gitSync.sshKeySecret

Name of a Secret containing the repo sshKeySecret.

~

 

 

dags.gitSync.subPath

Subpath within the repo where dags are located.

tests/dags

 

 

dags.gitSync.uid

Git sync container run as user parameter.

65533

 

 

dags.gitSync.wait

Interval between git sync attempts in seconds. High values are more likely to cause DAGs to become out of sync between different components. Low values cause more traffic to the remote git repository.

~

 

 

dags.mountPath

Where dags volume will be mounted. Works for both persistence and gitSync. If not specified, dags mount path will be set to $AIRFLOW_HOME/dags

~

 

 

keep

dags.persistence.accessMode

Access mode of the persistent volume.

ReadWriteOnce

 

 

keep

dags.persistence.annotations

Annotations for the dag PVC

{}

 

 

keep

dags.persistence.enabled

Enable persistent volume for storing dags.

False

 

 

keep

dags.persistence.existingClaim

The name of an existing PVC to use.

~

 

 

keep

dags.persistence.size

Volume size for dags.

1Gi

 

 

keep

dags.persistence.storageClassName

If using a custom StorageClass, pass name here (templated).

~

 

 

keep

dags.persistence.subPath

Subpath within the PVC where dags are located.

~

 

 

keep

elasticsearch.connection

Elasticsearch connection configuration.

{}

Examples:

connection:
  host: '...'
  pass: '...'
  port: '...'
  scheme: https
  user: '...'

 

 

Drop/move to Kustomize

 

 

 

 

 

 

 

 

elasticsearch.connection.host

Host

""

 

 

elasticsearch.connection.pass

Password

""

 

 

elasticsearch.connection.port

Port

80

 

 

elasticsearch.connection.scheme

Scheme

http

 

 

elasticsearch.connection.user

Username

""

 

 

elasticsearch.enabled

Enable Elasticsearch task logging.

False

 

 

elasticsearch.secretAnnotations

Extra annotations to apply to the elasticsearch secret.

{}

 

 

elasticsearch.secretName

A secret containing the connection string.

~

 

 

enableBuiltInSecretEnvVars.AIRFLOW_CONN_AIRFLOW_DB

Enable AIRFLOW_CONN_AIRFLOW_DB variable to be read from the Metadata Secret

True

What is this for?

SQLAlchemy connection string from K8s secret. Keeping with a better description and maybe renaming to reflect better.

I guess it should be the same as AIRFLOW__DATABASE__SQL_ALCHEMY_CONN so probably drop

enableBuiltInSecretEnvVars.AIRFLOW__API_AUTH__JWT_SECRET

Enable AIRFLOW__API_AUTH__JWT_SECRET variable to be read from the JWT Secret

True

What is this for?

Enable using K8S secret in env var to pass jwt_secret config

keep

enableBuiltInSecretEnvVars.AIRFLOW__API__SECRET_KEY

Enable AIRFLOW__API__SECRET_KEY variable to be read from the Api Secret Key Secret

True

What is this for?

Similar to above

secret_key

keep

enableBuiltInSecretEnvVars.AIRFLOW__CELERY__BROKER_URL

Enable AIRFLOW__CELERY__BROKER_URL variable to be read from the Celery Broker URL Secret

True

What is this for?

 

 

 

 

 

 

 

 

All these are the same as above, as setting some config that enables to read secret to pass to the env var. Maybe move them to Kustomize?

keep/move under workers.celery section

enableBuiltInSecretEnvVars.AIRFLOW__CELERY__CELERY_RESULT_BACKEND

Enable AIRFLOW__CELERY__CELERY_RESULT_BACKEND variable to be read from the Celery Result Backend Secret - Airflow 1.10.* variant

True

What is this for?

Drop

enableBuiltInSecretEnvVars.AIRFLOW__CELERY__RESULT_BACKEND

Enable AIRFLOW__CELERY__RESULT_BACKEND variable to be read from the Celery Result Backend Secret

True

What is this for?

keep/move under workers.celery section

enableBuiltInSecretEnvVars.AIRFLOW__CORE__FERNET_KEY

Enable AIRFLOW__CORE__FERNET_KEY variable to be read from the Fernet key Secret

True

What is this for?

keep

enableBuiltInSecretEnvVars.AIRFLOW__CORE__SQL_ALCHEMY_CONN

Enable AIRFLOW__CORE__SQL_ALCHEMY_CONN variable to be read from the Metadata Secret

True

What is this for?

Remove with Airflow 2 drop

enableBuiltInSecretEnvVars.AIRFLOW__DATABASE__SQL_ALCHEMY_CONN

Enable AIRFLOW__DATABASE__SQL_ALCHEMY_CONN variable to be read from the Metadata Secret

True

What is this for?

keep

enableBuiltInSecretEnvVars.AIRFLOW__ELASTICSEARCH__ELASTICSEARCH_HOST

Enable AIRFLOW__ELASTICSEARCH__ELASTICSEARCH_HOST variable to be read from the Elasticsearch Host Secret - Airflow <1.10.4 variant

True

What is this for?

Drop/move to Kustomize with elasticsearch section

enableBuiltInSecretEnvVars.AIRFLOW__ELASTICSEARCH__HOST

Enable AIRFLOW__ELASTICSEARCH__HOST variable to be read from the Elasticsearch Host Secret

True

What is this for?

enableBuiltInSecretEnvVars.AIRFLOW__OPENSEARCH__HOST

Enable AIRFLOW__OPENSEARCH__HOST variable to be read from the OpenSearch Host Secret

True

What is this for?

Drop/move to Kustomize with opensearch section

enableBuiltInSecretEnvVars.AIRFLOW__WEBSERVER__SECRET_KEY

Enable AIRFLOW__WEBSERVER__SECRET_KEY variable to be read from the Webserver Secret Key Secret

True

removed with dropping AF2 support

 

Drop with Airflow 2

env

Environment variables for all Airflow containers.

[]

Examples:

env:
- name: MYENVVAR
  value: something_fun

 

 

keep

extraEnv

Extra env ‘items’ that will be added to the definition of Airflow containers; a string is expected (templated).

~

Examples:

extraEnv: |-
  - name: AIRFLOW__CORE__LOAD_EXAMPLES
     value: True

 

 

keep

extraEnvFrom

Extra envFrom ‘items’ that will be added to the definition of Airflow containers; a string is expected (templated).

~

Examples:

extraEnvFrom: |-
  - secretRef:
      name: '{{ .Release.Name }}-airflow-connections'
extraEnvFrom: |-
  - configMapRef:
      name: '{{ .Release.Name }}-airflow-variables'

 

 

keep

fernetKeySecretName

The Fernet key secret name.

~

 

 

Move under the new encryption section

gid

Group of airflow user.

0

 

 

Probably remove and/or move to Kustomize. It is only used in volume-permissions sidecar in workers, which, looking at the comment, is only needed with the local path provisioner https://github.com/rancher/local-path-provisioner setup.

jwtSecretName

The JWT secret name.

~

 

 

Move under the new encryption section

logs.emptyDirConfig

Configuration for logs empty dir volume.

~

 

 

keep

logs.persistence.annotations

Annotations to add to logs PVC

{}

 

 

keep

logs.persistence.enabled

Enable persistent volume for storing logs.

False

 

 

keep

logs.persistence.existingClaim

The name of an existing PVC to use.

~

 

 

keep

logs.persistence.size

Volume size for logs.

100Gi

 

 

keep

logs.persistence.storageClassName

If using a custom StorageClass, pass name here (templated).

~

 

 

keep

logs.persistence.subPath

The subpath of the existing PVC to use.

~

 

 

keep

multiNamespaceMode

Whether Airflow can launch workers and/or pods in multiple namespaces. If true, it creates ClusterRole/ClusterRolebinding (with access to entire cluster)

False

 

 

Move under workers.kubernetes section as it is only related to KubernetesExecutor

opensearch.connection

OpenSearch connection configuration.

{}

Examples:

connection:
  host: '...'
  pass: '...'
  port: '...'
  scheme: https
  user: '...'

 

 

Drop/move to Kustomize

 

 

 

 

 

 

 

opensearch.connection.host

Host

""

 

 

opensearch.connection.pass

Password

""

 

 

opensearch.connection.port

Port

80

 

 

opensearch.connection.scheme

Scheme

http

 

 

opensearch.connection.user

Username

""

 

 

opensearch.enabled

Enable OpenSearch task logging.

False

 

 

opensearch.secretName

A secret containing the connection string.

~

 

 

podTemplate

The content of pod_template_file.yaml used for KubernetesExecutor workers (templated). The default (see files/pod-template-file.kubernetes-helm-yaml) already takes into account normal workers configuration parameters (e.g. workers.resources), so you normally won’t need to override this directly.

~

Examples:

podTemplate: |-
  apiVersion: v1
  kind: Pod
  metadata:
    name: placeholder-name
    labels:
      tier: airflow
      component: worker
      release: {{ .Release.Name }}
  spec:
    priorityClassName: high-priority
    containers:
      - name: base
      ...

 

 

Move to workers.kubernetes as it is only related to KubernetesExecutor

secret

Secrets for all Airflow containers.

[]

Examples:

secret:
- envName: SecretEnvVar
  secretKey: somekey
  secretName: somesecret

 

 

keep

uid

User of airflow user.

50000

 

 

Probably remove and/or move to Kustomize. It is only used in volume-permissions sidecar in workers, which, looking at the comment, is only needed with the local path provisioner https://github.com/rancher/local-path-provisioner setup.

volumeMounts

VolumeMounts for all Airflow containers.

[]

 

 

keep

volumes

Volumes for all Airflow containers.

[]

 

 

keep

webserverSecretKeySecretName

The Secret name containing Flask secret_key for the Webserver.

~

removed with dropping AF2 support

 

Drop with Airflow 2

Images

Parameter

Description

Default

Opinion Jens

Opinion Bugra

Opinion Przemek

images.airflow.digest

The airflow image digest. If set, it will override the tag.

~

 

 

Probably more in future: move to per-section config with ongoing decupling of airflow and future multi-team feature

images.airflow.pullPolicy

The airflow image pull policy.

IfNotPresent

 

 

images.airflow.repository

The airflow image repository.

~

 

 

images.airflow.tag

The airflow image tag.

~

 

 

images.flower.pullPolicy

The flower image pull policy.

IfNotPresent

 

 

Move to Kustomize with the flower section

images.flower.repository

The flower image repository.

~

 

 

images.flower.tag

The flower image tag.

~

 

 

images.gitSync.pullPolicy

The gitSync image pull policy.

IfNotPresent

 

 

 

Maybe we can bundle these with the gitSync definitions above. I see from Jed's comment in Slack that if we completely hammer it, bundling can make it easier, but on the other hand, not sure if it would be worth the effort.

Drop with gitSync drop or move as a method to sync external Dag dependencies

images.gitSync.repository

The gitSync image repository.

registry.k8s.io/git-sync/git-sync

 

images.gitSync.tag

The gitSync image tag.

v4.4.2

Note: Drop support for old versions!

images.migrationsWaitTimeout

The time (in seconds) to wait for the DB migrations to complete.

60

 

 

Move under the migrateDatabaseJob section

images.pgbouncer.pullPolicy

The PgBouncer image pull policy.

IfNotPresent

 

 

Related comment under the PgBouncer section

images.pgbouncer.repository

The PgBouncer image repository.

apache/airflow

 

 

images.pgbouncer.tag

The PgBouncer image tag.

airflow-pgbouncer-2025.03.05-1.23.1

 

 

images.pgbouncerExporter.pullPolicy

The PgBouncer exporter image pull policy.

IfNotPresent

 

 

images.pgbouncerExporter.repository

The PgBouncer exporter image repository.

apache/airflow

 

 

images.pgbouncerExporter.tag

The PgBouncer exporter image tag.

airflow-pgbouncer-exporter-2025.03.05-0.18.0

 

 

images.pod_template.pullPolicy

The pod_template image pull policy.

IfNotPresent

 

 

Probably more in future: move to per-section config with ongoing decupling of airflow and future multi-team feature

images.pod_template.repository

The pod_template image repository. If config.kubernetes.worker_container_repository is set, k8s executor will use config value instead.

~

 

 

images.pod_template.tag

The pod_template image tag. If config.kubernetes.worker_container_tag is set, k8s executor will use config value instead.

~

 

 

images.redis.pullPolicy

The redis image pull policy.

IfNotPresent

 

 

Opinion under respective sections

images.redis.repository

The redis image repository.

redis

 

 

images.redis.tag

The redis image tag.

7.2-bookworm

 

 

images.statsd.pullPolicy

The StatsD image pull policy.

IfNotPresent

 

 

images.statsd.repository

The StatsD image repository.

quay.io/prometheus/statsd-exporter

 

 

images.statsd.tag

The StatsD image tag.

v0.28.0

 

 

images.useDefaultImageForMigration

To avoid images with user code for running and waiting for DB migrations set this to true.

False

 

 

Probably more in future: move to per-section config with ongoing decupling of airflow and future multi-team feature

Ports

Parameter

Description

Default

Opinion Jens

Opinion Przemek

ports.airflowUI

Airflow UI port.

8080

keep all

Keep

ports.apiServer

API server port.

8080

 

Keep

ports.flowerUI

Flower UI port.

5555

 

Move with flower to Kustomize

ports.pgbouncer

PgBouncer port.

6543

 

Related comment under the PgBouncer section

ports.pgbouncerScrape

PgBouncer scrape port.

9127

 

ports.redisDB

Redis port.

6379

 

Keep

ports.statsdIngest

StatsD ingest port.

9125

 

Maybe move with statsd to Kustomize

ports.statsdScrape

StatsD scrape port.

9102

 

Maybe move with statsd to Kustomize

ports.triggererLogs

Triggerer logs port.

8794

 

Keep

ports.workerLogs

Worker logs port.

8793

 

Keep

Database

Parameter

Description

Default

Opinion Jens

Opinion Bugra

Opinion Przemek

data.brokerUrlSecretAnnotations

Annotations to add to the broker url secret.

{}

 

 

Move to workers.celery/redis as it is only for CeleryExecutor

data.metadataConnection.db

The name of the database.

postgres

 

 

Move to the new metastore/database section

 

 

 

 

 

 

 

 

data.metadataConnection.host

The database host.

~

 

 

data.metadataConnection.pass

The user’s password.

postgres

 

 

data.metadataConnection.port

The database port.

5432

 

 

data.metadataConnection.protocol

The database protocol.

postgresql

 

 

data.metadataConnection.secretAnnotations

Annotations to add to the metadata connection secret.

{}

 

 

data.metadataConnection.sslmode

The database SSL parameter.

disable

 

 

data.metadataConnection.user

The database user.

postgres

 

 

data.metadataSecretName

Metadata connection string secret.

~

 

 

data.resultBackendConnection

Result backend connection configuration.

~

 

 

Move to workers.celery as it is only for CeleryExecutor


 

 

 

 

 

 

 

 

 

data.resultBackendConnection.db

The name of the database.

~

 

 

data.resultBackendConnection.host

The database host.

~

 

 

data.resultBackendConnection.pass

The database password.

~

 

 

data.resultBackendConnection.port

The database port.

~

 

 

data.resultBackendConnection.protocol

The database protocol.

~

 

 

data.resultBackendConnection.sslmode

The database SSL parameter.

~

 

 

data.resultBackendConnection.user

The database user.

~

 

 

data.resultBackendConnectionSecretAnnotations

Annotations to add to the result backend connection secret.

{}

 

 

data.resultBackendSecretName

Result backend connection string secret.

~

 

 

postgresql.auth.enablePostgresUser

Assign a password to the ‘postgres’ admin user. Otherwise, remote access will be blocked for this user

True

Drop postgres inline support

 

 

 

 

 

 

This is not even needs a discussion :D 

Drop

postgresql.auth.password

Password for the custom user to create.

""

 

postgresql.auth.postgresPassword

Password for the ‘postgres’ admin user.

postgres

 

postgresql.auth.username

Name for a custom user to create

""

 

postgresql.enabled

Enable PostgreSQL subchart.

True

 

postgresql.image.repository

The PostgreSQL image repository.

bitnamilegacy/postgresql

 

postgresql.image.tag

The PostgreSQL image tag.

16.1.0-debian-11-r15

 

PgBouncer

Parameter

Description

Default

Opinion Jens

Opinion Bugra

Opinion Przemek

pgbouncer.affinity

Specify scheduling constraints for PgBouncer pods.

{}

 

I have a comment on this as a whole. While we are planning to hammer PostgreSQL. Why we should also consider dropping support for PgBouncer as well thinking face

It is just an idea thrown into the wild. Since it is specific to the PostgreSQL environment, maybe our focus can be on making core charts better rather than spending time providing support to another tool similar to PostgreSQL.

I am not discussing how useful it is for availability and managing/distributing database connections.

My ideation is more on I see no difference with having ProxySQL, which is a similar tool (open source under GNU, GPL-3).


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 

In addition to Bugra's comment - as I agree on that pgbouncer is just one possible tool which can be used, I think it goes potentially a little deeper in the sense of what we would want this chart to be. Do we want it to be only a baseline (base config and support for Apache Airflow features, nothing more), or do we want to make it more production-grade with a recommended setup (e.g. use pgbouncer) with integration tests, e.g. for upgrade/downgrades, etc. (maybe not like use this or that, but we are testing that kind of setup and it seems to work like with constraints file)?


On the whole section itself, I would remove it from the core chart as it is not an Airflow-related component (I had an idea that maybe the core chart should only consist of Airflow-related things, and the rest should be customisation).


 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

pgbouncer.annotations

Annotations to add to the PgBouncer deployment

{}

 

pgbouncer.args

Args to use for PgBouncer (templated).

~

 

pgbouncer.auth_file

The name of the file to load user names and passwords from

/etc/pgbouncer/users.txt

 

pgbouncer.auth_type

Method of authenticating users

scram-sha-256

 

pgbouncer.certificatesSecretAnnotations

Annotations to add to the PgBouncer certificates secret.

{}

 

pgbouncer.ciphers

The allowed ciphers, might be ‘fast’, ‘normal’ or list ciphers separated with ‘:’.

normal

 

pgbouncer.command

Command to use for PgBouncer (templated).

['pgbouncer', '-u', 'nobody', '/etc/pgbouncer/pgbouncer.ini']

 

pgbouncer.configSecretAnnotations

Annotations to add to the PgBouncer config secret.

{}

 

pgbouncer.configSecretName

The PgBouncer config Secret name.

~

 

pgbouncer.env

Add additional env vars to pgbouncer container.

[]

 

pgbouncer.extraContainers

Launch additional containers into pgbouncer.

[]

 

pgbouncer.extraIni

Add extra general PgBouncer ini configuration: https://www.pgbouncer.org/config.html

~

 

pgbouncer.extraIniMetadata

Add extra metadata database specific PgBouncer ini configuration: https://www.pgbouncer.org/config.html#section-databases

~

 

pgbouncer.extraIniResultBackend

Add extra result backend database specific PgBouncer ini configuration: https://www.pgbouncer.org/config.html#section-databases

~

 

pgbouncer.extraNetworkPolicies

Additional NetworkPolicies as needed.

[]

 

pgbouncer.extraVolumeMounts

Mount additional volumes into PgBouncer.

[]

 

pgbouncer.extraVolumes

Mount additional volumes into PgBouncer.

[]

 

pgbouncer.labels

Labels to add to the PgBouncer objects and pods.

{}

 

pgbouncer.logConnections

Log successful logins.

0

 

pgbouncer.logDisconnections

Log disconnections with reasons.

0

 

pgbouncer.maxClientConn

Maximum clients that can connect to PgBouncer (higher = more file descriptors).

100

 

pgbouncer.metadataPoolSize

Metadata pool size.

10

 

pgbouncer.metricsExporterSidecar.extraVolumeMounts

Mount additional volumes into PgBouncer Metrics Exporter.

[]

 

pgbouncer.metricsExporterSidecar.livenessProbe.initialDelaySeconds

Metrics Exporter liveness probe initial delay

10

 

pgbouncer.metricsExporterSidecar.livenessProbe.periodSeconds

Metrics Exporter liveness probe frequency

10

 

pgbouncer.metricsExporterSidecar.livenessProbe.timeoutSeconds

Metrics Exporter liveness probe command timeout

1

 

pgbouncer.metricsExporterSidecar.readinessProbe.initialDelaySeconds

Metrics Exporter readiness probe initial delay

10

 

pgbouncer.metricsExporterSidecar.readinessProbe.periodSeconds

Metrics Exporter readiness probe frequency

10

 

pgbouncer.metricsExporterSidecar.readinessProbe.timeoutSeconds

Metrics Exporter readiness probe command timeout

1

 

pgbouncer.metricsExporterSidecar.resources

Resources for the PgBouncer metric exporter.

{}

Examples:

resources:
  limits:
    cpu: 100m
    memory: 128Mi
  requests:
    cpu: 100m
    memory: 128Mi

 

pgbouncer.metricsExporterSidecar.sslmode

SSL mode for metricsExporterSidecar

disable

 

pgbouncer.metricsExporterSidecar.statsSecretAnnotations

Annotations to add to the PgBouncer stats secret.

{}

 

pgbouncer.metricsExporterSidecar.statsSecretKey

Key referencing the PGBouncer Metrics connection URI within an existing Secrets object. Defaults to connection if left null.

~

 

pgbouncer.metricsExporterSidecar.statsSecretName

Name of an existing Secrets object containing PgBouncer Metrics secrets.

~

 

pgbouncer.nodeSelector

Select certain nodes for PgBouncer pods.

{}

 

pgbouncer.podAnnotations

Add annotations for the PgBouncer Pod.

{}

 

pgbouncer.podDisruptionBudget.config.maxUnavailable

Max unavailable pods for PgBouncer.

1

 

pgbouncer.podDisruptionBudget.config.minAvailable

Min available pods for PgBouncer.

1

 

pgbouncer.podDisruptionBudget.enabled

Enabled PodDistributionBudget.

False

 

pgbouncer.priorityClassName

Specify priority for PgBouncer pods.

~

 

pgbouncer.replicas

Number of PgBouncer replicas to run in Deployment.

1

 

pgbouncer.resources

Resources for the PgBouncer pods.

{}

Examples:

resources:
  limits:
    cpu: 100m
    memory: 128Mi
  requests:
    cpu: 100m
    memory: 128Mi

 

pgbouncer.resultBackendPoolSize

Result backend pool size.

5

 

pgbouncer.service.clusterIp

Specific ClusterIP for the PgBouncer Service.

~

 

pgbouncer.service.extraAnnotations

Extra annotations for the PgBouncer Service.

{}

 

pgbouncer.serviceAccount.annotations

Annotations to add to the worker Kubernetes ServiceAccount.

{}

 

pgbouncer.serviceAccount.automountServiceAccountToken

Specifies if ServiceAccount’s API credentials should be mounted onto Pods.

True

 

pgbouncer.serviceAccount.create

Specifies whether a ServiceAccount should be created.

True

 

pgbouncer.serviceAccount.name

The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name.

~

 

pgbouncer.ssl.ca

Certificate Authority for server side

~

 

pgbouncer.ssl.cert

Server Certificate for server side

~

 

pgbouncer.ssl.key

Private key used to authenticate with the server

~

 

pgbouncer.sslmode

SSL mode for PgBouncer.

prefer

 

pgbouncer.tolerations

Specify Tolerations for PgBouncer pods.

[]

 

pgbouncer.topologySpreadConstraints

Specify topology spread constraints for PgBouncer pods.

[]

 

pgbouncer.uid

PgBouncer run as user parameter.

65534

 

pgbouncer.verbose

Increase PgBouncer verbosity.

0

 

API Server

Parameter

Description

Default

Opinion Jens

Opinion Bugra

Opinion Przemek

apiServer.affinity

Specify scheduling constraints for API server pods.

See values.yaml

 

 

keep

apiServer.allowPodLogReading

Allow API server to read k8s pod logs. Useful when you don’t have an external log store.

True

 

 

Maybe it could be automatically set based on the executor value 🤔. Maybe, as I'm not sure about whether it is needed or not, with e.g. remote logging set

apiServer.annotations

Annotations to add to the API server deployment

{}

 

 

keep 

apiServer.args

Args to use when running the Airflow API server (templated). When running behind a reverse proxy, add –proxy-headers to enable Uvicorn to respect X-Forwarded-Proto, X-Forwarded-For, and X-Forwarded-Port headers.

['bash', '-c', 'exec airflow api-server']

Examples:

args:
- bash
- -c
- exec airflow api-server --proxy-headers

 

 

keep

apiServer.command

Command to use when running the Airflow API server (templated).

~

 

 

keep

apiServer.configMapAnnotations

Extra annotations to apply to the API server configmap.

{}

 

 

keep

apiServer.enabled

Enable Airflow API server deployment.

True

 

 

keep

apiServer.env

Add additional env vars to API server. When running behind a reverse proxy, set FORWARDED_ALLOW_IPS to specify which IPs are trusted to send X-Forwarded-* headers. Use “*” for trusted environments, or specify proxy IP ranges for production.

[]

Examples:

env:
- - name: FORWARDED_ALLOW_IPS
    value: '*'

 

 

keep

apiServer.extraContainers

Launch additional containers into API server.

[]

 

 

keep

apiServer.extraInitContainers

Add additional init containers into API server.

[]

 

 

keep

apiServer.extraVolumeMounts

Mount additional volumes into API server.

[]

 

 

keep

apiServer.extraVolumes

Mount additional volumes into API server.

[]

 

 

keep

apiServer.hostAliases

HostAliases for the API server pod.

[]

Examples:

hostAliases:
- hostnames:
  - foo.local
  ip: 127.0.0.1
hostAliases:
- hostnames:
  - foo.remote
  ip: 10.1.2.3

 

 

keep

apiServer.hpa.behavior

HorizontalPodAutoscalerBehavior configures the scaling behavior of the target.

{}

Should we keep both KEDA and HPA? Or move both to a Kustomize?


 


 


 


 

My vote would be moving Keda to Kustomize, as it is really custom if we think of core components


 


 


 


 

Keep HPA, document Keda in Kustomize


 


 


 


 

apiServer.hpa.enabled

Enable HPA autoscaling for API server

False

apiServer.hpa.maxReplicaCount

Maximum number of API server replicas created by HPA if HPA is enabled.

5

apiServer.hpa.metrics

Specifications for which to use to calculate the desired replica count.

[{'type': 'Resource', 'resource': {'name': 'cpu', 'target': {'type': 'Utilization', 'averageUtilization': 50}}}]

apiServer.hpa.minReplicaCount

Minimum number of API server replicas created by HPA if HPA is enabled.

1

apiServer.labels

Labels to add to the API server objects and pods.

{}

 

 

keep

apiServer.livenessProbe.failureThreshold

API server Liveness probe failure threshold.

5

 

 

keep 

apiServer.livenessProbe.initialDelaySeconds

API server Liveness probe initial delay.

15

 

 

keep

apiServer.livenessProbe.periodSeconds

API server Liveness probe period seconds.

10

 

 

keep

apiServer.livenessProbe.scheme

API server Liveness probe scheme.

HTTP

 

 

keep

apiServer.livenessProbe.timeoutSeconds

API server Liveness probe timeout seconds.

5

 

 

keep

apiServer.networkPolicy.ingress.from

Peers for API server NetworkPolicyingress.

[]

 

 

keep

apiServer.networkPolicy.ingress.ports

Ports for API server NetworkPolicyingress (if from is set).

[{'port': '{{ .Values.ports.apiServer }}'}]

Examples:

ports:
- port: 8080

 

 

keep

apiServer.nodeSelector

Select certain nodes for API server pods.

{}

 

 

keep

apiServer.podAnnotations

Annotations to add to the API server pods.

{}

 

 

keep

apiServer.podDisruptionBudget.config.maxUnavailable

Max unavailable pods for API server.

1

 

 

Remove maxUnavailable and minAvailable and set config: {}


apiServer.podDisruptionBudget.config.minAvailable

Min available pods for API server.

1

 

 

apiServer.podDisruptionBudget.enabled

Enable pod disruption budget.

False

 

 

keep

apiServer.priorityClassName

Specify priority for API server pods.

~

 

 

keep

apiServer.readinessProbe.failureThreshold

API server Readiness probe failure threshold.

5

 

 

keep

apiServer.readinessProbe.initialDelaySeconds

API server Readiness probe initial delay.

15

 

 

keep

apiServer.readinessProbe.periodSeconds

API server Readiness probe period seconds.

10

 

 

keep

apiServer.readinessProbe.scheme

API server Readiness probe scheme.

HTTP

 

 

keep

apiServer.readinessProbe.timeoutSeconds

API server Readiness probe timeout seconds.

5

 

 

keep

apiServer.replicas

How many Airflow API server replicas should run. This setting is ignored when HPA (Horizontal Pod Autoscaler) is enabled

1

 

 

keep

apiServer.resources

Resources for API server pods.

{}

Examples:

resources:
  limits:
    cpu: 100m
    memory: 128Mi
  requests:
    cpu: 100m
    memory: 128Mi

 

 

keep

apiServer.service.annotations

Annotations for the API server Service.

{}

 

 

keep

apiServer.service.loadBalancerIP

API server Service loadBalancerIP.

~

 

 

keep

apiServer.service.loadBalancerSourceRanges

API server Service loadBalancerSourceRanges.

[]

Examples:

loadBalancerSourceRanges:
- 10.123.0.0/16

 

 

keep

apiServer.service.ports

Ports for the API server Service.

[{'name': 'api-server', 'port': '{{ .Values.ports.apiServer }}'}]

Examples:

ports:
- name: api-server
  port: 8080
  targetPort: api-server
ports:
- name: only_sidecar
  port: 9080
  targetPort: 8888

 

 

keep

apiServer.service.type

API server Service type.

ClusterIP

 

 

keep

apiServer.serviceAccount.annotations

Annotations to add to the API server Kubernetes ServiceAccount.

{}

 

 

keep

apiServer.serviceAccount.automountServiceAccountToken

Specifies if ServiceAccount’s API credentials should be mounted onto Pods.

True

 

 

keep

apiServer.serviceAccount.create

Specifies whether a ServiceAccount should be created.

True

 

 

keep

apiServer.serviceAccount.name

The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name.

~

 

 

keep

apiServer.startupProbe.failureThreshold

API server Startup probe failure threshold.

6

 

 

keep

apiServer.startupProbe.initialDelaySeconds

API server Startup probe initial delay seconds.

0

 

 

keep

apiServer.startupProbe.periodSeconds

API server Startup probe period seconds.

10

 

 

keep

apiServer.startupProbe.scheme

API server Startup probe scheme.

HTTP

 

 

keep

apiServer.startupProbe.timeoutSeconds

API server Startup probe timeout seconds.

20

 

 

keep

apiServer.strategy

Specifies the strategy used to replace old Pods by new ones.

~

 

 

keep

apiServer.tolerations

Specify Tolerations for API server pods.

[]

 

 

keep

apiServer.waitForMigrations.enabled

Enable wait-for-airflow-migrations init container.

True

 

 

Not sure if this should be a container-level flag (I don't have a specific use case in mind when it would make sense to have it only in one component and not in another)


apiServer.waitForMigrations.env

Add additional env vars to wait-for-airflow-migrations init container.

[]

 

 

Scheduler

Parameter

Description

Default

Opinion Jens

Opinion Bugra

Opinion Przemek

scheduler.affinity

Specify scheduling constraints for scheduler pods.

See values.yaml

 

 

keep

scheduler.annotations

Annotations to add to the scheduler deployment

{}

 

 

keep

scheduler.args

Args to use when running the Airflow scheduler (templated).

['bash', '-c', 'exec airflow scheduler']

 

 

keep

scheduler.command

Command to use when running the Airflow scheduler (templated).

~

 

 

keep

scheduler.enabled

Enable scheduler

True

 

 

keep

scheduler.env

Add additional env vars to scheduler.

[]

 

 

keep

scheduler.extraContainers

Launch additional containers into scheduler (templated).

[]

 

 

keep

scheduler.extraInitContainers

Add additional init containers into scheduler (templated).

[]

 

 

keep

scheduler.extraVolumeMounts

Mount additional volumes into scheduler.

[]

 

 

keep

scheduler.extraVolumes

Mount additional volumes into scheduler.

[]

 

 

keep

scheduler.hostAliases

HostAliases for the scheduler pod.

[]

Examples:

hostAliases:
- hostnames:
  - foo.local
  ip: 127.0.0.1
hostAliases:
- hostnames:
  - foo.remote
  ip: 10.1.2.3

 

 

keep

scheduler.labels

Labels to add to the scheduler objects and pods.

{}

 

 

keep

scheduler.livenessProbe.command

Command for LivenessProbe

~

 

 

keep

scheduler.livenessProbe.failureThreshold

Minimum consecutive failures for the probe to be considered failed after having succeeded. Minimum value is 1.

5

 

 

keep

scheduler.livenessProbe.initialDelaySeconds

Number of seconds after the container has started before liveness probes are initiated.

10

 

 

keep

scheduler.livenessProbe.periodSeconds

How often (in seconds) to perform the probe. Minimum value is 1.

60

 

 

keep

scheduler.livenessProbe.timeoutSeconds

Number of seconds after which the probe times out. Minimum value is 1 seconds.

20

 

 

keep

scheduler.nodeSelector

Select certain nodes for scheduler pods.

{}

 

 

keep

scheduler.podAnnotations

Annotations to add to the scheduler pods.

{}

 

 

keep

scheduler.podDisruptionBudget.config.maxUnavailable

Max unavailable pods for scheduler.

1

 

 

Remove maxUnavailable and minAvailable and set config: {}

scheduler.podDisruptionBudget.config.minAvailable

Min available pods for scheduler.

1

 

 

scheduler.podDisruptionBudget.enabled

Enable pod disruption budget.

False

 

 

keep

scheduler.priorityClassName

Specify priority for scheduler pods.

~

 

 

keep

scheduler.replicas

Airflow 2.0 allows users to run multiple schedulers. This feature is only recommended for MySQL 8+ and PostgreSQL

1

 

 

keep

scheduler.resources

Resources for scheduler pods.

{}

Examples:

resources:
  limits:
    cpu: 100m
    memory: 128Mi
  requests:
    cpu: 100m
    memory: 128Mi

 

 

keep

scheduler.safeToEvict

This setting tells Kubernetes that its ok to evict when it wants to scale a node down.

True

 

 

keep

scheduler.securityContext

Security context for the scheduler pod (deprecated, use securityContexts instead). If not set, the values from securityContext will be used.

{}

Examples:

securityContext:
  fsGroup: 0
  runAsGroup: 0
  runAsUser: 50000

 

 

keep

scheduler.serviceAccount.annotations

Annotations to add to the scheduler Kubernetes ServiceAccount.

{}

 

 

keep

scheduler.serviceAccount.automountServiceAccountToken

Specifies if ServiceAccount’s API credentials should be mounted onto Pods. When false, you can use serviceAccountTokenVolume to manually configure service account token volume for pod-launching executors.

True

 

 

keep

scheduler.serviceAccount.create

Specifies whether a ServiceAccount should be created.

True

 

 

keep

scheduler.serviceAccount.name

The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name.

~

 

 

keep

scheduler.serviceAccount.serviceAccountTokenVolume.audience

Intended audience of the token. Optional - defaults to the identifier of the Kubernetes API server.

~

 

 

Configuration options for other components are limited compared to this one. Maybe we should align the other component options with it?

scheduler.serviceAccount.serviceAccountTokenVolume.enabled

Enable manual service account token volume configuration.

False

 

 

scheduler.serviceAccount.serviceAccountTokenVolume.expirationSeconds

Token expiration time in seconds.

3600

 

 

scheduler.serviceAccount.serviceAccountTokenVolume.mountPath

Path where the service account token volume will be mounted.

/var/run/secrets/kubernetes.io/serviceaccount

 

 

scheduler.serviceAccount.serviceAccountTokenVolume.volumeName

Name of the service account token volume.

kube-api-access

 

 

scheduler.startupProbe.command

Command for LivenessProbe

~

 

 

keep

scheduler.startupProbe.failureThreshold

Minimum consecutive failures for the probe to be considered failed after having succeeded. Minimum value is 1.

6

 

 

keep

scheduler.startupProbe.initialDelaySeconds

Number of seconds after the container has started before startup probes are initiated.

0

 

 

keep

scheduler.startupProbe.periodSeconds

How often (in seconds) to perform the probe. Minimum value is 1.

10

 

 

keep

scheduler.startupProbe.timeoutSeconds

Number of seconds after which the probe times out. Minimum value is 1 seconds.

20

 

 

keep

scheduler.strategy

Specifies the strategy used to replace old Pods by new ones when deployed as a Deployment (when not using LocalExecutor and workers.persistence).

~

 

 

keep

scheduler.terminationGracePeriodSeconds

Grace period for scheduler to finish after SIGTERM is sent from Kubernetes.

10

 

 

keep

scheduler.tolerations

Specify Tolerations for scheduler pods.

[]

 

 

keep

scheduler.topologySpreadConstraints

Specify topology spread constraints for scheduler pods.

[]

 

 

keep

scheduler.updateStrategy

Specifies the strategy used to replace old Pods by new ones when deployed as a StatefulSet (when using LocalExecutor and workers.persistence).

~

 

 

keep

scheduler.waitForMigrations.enabled

Enable wait-for-airflow-migrations init container.

True

 

 

Not sure if this should be a container-level flag (I don't have a specific use case in mind when it would make sense to have it only in one component and not in another)

scheduler.waitForMigrations.env

Add additional env vars to wait-for-airflow-migrations init container.

[]

 

 

Webserver

Opinion Jens: Full section will be dropped in 1.21 with removal of Airflow 2 support. Big complexity reduction!

Opinion Przemek: Not sure if 1.21, but in general, yes - whole section drop with Airflow 2 support.

Workers

Parameter

Description

Default

Opinion Jens

Opinion Bugra

Opinion Przemek

workers.affinity

Specify scheduling constraints for Airflow Celery worker pods and pods created with pod-template-file.

See values.yaml

All "workers" options should go to "workers.celery", Deprecations /fallback removed

 

Move proper sections to workers.celery/workers.kubernetes sections and remove all other under the workers section (finish work started in https://github.com/apache/airflow/issues/28880)

workers.annotations

Annotations to add to the Airflow Celery worker deployment.

{}

ditto

 

workers.args

Args to use when running Airflow Celery workers (templated) (deprecated, use workers.celery.args instead).

['bash', '-c', 'exec \\\nairflow {{ semverCompare ">=2.0.0" .Values.airflowVersion | ternary "celery worker" "worker" }}\n{{- if and .Values.workers.queue (ne .Values.workers.queue "default") }}\n{{- " -q " }}{{ .Values.workers.queue }}\n{{- end }}']

ditto 

 

workers.celery.args

Args to use when running Airflow Celery workers (templated).

~

 

 

keep

workers.celery.command

Command to use when running Airflow Celery workers (templated).

~

 

 

keep

workers.celery.enableDefault

Enable the default worker defined by the workers and workers.celery configurations.

True

Should we have a "default" worker and sets in parallel? Can we not make the set havinf one default to reduce complexity?

 

keep

workers.celery.kerberosInitContainer.enabled

Enable Kerberos init container.

~

 

 

Move to Kustomize with the whole Kerberos stuff

workers.celery.kerberosInitContainer.resources

Resources on kerberos init container.

{}

Examples:

resources:
  limits:
    cpu: 100m
    memory: 128Mi
  requests:
    cpu: 100m
    memory: 128Mi

 

 

workers.celery.livenessProbe.command

Command for LivenessProbe

~

 

 

keep

workers.celery.livenessProbe.enabled

Enable liveness probe for Airflow Celery workers.

~

 

 

keep

workers.celery.livenessProbe.failureThreshold

Minimum consecutive failures for the probe to be considered failed after having succeeded. Minimum value is 1.

~

 

 

keep

workers.celery.livenessProbe.initialDelaySeconds

Number of seconds after the container has started before liveness probes are initiated.

~

 

 

keep

workers.celery.livenessProbe.periodSeconds

How often (in seconds) to perform the probe. Minimum value is 1.

~

 

 

keep

workers.celery.livenessProbe.timeoutSeconds

Number of seconds after which the probe times out. Minimum value is 1 seconds.

~

 

 

keep

workers.celery.persistence.annotations

Annotations to add to Airflow Celery worker volumes.

{}

 

 

keep

workers.celery.persistence.enabled

Enable persistent volumes.

~

 

 

keep

workers.celery.persistence.fixPermissions

Execute init container to chown log directory. This is currently only needed in kind, due to usage of local-path provisioner.

~

 

 

keep

workers.celery.persistence.size

Volume size for Airflow Celery worker StatefulSet.

~

 

 

keep

workers.celery.persistence.storageClassName

If using a custom StorageClass, pass name ref to all StatefulSets here (templated).

~

 

 

keep

workers.celery.podManagementPolicy

Specifies the policy for managing pods within the Airflow Celery worker. Only applicable to StatefulSet.

~

 

 

keep

workers.celery.queue

Queue name for the worker.

default

 

 

keep

workers.celery.replicas

Number of Airflow Celery workers.

~

 

 

keep

workers.celery.sets

List of worker sets. Each item can overwrite values from the parent workers and workers.celery sections.

[]

Here all properties from above and below should go in, with one "default"?

 


workers.celery.strategy

Specifies the strategy used to replace old Airflow Celery worker pods by new ones when deployed as a Deployment.

~

 

 

keep

workers.celery.updateStrategy

Specifies the strategy used to replace old Airflow Celery worker pods by new ones when deployed as a StatefulSet.

~

 

 

keep

workers.command

Command to use when running Airflow Celery workers and using pod-template-file (templated). Use workers.celery.command and/or workers.kubernetes.command to separate value between Celery workers and pod-template-file.

~

^^^ ditto

 

Move proper sections to workers.celery/workers.kubernetes sections and remove all other under the workers section (finish work started in https://github.com/apache/airflow/issues/28880)

workers.env

Add additional env vars to the Airflow Celery workers and pods created with pod-template-file.

[]

ditto

 

workers.extraContainers

Launch additional containers into Airflow Celery workers and pods created with pod-template-file (templated). Note, if used with KubernetesExecutor, you are responsible for signaling sidecars to exit when the main container finishes so Airflow can continue the worker shutdown process!

[]

ditto

 

workers.extraInitContainers

Add additional init containers into Airflow Celery workers and pods created with pod-template-file (templated).

[]

ditto

 

workers.extraPorts

Expose additional ports of Airflow Celery worker container.

[]

ditto

 

workers.extraVolumeMounts

Additional volume mounts attached to the Airflow Celery workers and pods created with pod-template-file.

[]

ditto

 

workers.extraVolumes

Additional volumes attached to the Airflow Celery workers and pods created with pod-template-file.

[]

ditto

 

workers.hostAliases

Specify HostAliases for Airflow Celery worker pods and pods created with pod-template-file.

[]

Examples:

hostAliases:
- hostnames:
  - test.hostname.one
  ip: 127.0.0.2
hostAliases:
- hostnames:
  - test.hostname.two
  ip: 127.0.0.3

ditto

 

workers.hpa.behavior

HorizontalPodAutoscalerBehavior configures the scaling behavior of the target.

{}

Should we have HPA and KEDA? How about moving this to Kustomize?

I really love this idea, as mentioned above, I would move them to Kustomize. 

workers.hpa.enabled

Allow HPA autoscaling (KEDA must be disabled).

False

 

 

workers.hpa.maxReplicaCount

Maximum number of Airflow Celery workers created by HPA.

5

 

 

workers.hpa.metrics

Specifications for which to use to calculate the desired replica count.

[{'type': 'Resource', 'resource': {'name': 'cpu', 'target': {'type': 'Utilization', 'averageUtilization': 80}}}]

 

 

workers.hpa.minReplicaCount

Minimum number of Airflow Celery workers created by HPA.

0

 

 

workers.keda.advanced

Advanced KEDA configuration.

{}

 

 

Move to Kustomize

 

 

 

 

workers.keda.advanced.horizontalPodAutoscalerConfig

HorizontalPodAutoscalerConfig specifies horizontal scale config.

{}

 

 

workers.keda.advanced.horizontalPodAutoscalerConfig.behavior

HorizontalPodAutoscalerBehavior configures the scaling behavior of the target.

{}

 

 

workers.keda.cooldownPeriod

How many seconds KEDA will wait before scaling to zero.

30

 

 

workers.keda.enabled

Allow KEDA autoscaling.

False

 

 

workers.keda.maxReplicaCount

Maximum number of Airflow Celery workers created by KEDA.

10

 

 

workers.keda.minReplicaCount

Minimum number of Airflow Celery workers created by KEDA.

0

 

 

workers.keda.namespaceLabels

Labels used in matchLabels for namespace in the PgBouncer NetworkPolicy.

{}

 

 

workers.keda.pollingInterval

How often KEDA polls the airflow DB to report new scale requests to the HPA.

5

 

 

workers.keda.query

Query to use for KEDA autoscaling. Must return a single integer.

SELECT ceil(COUNT(*)::decimal / {{ .Values.config.celery.worker_concurrency }}) FROM task_instance WHERE (state='running' OR state='queued') AND queue IN ( {{- range $i, $q := splitList "," .Values.workers.queue -}} {{- if $i }},{{ end }}'{{ $q | trim }}' {{- end -}} ) {{- if contains "CeleryKubernetesExecutor" .Values.executor }} AND queue != '{{ .Values.config.celery_kubernetes_executor.kubernetes_queue }}' {{- else if contains "KubernetesExecutor" .Values.executor }} AND executor IS DISTINCT FROM 'KubernetesExecutor' {{- else if contains "airflow.providers.edge3.executors.EdgeExecutor" .Values.executor }} AND executor IS DISTINCT FROM 'EdgeExecutor' {{- end }}

 

 

workers.keda.usePgbouncer

Weather to use PGBouncer to connect to the database or not when it is enabled. This configuration will be ignored if PGBouncer is not enabled.

True

 

 

workers.kerberosInitContainer.enabled

Enable Kerberos init container.

False

^^^ ditto

I agree, and Kerberos in Kustomize looks more manageable from both perspectives.

workers.kerberosInitContainer.resources

Resources on kerberos init container.

{}

Examples:

resources:
  limits:
    cpu: 100m
    memory: 128Mi
  requests:
    cpu: 100m
    memory: 128Mi

ditto

 

workers.kerberosSidecar.enabled

Enable Kerberos sidecar.

False

ditto 

 

workers.kerberosSidecar.resources

Resources on kerberos sidecar.

{}

Examples:

resources:
  limits:
    cpu: 100m
    memory: 128Mi
  requests:
    cpu: 100m
    memory: 128Mi

ditto

 

workers.kubernetes.command

Command to use in pod-template-file (templated).

~

 

 

keep

workers.kubernetes.kerberosInitContainer.enabled

Enable kerberos init container.

~

 

 

Move to Kustomize

workers.kubernetes.kerberosInitContainer.resources

Resources on kerberos init container.

{}

Examples:

resources:
  limits:
    cpu: 100m
    memory: 128Mi
  requests:
    cpu: 100m
    memory: 128Mi

 

 

workers.labels

Labels to add to the Airflow Celery workers objects and pods created with pod-template-file.

{}

^^^ ditto

 

Move proper sections to workers.celery/workers.kubernetes sections and remove all other under the workers section (finish work started in https://github.com/apache/airflow/issues/28880)

workers.livenessProbe.command

Command for LivenessProbe (deprecated, use workers.celery.livenessProbe.command instead)

~

ditto

 

workers.livenessProbe.enabled

Enable liveness probe for Airflow Celery workers (deprecated, use workers.celery.livenessProbe.enabled instead).

True

ditto

 

workers.livenessProbe.failureThreshold

Minimum consecutive failures for the probe to be considered failed after having succeeded. Minimum value is 1 (deprecated, use workers.celery.livenessProbe.failureThreshold instead).

5

ditto

 

workers.livenessProbe.initialDelaySeconds

Number of seconds after the container has started before liveness probes are initiated (deprecated, use workers.celery.livenessProbe.initialDelaySeconds instead).

10

ditto

 

workers.livenessProbe.periodSeconds

How often (in seconds) to perform the probe. Minimum value is 1 (deprecated, use workers.celery.livenessProbe.periodSeconds instead).

60

ditto

 

workers.livenessProbe.timeoutSeconds

Number of seconds after which the probe times out. Minimum value is 1 seconds (deprecated, use workers.celery.livenessProbe.timeoutSeconds instead).

20

ditto

 

workers.nodeSelector

Select certain nodes for Airflow Celery worker pods and pods created with pod-template-file.

{}

ditto

 

workers.persistence.annotations

Annotations to add to Airflow Celery worker volumes (deprecated, use workers.celery.persistence.annotations instead).

{}

ditto

 

workers.persistence.enabled

Enable persistent volumes (deprecated, use workers.celery.persistence.enabled instead).

True

ditto

 

workers.persistence.fixPermissions

Execute init container to chown log directory. This is currently only needed in kind, due to usage of local-path provisioner (deprecated, use workers.celery.persistence.fixPermissions instead).

False

ditto

 

Probably remove and/or move to Kustomize. It is only used in volume-permissions sidecar in workers, which, looking at the comment, is only needed with the local path provisioner https://github.com/rancher/local-path-provisioner setup.

workers.persistence.size

Volume size for Airflow Celery worker StatefulSet (deprecated, use workers.celery.persistence.size instead).

100Gi

ditto

 

Move proper sections to workers.celery/workers.kubernetes sections and remove all other under the workers section (finish work started in https://github.com/apache/airflow/issues/28880)

 

 

workers.persistence.storageClassName

If using a custom StorageClass, pass name ref to all StatefulSets here (templated) (deprecated, use workers.celery.persistence.storageClassName instead).

~

ditto

 

workers.podAnnotations

Annotations to add to the Airflow Celery workers and pods created with pod-template-file.

{}

ditto

 

workers.podDisruptionBudget.config.maxUnavailable

Max unavailable pods for worker.

1

ditto

 

workers.podDisruptionBudget.config.minAvailable

Min available pods for worker.

1

ditto

 

workers.podDisruptionBudget.enabled

Enable pod disruption budget.

False

ditto

 

workers.podManagementPolicy

Specifies the policy for managing pods within the Airflow Celery worker (deprecated, use workers.celery.podManagementPolicy instead). Only applicable to StatefulSet.

~

ditto

 

workers.priorityClassName

Specify priority for Airflow Celery worker pods and pods created with pod-template-file.

~

ditto

 

workers.replicas

Number of Airflow Celery workers (deprecated, use workers.celery.replicas instead).

1

ditto

 

workers.resources

Resource configuration for Airflow Celery workers and pods created with pod-template-file.

{}

Examples:

resources:
  limits:
    cpu: 100m
    memory: 128Mi
  requests:
    cpu: 100m
    memory: 128Mi

ditto

 

workers.runtimeClassName

Specify runtime for Airflow Celery worker pods and pods created with pod-template-file.

~

ditto

 

workers.safeToEvict

This setting tells Kubernetes that it’s ok to evict when it wants to scale a node down. It is used by Airflow Celery workers and pod-template-file.

False

ditto

 

workers.securityContext

Security context for the Airflow Celery worker pods and pods created with pod-template-file (deprecated, use securityContexts instead). If not set, the values from securityContext will be used.

{}

Examples:

securityContext:
  fsGroup: 0
  runAsGroup: 0
  runAsUser: 50000

ditto

 

workers.serviceAccount.annotations

Annotations to add to the worker Kubernetes ServiceAccount.

{}

ditto

 

workers.serviceAccount.automountServiceAccountToken

Specifies if ServiceAccount’s API credentials should be mounted onto Pods

True

ditto

 

workers.serviceAccount.create

Specifies whether a ServiceAccount should be created.

True

ditto

 

workers.serviceAccount.name

The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name.

~

ditto

 

workers.strategy

Specifies the strategy used to replace old Airflow Celery worker pods by new ones when deployed as a Deployment (deprecated, use workers.celery.strategy instead).

{'rollingUpdate': {'maxSurge': '100%', 'maxUnavailable': '50%'}}

ditto

 

workers.terminationGracePeriodSeconds

Grace period for tasks to finish after SIGTERM is sent from Kubernetes. It is used by Airflow Celery workers and pod-template-file.

600

ditto

 

workers.tolerations

Specify Tolerations for Airflow Celery worker pods and pods created with pod-template-file.

[]

ditto

 

workers.topologySpreadConstraints

Specify topology spread constraints for Airflow Celery worker pods and pods created with pod-template-file.

[]

ditto

 

workers.updateStrategy

Specifies the strategy used to replace old Airflow Celery worker pods by new ones when deployed as a StatefulSet (deprecated, use workers.celery.updateStrategy instead).

~

ditto

 

workers.volumeClaimTemplates

Specify additional volume claim template for Airflow Celery workers.

[]

Examples:

volumeClaimTemplates:
- accessModes:
  - ReadWriteOnce
  name: data-volume-1
  resources:
    requests:
      storage: 10Gi
  storageClassName: storage-class-1
volumeClaimTemplates:
- accessModes:
  - ReadWriteOnce
  name: data-volume-2
  resources:
    requests:
      storage: 20Gi
  storageClassName: storage-class-2

ditto

 

workers.waitForMigrations.enabled

Enable wait-for-airflow-migrations init container.

True

ditto

 

workers.waitForMigrations.env

Add additional env vars to wait-for-airflow-migrations init container.

[]

ditto

 

Triggerer

Parameter

Description

Default

Opinion Jens

Opinion Bugra

Opinion Przemek

triggerer.affinity

Specify scheduling constraints for triggerer pods.

See values.yaml

 

 

keep

triggerer.annotations

Annotations to add to the triggerer deployment

{}

 

 

keep

triggerer.args

Args to use when running the Airflow triggerer (templated).

['bash', '-c', 'exec airflow triggerer']

 

 

keep

triggerer.command

Command to use when running the Airflow triggerer (templated).

~

 

 

keep

triggerer.enabled

Enable triggerer

True

 

 

keep

triggerer.env

Add additional env vars to triggerer.

[]

 

 

keep

triggerer.extraContainers

Launch additional containers into triggerer (templated).

[]

 

 

keep

triggerer.extraInitContainers

Add additional init containers into triggerer (templated).

[]

 

 

keep

triggerer.extraVolumeMounts

Mount additional volumes into triggerer.

[]

 

 

keep

triggerer.extraVolumes

Mount additional volumes into triggerer.

[]

 

 

keep

triggerer.hostAliases

HostAliases for the triggerer pod.

[]

Examples:

hostAliases:
- hostnames:
  - foo.local
  ip: 127.0.0.1
hostAliases:
- hostnames:
  - foo.remote
  ip: 10.1.2.3

 

 

keep

triggerer.keda.advanced

Advanced KEDA configuration.

{}

Same like above: Should we have KEDA/HPA as Kustomize only?

 

 

 

 

 

 

 

 

 

 

I agree moving idea as above

Move to Kustomize

triggerer.keda.advanced.horizontalPodAutoscalerConfig

HorizontalPodAutoscalerConfig specifies horizontal scale config.

{}

 

triggerer.keda.advanced.horizontalPodAutoscalerConfig.behavior

HorizontalPodAutoscalerBehavior configures the scaling behavior of the target.

{}

 

triggerer.keda.cooldownPeriod

How many seconds KEDA will wait before scaling to zero.

30

 

triggerer.keda.enabled

Allow KEDA autoscaling.

False

 

triggerer.keda.maxReplicaCount

Maximum number of triggerers created by KEDA.

10

 

triggerer.keda.minReplicaCount

Minimum number of triggerers created by KEDA.

0

 

triggerer.keda.namespaceLabels

Labels used in matchLabels for namespace in the PgBouncer NetworkPolicy.

{}

 

triggerer.keda.pollingInterval

How often KEDA polls the airflow DB to report new scale requests to the HPA.

5

 

triggerer.keda.query

Query to use for KEDA autoscaling. Must return a single integer.

SELECT ceil(COUNT(*)::decimal / {{ include "triggerer.capacity" . }}) FROM trigger

 

triggerer.keda.usePgbouncer

Whether to use PGBouncer to connect to the database or not when it is enabled. This configuration will be ignored if PGBouncer is not enabled.

False

 

triggerer.labels

Labels to add to the triggerer objects and pods.

{}

 

 

keep

triggerer.livenessProbe.command

Command for LivenessProbe

~

 

 

keep

triggerer.livenessProbe.failureThreshold

Minimum consecutive failures for the probe to be considered failed after having succeeded. Minimum value is 1.

5

 

 

keep

triggerer.livenessProbe.initialDelaySeconds

Number of seconds after the container has started before liveness probes are initiated.

10

 

 

keep

triggerer.livenessProbe.periodSeconds

How often (in seconds) to perform the probe. Minimum value is 1.

60

 

 

keep

triggerer.livenessProbe.timeoutSeconds

Number of seconds after which the probe times out. Minimum value is 1 seconds.

20

 

 

keep

triggerer.nodeSelector

Select certain nodes for triggerer pods.

{}

 

 

keep

triggerer.persistence.annotations

Annotations to add to triggerer volumes.

{}

 

 

keep

triggerer.persistence.enabled

Enable persistent volumes.

True

 

 

keep

triggerer.persistence.fixPermissions

Execute init container to chown log directory. This is currently only needed in kind, due to usage of local-path provisioner.

False

 

 

Not used - drop

triggerer.persistence.size

Volume size for triggerer StatefulSet.

100Gi

 

 

keep

triggerer.persistence.storageClassName

If using a custom StorageClass, pass name ref to all StatefulSets here (templated).

~

 

 

keep

triggerer.podAnnotations

Annotations to add to the triggerer pods.

{}

 

 

keep

triggerer.podDisruptionBudget.config.maxUnavailable

Max unavailable pods for triggerer.

1

 

 

Remove maxUnavailable and minAvailable and set config: {}

triggerer.podDisruptionBudget.config.minAvailable

Min available pods for triggerer.

1

 

 

triggerer.podDisruptionBudget.enabled

Enable pod disruption budget.

False

 

 

keep

triggerer.priorityClassName

Specify priority for triggerer pods.

~

 

 

keep

triggerer.replicas

Number of triggerers to run.

1

 

 

keep

triggerer.resources

Resources for triggerer pods.

{}

Examples:

resources:
  limits:
    cpu: 100m
    memory: 128Mi
  requests:
    cpu: 100m
    memory: 128Mi

 

 

keep

triggerer.safeToEvict

This setting tells Kubernetes that its ok to evict when it wants to scale a node down.

True

 

 

keep

triggerer.securityContext

Security context for the triggerer pod (deprecated, use securityContexts instead). If not set, the values from securityContext will be used.

{}

Examples:

securityContext:
  fsGroup: 0
  runAsGroup: 0
  runAsUser: 50000

 

 

Drop

triggerer.serviceAccount.annotations

Annotations to add to the triggerer Kubernetes ServiceAccount.

{}

 

 

keep

triggerer.serviceAccount.automountServiceAccountToken

Specifies if ServiceAccount’s API credentials should be mounted onto Pods.

True

 

 

keep

triggerer.serviceAccount.create

Specifies whether a ServiceAccount should be created.

True

 

 

keep

triggerer.serviceAccount.name

The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name.

~

 

 

keep

triggerer.strategy

Specifies the strategy used to replace old Pods by new ones when deployed as a Deployment.

{'rollingUpdate': {'maxSurge': '100%', 'maxUnavailable': '50%'}}

 

 

keep

triggerer.terminationGracePeriodSeconds

Grace period for tasks to finish after SIGTERM is sent from Kubernetes.

60

 

 

keep

triggerer.tolerations

Specify Tolerations for triggerer pods.

[]

 

 

keep

triggerer.topologySpreadConstraints

Specify topology spread constraints for triggerer pods.

[]

 

 

keep

triggerer.updateStrategy

Specifies the strategy used to replace old Pods by new ones when deployed as a StatefulSet.

~

 

 

keep

triggerer.waitForMigrations.enabled

Enable wait-for-airflow-migrations init container.

True

 

 

Not sure if this should be a container-level flag (I don't have a specific use case in mind when it would make sense to have it only in one component and not in another)

triggerer.waitForMigrations.env

Add additional env vars to wait-for-airflow-migrations init container.

[]

 

 

DagProcessor

Parameter

Description

Default

Opinion Jens

Opinion Bugra

Opinion Przemek

dagProcessor.affinity

Specify scheduling constraints for dag processor pods.

See values.yaml

 

 

keep

dagProcessor.annotations

Annotations to add to the dag processor deployment

{}

 

 

keep

dagProcessor.args

Args to use when running the Airflow dag processor (templated).

['bash', '-c', 'exec airflow dag-processor']

Same like with workers: There is this PR to start a deployment per bundle. Should structure be here made similar like with workers? (one default can add more?)


Having the same, not different, structure would be probably easier (in terms of logic) to maintain, but I would wait until we will have on main branch finished work with workers.celery/workers.kubernetes sections to see how logic and complexity will look like without the whole complexity due to backward compatibility

dagProcessor.command

Command to use when running the Airflow dag processor (templated).

~

 

 

keep

dagProcessor.dagBundleConfigList

Define Dag bundles in a structured YAML format. This will be automatically converted to JSON string format for config.dag_processor.dag_bundle_config_list.

[{'name': 'dags-folder', 'classpath': 'airflow.dag_processing.bundles.local.LocalDagBundle', 'kwargs': {}}]

 

 

keep

dagProcessor.dagBundleConfigList[].kwargs

Keyword arguments for the Dag bundle.

{}

 

 

keep

dagProcessor.enabled

Enable standalone dag processor (requires Airflow 2.3.0+).

~

 

 

keep

dagProcessor.env

Add additional env vars to dag processor.

[]

 

 

keep

dagProcessor.extraContainers

Launch additional containers into dag processor (templated).

[]

 

 

keep

dagProcessor.extraInitContainers

Add additional init containers into dag processor (templated).

[]

 

 

keep

dagProcessor.extraVolumeMounts

Mount additional volumes into dag processor.

[]

 

 

keep

dagProcessor.extraVolumes

Mount additional volumes into dag processor.

[]

 

 

keep

dagProcessor.labels

Labels specific to dag processor objects and pods

{}

 

 

keep

dagProcessor.livenessProbe.command

Command for LivenessProbe

~

 

 

keep

dagProcessor.livenessProbe.failureThreshold

Minimum consecutive failures for the probe to be considered failed after having succeeded. Minimum value is 1.

5

 

 

keep

dagProcessor.livenessProbe.initialDelaySeconds

Number of seconds after the container has started before liveness probes are initiated.

10

 

 

keep

dagProcessor.livenessProbe.periodSeconds

How often (in seconds) to perform the probe. Minimum value is 1.

60

 

 

keep

dagProcessor.livenessProbe.timeoutSeconds

Number of seconds after which the probe times out. Minimum value is 1 seconds.

20

 

 

keep

dagProcessor.nodeSelector

Select certain nodes for dag processor pods.

{}

 

 

keep

dagProcessor.podAnnotations

Annotations to add to the dag processor pods.

{}

 

 

keep

dagProcessor.podDisruptionBudget.config.maxUnavailable

Max unavailable pods for dag processor.

1

 

 

Remove maxUnavailable and minAvailable and set config: {}

dagProcessor.podDisruptionBudget.config.minAvailable

Min available pods for dag processor.

1

 

 

dagProcessor.podDisruptionBudget.enabled

Enable pod disruption budget.

False

 

 

keep

dagProcessor.priorityClassName

Specify priority for dag processor pods.

~

 

 

keep

dagProcessor.replicas

Number of dag processors to run.

1

 

 

keep

dagProcessor.resources

Resources for dag processor pods.

{}

Examples:

resources:
  limits:
    cpu: 100m
    memory: 128Mi
  requests:
    cpu: 100m
    memory: 128Mi

 

 

keep

dagProcessor.safeToEvict

This setting tells Kubernetes that its ok to evict when it wants to scale a node down.

True

 

 

keep

dagProcessor.securityContext

Security context for the dag processor pod (deprecated, use securityContexts instead). If not set, the values from securityContext will be used.

{}

Examples:

securityContext:
  fsGroup: 0
  runAsGroup: 0
  runAsUser: 50000

 

 

keep

dagProcessor.serviceAccount.annotations

Annotations to add to the dag processor Kubernetes ServiceAccount.

{}

 

 

keep

dagProcessor.serviceAccount.automountServiceAccountToken

Specifies if ServiceAccount’s API credentials should be mounted onto Pods.

True

 

 

keep

dagProcessor.serviceAccount.create

Specifies whether a ServiceAccount should be created.

True

 

 

keep

dagProcessor.serviceAccount.name

The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name.

~

 

 

keep

dagProcessor.strategy

Specifies the strategy used to replace old Pods by new ones when deployed as a Deployment.

{'rollingUpdate': {'maxSurge': '100%', 'maxUnavailable': '50%'}}

 

 

keep

dagProcessor.terminationGracePeriodSeconds

Grace period for tasks to finish after SIGTERM is sent from Kubernetes.

60

 

 

keep

dagProcessor.tolerations

Specify Tolerations for dag processor pods.

[]

 

 

keep

dagProcessor.topologySpreadConstraints

Specify topology spread constraints for dag processor pods.

[]

 

 

keep

dagProcessor.waitForMigrations.enabled

Enable wait-for-airflow-migrations init container.

True

 

 

Not sure if this should be a container-level flag (I don't have a specific use case in mind when it would make sense to have it only in one component and not in another)

dagProcessor.waitForMigrations.env

Add additional env vars to wait-for-airflow-migrations init container.

[]

 

 

Flower

Parameter

Description

Default

Opinion Jens

Opinion Bugra

Opinion Przemek

flower.affinity

Specify scheduling constraints for Flower pods.

{}

 

 

Move to Kustomize

flower.annotations

Annotations to add to the flower deployment

{}

 

 

flower.args

Args to use when running flower (templated).

['bash', '-c', 'exec \\\nairflow {{ semverCompare ">=2.0.0" .Values.airflowVersion | ternary "celery flower" "flower" }}']

 

 

flower.command

Command to use when running flower (templated).

~

 

 

flower.enabled

Enable Flower.

False

Is flower a component that should be supported by helm chart? Not complex but could be carved-out into a separate deployment add-on in Kustomize...

I agree to this as well. Let's move them out. It won't add much load but I also mentioned a similar thing for Pgbouncer too. How about approaching all these as the same?

flower.env

Add additional env vars to flower.

[]

 

 

flower.extraContainers

Launch additional containers into the flower pods.

[]

 

 

flower.extraNetworkPolicies

Additional NetworkPolicies as needed (Deprecated - renamed to flower.networkPolicy.ingress.from).

[]

 

 

flower.extraVolumeMounts

Mount additional volumes into the flower pods.

[]

 

 

flower.extraVolumes

Mount additional volumes into the flower pods.

[]

 

 

flower.labels

Labels to add to the flower objects and pods.

{}

 

 

flower.livenessProbe.failureThreshold

Flower Liveness probe failure threshold.

10

 

 

flower.livenessProbe.initialDelaySeconds

Flower Liveness probe initial delay.

10

 

 

flower.livenessProbe.periodSeconds

Flower Liveness probe period seconds.

5

 

 

flower.livenessProbe.timeoutSeconds

Flower Liveness probe timeout seconds.

5

 

 

flower.networkPolicy.ingress.from

Peers for flower NetworkPolicyingress.

[]

 

 

flower.networkPolicy.ingress.ports

Ports for flower NetworkPolicyingress (if from is set).

[{'port': '{{ .Values.ports.flowerUI }}'}]

Examples:

ports:
- port: 5565

 

 

flower.nodeSelector

Select certain nodes for Flower pods.

{}

 

 

flower.password

Password use to access Flower.

~

 

 

flower.podAnnotations

Annotations to add to the Flower pods.

{}

 

 

flower.priorityClassName

Specify priority for Flower pods.

~

 

 

flower.readinessProbe.failureThreshold

Flower Readiness probe failure threshold.

10

 

 

flower.readinessProbe.initialDelaySeconds

Flower Readiness probe initial delay.

10

 

 

flower.readinessProbe.periodSeconds

Flower Readiness probe period seconds.

5

 

 

flower.readinessProbe.timeoutSeconds

Flower Readiness probe timeout seconds.

5

 

 

flower.resources

Resources for Flower pods.

{}

Examples:

resources:
  limits:
    cpu: 100m
    memory: 128Mi
  requests:
    cpu: 100m
    memory: 128Mi

 

 

flower.secretAnnotations

Annotations to add to the flower secret.

{}

 

 

flower.secretName

A secret containing the user and password pair.

~

 

 

flower.securityContext

Security context for the flower pod (deprecated, use securityContexts instead). If not set, the values from securityContext will be used.

{}

Examples:

securityContext:
  fsGroup: 0
  runAsGroup: 0
  runAsUser: 50000

 

 

flower.service.annotations

Annotations for the flower Service.

{}

 

 

flower.service.loadBalancerIP

Flower Service loadBalancerIP.

~

 

 

flower.service.loadBalancerSourceRanges

Flower Service loadBalancerSourceRanges.

[]

Examples:

loadBalancerSourceRanges:
- 10.123.0.0/16

 

 

flower.service.ports

Ports for the flower Service.

[{'name': 'flower-ui', 'port': '{{ .Values.ports.flowerUI }}'}]

Examples:

ports:
- name: flower-ui
  port: 8080
  targetPort: flower-ui

 

 

flower.service.type

Flower Service type.

ClusterIP

 

 

flower.serviceAccount.annotations

Annotations to add to the worker Kubernetes ServiceAccount.

{}

 

 

flower.serviceAccount.automountServiceAccountToken

Specifies if ServiceAccount’s API credentials should be mounted onto Pods.

True

 

 

flower.serviceAccount.create

Specifies whether a ServiceAccount should be created.

True

 

 

flower.serviceAccount.name

The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name.

~

 

 

flower.startupProbe.failureThreshold

Flower Startup probe failure threshold.

6

 

 

flower.startupProbe.initialDelaySeconds

Flower Startup probe initial delay seconds.

0

 

 

flower.startupProbe.periodSeconds

Flower Startup probe period seconds.

10

 

 

flower.startupProbe.timeoutSeconds

Flower Startup probe timeout seconds.

20

 

 

flower.tolerations

Specify Tolerations for Flower pods.

[]

 

 

flower.topologySpreadConstraints

Specify topology spread constraints for Flower pods.

[]

 

 

flower.username

Username use to access Flower.

~

 

 

Redis

Parameter

Description

Default

Opinion Jens

Opinion Bugra

Opinion Przemek

data.brokerUrl

Direct url to the redis broker (when using an external redis instance) (can only be set during install, not upgrade).

~

 

 

Move to workers.celery/redis as it is only for CeleryExecutor

data.brokerUrlSecretName

Redis broker URL secret.

~

 

 

redis.affinity

Specify scheduling constraints for Redis pods.

{}

 

 

From one perspective, it is outside Airflow components (additional thing), from another, it is core for CeleryExecutor setup. I would think of moving it to Kustomize, but it could also stay in the core chart

redis.annotations

Annotations for the redis.

{}

 

 

redis.emptyDirConfig

Configuration for redis empty dir volume.

~

 

 

redis.enabled

Enable the Redis provisioned by the chart (you can also use an external Redis instance with data.brokerUrl or data.brokerUrlSecretName).

True

 

 

redis.labels

Labels to add to the redis objects and pods.

{}

 

 

redis.nodeSelector

Select certain nodes for Redis pods.

{}

 

 

redis.password

If password is set, create secret with it, else generate a new one on install (can only be set during install, not upgrade).

~

 

 

redis.passwordSecretAnnotations

Annotations to add to the redis password secret.

{}

 

 

redis.passwordSecretName

Redis password secret.

~

 

 

redis.persistence.annotations

Annotations to add to redis volumes.

{}

 

 

redis.persistence.enabled

Enable persistent volumes.

True

 

 

redis.persistence.existingClaim

The name of an existing PVC to use.

~

 

 

redis.persistence.size

Volume size for Redis StatefulSet.

1Gi

 

 

redis.persistence.storageClassName

If using a custom StorageClass, pass name ref to all StatefulSets here (templated).

~

 

 

redis.podAnnotations

Annotations to add to the redis pods.

{}

 

 

redis.priorityClassName

Specify priority for redis pods.

~

 

 

redis.resources

Resources for the Redis pods

{}

Examples:

resources:
  limits:
    cpu: 100m
    memory: 128Mi
  requests:
    cpu: 100m
    memory: 128Mi

 

 

redis.safeToEvict

This setting tells Kubernetes that its ok to evict when it wants to scale a node down.

True

 

 

redis.securityContext

Security context for the cleanup job pod (deprecated, use securityContexts instead). If not set, the values from securityContext will be used.

{}

Examples:

securityContext:
  fsGroup: 0
  runAsGroup: 0
  runAsUser: 50000

 

 

redis.service.clusterIP

If using ClusterIP service type, custom IP address can be specified.

~

 

 

redis.service.nodePort

If using NodePort service type, custom node port can be specified.

~

 

 

redis.service.type

Service type.

ClusterIP

 

 

redis.serviceAccount.annotations

Annotations to add to the worker Kubernetes ServiceAccount.

{}

 

 

redis.serviceAccount.automountServiceAccountToken

Specifies if ServiceAccount’s API credentials should be mounted onto Pods

True

 

 

redis.serviceAccount.create

Specifies whether a ServiceAccount should be created.

True

 

 

redis.serviceAccount.name

The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name.

~

 

 

redis.terminationGracePeriodSeconds

Grace period for Redis to exit after SIGTERM is sent from Kubernetes.

600

 

 

redis.tolerations

Specify Tolerations for Redis pods.

[]

 

 

redis.topologySpreadConstraints

Specify topology spread constraints for Redis pods.

[]

 

 

redis.uid

Redis run as user parameter.

0

 

 

StatsD

Parameter

Description

Default

Opinion Jens

Opinion Bugra

Opinion Przemek

statsd.affinity

Specify scheduling constraints for StatsD pods.

{}

 

 

Move to Kustomize & document OpenTelemetry setup too

statsd.annotations

Annotations to add to the StatsD deployment.

{}

 

 

statsd.args

Args to use when running statsd-exporter (templated).

['--statsd.mapping-config=/etc/statsd-exporter/mappings.yml']

 

 

statsd.cache.size

Maximum number of metric mappings to cache in memory. Higher values improve performance for frequently used metrics but consume more memory.

1000

 

 

statsd.cache.ttl

Time-to-live for cached metric mappings. Determines how long mappings remain in cache before expiring. Set to ‘0s’ to disable expiration.

0s

 

 

statsd.cache.type

Cache eviction strategy for metric mappings. lru (Least Recently Used) evicts oldest accessed items, ‘random’ evicts randomly selected items.

lru

 

 

statsd.configMapAnnotations

Extra annotations to apply to the statsd configmap.

{}

 

 

statsd.enabled

Enable StatsD.

True

 

 

statsd.env

Add additional env vars to statsd container.

[]

 

 

statsd.extraMappings

Additional mappings for StatsD exporter.If set, will merge default mapping and extra mappings, default mapping has higher priority. So, if you want to change some default mapping, please use overrideMappings

[]

 

 

statsd.extraNetworkPolicies

Additional NetworkPolicies as needed.

[]

 

 

statsd.labels

Labels specific to statsd objects and pods

{}

 

 

statsd.nodeSelector

Select certain nodes for StatsD pods.

{}

 

 

statsd.overrideMappings

Override mappings for StatsD exporter.If set, will ignore setting item in default and extraMappings. So, If you use it, ensure all mapping item contains in it.

[]

 

 

statsd.podAnnotations

Annotations to add to the StatsD pods.

{}

 

 

statsd.priorityClassName

Specify priority for StatsD pods.

~

 

 

statsd.resources

Resources for StatsD pods.

{}

Examples:

resources:
  limits:
    cpu: 100m
    memory: 128Mi
  requests:
    cpu: 100m
    memory: 128Mi

 

 

statsd.securityContext

Security context for the StatsD pod (deprecated, use securityContexts instead).

{}

Examples:

securityContext:
  fsGroup: 0
  runAsGroup: 0
  runAsUser: 50000

 

 

statsd.service.extraAnnotations

Extra annotations for the StatsD Service.

{}

 

 

statsd.serviceAccount.annotations

Annotations to add to the StatsD Kubernetes ServiceAccount.

{}

 

 

statsd.serviceAccount.automountServiceAccountToken

Specifies if ServiceAccount’s API credentials should be mounted onto Pods.

True

 

 

statsd.serviceAccount.create

Specifies whether a ServiceAccount should be created.

True

 

 

statsd.serviceAccount.name

The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name.

~

 

 

statsd.terminationGracePeriodSeconds

Grace period for statsd to finish after SIGTERM is sent from Kubernetes.

30

 

 

statsd.tolerations

Specify Tolerations for StatsD pods.

[]

 

 

statsd.topologySpreadConstraints

Specify topology spread constraints for StatsD pods.

[]

 

 

statsd.uid

StatsD run as user parameter.

65534

 

 

Jobs

Parameter

Description

Default

Opinion Jens

Opinion Bugra

Opinion Przemek

cleanup.affinity

Specify scheduling constraints for cleanup pods.

{}

 

 

Move it under workers.kubernetes section as it is needed only for KubernetesExecutor

cleanup.args

Args to use when running the cleanup cronjob (templated).

['bash', '-c', 'exec airflow kubernetes cleanup-pods --namespace={{ .Release.Namespace }}']

 

 

cleanup.command

Command to use when running the cleanup cronjob (templated).

~

 

 

cleanup.enabled

Enable cleanup.

False

 

 

cleanup.env

Add additional env vars to cleanup.

[]

 

 

cleanup.jobAnnotations

Annotations to add to the cleanup cronjob.

{}

 

 

cleanup.labels

labels to add to cleanup pods.

{}

 

 

cleanup.nodeSelector

Select certain nodes for cleanup pods.

{}

 

 

cleanup.podAnnotations

Annotations to add to cleanup pods.

{}

 

 

cleanup.priorityClassName

Specify priority for cleanup pods.

~

 

 

cleanup.resources

Resources for cleanup pods

{}

Examples:

resources:
  limits:
    cpu: 100m
    memory: 128Mi
  requests:
    cpu: 100m
    memory: 128Mi

 

 

cleanup.schedule

Cleanup schedule (templated).

*/15 * * * *

 

 

cleanup.securityContext

Security context for the cleanup job pod (deprecated, use securityContexts instead). If not set, the values from securityContext will be used.

{}

Examples:

securityContext:
  fsGroup: 0
  runAsGroup: 0
  runAsUser: 50000

 

 

cleanup.serviceAccount.annotations

Annotations to add to the cleanup CronJob Kubernetes ServiceAccount.

{}

 

 

cleanup.serviceAccount.automountServiceAccountToken

Specifies if ServiceAccount’s API credentials should be mounted onto Pods

True

 

 

cleanup.serviceAccount.create

Specifies whether a ServiceAccount should be created.

True

 

 

cleanup.serviceAccount.name

The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name.

~

 

 

cleanup.tolerations

Specify Tolerations for cleanup pods.

[]

 

 

cleanup.topologySpreadConstraints

Specify topology spread constraints for cleanup pods.

[]

 

 

createUserJob.affinity

Specify scheduling constraints for the create user job pod.

{}

 

 

Move under the new flask section


Agreed on making createUserJob.enabled false by default

createUserJob.annotations

Annotations to add to the create user job pod.

{}

 

 

createUserJob.applyCustomEnv

Specify if you want additional configured env vars applied to this job

True

 

 

createUserJob.args

Args to use when running create user job (templated).

['bash', '-c', 'exec \\\nairflow {{ semverCompare ">=2.0.0" .Values.airflowVersion | ternary "users create" "create_user" }} "$@"', '--', '-r', '{{ if .Values.webserver.defaultUser }}{{ .Values.webserver.defaultUser.role }}{{ else }}{{ .Values.createUserJob.defaultUser.role }}{{ end }}', '-u', '{{ if .Values.webserver.defaultUser }}{{ .Values.webserver.defaultUser.username }}{{ else }}{{ .Values.createUserJob.defaultUser.username }}{{ end }}', '-e', '{{ if .Values.webserver.defaultUser }}{{ .Values.webserver.defaultUser.email }}{{ else }}{{ .Values.createUserJob.defaultUser.email }}{{ end }}', '-f', '{{ if .Values.webserver.defaultUser }}{{ .Values.webserver.defaultUser.firstName }}{{ else }}{{ .Values.createUserJob.defaultUser.firstName }}{{ end }}', '-l', '{{ if .Values.webserver.defaultUser }}{{ .Values.webserver.defaultUser.lastName }}{{ else }}{{ .Values.createUserJob.defaultUser.lastName }}{{ end }}', '-p', '{{ if .Values.webserver.defaultUser }}{{ .Values.webserver.defaultUser.password }}{{ else }}{{ .Values.createUserJob.defaultUser.password }}{{ end }}']

 

 

createUserJob.command

Command to use when running create user job (templated).

~

 

 

createUserJob.defaultUser.email

Default user email address.

admin@example.com

 

 

createUserJob.defaultUser.firstName

Default user firstname.

admin

 

 

createUserJob.defaultUser.lastName

Default user lastname.

user

 

 

createUserJob.defaultUser.password

Default user password.

admin

 

 

createUserJob.defaultUser.role

Default user role.

Admin

 

 

createUserJob.defaultUser.username

Default user username.

admin

 

 

createUserJob.enabled

Whether the create user job should be created.

True

Should we make this Fals eas default not to be insecure by default?

Make sense. The only thing people can expect this is true by default. Maybe we can add docs there to route them to 

createUserJob.env

Add additional env vars to the create user job pod.

[]

 

 

createUserJob.extraContainers

Launch additional containers for the create user job pod

[]

 

 

createUserJob.extraInitContainers

Add additional init containers into create user job pod (templated).

[]

 

 

createUserJob.extraVolumeMounts

Mount additional volumes into create user job

[]

 

 

createUserJob.extraVolumes

Mount additional volumes into create user job

[]

 

 

createUserJob.jobAnnotations

Annotations to add to the create user job job.

{}

 

 

createUserJob.labels

Labels to add to the create user job objects and pods.

{}

 

 

createUserJob.nodeSelector

Select certain nodes for the create user job pod.

{}

 

 

createUserJob.priorityClassName

Specify priority for the create user job pod.

~

 

 

createUserJob.resources

Resources for the create user job pod

{}

Examples:

resources:
  limits:
    cpu: 100m
    memory: 128Mi
  requests:
    cpu: 100m
    memory: 128Mi

 

 

createUserJob.restartPolicy

Restart policy for the database migration job.

OnFailure

 

 

createUserJob.securityContext

Security context for the create user job pod (deprecated, use securityContexts instead). If not set, the values from securityContext will be used.

{}

Examples:

securityContext:
  fsGroup: 0
  runAsGroup: 0
  runAsUser: 50000

 

 

createUserJob.serviceAccount.annotations

Annotations to add to the create user job Kubernetes ServiceAccount.

{}

 

 

createUserJob.serviceAccount.automountServiceAccountToken

Specifies if ServiceAccount’s API credentials should be mounted onto Pods.

True

 

 

createUserJob.serviceAccount.create

Specifies whether a ServiceAccount should be created.

True

 

 

createUserJob.serviceAccount.name

The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name.

~

 

 

createUserJob.tolerations

Specify Tolerations for the create user job pod.

[]

 

 

createUserJob.topologySpreadConstraints

Specify topology spread constraints for the create user job pod.

[]

 

 

createUserJob.ttlSecondsAfterFinished

Limit the lifetime of the job object after it finished execution

300

 

 

createUserJob.useHelmHooks

Specify if you want to use the default Helm Hook annotations

True

 

 

databaseCleanup.affinity

Specify scheduling constraints for database cleanup pods.

{}

 

 

keep

databaseCleanup.applyCustomEnv

Specify if you want additional configured env vars applied to database cleanup job

True

 

 

keep

databaseCleanup.args

Args to use when running the database cleanup cronjob (templated).

['-c', 'CLEAN_TS=$(date -d "-{{ .Values.databaseCleanup.retentionDays }} days" +"%Y-%m-%dT%H:%M:%S"); echo "Cleaning up metadata DB entries older than ${CLEAN_TS}"; exec airflow db clean --clean-before-timestamp "${CLEAN_TS}" --yes {{- if .Values.databaseCleanup.skipArchive }} --skip-archive{{ end }} {{- if .Values.databaseCleanup.verbose }} --verbose{{ end }} {{- with .Values.databaseCleanup.batchSize }} --batch-size {{ . }}{{ end }} {{- with .Values.databaseCleanup.tables }} --tables {{ . | join "," }}{{ end }}']

 

 

keep

databaseCleanup.batchSize

Maximum number of rows to delete or archive in a single transaction.

~

 

 

keep

databaseCleanup.command

Command to use when running the database cleanup cronjob (templated).

['bash']

 

 

keep

databaseCleanup.enabled

Enable database cleanup.

False

 

 

keep

databaseCleanup.env

Add additional env vars to database cleanup.

[]

 

 

keep

databaseCleanup.jobAnnotations

Annotations to add to the database cleanup cronjob.

{}

 

 

keep

databaseCleanup.labels

labels to add to database cleanup pods.

{}

 

 

keep

databaseCleanup.nodeSelector

Select certain nodes for database cleanup pods.

{}

 

 

keep

databaseCleanup.podAnnotations

Annotations to add to database cleanup pods.

{}

 

 

keep

databaseCleanup.priorityClassName

Specify priority for database cleanup pods.

~

 

 

keep

databaseCleanup.resources

Resources for database cleanup pods

{}

Examples:

resources:
  limits:
    cpu: 100m
    memory: 128Mi
  requests:
    cpu: 100m
    memory: 128Mi

 

 

keep

databaseCleanup.retentionDays

Number of days to retain records in the metadata database.

90

 

 

keep

databaseCleanup.schedule

Database cleanup schedule (templated).

0 0 * * 0

 

 

keep

databaseCleanup.serviceAccount.annotations

Annotations to add to the database cleanup CronJob Kubernetes ServiceAccount.

{}

 

 

keep

databaseCleanup.serviceAccount.automountServiceAccountToken

Specifies if ServiceAccount’s API credentials should be mounted onto Pods

True

 

 

keep

databaseCleanup.serviceAccount.create

Specifies whether a ServiceAccount should be created.

True

 

 

keep

databaseCleanup.serviceAccount.name

The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name.

~

 

 

keep

databaseCleanup.skipArchive

Don’t preserve purged records in an archive table.

False

 

 

keep

databaseCleanup.tables

Table names to perform maintenance on. Supported values in: https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#clean

[]

 

 

keep

databaseCleanup.tolerations

Specify Tolerations for database cleanup pods.

[]

 

 

keep

databaseCleanup.topologySpreadConstraints

Specify topology spread constraints for database cleanup pods.

[]

 

 

keep

databaseCleanup.verbose

Make logging output more verbose.

True

 

 

keep

migrateDatabaseJob.affinity

Specify scheduling constraints for the migrate database job pod.

{}

 

 

keep

migrateDatabaseJob.annotations

Annotations to add to the migrate database job pod.

{}

 

 

keep

migrateDatabaseJob.applyCustomEnv

Specify if you want additional configured env vars applied to this job

True

 

 

keep

migrateDatabaseJob.args

Args to use when running migrate database job (templated).

['bash', '-c', 'exec \\\nairflow {{ semverCompare ">=2.7.0" .Values.airflowVersion | ternary "db migrate" (semverCompare ">=2.0.0" .Values.airflowVersion | ternary "db upgrade" "upgradedb") }}']

 

 

keep

migrateDatabaseJob.command

Command to use when running migrate database job (templated).

~

 

 

keep

migrateDatabaseJob.enabled

Enable migrate database job.

True

 

 

keep

migrateDatabaseJob.env

Add additional env vars to migrate database job.

[]

 

 

keep

migrateDatabaseJob.extraContainers

Launch additional containers for the migrate database job pod

[]

 

 

keep

migrateDatabaseJob.extraInitContainers

Add additional init containers into migrate database job (templated).

[]

 

 

keep

migrateDatabaseJob.extraVolumeMounts

Mount additional volumes into migrate database job

[]

 

 

keep

migrateDatabaseJob.extraVolumes

Mount additional volumes into migrate database job

[]

 

 

keep

migrateDatabaseJob.jobAnnotations

Annotations to add to the migrate database job.

{}

 

 

keep

migrateDatabaseJob.labels

Labels to add to the migrate database job objects and pods.

{}

 

 

keep

migrateDatabaseJob.nodeSelector

Select certain nodes for the migrate database job pod.

{}

 

 

keep

migrateDatabaseJob.priorityClassName

Specify priority for the migrate database job pod.

~

 

 

keep

migrateDatabaseJob.resources

Resources for the migrate database job pod

{}

Examples:

resources:
  limits:
    cpu: 100m
    memory: 128Mi
  requests:
    cpu: 100m
    memory: 128Mi

 

 

keep

migrateDatabaseJob.restartPolicy

Restart policy for the database migration job.

OnFailure

 

 

keep

migrateDatabaseJob.securityContext

Security context for the migrate database job pod (deprecated, use securityContexts instead). If not set, the values from securityContext will be used.

{}

Examples:

securityContext:
  fsGroup: 0
  runAsGroup: 0
  runAsUser: 50000

 

 

keep

migrateDatabaseJob.serviceAccount.annotations

Annotations to add to the migrate database job Kubernetes ServiceAccount.

{}

 

 

keep

migrateDatabaseJob.serviceAccount.automountServiceAccountToken

Specifies if ServiceAccount’s API credentials should be mounted onto Pods.

True

 

 

keep

migrateDatabaseJob.serviceAccount.create

Specifies whether a ServiceAccount should be created.

True

 

 

keep

migrateDatabaseJob.serviceAccount.name

The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name.

~

 

 

keep

migrateDatabaseJob.tolerations

Specify Tolerations for the migrate database job pod.

[]

 

 

keep

migrateDatabaseJob.topologySpreadConstraints

Specify topology spread constraints for migrate database job pod.

[]

 

 

keep

migrateDatabaseJob.ttlSecondsAfterFinished

Limit the lifetime of the job object after it finished execution

300

 

 

keep

migrateDatabaseJob.useHelmHooks

Specify if you want to use the default Helm Hook annotations

True

 

 

Modify the chart deployment logic in a way that this will not be needed in the sense that:

  1. Deployment with helm --wait flag will work by default
  2. The order of components deployment will be proper, e.g. api server should be one of the first components to be deployed and e.g. workers after api server

Kubernetes

Parameter

Description

Default

Opinion Jens

Opinion Bugra

Opinion Przemek

affinity

Specify scheduling constraints for all pods.

{}

 

 

keep

airflowConfigAnnotations

Extra annotations to apply to the main Airflow configmap.

{}

 

 

keep

airflowPodAnnotations

Extra annotations to apply to all Airflow pods.

{}

 

 

keep

apiServer.containerLifecycleHooks

Container Lifecycle Hooks definition for the API server. If not set, the values from global containerLifecycleHooks will be used.

{}

Examples:

containerLifecycleHooks:
  postStart:
    exec:
      command:
      - /bin/sh
      - -c
      - echo postStart handler > /usr/share/message
  preStop:
    exec:
      command:
      - /bin/sh
      - -c
      - echo preStop handler > /usr/share/message

 

 

keep

apiServer.revisionHistoryLimit

Number of old ReplicaSets to retain.

~

 

 

keep

apiServer.securityContexts.container

Container security context definition for the API server.

{}

Examples:

container:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL

 

 

keep

apiServer.securityContexts.pod

Pod security context definition for the API server.

{}

Examples:

pod:
  fsGroup: 0
  runAsGroup: 0
  runAsUser: 50000

 

 

keep

apiServer.topologySpreadConstraints

Specify topology spread constraints for API server pods.

[]

 

 

keep

apiServer.waitForMigrations.securityContexts.container

Container security context definition for the wait for migrations.

{}

Examples:

container:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL

 

 

Related comment under dagProcessor section next to waitForMigration section

cleanup.containerLifecycleHooks

Container Lifecycle Hooks definition for the cleanup. If not set, the values from global containerLifecycleHooks will be used.

{}

Examples:

containerLifecycleHooks:
  postStart:
    exec:
      command:
      - /bin/sh
      - -c
      - echo postStart handler > /usr/share/message
  preStop:
    exec:
      command:
      - /bin/sh
      - -c
      - echo preStop handler > /usr/share/message

 

 

keep

cleanup.failedJobsHistoryLimit

The failed jobs history limit specifies the number of failed jobs to retain.

~

 

 

keep

cleanup.securityContexts.container

Container security context definition for the cleanup.

{}

Examples:

container:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL

 

 

keep

cleanup.securityContexts.pod

Pod security context definition for the cleanup.

{}

Examples:

pod:
  fsGroup: 0
  runAsGroup: 0
  runAsUser: 50000

 

 

keep

cleanup.successfulJobsHistoryLimit

The successful jobs history limit specifies the number of finished jobs to retain.

~

 

 

keep

containerLifecycleHooks

Default Container Lifecycle Hooks definition. The values in this parameter will be used when containerLifecycleHooks is not defined for specific containers.

{}

Examples:

containerLifecycleHooks:
  postStart:
    exec:
      command:
      - /bin/sh
      - -c
      - echo postStart handler > /usr/share/message
  preStop:
    exec:
      command:
      - /bin/sh
      - -c
      - echo preStop handler > /usr/share/message

 

 

keep

createUserJob.containerLifecycleHooks

Container Lifecycle Hooks definition for the create user job. If not set, the values from global containerLifecycleHooks will be used.

{}

Examples:

containerLifecycleHooks:
  postStart:
    exec:
      command:
      - /bin/sh
      - -c
      - echo postStart handler > /usr/share/message
  preStop:
    exec:
      command:
      - /bin/sh
      - -c
      - echo preStop handler > /usr/share/message

 

 

Related comment under jobs section next to createUserJob section

createUserJob.securityContexts.container

Container security context definition for the create user job.

{}

Examples:

container:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL

 

 

createUserJob.securityContexts.pod

Pod security context definition for the create user job.

{}

Examples:

pod:
  fsGroup: 0
  runAsGroup: 0
  runAsUser: 50000

 

 

dagProcessor.containerLifecycleHooks

Container Lifecycle Hooks definition for the dag processor. If not set, the values from global containerLifecycleHooks will be used.

{}

Examples:

containerLifecycleHooks:
  postStart:
    exec:
      command:
      - /bin/sh
      - -c
      - echo postStart handler > /usr/share/message
  preStop:
    exec:
      command:
      - /bin/sh
      - -c
      - echo preStop handler > /usr/share/message

 

 

keep

dagProcessor.securityContexts.container

Container security context definition for the dag processor.

{}

Examples:

container:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL

 

 

keep

dagProcessor.securityContexts.pod

Pod security context definition for the dag processor.

{}

Examples:

pod:
  fsGroup: 0
  runAsGroup: 0
  runAsUser: 50000

 

 

keep

dagProcessor.waitForMigrations.securityContexts.container

Container security context definition for the wait for migrations.

{}

Examples:

container:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL

 

 

Related comment under dagProcessor section next to waitForMigration section

dags.gitSync.containerLifecycleHooks

Container Lifecycle Hooks definition for the git sync sidecar. If not set, the values from global containerLifecycleHooks will be used.

{}

Examples:

containerLifecycleHooks:
  postStart:
    exec:
      command:
      - /bin/sh
      - -c
      - echo postStart handler > /usr/share/message
  preStop:
    exec:
      command:
      - /bin/sh
      - -c
      - echo preStop handler > /usr/share/message

 

gitSync related, could be useful to tag if we do bundle or deprecate


 

Related comment next to other dags.gitSync fields

dags.gitSync.securityContexts.container

Container security context definition for the git sync sidecar.

{}

Examples:

container:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL

 

databaseCleanup.containerLifecycleHooks

Container Lifecycle Hooks definition for the database cleanup. If not set, the values from global containerLifecycleHooks will be used.

{}

Examples:

containerLifecycleHooks:
  postStart:
    exec:
      command:
      - /bin/sh
      - -c
      - echo postStart handler > /usr/share/message
  preStop:
    exec:
      command:
      - /bin/sh
      - -c
      - echo preStop handler > /usr/share/message

 

 

keep

databaseCleanup.failedJobsHistoryLimit

The failed jobs history limit specifies the number of failed jobs to retain.

1

 

 

keep

databaseCleanup.securityContexts.container

Container security context definition for the database cleanup.

{}

Examples:

container:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL

 

 

keep

databaseCleanup.securityContexts.pod

Pod security context definition for the database cleanup.

{}

Examples:

pod:
  fsGroup: 0
  runAsGroup: 0
  runAsUser: 50000

 

 

keep

databaseCleanup.successfulJobsHistoryLimit

The successful jobs history limit specifies the number of finished jobs to retain.

1

 

 

keep

extraConfigMaps

Extra ConfigMaps that will be managed by the chart.

{}

Examples:

extraConfigMaps:
  '{{ .Release.Name }}-airflow-variables':
    data: |-
      AIRFLOW_VAR_HELLO_MESSAGE: 'Hi!'
      AIRFLOW_VAR_KUBERNETES_NAMESPACE: '{{ .Release.Namespace }}'

 

 

keep

extraSecrets

Extra secrets that will be managed by the chart.

{}

Examples:

extraSecrets:
  '{{ .Release.Name }}-airflow-connections':
    data: |-
      AIRFLOW_CONN_GCP: 'base64_encoded_gcp_conn_string'
      AIRFLOW_CONN_AWS: 'base64_encoded_aws_conn_string'
    stringData: 'AIRFLOW_CONN_OTHER: ''other_conn'''

 

 

keep

flower.containerLifecycleHooks

Container Lifecycle Hooks definition for the network policy. If not set, the values from global containerLifecycleHooks will be used.

{}

Examples:

containerLifecycleHooks:
  postStart:
    exec:
      command:
      - /bin/sh
      - -c
      - echo postStart handler > /usr/share/message
  preStop:
    exec:
      command:
      - /bin/sh
      - -c
      - echo preStop handler > /usr/share/message

 

 

 

Flower settings, above comment related 

Related comment under flower section

flower.revisionHistoryLimit

Number of old ReplicaSets to retain.

~

 

flower.securityContexts.container

Container security context definition for the network policy.

{}

Examples:

container:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL

 

flower.securityContexts.pod

Pod security context definition for the network policy.

{}

Examples:

pod:
  fsGroup: 0
  runAsGroup: 0
  runAsUser: 50000

 

imagePullSecrets

List of existing Kubernetes secrets containing Base64 encoded credentials to connect to private registries (will get passed to imagePullSecrets).

[]

 

 

keep

labels

Add common labels to all objects and pods defined in this chart.

{}

 

 

keep

limits

Define default/max/min values for pods and containers in namespace.

[]

 

 

keep

migrateDatabaseJob.containerLifecycleHooks

Container Lifecycle Hooks definition for the migrate database job. If not set, the values from global containerLifecycleHooks will be used.

{}

Examples:

containerLifecycleHooks:
  postStart:
    exec:
      command:
      - /bin/sh
      - -c
      - echo postStart handler > /usr/share/message
  preStop:
    exec:
      command:
      - /bin/sh
      - -c
      - echo preStop handler > /usr/share/message

 

 

Related comment under jobs section next to migrateDatabaseJob other fields

migrateDatabaseJob.securityContexts.container

Container security context definition for the migrate database job.

{}

Examples:

container:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL

 

 

migrateDatabaseJob.securityContexts.pod

Pod security context definition for the migrate database job.

{}

Examples:

pod:
  fsGroup: 0
  runAsGroup: 0
  runAsUser: 50000

 

 

networkPolicies.enabled

Enabled network policies.

False

 

 

keep

nodeSelector

Select certain nodes for all pods.

{}

 

 

keep

pgbouncer.containerLifecycleHooks

Container Lifecycle Hooks definition for the PgBouncer. If not set, the values from global containerLifecycleHooks will be used.

{'preStop': {'exec': {'command': ['/bin/sh', '-c', 'killall -INT pgbouncer && sleep 120']}}}

Examples:

containerLifecycleHooks:
  postStart:
    exec:
      command:
      - /bin/sh
      - -c
      - echo postStart handler > /usr/share/message
  preStop:
    exec:
      command:
      - /bin/sh
      - -c
      - echo preStop handler > /usr/share/message

 

 

Related comment under PgBouncer section

pgbouncer.metricsExporterSidecar.containerLifecycleHooks

Container Lifecycle Hooks definition for the metrics exporter sidecar. If not set, the values from global containerLifecycleHooks will be used.

{}

Examples:

containerLifecycleHooks:
  postStart:
    exec:
      command:
      - /bin/sh
      - -c
      - echo postStart handler > /usr/share/message
  preStop:
    exec:
      command:
      - /bin/sh
      - -c
      - echo preStop handler > /usr/share/message

 

 

pgbouncer.metricsExporterSidecar.securityContexts.container

Container security context definition for the metrics exporter sidecar.

{}

Examples:

container:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL

 

 

pgbouncer.revisionHistoryLimit

Number of old ReplicaSets to retain.

~

 

 

pgbouncer.securityContexts.container

Container security context definition for the PgBouncer.

{}

Examples:

container:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL

 

 

pgbouncer.securityContexts.pod

Pod security context definition for the PgBouncer.

{}

Examples:

pod:
  fsGroup: 0
  runAsGroup: 0
  runAsUser: 65534

 

 

priorityClasses

Priority Classes created by helm charts

[]

Examples:

priorityClasses:
- name: class1
  preemptionPolicy: PreemptLowerPriority
  value: 10000
priorityClasses:
- name: class2
  preemptionPolicy: Never
  value: 100000

 

 

keep

quotas

Define any ResourceQuotas for namespace.

{}

 

 

keep

rbac.create

Specifies whether RBAC resources should be created.

True

 

 

keep

rbac.createSCCRoleBinding

Specifies whether SCC RoleBinding resource should be created (refer to Production Guide).

False

 

 

keep

redis.containerLifecycleHooks

Container Lifecycle Hooks definition for the redis. If not set, the values from global containerLifecycleHooks will be used.

{}

Examples:

containerLifecycleHooks:
  postStart:
    exec:
      command:
      - /bin/sh
      - -c
      - echo postStart handler > /usr/share/message
  preStop:
    exec:
      command:
      - /bin/sh
      - -c
      - echo preStop handler > /usr/share/message

 

 

Related comment under redis section

redis.securityContexts.container

Container security context definition for the redis.

{}

Examples:

container:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL

 

 

redis.securityContexts.pod

Pod security context definition for the redis.

{}

Examples:

pod:
  fsGroup: 0
  runAsGroup: 0
  runAsUser: 999

 

 

registry.connection

Credentials to connect to a private registry, these will get Base64 encoded and stored in a secret (will get passed to imagePullSecrets) (create manually the credentials secret and add to imagePullSecrets instead).

{}

Examples:

connection:
  email: '...'
  host: '...'
  pass: '...'
  user: '...'

 

 

keep

registry.connection.email

Email Address

""

 

 

keep

registry.connection.host

Registry Server URL (e.g. https://index.docker.io/v1/ for DockerHub)

""

 

 

keep

registry.connection.pass

Password

""

 

 

keep

registry.connection.user

Username

""

 

 

keep

registry.secretName

Name of the Kubernetes secret containing Base64 encoded credentials to connect to a private registry (will get passed to imagePullSecrets) (Deprecated - renamed to registry.secretNames).

~

 

 

registry.secretNames is not used in chart

revisionHistoryLimit

Global number of old ReplicaSets to retain. Can be overridden by each deployment’s revisionHistoryLimit

~

 

 

keep

scheduler.containerLifecycleHooks

Container Lifecycle Hooks definition for the scheduler. If not set, the values from global containerLifecycleHooks will be used.

{}

Examples:

containerLifecycleHooks:
  postStart:
    exec:
      command:
      - /bin/sh
      - -c
      - echo postStart handler > /usr/share/message
  preStop:
    exec:
      command:
      - /bin/sh
      - -c
      - echo preStop handler > /usr/share/message

 

 

keep

scheduler.revisionHistoryLimit

Number of old ReplicaSets to retain.

~

 

 

keep

scheduler.securityContexts.container

Container security context definition for the scheduler.

{}

Examples:

container:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL

 

 

keep

scheduler.securityContexts.pod

Pod security context definition for the scheduler.

{}

Examples:

pod:
  fsGroup: 0
  runAsGroup: 0
  runAsUser: 50000

 

 

keep

scheduler.waitForMigrations.securityContexts.container

Container security context definition for the wait for migrations.

{}

Examples:

container:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL

 

 

Related comment under dagProcessor section next to waitForMigration section

securityContext

Default pod security context definition (deprecated, use securityContexts instead). The values in this parameter will be used when securityContext is not defined for specific Pods

{}

Examples:

securityContext:
  fsGroup: 0
  runAsGroup: 0
  runAsUser: 50000

 

 

Drop

securityContexts.containers

Default container security context definition. The values in this parameter will be used when securityContexts is not defined for specific containers

{}

Examples:

containers:
  allowPrivilegeEscalation: false

 

 

keep

securityContexts.pod

Default pod security context definition. The values in this parameter will be used when securityContexts is not defined for specific Pods.

{}

Examples:

pod:
  fsGroup: 0
  runAsGroup: 0
  runAsUser: 50000

 

 

keep

statsd.containerLifecycleHooks

Container Lifecycle Hooks definition for the statsd. If not set, the values from global containerLifecycleHooks will be used.

{}

Examples:

containerLifecycleHooks:
  postStart:
    exec:
      command:
      - /bin/sh
      - -c
      - echo postStart handler > /usr/share/message
  preStop:
    exec:
      command:
      - /bin/sh
      - -c
      - echo preStop handler > /usr/share/message

 

 

Related comment under statsd section

statsd.revisionHistoryLimit

Number of old ReplicaSets to retain.

~

 

 

statsd.securityContexts.container

Container security context definition for the statsd.

{}

Examples:

container:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL

 

 

statsd.securityContexts.pod

Pod security context definition for the statsd.

{}

Examples:

pod:
  fsGroup: 0
  runAsGroup: 0
  runAsUser: 50000

 

 

tolerations

Specify Tolerations for all pods.

[]

 

 

keep

topologySpreadConstraints

Specify topology spread constraints for all pods.

[]

 

 

keep

triggerer.containerLifecycleHooks

Container Lifecycle Hooks definition for the triggerer. If not set, the values from global containerLifecycleHooks will be used.

{}

Examples:

containerLifecycleHooks:
  postStart:
    exec:
      command:
      - /bin/sh
      - -c
      - echo postStart handler > /usr/share/message
  preStop:
    exec:
      command:
      - /bin/sh
      - -c
      - echo preStop handler > /usr/share/message

 

 

keep

triggerer.revisionHistoryLimit

Number of old ReplicaSets to retain.

~

 

 

keep

triggerer.securityContexts.container

Container security context definition for the triggerer.

{}

Examples:

container:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL

 

 

keep

triggerer.securityContexts.pod

Pod security context definition for the triggerer.

{}

Examples:

pod:
  fsGroup: 0
  runAsGroup: 0
  runAsUser: 50000

 

 

keep

triggerer.waitForMigrations.securityContexts.container

Container security context definition for the wait for migrations.

{}

Examples:

container:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL

 

 

Related comment under dagProcessor section next to waitForMigration section

webserver.containerLifecycleHooks

Container Lifecycle Hooks definition for the webserver. If not set, the values from global containerLifecycleHooks will be used.

{}

Examples:

containerLifecycleHooks:
  postStart:
    exec:
      command:
      - /bin/sh
      - -c
      - echo postStart handler > /usr/share/message
  preStop:
    exec:
      command:
      - /bin/sh
      - -c
      - echo preStop handler > /usr/share/message

 

 

 

 

 

 

We should either remove these or convert to an API server but I assume just removing is the way to go


 

 

 

 

Drop with Airflow 2

webserver.securityContexts.container

Container security context definition for the webserver.

{}

Examples:

container:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL

 

webserver.securityContexts.pod

Pod security context definition for the webserver.

{}

Examples:

pod:
  fsGroup: 0
  runAsGroup: 0
  runAsUser: 50000

 

webserver.topologySpreadConstraints

Specify topology spread constraints for webserver pods.

[]

 

webserver.waitForMigrations.securityContexts.container

Container security context definition for the wait for migrations.

{}

Examples:

container:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL

 

workers.celery.kerberosInitContainer.containerLifecycleHooks

Container Lifecycle Hooks definition for the kerberos init container. If not set, the values from workers.kerberosInitContainer.containerLifecycleHooks will be used.

{}

Examples:

containerLifecycleHooks:
  postStart:
    exec:
      command:
      - /bin/sh
      - -c
      - echo postStart handler > /usr/share/message
  preStop:
    exec:
      command:
      - /bin/sh
      - -c
      - echo preStop handler > /usr/share/message

 

Kerberos, if we move to Kustomize


 

Related comment under kerberos section

workers.celery.kerberosInitContainer.securityContexts.container

Container security context definition for the kerberos init container.

{}

Examples:

container:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL

 

workers.celery.persistence.securityContexts.container

Container security context definition for the persistence.

{}

Examples:

container:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL

 

 

keep

workers.celery.securityContexts.container

Container security context definition.

{}

Examples:

container:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL

 

 

keep

workers.celery.securityContexts.pod

Pod security context definition.

{}

Examples:

pod:
  fsGroup: 0
  runAsGroup: 0
  runAsUser: 50000

 

 

keep

workers.containerLifecycleHooks

Container Lifecycle Hooks definition for Airflow Celery workers and pods created with pod-template-file. If not set, the values from global containerLifecycleHooks will be used.

{}

Examples:

containerLifecycleHooks:
  postStart:
    exec:
      command:
      - /bin/sh
      - -c
      - echo postStart handler > /usr/share/message
  preStop:
    exec:
      command:
      - /bin/sh
      - -c
      - echo preStop handler > /usr/share/message

 

 

Move to proper workers.celery/workers.kubernetes sections

workers.kerberosInitContainer.containerLifecycleHooks

Container Lifecycle Hooks definition for the kerberos init container. If not set, the values from global containerLifecycleHooks will be used.

{}

Examples:

containerLifecycleHooks:
  postStart:
    exec:
      command:
      - /bin/sh
      - -c
      - echo postStart handler > /usr/share/message
  preStop:
    exec:
      command:
      - /bin/sh
      - -c
      - echo preStop handler > /usr/share/message

 

 

 

 

 

 

Kerberos, if we move Kustomize. 

Related comment under kerberos section

workers.kerberosInitContainer.securityContexts.container

Container security context definition for the kerberos init container.

{}

Examples:

container:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL

 

workers.kerberosSidecar.containerLifecycleHooks

Container Lifecycle Hooks definition for the kerberos sidecar. If not set, the values from global containerLifecycleHooks will be used.

{}

Examples:

containerLifecycleHooks:
  postStart:
    exec:
      command:
      - /bin/sh
      - -c
      - echo postStart handler > /usr/share/message
  preStop:
    exec:
      command:
      - /bin/sh
      - -c
      - echo preStop handler > /usr/share/message

 

workers.kerberosSidecar.securityContexts.container

Container security context definition for the kerberos sidecar.

{}

Examples:

container:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL

 

workers.kubernetes.kerberosInitContainer.containerLifecycleHooks

Container Lifecycle Hooks definition for the kerberos init container. If not set, the values from workers.kerberosInitContainer.containerLifecycleHooks will be used.

{}

Examples:

containerLifecycleHooks:
  postStart:
    exec:
      command:
      - /bin/sh
      - -c
      - echo postStart handler > /usr/share/message
  preStop:
    exec:
      command:
      - /bin/sh
      - -c
      - echo preStop handler > /usr/share/message

 

workers.kubernetes.kerberosInitContainer.securityContexts.container

Container security context definition for the kerberos init container.

{}

Examples:

container:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL

 

workers.kubernetes.securityContexts.container

Container security context definition.

{}

Examples:

container:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL

 

 

keep

workers.kubernetes.securityContexts.pod

Pod security context definition.

{}

Examples:

pod:
  fsGroup: 0
  runAsGroup: 0
  runAsUser: 50000

 

 

keep

workers.persistence.securityContexts.container

Container security context definition for the persistence.

{}

Examples:

container:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL

 

 

Move to proper workers.celery/workers.kubernetes sections

workers.revisionHistoryLimit

Max number of old Airflow Celery workers ReplicaSets to retain.

~

 

 

workers.securityContexts.container

Container security context definition.

{}

Examples:

container:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL

 

 

workers.securityContexts.pod

Pod security context definition.

{}

Examples:

pod:
  fsGroup: 0
  runAsGroup: 0
  runAsUser: 50000

 

 

workers.waitForMigrations.securityContexts.container

Container security context definition for the wait-for-airflow-migrations container.

{}

Examples:

container:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL

 

 

Related comment under dagProcessor section next to waitForMigration section

Ingress

Parameter

Description

Default

Opinion Jens

Opinion Jens

Opinion Przemek

ingress.apiServer.annotations

Annotations for the API server Ingress.

{}

 

 

keep

ingress.apiServer.enabled

Enable API server ingress resource.

False

 

 

keep

ingress.apiServer.host

The hostname for the API server Ingress. (Deprecated - renamed to ingress.apiServer.hosts)

""

 

 

drop

ingress.apiServer.hosts

The hostnames or hosts configuration for the API server Ingress.

[]

 

 

keep

ingress.apiServer.ingressClassName

The Ingress Class for the API server Ingress.

""

 

 

keep

ingress.apiServer.path

The path for the API server Ingress.

/

 

 

keep

ingress.apiServer.pathType

The pathType for the API server Ingress (required for Kubernetes 1.19 and above).

ImplementationSpecific

 

 

keep

ingress.apiServer.precedingPaths

HTTP paths to add to the API server Ingress before the default path.

[]

 

 

keep

ingress.apiServer.succeedingPaths

HTTP paths to add to the API server Ingress after the default path.

[]

 

 

keep

ingress.apiServer.tls.enabled

Enable TLS termination for the API server Ingress.

False

 

 

drop - missing deprecation info

ingress.apiServer.tls.secretName

The name of a pre-created Secret containing a TLS private key and certificate.

""

 

 

drop - missing deprecation info

ingress.enabled

Enable all ingress resources (deprecated - use ingress.web.enabled and ingress.flower.enabled).

~

 

 

drop

ingress.flower.annotations

Annotations for the flower Ingress.

{}

 

 

Move to Kustomize

ingress.flower.enabled

Enable flower ingress resource.

False

 

 

ingress.flower.host

The hostname for the flower Ingress. (Deprecated - renamed to ingress.flower.hosts)

""

 

 

ingress.flower.hosts

The hostnames or hosts configuration for the flower Ingress.

[]

 

 

ingress.flower.ingressClassName

The Ingress Class for the flower Ingress.

""

 

 

ingress.flower.path

The path for the flower Ingress.

/

 

 

ingress.flower.pathType

The pathType for the flower Ingress (required for Kubernetes 1.19 and above).

ImplementationSpecific

 

 

ingress.flower.tls.enabled

Enable TLS termination for the flower Ingress.

False

 

 

ingress.flower.tls.secretName

The name of a pre-created Secret containing a TLS private key and certificate.

""

 

 

ingress.pgbouncer.annotations

Annotations for the PgBouncer Ingress.

{}

Why should pgbouncer have an ingress?

These are all deployed into the same cluster.

So there is no reason to have.

It is even a possible open door.


 

 

 

 

 

 

Related comment under PgBouncer section.

Ingress needed for setups where components are separated from each other, looking at different setups (basically, all enabled flags allow for that)

ingress.pgbouncer.enabled

Enable PgBouncer ingress resource.

False

 

ingress.pgbouncer.host

The hostname for the PgBouncer Ingress. (Deprecated - renamed to ingress.pgbouncer.hosts)

""

 

ingress.pgbouncer.hosts

The hostnames or hosts configuration for the PgBouncer Ingress.

[]

 

ingress.pgbouncer.ingressClassName

The Ingress Class for the PgBouncer Ingress.

""

 

ingress.pgbouncer.path

The path for the PgBouncer Ingress.

/metrics

 

ingress.pgbouncer.pathType

The pathType for the PgBouncer Ingress (required for Kubernetes 1.19 and above).

ImplementationSpecific

 

ingress.statsd.annotations

Annotations for the statsd Ingress.

{}

Why should statsd have an ingress?

This could make sense if the deployed logging tool is outside of the cluster,

I would understand if they need to reach out to statsd


 

 

 

 

 

 

Move to Kustomize.

Ingress needs for setups where Observability tools gather metrics from the endpoint instead of having dedicated agents inside the Kubernetes cluster

ingress.statsd.enabled

Enable statsd ingress resource.

False

 

ingress.statsd.host

The hostname for the statsd Ingress. (Deprecated - renamed to ingress.statsd.hosts)

""

 

ingress.statsd.hosts

The hostnames or hosts configuration for the statsd Ingress.

[]

 

ingress.statsd.ingressClassName

The Ingress Class for the statsd Ingress.

""

 

ingress.statsd.path

The path for the statsd Ingress.

/metrics

 

ingress.statsd.pathType

The pathType for the statsd Ingress (required for Kubernetes 1.19 and above).

ImplementationSpecific

 

ingress.web.annotations

Annotations for the web Ingress.

{}

Will be removed with deprecation of Airflow 2

 

 

 

 

 

 

 

 

 

 

Yes, API server separatly defined.

Drop with Airflow 2

ingress.web.enabled

Enable web ingress resource.

False

 

ingress.web.host

The hostname for the web Ingress. (Deprecated - renamed to ingress.web.hosts)

""

 

ingress.web.hosts

The hostnames or hosts configuration for the web Ingress.

[]

 

ingress.web.ingressClassName

The Ingress Class for the web Ingress.

""

 

ingress.web.path

The path for the web Ingress.

/

 

ingress.web.pathType

The pathType for the web Ingress (required for Kubernetes 1.19 and above).

ImplementationSpecific

 

ingress.web.precedingPaths

HTTP paths to add to the web Ingress before the default path.

[]

 

ingress.web.succeedingPaths

HTTP paths to add to the web Ingress after the default path.

[]

 

ingress.web.tls.enabled

Enable TLS termination for the web Ingress.

False

 

ingress.web.tls.secretName

The name of a pre-created Secret containing a TLS private key and certificate.

""

 

Kerberos

Parameter

Description

Default

Opinion Jens

Opinion Bugra

Opinion Przemek

kerberos.ccacheFileName

Name for kerberos credentials cache file.

cache

 

 

 

 

 

 

 

 

 

We should move these to Kustomize

If the above comments (under Workers section) also agreed?

Move to Kustomize

kerberos.ccacheMountPath

Path to mount shared volume for kerberos credentials cache.

/var/kerberos-ccache

 

kerberos.config

Contents of krb5.conf.

See values.yaml

 

kerberos.configPath

Path to mount krb5.conf kerberos configuration file.

/etc/krb5.conf

 

kerberos.enabled

Enable kerberos.

False

 

kerberos.keytabBase64Content

Kerberos keytab base64 encoded content.

~

 

kerberos.keytabPath

Path to mount the keytab for refreshing credentials in the kerberos sidecar.

/etc/airflow.keytab

 

kerberos.principal

Principal to use when refreshing kerberos credentials.

airflow@FOO.COM

 

kerberos.reinitFrequency

How often (in minutes) airflow kerberos will reinitialize the credentials cache.

3600

 

  • No labels