Authored by George Ke with Maxime Beauchemin on 2016-07-26

Overview

We want to add the ability to impersonate a unix user for a subprocess run.

Motivation

We need feature parity with Chronos tasks for the upcoming Chronos migration at Airbnb.

Currently, all tasks are run as the same `airflow` user on a given worker. This makes it impossible to log who ran which task. With impersonation, commands run by operators can be ran as and logged as the user who wrote the task.

Security is also a concern, as right now all subprocesses have the same privileges as the parent process. Although we are not specifically tackling security, we want to make sure we don't regress or leave blatant security holes.

Additionally, we want to keep in mind a long-term goal of having a system that manages secrets that Airflow and its tasks need. We want to have the config & secrets that Airflow needs to be completely separate from the config & secrets operators need. Ideally, we would have a vault that manages secrets and provides what's needed to a user with a lease attached to it.  
 

Solutions

There have been many solutions proposed, listed below in chronological order. 
Solution 1: Simple
Sharing `airflow.cfg` and logging files with the user, (Link to commit)
  • Overview:
  • Running the subtask command (`airflow run --raw`) with `sudo -u`
  • user must be able to read `airflow.cfg` and be able to write to logging files
  • Issues:
  • Security concerns will allowing all users to read `airflow.cfg`
  • `airflow.cfg` might be incomplete or not exist as we allow env vars as configuration
 
Solution 2: Pickles
Passing in a pickled subset of `airflow.cfg` to `airflow run --raw` (Link to commit)
  • Overview:
  • The pickled subset config is passed as a command line argument
  • You can also specify a custom logging directory on the operator level if the default logging files are not writable
  • Issues:
  • The `airflow run --raw` command will have secrets it, which can be seen with `ps`, in the syslog, as well as in the bash history
Solution 3: Forking
Forking the parent process and calling Python's `os.setuid` to change the user, (Link to commit)
  • Overview:
  • A subset of `airflow.cfg` is passed into the forked process
  • Only unix user is changed, so certain operators that use other indicators (e.g. smtp or hiveserver2) other than unix user for impersonation would implement them at an operator or hook level
  • In terms of logging, the parent can pass file descriptors to the child that it can write to
  • Advantages:
  • Subset of configs is safely passed to the subprocess without going through the command line
  • A clean blanket solution that has a clear guarantee: that the Unix user will be set to the user passed in for the duration of the subprocess run
  • Concerns:
  • Complexity/nuances with forked processes (e.g. shared file descriptors between parent and child)
  • Security concerns. Because the forked process has the same memory as the parent, it has access to the `airflow.cfg` that's been loaded in memory.  A bad python operator could read these secrets and/or dump them somewhere that the user can read. This makes code review as the gateway for enforcing security.
Solution 4: Operators
Operator and Hook level impersonation
  • Overview:
  • Add a `run_as_user` parameter on the `BaseHook` 
  • For each operator and associated hook that runs commands, we use `run_as_user` to do impersonation for each command in the operator/hook
  • e.g. `sudo -u` for bash operator
  • Advantages:
  • No secrets need to be passed to the subprocess as it has the same privileges as the parent. It's still secure as the commands that run in the operator are ran as the user
  • Explicitly define what operators are used
  • Hooks to APIs where impersonation isn't inferred from the unix username need this type of implementation regardless of the higher level solution.
  • Concerns:
  • Making changes to all operators/hook with commands could be unwieldy and is specific to each operator/hook
  • Things could be missed due to the large number of things being changed
  • What the `run_as_user` parameter does could be foggy as operators do different things with it. There are not clear guarantees across operators.
Solution 5: Custom File
Parent process creates a temporary custom config file for the child to read.
  • Overview:
  • `airflow run --raw` for the child will be prepended with `sudo -H -u`
  • The custom file will be a subset of `airflow.cfg` and only readable by the user. It will be deleted after the subprocess run.
  • Advantages:
  • Better security than pickling. There will be no trace of secrets left in logs.
  • Allows for an easier upgrade to using a vault of secrets when we decide to tackle that task (the custom file can be replaced with a call to the vault) as we are adding a codepath to isolate airflow secrets with operator secrets
  • We're able to serialize the configuration object and get configuration from any source (env vars, ...)
  • Concerns:
  • You can ssh into the machine and view the temp file
  • Retort: allowing users to ssh into a machine is already insecure from a pure security standpoint. Also, you can create users on a machine that you cannot ssh as.
      • Also you can still view the secrets in the fork model (e.g. just have your task print the secrets to the log), so I wouldn't say this is a concern, it's actually more secure than forking the way I see it (since the task is run in complete isolation and memory space than the parent task).

Other Considerations

  1. Right now all airflow boxes are run as `airflow` user. To do impersonation, we need to run as a `sudo`er. This means that all existing jobs will be run as `sudo`er instead of `airflow`. This avoid this, we can allow the scheduler to be run as a `sudo`er but lower down to `airflow` before we run any tasks. This may either require a default "lower user" in `airflow.cfg` or having all operators set `unix_impersonate` so that it they don't run as `sudo`.

Conclusion

We are strongly leaning towards Solution 5 as it solves our following needs: 

We want clear guarantees for unix impersonation, implemented at the BaseOperator level and baked into the framework.  The guarantee we want is simple: when using the `BaseOperator.unix_impersonate` (var name may change) the operator's execute method and hooks of a task instance will be executed as that user. Solution 5 is a blanket solution that fulfills this guarantee.

On top of that, individual hooks and operators that require impersonation beyond the Unix user itself should expose a `run_as` parameter with custom logic.
For instance someone maybe want the HiveOperator to run as the unix user, in which case `BaseOperator.unix_impersonate` would be applied. In another environment, someone may want to use the `HiveOperator.run_as` which essentially prepends the HQL command with a `SET proxy.user = {run_as}`. Both scenarios are legitimate, people may want either, and both will be possible with Solution 5.
  • No labels

6 Comments

  1. Bolke de Bruin

    Jul 27, 2016

    I like #5 as well as it allows follow up options to be developed: vault / api or even further using the config file as a means to transfer info (this is a sort of poor mans IPC btw, which can also be an option, this removes concerns about being able to read a config by a user). The downside is additional file io and the requirement to clean up those temporary files (and create them securely). 

    For me the API seems to most secure way to go as it allows you to disallow access, but creating a file with just the relevant information per task might be better from a reliability point of view (API does not have to be up when the task is running). 

     

    On your option #3 (forking) I disagree with the issues around memory - just unset the variables and file descriptors (configurable). 

    1. Maxime Beauchemin

      Jul 27, 2016

      Right, security-wise #3 it probably safer than other options (no files shuffling). The real concerns would be subtleties around the way Python manages forking and how that can be tricky to understand and debug. I'm not sure how concerning that is, but our gut feeling was to steer away from this approach.

      Out of scope for this effort, but next steps around security would be the vault/api integration, and semantics around what individual hooks need from the config/connections/vault and have the parent process prepare only that.

    2. Dan Davydov

      Jul 27, 2016

      Agreed with Bolke re. vault/api, I also think that the endgame is to completely remove the need for a vault in airflow, storing secrets like database credentials should be completely decoupled from airflow, and airflow should not have to pass workers any information. That being said we may want to keep the current config-passing model for the base setup for user convenience.

    3. Dan Davydov

      Jul 27, 2016

      Also note that the temporary files will not need to be cleaned up in normal circumstances (as they will be created via context manager): https://docs.python.org/3/library/tempfile.html , though you are right in that there are still cases where these files could persist.

  2. Bolke de Bruin

    Jul 27, 2016

    From a security point of view I like #5 better actually as it decouples the different parts of airflow. We could also opt for a hybrid solution: let the task wait for some config passed over stdin. 

    from a services security point of view I like Kerberos / oauth2 / secret key integration to make sure each part of airflow proves that it is part of the setup. But that is obviously out of scope. 

  3. Bolke de Bruin

    Jul 27, 2016

    And we can still do the "vault" things with the config file. Instead that an hook/operator accesses the db to find its connections pass the connection credentials in the temporary config file.