Delete comments <a href="http://www.aptis.info/portfolio/delete-all-comments/" target="_blank" title="Help"> <span class="aui-icon aui-icon-small aui-iconfont-help aptis-delete-all-comments-title-icon">Help</span> </a>

ash tfeng would be great if I can gather some feedbacks from you first before I...

Chao-Han Tsai | Mar 22, 2019

Don't most cloud services and similar offer the ability to mount a shared files...

Ezra Epstein | Mar 22, 2019

> Don't most cloud services and similar offer the ability to mount a shared fil...

Chao-Han Tsai | Mar 23, 2019

Not all cloud services support an easy way to mount things as a shared filesyst...

Ian Davison | Mar 26, 2019

I faced the issue when we moved to k8s and there is no One-to-Many easy solutio...

Valeriys Soloviov | Aug 13, 2019

Most data centers - self hosted - have NAS, so it's a non-issue. For others, t...

Ezra Epstein | Mar 25, 2019

Does this assume that airflow is deployed along with the dags?

Sergio Kef | Mar 26, 2019

It doesn't assume that. Both the model of deploying together and deploying the...

Ezra Epstein | Mar 26, 2019

I do agree that the DAG deployment pipeline is not complex. But I think if Airf...

Chao-Han Tsai | Mar 26, 2019

I like the idea, I think this goes the right direction. I would also add that ...

Sergio Kef | Mar 26, 2019

I think the idea of coupling a great workflow tool (Airflow) with one particula...

Ezra Epstein | Mar 26, 2019

This isn't coupling Airflow to any sort of deployment approach, the URIs can be...

Ian Davison | Mar 26, 2019

Airflow would not be limited to one particular approach of deployment. Airflow ...

Chao-Han Tsai | Mar 27, 2019

The big issues that need to be overcome with a remote DAG fetcher is how to val...

Ian Davison | Mar 26, 2019

I am proposing that we register a list of URI to the root directory of remote r...

Chao-Han Tsai | Mar 27, 2019

This approach will cause problems if you have a very large DAG repository and y...

Ian Davison | Mar 27, 2019

I see. ianjdavison can you tell me a bit more about how k8s executor get DAGs c...

Chao-Han Tsai | Mar 27, 2019

Looking at the code it seems like it pulls the code from git currently before i...

Chao-Han Tsai | Mar 27, 2019

ianjdavison I would prefer bringing in the Remote DAG fetching and DAG manifes...

Chao-Han Tsai | Mar 28, 2019

For the k8s executor for the task pods people will supply DAGs in one of three ...

Ian Davison | Mar 29, 2019

> On of the hopes is that with a DAG fetcher and a manifest for the k8s executo...

Chao-Han Tsai | Mar 29, 2019

PVC = persistent volume claim.

Ash Berlin-Taylor | May 02, 2019

tfeng ash kaxilnaik xd-deng fokko Would be great if I can get some feedbacks ...

Chao-Han Tsai | Mar 28, 2019

milton0825 , sorry for the delay, just skim through the AIP. Here are my questi...

Tao Feng | Apr 19, 2019

tfeng thanks for the review! > How do we make sure the cache DAG in sync with ...

Chao-Han Tsai | Apr 19, 2019

> Where is dag_repositories.json located? If we want to add a new dag location,...

Chao-Han Tsai | Apr 19, 2019

Correct me if my understanding is incorrect. Is it the idea to have schedul...

Chao-Han Tsai | Apr 19, 2019

How will the DAG model/ORM class be updated (as we are moving toward using that...

Ash Berlin-Taylor | May 02, 2019

> How will the DAG model/ORM class be updated (as we are moving toward using th...

Chao-Han Tsai | May 03, 2019

> We need to give some thought to how errors are handled and surfaced to any op...

Chao-Han Tsai | May 03, 2019

ash thanks for the review. Very insightful : )

Chao-Han Tsai | May 03, 2019

How do we map from the conn_id's you mentioned above to, say, a GitDagFetcherCl...

Ash Berlin-Taylor | May 02, 2019

> How do we map from the conn_id's you mentioned above to, say, a GitDagFetcher...

Chao-Han Tsai | May 03, 2019

> how the cache would work. I am thinking about letting user specifying a $cac...

Chao-Han Tsai | May 03, 2019

I think rather than symlinking we should update the DagBag etc to be able to re...

Ash Berlin-Taylor | May 03, 2019

I think even with the symlink approach, the dags folder and repo_cache can stil...

Chao-Han Tsai | May 04, 2019

Any git sync want to allow a commitish, which could be: a specific commit, a br...

James Meickle | May 07, 2019

> what the API for the DagFetcher class is class BaseDagFetcher(): def fetc...

Chao-Han Tsai | May 03, 2019

> what changes need making to the hooks(?) to support this. Can you shed some ...

Chao-Han Tsai | May 03, 2019

I guess how do the hooks (which are what the conn_id points at right?) in the r...

Ash Berlin-Taylor | May 03, 2019

During the fetch phase, we will initialize one process per repo. The process wi...

Chao-Han Tsai | May 04, 2019

> what we will use to write the Git based fetcher (I'd very much prefer it if w...

Chao-Han Tsai | May 03, 2019

> GitPython is not suited for long-running processes (like daemons) as it tends...

Ash Berlin-Taylor | May 03, 2019

Thanks for pointing that out! I did not aware that GitPython isn't suitable for...

Chao-Han Tsai | May 04, 2019

One thing I would like to point out is that we will spin up multiple processes ...

Chao-Han Tsai | May 04, 2019

> how we minimize scheduler delay We will need to block the scheduler to mutat...

Chao-Han Tsai | May 03, 2019

I wrote a git poller that does this async: detect there's been a change, pull i...

James Meickle | May 07, 2019

I see. We can make GitDagFectcher async and swap symlink like you said : )

Chao-Han Tsai | May 09, 2019

> anything we need to do to ensure that the versions between scheduler and work...

Chao-Han Tsai | May 03, 2019

Not all remote stores will be versioned (like S3 buckets, which can or can not ...

James Meickle | May 07, 2019

For a storage system like S3, we can enforce certain rules like storing the sna...

Chao-Han Tsai | May 09, 2019

Similarly to few other voices in the discussion. I do not really see an added v...

Jarek Potiuk | May 04, 2019

jarek.potiuk thanks for the review! I think the value of supporting remote dag...

Chao-Han Tsai | May 05, 2019

But this will only work (snapshotting the dags used) if we use GIT. If we use S...

Jarek Potiuk | May 05, 2019

> But this will only work (snapshotting the dags used) if we use GIT. If we use...

Chao-Han Tsai | May 09, 2019

> I think extending AIP-20 DAG manifest https://cwiki.apache.org/confluence/dis...

Chao-Han Tsai | May 09, 2019

Interesting. How does the scheduling to execution flow looks like if we are usi...

Chao-Han Tsai | May 09, 2019

It would definitely be good to have a remote fetcher on VMs where we don't want...

Kaxil Naik | May 08, 2019

I can see the benefit of this – as right now I'm having difficulty in that my e...

Ken Melms | Jun 14, 2019

I came from Oozie Job World (with Py/Spark). When I submit the job to execution...

Valeriys Soloviov | Aug 13, 2019

milton0825 any movement here? We see need for this with more than one group of...

David Pryce | Jan 10, 2020

I agree this and the general development of an external registry for DAGs could...

Kyle Prifogle | Jul 22, 2020

I have a question: Why does airflow dag not support reading directly from db, ...

luoyuliuyin | May 06, 2021

Superseded by AIP-66 https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP...

Kaxil Naik | Jun 24, 2024

Delete comments Help