Delete comments <a href="http://www.aptis.info/portfolio/delete-all-comments/" target="_blank" title="Help"> <span class="aui-icon aui-icon-small aui-iconfont-help aptis-delete-all-comments-title-icon">Help</span> </a>

If you propose to have mount options on global level we need to be careful that...

Jens Scheffler | Oct 08, 2023

It should just be a lightweight hash table and only gets initialised when a met...

Bolke de Bruin | Oct 08, 2023

I like the general idea of AFS as proposed here but I have a bit of doubts whet...

Jens Scheffler | Oct 08, 2023

The idea behind using mount points is that it comes more natural to what the us...

Bolke de Bruin | Oct 08, 2023

Thinking about your Path idea, can you give an example what you would like? I c...

Bolke de Bruin | Oct 08, 2023

Mount are already a PurePath and behave like a concrete Path. I can make it a c...

Bolke de Bruin | Oct 08, 2023

With "Path like" I was referring to the API which pathlib.Path provides - but ...

Jens Scheffler | Oct 08, 2023

Indeed, I think we should not support nested mounts (it raises an error current...

Bolke de Bruin | Oct 08, 2023

+1 to the original Jens comment. Overall I'd suggest to avoid referencing the ...

Igor Kholopov | Oct 08, 2023

I disagree with your statement with that we are not mounting a filesystem, beca...

Bolke de Bruin | Oct 09, 2023

I think calling it a FileSystem is a bad idea because it might be misleading. A...

Fokko Driesprong | Oct 10, 2023

Haha I am happy to rename and to put up a poll for that . Note that fsspec can ...

Bolke de Bruin | Oct 10, 2023

And samba is great .

Bolke de Bruin | Oct 10, 2023

Related to comment above: If a mount can be made as top level code, then the DA...

Jens Scheffler | Oct 08, 2023

Validating connections requires a DB request per connection per times processed...

Bolke de Bruin | Oct 08, 2023

That is a design question. I like a proper validation and fail fast other than ...

Jens Scheffler | Oct 08, 2023

True. Maybe even a configurable option like [mount] eager=True, but maybe we sh...

Bolke de Bruin | Oct 08, 2023

These implementations must be hosted within the respective provider packages. ...

Jens Scheffler | Oct 08, 2023

Already the case

Bolke de Bruin | Oct 08, 2023

For the ZipFileSystem - or let's say I want to define a 7zFileSystem - would th...

Jens Scheffler | Oct 08, 2023

mount accepts a fs_type option of type AbstractFilesystem (see fsspec for its ...

Bolke de Bruin | Oct 08, 2023

AFS interface should get one of the standard "citicens" of the Provider Package...

Jens Scheffler | Oct 08, 2023

Absolutely, this is the case in the implementation, but I'll make it explicit.

Bolke de Bruin | Oct 08, 2023

It'd be good to understand what prevents the user from using fsspec directly in...

Igor Kholopov | Oct 08, 2023

Good point. I'll add that as a consideration. We are in the same boat as Apache...

Bolke de Bruin | Oct 09, 2023

Dependencies are always a concern, but fsspec spawns out of Dask and has connec...

Tzu-ping Chung | Oct 11, 2023

It is also used by PyTorch too

Kaxil Naik | Oct 19, 2023

And Dagster

Bolke de Bruin | Oct 20, 2023

How does this interact with Datasets? They currently have a URI, and if the URI...

Ash Berlin-Taylor | Oct 09, 2023

In general they are two separate concepts (see also the discussion here: https:...

Bolke de Bruin | Oct 09, 2023

I don't have something in mind, other than a "these both take URLs and are both...

Ash Berlin-Taylor | Oct 09, 2023

I might, there are some inklings of ideas

Bolke de Bruin | Oct 10, 2023

I thought about this as well (see Progress Tracking PR) but similarly there’s n...

Tzu-ping Chung | Oct 11, 2023

As Airflow has an inherit platform complexity, the idea of linking dataset and ...

Jens Scheffler | Oct 09, 2023

I agree, but the idea needs some iterations and fleshing out. AFS is intended a...

Bolke de Bruin | Oct 10, 2023

Okay, thanks. Does not need to be "all complete with one PR" but in general it ...

Jens Scheffler | Oct 11, 2023

Would such a AS-Path be serializable in a way a file/path reference from a remo...

Jens Scheffler | Oct 10, 2023

Yes, that's the plan

Bolke de Bruin | Oct 11, 2023

One thing that came into my mind: Main driver is as described (100% agree on t...

Jens Scheffler | Oct 10, 2023

For now, if it is detected that copying/moving happens within the same Store th...

Bolke de Bruin | Oct 19, 2023

Yeah, we are going to follow-up by contributing the work on UTO & Astro SDK (th...

Kaxil Naik | Oct 19, 2023

Probably want to remove Hyperlink

Kaxil Naik | Oct 19, 2023

Bolke de Bruin | Oct 19, 2023

Did you miss adding something in this section bolke ? Currently it just con...

Kaxil Naik | Oct 19, 2023

I'm not sure what to add . The PR contains the real implementation details. The...

Bolke de Bruin | Oct 20, 2023

Then you can "resolve" this comment by adding a link (again to the PR and a not...

Jens Scheffler | Oct 21, 2023

Would have liked to see a example of each of the points mentioned here. May...

Kaxil Naik | Oct 19, 2023

It is not implemented (on the side of XCom) yet. But what we could do for XCom ...

Bolke de Bruin | Oct 20, 2023

Yeah, I mean even a conceptual section with high-level example of what the inte...

Kaxil Naik | Oct 20, 2023

You do have lineage listed in the image https://cwiki.apache.org/confluence/pag...

Kaxil Naik | Oct 20, 2023

It is in the blocks of the image above AIrflow Storage / FS. It means it could ...

Bolke de Bruin | Oct 21, 2023

Would it also mean that if you use an "older" Airflow version e.g. 2.7.3 you ar...

Jens Scheffler | Oct 21, 2023

I'm not entirely how dependency resolution works for the providers. As long as ...

Bolke de Bruin | Oct 21, 2023

Means: XCom storing it's content on a FS Backend or storing AS links in XCom da...

Jens Scheffler | Oct 21, 2023

XCom storing its content. References are supported.

Bolke de Bruin | Oct 21, 2023

I assume with merge of the linked PR and checking the details - except that it ...

Jens Scheffler | Nov 25, 2023

I believe this AIP is don meanwhile?

Jens Scheffler | Jan 19, 2024

bolke Looks like the AIP is done? Can we update the status if so

Pankaj Koti | Apr 25, 2024

Delete comments Help