Delete comments <a href="http://www.aptis.info/portfolio/delete-all-comments/" target="_blank" title="Help"> <span class="aui-icon aui-icon-small aui-iconfont-help aptis-delete-all-comments-title-icon">Help</span> </a>

pwason please start a mailing list DISCUSS thread when ready

Vinoth Chandar | Feb 14, 2020

+1 we already maintain this for the keys... We can also look into indexing the ...

Vinoth Chandar | Feb 19, 2020

The Column indexing is per-column and any column can be indexed as required by ...

Prashant Wason | Feb 24, 2020

curious to see some microbenchmarks or theoretical explanation for this,, (alth...

Vinoth Chandar | Feb 19, 2020

I am trying to find some way of collecting granular metrics from hoodie dataset...

Prashant Wason | Feb 24, 2020

for ease of review, can you please layout some pros/cons & tradeoffs for each a...

Vinoth Chandar | Feb 19, 2020

I have added a comparison table at the end. Please add more rows there for any ...

Prashant Wason | Feb 24, 2020

One high level comment is that : could we try to design this using the same bas...

Vinoth Chandar | Feb 19, 2020

>> could we try to design this using the same base/log file concepts that we ha...

Prashant Wason | Feb 24, 2020

pwason I know you folks are blocked on this.. You will have a detailed review b...

Vinoth Chandar | Feb 28, 2020

At some point, i want to also understand how the query engines are going to tap...

Vinoth Chandar | Feb 29, 2020

Yes, that is our goal too. We have explained some part of it below in the Workf...

Prashant Wason | Mar 02, 2020

wont there be a race? Multiple executors could be working on the same partition...

Vinoth Chandar | Feb 29, 2020

if we could do it at the input format level, that would be cool. Continuing fro...

Vinoth Chandar | Feb 29, 2020

+1 we can build tons of new features on top of this

Vinoth Chandar | Feb 29, 2020

I feel we can also apply such delta log approach to Approach 1. i.e instead of ...

Vinoth Chandar | Feb 29, 2020

Another difference is the number of files that are touched/modified/created dur...

Prashant Wason | Mar 02, 2020

Other differences: Perf: Reading a single large file may be faster than readi...

Prashant Wason | Mar 02, 2020

pwason this is so helpful! this RFC is one of best written RFC so far.. Great g...

Vinoth Chandar | Feb 29, 2020

Thanks for the encouragement.

Prashant Wason | Mar 02, 2020

question.. if we aggressively compact delta manifests, this will be very simila...

Vinoth Chandar | Feb 29, 2020

As also reported in another benchmark, point looks are faster if you have to re...

Prashant Wason | Mar 02, 2020

>> question.. if we aggressively compact delta manifests, this will be very sim...

Prashant Wason | Mar 02, 2020

This is sweet..

Vinoth Chandar | Feb 29, 2020

Additionally, we want this to lock-free. The Column Index can be created / drop...

Prashant Wason | Mar 02, 2020

This gives us more knobs to control ... we can always aggressively compact to c...

Vinoth Chandar | Feb 29, 2020

this is the special partition column index, right?

Vinoth Chandar | Feb 29, 2020

Yes.

Prashant Wason | Mar 02, 2020

I am worried about the amount of column indexes to be updated for large tables....

Vinoth Chandar | Mar 02, 2020

Column Indexes need to be updated only for the indexed columns. In Approach #2,...

Prashant Wason | Mar 02, 2020

Here's my take on the two approaches. Between the two, I do favor the approach ...

Vinoth Chandar | Mar 02, 2020

More discussions between nagarwal pwason vbalaji vinoth + balajee Agree on : ...

Vinoth Chandar | Mar 06, 2020

shivnarayan can you share a pointer to your hfile benchmarks here?

Vinoth Chandar | Mar 06, 2020

vinoth vbalaji balajee fyi

Nishith Agarwal | Apr 21, 2020

pwason what about delta commits, which may not create a new file but just log a...

Vinoth Chandar | Apr 28, 2020

Yep, they can also be handled in similar manner. This is not an exhaustive list...

Prashant Wason | Apr 28, 2020

Again I feel we need not re-invent these things again.. Love to understand why ...

Vinoth Chandar | Apr 28, 2020

We can choose method to save the deltas. MOR type log files saves "blocks" of...

Prashant Wason | Apr 28, 2020

Even this can be implemented as a read of just the last block in the logs..

Vinoth Chandar | Apr 28, 2020

Yes it can be. But how do we handle the following: Rollbacks where certain ...

Prashant Wason | Apr 28, 2020

tbh I am not sure about these even in the current scheme. we need to solve for ...

Vinoth Chandar | Apr 28, 2020

why just -100%. not following this

Vinoth Chandar | Apr 28, 2020

Rounding error. Original time taken: 7798ms Reduced to: 0.20ms 0.20ms is near...

Prashant Wason | Apr 28, 2020

end of the day, this is a disk read. So not sure if it will ever be faster..

Vinoth Chandar | Apr 28, 2020

The operation can be made faster by reducing the number of disk reads required....

Prashant Wason | Apr 28, 2020

3 is not just network I/O.. its disk read + network I/O

Vinoth Chandar | Apr 28, 2020

Correct. But a contiguous larger disk read will be faster (read + network tran...

Prashant Wason | Apr 28, 2020

shivnarayan did some of this tuning alrady for RFC-8 .. lets cross share? Is a...

Vinoth Chandar | Apr 28, 2020

I have not done any tuning wrt block size. Need some work to be done there. I d...

sivabalan narayanan | Apr 29, 2020

Open items FileID generation in the face of concurrent writers, they may try...

Vinoth Chandar | Apr 28, 2020

For implementing phase 1 (just eliminating listing using partition index) Ho...

Vinoth Chandar | May 08, 2020

the pic below, f2_w2_c3.parquet should be [400, 900], [23457, 56789], [555-555...

Shaofeng Li | May 24, 2020

when and how to build this file?

Shaofeng Li | May 24, 2020

pwason : when you get a chance, can you update the RFC with final design detail...

sivabalan narayanan | Aug 25, 2020

pwason thanks for the details. +1 on siva's comment. but let's lower the prio...

Vinoth Chandar | Aug 25, 2020

for devs to review the patch, would be good to have this design in sync w/ fina...

sivabalan narayanan | Aug 26, 2020

for eg, the final design section does not talk much about column index. And fro...

sivabalan narayanan | Aug 26, 2020

Column Indexes have not been implemented yet. I have added a NOTE.

Prashant Wason | Sep 17, 2020

shivnarayan The initial designs were based on using HFIles as raw files to save...

Prashant Wason | Sep 17, 2020

pwason can we use a different field for this? instead of overloading the map. t...

Vinoth Chandar | Aug 25, 2020

+1

sivabalan narayanan | Aug 26, 2020

This is a mapping of filenames to size so suits to a map naturally. Or maybe y...

Prashant Wason | Sep 17, 2020

this will also solve the problem of sharing the timeline amongst column indexes...

Vinoth Chandar | Aug 25, 2020

pwason something to think about here, is that the different partitions will use...

Vinoth Chandar | Sep 16, 2020

Yes, column indexes will be implemented later. It would be ideal if we could ...

Prashant Wason | Sep 17, 2020

so during finalizing of the write?

Vinoth Chandar | Aug 25, 2020

Yes

Prashant Wason | Sep 17, 2020

also need to handle rollback, though its more or less similar

Vinoth Chandar | Aug 25, 2020

so we will try to sync before we begin the next commit on the dataset?

Vinoth Chandar | Aug 25, 2020

Yes.

Prashant Wason | Sep 17, 2020

so if we fail after committing to dataset and before committing to metadata, qu...

Vinoth Chandar | Aug 25, 2020

not sure I get you. the commit to dataset(i meant, the finalize write section) ...

sivabalan narayanan | Aug 26, 2020

Commit to metadata is performed before the commit to dataset is completed. So m...

Prashant Wason | Sep 17, 2020

shivnarayan You got it right. I have explained how this is handled in my last c...

Prashant Wason | Sep 17, 2020

this would change eventually. but we can handle this separately. not an issue f...

Vinoth Chandar | Aug 25, 2020

Sure. Multi writer for Metadata Table would be similar to how Multi Writer woul...

Prashant Wason | Sep 17, 2020

I will take a closer look at this part in the PR.

Vinoth Chandar | Aug 25, 2020

pwason so the idea for these background operations are to log them ahead of tim...

Vinoth Chandar | Aug 25, 2020

+1. still on the above. also I wonder if we should also have "clean" log the de...

Vinoth Chandar | Sep 16, 2020

If we update metadata table after the clean has completed, there will be a peri...

Prashant Wason | Sep 17, 2020

I know chances are slim, but is there a chance of clash between instants betwee...

sivabalan narayanan | Aug 26, 2020

Yes, there is a rare chance if async compaction/clean is enabled on the metadat...

Prashant Wason | Sep 17, 2020

Summarizing discussion among folks from Uber, AWS and OSS on how to implement r...

Nishith Agarwal | Feb 10, 2021

vinoth pwason Updated the title of the RFC to reflect the actual work and detai...

Nishith Agarwal | Jan 24, 2022

Delete comments Help