Delete comments <a href="http://www.aptis.info/portfolio/delete-all-comments/" target="_blank" title="Help"> <span class="aui-icon aui-icon-small aui-iconfont-help aptis-delete-all-comments-title-icon">Help</span> </a>

I like where you are going with this in spirit.. lets work through this.. We a...

Vinoth Chandar | May 14, 2020

Thanks, Vinoth Chandar Just like https://issues.apache.org/jira/projects/HUDI/i...

liwei | May 16, 2020

satish.kotha Who is looking into how to support collapse style operations at ub...

Nishith Agarwal | May 15, 2020

Thanks, Nishith Agarwal our issue just like ( https://issues.apache.org/jira...

liwei | May 16, 2020

reason we did not do this was because of the query performance hit, that would ...

Vinoth Chandar | May 25, 2020

this part needs more discussion.. you seem to suggest that we will compact mu...

Vinoth Chandar | May 25, 2020

This should be left to the cleaner..

Vinoth Chandar | May 25, 2020

satish.kotha nagarwal lw309637554 leesf vbalaji bhavanisudhaccing everyone her...

Vinoth Chandar | May 25, 2020

Thanks Vinoth Chandar , I am strongly agree your suggesting.I have some think...

liwei | May 25, 2020

vinoth Thanks for your detail explanation, a doubt about the limitations/consid...

Shaofeng Li | May 26, 2020

leesf you are right.. Append only will not be affected by that limitation.. Jus...

Vinoth Chandar | May 30, 2020

Hi vinoth nagarwal Sorry for delay here. I am just starting to look at this an...

Satish Kotha | Jun 04, 2020

lw309637554 Thanks for your suggestions. The pictures/figures on the RFC are ve...

Nishith Agarwal | May 26, 2020

I think its probably okay to assume MOR for a lot of these optimizations, since...

Vinoth Chandar | May 30, 2020

+1. Supporting clustering at file-groups level would help us be nimble and give...

Balaji Varadarajan | May 30, 2020

+1 on the idea of having a special file group, for logging concurrent updates d...

Balajee Nagasubramaniam | Aug 04, 2020

or alternatively, query engines can merge them as well

Vinoth Chandar | Aug 18, 2020

vinoth nagarwal Thanks for your feedback, some new ideas about the clustering/c...

Shaofeng Li | May 27, 2020

if using `upsert` api(contains both insert and update), will also reject? And...

Shaofeng Li | Jun 18, 2020

I am referring to this code https://github.com/apache/hudi/blob/master/hudi-cli...

Satish Kotha | Jun 21, 2020

it means we handle the clustering in record level rather than file level?

Shaofeng Li | Jun 18, 2020

Yes, this adds some complexity. But this is suggested to improve data locality....

Satish Kotha | Jun 21, 2020

satish.kotha maybe we could provide strategies including file level clustering...

Shaofeng Li | Jun 21, 2020

I am thinking whether we should create a normal t6.commit that put subset commi...

Shaofeng Li | Jun 18, 2020

This is a good suggestion. However, I want to use same framework for RFC-18, wh...

Satish Kotha | Jun 21, 2020

yes, #1 sounds reasonable.

Shaofeng Li | Jun 21, 2020

leesf After discussing with others, we decided to use approach #3 instead, as t...

Satish Kotha | Jul 15, 2020

satish.kotha option #3 would make the cost time lower and the POC looks very ...

Shaofeng Li | Jul 17, 2020

just like async compaction, after generated compaction.requested, it would run ...

Shaofeng Li | Jun 18, 2020

Yes. I think similar framework can be reused. We could also leverage strategies...

Satish Kotha | Jun 21, 2020

so when compaction is requested and generating CompactionPlan, it should read t...

Shaofeng Li | Jun 18, 2020

when have clustering and compaction concepts. I think it will increase user und...

liwei | Jun 19, 2020

leesf yes, agree. I can add more step-by-step details. lw309637554 Agree te...

Satish Kotha | Jun 21, 2020

if these upsert rejected,this commit of the client will failed?

liwei | Jun 19, 2020

yes for phase1. Note that if upsert does not modify file groups being clustered...

Satish Kotha | Jun 21, 2020

on MOR mode,the clustering will merge the parquet and log files ?

liwei | Jun 19, 2020

yes, thats my thought. But, we could consider delaying clustering until compac...

Satish Kotha | Jun 21, 2020

At the first step,I think RFC 19 can not rely on RFC-08?

liwei | Jun 19, 2020

I think this is not really dependent on RFC-08. For some datasets, we already ...

Satish Kotha | Jun 21, 2020

satish.kotha Hi, thanks so much to provide "Option#2: Clustering with snapshot...

liwei | Jun 19, 2020

> Can we unify clustering and compaction into one concept such as clustering ...

Vinoth Chandar | Jun 21, 2020

lw309637554 leesf satish.kotha I think our discussions are winding down to the...

Vinoth Chandar | Jun 22, 2020

satish.kotha vinoth lw309637554 nagarwal After the discussion, I think we ...

Shaofeng Li | Jun 24, 2020

nagarwal vinoth Please take a look at these numbers and let me know if you ...

Satish Kotha | Jul 15, 2020

satish.kotha guess the assumption here is that the DFS read will be as fast as ...

Vinoth Chandar | Jul 18, 2020

Clustering with conflicting updates needs more thought like you mentioned .. le...

Vinoth Chandar | Jul 18, 2020

vbalaji nagarwal vinoth leesf Please review this new section I added and let m...

Satish Kotha | Oct 14, 2020

hi, what is the plan about query optimize base on clustering. spark and presto ...

liwei | Oct 23, 2020

Once we have the clustering plan that sorts data by a list offields, query engi...

Vinoth Chandar | Oct 24, 2020

to ensure our implementation is generic, I feel we should allow this from the g...

Vinoth Chandar | Oct 22, 2020

If we do clustering for files that have log files, then we have to introduce ad...

Satish Kotha | Oct 25, 2020

Discussed offline. It makes sense to combine clustering and compaction because ...

Satish Kotha | Oct 27, 2020

can this be just a list of fields, that can used as a composite sort key?

Vinoth Chandar | Oct 22, 2020

sure. I went with record key ranges because this information is available in da...

Satish Kotha | Oct 25, 2020

having this be configurable like that, could help us support generic clustering...

Vinoth Chandar | Oct 27, 2020

Again, it would be nice if we just treat (a) as a special case of (c)

Vinoth Chandar | Oct 22, 2020

base on this strategy, can also support index like z-ordering?

liwei | Oct 23, 2020

Basic idea of 'group' here is a list of fileIds that have same value for specif...

Satish Kotha | Oct 25, 2020

we can just do 2a,b for now IMO and leave these for later

Vinoth Chandar | Oct 22, 2020

agree

Satish Kotha | Oct 25, 2020

just `inputGroups`? I assume each group is a set of files grouped under a given...

Vinoth Chandar | Oct 22, 2020

yes. Preferably files have similar values are grouped together. Each group take...

Satish Kotha | Oct 25, 2020

this would be specific to a given strategy right. I view this as input to the s...

Vinoth Chandar | Oct 22, 2020

we should just store the number of output fileGroups or something?

Vinoth Chandar | Oct 22, 2020

I removed it and sent initial PR https://github.com/apache/hudi/pull/2202 https...

Satish Kotha | Oct 25, 2020

can we also flesh this out? I think this is where the detail is. I feel all w...

Vinoth Chandar | Oct 22, 2020

Good point. I was thinking of keeping the interface simple: cluster(HoodieClus...

Satish Kotha | Oct 25, 2020

this seems flexible enough.

Vinoth Chandar | Oct 22, 2020

+1

Vinoth Chandar | Oct 22, 2020

IMO we should fix the input/clusteringGroups at planning time and only thing th...

Vinoth Chandar | Oct 22, 2020

good point. I'll remove this section.

Satish Kotha | Oct 25, 2020

I think we should not do this. Clustering can be run differently for different ...

Vinoth Chandar | Oct 22, 2020

+1

Satish Kotha | Oct 25, 2020

satish.kotha Now that we are done with this almost, mind taking a pass at the R...

Vinoth Chandar | Dec 23, 2020

vinoth Makes sense. I'll reorganize this page next week cc: lw309637554

Satish Kotha | Dec 23, 2020

vinoth I edited this page to reflect all aspects discovered during implementat...

Satish Kotha | Jan 20, 2021

satish.kotha thanks for jumping on this. this looks great for now. Gives enough...

Vinoth Chandar | Jan 20, 2021

satish.kotha when https://issues.apache.org/jira/browse/HUDI-1399 https://issue...

liwei | Dec 23, 2020

sounds good. thanks!

Satish Kotha | Dec 23, 2020

lw309637554 can you update the wiki?

Satish Kotha | Jan 20, 2021

ok

liwei | Jan 20, 2021

satish.kotha have add it

liwei | Jan 22, 2021

satish.kotha are these points upto date? uditmeand team want to understand the ...

Vinoth Chandar | Apr 08, 2021

Yes, this is still accurate. I opened a draft PR https://github.com/apache/hudi...

Satish Kotha | Apr 09, 2021

Delete comments Help