Delete comments <a href="http://www.aptis.info/portfolio/delete-all-comments/" target="_blank" title="Help"> <span class="aui-icon aui-icon-small aui-iconfont-help aptis-delete-all-comments-title-icon">Help</span> </a>

Thanks for documenting this clearly once again!

Vinoth Chandar | Dec 15, 2019

+1 Very clear

Nishith Agarwal | Dec 17, 2019

+1

Vinoth Chandar | Dec 15, 2019

gtk.. for a lot of the smaller datasets, its fair to say this bootstrap cost wi...

Vinoth Chandar | Dec 15, 2019

This is phenomenal So, if one doesn't have a (near) real time use-case for th...

sivabalan narayanan | Dec 16, 2019

These are uplifting results, although, in the old (24hrs) bootstrap case, no re...

Nishith Agarwal | Dec 17, 2019

Yes, there is a read side merge cost. The implementation is aiming to only take...

Balaji Varadarajan | Feb 18, 2020

Is it possible to just hand nulls for the _hoodie* metadata columns, as a query...

Vinoth Chandar | Feb 19, 2020

Sounds like a good option to have. Will investigate (added HUDI-619 for this)

Balaji Varadarajan | Feb 19, 2020

can we call out the requirements for this original dataset? I imagine the user ...

Vinoth Chandar | Dec 15, 2019

CLI integration?

Vinoth Chandar | Dec 15, 2019

What about the datasource?

Vinoth Chandar | Dec 15, 2019

+1, also, are we planning to tackle a scenario of folks who have already starte...

Nishith Agarwal | Dec 17, 2019

Regarding Spark DataSource, CLI and DeltaStreamer support, I will be working wi...

Balaji Varadarajan | Feb 19, 2020

can this be just another KeyGenerator config?

Vinoth Chandar | Dec 15, 2019

Yes. Agree

Balaji Varadarajan | Feb 19, 2020

000000000? instead in actual code? to stay consistent with our timestamp format...

Vinoth Chandar | Dec 15, 2019

Sure

Balaji Varadarajan | Feb 19, 2020

do we support changes though? in this design, we just bootstrap once right

Vinoth Chandar | Dec 15, 2019

same doubt as vinoth. when do we update these?

sivabalan narayanan | Dec 16, 2019

The initial proposal was to keep bootstrap index consistent with changing data....

Balaji Varadarajan | Mar 02, 2020

is nt this structure simply a MapFIle? https://diveintohadoop.wordpress.com/201...

Vinoth Chandar | Dec 15, 2019

Implementing with HFile structure. for now. Based on performance issues we see ...

Balaji Varadarajan | Mar 02, 2020

why? they would be cleaned only if there was a newer slice that was non-skeleto...

Vinoth Chandar | Dec 15, 2019

+1

Nishith Agarwal | Dec 18, 2019

Yes, removing the skeleton file is all we need for correctness. Also, the exter...

Balaji Varadarajan | Mar 02, 2020

While I understand how this can benefit us in general, this does increase the s...

Vinoth Chandar | Dec 15, 2019

Agree, this can be solved in a follow up.

Balaji Varadarajan | Mar 02, 2020

Overall design seems sound to me! had some implementation questions.. which we ...

Vinoth Chandar | Dec 15, 2019

correct me if I am wrong. if my understanding is right, the set of partitions t...

sivabalan narayanan | Dec 16, 2019

In the diagram below, why the result file is named as skeleton file? I thought ...

sivabalan narayanan | Dec 16, 2019

Yes, It is a regular hudi file. Thanks for catching it.

Balaji Varadarajan | Mar 02, 2020

so, there will be atmost only one boostrap commit file per old file id(origina...

sivabalan narayanan | Dec 16, 2019

Yeah, there will be only 1 bootstrap commit file which will track the bootstrap...

Balaji Varadarajan | Mar 02, 2020

a naive question. what will be the commit time for these old records? the boots...

sivabalan narayanan | Dec 16, 2019

Yes, it will be bootstrap commit time. Not sure if I follow your second comment...

Balaji Varadarajan | Feb 18, 2020

Will review the comments and address them once I am back from vacation.

Balaji Varadarajan | Dec 17, 2019

Can we also clearly state that user will require to have a downtime on the exis...

Nishith Agarwal | Dec 17, 2019

Yes. Added the step

Balaji Varadarajan | Feb 19, 2020

High level questions & thoughts : 1) This assumes that the source data is part...

Nishith Agarwal | Dec 17, 2019

Yes, the design is expected to work for non-partitioned dataset also. Partiti...

Balaji Varadarajan | Feb 18, 2020

Out of curiosity (and precedents before with cleaner, rollback..), do we see pa...

Nishith Agarwal | Dec 18, 2019

Yeah, it is possible to have non-uniform partitions. It would reflect in the re...

Balaji Varadarajan | Mar 02, 2020

Does this mean a Hudi enabled dataset will actually consist of 2 paths 1) The o...

Nishith Agarwal | Dec 18, 2019

We should be keeping a separate hive table corresponding to the bootstrapped da...

Balaji Varadarajan | Mar 02, 2020

Can you add a one liner on how the bootstrap file slice and the old file slice ...

Nishith Agarwal | Dec 19, 2019

Done

Balaji Varadarajan | Mar 02, 2020

Left a few comments for clarification, rest looks super awesome, great document...

Nishith Agarwal | Dec 19, 2019

vbalaji We should also have a section on how we are proposing to handle duplica...

Nishith Agarwal | Jan 13, 2020

As uniqueness is a fundamental assumption in Apache Hudi, I have documented thi...

Balaji Varadarajan | Feb 18, 2020

why should we bother specially about duplicates here? today, you can bulk_inser...

Vinoth Chandar | Feb 19, 2020

I am hoping that this would also work with DataSource API for inserting/upserti...

Udit Mehrotra | Jan 31, 2020

yes.. this is being added at the hudi-client level.. and exposed to both..

Vinoth Chandar | Feb 04, 2020

This would prevent us from using Spark's native parquet readers for reading Cop...

Udit Mehrotra | Feb 01, 2020

uditme good point.. The path filter would be problematic here for SparkSQL sinc...

Vinoth Chandar | Feb 04, 2020

Nice proposal Balaji ! Left a couple of comments. In addition, we might want to...

Udit Mehrotra | Feb 01, 2020

vinoth nagarwal : uditme has a proposal for supporting Data Source for Bootstra...

Balaji Varadarajan | Mar 07, 2020

uditme Left few comments.. Mostly on the same page.. I wonder if we should fir...

Vinoth Chandar | Mar 09, 2020

I think it all depends upon our priority. If we want to release Bootstrapping f...

Udit Mehrotra | Mar 09, 2020

nit: optional skeleton file..?

Vinoth Chandar | Mar 09, 2020

Done. Thanks!

Udit Mehrotra | Mar 09, 2020

are these APIs based on v1 or v2 datasource APIs?

Vinoth Chandar | Mar 09, 2020

These API's are based on DataSource V1. I did some exploration for DataSource v...

Udit Mehrotra | Mar 09, 2020

This is an general design issue I feel.. but is there a way to generate the spl...

Vinoth Chandar | Mar 09, 2020

I did check the code for RealtimeInputFormat, and I do not see us doing anythin...

Udit Mehrotra | Mar 09, 2020

Also based on my discussions with Balaji, we would be disabling the file splitt...

Udit Mehrotra | Mar 09, 2020

In general, I think that this design does give us the ability to control the pa...

Udit Mehrotra | Mar 09, 2020

I think you are referring to the ability to now hand a Iterator[Row] back in co...

Vinoth Chandar | Mar 09, 2020

Yes, and being able to control what goes into each partition for each task. Lik...

Udit Mehrotra | Mar 09, 2020

Control over partition is otherwise not possible in Spark. I explored other app...

Udit Mehrotra | Mar 09, 2020

For SparkSQL, we will still fallback to the input format or somehow Spark will ...

Vinoth Chandar | Mar 09, 2020

Yes this is not a solution for SparkSQL. For that Spark would still default to ...

Udit Mehrotra | Mar 09, 2020

Delete comments Help