Delete comments <a href="http://www.aptis.info/portfolio/delete-all-comments/" target="_blank" title="Help"> <span class="aui-icon aui-icon-small aui-iconfont-help aptis-delete-all-comments-title-icon">Help</span> </a>

s/Kafka/Stream. The interface and protocol is stream agnostic and not specific ...

Subbu Subramaniam | May 06, 2019

Corrected.

Chinmay Soman | May 06, 2019

offsets are not guaranteed to be continuous right? Offset 1 to 100 might not ne...

Neha Pawar | May 06, 2019

Please see my reply to Subbu's comment (regarding this same question).

Chinmay Soman | May 07, 2019

Does this mean all replicas will have to create a segment on their own? If one ...

Neha Pawar | May 06, 2019

I see it explained later

Neha Pawar | May 06, 2019

Pinot needs a storage system that is attached to the controllers and is common ...

Subbu Subramaniam | May 06, 2019

In this proposal, the goal is to remove the need for a storage system attached ...

Chinmay Soman | May 07, 2019

So, are we considering pinot clusters that do not have offline segments at all?...

Subbu Subramaniam | May 07, 2019

Does it work if controller just stores the segments on local disk? The download...

Jackie Jiang | Nov 15, 2019

The local disk on controller may not be able to hold all segments for all table...

Ting Chen | Feb 06, 2020

If deepstore is not available, data pushes to offline tables cannot be complete...

Subbu Subramaniam | May 06, 2019

Good point. The caveat here is whether the deep store is in the same region or ...

Chinmay Soman | May 07, 2019

Support for realtime segments in deepstore is not yet done. The current method ...

Subbu Subramaniam | May 07, 2019

Even when the servers can upload the segments to the deep store directly, we st...

Ting Chen | Feb 06, 2020

You mean segment metadata?

Subbu Subramaniam | May 06, 2019

Yes. Will reword.

Chinmay Soman | May 07, 2019

This assumes that offset is directly proportional to the number of rows. We can...

Subbu Subramaniam | May 06, 2019

Good point. For numerical offsets, it doesn't really matter if the offsets are ...

Chinmay Soman | May 07, 2019

Btw, even in the existing approach, the controller has to tell the other replic...

Chinmay Soman | May 07, 2019

The key is that the end offset is only needed (as things are) when the consumin...

Subbu Subramaniam | May 07, 2019

Isn't the same effect achieved by having an arbitrarily high time threshold?

Subbu Subramaniam | May 06, 2019

No. Time is not a good way for all replicas to agree on the exact segment. By h...

Chinmay Soman | May 07, 2019

If the time threshold is set to some arbitrary high value such that we always ...

Subbu Subramaniam | May 15, 2019

It is not clear whether you are proposing that both of these variations be impl...

Subbu Subramaniam | May 06, 2019

This is a typo. Will reword

Chinmay Soman | May 07, 2019

By definition they are all at the same number of rows, right?

Subbu Subramaniam | May 06, 2019

I'm afraid I don't follow. Can you elaborate your question ? This scenario is...

Chinmay Soman | May 07, 2019

You mean, the other servers are so slow that they were not able to get to the e...

Subbu Subramaniam | May 15, 2019

I don't understand. This is the same as current protocol so is this part saying...

Subbu Subramaniam | May 06, 2019

Agree this part is very confusing. Let me remove this.

Chinmay Soman | May 07, 2019

No extra code needs to be developed for this quorum.We can modify the existing ...

Subbu Subramaniam | May 06, 2019

Yes. that will work. Good point.

Chinmay Soman | May 07, 2019

It is not clear to me what problem we are trying to solve here. From the openi...

Subbu Subramaniam | May 06, 2019

Added the most common failure scenarios we expected. After talking to our HDFS ...

Ting Chen | May 07, 2019

Wondering why we will face this restriction? The mechanism to finish on thresho...

Neha Pawar | May 06, 2019

Yes added a note saying this modification is optional. The main change is in ho...

Chinmay Soman | May 07, 2019

The motivation is to remove Pinot's strict dependence on deep store for realtim...

Chinmay Soman | May 07, 2019

Some things to think about when we start having a list of "segmentLocations" wh...

Neha Pawar | May 15, 2019

1) The key issue here seem to be replacing all the servers. If the replacement ...

Ting Chen | Aug 26, 2019

What if we rebalance and all consuming segments are now on completely different...

Neha Pawar | May 15, 2019

There is a possibility this throws off our estimations of segment sizes. We re...

Neha Pawar | May 15, 2019

Need to evaluate how this affects query serving at 1000s of qps use case

Subbu Subramaniam | Sep 25, 2019

By the server? Please clarify

Subbu Subramaniam | Sep 25, 2019

Yes. By the server. After it uploads the segment to the deep storage, it can up...

Ting Chen | Sep 25, 2019

synchronously? asynchronously? There will be zk concention for the same znode,...

Subbu Subramaniam | Sep 25, 2019

The update to zk will be done synchronously. Yes retries will be needed if ther...

Ting Chen | Sep 25, 2019

There is now a configuration introduced (we found it very useful in high qps sc...

Subbu Subramaniam | Sep 25, 2019

Meeting minutes Sep 30, 2019. Need to handle these cases: (1) have lead commit...

Subbu Subramaniam | Sep 30, 2019

We are going with Externalview based peer identification, so there should be no...

Subbu Subramaniam | Nov 13, 2019

Please update this

Subbu Subramaniam | Feb 06, 2020

Why just "initially?" We should always use split commit, so this can be done fo...

Subbu Subramaniam | Nov 13, 2019

What I meant is that right now controller's LLCSegmentCompletionHandlers has tw...

Ting Chen | Nov 13, 2019

Is this a controller to server API? So, we introduce another REST API in the se...

Subbu Subramaniam | Nov 13, 2019

The segment upload should be invoked on server API. My current thinking is that...

Ting Chen | Feb 06, 2020

This is already supported in split commit right now.

Subbu Subramaniam | Nov 13, 2019

ack. Let me try that.

Ting Chen | Nov 13, 2019

Have you tried it yet? If so, please change the sentence accordingly

Subbu Subramaniam | Feb 06, 2020

This has been tested in my POC env. However, in our Prod setting, server uploa...

Ting Chen | Feb 07, 2020

Hold off on this, a PR is pending on re-factoring the committers. Also, the ser...

Subbu Subramaniam | Nov 13, 2019

Also, we need to add code in the server to invoke the appropriate upload to dee...

Subbu Subramaniam | Nov 13, 2019

Lastly, the server should schedule an upload only after it has committed the s...

Subbu Subramaniam | Nov 13, 2019

If the server commits metadata before scheduling a background upload to deep st...

Ting Chen | Nov 13, 2019

It all depends on what kind of deep store we use. One kind allows us to store a...

Subbu Subramaniam | Feb 06, 2020

either way, PR 4713 is done, and allows for you to add another committer logic ...

Subbu Subramaniam | Feb 06, 2020

We should never hit this situation, because we will attempt the upload only aft...

Subbu Subramaniam | Nov 13, 2019

It may be worth noting that the server download case may be more common than we...

Subbu Subramaniam | Nov 13, 2019

We will first try to do a best effort upload of the segment to a configured ext...

Ting Chen | Nov 13, 2019

We should have deterministic mechanisms instead of wit N seconds etc.

Subbu Subramaniam | Nov 15, 2019

Consider adding a new download scheme called (say) "peer". So, a URI can be con...

Subbu Subramaniam | Nov 15, 2019

Good idea. A new PeerServerSegmentFetcher https://github.com/apache/incubator-p...

Ting Chen | Apr 01, 2020

mcvsubbu nehapawar could you take a look at the modified LLC protocols?

Ting Chen | Feb 06, 2020

The pictures seem to assume that COMMIT_SUCCESS is the same as CONSUMING to ONL...

Subbu Subramaniam | Feb 06, 2020

For the race condition, it seems an existing issue where the commit server may ...

Ting Chen | Feb 06, 2020

Can you clarify this further? It has been a while that the details escape me at...

Subbu Subramaniam | Feb 06, 2020

We have decided that the committer server will attempt to upload to deep store ...

Ting Chen | Feb 06, 2020

this is not a good condition to put. In practice we are having to move more and...

Neha Pawar | Feb 06, 2020

Let me rephrase this part. My point here is that if users care about their serv...

Ting Chen | Feb 06, 2020

Are you proposing a server level configuration to do one or the other? We are t...

Subbu Subramaniam | Feb 06, 2020

No there is no new configuration proposed for the segment completion mode here....

Ting Chen | Feb 07, 2020

Validation manager consciously tried to minimize zk access. The new check will ...

Neha Pawar | Feb 06, 2020

Agree. This job needs to be optimized to reduce zk access cost. I will put a no...

Ting Chen | Feb 07, 2020

With the new proposal, it may be that there is no deepstore uri in the segment ...

Subbu Subramaniam | Feb 06, 2020

Yes. Let me make the proposal clear though: Case 1: no deep store configured. ...

Ting Chen | Feb 07, 2020

One change in the controller is to accept a segment commit without actualy URI ...

Subbu Subramaniam | Feb 06, 2020

right. added the section.

Ting Chen | Feb 07, 2020

Can you describe what happens in this specific sceanrio described below? Say, ...

Subbu Subramaniam | Apr 03, 2020

Should we make this a table config?

Subbu Subramaniam | Apr 03, 2020

Perhaps "PEER" instead of "SERVER"?

Subbu Subramaniam | May 11, 2020

Done.

Ting Chen | May 12, 2020

You may also want to add a note that the goal of the config change is to ensure...

Subbu Subramaniam | May 11, 2020

Thanks. Notes added.

Ting Chen | May 12, 2020

we dont need a separate PeerSegmentFetcher since this is nothig but an http or...

Subbu Subramaniam | May 12, 2020

Even we do not call it PeerSegmentFetcher (which would a subclass of the Segmen...

Ting Chen | May 13, 2020

Delete comments Help