Motivation

• Users should be able to control thin client retry behavior in case of connection loss.
• The API should be unified across all thin client implementations.

Description

• Automatic reconnect on failure is present in all thin clients (see Thin clients features).
• Java thin client also implements automatic operation retry. For example, if a cache.put() fails due to a connection issue, Ignite will try to establish a new connection (or use an existing one to another server), and execute cache.put() again. This behavior is dangerous for non-idempotent operations. In a case when the server completes an operation, but the connection fails during the response, the client will still retry the operation. For example, an SQL query that increments a column by 1 can be executed two or more times, leading to unexpected results.
• Non-Java thin clients do not include automatic operation retry, so the users have to implement it like this:

while (maxRetries-- > 0)
{
    try
    {
        cache.Put(1, "Hello");
        break;
    }
    catch (Exception e) when (e.GetBaseException() is SocketException)
    {
        // Retry.
    }
}

Summary

• Java thin client has retry, but it can be dangerous and can't be fine-tuned.
• Other thin clients don't have retry and the user has to write cumbersome code.

Proposal

All changes are on the client side, server is not affected.

1. Add ClientConfiguration.retryPolicy property. By default, there is no retry.

2. Introduce ClientRetryPolicy interface

/**
 * Client retry policy determines whether client operations that have failed due to a connection issue should be retried.
 */
public interface ClientRetryPolicy {
    /**
     * Gets a value indicating whether a client operation that has failed due to a connection issue should be retried.
     *
     * @param client Ignite client instance.
     * @param operationType Operation type.
     * @param iteration Current iteration index (greater or equal to 0).
     * @param exception Exception that caused the operation to fail.
     * @return {@code true} if the operation should be retried on another connection, {@code false} otherwise.
     */
    public boolean shouldRetry(IgniteClient client, ClientOperationType operationType, int iteration, ClientConnectionException exception);
}

3. Provide predefined implementations, for example: ClientRetryAllPolicy, ClientRetryReadPolicy, ClientRetryIdempotentPolicy.

Usage example:

ClientConfiguration cfg = new ClientConfiguration()
        .setRetryPolicy(new ClientRetryReadPolicy());

Ignition.startClient(cfg);

Cross-Language Support

An interface is a good choice for Java and C# thin clients. For other languages, like C++ and Python, we may consider a different approach, such as a callback or a function pointer.

Risks and Assumptions

Ideally, we could pass more information to the shouldRetry method: cache keys and values, SQL query text, and so on, according to the operation at hand. However, this will require extra heap allocations - for every client method call, we'll need to store the arguments somewhere. This means overhead for the entire thin client API, which is not acceptable. We could also make this behavior optional with a property like ClientRetryPolicy.requiresArguments. For now, this is considered out of scope.

Discussion Links

• https://lists.apache.org/thread/w2pqdz2p9jn6rqq2lw0zv9ohv0o7m9wx

Reference Links

• Code sample: https://github.com/apache/ignite/pull/9586
• Cassandra retry policy: https://docs.datastax.com/en/drivers/java/2.0/com/datastax/driver/core/policies/RetryPolicy.html
• gRPC Retry Policy  (an example of similar approach)
  • https://devblogs.microsoft.com/dotnet/grpc-in-dotnet-6/#transient-fault-handling-with-retries
  • https://docs.microsoft.com/en-us/aspnet/core/grpc/retries?view=aspnetcore-6.0

Tickets