Page History

...

1. READ_COMMITTED isolation level uses pessimistic locks on the data being changed. 
   1. Pros
      1. Every writer always reads the latest committed data
   2. Cons
      1. Starvation since writers contend for pessimistic locks based on when the lock is released
      2. Non-Repeatable reads (same record or file read multiple times during the same write/transaction) → this is lower level of guarantee than what Hudi already provides with snapshot isolation
      3. Requires locks at record or file level to allow concurrent transactions to complete in parallel
2. SERIALIZABLE isolation uses different techniques of achieving serializability such as 2 phase locking (2PL), timestamp ordering for conflict avoidance. Such techniques are fairly complex and probably a non-scalable, overkill for a system like Hudi. The reasons for this is out of the scope of this RFC but can be understood by doing research on the need of 2PL and how it is implemented.
   1. Pros
      1. Highest form of isolation and provides many more guarantees than update conflicts (such as no phantom reads, conflict and view serializability etc)
   2. Cons
      1. Complex to implement and is generally something that is required for OLTP type systems 
      2. Out of scope of this RFC 
3. SERIALIZABLE_SNAPSHOT_ISOLATION is a variant of serializable isolation without some of the other guarantees provided by serializable isolation level but still providing methodologies to handle update conflicts. This isolation level uses MVCC based snapshot isolation along with a way to resolve update conflicts. There could be multiple ways to achieve this. 
   1. PESSIMISTIC LOCKING
      1. Acquire a exclusive_lock (reads & writes are blocked) for the entirety of the transaction. For Hudi this could be table level, partition level or file level - each of them have different trade-offs. Additionally, for a system like Hudi, where a writer modifies a batch of data and has no prior understanding of what that batch is, this would mean table level locks for the entirety of duration for the writer. 
         1. Pros
            1. Achieves our goal of handling update conflicts
         2. Cons
            1. May not scale depending on level of locks chosen
            2. Can eventually make all transactions serial by using table level locks for the entirety of the transaction thus turning it into a serial operation anyways. 
   2. OPTIMISITIC LOCKING
      1. Allow two concurrent transactions to proceed by reading the latest committed snapshot at the beginning of their transaction. Before committing the transactions, acquire a table level exclusive_lock (reads & writes are blocked) and perform conflict resolution to check if there are conflicting updates. If yes, allow for first transaction (may be using timestamp ordering) to continue to commit and abort the second transaction (which has to be retried). 
         1. Pros
            1. Achieves our goal of handling update conflicts
            2. Scales well since only a table level lock is needed
            3. Lock is acquired only for a small duration of time so allows transactions to run and commit in parallel if no conflicts
            4. Works well for cases where there is none to light contentions
         2. Cons
            1. Requires an external server for locks 
            2. Non performant when there is heavy contention
   3. LOCK FREE
      1. Since MVCC keeps a version of every modified data, two concurrent transactions could be allowed to proceed and commit concurrently. With all versions of data present, a conflict resolution and reconciliation mechanism can be applied later by the reader to get the latest, correct state of the datum.  
         1. Pros
            1. Achieves our goal of handling update conflicts
            2. No locks are required
            3. Scales well
         2. Cons
            1. Reader has to do additional work of reconciliation which may require additional metadata thus a possibility of increasing disk and compute cost. 

...