Report #1

Week 1: May 27 - June 2

Activities:

  • Create a new branch GORA-527
  • Create a basic structure for the Redis datastore.
  • Investigate about Redis data types, embed REDIS for tests and connection.


Report #2

Week 2: June 3 - June 9

Activities: 

  • Analyze libraries to connect datastore to Redis DB
  • Use Reddisson library instead of Jedis
  • Read configuration files

Future week:

  • Started to an analysis of Redisson codecs in order to find the best codec that aligns with Gora's use case. 

Report #3

Week 3: June 10 - June 16

Analysis of Redis libraries

  • Port to test containers (Docker redis)
  • Add connections mode configuration (cluster, replicated, ...)
  • Implement put, exists, get, delete
  • Fix Alfonso comments (reading configuration file as in HBase)

Tasks completed:

Next Week:

  • Use strings data types apart from hashed
  • Add a secondary index as explained in the analysis of the library

Report #4

Week 4: June 17 - June 23

    Tasks:

  • Implement two storage strategies HASH, STRING (Madhawa's suggestion)
  • Create two DataStoreTest classes one for each strategy -Ignore schema-related methods getSchemaName createSchema deleteSchema schemaExists
  • Add secondary index using ScoredSortedSet for querying (put, delete methods)
  • Enable tests for methods: get, put, exists (passing).

Next Week:

  • Research and implement unit tests to check the Redis support and transactions with the redisson library. The idea of this test is to make sure that there won't be consistency problems when there is a secondary index.
  • Test different connections types with redis. For now, I've only tested Single Serve mode. 
  • Implement unit tests for Cluster, Replicated, etc.

Find the commit in GitHub

QuestionAnswer

Redis doesn't have a schema as other data stores, so is it okay to leave functions such as createSchema, deleteSchema y schemaExists with no implementation?

After discussing with the mentors, it is safe to leave these methods without implementation. See the answer from Carlos.

Report #5

Week 5: June 24 - June 30

    Tasks:

  • Change Redis docker image with one that supports clusters.
  • Add test for connections in the cluster
  • Group I/O operations in batch in order to check the consistency of secondary indices in methods put and delete.

Next Week:

  • Implement unit tests for Sentinel and replicated mode.
  • Implement queries

See the commit


Report #6

Week 6: July 1 - July 7

    Tasks:

  • Add support to List and Hash datatypes from Redis

  • Improve configuration of Redis docker container

  • Start implementing queries.

Suggestion: 

   One of the mentors suggested that we should support List and Maps.

QuestionAnswer
What is the expected behavior when calling the method put() twice?  
E.g.

obj1 : {name: Mock, address:Foo, boss: Bar}
redisDataStore.put('001', obj1);

obj2 : {name: Mock, address:null, boss:null}
redisDataStore.put('001', obj2);

Should we throw an exception saying that the obj is in the DB? or should we write the tuple?

Carlos answer:

The expected behavior is an update operation. Please give a look to the tests "testUpdate" and "testEmptyUpdate" of DataStoreTestBase[1], where you can find some examples. Also, notice that some fields could be deleted during the process  [2], that could be an issue in redis because you would need to delete the corresponding key-value entries within the put method.


[1] https://github.com/apache/gora/blob/master/gora-core/src/test/java/org/apache/gora/store/DataStoreTestBase.java#L203
[2] https://github.com/apache/gora/blob/master/gora-core/src/test/java/org/apache/gora/store/DataStoreTestUtil.java#L719

Have any of you worked with the Sentinel mode of Redis? I am having problems with the following exception:


org.apache.gora.util.GoraException: org.redisson.client.RedisNodeNotFoundException:

Node: NodeSource [slot=7484, addr=redis://172.17.0.2:7001, redisClient=null, redirect=MOVED, entry=null]

hasn't been discovered yet

We 'll come back to this problem after implementing other features for the redis datastore

Next Week:

Implement queries and solve problems found with sentinel mode in Redis.


See the commit

Report #7

Week 7: July 8 - July 14

    Tasks:

  • Implement queries
  • Enable some query-related tests

Next Week:

  • Analyze update operations suggested by Carlos.
  • Analyze data type serialization for complex information such as Map or bytes. These tests are currently failing and I want to determine the problem. 


Seel commit


Report #8

Week 8: July 15 - July 21

Tasks

  • Implement update features
  • Improve maps serialization
  • Implement deleteByQuery
  • Implement deleteSchema

Next week: Fix remaining tests in DataStoreTestBase

See commit 


Report #9

Week 9: July 22 - July 28

Tasks

  • Enable all tests for the DataStore
  • Fix suggestions of Carlos. For instance, 1) Use enums for datatypes; 2) Use constants with strings
  • Enable map reduce tests

Next week: Work in the documentation and add examples.

See commit 

Sent PR, see.


Report #10

Week 10: July 29 - August 04

Tasks

  • Add documentation and examples

Next week: Work in the documentation and add examples.

See documentation 

Report #11

Week 11: August 04 - August 10

  • Implement suggestions provided by mentors
  • Fix typos
  • Delete unnecessary code
  • Improve format

See commit


  • No labels