Report #1
Week 1: May 27 - June 2
Activities:
- Create a new branch GORA-527
- Create a basic structure for the Redis datastore.
- Investigate about Redis data types, embed REDIS for tests and connection.
Report #2
Week 2: June 3 - June 9
Activities:
- Analyze libraries to connect datastore to Redis DB
- Use Reddisson library instead of Jedis
- Read configuration files
Future week:
- Started to an analysis of Redisson codecs in order to find the best codec that aligns with Gora's use case.
Report #3
Week 3: June 10 - June 16
- Port to test containers (Docker redis)
- Add connections mode configuration (cluster, replicated, ...)
- Implement put, exists, get, delete
- Fix Alfonso comments (reading configuration file as in HBase)
Tasks completed:
Next Week:
- Use strings data types apart from hashed
- Add a secondary index as explained in the analysis of the library
Report #4
Week 4: June 17 - June 23
Tasks:
- Implement two storage strategies HASH, STRING (Madhawa's suggestion)
- Create two DataStoreTest classes one for each strategy -Ignore schema-related methods getSchemaName createSchema deleteSchema schemaExists
- Add secondary index using ScoredSortedSet for querying (put, delete methods)
- Enable tests for methods: get, put, exists (passing).
Next Week:
- Research and implement unit tests to check the Redis support and transactions with the redisson library. The idea of this test is to make sure that there won't be consistency problems when there is a secondary index.
- Test different connections types with redis. For now, I've only tested Single Serve mode.
- Implement unit tests for Cluster, Replicated, etc.
Find the commit in GitHub
Question | Answer |
---|---|
Redis doesn't have a schema as other data stores, so is it okay to leave functions such as createSchema, deleteSchema y schemaExists with no implementation? | After discussing with the mentors, it is safe to leave these methods without implementation. See the answer from Carlos. |
Report #5
Week 5: June 24 - June 30
Tasks:
- Change Redis docker image with one that supports clusters.
- Add test for connections in the cluster
- Group I/O operations in batch in order to check the consistency of secondary indices in methods put and delete.
Next Week:
- Implement unit tests for Sentinel and replicated mode.
- Implement queries
See the commit
Report #6
Week 6: July 1 - July 7
Tasks:
Add support to List and Hash datatypes from Redis
Improve configuration of Redis docker container
Start implementing queries.
Suggestion:
One of the mentors suggested that we should support List and Maps.
Question | Answer |
---|---|
What is the expected behavior when calling the method put() twice? E.g. obj1 : {name: Mock, address:Foo, boss: Bar} obj2 : {name: Mock, address:null, boss:null} Should we throw an exception saying that the obj is in the DB? or should we write the tuple? | Carlos answer: [1] https://github.com/apache/gora/blob/master/gora-core/src/test/java/org/apache/gora/store/DataStoreTestBase.java#L203 |
Have any of you worked with the Sentinel mode of Redis? I am having problems with the following exception: org.apache.gora.util.GoraException: org.redisson.client.RedisNodeNotFoundException: Node: NodeSource [slot=7484, addr=redis://172.17.0.2:7001, redisClient=null, redirect=MOVED, entry=null] hasn't been discovered yet | We 'll come back to this problem after implementing other features for the redis datastore |
Next Week:
Implement queries and solve problems found with sentinel mode in Redis.
Report #7
Week 7: July 8 - July 14
Tasks:
- Implement queries
- Enable some query-related tests
Next Week:
- Analyze update operations suggested by Carlos.
- Analyze data type serialization for complex information such as Map or bytes. These tests are currently failing and I want to determine the problem.
Seel commit
Report #8
Week 8: July 15 - July 21
Tasks
- Implement update features
- Improve maps serialization
- Implement deleteByQuery
- Implement deleteSchema
Next week: Fix remaining tests in DataStoreTestBase
See commit
Report #9
Week 9: July 22 - July 28
Tasks
- Enable all tests for the DataStore
- Fix suggestions of Carlos. For instance, 1) Use enums for datatypes; 2) Use constants with strings
- Enable map reduce tests
Next week: Work in the documentation and add examples.
See commit
Sent PR, see.
Report #10
Week 10: July 29 - August 04
Tasks
- Add documentation and examples
Next week: Work in the documentation and add examples.
See documentation
Report #11
Week 11: August 04 - August 10
- Implement suggestions provided by mentors
- Fix typos
- Delete unnecessary code
- Improve format
See commit