Please refer to Geode 1.2.0 documentation with final implementation is here.
Allow user to create Lucene Indexes on data stored in Geode
- Update the indexes asynchronously to avoid impacting write latency
Allow user to perform text (Lucene) search on Geode data using the Lucene index. Results from the text searches may be stale due to asynchronous index updates.
Provide highly available of indexes using Geode's HA capabilities
- Performance comparable to RAMFSDirectory
Building next/better Solr/Elasticsearch.
Enhancing the current Geode OQL to use Lucene index.
Similar efforts done by other data products
Hibernate Search: Hibernate search
Solandra: Solandra embeds Solr in Cassandra.
- Documents: In Lucene, a Document is the unit of search and index. An index consists of one or more Documents.
- Fields: A Document consists of one or more Fields. A Field is simply a name-value pair.
- Indexing involves adding Documents to an IndexWriter, and searching involves retrieving Documents from an index via an IndexSearcher.
- A region and list of to-be-indexed fields
- [ Optional ] Specified Analyzer for fields or Standard Analyzer if not specified with fields
A single index will not support multiple regions. Join queries between regions are not supported
- Heterogeneous objects in single region will be supported
- Only top level fields of nested objects can be indexed, not nested collections
- The index needs to be created before the region is created (for phase1)
- Pagination of results will be supported
Now that this feature has been implemented, please refer to the javadocs for details on the Java API.
Spring Data GemFire Support
A closer look at Partitioned region data flow
- FileRegion : holds the meta data about indexing files
- ChunkRegion : Holds the actual data chunks for a given index file.
- create document for indexed fields. Indexed field values are obtained from AsyncEvent through reflection (in case of domain object) or by PdxInstance interface (in case pdx or JSON); constructing Lucene document object and adding it to the LuceneIndex associated with that region.
- determine the bucket id of the entry.
- Get the RegionDirectory for that bucket, save the document into RegionDirectory.
Storage with different region types
Walkthrough creating index in Geode region
LuceneIndex can be created and destroy. We don't support creating index on a region with data for now.
Handling failures, restarts, and rebalance
The index region and async event queue will be restored with its colocated data region's buckets. So during failover the new primary should be able to read/write index as usual.
In the case of partitioned regions, the query must be sent out to all the primaries. The results will then need to be aggregated back together. Lucene search will use FunctionService to distribute query to primaries.
Input to primaries
- Serialized Query
- CollectorManager to be used for local aggregation
- Result limit
Output from primaries
- Merged collector created from results of search on local bucket indexes.
We are still investigating options for how to aggregate the data, see Text Search Aggregation Options.
In case of replicated regions, query will be sent to one of the members and get the results there. Aggregation will be handled in that member before returned to the caller.
Result collection and paging
A Lucene Service MBean is available and accessed through an ObjectName like:
This MBean provides operations these operations:
A LuceneIndexMetrics data bean includes raw stat values like:
- no rates or average latencies are available
- no aggregation (which means no rollups across members in the GemFire -> Distributed MBean)