Goals
- Allow for OODT filemanager linear scalaing, distribution and seamless querying of multiple filemanagers on separate networks.
Background and strategic fit
To truely fit the "Distributed" aspect of OODT, the file manager component should allow for multiple file managers to be started and they allow for both local and remote querying of the file managers seamlessly. For example, I have an OODT installation in the UK and an OODT installation on a separate site in Australia, it makes more sense from a data transfer and performance perspective to allow those 2 FM's to operate independantly of each other, but allow for querying of both repositories as if they were one, so I could choose to retieve information from the Australia FM. This also allows users to say "give me all Excel files from all my sites", without having to point fm-client at all the different instances.
Assumptions
Requirements
| # | Title | User Story | Importance | Notes |
|---|---|---|---|---|
| 1 | Mutliple file manager configuration and monitoring via PCS | Developers should be able to define multiple file managers in the PCS so they are exposed to services like OPSUI | ||
| 2 | Query execution location | A user using a file manager client, should be able to tell the file manager client it only wants to search the local file manager, or search a specific remote, or search every available file manager for data. | ||
| 3 | Coping with broken communications | The File Manager should be able to cope with remote nodes going offline or becoming unavailable and fail gracefully. | ||
| 4 | Schema alignment/matching | A user wishes to execute a query without knowing the underlying data model, on more than one file manager server. The query should be executed regardless and the relevant content returned. | Must Have |
User interaction and design
If you use filemgr client then a very simplistic implementation would be to extend the service to allow definitions of multiple local and remote file managers and allow the user to execute a query over each file manager and concat the result. This would be very quick to implement but doesn't scale well or support other services.
Phase 1
Currently OPSUI and the PCS platform only allows you to define 1 file manager per opsui instance, but to implement this feature successfully the OPSUI configuration needs to allow for multiple file managers, and when a user looks at the summary page the summary should show a grouping of all the available file managers, whilst also allowing the ability to filter by file manager.
In summary the following changes will be made:
- Registration of multiple file managers within PCS/OPSUI
- Changes made to the OPSUI monitoring pages to reflect multiple filemanagers
- Changes made to the Product listing pages to allow display of content from multiple filemanagers and filter by file manager
- Additional configuration within filemgr client to allow for lookup of available filemanagers from PCS and query them.
Phase 2
Phase 2 of the enhancement would involve adding further distributed capabilities to the filemanager(and possibly PCS platform as a whole), by adding in an optional zookeeper configuration that would allow for nodes self registering, graceful handling of nodes disappearing and also leader election and so forth. I feel enhancing OODT with industry standard distributed configuration management that is already widely used in "Big Data" type deployments will help with scalability of the platform and resiliency over distributed locations.
Questions
Below is a list of questions to be addressed as a result of this requirements document:
| Question | Outcome |
|---|---|
1 Comment
Lewis John McGibbney
Tom Barber I added a number 4 to the requirements table. We discussed this offline.. please feel free to reword.