Proposal For Client-Side Changes To Support Dynamic Server Configuration
In the existing client implementation, a static server list is provided to the ZooKeeper constructor while creating a client object. If the server list contains host names, then the name to IP mapping is performed only once. The server list as well as the resolved IP addresses remains unchanged. With dynamic reconfiguration , of ZooKeeper servers, ZooKeeper clients need a way to determine the most up-to-date server list.
A. Overview of proposed changes
A client can potentially resolve the new server list by periodically querying an external resource. For example, the ZooKeeper client library can take a reference to a HTTP URL (that records the server list) and periodically refresh the list from the URL. In addition to high availability of the resource, there three main questions to answer:
- What resources should we support?
- How would we update them? What would be the security mechanisms used to access them?
- How would we ensure that the external resource has the correct version of the server list, especially, when concurrent server reconfigurations are in progress?
It is difficult to implement a generic mechanism in the client library to address these problems. There are many options for external resource (HTTP, DNS, NFS, LDAP, virtual IP, etc). From security mechanisms view point; there are many solutions as well and some of them could require tedious setup. Further, the process of updating a resource is not simple. The process that updates the external resources must ensure that only the most recent server list gets recorded at the resource. We could add the capability in the client library that records a union of the old and new server configuration at the URL, performs the reconfiguration, and then updates the URL to with the most modified server list. However, the client library would also need to handle failures that may occur in this 3-step process. Further, if the application performs concurrent reconfigurations (which is supported by the server), then the client library would need to resolve race conditions to ensure that the result of last successful configuration is correctly recorded. In order to synchronize updates, the client library would need the external resource to support some atomic primitives (e.g., for locking and fencing).
Considering the complexity of the problem, in the first version, we propose to not implement any mechanisms to update the external resource. The ZooKeeper application would need to implement its own mechanism to update the URL. We expect that in most cases the change in ZooKeeper configuration will be an administrative process. Therefore, the Administrator can easily ensure that the URL is kept up-to-date. When we gain more experience in the future, we could consider adding this part the client library.
We propose to use two additional lookup mechanisms (as proposed in ZOOKEEPER-390):
- HTTP URL
- File URL that could potentially point to a path exported by a remote file server
Since the external resource needs to be highly available not all applications would be able to benefit from this approach. Applications that do not have access to such a resource may choose to implement their own mechanism by broadcasting the server list to its ZooKeeper clients. We propose to add new methods to the client that the application can use to set the server list.
To summarize, in the first round of development, we propose to:
- Add support for HTTP and file URL
- Add support to periodically retrieve the server list from the URL
- Allow application to set the server list
- Leave the responsibility of updating the URL to the application
B. Automatic refresh of server configuration
We will support Web server and a file as external resource. The reference to the external resource will be provided to the ZooKeeper constructor as string (see ZOOKEEPER-390).
The following formats will be supported by the client library:
- h1:port1,h2:port2/path (current format unchanged)
- zk://h1:port1,h2:port2!/path (new format to specify host:port connection string)
- http://abc.com/serves.txt!/path (use http to retrieve the server list)
- file:///a/b/c/server.txt!/path (read the server list from a file)
The optional path at the end of the connection string specifies the chroot. Note that to allow applications to be compiled with the new library without any modification, the current connection string format will also be supported in addition to the URL. If existing ZooKeeper applications do not require dynamic membership or periodic refresh mechanisms and prefer to use the host:port connection string format, then the application's clients will work with the new version without any modifications to the clients. Existing ZooKeeper constructors remain unchanged.
If the URL protocol is either HTTP or file, then the client library will periodically check the resource and update its in-core server list. If the protocol is “
zk” or the traditional connection string is used, then the client library will periodically resolve the DNS names, if needed. The refresh and/or name resolution will be enabled by default.
Each client will retrieve the list from the URL after
n + r seconds. By default
n will be set to 30 minutes and
r will be randomly chosen from 0 to 5 minutes.
Following methods will be added to the ZooKeeper class:
ZooKeeper.disableURLRefresh()-Tells the client library to disable refresh if enabled.
ZooKeeper.enableURLRefresh()- Tells the client library to enable refresh if disabled.
ZooKeeper.setRefreshInterval(int n, int r)- Sets the frequency of refresh to n seconds.
TBD: We could add a new constructor to specify these options, but it would be good to avoid creating new constructors.
If the server list retrieved from the URL does not include the server that the client is currently connected to, then the client will disconnect from the server and attempt to reconnect to other servers (using the new list). Otherwise, the new list will be used next time when the client attempts to connect to a server.
The server list stored in the URL will be the usual ZooKeeper configuration file (
zoo.cfg) with the following exceptions:
- A new optional property “
version” will added to the configuration file. The version number should match with the version number of the current server configuration returned by
ZooKeeper.getConfig()(see , for further details). The version number, when specified, will be used to ensure that the client will be choose the new server list only if the version number of the list is higher than the current version number.
- The “
server.x=host:serverPort:electionPort” property will be changed to “
Rest of the properties specified in the URL will be ignored.
The client will refresh from the URL every (
n + r) seconds. In addition, the client will trigger a refresh (by waking up the refresh thread) if
m successive connection attempts fail. The default value of
m will be set to (
s/3 + 1), where
s is the size of the server list. The value can be set by “
zookeeper.refreshAfterNumFailure” system property. If the application wishes to refresh every time the client disconnects from the server then this property can be assigned a value 0.
C. Letting the client application specify the server configuration
For applications that want to use their own independent discovery and refresh mechanism, we will add a new method to ZooKeeper class.
resetServerList(Properties serverConfig): This method will set the server list for the client. If the server list does not include the server that the client is currently connected to, then the client will disconnect from the server and attempt to reconnect to other servers (using the new list). Otherwise, the new list will be used next time when the client attempts to connect to a server. The format for serverConfig argument is described above in section B. If
null and if refresh is enabled, then the client will trigger a refresh from the URL.
Note that this allows the application to update the server list (in a best effort manner) by using watches. The application can set a watch on the configuration znode (
/zookeeper/config) that records the last committed server configuration. Once the watch is triggered, the application can read the configuration and call
D. Future extensions
- A tool to update a password protected HTTP URL.
- Avoid URL refresh from the same JVM for each client. This problem is similar to implementing shared sessions.
- Support for DNS based discovery at the client?
- For applications that want to use protocols other than file or URL, we could provide a callback mechanism. When the client library needs to retrieve the server list, it will invoke the callback method in the application. It will be responsibility of the application to discover the server list.
- For applications do not have an external web server or file server to record the configuration, we can implement the solution using virtual IPs described in ZOOKEEPER-1031.
E. Related JIRAs
 JIRA for dynamic cluster membership. https://issues.apache.org/jira/browse/ZOOKEEPER-107
 Cluster Membership. https://cwiki.apache.org/confluence/display/ZOOKEEPER/ClusterMembership