1 Introduction
Federation of MBeanServers means that Member-1 has a master MBeanServer which provides a proxied view of all the MBeans that Member-1 hosts. Federation will also involve cascading of operations and notifications across master and managed members.
This specification defines Federation as service for gemfire mbeans and some platform mbeans as well as any MBean as desired by the User.
1.1 Requirements
The Mbean federation strategy focus on the below mentioned requirements.
- It should be scalable.
- Performance impact of federation should be minimal. Push-based model is advisable as opposed to standard way of proxy like RMI Pull-method based proxies.
- User should be able to monitor the Distributed System from any third party tool compliant with JMX. User should be able to reach all mbeans from managing node as well as managed nodes
- Management API should provide mechanism for filtering out mbeans in-order to limit impact on scalability in large-systems.
1.2 Assumptions
- Strict consistency across managing member and managed members is not desired.
- Strict real time data is not mandated, its alright as the system would give a close enough picture.
1.3 Scope
This spec only points out federation strategy for Mbeans. The Mbean modeling and details about the Mbeans are mentioned in related document section.
1.4 Related Document
- https://cwiki.apache.org/confluence/display/GEODE/Proposed+GemFire+MBeans
- https://cwiki.apache.org/confluence/display/GEODE/MBeanServer+Federation+Strategy
- https://cwiki.apache.org/confluence/download/attachments/61309759/JMXFederation-POC.pdf
1.5 Terminology
Managed Node : Each member of Distributed system is a Managed node. It can be monitored independently as well as from a single agent view.
Managing Node : The member which will provide the single agent view to monitor and manage all the members. There can multiple managing nodes in the system
Hidden Region : This term is used for internal meta-data region that will be used for replication of MBean states.
2 Overview Of GemFire Federation
2.1 GemFire Services For Management & Monitoring
GemFire will provide following services for Management and Monitoring.
- Single Agent View of Distributed System MBeans
- Management Resources : Connectors and predefined MBeans.
- Inbuilt Federation Mechanism
2.2 Single-Agent View
With a single-agent view, the application or manager interacts with one MBeanServer. Application and manager development is much easier because they don't have to find the right MBeanServer to make a request on an MBean. That MBeanServer is responsible for aggregating all of the MBeans in the MBeanServers with which it is federated. This is very convenient because the application or manager can interact with a local MBeanServer that has services and adapters to interact with other local or remote MBeanServers. The location of the federated MBeanServers does not change how the application or manager is programmed to interact with its MBeanServer. For the sake of this discussion, we will refer to the single-agent MBeanServer as Managing Node. The MBeanServers that the master interacts with (and the application does not) will be referred to as Managed Node.
GemFire achieves Single-Agent View by creating MBean proxies at Managing Node side and keeping their state in sync with their Managed node counterpart.
2.3 Management Resources
2.3.1 Managed Node
Any manageable node which is not managing other members is managed node.
Managed node has following resources allocated so that it can answer jmx queries both locally and remotely.
- An RMI Connector which will allow JMX Client to connect directly to the node to and get access to all local MBeans.
- Local MBeans : representing the local monitored components on this node.Built in platform MBeans
- MemberMBean
- RegionMBean
- ClientServiceMBean
- LockServiceMBean
- DiskStoreMBean
- GatewaySenderMBean
- GatewayReceiverMBean
- LocatorMBean
- Management MBean with Scope=LOCAL
- Built in platform MBeans
- Custom MBeans
2.3.2 Managing Node
Managing node has following extra resources allocated so that it can answer jmx queries
- An RMI Connector which will allow JMX Client to connect to and get access to all MBeans of the distributed system
- Local MBeans : representing the local monitored components on this node. Same as any other managed node
- Aggregate MBeans :Management MBean with Scope=ALL which allows various DS-wide operations.
- DistributedSystemMBean
- DistributedRegionMBean
- MemberGroupMBean
- DistributedLockServiceMBean
- Management MBean with Scope=ALL which allows various DS-wide operations.
- Proxy to MBeans on managed nodes
- Built in platform MBeans
- Custom MBeans
2.4 Inbuilt Federation Mechanism
GemFire federation mechanism is the process by virtue of which we can achieve single agent view.The protocol used is internal to GemFire and does not interfere with any application and manager implementation.
GemFire federation takes care of the following functionality
- MBean proxy creation
- MBean state propagation
- notifications propagation
- operation invocation.
GemFire federation is not limited to predefined MBeans only. In fact users can write their own MBeans and GemFire will provide a federated view of their MBean.
2.4.1 MBean/Proxy Naming Convention
Each GemFire MBean will follow a particular naming convention for easier grouping
e.g. GemFire:type=Member,service=LockService,name=<dlsName>,memberName=<memberName>
At the managing node this MBean will be registered with GemFire/<memberId> as domain
e.g. GemFire:type=Member,service=LockService,name=<dlsName>,memberName=<memberName>
2.5 Application Program Interfaces
GemFire Management API represent the Gemfire System view to a JMX user. However it does not intend to provide functionality which is otherwise present in JMX. It only provides a gateway into various services exclusively offered by GemFire Management.
The entry point to GemFire Management is through interface ManagementService
ManagementService service = ManagementService.getManagementService(cache);
The implementation of getManagementService is a singleton for now but it allows us to eventually support multiple cache instances (probably multiple client caches and on peer cache). The resulting ManagementService instance will be specific to the provided cache and its distributed system. Also the instance will behave differently when the cache is a client cache vs a peer cache. When its a client cache then execution of a command string should be forwarded to the cache server instead of processed in the local JVM. More detailing needs to be done for client cache.
2.5.1 APIs useful at Managed Node
public class ManagementService { /** * It will a singleton instance across a Cache. This instance will * act as th einterface between user and GemFire Management. * * @param Cache * @return ManagementService */ public static ManagementService getManagementService(Cache cache) { } /** * This will register the MBean in default GemFire domain. * Any MBean which wishes to register itself in Gemfire * domain should use this method. * Note: The ObjectName will be modified here. Hence returning the * modified ObjectName. */ public ObjectName registerMBean(Object object, ObjectName objectName) { } /** * For consistency and clean up of GemFire resources * use this method to unregister the MBean from GemFire Domain */ public void unregisterMBean(ObjectName objectName) { } /** * This method is used to mark an MBean for federation. * * Users may want to only register their MBean in GemFire domain. In that * case they don't have to call federate(); * * @param <T> * @param name MBean name * @param interfaceClass MBean interface class * @param notificationEmitter is the MBean a notification emitter * @return */ public <T> boolean federate(ObjectName name, Class<T> interfaceClass, boolean notificationEmitter) { } /** * This method is used to process a command * This takes an String is a parameter. * * @param commandString * @return */ public Object processComand(String commandString) { } /** * Interfaces to get all GemFire specific MBeans * */ public RegionMXBean getLocalRegionMXBean(Region region) { } public LockServiceMXBean getLocalLockServiceMXBean(String name){ } ........ }
2.5.2 APIs useful for MBean Federation
Usage:
MyCustomMXBean myMbean = new MyCustomMBean(); ManagementService service = ManagementService.getManagementService(cache); //To register in Gemfire domain // GemFire will modify the given ObjectName to append memberName ObjectName newObjectName service.register(myMbean, name); /** * To allow MyCustomMBean to be federated */ service.federate(newObjectName, MyCustomMXBean.class, false);
2.5.3 APIs useful at Managing Node
public class ManagementService { /** * This method will be invoked whenever a member wants to be a managing node. The * exception Management exception has to be handled by the caller. */ public void startManager() { } /** * This method will be invoked whenever a member stops being a managing node. * The exception Management exception has to be handled by the caller. * This will stop all the activities and clean up resources pertaining to * Managing Node */ public void stopManager() { } ........ /** * This will return a set of MBean proxy names for a given member name * * @param memberName */ public Set<ObjectName> queryProxyMBeans(DistributedMember memberName) { } /** * The method will return the proxy instance represented by the ObjectName. * * It will be an actual reference to the proxy instance rather than an * ObjectInstance. * * @param <T> * @param objectName * @return * @throws ManagementException * @throws InstanceNotFoundException */ public <T> T getMBeanProxy(DistributedMember memberName, ObjectName objectName, Class<T> interfaceClass) { } /** * This will return the last updated time of the proxyMBean * * @param memberName */ public long getLastUpdateTime(ObjectName objectName) { } public Object processComand(String commandString) { } /** * Methods to get remote MBean proxy reference */ public RegionMXBean getRemoteRegionMXBean(DistributedMember memberName, Region region) { } public LockServiceMXBean getRemoteLockServiceMXBean(DistributedMember memberName, String name) {} ....................... }
3 Architectural Description
Internal Details The following section is internal details of GemFire and not visible to end user directly |
For JMX data distribution, Gemfire's inbuilt Replicated Region support will be used. This will help in data distribution in one direction i.e. from Managed Node to Managing Node.
Replicated region fits into push based monitoring support easily.
For information flow from managed node to managing node, Gemfire's "Function Service" will be used. Functions are primarily for all operations and propagating attribute setters.
3.1 Transition from Managed Node to Managing Node
The mechanism for transition from Managed Node to Managing node will be part of Beaverton specs.
a) StartManagingNode : Will create Management resources namely Regions, proxies for all the connected members
b) StopManagingNode : Will remove all the Management resources.It will behave as a normal Managed Node and start pushing its local data to the hidden regions
3.2 Management Listener
A new member wide listener component to be introduced to listen on different GFE resource creation/removal/destroy. On particular events it will create/ destroy Mbeans. This will save us from unnecessary existence of Mbeans. However this will require some internal GFE changes.
Other internal components can also be initialized by this listener. e.g. HiddenRegion can be created by this listener on cache create event.
public final class InternalDistributedSystem{ private void initialize() { ...................... ManagementListener listener = new ManagementListener(); this.managementListener = listener; } } public enum ManagementEvent { CACHE_CREATE, REGION_CREATE, DISKSTORE_CREATE, CACHESERVER_CREATE, GATEWAYRECEIVER_CREATE, GATEWAYSENDER_CREATE, LOCKSERVICE_CREATE, CACHE_REMOVE, REGION_REMOVE, DISKSTORE_REMOVE, CACHESERVER_REMOVE, GATEWAYRECEIVER_REMOVE, GATEWAYSENDER_REMOVE, LOCKSERVICE_REMOVE } public class ManagementListener { public void handleEvent(ManagementEvent event){ } } class GemFireCacheImpl{ init(){ .............. system.getManagementListener().handleEvent(ManagementEvent.CACHE_CREATE, this); .......... } }
3.3 Federation Strategy
Three main functionalities of any MBean : Attributes, Operation and Notifications are federated in following way
- Attribute value are federated using GemFire Hidden Region
Central component for Federation is FederatedComponent which will store object state as well as some meta data like MBean interface , notification enabled etc. - Operations are federated using GemFire Function Service
- Notification are federated using Hidden Region and Function Service together.
Each member of the system will be represented by one hidden region. The regions can be named "MONITORING_REGION_<memberName>". Whichever node becomes a Managing Node it will replicate all the hidden regions created by its managed member. Hidden region will store MBean state and other meta-data about mbean and its notifications.
3.4 Federation at managing member
This section describes the events that happen when a node makes transition from Normal Node to Managing node
3.4.1 Query cluster to know all manageable members
Managing member has to discover set of members in the current distributed system. Very first step would be to know all manageable members. This includes members with fixed-persistent memberName as well as member with non-persistent ID. This section assumes mechanism for querying all member of system. The results of this operation should return memberName and its descriptor. The descriptor would tell following attributes of the member
- memberName
- Type : Persistent, Non-Persistent
- Services running : Peer, CacheServer, Locator, Managing or Managed
3.4.2 Creation of Hidden Regions
Managing node has to communicate to all managed node and exchange all mbean informration i.e. what all mbeans are hosted by each managed node. Once this set is established managing node can finalize the list of proxies that it needs to create.
As we will be using GemFire hidden regions while the Managing member comes up GII functionality will take care of this.
If Managing member is up and a new member comes up then MembershipListener will take care of this.
3.4.3 Create Proxy for remote MBeans
For this purpose Java's Proxy Design pattern is used. Our proxies are similar to JMX proxies. Only underlying InvocationHandler is different.
JMX delegates all call to remote RMI connnection. Calls on Gemfire Management proxies will be delegated to the state containing FederatedComponent and FunctionService.
Reference : javax.management.MBeanServerInvocationHandler
3.4.4 Method Invocation in Proxy MBean
- All getter method on the proxy will be delegated to the contained FederationComponent
- All setter metohd on the proxy will be delegated to an instance of GenericMBeanFunction
- All other opertaions on the proxy will be delegated to an instance of GenericMBeanFunction. Flow same as setter functions
Method Invocationin Proxy MBean(If proxy is provided explicitly)
GemFire MBeans will implement an interface which will serve two purposes
a) System can identify GemFire specific MBeans
b) Any proxy implementation can be given by this MBean which will by pass above generic proxy implementation
All the getters, setters, and operations will be delegated to this proxy. As proxy will be written by the developer who develops the MBean he will have the information how to access the data locally.
Following listeners will trigger creation and deletion of proxy mbeans
- MembershipListener ( Whenever a member joins or departs we need to create /destroy proxies)
- ManagementCacheListener ( Whenever a region operation happens on the MONITORING_REGION)
- ManagingNodeTransitionListener ( Whenever a member becomes a Managing node)
Data Model
Gemfire MBean state consists of Statistical information maintained by Gemfire Statistics and other information like configuration and other dynamic information not covered by gemfire statistics.
Non-Gemfire MBeans (Platform MBeans, User Defined MBeans) will be represented as it is.
Data Model - Managed Node
1 Data model
Each MBean state will be represented by a FederatedComponent as per the proposed model. FederatedComponent will also be behave as an entry for the hidden region for data distribution.
The relationship between MBean, FederatedComponent is depicted below.
2 Bridge Data model
For Gemfire MBeans an intermediary MBeanBridge will be implemented to act as a conduit between GFE resource and MBeans.
Note MBeanBridge is only for Gemfire MBeans for ease of programming. Other MBeans are free to have their implementation as they require. |
3 Federated MBeans
If a MBean is intended to be federated then Gemfire should have some means to get the state data to be propagated by hidden region.
For that an FederatedComponent is used.
It uses MBeanInfo to dynamically get attribute & attribute values and keep it in an ObjectState map as a key value pair.
In addition it also contains some meta data like MBean interface, notification capabilities etc.
4 Delta Approach and other optimizations
This is deferred till implementation phase after checking the number of attributes which gets change frequently. If the percentage. is quite high then Delta might not be a good idea.
Data Model - Managing Node
1 Proxy Data model
This will include proxies for Global scoped User Defined MBeans and of course Gemfire MBeans
The below diagram depicts the relationship between various components for Proxies.
- Delta Approach and other optimizations
This is deferred till implementation phase after checking the number of attributes which gets change frequently.
Push Model for MBean State
Generic MBeans
All user MBeans and Platform MBeans can be federated provided they follow the above mentioned contract with Gemfire.
Gemfire will internally scan through all such MBeans and push their state to the hidden region in a given frequency.
Gemfire Specific MBeans
MBean state consists of Statistical information maintained by Gemfire Statistics and other information like configuration and other dynamic information not covered by gemfire statistics. Following diagram shows how M&M layer gets data from gemfire statistics. ManagementThread is used for pushing updates from managed node to manging node via hidden region replication. ManagementThread will group and batch all updated in-order to reduce network traffic.
Note MBeanBridge is only for Gemfire MBeans for ease of programming.They contain the MBean state. They can be updated by the Listeners directly as we are free to code the listener object |
- Non-Statistics Part
This part of MBean state is collected from Bridge component of underlying gemfire resources. - Statistics Part
Statistics Listener provided by Statistics framework is used for updating mbean state. There is not direct invocation or signaling mechanism between StatisticsSampler Thread and Management Thread. ManagementThread is responsible for getting updating statistics part of Mbean State.
StatisticsListener listener = new StatisticsListener() { public void handleNotification(StatisticsNotification notification) { // handle the notification } }; ValueMonitor monitor = new ValueMonitor() .addStatistic(GemFireStatistics.getCreatesDescriptor()) .addStatistic(GemFireStatistics.getDestroysDescriptor()) .addStatistic(GemFireStatistics.getEntriesDescriptor()) .addStatistic(GemFireStatistics.getServerStatistics().getCurrentClientsDescriptor() .addStatistic(GemFireStatistics.getServerStatistics().getSubscriptionQueueSizeDescriptor()); monitor.registerListener(listener);
MBean Operation on Remote MBeans
MBean Operation-Function Mapping
For the time being one function class GenericMBeanFunction is proposed.
This will be used :
- For all setter functions of a MBean
- For all operations of a MBean
- Notification client add/ remove
Function Attributes
- hasResult : It will always be true. Even if MBean operation has void as return type. This is to handle error conditions and suitable messaging
- isHa : Always false
Function Registration
ManagementService instance will register the function
Failed Function Results
If by any means the function execution fails (e.g. by unreachable members) it will return UNDEFINED string as a result.
Notifications
To understand the proposed notification federation model let's look briefly at the distributed management scope.
It can be depicted roughly in the below mentioned layers.
JMXClient Layer | <- RMI Protocol -> | Managing Proxy Layer | <- GemFire Protocols -> | Managed MBean Layer |
---|
Each layer runs on different VMs.
GemFire implementation of proxy will be a little different than other JMX federation approaches.
GemFire proxies inherently are local MBeans.It will emit notification when an event occurs.
This event is mapped to notification emitted by the actual MBean on Managed Node side. So some component(in our model its NotificationHub) at Managed Node
side should capture the notifications and send a message to Managing Node for actual JMX notification emit.
Description
Federation of notification is implemented using NotificationHub. Hub subscribes for notifications in the managed node and forwards them to managing node. On managing-node HubClient will fan-out the notification to correct MBean proxy. Proxy then uses standard JMX NotificationBroadcasterSupport class for forwarding up to JMX stack and JMX RMI Connector.
This approach allows us to achieve following requirements- Does not disturb standard JMX notification flow
- Supports standard JMX notifications for RMI connector on managed node as well as manging node
Notification federation will be supported in a sub region of MONITORING_REGION. It will be named NOTIFICATION_REGION which will be an empty region.
If the region is not an empty region we would have to delete the notification from the region which will cause an extra message.
Assumptions
- MBean is using NotificationBroadcasterSupport for emitting notifications. Thus on client side Proxy generated also extends NotificationBroadcasterSupport.
- All notifications generated by MBean marked as Federated are emitted to Managing node. There is no filtering on type of notification within a single MBean
Flow at Managed Member
Flow at Managing Member
Notification Subscription
NotificationHub should not send all the notifications originating in its VM . Rather it should send messages for only those notifications for which a client has expressed interest.
Whenever client express interest for notifications of a particular MBean an instance of NotificationListener will be registered at Managed Node side.
This listener will do the job of listening to notifications locally and put data in the hidden notification region.
- addNotificationListener
- removeNotificationListener
Proxy Class : MBeanProxyInvocationHandler
MBeanServer mbeanServer = ManagementFactory.getPlatformMBeanServer(); public Object invoke(Object proxy, Method method, Object[] args) throws Throwable { final Class methodClass = method.getDeclaringClass(); if (methodClass.equals(NotificationBroadcaster.class) || methodClass.equals(NotificationEmitter.class)) return invokeBroadcasterMethod(proxy, method, args); ............. ............... } private Object invokeBroadcasterMethod(Object proxy, Method method, Object[] args) throws Exception { if (methodName.equals("addNotificationListener")) { mbeanServer.addNotificationListener(mbeanName, listener, filter, handback); delegateToFucntionService();// For registering a NotificationListener at Managed Node side } if (methodName.equals("removeNotificationListener")) { mbeanServer.removeNotificationListener(mbeanName, listener, filter, handback); delegateToFucntionService();// For removing NotificationListener from Managed Node side } }
Some conceptual details for remote notifications can be found at
Remote Notifications Section: http://mx4j.sourceforge.net/docs/ch03s04.html
Gemfire alerts
Gemfire alert mechanism will also use Federation services backed by replicated regions. Current approach uses peer-to-peer messaging. Earlier alerts were consumed by remote listeners installed on JMX Agent.Now those messages or alerts will be consumed by NotificationHub.
A default AlertNotificationBroadcaster will be registered for each node. ManagerLogWriter.notifyAlertListeners will invoke the notification broadcaster.
This broadcaster will collect all the alerts depending on the alert level depending on the "alertLevel" attribute of ManagementMBean.The alerts will be propagated to the notification region.
Following diagram shows two flows
- Gemfire Warning messages getting federated as JMX Notifications across DS
- Custom Statistics based Monitor generating federated JMX notifications. Monitor concept is derived from ResourceMonitor and also similar to JDK provided Monitor and GaugeMonitor MBeans. Difference is all logic is executed on ManagementThread instead of JDK Monitor where each object uses its own thread.
Diagram also shows EmailNotifier as another consumer of notification. Such notifiers can be installed on Managing Server where HubClient can use this to send emails to gemfire users or any other notification consumption mechanism other than JMX
Network Partitioning Handling
- Managing Node on Winning Side : It will only emit notifications about crashed members
- Managing Node on Loosing Side : Managing Node will shutdown itself. Before shutdown the alert emitted will contain info about Network Partition with failure set information
3.Future Direction
Management Filters
Management filter are defined to restrict or narrow-down the management and monitoring to limited set of resources or limited number of members. Filter will help to scale M&M in a large distributed system consisting of hundreds or member.
Typical filter consists of inclusion and exclusion lists. It mimics firewall rules, hence inclusion and exclusion. Inclusion is processed first and defines what to include. Exclusion is processed second and defines what to exclude out of what was already defined is being included.
Types of Filters
Three types of filters are defined :
FilterName Functionality Inclusion List Exclusion List Description Local MBean Filter Determines what local MBeans will be registered if ObjectName matches any then it will be created/registered; default: * (all) if ObjectName matches any then it will be ignored; default: null localMBeanInclusion: *
localMBeanExclusion: null
So by default "*" causes all local MBeans to be included. This means means that every single MBean relevant to the local member will be instantiated and registered. For example, every Region will result in a RegionMBean. Since the exclusion is "null", none of the MBean types would be excluded. But you could change the localMBeanExclusion to be regex pattern ".*Region.*" which results in excluding of the RegionMBeans for each Region in the local member (because the ObjectName for RegionMBeans contains "Region").
This is applicable only when SCOPE attribute of ManagementMBean is LOCALRemote MBean Filter determines what remote MBeans will be federated by proxy MBeans if ObjectName matches any then it will be created/registered; default: * (all) if ObjectName matches any then it will be skipped; default: null remoteMBeanInclusion: *
remoteMBeanExclusion: null
The above are ignored if Scope is "Local". But if Scope is "All" then the local JVM is a management node and it will create a hidden region for every other member in the distributed system. It will then create federated versions of every MBean in those members as determined by remoteMBeanInclusion/remoteMBeanExclusion. remoteMBeanInclusion "*" means that all remote MBeans will be federated. remoteMBeanExclusion "null" means that no remote MBeans will be excluded from this.Managed Member Filter determines what members will be managed (federated) if memberName matches any then it will be managed (federated); default: * (all) if memberName matches any then it will be ignored; default: null memberInclusion: *
memberExclusion: null
memberGroupInclusion: *
memberGroupExclusion: null
Thus by default all members are included and federated if the local JVM is a management node. Meaning this node will create federated views of the MBeans (that match the remote mbean filter) in every member. If an exclusion other than null is set then any member matching that exclusion would be excluded from the federated view. The values of all these inclusion/exclusion properties should support regex pattern such as "Publisher.*" as well as constant strings such as "Server-1" as well as LISTS such as "Publisher.*;Subscriber.*" (note that someone needs to pick a reasonable delimiter for lists of regex patterns) or even a mix such as "Publisher-A;Subscriber-B;Server.*".
memberGroupInclusion/memberGroupExclusion are more important for large memberships and allows someone to limit their monitoring or managing to one server group or one logical group of members (or even more than one server group).
Exact syntax of these filters is still TBD.
Example: if you defined the following... memberInclusion: ".\*Publisher.\*;.\*Subscriber.\*" memberExclusion: "Trade.\*" ...you would end up including members with member-ids such as: OptionPublisher-1 OptionPublisher-2 OptionPublisher-3 OptionSubscriber-1 StockPublisher-1 StockPublisher-2 But the following would be excluded: TradePublisher-1 TradePublisher-2 TradeSubscriber-1
How filters are specified
- Gemfire configuration mechanism
Behavior on Filter
Filters will be picked up when a node starts up and those filters will be in force during the lifetime of the member node. Change in filters will require member node restart.