Description
This document describes how to change the communication between WAN sites from Non-SSL to SSL one site at a time without stopping the cluster.
Steps
The steps to do this are:
Initially Non-SSL sites A and B are running.
- Stop accepting client connections in non-SSL site A (clients connect to non-SSL site B)
- Stop non-SSL site A once all queues are drained
- Stop receivers in non-SSL site B
- Pause senders in non-SSL site B
- Do puts in non-SSL site B (events are queued)
- Start SSL site A
- Start non-SSL proxy in site A
- Resume senders in non-SSL site B (site B will connect to the non-SSL proxy in site A, and its queues will drain)
- Stop non-SSL site B once all queues are drained
- Stop non-SSL proxy in site A
- Start SSL site B
Now SSL sites A and B are running.
Note: There are two race conditions in the above steps, namely:
In step 2, events in site A's queue may not have been sent but clients are connected to site B.
In step 9, queues may never drain as long as clients are connected to site B.
Example
The attached ssl-proxy-clean_09-12-2016.tgz example encapsulates the above steps.
Initialization
Modify the GEMFIRE environment variable in the setenv.sh script to point to a GemFire 8.2.x installation.
In each terminal window, source the setenv.sh script like:
. ./setenv.sh
Make sure that ANT_HOME/bin and JAVA_HOME/bin are also on the PATH.
Set the following GemFire properties in the config/gemfire-security.properties file to appropriate values:
gateway-ssl-keystore
gateway-ssl-keystore-password
gateway-ssl-truststore
gateway-ssl-truststore-password
Start Locators
Start locators in both sites using the startlocators.sh script like:
./startlocators.sh Starting locator ny on port 10331 ................................. Locator in /path/to/locator-ny on boglesbymac-2[10331] as locator-ny is currently online. Process ID: 50276 Uptime: 17 seconds GemFire Version: 8.2.1.2 Java Version: 1.7.0_80 Log File: /path/to/locator-ny/locator-ny.log JVM Arguments: -DgemfirePropertyFile=/path/to/config/gemfire-ny-locator.properties -Dgemfire.enable-cluster-configuration=true -Dgemfire.load-cluster-configuration-from-dir=false -Dgemfire.launcher.registerSignalHandlers=true -Djava.awt.headless=true -Dsun.rmi.dgc.server.gcInterval=9223372036854775806 Class-Path: ... Successfully connected to: [host=boglesbymac-2, port=1091] Cluster configuration service is up and running. Starting locator ln on port 10332 ................................ Locator in /path/to/locator-ln on boglesbymac-2[10332] as locator-ln is currently online. Process ID: 50293 Uptime: 16 seconds GemFire Version: 8.2.1.2 Java Version: 1.7.0_80 Log File: /path/to/locator-ln/locator-ln.log JVM Arguments: -DgemfirePropertyFile=/path/to/config/gemfire-ln-locator.properties -Dgemfire.enable-cluster-configuration=true -Dgemfire.load-cluster-configuration-from-dir=false -Dgemfire.launcher.registerSignalHandlers=true -Djava.awt.headless=true -Dsun.rmi.dgc.server.gcInterval=9223372036854775806 Class-Path: ... Successfully connected to: [host=boglesbymac-2, port=1092] Cluster configuration service is up and running.
Start Non-SSL Servers
Start the ln-1 non-SSL server using the ln-1 ant task like:
ant ln-1 ... gateway: [java] Created GemFireCache[id = 1623717470; isClosing = false; isShutDownAll = false; closingGatewayHubsByShutdownAll = false; created = Wed Aug 31 17:09:03 PDT 2016; server = false; copyOnRead = false; lockLease = 120; lockTimeout = 60] [java] Retrieved region: Partitioned Region @165caedc [path='/data'; dataPolicy=PERSISTENT_PARTITION; gatewayEnabled=false; prId=1; isDestroyed=false; isClosed=false; retryTimeout=3600000; serialNumber=10; partition attributes=PartitionAttributes@159930692[redundantCopies=1;localMaxMemory=444;totalMaxMemory=2147483647;totalNumBuckets=113;partitionResolver=null;colocatedWith=null;recoveryDelay=-1;startupRecoveryDelay=0;FixedPartitionAttributes=null;partitionListeners=null]; on VM boglesbymac-2(ln-1:50309)<v1>:62011] [java] ParallelGatewaySenderImpl[id=ny; socketReadTimeout=0] [java] Region named data size: 0 [java] Region /ny_PARALLEL_GATEWAY_SENDER_QUEUE (a ParallelGatewaySenderQueueMetaRegion) size=0; primarySize=0; entrySetSize=0; entrySetArrayLength=0; primaryEntrySetSize=0; primaryEntrySetArrayLength=0
Start the ln-2 non-SSL server using the ln-2 ant task like:
ant ln-2 ... gateway: [java] Created GemFireCache[id = 745001724; isClosing = false; isShutDownAll = false; closingGatewayHubsByShutdownAll = false; created = Wed Aug 31 17:09:03 PDT 2016; server = false; copyOnRead = false; lockLease = 120; lockTimeout = 60] [java] Retrieved region: Partitioned Region @3a7cbada [path='/data'; dataPolicy=PERSISTENT_PARTITION; gatewayEnabled=false; prId=1; isDestroyed=false; isClosed=false; retryTimeout=3600000; serialNumber=10; partition attributes=PartitionAttributes@1866528453[redundantCopies=1;localMaxMemory=444;totalMaxMemory=2147483647;totalNumBuckets=113;partitionResolver=null;colocatedWith=null;recoveryDelay=-1;startupRecoveryDelay=0;FixedPartitionAttributes=null;partitionListeners=null]; on VM boglesbymac-2(ln-2:50310)<v2>:34250] [java] ParallelGatewaySenderImpl[id=ny; socketReadTimeout=0] [java] Region named data size: 0 [java] Region /ny_PARALLEL_GATEWAY_SENDER_QUEUE (a ParallelGatewaySenderQueueMetaRegion) size=0; primarySize=0; entrySetSize=0; entrySetArrayLength=0; primaryEntrySetSize=0; primaryEntrySetArrayLength=0
Start the ny-1 non-SSL server using the ny-1 task like:
ant ny-1 ... gateway: [java] Created GemFireCache[id = 1188435730; isClosing = false; isShutDownAll = false; closingGatewayHubsByShutdownAll = false; created = Wed Aug 31 17:09:04 PDT 2016; server = false; copyOnRead = false; lockLease = 120; lockTimeout = 60] [java] Retrieved region: Partitioned Region @4e69f4e7 [path='/data'; dataPolicy=PERSISTENT_PARTITION; gatewayEnabled=false; prId=1; isDestroyed=false; isClosed=false; retryTimeout=3600000; serialNumber=10; partition attributes=PartitionAttributes@1670717230[redundantCopies=1;localMaxMemory=444;totalMaxMemory=2147483647;totalNumBuckets=113;partitionResolver=null;colocatedWith=null;recoveryDelay=-1;startupRecoveryDelay=0;FixedPartitionAttributes=null;partitionListeners=null]; on VM boglesbymac-2(ny-1:50317)<v1>:65157] [java] ParallelGatewaySenderImpl[id=ln; socketReadTimeout=0] [java] Region named data size: 0 [java] Region /ln_PARALLEL_GATEWAY_SENDER_QUEUE (a ParallelGatewaySenderQueueMetaRegion) size=0; primarySize=0; entrySetSize=0; entrySetArrayLength=0; primaryEntrySetSize=0; primaryEntrySetArrayLength=0
Start the ny-2 non-SSL server using the ny-2 ant task like:
ant ny-2 ... gateway: [java] Created GemFireCache[id = 1765036803; isClosing = false; isShutDownAll = false; closingGatewayHubsByShutdownAll = false; created = Wed Aug 31 17:09:06 PDT 2016; server = false; copyOnRead = false; lockLease = 120; lockTimeout = 60] [java] Retrieved region: Partitioned Region @53d2e924 [path='/data'; dataPolicy=PERSISTENT_PARTITION; gatewayEnabled=false; prId=1; isDestroyed=false; isClosed=false; retryTimeout=3600000; serialNumber=10; partition attributes=PartitionAttributes@1233228555[redundantCopies=1;localMaxMemory=444;totalMaxMemory=2147483647;totalNumBuckets=113;partitionResolver=null;colocatedWith=null;recoveryDelay=-1;startupRecoveryDelay=0;FixedPartitionAttributes=null;partitionListeners=null]; on VM boglesbymac-2(ny-2:50324)<v2>:18596] [java] ParallelGatewaySenderImpl[id=ln; socketReadTimeout=0] [java] Region named data size: 0 [java] Region /ln_PARALLEL_GATEWAY_SENDER_QUEUE (a ParallelGatewaySenderQueueMetaRegion) size=0; primarySize=0; entrySetSize=0; entrySetArrayLength=0; primaryEntrySetSize=0; primaryEntrySetArrayLength=0
Stop ln Non-SSL Servers
Stop the ln non-SSL servers using the shutdown_ln_servers.sh script like:
./shutdown_ln_servers.sh (1) Executing - connect --locator=localhost[10332] Connecting to Locator at [host=localhost, port=10332] .. Connecting to Manager at [host=boglesbymac-2, port=1092] .. Successfully connected to: [host=boglesbymac-2, port=1092] (2) Executing - shutdown Shutdown is triggered
Stop ny Gateway Receiver
Stop the ny Gateway Receiver in both ny members using the stop_ny_gateway_receiver.sh script like:
./stop_ny_gateway_receiver.sh (1) Executing - connect --locator=localhost[10331] Connecting to Locator at [host=localhost, port=10331] .. Connecting to Manager at [host=boglesbymac-2, port=1091] .. Successfully connected to: [host=boglesbymac-2, port=1091] (2) Executing - stop gateway-receiver Member | Result | Message ----------------------------------- | ------ | ---------------------------------------------------------------------------- boglesbymac-2(ny-1:50317)<v1>:65157 | OK | GatewayReceiver is stopped on member boglesbymac-2(ny-1:50317)<v1>:65157 boglesbymac-2(ny-2:50324)<v2>:18596 | OK | GatewayReceiver is stopped on member boglesbymac-2(ny-2:50324)<v2>:18596
Pause ny Gateway Sender
Pause the ny Gateway Sender in both ny members using the pause_ny_gateway_sender.sh script like:
./pause_ny_gateway_sender.sh (1) Executing - connect --locator=localhost[10331] Connecting to Locator at [host=localhost, port=10331] .. Connecting to Manager at [host=boglesbymac-2, port=1091] .. Successfully connected to: [host=boglesbymac-2, port=1091] (2) Executing - pause gateway-sender --id=ln Member | Result | Message ----------------------------------- | ------ | ------------------------------------------------------------------------ boglesbymac-2(ny-1:50317)<v1>:65157 | OK | GatewaySender ln is paused on member boglesbymac-2(ny-1:50317)<v1>:65157 boglesbymac-2(ny-2:50324)<v2>:18596 | OK | GatewaySender ln is paused on member boglesbymac-2(ny-2:50324)<v2>:18596
Run ny Client Loader
Run the ny Client Loader to load entries into the ny site using the ny-client-load ant task like:
ant ny-client-load ... client: [java] Created GemFireCache[id = 2019921345; isClosing = false; isShutDownAll = false; closingGatewayHubsByShutdownAll = false; created = Wed Aug 31 17:14:52 PDT 2016; server = false; copyOnRead = false; lockLease = 120; lockTimeout = 60] [java] Retrieved region: com.gemstone.gemfire.internal.cache.LocalRegion[path='/data';scope=LOCAL';dataPolicy=EMPTY; concurrencyChecksEnabled] [java] Loaded 1000 entries in 2218 ms [java] Loaded 1000 entries in 2218 ms
The ny server terminal windows will log messages like:
[java] Region named data size: 787 [java] Region /ln_PARALLEL_GATEWAY_SENDER_QUEUE (a ParallelGatewaySenderQueueMetaRegion) size=868; primarySize=440; entrySetSize=870; entrySetArrayLength=879; primaryEntrySetSize=445; primaryEntrySetArrayLength=445 [java] Region named data size: 1000 [java] Region /ln_PARALLEL_GATEWAY_SENDER_QUEUE (a ParallelGatewaySenderQueueMetaRegion) size=1000; primarySize=508; entrySetSize=1000; entrySetArrayLength=1000; primaryEntrySetSize=508; primaryEntrySetArrayLength=508
Start ln SSL Servers
Start the ln-1 SSL server using the ln-ssl-1 ant task like:
ant ln-ssl-1 ... gateway-ssl: [java] Created GemFireCache[id = 1763038120; isClosing = false; isShutDownAll = false; closingGatewayHubsByShutdownAll = false; created = Wed Aug 31 17:18:48 PDT 2016; server = false; copyOnRead = false; lockLease = 120; lockTimeout = 60] [java] Retrieved region: Partitioned Region @6417f586 [path='/data'; dataPolicy=PERSISTENT_PARTITION; gatewayEnabled=false; prId=3; isDestroyed=false; isClosed=false; retryTimeout=3600000; serialNumber=10; partition attributes=PartitionAttributes@1145119737[redundantCopies=1;localMaxMemory=444;totalMaxMemory=2147483647;totalNumBuckets=113;partitionResolver=null;colocatedWith=null;recoveryDelay=-1;startupRecoveryDelay=0;FixedPartitionAttributes=null;partitionListeners=null]; on VM boglesbymac-2(ln-1:50522)<v5>:52207] [java] ParallelGatewaySenderImpl[id=ny; socketReadTimeout=0] [java] Region named data size: 0 [java] Region /ny_PARALLEL_GATEWAY_SENDER_QUEUE (a ParallelGatewaySenderQueueMetaRegion) size=0; primarySize=0; entrySetSize=0; entrySetArrayLength=0; primaryEntrySetSize=0; primaryEntrySetArrayLength=0
Start the ln-2 SSL server using the ln-ssl-2 ant task like:
ant ln-ssl-2 ... gateway-ssl: [java] Created GemFireCache[id = 1155454536; isClosing = false; isShutDownAll = false; closingGatewayHubsByShutdownAll = false; created = Wed Aug 31 17:18:49 PDT 2016; server = false; copyOnRead = false; lockLease = 120; lockTimeout = 60] [java] Retrieved region: Partitioned Region @4997439e [path='/data'; dataPolicy=PERSISTENT_PARTITION; gatewayEnabled=false; prId=3; isDestroyed=false; isClosed=false; retryTimeout=3600000; serialNumber=10; partition attributes=PartitionAttributes@1413562044[redundantCopies=1;localMaxMemory=444;totalMaxMemory=2147483647;totalNumBuckets=113;partitionResolver=null;colocatedWith=null;recoveryDelay=-1;startupRecoveryDelay=0;FixedPartitionAttributes=null;partitionListeners=null]; on VM boglesbymac-2(ln-2:50529)<v6>:14156] [java] ParallelGatewaySenderImpl[id=ny; socketReadTimeout=0] [java] Region named data size: 0 [java] Region /ny_PARALLEL_GATEWAY_SENDER_QUEUE (a ParallelGatewaySenderQueueMetaRegion) size=0; primarySize=0; entrySetSize=0; entrySetArrayLength=0; primaryEntrySetSize=0; primaryEntrySetArrayLength=0
Start ln non-SSL Proxy
Start the ln non-SSL proxy server using the ln-proxy ant task like:
ant ln-proxy ... gateway-proxy: [java] Created GemFireCache[id = 1426898936; isClosing = false; isShutDownAll = false; closingGatewayHubsByShutdownAll = false; created = Tue Sep 06 10:37:42 PDT 2016; server = false; copyOnRead = false; lockLease = 120; lockTimeout = 60] [java] Retrieved region: Partitioned Region @237ac531 [path='/data'; dataPolicy=PARTITION; gatewayEnabled=false; prId=3; isDestroyed=false; isClosed=false; retryTimeout=3600000; serialNumber=7; partition attributes=PartitionAttributes@172353175[redundantCopies=1;localMaxMemory=0;totalMaxMemory=2147483647;totalNumBuckets=113;partitionResolver=null;colocatedWith=null;recoveryDelay=-1;startupRecoveryDelay=0;FixedPartitionAttributes=null;partitionListeners=null]; on VM boglesbymac-2(ln-1-proxy:11650)<v7>:18044] [java] Region named data size: 1000
Resume ny Gateway Sender
Resume the ny GatewaySender in all ny members using the resume_ny_gateway_sender.sh script like:
./resume_ny_gateway_sender.sh (1) Executing - connect --locator=localhost[10331] Connecting to Locator at [host=localhost, port=10331] .. Connecting to Manager at [host=boglesbymac-2, port=1091] .. Successfully connected to: [host=boglesbymac-2, port=1091] (2) Executing - resume gateway-sender --id=ln Member | Result | Message ----------------------------------- | ------ | ------------------------------------------------------------------------- boglesbymac-2(ny-1:11292)<v1>:14874 | OK | GatewaySender ln is resumed on member boglesbymac-2(ny-1:11292)<v1>:14874 boglesbymac-2(ny-2:11306)<v2>:57278 | OK | GatewaySender ln is resumed on member boglesbymac-2(ny-2:11306)<v2>:57278
The ny Gateway Senders will connect to the ln non-SSL proxy server in the ln site and the ny queues will drain.
The ln server terminal windows will log messages like:
[java] Region named data size: 1000
The ny server terminal windows will log messages like:
[java] Region /ln_PARALLEL_GATEWAY_SENDER_QUEUE (a ParallelGatewaySenderQueueMetaRegion) size=0; primarySize=0; entrySetSize=0; entrySetArrayLength=0; primaryEntrySetSize=0; primaryEntrySetArrayLength=0
Stop ny Non-SSL Servers
Stop the ny non-SSL servers using the shutdown_ny_servers.sh script like:
./shutdown_ny_servers.sh (1) Executing - connect --locator=localhost[10331] Connecting to Locator at [host=localhost, port=10331] .. Connecting to Manager at [host=boglesbymac-2, port=1091] .. Successfully connected to: [host=boglesbymac-2, port=1091] (2) Executing - shutdown Shutdown is triggered
Stop ln non-SSL Proxy Server
Stop the ln non-SSL proxy server using kill -9 or CMD/CNTL-C.
Start ny SSL Servers
Start the ny-1 SSL server using the ny-ssl-1 ant task like:
ant ny-ssl-1 ... gateway-ssl: [java] Created GemFireCache[id = 2073241813; isClosing = false; isShutDownAll = false; closingGatewayHubsByShutdownAll = false; created = Wed Aug 31 17:23:59 PDT 2016; server = false; copyOnRead = false; lockLease = 120; lockTimeout = 60] [java] Retrieved region: Partitioned Region @52dcd527 [path='/data'; dataPolicy=PERSISTENT_PARTITION; gatewayEnabled=false; prId=3; isDestroyed=false; isClosed=false; retryTimeout=3600000; serialNumber=10; partition attributes=PartitionAttributes@193834439[redundantCopies=1;localMaxMemory=444;totalMaxMemory=2147483647;totalNumBuckets=113;partitionResolver=null;colocatedWith=null;recoveryDelay=-1;startupRecoveryDelay=0;FixedPartitionAttributes=null;partitionListeners=null]; on VM boglesbymac-2(ny-1:50581)<v8>:43308] [java] ParallelGatewaySenderImpl[id=ln; socketReadTimeout=0] [java] Region named data size: 1000 [java] Region /ln_PARALLEL_GATEWAY_SENDER_QUEUE (a ParallelGatewaySenderQueueMetaRegion) size=0; primarySize=0; entrySetSize=0; entrySetArrayLength=0; primaryEntrySetSize=0; primaryEntrySetArrayLength=0
Start the ny-2 SSL server using the ny-ssl-2 ant task like:
ant ny-ssl-2 ... gateway-ssl: [java] Created GemFireCache[id = 790162063; isClosing = false; isShutDownAll = false; closingGatewayHubsByShutdownAll = false; created = Tue Sep 06 12:59:37 PDT 2016; server = false; copyOnRead = false; lockLease = 120; lockTimeout = 60] [java] Retrieved region: Partitioned Region @56986d66 [path='/data'; dataPolicy=PERSISTENT_PARTITION; gatewayEnabled=false; prId=5; isDestroyed=false; isClosed=false; retryTimeout=3600000; serialNumber=10; partition attributes=PartitionAttributes@1937993113[redundantCopies=1;localMaxMemory=444;totalMaxMemory=2147483647;totalNumBuckets=113;partitionResolver=null;colocatedWith=null;recoveryDelay=-1;startupRecoveryDelay=0;FixedPartitionAttributes=null;partitionListeners=null]; on VM boglesbymac-2(ny-2:13344)<v10>:21376] [java] ParallelGatewaySenderImpl[id=ln; socketReadTimeout=0] [java] Region named data size: 1000 [java] Region /ln_PARALLEL_GATEWAY_SENDER_QUEUE (a ParallelGatewaySenderQueueMetaRegion) size=0; primarySize=0; entrySetSize=0; entrySetArrayLength=0; primaryEntrySetSize=0; primaryEntrySetArrayLength=0
At this point, both sites will be up and communicating with each other using SSL sockets.
Run ln Client Loader
Run the ln client loader using the ln-client-load task like:
ant ln-client-load ... client: [java] Created GemFireCache[id = 1417219418; isClosing = false; isShutDownAll = false; closingGatewayHubsByShutdownAll = false; created = Wed Aug 31 17:26:29 PDT 2016; server = false; copyOnRead = false; lockLease = 120; lockTimeout = 60] [java] Retrieved region: com.gemstone.gemfire.internal.cache.LocalRegion[path='/data';scope=LOCAL';dataPolicy=EMPTY; concurrencyChecksEnabled] [java] Loaded 1000 entries in 2300 ms [java] Loaded 1000 entries in 2301 ms [java] Region /ny_PARALLEL_GATEWAY_SENDER_QUEUE (a ParallelGatewaySenderQueueMetaRegion) size=80; primarySize=47; entrySetSize=80; entrySetArrayLength=82; primaryEntrySetSize=49; primaryEntrySetArrayLength=49 [java] Region /ny_PARALLEL_GATEWAY_SENDER_QUEUE (a ParallelGatewaySenderQueueMetaRegion) size=55; primarySize=31; entrySetSize=48; entrySetArrayLength=154; primaryEntrySetSize=25; primaryEntrySetArrayLength=23 [java] Region /ny_PARALLEL_GATEWAY_SENDER_QUEUE (a ParallelGatewaySenderQueueMetaRegion) size=0; primarySize=0; entrySetSize=0; entrySetArrayLength=0; primaryEntrySetSize=0; primaryEntrySetArrayLength=0
Run ny Client Loader
Run the ny client loader using the ny-client-load task like:
ant ny-client-load ... client: [java] Created GemFireCache[id = 1826080114; isClosing = false; isShutDownAll = false; closingGatewayHubsByShutdownAll = false; created = Wed Aug 31 17:27:52 PDT 2016; server = false; copyOnRead = false; lockLease = 120; lockTimeout = 60] [java] Retrieved region: com.gemstone.gemfire.internal.cache.LocalRegion[path='/data';scope=LOCAL';dataPolicy=EMPTY; concurrencyChecksEnabled] [java] Loaded 1000 entries in 2137 ms [java] Loaded 1000 entries in 2137 ms [java] Region /ln_PARALLEL_GATEWAY_SENDER_QUEUE (a ParallelGatewaySenderQueueMetaRegion) size=195; primarySize=98; entrySetSize=198; entrySetArrayLength=205; primaryEntrySetSize=99; primaryEntrySetArrayLength=100 [java] Region /ln_PARALLEL_GATEWAY_SENDER_QUEUE (a ParallelGatewaySenderQueueMetaRegion) size=75; primarySize=0; entrySetSize=63; entrySetArrayLength=75; primaryEntrySetSize=0; primaryEntrySetArrayLength=0 [java] Region /ln_PARALLEL_GATEWAY_SENDER_QUEUE (a ParallelGatewaySenderQueueMetaRegion) size=0; primarySize=0; entrySetSize=0; entrySetArrayLength=0; primaryEntrySetSize=0; primaryEntrySetArrayLength=0
Shutdown Sites
Stop servers and locators in both sites using the shutdownallsites.sh script like:
./shutdownallsites.sh (1) Executing - connect --locator=localhost[10331] Connecting to Locator at [host=localhost, port=10331] .. Connecting to Manager at [host=boglesbymac-2, port=1091] .. Successfully connected to: [host=boglesbymac-2, port=1091] (2) Executing - shutdown --include-locators=true Shutdown is triggered (1) Executing - connect --locator=localhost[10332] Connecting to Locator at [host=localhost, port=10332] .. Connecting to Manager at [host=boglesbymac-2, port=1092] .. Successfully connected to: [host=boglesbymac-2, port=1092] (2) Executing - shutdown --include-locators=true Shutdown is triggered