CONFIGURATIONS
1. Management server
Processor
Dual core Intel(R) Xeon(R) CPU processor, 2.27GHz, ht enabled, 4 processor
Operating System
CentOS release 5.5 (Final), x86_64
Java version
java version "1.6.0"
OpenJDK Runtime Environment (build 1.6.0-b09)
OpenJDK 64-Bit Server VM (build 1.6.0-b09, mixed mode)
2. Database
Processor
Quad-Core AMD Opteron(tm) Processor, 2.1GHz, ht enabled, 8 processor
Operating System
CentOS release 6.2 (Final), x86_64
Configuration Parameters
DB configurations for this run is detailed in the my.cnf attached: my.cnf.txt
Mysql version
MySQL-server-5.5.21-1.linux2.6.x86_64
RESULTS
Tests on Basic zone:
Set up a clustered Management Server (with 3 MS)
Deploy a basic zone with 20,000 simulated virtual machines. Create security group ingress rules to allow traffic. Use local storage.
Test Scenario 1:
Deploy the VMs in batches of 5000 and track CPU Utilization, DB Connections, and the time taken to deploy the VMs as well as response time for the async job.
Test Scenario 2:
Stop a management server and track the time taken for hosts to get re distributed with other management servers
Stop all 4 management servers and restart all of them and measure the time taken for all of the hosts to get to connected state
Due to following issues, the runs haven't been completed
https://issues.apache.org/jira/browse/CLOUDSTACK-4371
https://issues.apache.org/jira/browse/CLOUDSTACK-4350
https://issues.apache.org/jira/browse/CLOUDSTACK-4350 
https://issues.apache.org/jira/browse/CLOUDSTACK-2918
https://issues.apache.org/jira/browse/CLOUDSTACK-3441
Tests on Advanced zone
Deploy an Advanced Zone with over a 2000 hosts and > 10000 simulated virtual machines. Have a clustered Management server set up (2 MS)
Add around 4000 accounts and set up as many networks.
Deploy the simulated VMs across these networks (around 3 VMs per network)
Test Scenario 3:
Deploy VMs in batches and in sequence and following measures should be tracked in the Management Server:
- CPU Utilization during deploy VM
- Time for Async Job to complete
- Time to return job id
- DB connections
Following config parameters were used in both the management servers
- Java heap size = 5 GB
- db.cloud.maxActive = 250
- db.cloud.url.params=prepStmtCacheSize=517&cachePrepStmts=true&prepStmtCacheSqlLimit=4096&includeInnodbStatusInDeadlockExceptions=true&logSlowQueries=true
CPU Utilization during deploy VM
CPU Utilization measured in the Management server during deployment of 12K simulator VMs and post deploy when there were no external APIs being fired
The area marked in red is during Deploy VM
Response Time of async job:
Following graph shows the response time calculated for deploy VM jobs. It’s more or less constant between 3 and 5 seconds except for few spikes 
Test Scenario 5:
Restart Management Servers and track the time for host re balancing. Also track the numbers when few hosts are put into maintenance mode.
Following shows the time taken for each restart:
TC1: Stop both management servers. Then start up both. Time taken for all hosts to get to Up state: ~13 mins
TC2: Stop one management server. Time taken for hosts of that management server to get to Up state: ~5 mins
TC3: Start up the stopped management server. Time taken for hosts to rebalance between the 2 management server: ~16 mins
Now, add few hosts into maintenance mode and repeat the above restarts. Following are the time taken in each scenario:
TC1: ~10 mins
TC2: ~5 mins
TC3: ~13 mins
LIST API RESPONSE TIME
Configuration:
Advanced zone, 112 Pods [Each Pod having 2 Clusters]
224 Clusters [Each cluster having 8 hosts and one primary storage]
2000 Hosts
4000 User accounts [Each account having one network]
4000 User instances
8000 Virtual Routers [Since we are using Redundant Virtual Router offering]
API |
Params |
Time taken in 4.1 |
Time taken in 4.2 |
Time taken in 4.2 |
|
|
|
|
(with fix) |
listAccounts |
listAll=true&pagesize=100&page=3 |
0m16.923s |
0m20.017s |
0m16.680s |
(# of obj: 4000) |
listAll=true&pagesize=800&page=2 |
0m26.555s |
0m34.438s |
0m36.118s |
|
listAll=true&pagesize=1500&page=2 |
0m40.140s |
1m1.036s |
0m55.497s |
|
listAll=true&pagesize=3000&page=1 |
1m3.906s |
2m2.921s |
1m24.601s |
|
listAll=true |
1m25.348s |
2m22.628s |
1m44.621s |
|
|
|
|
|
listRouters |
listAll=true&pagesize=100&page=1 |
0m2.398s |
0m3.808s |
0m3.261s |
(# of obj: 8000) |
listAll=true&pagesize=800&page=1 |
0m10.372s |
0m23.124s |
0m22.164s |
|
listAll=true&pagesize=1500&page=1 |
0m20.857s |
0m25.727s |
0m38.656s |
|
listAll=true&pagesize=3000&page=1 |
0m35.034s |
1m39.030s |
1m22.330s |
|
listAll=true&pagesize=5000&page=1 |
1m0.932s |
2m31.960s |
2m5.248s |
|
listAll=true |
1m30.935s |
4m26.163s |
3m30.292s |
|
|
|
|
|
listEvents |
listAll=true&pagesize=100&page=1 |
0m5.967s |
0m2.169s |
0m3.423s |
(# of obj: 8000) |
listAll=true&pagesize=800&page=1 |
0m3.006s |
0m6.356s |
0m5.192s |
|
listAll=true&pagesize=1500&page=1 |
0m2.111s |
0m10.402s |
0m7.622s |
|
listAll=true&pagesize=3000&page=1 |
0m4.122s |
0m13.979s |
0m13.520s |
|
listAll=true&pagesize=5000&page=1 |
0m6.575s |
0m20.277s |
0m21.931s |
|
listAll=true |
0m15.399s |
2m16.445s |
1m54.454s |
|
|
|
|
|
listHosts |
listAll=true&pagesize=100&page=1 |
0m1.906s |
0m1.030s |
0m1.159s |
(# of obj: 1985) |
listAll=true&pagesize=800&page=1 |
0m13.316s |
0m7.949s |
0m7.091s |
|
listAll=true&pagesize=1500&page=1 |
0m27.115s |
0m19.168s |
0m9.309s |
|
listAll=true |
|
0m42.024s |
0m41.504s |
|
|
|
|
|
listUsers |
listAll=true&pagesize=100&page=1 |
0m0.253s |
0m0.406s |
0m0.704s |
(# of obj: 4000) |
listAll=true&pagesize=800&page=1 |
0m2.105s |
0m3.526s |
0m3.484s |
|
listAll=true&pagesize=1500&page=1 |
0m2.485s |
0m12.411s |
0m6.792s |
|
listAll=true&pagesize=3000&page=1 |
0m6.975s |
0m14.739s |
0m14.165s |
|
listAll=true |
0m6.787s |
0m22.466s |
0m19.312s |
|
|
|
|
|
listVirtualMachines |
listAll=true&pagesize=100&page=1 |
0m1.671s |
0m9.009s |
0m2.464s |
(# of obj: 8000) |
listAll=true&pagesize=800&page=1 |
0m9.191s |
0m25.621s |
0m14.625s |
|
listAll=true&pagesize=1500&page=1 |
0m13.068s |
0m20.737s |
0m26.461s |
|
listAll=true&pagesize=3000&page=1 |
0m22.950s |
0m18.503s |
0m51.964s |
|
listAll=true&pagesize=5000&page=1 |
0m35.443s |
0m18.503s |
|
|
listAll=true |
|
1m16.001s |
|
|
|
|
|
|
listVolumes |
listAll=true&pagesize=100&page=1 |
0m0.830s |
0m0.889s |
0m1.092s |
(# of obj: 8000) |
listAll=true&pagesize=800&page=1 |
0m3.900s |
0m7.091s |
0m7.148s |
|
listAll=true&pagesize=1500&page=1 |
0m5.887s |
0m13.906s |
0m13.077s |
|
listAll=true&pagesize=3000&page=1 |
0m11.037s |
0m8.737s |
0m27.093s |
|
listAll=true&pagesize=5000&page=1 |
0m18.859s |
0m40.966s |
1m9.951s |
|
listAll=true |
|
0m0.526s |
|
|
|
|
|
|
listStoragePools |
listAll=true&pagesize=100&page=1 |
0m0.313s |
0m0.526s |
0m0.346s |
(# of obj: 248) |
listAll=true |
0m0.454s |
0m1.115s |
0m1.182s |
|
|
|
|
|
listProjects |
listAll=true&pagesize=100&page=1 |
0m0.304s |
0m27.884s |
|
|
listAll=true&pagesize=800&page=1 |
0m1.019s |
4m5.661s |
|
|
listAll=true&pagesize=1500&page=1 |
0m1.696s |
|
|
|
listAll=true&pagesize=3000&page=1 |
0m2.986s |
|
|
|
listAll=true |
0m5.213s |
|
|
|
|
|
|
|
listProjectAccounts |
listAll=true&projectid=3 |
0m33.247s |
|
|
|
listAll=true&projectid=4 |
0m15.992s |
|
|
|
listAll=true |
|
0m13.772s |
|
|
|
|
|
|
listProjectInvitations |
listAll=true&pagesize=100&page=1 |
0m0.129s |
|
|
|
listAll=true&pagesize=800&page=1 |
0m0.515s |
|
|
|
listAll=true |
0m0.843s |
|
|
|
|
|
|
|
listAsyncJobs |
listAll=true&pagesize=100&page=120 |
0m0.544s |
|
0m2.210s |
(# of obj: 12000) |
listAll=true&pagesize=800&page=5 |
0m8.027s |
0m21.987s |
0m18.309s |
|
listAll=true&pagesize=1500&page=6 |
0m1.856s |
0m29.040s |
0m36.051s |
|
listAll=true&pagesize=3000&page=3 |
0m3.396s |
0m27.300s |
1m3.663s |
|
listAll=true&pagesize=5000&page=3 |
|
1m49.942s |
1m50.274s |
|
listAll=true&pagesize=8000&page=1 |
0m7.731s |
|
|
|
listAll=true |
0m9.139s |
1m32.254s |
|
Overview
Content Tools
Apps