Skip to end of metadata
Go to start of metadata

CONFIGURATIONS

1.      Management server

Processor

Dual core Intel(R) Xeon(R) CPU processor, 2.27GHz, ht enabled, 4 processor

Operating System

CentOS release 5.5 (Final), x86_64

Java version

java version "1.6.0"

OpenJDK  Runtime Environment (build 1.6.0-b09)

OpenJDK 64-Bit Server VM (build 1.6.0-b09, mixed mode)

2.      Database

 Processor

Quad-Core AMD Opteron(tm) Processor, 2.1GHz, ht enabled, 8 processor

Operating System

CentOS release 6.2 (Final), x86_64

Configuration Parameters

DB configurations for this run is detailed in the my.cnf attached: my.cnf.txt

Mysql version

MySQL-server-5.5.21-1.linux2.6.x86_64

RESULTS 

Tests on Basic zone:

Set up a clustered Management Server (with 3 MS)

Deploy a basic zone with 20,000 simulated virtual machines. Create security group ingress rules to allow traffic. Use local storage.

 

Test Scenario 1:

Deploy the VMs in batches of 5000 and track CPU Utilization, DB Connections, and the time taken to deploy the VMs as well as response time for the async job.

Test Scenario 2:

Stop a management server and track the time taken for hosts to get re distributed with other management servers 

Stop all 4 management servers and restart all of them and measure the time taken for all of the hosts to get to connected state

Due to following issues, the runs haven't been completed

https://issues.apache.org/jira/browse/CLOUDSTACK-4371

https://issues.apache.org/jira/browse/CLOUDSTACK-4350

https://issues.apache.org/jira/browse/CLOUDSTACK-4350&nbsp
https://issues.apache.org/jira/browse/CLOUDSTACK-2918

https://issues.apache.org/jira/browse/CLOUDSTACK-3441

Tests on Advanced zone

Deploy an Advanced Zone with over a 2000 hosts and > 10000 simulated virtual machines. Have a clustered Management server set up (2 MS)

Add around 4000 accounts and set up as many networks.

Deploy the simulated VMs across these networks (around 3 VMs per network)

Test Scenario 3:

Deploy VMs in batches and in sequence and following measures should be tracked in the Management Server:

  1. CPU Utilization during deploy VM
  2. Time for Async Job to complete
  3. Time to return job id
  4. DB connections

Following config parameters were used in both the management servers

-        Java heap size = 5 GB

-        db.cloud.maxActive = 250

-        db.cloud.url.params=prepStmtCacheSize=517&cachePrepStmts=true&prepStmtCacheSqlLimit=4096&includeInnodbStatusInDeadlockExceptions=true&logSlowQueries=true

CPU Utilization during deploy VM

CPU Utilization measured in the Management server during deployment of 12K simulator VMs and post deploy when there were no external APIs being fired 

The area marked in red is during Deploy VM

Response Time of async job:

Following graph shows the response time calculated for deploy VM jobs. It’s more or less constant between 3 and 5 seconds except for few spikes

Test Scenario 5:

Restart Management Servers and track the time for host re balancing. Also track the numbers when few hosts are put into maintenance mode.

Following shows the time taken for each restart:

TC1: Stop both management servers. Then start up both. Time taken for all hosts to get to Up state: ~13 mins

TC2: Stop one management server. Time taken for hosts of that management server to get to Up state: ~5 mins

TC3: Start up the stopped management server. Time taken for hosts to rebalance between the 2 management server: ~16 mins

Now, add few hosts into maintenance mode and repeat the above restarts. Following are the time taken in each scenario:

TC1: ~10 mins

TC2:  ~5 mins

TC3:  ~13 mins

LIST API RESPONSE TIME

Configuration:

Advanced zone, 112 Pods [Each Pod having 2 Clusters]

224 Clusters [Each cluster having 8 hosts and one primary storage]

2000 Hosts

4000 User accounts [Each account having one network]

4000 User instances

8000 Virtual Routers [Since we are using Redundant Virtual Router offering]

API

Params

Time taken in 4.1

Time taken in 4.2

Time taken in 4.2

 

 

 

 

(with fix)

listAccounts

listAll=true&pagesize=100&page=3

0m16.923s

0m20.017s

0m16.680s

(# of obj: 4000)

listAll=true&pagesize=800&page=2

0m26.555s

0m34.438s

0m36.118s

 

listAll=true&pagesize=1500&page=2

0m40.140s

1m1.036s

0m55.497s

 

listAll=true&pagesize=3000&page=1

1m3.906s

2m2.921s

1m24.601s

 

listAll=true

1m25.348s

2m22.628s

1m44.621s

 

 

 

 

 

listRouters

listAll=true&pagesize=100&page=1

0m2.398s

0m3.808s

0m3.261s

(# of obj: 8000)

listAll=true&pagesize=800&page=1

0m10.372s

0m23.124s

0m22.164s

 

listAll=true&pagesize=1500&page=1

0m20.857s

0m25.727s

0m38.656s

 

listAll=true&pagesize=3000&page=1

0m35.034s

1m39.030s

1m22.330s

 

listAll=true&pagesize=5000&page=1

1m0.932s

2m31.960s

2m5.248s

 

listAll=true

1m30.935s

4m26.163s

3m30.292s

 

 

 

 

 

listEvents

listAll=true&pagesize=100&page=1

0m5.967s

0m2.169s

0m3.423s

(# of obj: 8000)

listAll=true&pagesize=800&page=1

0m3.006s

0m6.356s

0m5.192s

 

listAll=true&pagesize=1500&page=1

0m2.111s

0m10.402s

0m7.622s

 

listAll=true&pagesize=3000&page=1

0m4.122s

0m13.979s

0m13.520s

 

listAll=true&pagesize=5000&page=1

0m6.575s

0m20.277s

0m21.931s

 

listAll=true

0m15.399s

2m16.445s

1m54.454s

 

 

 

 

 

listHosts

listAll=true&pagesize=100&page=1

0m1.906s

0m1.030s

0m1.159s

(# of obj: 1985)

listAll=true&pagesize=800&page=1

0m13.316s

0m7.949s

0m7.091s

 

listAll=true&pagesize=1500&page=1

0m27.115s

0m19.168s

0m9.309s

 

listAll=true

 

0m42.024s

0m41.504s

 

 

 

 

 

listUsers

listAll=true&pagesize=100&page=1

0m0.253s

0m0.406s

0m0.704s

(# of obj: 4000)

listAll=true&pagesize=800&page=1

0m2.105s

0m3.526s

0m3.484s

 

listAll=true&pagesize=1500&page=1

0m2.485s

0m12.411s

0m6.792s

 

listAll=true&pagesize=3000&page=1

0m6.975s

0m14.739s

0m14.165s

 

listAll=true

0m6.787s

0m22.466s

0m19.312s

 

 

 

 

 

listVirtualMachines

listAll=true&pagesize=100&page=1

0m1.671s

0m9.009s

0m2.464s

(# of obj: 8000)

listAll=true&pagesize=800&page=1

0m9.191s

0m25.621s

0m14.625s

 

listAll=true&pagesize=1500&page=1

0m13.068s

0m20.737s

0m26.461s

 

listAll=true&pagesize=3000&page=1

0m22.950s

0m18.503s

0m51.964s

 

listAll=true&pagesize=5000&page=1

0m35.443s

0m18.503s

 

 

listAll=true

 

1m16.001s

 

 

 

 

 

 

listVolumes

listAll=true&pagesize=100&page=1

0m0.830s

0m0.889s

0m1.092s

(# of obj: 8000)

listAll=true&pagesize=800&page=1

0m3.900s

0m7.091s

0m7.148s

 

listAll=true&pagesize=1500&page=1

0m5.887s

0m13.906s

0m13.077s

 

listAll=true&pagesize=3000&page=1

0m11.037s

0m8.737s

0m27.093s

 

listAll=true&pagesize=5000&page=1

0m18.859s

0m40.966s

1m9.951s

 

listAll=true

 

0m0.526s

 

 

 

 

 

 

listStoragePools

listAll=true&pagesize=100&page=1

0m0.313s

0m0.526s

0m0.346s

(# of obj: 248)

listAll=true

0m0.454s

0m1.115s

0m1.182s

 

 

 

 

 

listProjects

listAll=true&pagesize=100&page=1

0m0.304s

0m27.884s

 

 

listAll=true&pagesize=800&page=1

0m1.019s

4m5.661s

 

 

listAll=true&pagesize=1500&page=1

0m1.696s

 

 

 

listAll=true&pagesize=3000&page=1

0m2.986s

 

 

 

listAll=true

0m5.213s

 

 

 

 

 

 

 

listProjectAccounts

listAll=true&projectid=3

0m33.247s

 

 

 

listAll=true&projectid=4

0m15.992s

 

 

 

listAll=true

 

0m13.772s

 

 

 

 

 

 

listProjectInvitations

listAll=true&pagesize=100&page=1

0m0.129s

 

 

 

listAll=true&pagesize=800&page=1

0m0.515s

 

 

 

listAll=true

0m0.843s

 

 

 

 

 

 

 

listAsyncJobs

listAll=true&pagesize=100&page=120

0m0.544s

 

0m2.210s

(# of obj: 12000)

listAll=true&pagesize=800&page=5

0m8.027s

0m21.987s

0m18.309s

 

listAll=true&pagesize=1500&page=6

0m1.856s

0m29.040s

0m36.051s

 

listAll=true&pagesize=3000&page=3

0m3.396s

0m27.300s

1m3.663s

 

listAll=true&pagesize=5000&page=3

 

1m49.942s

1m50.274s

 

listAll=true&pagesize=8000&page=1

0m7.731s

 

 

 

listAll=true

0m9.139s

1m32.254s

 

  • No labels