This page describes the workload portion of Olio i.e. the load that is generated by the Olio driver against the Olio application.
- Concurrent Users: The number of users emulated during a test run of the workload. The workload is scaled by increasing the number of Concurrent Users. We also use the term Active Users to represent the set of users actually performing work against the application.
- Users : The term Users is usually a short-hand for Concurrent Users or Active Users.
- Registered Users: This is the number of users known to the application i.e. the number of users in the database. The number of registered users is 100 times the number of Concurrent Users and reflects the fact that social networking applications have a large number of users but typically only a fraction of them are actively using the application.
- Operations: An operation is a request for a particular page made by the emulated user. Each operation may involve multiple HTTP request/response cycles depending on the embedded objects on a page. On average, there are 20 HTTP requests made per operation.
- Response Time: This is the time taken between the first byte of the first http request for an operation to the time the last byte of the last http request has been received. As such, this is a measure of the server response time and does not include any client-side processing.
- Cycle Time: This is the inter-arrival time between subsequent operations made by a concurrent user. To more realistically represent real workloads, each emulated user will wait for some time before submitting the next operation. In Olio, the mean cycle time is 5 seconds. This allows us to drive a larger emulated user load although to be truly realistic, the cycle time should be upto 30 seconds or larger. Using larger cycle times will require emulating 100's of thousands (or even millions) of users, so to provide a workload that is easier to run, we use 5 seconds.
- Operations Mix: This is the ratio of the various operations submitted by concurrent users. Olio uses a state-transition matrix mix to define the probability of transitioning from one operation to the next. See the Faban documentation for more details.
There are 7 different Operations defined by Olio. A brief description of each is given below.
The AddPerson and AddEvent operations are relatively expensive in terms of cpu time consumed, since they involve image thumbnail generation.
Requirements and Metrics
The primary performance metric is a throughput metric of operations/second representing the total number of operations performed during the steady-state interval divided by the number of seconds in the steady-state interval. The metric is valid only if all of the requirements for a run have passed as indicated by the summary report.
There are requirements for 90% response time (the time in which 90% of all operations in staeady state should have completed) for all of the operations as specified below :
90% Response Time
A valid run needs to meet the following operation mix requirements. Note that the driver is designed to meet these requirements as long as there are a reasonable number of operations that were executed during the run.Note that since the AddPerson is a very small percentage of the mix, you will need to ensure that a sufficient number of operations are executed to meet this requirement. In practice, we have had no problem in achieving passing runs even with low concurrent users (50 or 100 users) as long as the steady state period is at 600-900 seconds.
Olio uses a negative exponential distribution of cycle times (see Cycle timings section in Faban documentation) to simulate a realistic web workload. The distribution uses a mean of 5 seconds, starting at 0 seconds and is truncated at 25 seconds for all of the operations except the AddPerson and AddEvent operations.