Measuring Performance in ColdFusion

   Performance is one of the most important criteria for any product or application. If the performance of the Application is low then it is not going to be useful. In this blog post I will discuss the different metrics involved in measuring performance, how to write a test case which will measure the performance accurately and the steps taken to measure ColdFusion performance.

Metrics involved in measuring Performance:

In ColdFusion we mainly concentrate on LOAD testing. There are three main parameters when it comes to LOAD testing. They are:

     1. Performance
     2. Stress
     3. Reliability

When we measure performance then we consider "TIME" taken to execute a test case as the most important factor. In other words the Average Response Time(ART) must be less with a greater throughput(no of requests processed per second).

When we measure Stress, then we are no longer interested in measuring the time taken except that it must be constant. The factors which are of at most importance are:
    a. There should be no exceptions/data loss or crash
    b. The tests should be run with a large number of users.
    c. Here we measure the maximum capacity of the product or tool. In other words, we measure the   maximum limit of the tool. For example, if we are interested in measuring the stress of ColdFusion then we increase the number of users till the point ColdFusion server hangs or stops responding. This is done so that the maximum limit of the product is known.

When we measure Reliability, then we run the tests for a longer period of time with constant users. Here, we are more concerned to see if the throughput is constant and no memory leaks are happening. It is generally run for 2 days.


Steps taken before writing a performance test case:


To write a test case for measuring performance we must adhere to the below rules. They are:
1. The CPU Usage for executing a test case should be around 60% for 10 Virtual users.
2. The xmx and xms value should be high so that the major and minor collections can be avoided. It should preferably be 1024.
3. The ART(Average Response Time) should be around 70-100 milli seconds for a single user or else the Jmeter results will not be reliable.


h3. Steps to be performed before running performance tests:

1. JRE should be the same(ie 64-bit/32-bit) on both machines(the two version which have to be compared). The preferred should be Server JDK since it is faster. The JRE used can be seen in the Administrator under "Settings Summary".
2. The different Admin Settings are:
     a. In Request Tuning -> Maximum number of simultaneous Template requests should be 50 with the remaining set to 1
     b. In caching -> Trusted Cache should be ticked/enabled.
     c. In Debugging & Logging -> Debug Output Settings: the debugging should be turned off.
     d. Ensure that no monitoring services are running
3. In the jvm.config in java.args change the xmx and xms value to 1024. Check for -server in the java args. The -Xbatch should be removed. The -Djava.security.manager should be removed. The disableJSafe option should be set to true.
4. Restart the server.
5. Run the test case few times on the browser for Initial warmup.
6. Launch Jmeter. Specify the required details such as no of users, server, path to test case etc.
7. Run the test case through Jmeter. Stop after 10 seconds.
8. Rerun the test case again for 5 minutes.
9. Record the throughput and ART. Repeat this process 3 times for getting a reasonable consistent reading.