Load Testing as a Service: A Look at Load Impact (beta)
I admit it. I’m a load / performance testing junkie. During my years with Network Computing I burned through any number of solutions designed to throw more traffic at products than money Congress is throwing at failed banks these days. And I do mean burned, as the last time I was in the lab there were no less than three non-functioning Spirent Avalanche systems that had given up the ghost after being forced to their absolute limits over years of use and abuse.
When I received a note telling me about LoadImpact.com, a load testing as a service site, naturally I was intrigued. Generate load? From across the Internet? Inconceivable! That’s a no-no, after all, at least the kind of load that I’m used to generating. So I decided to check it out. Here’s the low down.
PICK YOUR POISON
Load Impact offers four service levels, with the lowest-capable load generating service being free (as in gratis). Pricing is per month with varying numbers of users, configuration and result management, and differences in what you can/cannot configure for any given test.
I tested out the “Light” version, of course, but the site allows you to see all the configuration options – they’re just not enabled for modification if you aren’t using the right version. Load Impact does a nice job of laying out the various versions in an at-a-glance format.
LOAD ‘EM UP
First you’ve got to create a test. When configuring a test you have two options: Basic and Advanced. Basic requires the page you want to test. That’s it. Load Impact supports both HTTP and HTTPS (SSL) and can handle sites requiring HTTP Basic Auth. From there it uses defaults. The Advanced settings offer you a little more control in the Light version, allowing you to analyze the page and then, if you so desire, you can edit the “load script”. The load script is pretty simple:
# Initial sleep to distribute clients
sleep 500-5000
request 2 GET:http://www.nandgate.com/
request 2 GET:http://www.nandgate.com/base.css
request 2 GET:http://www.nandgate.com/bigip/f5-logo.png
request 2 GET:http://www.nandgate.com/donlori-w.jpg
Don’t let the simplicity fool you. You can do quite a bit with the script to change the behavior of the clients and customize it to closely match the behavior of (or at least what you expect will be the behavior of) real visitors. The script format is documented in the FAQ. The format allows you to control the number of simultaneous connections that can be opened (in this case always 2 by default, which appears to try to emulate the behavior of IE6 and below) to better mimic the behavior of a “real” web browser. To simulate a POST method, you send parameters as though it was a GET request; it is assumed that the Load Impact client interprets the script and translates that into an HTTP compliant POST operation. The ability to edit the script allows you to simulate behavior rather than just the loading of a page as you can continue the script and have a client load multiple pages and sleep in between if desired.
In more advanced (paid) versions you can also use a recorder to help you build out more complex scripts – a feature common in most commercial load testing products.
Load Impact clients can also handle cookies, so if you’re testing a web application that uses cookies (and what web application today doesn’t?) you’re good to go. This is also good news for testing web applications that are load balanced and using persistence-based routing as such functionality is often implemented via cookies. The actual load balancing algorithms and functions for hosted and cloud-based applications are rarely made public (and too often no one asks) so supporting cookies ensures that the most common of those methods will be supported.
You can change the number of clients and the “ramp up” (step) values, as long as they stay within the specified limits for the version you are using. The default for the Light (free) version is to start with 10, stop at 50, and ramp up by 10 clients. You cannot configure it to “burst” all the clients at one time, though you could start with 49, end at 50, and ramp by 1 to simulate a situation close enough to bursting. The load generators are using HTTP 1.1 and each test consists of several subtests: one per step. Clients use persistent connections throughout each subtest, but open new connections at the onset of each subtest.
A nice option, though not configurable in the “Light” version, is “page view time”. In high-end load testing terms this is usually referred to as “think time”. “Page view time” is used to simulate the fact that a real client wouldn’t just load the page and leave. They’d visit for a while - hence the reason we call them visitors, right?
You can also change the client bandwidth in paid versions, though a change affects all clients, not just a specified subset. Load Impact has plans to broaden this capability and allow more granular assignment of bandwidth in the future in a manner similar to “page view time” by allowing you to configure a range and then randomly assigning bandwidth values within that range to clients.
An important note about Load Impact clients:
(*) Note that Load Impact clients always behave as web browsers with an empty client-side cache. They will always load all objects on a page, never caching anything. This means that they are sometimes "heavier" than a real client would be.
This means that if you were hoping to test the impact of a web acceleration solution that takes advantage of client-side caching (like F5’s WebAccelerator) that Load Impact will not be valuable in assessing the improvement over subsequent visits due to client-side caching. This also means Load Impact will not accurately test your web server configuration or help you assess the impact on the use of cache-control HTTP headers or HTML meta-tags. It will, however, help assess any improvements in performance that may be due to server-side caching or other optimization/acceleration techniques. This means Load Impact may be invaluable to you when assessing technology designed to help improve application performance before purchasing. The Light version will do well enough for assessing impact on performance and is free, so be sure to remember to include it on the list of options during a proof of concept run with new technology.
WYSIWYG
Once the test is running you’ll see the graph and data updated in real-time. A nice to have, as you don’t have visibility into the status of the clients, but not very interesting to watch as it tends to update slowly. The end result is a nice little graph depicting “Delay” in seconds as it relates to the number of
clients. Initially you see only the summary, the total delay for the entire page. But you can drill into each of the file types and select up to three objects to be included in the graph. As you drill down and select different objects, a spreadsheet with delay as it relates to load is updated. In this case, choosing the base.css and two image files showed the delay in ms rather than in full seconds, as would be expected. You can export the data in CSV format, which should allow you to chart all the composite objects performance together to get a better view of what’s causing any problems you might see.
When you’ve selected more than one object you can also play with the “min”, “max”, and “average” settings to dig around a bit more into the actual data that was generated. Load Impact saves one average delay value per client step, so if you ramped from 10-50 on a step of 10, you’ll see 5 distinct data points plotted, each representing the average of all clients at that step.
Load Impact also notes that “Delay” is not the time it would take for a client (browser) to load the page.
This value is the sum of all individual object load times. Please note that this is not the same as the time the page will load and render in a web browser. Because objects are loaded across several connections in parallel, the total sum of all individual load times will not be the same as the time a web browser takes to load a page. The current "test summary" metric is intended for optimization work, as it accurately shows changes in object load times. We have, however, realized that many people expect it to show page load times as experienced by a user, and therefore we are going to implement such a metric as well.
It’s also important to note that the data collected is not normalized against intercontinental latency. Load Impact has plans to increase the number of load generators available in the future but right now there are two options: one located in Stockholm and one in Chicago. In the Light version all load is generated from Stockholm, which necessarily incurs intercontinental latency that may or may not be applicable to your application. So if you’re using the Light version and hosting somewhere other than Europe and expect most visitors to be outside Europe you may want to manually adjust the data collected by downloading the CSV version of the data.
The next version of Load Impact, which is expected to be available in the next couple of weeks, will offer more granular performance metrics including “a new response time metric and a bandwidth usage metric.” Initially these will be summary only with per-object reporting in the future. Plans for TCP metrics like TTFB (Time to First Byte) and TTLB (Time to Last Byte) as well as IP metrics such as packet loss, delay, and jitter are also in the works. More data is always good for troubleshooting exactly where application performance problems are occurring: at the network or application layers.
RUN IT AGAIN SAM
You can save and re-run a number of tests based on the version you choose, which is a nice way to test sites under low traffic, medium traffic, and high traffic volumes. With the free service version the limit of 50 users makes the definitions of low-medium-high somewhat irrelevant, but even at 5000 concurrent users with the Advanced version Load Impact is definitely not going to be able to generate a high enough load to be considered a mainstay for high-volume site testing.
You’ll also want to consider that the transactions generated by the client are not synthetic unless you use parameters you will recognize as synthetic and can account for them. The traffic generated by Load Impact is real, and therefore it will register as real hits for purposes of advertising, campaign tracking, and any other web site measurement based on visitors you may have in place. If you are using script-based analytics or advertising you can relax; while scripts are downloaded they are not executed or evaluated by the load generator clients.
Be careful about the potential to trigger security mechanisms if you’re testing a hosted or cloud-based application with Load Impact, or even your own web application. At this time all clients appear to be arriving from the same IP address, and depending on the configuration of security systems this could trigger alarms and notifications of an attack and potentially blacklist (usually temporarily) the load generator’s IP address. Of course if you’re implementing a security solution or are trying to evaluate rate shaping/QoS functionality you may want to consider using Load Impact to generate some load to evaluate the solution’s effectiveness as well.
To address the possibility of overwhelming third party sites hosting scripts commonly used by web applications Load Impact limits the number of times a resource can be loaded by the entire system in a 24 hour period. While this doesn’t completely mitigate the potential for tripping up security infrastructure, it is a good idea for Load Impact to impose such a limit in the interests of being a good Internet citizen.
The FAQ and forums have a lot of good questions and answers as well as tips to help you build out a test scenario. While the help / information available within the actual application are a bit light, you should be able to find what you need in the forums.
There aren’t enough knobs to make me truly happy yet (I am a load testing junkie, after all) but the roadmap for additional features and metrics is definitely on track to get me there. For most folks this is a good way to start out testing and for many it is probably more than enough than they’ll ever need. For developers trying out their application in a cloud or hosted environment, this is definitely a good way to get a handle on how it’s going to perform. Even for developers wondering how their locally hosted web application is performing might want to consider giving Load Impact a whirl; it provides the perspective of a user outside the local environment which is something not always available to developers using in-house solutions.
It’s free to try out, so go on over and give it a run yourself.
