If you have any questions or comments about the performance report or it's supporting documents, please feel free to post them here.
I'm one of the engineers who helped to create the performance report, and I'll be actively monitoring this forum to answer questions.
Mike Lowell
Nimbostratus
The good news is we can change the vast majority of the knobs and dials and the impact is barely measurable, sometimes higher, sometimes lower, but generally small enough to fit inside a margin of error.
For the few knobs that can have an impact, we usually have some idea of "how much" the impact can be. On the BIG-IP 8800 the performance for "full" and "partial" PVA acceleration is typically within 1% of being identical, due to the processing power of the BIG-IP 8800. The real change switching from "full" to "partial" is that CPU usage goes from 0% (i.e. handled entirely within PVA) to probably 30-40% for small responses, less than 20% for 8KB, and less than 5% for 64KB and larger.
Generically speaking, for configurations that have "no" acceleration (software only), top-end performance is likely to be reduced by 5-35%, depending on response size. As an interesting data point, even assuming a worst-case-scenario the L4 performance of the BIG-IP 8800 is still a healthy margin above the competition. :)
As always, I personally suggest that customers run their own performance tests to be sure all environmental factors are properly considered. For example, it's often not well appreciated how much server performance factors in. Similarly, many customers don't have a good idea of their response size mix, etc -- there are hundreds of factors like this. The only way to be sure everything is factored in is to test in your particular environment. Having said that, the relative performance between vendors should be similar to what you see in the report.
I hope this helps!
Mike Lowell
p.s.
I'll provide your feedback to the right people to look new documents in the future. Thanks!
Nimbostratus
Was there testing done to determine the maximum number of persistent SSL connections that could be established? I have an application that requires a large number of concurrent SSL connections (small number of TPS/connection) and was wondering if BIG-IP would fit the bill.
Thanks,
Jeff
His question was, how does F5 compare in terms of the performance/price ratio? This gets interesting when you talk about disruptive innovators in the marketplace.
Thanks and I look forward to your thoughts.
Ron
Concurrent SSL connections was not tested. In general, the "hard limit" for SSL concurrency is based on the amount of memory that can be allocated to store SSL connection information. The practical limit for SSL concurrency depends on your threshold for latency.
As concurrency increases so do the processing requirements for handling each connection. Depending on the particular implementation, factors that contribute to this increase in per-connection overhead may include: connection hash table collisions, memory fragmentation, session cache overflows, connection maintenance functions (checking for connection timeouts, for example), and more.
Depending on implementation, the effects of increased overhead may start to appear with thousands, tens of thousands, or hundreds of thousands of connections. It's highly dependent on vendor implementation.
All things considered, you're more likely to hit a "practical limit" that causes poor performance (high latency) than you are to hit a "hard limit" based on memory limits. With that in mind, the numbers that vendors have published thus far are based "hard limits".
Without having more exact information to give you, I'd like to point out that you're likely to see practical performance limits that mirror the other SSL performance recorded in the performance report. In other words, if vendorX can process 48,000 SSL TPS, vendorY 28,000, and vendorZ 15,000, it's reasonable to expect vendorX has more processing power dedicated to SSL, and thus is likely to have an advantage in practical SSL concurrency as well.
I hope that helps. Good luck!
Mike Lowell
Price analysis wasn't within the scope of the report, but we have looked at raw price and price/performance for other purposes. Briefly stated, we feel we're very competitive in this regard and encourage others to consider price/performance and TCO when evaluating ADC solutions.
Mike Lowell
(Note: Mike's in-line replies are indicated with a leading '>')
From: Nalik, Joerg [mailto:joerg.nalik@sap.com]
Sent: Monday, September 24, 2007 6:39 PM
To: Ron Carovano, Jr.
Subject: RE: Comparative performance testing
Hi Ron,
first of all I think this is a very nice paper,
> Thanks!
perhaps you can tell me in a few weeks what its impact is: to your customers and your competitors.
Now as a trained scientist and with some experience in this area I could start nagging about the paper until you feel really bad :) Not too much point to that because not many scientist will care about the paper and anybody about them. So here some ideas only:
Many measurements show higher F5 throughput. So why not buying 2 competitor boxes for one F5 box and getting the same?
This points to my price/performance question (not answered on dev central).
> Keeping the report as short as possible was a key goal for us. To make this report useful/digestible for the largest
> target audience we had to do a lot of trimming. I would have liked to include price/performance information because
> it *heavily* favors F5.
On some measurements it says Cisco doesn't offer some feature in their ACE. Do they offer it in another product line?
Not everybody needs to slice and dice features and products like F5 does.
> Agreed. Part of the limitation here was one of scope (i.e. keep the document short, focused on the largest target
> audience). From a customers point of view having multiple boxes is often less desirable because it typically
> increases total cost of ownership. TCO increases because the box count increases (at least 2x of each box to provide
> redundancy), and in the case of Cisco the devices providing each function were from different acquisitions so
> managing the devices is more complex (i.e. you have to learn how to manage each device in it's own way, they are not
> similar). TCO also increases because overhead increases: with all the features in a single box there are many
> efficiencies to take advantage of. Having separate devices requires that multiple devices implement many of the same
> technologies (TCP/IP stacks, HTTP parsing, etc) which increases overhead, latency, and risk (more points of failure).
> Multiple boxes also means more rack space, more heat, more power, etc.
>
> It would be an oversimplification to say that F5's way of slicing features is always better, though -- I do agree with
> you.
As an application guy this doesn't tell me too much because your methodology is too deep down in network terms. As business guy it's even worse. So cool thing that F5 also publishes results for SAP apps which are better to relate to a customers' business, which in the end also provides the budget for network gear.
> Agreed. This was again the result of our limitations of scope and target audience. Having said that, the relative
> performance of the devices as you saw in the report is very likely to be similar to most application environments.
> i.e. if BIG-IP was twice as fast as vendor X in the report, it's likely this 2x advantage will also apply to an SAP
> deployment. It would of course be an oversimplification to say that's always true, though.
The database and hardware server folks do benchmarks like this all the time. In order to make them valuable, they all agreed on a set of standardize benchmark scenarios and methodologies, which they defined and acknowledged together. Therefore their benchmark results have high credibility to customers and are apple to apple comparable. The benchmark F5 publishes tries to demonstrate transparency but in the end it is a paper ordered and paid for by F5, only done with methods and scenarios only F5 (or your contractor) uses. Therefore, F5 competition can just ignore those results and claim they are somehow manufactured and not credible. I actually asked a few network vendors last year why there is not network benchmark standard? The whole concept seemed to be alien to the network industry. I credit it to your work with SAP and maybe other app folks that the benchmark idea is making it slowly into the network world.
> I've a huge fan of standardized testing. At F5 we've been looking at this more and more in recent months. I hope to
> see standardized testing for the ADC space within the next 2 years, and I think F5 will play an important role in
> driving this. BTW, did you see the "how to create performance testing methodologies" doc linked in the report? I
> think it may be interesting to you.
>
> Lastly, I wanted to give you my feelings regarding the report. First, I can say without hesitation I'm much more of a
> scientist than anything else, I agree with you that there are very few scientists in this business. If I had my way
> the report would have been 100-200 pages long as a result of including significantly more detail (about everything),
> as well as by the inclusion of many more related angles (app testing, pricing, and much more). Sadly, this would have
> taken ~1 year to complete, and would have had a significantly smaller target audience (difficult to justify given
> business priorities). In the end, however, I'm glad we were able to produce something useful and be open and honest
> about it, i.e. more than just ridiculous vendor claims :) I hope this raises the bar for other vendors in our space,
> as that would significantly help our efforts to reach standardization.
>
>Thanks for your comments!
- Joerg
Nimbostratus
We are using SSL and we have been told that POC device has 9k SSL packet size limit because of memory constraint on POC device, and some how it couldn’t able to re-assemble the SSL packets because the BigIP might be sending data in 16K SSL packets. It’s difficult to buy the theory, as it’s failing exactly at 19,475 bytes.
We could able to download around 19k of response with out any problems, if the response size is greater then we are having problems in reading the response, it would read only 19,475 and fails to read any more data.
I have the following questions, appreciate if you can suggest some solution .
1. With respect to this problem what specific parameters we may have tweak in order to
download big xml response of size 60k+?
2. What is this magic number, why it failing at 19475 (155 header + 19320 data) every
time?
3. Does it something to do with mismatching of SSL packet size at BigIP and POC device?
Appreciate your help.
You mention TCP/HTTP, so I'm guessing this is XML being transported over HTTP. If that's the case, then no BIG-IP tweaking should be required for normal clients/servers.
Unless you ask BIG-IP to do otherwise, it'll treat XML responses just like image responses or web pages or zip files, or anything else. 60KB isn't even that big. :)
With regards to the magic size, I suspect this has something to do with the client or server. The closest magic number on BIG-IP is 32768, which is the maximum size of HTTP headers we'll accept by default (this is configurable on the HTTP profile).
Your comment about SSL packet size (actually SSL record size) is likely the key. BIG-IP will attempt to send large SSL records because it's normally in the best interests of all that are involved (it's faster for both BIG-IP and receiving devices to deal with fewer large SSL records than many small records).
Whether BIG-IP is going to use large records or small records depends on the speed of the clients and servers (how quickly we're getting data from the server and how quickly the client can receive it from us).
It's very normal to see multiple SSL record sizes used throughout a single transaction. With that in mind, if the client really cannot accept 16KB SSL records, then it seems reasonable to think that the 19KB magic number could actually be a combination of a 1 or few small records (1KB-4KB) and perhaps followed by a 16KB record which doesn't work.
It would be very unusual for a device to not support 16KB records, but if that really is the case perhaps the problem could be worked around by limiting BIG-IP's TCP 'send buffer' parameter on the TCP profile, or perhaps reducing the 'proxy buffer high' to something small like 4096. This is just a guess, I don't have an easy way to test this.
Given your situation, I encourage you to contact the F5 Networks support team to work through the issue.
Good luck!
Mike Lowell
Nimbostratus
We already changed the value of 'send buffer' to 4k and 8k and we also changed the value for "recv window", it didn't worked. One interesting thing we have observed is read pattern in case of successful response and partial response. When we get complete response it reads 500 bytes every time except the last one, which reads what ever remaining bytes. In case of partial response it also reads 500 bytes, but in between it reads 175 or 400 bytes, kind of little irregular read pattern and looks like the data is getting obstructed some where in BigIP, as it could able to read only 175 or 400 bytes instead of 500 bytes what we are trying to read every time. Does it give any hint to you? As we are communicating over SSL, is there any specific profile for SSL need to be tweaked for this problem. We don't suspect the server problem, as we could able to read same response from internet explorer and a java client program. We'll also try to contact BigIp support, however any suggestion from you is really great to us. Please let us know if you have any ideas? Thanks again Mike.
