Stresstesting a Lasso webapp

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Stresstesting a Lasso webapp

Lars A. Gundersen-2
Had a pretty rough few days here. One of our customers started a survey with our survey webapp with about 10 000 participants, which were served the link to the survey almost at the same time. When they started clicking on the link en masse the response from our servers slowed down to a crawl.
I did a number of things to mitigate it, and it did help a bit, but I realize I have fundamentally neglected to do proper stress testing on the whole usage chain. Basically it runs nice until a certain number (a too low number) of clients connects concurrently, then the CPU just completely peaks and the response time jumps through the roof.
I found https://loadimpact.com/ which is very nice, but I think the price is a bit steep. Are there any other services like that, or other ways of doing stresstesting on your own? Any other tips on stresstesting in general?

Lars
#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Stresstesting a Lasso webapp

Richard Taubo
Hi!

On 11. mars 2012, at 15.32, Lars A. Gundersen wrote:
> Had a pretty rough few days here. One of our customers started a survey with our survey webapp with about 10 000 participants, which were served the link to the survey almost at the same time. When they started clicking on the link en masse the response from our servers slowed down to a crawl.
> I did a number of things to mitigate it, and it did help a bit, but I realize I have fundamentally neglected to do proper stress testing on the whole usage chain. Basically it runs nice until a certain number (a too low number) of clients connects concurrently, then the CPU just completely peaks and the response time jumps through the roof.
> I found https://loadimpact.com/ which is very nice, but I think the price is a bit steep. Are there any other services like that, or other ways of doing stresstesting on your own? Any other tips on stresstesting in general?


I think I would be looking into e.g. jmeter, http://jmeter.apache.org/
For at getting started tutorial see: http://vimeo.com/4025715 (there are few other tutorials there as well).


Best regards,
Richard Taubo
#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Stresstesting a Lasso webapp

Ke Carlton-3
In reply to this post by Lars A. Gundersen-2
There's quite a few free tools available, I use:

  ab
  httpload

Typically for that sort of concurrency you'll need multiple servers behind
a load balancer (for any serious commercial venture I would recommend this
as a matter of course for L8 or L9).

L9 could probably cope with single server once optimised -- but again you
need the reduncy so you only need to tune for two or more servers (make
tuning a lot easier). Redundancy is good.

On another note, a lot of your issues could've been avoided by staggering
the delivery of the emails.

Stress aside (!), the end result will be productive. Afterwards you will
have a better idea about where you stand in terms of capacity -- what
volume you can handle, what you need to improve and eventually you will be
able to predict when you need to expand again...

Ke

On Sunday, 11 March 2012, Lars A. Gundersen <[hidden email]> wrote:
> Had a pretty rough few days here. One of our customers started a survey
with our survey webapp with about 10 000 participants, which were served
the link to the survey almost at the same time. When they started clicking
on the link en masse the response from our servers slowed down to a crawl.
> I did a number of things to mitigate it, and it did help a bit, but I
realize I have fundamentally neglected to do proper stress testing on the
whole usage chain. Basically it runs nice until a certain number (a too low
number) of clients connects concurrently, then the CPU just completely
peaks and the response time jumps through the roof.
> I found https://loadimpact.com/ which is very nice, but I think the price
is a bit steep. Are there any other services like that, or other ways of
doing stresstesting on your own? Any other tips on stresstesting in general?
>
> Lars
> #############################################################
> This message is sent to you because you are subscribed to
>  the mailing list Lasso
> [hidden email]
> To unsubscribe, E-mail to: <[hidden email]>
> Send administrative queries to  <[hidden email]>
>
#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Stresstesting a Lasso webapp

Tim Taplin
In reply to this post by Lars A. Gundersen-2
I have found siege to be a valuable tool in benchmarking. It tends to have a fairly dark view of things, but its useful to be able to record a session and replay it with different numbers of concurrent sessions and repeat counts. By saving your results, making a few adjustments and rerunning your tests, you can get a feel for how much improvement you're getting for your changes.

Biggest improvements likely with mysql indexing and putting tags into lassostartup instead of including the files repeatedly. This is assuming that you've already got most of your application logic in custom tags.

Tim Taplin
ps. by "Dark View" I mean dont be discouraged if the numbers you get from siege seem painfully slow, just focus on the improvements you can make


On Mar 11, 2012, at 8:32 AM, Lars A. Gundersen wrote:

> Had a pretty rough few days here. One of our customers started a survey with our survey webapp with about 10 000 participants, which were served the link to the survey almost at the same time. When they started clicking on the link en masse the response from our servers slowed down to a crawl.
> I did a number of things to mitigate it, and it did help a bit, but I realize I have fundamentally neglected to do proper stress testing on the whole usage chain. Basically it runs nice until a certain number (a too low number) of clients connects concurrently, then the CPU just completely peaks and the response time jumps through the roof.
> I found https://loadimpact.com/ which is very nice, but I think the price is a bit steep. Are there any other services like that, or other ways of doing stresstesting on your own? Any other tips on stresstesting in general?
>
> Lars
> #############################################################
> This message is sent to you because you are subscribed to
>  the mailing list Lasso
> [hidden email]
> To unsubscribe, E-mail to: <[hidden email]>
> Send administrative queries to  <[hidden email]>

#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Stresstesting a Lasso webapp

Kyle Jessup-2
In reply to this post by Ke Carlton-3

On Mar 11, 2012, at 5:07 PM, Ke Carlton wrote:

> There's quite a few free tools available, I use:
>
>  ab
>  httpload

httperf is good as well.
-Kyle

> Typically for that sort of concurrency you'll need multiple servers behind
> a load balancer (for any serious commercial venture I would recommend this
> as a matter of course for L8 or L9).
>
> L9 could probably cope with single server once optimised -- but again you
> need the reduncy so you only need to tune for two or more servers (make
> tuning a lot easier). Redundancy is good.
>
> On another note, a lot of your issues could've been avoided by staggering
> the delivery of the emails.
>
> Stress aside (!), the end result will be productive. Afterwards you will
> have a better idea about where you stand in terms of capacity -- what
> volume you can handle, what you need to improve and eventually you will be
> able to predict when you need to expand again...
>
> Ke
>
> On Sunday, 11 March 2012, Lars A. Gundersen <[hidden email]> wrote:
>> Had a pretty rough few days here. One of our customers started a survey
> with our survey webapp with about 10 000 participants, which were served
> the link to the survey almost at the same time. When they started clicking
> on the link en masse the response from our servers slowed down to a crawl.
>> I did a number of things to mitigate it, and it did help a bit, but I
> realize I have fundamentally neglected to do proper stress testing on the
> whole usage chain. Basically it runs nice until a certain number (a too low
> number) of clients connects concurrently, then the CPU just completely
> peaks and the response time jumps through the roof.
>> I found https://loadimpact.com/ which is very nice, but I think the price
> is a bit steep. Are there any other services like that, or other ways of
> doing stresstesting on your own? Any other tips on stresstesting in general?
>>
>> Lars
>> #############################################################
>> This message is sent to you because you are subscribed to
>> the mailing list Lasso
>> [hidden email]
>> To unsubscribe, E-mail to: <[hidden email]>
>> Send administrative queries to  <[hidden email]>
>>
> #############################################################
> This message is sent to you because you are subscribed to
>  the mailing list Lasso
> [hidden email]
> To unsubscribe, E-mail to: <[hidden email]>
> Send administrative queries to  <[hidden email]>

#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>