Lasso 8.6 consuming too much CPU & RAM

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Lasso 8.6 consuming too much CPU & RAM

Roy Dacke
We have been on Lasso for years, over a decade and are now having a problem w 8.6 on CentOS 6.7.

Lasso consumes all the CPU and RAM and has to be restarted repeatedly. Sometimes within a minute of restarting, Lasso needs to be restarted again.

It is a problem that has plagued us for a very long time, years— originally we were on Mac for our Lasso. We applied scripts that check for various conditions leading to failure such as too little RAM available, too little CPU available, too many Lasso threads, too big largest Lasso thread, too many apache threads… It got crazy.  The scripts would work for a few months, then suddenly I’d be back to restarting Lasso all the time. But I’d tweak the scripts in some way, things would work ok again for a few months.

Finally got fed up with this and migrated one group of four Lasso servers in a load balanced group over to CentOS, thinking this would be the magic bullet.  And it was for a couple weeks, I slowly rotated the new machines in and after a week that group was 100% CentOS, it ran great for two weekends (we do grocery store websites, so most traffic is on weekends) then it started breaking RAM limits causing swap. I had to make a simple script to check for that and refresh Lasso.

This worked for a week but failed over the next weekend, hitting CPU limits.  Updated my script to check for that too, but now it wants to restart way too frequently and even when it does, the CPU shoots up within a minute or two… sometimes, other times any given server will be stable for hours or a whole day, sometimes not and I find myself attached to my computer rebooting services constantly (and sometimes entire VMs). Seems like when Lasso doesn’t want to let go of it’s CPU even after restarting the service (and apache) sometimes rebooting the whole VM calms it down for some minutes or hours. I just never know, and it doesn’t necessarily seem to track exactly w high traffic volumes. Any one of those four servers could get flaky at any time and need rebooting— or all four at once.

We need a solution for this ASAP, before the weekend. Lassosoft Support can not see us until next Wednesday.





#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Lasso 8.6 consuming too much CPU & RAM

decorior
Hi Ron:

Are you using mySQL?

If so I would turn on the mySQL slow-log. This has been the cause of the things that you see for us.

It has been as simple as a table that was not indexed.

Deco

> On Feb 10, 2016, at 3:53 PM, Roy Dacke <[hidden email]> wrote:
>
> We have been on Lasso for years, over a decade and are now having a problem w 8.6 on CentOS 6.7.
>
> Lasso consumes all the CPU and RAM and has to be restarted repeatedly. Sometimes within a minute of restarting, Lasso needs to be restarted again.
>
> It is a problem that has plagued us for a very long time, years— originally we were on Mac for our Lasso. We applied scripts that check for various conditions leading to failure such as too little RAM available, too little CPU available, too many Lasso threads, too big largest Lasso thread, too many apache threads… It got crazy.  The scripts would work for a few months, then suddenly I’d be back to restarting Lasso all the time. But I’d tweak the scripts in some way, things would work ok again for a few months.
>
> Finally got fed up with this and migrated one group of four Lasso servers in a load balanced group over to CentOS, thinking this would be the magic bullet.  And it was for a couple weeks, I slowly rotated the new machines in and after a week that group was 100% CentOS, it ran great for two weekends (we do grocery store websites, so most traffic is on weekends) then it started breaking RAM limits causing swap. I had to make a simple script to check for that and refresh Lasso.
>
> This worked for a week but failed over the next weekend, hitting CPU limits.  Updated my script to check for that too, but now it wants to restart way too frequently and even when it does, the CPU shoots up within a minute or two… sometimes, other times any given server will be stable for hours or a whole day, sometimes not and I find myself attached to my computer rebooting services constantly (and sometimes entire VMs). Seems like when Lasso doesn’t want to let go of it’s CPU even after restarting the service (and apache) sometimes rebooting the whole VM calms it down for some minutes or hours. I just never know, and it doesn’t necessarily seem to track exactly w high traffic volumes. Any one of those four servers could get flaky at any time and need rebooting— or all four at once.
>
> We need a solution for this ASAP, before the weekend. Lassosoft Support can not see us until next Wednesday.
>
>
>
>
> #############################################################
>
> This message is sent to you because you are subscribed to
>  the mailing list Lasso [hidden email]
> Official list archives available at http://www.lassotalk.com
> To unsubscribe, E-mail to: <[hidden email]>
> Send administrative queries to  <[hidden email]>


#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Lasso 8.6 consuming too much CPU & RAM

Ke Carlton-3
In reply to this post by Roy Dacke
This could be any number of things:

   - Bad SQL queries / indexes
   - Loading too many rows without limits
   - Loading unnecessary columns
   - Unoptimised code
   - Too much logging
   - Thread locking
   - Image processing
   - PDF processing
   - Infinite loops
   - The list goes on...

How long does a typical page take to process?

20ms, 50ms, 100ms, 1 second+  — because that directly limits capacity.

I would recommend adding some tracing code to the system and isolate the
slow parts by monitoring / testing the live system. L-Debug for Lasso 8 is
here https://github.com/Zeroloop/l-debug/tree/Lasso-8.6 — you can use that
in console mode or the timerstack in standard mode.

I believe there's been a number of threads that have covered this over the
years — most of them should be on
http://lasso.2283332.n4.nabble.com/Lasso-Talk-f3096191.html or
http://www.lassotalk.com/

Here's a good starting point for a basic logging approach:
http://lasso.2283332.n4.nabble.com/atBegin-questions-td4635795.html

Best regards,

Ke

On Thu, Feb 11, 2016 at 11:53 AM Roy Dacke <[hidden email]> wrote:

> We have been on Lasso for years, over a decade and are now having a
> problem w 8.6 on CentOS 6.7.
>
> Lasso consumes all the CPU and RAM and has to be restarted repeatedly.
> Sometimes within a minute of restarting, Lasso needs to be restarted again.
>
> It is a problem that has plagued us for a very long time, years—
> originally we were on Mac for our Lasso. We applied scripts that check for
> various conditions leading to failure such as too little RAM available, too
> little CPU available, too many Lasso threads, too big largest Lasso thread,
> too many apache threads… It got crazy.  The scripts would work for a few
> months, then suddenly I’d be back to restarting Lasso all the time. But I’d
> tweak the scripts in some way, things would work ok again for a few months.
>
> Finally got fed up with this and migrated one group of four Lasso servers
> in a load balanced group over to CentOS, thinking this would be the magic
> bullet.  And it was for a couple weeks, I slowly rotated the new machines
> in and after a week that group was 100% CentOS, it ran great for two
> weekends (we do grocery store websites, so most traffic is on weekends)
> then it started breaking RAM limits causing swap. I had to make a simple
> script to check for that and refresh Lasso.
>
> This worked for a week but failed over the next weekend, hitting CPU
> limits.  Updated my script to check for that too, but now it wants to
> restart way too frequently and even when it does, the CPU shoots up within
> a minute or two… sometimes, other times any given server will be stable for
> hours or a whole day, sometimes not and I find myself attached to my
> computer rebooting services constantly (and sometimes entire VMs). Seems
> like when Lasso doesn’t want to let go of it’s CPU even after restarting
> the service (and apache) sometimes rebooting the whole VM calms it down for
> some minutes or hours. I just never know, and it doesn’t necessarily seem
> to track exactly w high traffic volumes. Any one of those four servers
> could get flaky at any time and need rebooting— or all four at once.
>
> We need a solution for this ASAP, before the weekend. Lassosoft Support
> can not see us until next Wednesday.
>
>

#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Lasso 8.6 consuming too much CPU & RAM

stevepiercy
As far as too much logging, specifically, in Lasso SiteAdmin:

Setup > Email > Setup

Retain Successful Messages = No
Retain Error Messages = No
Retain Encoded Messages in Log = No

If any were Yes, then also delete all the records.

Setup > Errors > Setup

Uncheck all.

If any were checked, then also delete the records or log files.  
See also:
http://www.stevepiercy.com/articles/mining-the-lasso-error-stack-for-meaning/#error-log-configuration
and
http://www.stevepiercy.com/articles/mining-the-lasso-error-stack-for-meaning/#viewing-log-files

In production, you should only log that which you absolutely
need and only during the time you need it.  And when you log,
save to a file that changes its name to include the current date
to give at least some semblance of log rolling.  A massive log
file or error database can ruin your whole day.

When I get a new client with their own environment, these are
some of the first things I check, then optimize to specs.

--steve


On 2/11/16 at 12:39 AM, [hidden email] (Ke Carlton) pronounced:

>This could be any number of things:
>
>- Bad SQL queries / indexes
>- Loading too many rows without limits
>- Loading unnecessary columns
>- Unoptimised code
>- Too much logging
>- Thread locking
>- Image processing
>- PDF processing
>- Infinite loops
>- The list goes on...
>
>How long does a typical page take to process?
>
>20ms, 50ms, 100ms, 1 second+  — because that directly limits capacity.
>
>I would recommend adding some tracing code to the system and isolate the
>slow parts by monitoring / testing the live system. L-Debug for Lasso 8 is
>here https://github.com/Zeroloop/l-debug/tree/Lasso-8.6 — you can use that
>in console mode or the timerstack in standard mode.
>
>I believe there's been a number of threads that have covered this over the
>years — most of them should be on
>http://lasso.2283332.n4.nabble.com/Lasso-Talk-f3096191.html or
>http://www.lassotalk.com/
>
>Here's a good starting point for a basic logging approach:
>http://lasso.2283332.n4.nabble.com/atBegin-questions-td4635795.html
>
>Best regards,
>
>Ke
>
>On Thu, Feb 11, 2016 at 11:53 AM Roy Dacke <[hidden email]> wrote:
>
>>We have been on Lasso for years, over a decade and are now having a
>>problem w 8.6 on CentOS 6.7.
>>
>>Lasso consumes all the CPU and RAM and has to be restarted repeatedly.
>>Sometimes within a minute of restarting, Lasso needs to be restarted again.
>>
>>It is a problem that has plagued us for a very long time, years—
>>originally we were on Mac for our Lasso. We applied scripts that check for
>>various conditions leading to failure such as too little RAM available, too
>>little CPU available, too many Lasso threads, too big largest Lasso thread,
>>too many apache threads… It got crazy.  The scripts would work for a few
>>months, then suddenly I’d be back to restarting Lasso all the time. But I’d
>>tweak the scripts in some way, things would work ok again for a few months.
>>
>>Finally got fed up with this and migrated one group of four Lasso servers
>>in a load balanced group over to CentOS, thinking this would be the magic
>>bullet.  And it was for a couple weeks, I slowly rotated the new machines
>>in and after a week that group was 100% CentOS, it ran great for two
>>weekends (we do grocery store websites, so most traffic is on weekends)
>>then it started breaking RAM limits causing swap. I had to make a simple
>>script to check for that and refresh Lasso.
>>
>>This worked for a week but failed over the next weekend, hitting CPU
>>limits.  Updated my script to check for that too, but now it wants to
>>restart way too frequently and even when it does, the CPU shoots up within
>>a minute or two… sometimes, other times any given server will be stable for
>>hours or a whole day, sometimes not and I find myself attached to my
>>computer rebooting services constantly (and sometimes entire VMs). Seems
>>like when Lasso doesn’t want to let go of it’s CPU even after restarting
>>the service (and apache) sometimes rebooting the whole VM calms it down for
>>some minutes or hours. I just never know, and it doesn’t necessarily seem
>>to track exactly w high traffic volumes. Any one of those four servers
>>could get flaky at any time and need rebooting— or all four at once.
>>
>>We need a solution for this ASAP, before the weekend. Lassosoft Support
>>can not see us until next Wednesday.
>>
>>
>
>#############################################################
>
>This message is sent to you because you are subscribed to
>the mailing list Lasso [hidden email]
>Official list archives available at http://www.lassotalk.com
>To unsubscribe, E-mail to: <[hidden email]>
>Send administrative queries to  <[hidden email]>

-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
Steve Piercy              Website Builder              Soquel, CA
<[hidden email]>               <http://www.StevePiercy.com/>


#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>