lasso 8.6.2 consuming too many resources

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

lasso 8.6.2 consuming too many resources

Todd Vainisi-2
Hi List,

My lasso instance has a tendency to continually use more and more server resources until it can no longer perform certain functions (like interacting with google cal or zipping files in a shell).  I have to issue a ‘lasso8ctl restart’ and then things work again.  Sometimes it takes a couple days to reach this point, sometimes it only takes a few hours.  Lately its been closer to the 6-12 hours.  The constant monitoring is driving me nuts and it doesn’t prevent users from experiencing errors prior to my restart.

I’ve written in about this before and gotten conflicting advice on what to do about it.  I’ve been told to write a process that will automatically restart the server every few days, but then my co-locator told me that it was a bad idea because sometimes it fails to restart (though I’ve never seen this happen when I do it manually) and then everything is down until I look.

I’d like to be more proactive and figure out what the actual cause of the consumption is and fix it rather than look for methods to react to a broken service.  My websites still mostly run when this problem is occurring - only certain features fail.  So I tried running those pages lots of times to see if I could change the resource consumption.  I couldn’t.

Is there a log file that would help me to understand what causes this?  I can’t keep checking on my server twice a day forever (and really I need to do it more often than that).

Here’s the output for the lasso8 process in top:
  PID USER    PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
12292 lasso     20   0 6718m 2.3g  18m S     0.7     17.0       17:35.44 Lasso8Service

It’s the 2.3g number that is so problematic, I believe.  That starts off at about 70mg after the restart.

Todd Vainisi





#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: lasso 8.6.2 consuming too many resources

stevepiercy
Unfortunately there is no silver bullet.

Use monitoring software to:
* send you an alert when memory consumption exceeds a threshold
you set
* graph a history of memory consumption
Look for patterns or unusual events.  Check log files for
unusual events.

Time your code execution.

What does your app do?  Huge PDFs or other media manipulation?  
Queries with huge amounts of data going back and forth?  Storing
too much data in sessions?

--steve


On 6/17/15 at 10:26 AM, [hidden email] (Todd
Vainisi) pronounced:

>Hi List,
>
>My lasso instance has a tendency to continually use more and
>more server resources until it can no longer perform certain
>functions (like interacting with google cal or zipping files in
>a shell).  I have to issue a ‘lasso8ctl restart’ and then
>things work again.  Sometimes it takes a couple days to reach
>this point, sometimes it only takes a few hours.  Lately its
>been closer to the 6-12 hours.  The constant monitoring is
>driving me nuts and it doesn’t prevent users from
>experiencing errors prior to my restart.
>
>I’ve written in about this before and gotten conflicting
>advice on what to do about it.  I’ve been told to write a
>process that will automatically restart the server every few
>days, but then my co-locator told me that it was a bad idea
>because sometimes it fails to restart (though I’ve never seen
>this happen when I do it manually) and then everything is down
>until I look.
>
>I’d like to be more proactive and figure out what the actual
>cause of the consumption is and fix it rather than look for
>methods to react to a broken service.  My websites still mostly
>run when this problem is occurring - only certain features
>fail.  So I tried running those pages lots of times to see if I
>could change the resource consumption.  I couldn’t.
>
>Is there a log file that would help me to understand what
>causes this?  I can’t keep checking on my server twice a day
>forever (and really I need to do it more often than that).
>
>Here’s the output for the lasso8 process in top:
>PID USER    PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>12292 lasso     20   0 6718m 2.3g  18m S     0.7     17.0       17:35.44 Lasso8Service
>
>It’s the 2.3g number that is so problematic, I believe.  That
>starts off at about 70mg after the restart.
>
>Todd Vainisi
>
>
>
>
>
>#############################################################
>
>This message is sent to you because you are subscribed to
>the mailing list Lasso [hidden email]
>Official list archives available at http://www.lassotalk.com
>To unsubscribe, E-mail to: <[hidden email]>
>Send administrative queries to  <[hidden email]>

-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
Steve Piercy              Website Builder              Soquel, CA
<[hidden email]>               <http://www.StevePiercy.com/>


#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: lasso 8.6.2 consuming too many resources

Mason Miller
In reply to this post by Todd Vainisi-2
I use a perl script to request a lasso page that returns the string 'true' if all of the tests I have on the page perform acceptably. If it returns false or fails completely, the perl script uses SCP to restart lasso.

Mason

Sent from my iPhone

> On Jun 17, 2015, at 10:26 AM, Todd Vainisi <[hidden email]> wrote:
>
> Hi List,
>
> My lasso instance has a tendency to continually use more and more server resources until it can no longer perform certain functions (like interacting with google cal or zipping files in a shell).  I have to issue a ‘lasso8ctl restart’ and then things work again.  Sometimes it takes a couple days to reach this point, sometimes it only takes a few hours.  Lately its been closer to the 6-12 hours.  The constant monitoring is driving me nuts and it doesn’t prevent users from experiencing errors prior to my restart.
>
> I’ve written in about this before and gotten conflicting advice on what to do about it.  I’ve been told to write a process that will automatically restart the server every few days, but then my co-locator told me that it was a bad idea because sometimes it fails to restart (though I’ve never seen this happen when I do it manually) and then everything is down until I look.
>
> I’d like to be more proactive and figure out what the actual cause of the consumption is and fix it rather than look for methods to react to a broken service.  My websites still mostly run when this problem is occurring - only certain features fail.  So I tried running those pages lots of times to see if I could change the resource consumption.  I couldn’t.
>
> Is there a log file that would help me to understand what causes this?  I can’t keep checking on my server twice a day forever (and really I need to do it more often than that).
>
> Here’s the output for the lasso8 process in top:
>  PID USER    PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 12292 lasso     20   0 6718m 2.3g  18m S     0.7     17.0       17:35.44 Lasso8Service
>
> It’s the 2.3g number that is so problematic, I believe.  That starts off at about 70mg after the restart.
>
> Todd Vainisi
>
>
>
>
>
> #############################################################
>
> This message is sent to you because you are subscribed to
>  the mailing list Lasso [hidden email]
> Official list archives available at http://www.lassotalk.com
> To unsubscribe, E-mail to: <[hidden email]>
> Send administrative queries to  <[hidden email]>

#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: lasso 8.6.2 consuming too many resources

Wade Maxfield
In reply to this post by Todd Vainisi-2
> On 18/06/2015, at 02:26, Todd Vainisi <[hidden email]> wrote:
>
> I’ve written in about this before and gotten conflicting advice on what to do about it.  I’ve been told to write a process that will automatically restart the server every few days, but then my co-locator told me that it was a bad idea because sometimes it fails to restart (though I’ve never seen this happen when I do it manually) and then everything is down until I look.
>


I use monit to monitor this type of thing and restart the lasso SITE that has the issue rather than the overall Lasso8service.  As long as the primary lasso8service is running then a request that comes in to the site in question will cause the lasso8service for that site to be started if it isn't already.  The first request (that causes this start) will generally fail but you're good to go after that, so as part of your restart script you should call the site in question (probably a second time after a 5-10s delay), that way the first request that normally fails will be from you and of no consequence.


> I’d like to be more proactive and figure out what the actual cause of the consumption is and fix it rather than look for methods to react to a broken service.  My websites still mostly run when this problem is occurring - only certain features fail.  So I tried running those pages lots of times to see if I could change the resource consumption.  I couldn’t.
>

Tough to pinpoint, it's all down to your code. PDF tags, cache tags, and lots of image tags can all add to the memory requirements.


> Is there a log file that would help me to understand what causes this?  I can’t keep checking on my server twice a day forever (and really I need to do it more often than that).

This is where monit saves you the trouble.  Even if you don't use it to restart things you can use it to alert you and let you do the manual restart.  

 - Wade






#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: lasso 8.6.2 consuming too many resources

Marc Vos-3
Something like this happened to me on CentOS. You might find the error if you disable all 'handle - /handle' error trapping and set error-reporting to 'full' and then see what happens. This way I found the error in one of my own tags where I least expected it.

- -
Regards,
Marc

http://marc.vos.net/
http://nl.linkedin.com/in/mhevos



> On 18 jun. 2015, at 02:45, Wade Maxfield <[hidden email]> wrote:
>
>> On 18/06/2015, at 02:26, Todd Vainisi <[hidden email]> wrote:
>>
>> I’ve written in about this before and gotten conflicting advice on what to do about it.  I’ve been told to write a process that will automatically restart the server every few days, but then my co-locator told me that it was a bad idea because sometimes it fails to restart (though I’ve never seen this happen when I do it manually) and then everything is down until I look.
>>
>
>
> I use monit to monitor this type of thing and restart the lasso SITE that has the issue rather than the overall Lasso8service.  As long as the primary lasso8service is running then a request that comes in to the site in question will cause the lasso8service for that site to be started if it isn't already.  The first request (that causes this start) will generally fail but you're good to go after that, so as part of your restart script you should call the site in question (probably a second time after a 5-10s delay), that way the first request that normally fails will be from you and of no consequence.
>
>
>> I’d like to be more proactive and figure out what the actual cause of the consumption is and fix it rather than look for methods to react to a broken service.  My websites still mostly run when this problem is occurring - only certain features fail.  So I tried running those pages lots of times to see if I could change the resource consumption.  I couldn’t.
>>
>
> Tough to pinpoint, it's all down to your code. PDF tags, cache tags, and lots of image tags can all add to the memory requirements.
>
>
>> Is there a log file that would help me to understand what causes this?  I can’t keep checking on my server twice a day forever (and really I need to do it more often than that).
>
> This is where monit saves you the trouble.  Even if you don't use it to restart things you can use it to alert you and let you do the manual restart.  
>
> - Wade
>
>
>
>
>
>
> #############################################################
>
> This message is sent to you because you are subscribed to
>  the mailing list Lasso [hidden email]
> Official list archives available at http://www.lassotalk.com
> To unsubscribe, E-mail to: <[hidden email]>
> Send administrative queries to  <[hidden email]>


#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>