Lasso filling the hard disk until server hangs up

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Lasso filling the hard disk until server hangs up

bienvenue
Hi list

Another unexpected an strange behavior happened twice this week on two distinct CentOS 6.5 / apache VM and Lasso 8.6.2

It seems like if the hard disk has suddenly been filled with datas, maybe from lasso SQLite ?.
At the rythm of aroung 1 Go / minute...
Then the system hangs up.
No way to access Lasso Site or Lasso Server admin…

On the first VM we tried to restart several time the Lasso service from the terminal, but the only way to come back to normal was to restart the machine
The second time it happened on the other machine, restarting the service did the job.

We are still investigating
Rather difficult as we did not found any logged error.
Feedback from users who called us for assistance did not give us any hints…
I've just seen once something like "error memory storage" thrown by the app in the browser

I will try to post more information if we find out what happens.
But if one of us did experiment such an experience or has some thought about…

Regards
JPK
----
E SYSTEMES - Lille France

#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Lasso filling the hard disk until server hangs up

Steffan A. Cline
You say it's filling with dates.

What do you mean?

Are you looking in /tmp?

If you have a datasource being queried that is not responding, it will
make lasso seem unresponsive too.

Are these files you're talking about owned by lasso?



Thanks,
Steffan

---------------------------------------------------------------
T E L  6 0 2 . 7 9 3 . 0 0 1 4 | F A X  6 0 2 . 9 7 1 . 1 6 9 4
Steffan A. Cline   [hidden email]
http://www.ExecuChoice.net                 Phoenix, Arizona USA
                 
---------------------------------------------------------------






On 1/30/15, 11:11 AM, "bienvenue" <[hidden email]> wrote:

>Hi list
>
>Another unexpected an strange behavior happened twice this week on two
>distinct CentOS 6.5 / apache VM and Lasso 8.6.2
>
>It seems like if the hard disk has suddenly been filled with datas, maybe
>from lasso SQLite ?.
>At the rythm of aroung 1 Go / minute...
>Then the system hangs up.
>No way to access Lasso Site or Lasso Server adminŠ
>
>On the first VM we tried to restart several time the Lasso service from
>the terminal, but the only way to come back to normal was to restart the
>machine
>The second time it happened on the other machine, restarting the service
>did the job.
>
>We are still investigating
>Rather difficult as we did not found any logged error.
>Feedback from users who called us for assistance did not give us any
>hintsŠ
>I've just seen once something like "error memory storage" thrown by the
>app in the browser
>
>I will try to post more information if we find out what happens.
>But if one of us did experiment such an experience or has some thought
>aboutŠ
>
>Regards
>JPK
>----
>E SYSTEMES - Lille France
>
>#############################################################
>
>This message is sent to you because you are subscribed to
>  the mailing list Lasso [hidden email]
>Official list archives available at http://www.lassotalk.com
>To unsubscribe, E-mail to: <[hidden email]>
>Send administrative queries to  <[hidden email]>
>



#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Lasso filling the hard disk until server hangs up

bienvenue
Hi Steffan

I meant the disk space is filled with "datas". In a few minutes, no available disk storage and the server hangs up.
Restarting the lasso service in one case or the machine itself in the other did erase those datas and allowed the server to run as expected.

We were in emmergency to allow the app users to resume working.
Just obviously noticed the cause was the lack of disk storage space…
We tried, but did not find out the kind of the datas written on disk

We suppose something went wrong with Lasso as in one case restarting the service did solve the case.

On those servers there are only one web lasso driven app running on its own instance

Regards
JPK
----
E SYSTEMES - Lille France

Le 30 janv. 2015 à 19:25, Steffan A. Cline <[hidden email]> a écrit :

> You say it's filling with dates.
>
> What do you mean?
>
> Are you looking in /tmp?
>
> If you have a datasource being queried that is not responding, it will
> make lasso seem unresponsive too.
>
> Are these files you're talking about owned by lasso?
>
>
>
> Thanks,
> Steffan
>
>
> On 1/30/15, 11:11 AM, "bienvenue" <[hidden email]> wrote:
>
>> Hi list
>>
>> Another unexpected an strange behavior happened twice this week on two
>> distinct CentOS 6.5 / apache VM and Lasso 8.6.2
>>
>> It seems like if the hard disk has suddenly been filled with datas, maybe
>> from lasso SQLite ?.
>> At the rythm of aroung 1 Go / minute...
>> Then the system hangs up.
>> No way to access Lasso Site or Lasso Server adminŠ
>>
>> On the first VM we tried to restart several time the Lasso service from
>> the terminal, but the only way to come back to normal was to restart the
>> machine
>> The second time it happened on the other machine, restarting the service
>> did the job.
>>
>> We are still investigating
>> Rather difficult as we did not found any logged error.
>> Feedback from users who called us for assistance did not give us any
>> hintsŠ
>> I've just seen once something like "error memory storage" thrown by the
>> app in the browser
>>
>> I will try to post more information if we find out what happens.
>> But if one of us did experiment such an experience or has some thought
>> aboutŠ
>>
>> Regards
>> JPK
>> ----
>> E SYSTEMES - Lille France


#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Lasso filling the hard disk until server hangs up

stevepiercy
This smells like a log management issue.  It could be too much
logging in Lasso, not rotating Apache logs, or some other
logging event.

In Lasso 8 SiteAdmin, navigate to Utility > Errors > Setup, and
check only the bare minimum required to a single place.  In
production I usually only log critical errors to console, and
read /tmp/LassoService8.out as needed.

See also rotatelogs for Apache.
https://httpd.apache.org/docs/2.2/programs/rotatelogs.html

--steve


On 1/31/15 at 9:01 AM, [hidden email] (bienvenue) pronounced:

>Hi Steffan
>
>I meant the disk space is filled with "datas". In a few
>minutes, no available disk storage and the server hangs up.
>Restarting the lasso service in one case or the machine itself
>in the other did erase those datas and allowed the server to
>run as expected.
>
>We were in emmergency to allow the app users to resume working.
>Just obviously noticed the cause was the lack of disk storage space…
>We tried, but did not find out the kind of the datas written on disk
>
>We suppose something went wrong with Lasso as in one case
>restarting the service did solve the case.
>
>On those servers there are only one web lasso driven app running on its own instance
>
>Regards
>JPK
>----
>E SYSTEMES - Lille France
>
>Le 30 janv. 2015 à 19:25, Steffan A. Cline <[hidden email]> a écrit :
>
>>You say it's filling with dates.
>>
>>What do you mean?
>>
>>Are you looking in /tmp?
>>
>>If you have a datasource being queried that is not responding, it will
>>make lasso seem unresponsive too.
>>
>>Are these files you're talking about owned by lasso?
>>
>>
>>
>>Thanks,
>>Steffan
>>
>>
>>On 1/30/15, 11:11 AM, "bienvenue" <[hidden email]> wrote:
>>
>>> Hi list
>>>   Another unexpected an strange behavior happened twice this
>>>week on two
>>> distinct CentOS 6.5 / apache VM and Lasso 8.6.2
>>>   It seems like if the hard disk has suddenly been filled
>>>with datas, maybe
>>> from lasso SQLite ?.
>>> At the rythm of aroung 1 Go / minute...
>>> Then the system hangs up.
>>> No way to access Lasso Site or Lasso Server adminŠ
>>>   On the first VM we tried to restart several time the Lasso
>>>service from
>>> the terminal, but the only way to come back to normal was to restart the
>>> machine
>>> The second time it happened on the other machine, restarting the service
>>> did the job.
>>>   We are still investigating
>>> Rather difficult as we did not found any logged error.
>>> Feedback from users who called us for assistance did not give us any
>>> hintsŠ
>>> I've just seen once something like "error memory storage" thrown by the
>>> app in the browser
>>>   I will try to post more information if we find out what happens.
>>> But if one of us did experiment such an experience or has some thought
>>> aboutŠ
>>>   Regards
>>> JPK
>>> ----
>>> E SYSTEMES - Lille France
>
>
>#############################################################
>
>This message is sent to you because you are subscribed to
>the mailing list Lasso [hidden email]
>Official list archives available at http://www.lassotalk.com
>To unsubscribe, E-mail to: <[hidden email]>
>Send administrative queries to  <[hidden email]>

-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
Steve Piercy              Website Builder              Soquel, CA
<[hidden email]>               <http://www.StevePiercy.com/>


#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Lasso filling the hard disk until server hangs up

bienvenue
Hi Steve

A logging event. That's what we guessed.

As a hint, when the first "crash" occured the lasso error logging in the admin was set to record all possible stuff in error db. Theese servers setup are a recent install (migration from OS X server to CentOS VM). We did set those preferences to easily check instal issues, and then unfortunatly forgot to unset it when all issues were resolved…
As far as I know, it was not the case on the second VM where error logging was set to the minimum…

Even if I do not wish that this behaviour occurs again, next time if ever, we'll try to save LassoService8.out before restart anything.

Regards
JPK
----
E SYSTEMES - Lille France

Le 31 janv. 2015 à 10:29, Steve Piercy - Website Builder <[hidden email]> a écrit :

> This smells like a log management issue.  It could be too much logging in Lasso, not rotating Apache logs, or some other logging event.
>
> In Lasso 8 SiteAdmin, navigate to Utility > Errors > Setup, and check only the bare minimum required to a single place.  In production I usually only log critical errors to console, and read /tmp/LassoService8.out as needed.
>
> See also rotatelogs for Apache.
> https://httpd.apache.org/docs/2.2/programs/rotatelogs.html
>
> --steve
>
>
> On 1/31/15 at 9:01 AM, [hidden email] (bienvenue) pronounced:
>
>> Hi Steffan
>>
>> I meant the disk space is filled with "datas". In a few minutes, no available disk storage and the server hangs up.
>> Restarting the lasso service in one case or the machine itself in the other did erase those datas and allowed the server to run as expected.
>>
>> We were in emmergency to allow the app users to resume working.
>> Just obviously noticed the cause was the lack of disk storage space…
>> We tried, but did not find out the kind of the datas written on disk
>>
>> We suppose something went wrong with Lasso as in one case restarting the service did solve the case.
>>
>> On those servers there are only one web lasso driven app running on its own instance
>>
>> Regards
>> JPK
>> ----
>> E SYSTEMES - Lille France
>>
>> Le 30 janv. 2015 à 19:25, Steffan A. Cline <[hidden email]> a écrit :
>>
>>> You say it's filling with dates.
>>>
>>> What do you mean?
>>>
>>> Are you looking in /tmp?
>>>
>>> If you have a datasource being queried that is not responding, it will
>>> make lasso seem unresponsive too.
>>>
>>> Are these files you're talking about owned by lasso?
>>>
>>>
>>>
>>> Thanks,
>>> Steffan
>>>
>>>
>>> On 1/30/15, 11:11 AM, "bienvenue" <[hidden email]> wrote:
>>>
>>>> Hi list
>>>>  Another unexpected an strange behavior happened twice this week on two
>>>> distinct CentOS 6.5 / apache VM and Lasso 8.6.2
>>>>  It seems like if the hard disk has suddenly been filled with datas, maybe
>>>> from lasso SQLite ?.
>>>> At the rythm of aroung 1 Go / minute...
>>>> Then the system hangs up.
>>>> No way to access Lasso Site or Lasso Server adminŠ
>>>>  On the first VM we tried to restart several time the Lasso service from
>>>> the terminal, but the only way to come back to normal was to restart the
>>>> machine
>>>> The second time it happened on the other machine, restarting the service
>>>> did the job.
>>>>  We are still investigating
>>>> Rather difficult as we did not found any logged error.
>>>> Feedback from users who called us for assistance did not give us any
>>>> hintsŠ
>>>> I've just seen once something like "error memory storage" thrown by the
>>>> app in the browser
>>>>  I will try to post more information if we find out what happens.
>>>> But if one of us did experiment such an experience or has some thought
>>>> aboutŠ
>>>>  Regards
>>>> JPK
>>>> ----
>>>> E SYSTEMES - Lille France
>>
>>
>> #############################################################
>>
>> This message is sent to you because you are subscribed to
>> the mailing list Lasso [hidden email]
>> Official list archives available at http://www.lassotalk.com
>> To unsubscribe, E-mail to: <[hidden email]>
>> Send administrative queries to  <[hidden email]>
>
> -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
> Steve Piercy              Website Builder              Soquel, CA
> <[hidden email]>               <http://www.StevePiercy.com/>
>
>
> #############################################################
>
> This message is sent to you because you are subscribed to
> the mailing list Lasso [hidden email]
> Official list archives available at http://www.lassotalk.com
> To unsubscribe, E-mail to: <[hidden email]>
> Send administrative queries to  <[hidden email]>


#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Lasso filling the hard disk until server hangs up

stevepiercy
If I recall correctly, you can stop Lasso, delete the SQLite
error database, restart Lasso, and the error dbs will recreate.  
I store no errors in the dbs, so my memory is vague.

If you want something more permanent, you can save to
LassoErrors.txt as well, assuming you manage it by pruning or
rotation, and configure it to write minimal errors.

--steve


On 1/31/15 at 11:31 AM, [hidden email] (bienvenue) pronounced:

>Hi Steve
>
>A logging event. That's what we guessed.
>
>As a hint, when the first "crash" occured the lasso error
>logging in the admin was set to record all possible stuff in
>error db. Theese servers setup are a recent install (migration
>from OS X server to CentOS VM). We did set those preferences to
>easily check instal issues, and then unfortunatly forgot to
>unset it when all issues were resolved…
>As far as I know, it was not the case on the second VM where
>error logging was set to the minimum…
>
>Even if I do not wish that this behaviour occurs again, next
>time if ever, we'll try to save LassoService8.out before
>restart anything.
>
>Regards
>JPK
>----
>E SYSTEMES - Lille France
>
>Le 31 janv. 2015 à 10:29, Steve Piercy - Website Builder
><[hidden email]> a écrit :
>
>>This smells like a log management issue.  It could be too much logging in Lasso, not
>rotating Apache logs, or some other logging event.
>>
>>In Lasso 8 SiteAdmin, navigate to Utility > Errors > Setup, and check only the bare
>minimum required to a single place.  In production I usually
>only log critical errors to console, and read
>/tmp/LassoService8.out as needed.
>>
>>See also rotatelogs for Apache.
>>https://httpd.apache.org/docs/2.2/programs/rotatelogs.html
>>
>>--steve
>>
>>
>>On 1/31/15 at 9:01 AM, [hidden email] (bienvenue) pronounced:
>>
>>> Hi Steffan
>>>   I meant the disk space is filled with "datas". In a few
>>>minutes, no available disk
>storage and the server hangs up.
>>> Restarting the lasso service in one case or the machine itself in the other did
>erase those datas and allowed the server to run as expected.
>>>   We were in emmergency to allow the app users to resume working.
>>> Just obviously noticed the cause was the lack of disk storage space…
>>> We tried, but did not find out the kind of the datas written on disk
>>>   We suppose something went wrong with Lasso as in one case
>>>restarting the service did
>solve the case.
>>>   On those servers there are only one web lasso driven app
>>>running on its own instance
>>>   Regards
>>> JPK
>>> ----
>>> E SYSTEMES - Lille France
>>>   Le 30 janv. 2015 à 19:25, Steffan A. Cline
>>><[hidden email]> a écrit :
>>>
>>>> You say it's filling with dates.
>>>>   What do you mean?
>>>>   Are you looking in /tmp?
>>>>   If you have a datasource being queried that is not
>>>>responding, it will
>>>> make lasso seem unresponsive too.
>>>>   Are these files you're talking about owned by lasso?
>>>>     Thanks,
>>>> Steffan
>>>>    On 1/30/15, 11:11 AM, "bienvenue"
>>>><[hidden email]> wrote:
>>>>
>>>>> Hi list
>>>>>  Another unexpected an strange behavior happened twice this week on two
>>>>> distinct CentOS 6.5 / apache VM and Lasso 8.6.2
>>>>>  It seems like if the hard disk has suddenly been filled with datas, maybe
>>>>> from lasso SQLite ?.
>>>>> At the rythm of aroung 1 Go / minute...
>>>>> Then the system hangs up.
>>>>> No way to access Lasso Site or Lasso Server adminŠ
>>>>>  On the first VM we tried to restart several time the Lasso service from
>>>>> the terminal, but the only way to come back to normal was to restart the
>>>>> machine
>>>>> The second time it happened on the other machine, restarting the service
>>>>> did the job.
>>>>>  We are still investigating
>>>>> Rather difficult as we did not found any logged error.
>>>>> Feedback from users who called us for assistance did not give us any
>>>>> hintsŠ
>>>>> I've just seen once something like "error memory storage" thrown by the
>>>>> app in the browser
>>>>>  I will try to post more information if we find out what happens.
>>>>> But if one of us did experiment such an experience or has some thought
>>>>> aboutŠ
>>>>>  Regards
>>>>> JPK
>>>>> ----
>>>>> E SYSTEMES - Lille France
>>>    #############################################################
>>>   This message is sent to you because you are subscribed to
>>> the mailing list Lasso [hidden email]
>>> Official list archives available at http://www.lassotalk.com
>>> To unsubscribe, E-mail to: <[hidden email]>
>>> Send administrative queries to  <[hidden email]>
>>
>>-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
>>Steve Piercy              Website Builder              Soquel, CA
>><[hidden email]>               <http://www.StevePiercy.com/>
>>
>>
>>#############################################################
>>
>>This message is sent to you because you are subscribed to
>>the mailing list Lasso [hidden email]
>>Official list archives available at http://www.lassotalk.com
>>To unsubscribe, E-mail to: <[hidden email]>
>>Send administrative queries to  <[hidden email]>
>
>
>#############################################################
>
>This message is sent to you because you are subscribed to
>the mailing list Lasso [hidden email]
>Official list archives available at http://www.lassotalk.com
>To unsubscribe, E-mail to: <[hidden email]>
>Send administrative queries to  <[hidden email]>

-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
Steve Piercy              Website Builder              Soquel, CA
<[hidden email]>               <http://www.StevePiercy.com/>


#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>