Lasso crashing on demand.. How to diagnose?

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Lasso crashing on demand.. How to diagnose?

Daniel Parker
Lasso 9.2.7 CentOS 6

I am having an issue where Lasso is randomly crashing randomly (No lasso page will load) every 1 - 3 days. I have not been able to figure out what exactly is causing it, but I can now reproduce it on demand.

If I visit:

/lasso9/Admin/logbook

And click on the settings button, make a change (assuming the setting window appears, sometimes it does not), then click save. Lasso will crash on my server. If I do the same on a development machine it works.

What actually happens during this process? Am I right in thinking that it will be writing to SQlite and that is where i should be looking?

Any suggestions on how to diagnose / fix or should I just rebuild the server? (I would prefer not to do this)

Thanks
Daniel Parker
#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Lasso crashing on demand.. How to diagnose?

Jonathan Guthrie-3
Daniel how much are you logging?
On a production server you should have log to SQLite turned off.
Your SQLite log file might be really large.

cd /var/lasso/instances/SQLiteDBs
ls -lh

And have a look how large your SQLite DB's are...

Thanks
Jono

On Jun 9, 2014, at 4:04 PM, Daniel Parker <[hidden email]> wrote:

> Lasso 9.2.7 CentOS 6
>
> I am having an issue where Lasso is randomly crashing randomly (No lasso page will load) every 1 - 3 days. I have not been able to figure out what exactly is causing it, but I can now reproduce it on demand.
>
> If I visit:
>
> /lasso9/Admin/logbook
>
> And click on the settings button, make a change (assuming the setting window appears, sometimes it does not), then click save. Lasso will crash on my server. If I do the same on a development machine it works.
>
> What actually happens during this process? Am I right in thinking that it will be writing to SQlite and that is where i should be looking?
>
> Any suggestions on how to diagnose / fix or should I just rebuild the server? (I would prefer not to do this)
>
> Thanks
> Daniel Parker
> #############################################################
> This message is sent to you because you are subscribed to
>  the mailing list Lasso [hidden email]
> Official list archives available at http://www.lassotalk.com
> To unsubscribe, E-mail to: <[hidden email]>
> Send administrative queries to  <[hidden email]>

Jono

----------------------------
Jonathan Guthrie
[hidden email]
@iamjono
LassoSoft Inc.
AIM Chatroom: lassochat

#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Lasso crashing on demand.. How to diagnose?

Daniel Parker

On 10/06/2014, at 8:10 am, Jonathan Guthrie <[hidden email]> wrote:

> Daniel how much are you logging?
> On a production server you should have log to SQLite turned off.
> Your SQLite log file might be really large.
>
> cd /var/lasso/instances/SQLiteDBs
> ls -lh
>
> And have a look how large your SQLite DB's are...

-rw-r--r-- 1 _lasso root    98K May 26 09:06 database_registry
-rw-r--r-- 1 _lasso root      0 Sep 17  2011 jdbc_datasources
-rw-r--r-- 1 _lasso root   4.0K Sep 17  2011 lassoapps_settings
-rw-r--r-- 1 _lasso root    19M Jun 10 07:58 lasso_email
-rw-r--r-- 1 _lasso root   4.0K Aug 15  2012 lasso_internal
-rw-r--r-- 1 _lasso root   2.5G Jun 10 07:44 lasso_logbook
-rw-r--r-- 1 _lasso root    11M Jun 10 07:57 lasso_session
-rw-r--r-- 1 _lasso root   2.0K Aug 15  2012 mysql_session_driver_data
-rw-r--r-- 1 _lasso _lasso 2.0K Jun 24  2012 odbc_session_driver_data
-rw-r--r-- 1 _lasso root   7.0K Sep 17  2011 security_auth_registry

I'm guessing that 2.5GB is bad.

Daniel

#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Lasso crashing on demand.. How to diagnose?

stevepiercy
On 6/10/14 at 8:19 AM, [hidden email] (Daniel Parker) pronounced:

>On 10/06/2014, at 8:10 am, Jonathan Guthrie <[hidden email]> wrote:
>
>>Daniel how much are you logging?
>>On a production server you should have log to SQLite turned off.
>>Your SQLite log file might be really large.
>>
>>cd /var/lasso/instances/SQLiteDBs
>>ls -lh
>>
>>And have a look how large your SQLite DB's are...
>
>-rw-r--r-- 1 _lasso root    98K May 26 09:06 database_registry
>-rw-r--r-- 1 _lasso root      0 Sep 17  2011 jdbc_datasources
>-rw-r--r-- 1 _lasso root   4.0K Sep 17  2011 lassoapps_settings
>-rw-r--r-- 1 _lasso root    19M Jun 10 07:58 lasso_email
>-rw-r--r-- 1 _lasso root   4.0K Aug 15  2012 lasso_internal
>-rw-r--r-- 1 _lasso root   2.5G Jun 10 07:44 lasso_logbook
>-rw-r--r-- 1 _lasso root    11M Jun 10 07:57 lasso_session
>-rw-r--r-- 1 _lasso root   2.0K Aug 15  2012 mysql_session_driver_data
>-rw-r--r-- 1 _lasso _lasso 2.0K Jun 24  2012 odbc_session_driver_data
>-rw-r--r-- 1 _lasso root   7.0K Sep 17  2011 security_auth_registry
>
>I'm guessing that 2.5GB is bad.

Yup.

I would do this:

killall lassoserver
sudo mv lasso_logbook lasso_logbook.freaking_huge_monstrosity
lassoserver &

http://www.lassosoft.com/Setup-Guide-Configuring-on-Mac-via-Subversion#heading9

and see if Lasso recreates the db automatically.  If not, then
you should move the db back in place (stop lassoserver first),
then use an SQLite db admin tool to remove the unneeded records,
then start lassoserver.

Suggest submitting a feature request, i.e., warn the
administrator of a too large SQLite db with instructions of how
to avoid repeating the issue, or perhaps automatically delete
old records to keep it trim with a notification to the developer.

--steve

-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
Steve Piercy              Website Builder              Soquel, CA
<[hidden email]>               <http://www.StevePiercy.com/>

#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Lasso crashing on demand.. How to diagnose?

Jonathan Guthrie-3
In reply to this post by Daniel Parker
Yes, 2.5 Gb is bad.

Either delete the file or:

Make sure Lasso 9 instance is stopped.

sqlite3 /var/lasso/instances/SQLiteDBs/lasso_logbook
DELETE FROM logbook;
VACUUM;
.quit

Then start the instance, go to the log settings and turn OFF log any statements to database, and save.

And you might wish to reconsider how many log_* statements you're using :)


On Jun 9, 2014, at 4:19 PM, Daniel Parker <[hidden email]> wrote:

> -rw-r--r-- 1 _lasso root    98K May 26 09:06 database_registry
> -rw-r--r-- 1 _lasso root      0 Sep 17  2011 jdbc_datasources
> -rw-r--r-- 1 _lasso root   4.0K Sep 17  2011 lassoapps_settings
> -rw-r--r-- 1 _lasso root    19M Jun 10 07:58 lasso_email
> -rw-r--r-- 1 _lasso root   4.0K Aug 15  2012 lasso_internal
> -rw-r--r-- 1 _lasso root   2.5G Jun 10 07:44 lasso_logbook
> -rw-r--r-- 1 _lasso root    11M Jun 10 07:57 lasso_session
> -rw-r--r-- 1 _lasso root   2.0K Aug 15  2012 mysql_session_driver_data
> -rw-r--r-- 1 _lasso _lasso 2.0K Jun 24  2012 odbc_session_driver_data
> -rw-r--r-- 1 _lasso root   7.0K Sep 17  2011 security_auth_registry
>
> I'm guessing that 2.5GB is bad.



Jono

----------------------------
Jonathan Guthrie
[hidden email]
@iamjono
LassoSoft Inc.
AIM Chatroom: lassochat

#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Lasso crashing on demand.. How to diagnose?

Daniel Parker
In reply to this post by stevepiercy

On 10/06/2014, at 9:02 am, Steve Piercy - Web Site Builder <[hidden email]> wrote:

> killall lassoserver
> sudo mv lasso_logbook lasso_logbook.freaking_huge_monstrosity
> lassoserver &

Fixed it.. Yay.. Thanks.

Daniel
#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Lasso crashing on demand.. How to diagnose?

Daniel Parker
In reply to this post by Jonathan Guthrie-3

On 10/06/2014, at 9:11 am, Jonathan Guthrie <[hidden email]> wrote:

> And you might wish to reconsider how many log_* statements you're using :)

Yes.. funny thing was I was increasing the usage of them to try to diagnose the issue. All fixed now. Thanks.

Daniel
#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Lasso crashing on demand.. How to diagnose?

Steve Upton
In reply to this post by Jonathan Guthrie-3

On Jun 9, 2014, at 2:11 PM, Jonathan Guthrie <[hidden email]> wrote:

> Yes, 2.5 Gb is bad.

Can you explain “bad” (other than "Try to imagine all life as you know it stopping instantaneously and every molecule in your body exploding at the speed of light” )

I have a log that’s about 950 MB and am also logging to a flat text file that grows rather large (multi-gig) sometimes.

How does that impact a site?

thanks,

Steve



#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Lasso crashing on demand.. How to diagnose?

Bil Corry-3
Log files should be rotated periodically so that they don't become too
large for the file system (or too large for the tools used to parse them),
and it's easier to archive or remove older logs that you no longer need.
 If you need to search through multiple log files as if they are one, I
suggest looking into Splunk.


- Bil


On Tue, Jun 10, 2014 at 8:30 PM, Steve Upton <[hidden email]> wrote:

>
> On Jun 9, 2014, at 2:11 PM, Jonathan Guthrie <[hidden email]> wrote:
>
> > Yes, 2.5 Gb is bad.
>
> Can you explain “bad” (other than "Try to imagine all life as you know it
> stopping instantaneously and every molecule in your body exploding at the
> speed of light” )
>
> I have a log that’s about 950 MB and am also logging to a flat text file
> that grows rather large (multi-gig) sometimes.
>
> How does that impact a site?
>
> thanks,
>
> Steve
>
>
>
> #############################################################
> This message is sent to you because you are subscribed to
>   the mailing list Lasso [hidden email]
> Official list archives available at http://www.lassotalk.com
> To unsubscribe, E-mail to: <[hidden email]>
> Send administrative queries to  <[hidden email]>
>
#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Lasso crashing on demand.. How to diagnose?

Steve Upton

On Jun 10, 2014, at 1:46 PM, Bil Corry <[hidden email]> wrote:

> Log files should be rotated periodically so that they don't become too
> large for the file system (or too large for the tools used to parse them),
> and it's easier to archive or remove older logs that you no longer need.
> If you need to search through multiple log files as if they are one, I
> suggest looking into Splunk.
>

That’s cool, and understood.

But does it affect the speed of the server, or it’s stability in some way?

Steve

#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Lasso crashing on demand.. How to diagnose?

stevepiercy
On 6/11/14 at 1:10 PM, [hidden email] (Steve Upton) pronounced:

>On Jun 10, 2014, at 1:46 PM, Bil Corry <[hidden email]> wrote:
>
>>Log files should be rotated periodically so that they don't become too
>>large for the file system (or too large for the tools used to parse them),
>>and it's easier to archive or remove older logs that you no longer need.
>>If you need to search through multiple log files as if they are one, I
>>suggest looking into Splunk.
>>
>
>That’s cool, and understood.
>
>But does it affect the speed of the server, or it’s stability in some way?

Yes, both.

As a rough demo, launch Activity Monitor and watch it while you
try opening a 1GB file in BBEdit.  Try search.  Try edit and
save.  It takes more resources to read/write to larger files.  
If the file gets too large, then your app will crash, whether
it's BBEdit, Lasso, or the internal SQLite database.

--steve

-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
Steve Piercy              Website Builder              Soquel, CA
<[hidden email]>               <http://www.StevePiercy.com/>

#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Lasso crashing on demand.. How to diagnose?

Bil Corry-3
The other issue is if your server is attacked, the attacker has only a
single file to delete and all your forensic evidence is gone.  At least if
you rotate and archive log files (or better, centrally consolidate logs),
you'll have at least something to work from.

If having a single log file is what you really want, I suggest just moving
it to another location, and periodically add the latest log entries to it.

- Bil


On Thu, Jun 12, 2014 at 12:07 AM, Steve Piercy - Web Site Builder <
[hidden email]> wrote:

> On 6/11/14 at 1:10 PM, [hidden email] (Steve Upton) pronounced:
>
>
>  On Jun 10, 2014, at 1:46 PM, Bil Corry <[hidden email]> wrote:
>>
>>  Log files should be rotated periodically so that they don't become too
>>> large for the file system (or too large for the tools used to parse
>>> them),
>>> and it's easier to archive or remove older logs that you no longer need.
>>> If you need to search through multiple log files as if they are one, I
>>> suggest looking into Splunk.
>>>
>>>
>> That’s cool, and understood.
>>
>> But does it affect the speed of the server, or it’s stability in some way?
>>
>
> Yes, both.
>
> As a rough demo, launch Activity Monitor and watch it while you try
> opening a 1GB file in BBEdit.  Try search.  Try edit and save.  It takes
> more resources to read/write to larger files.  If the file gets too large,
> then your app will crash, whether it's BBEdit, Lasso, or the internal
> SQLite database.
>
>
> --steve
>
> -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
> Steve Piercy              Website Builder              Soquel, CA
> <[hidden email]>               <http://www.StevePiercy.com/>
>
> #############################################################
> This message is sent to you because you are subscribed to
>  the mailing list Lasso [hidden email]
> Official list archives available at http://www.lassotalk.com
> To unsubscribe, E-mail to: <[hidden email]>
> Send administrative queries to  <[hidden email]>
>
#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>