Scream for Help

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Scream for Help

Pier Kuipers
Hi everyone,

The unthinkable has happened: Lasso on production server won's start up.
If ther is anyone on the planet who is awake and canhelp, I would greatly
appreciate it..!

Yours in Panic,

Pier

--
Pier Kuipers
Mobile: +353 87 294 3063
Blog: http://www.pierkuipers.com
Facebook: http://www.facebook.com/pier.kuipers
Twitter: @pkvisualid
AIM: pkvisualid
#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Scream for Help

decorior
8 or 9?
On Oct 11, 2012, at 8:02 PM, Pier Kuipers <[hidden email]> wrote:

> Hi everyone,
>
> The unthinkable has happened: Lasso on production server won's start up.
> If ther is anyone on the planet who is awake and canhelp, I would greatly
> appreciate it..!
>
> Yours in Panic,
>
> Pier
>
> --
> Pier Kuipers
> Mobile: +353 87 294 3063
> Blog: http://www.pierkuipers.com
> Facebook: http://www.facebook.com/pier.kuipers
> Twitter: @pkvisualid
> AIM: pkvisualid
> #############################################################
> This message is sent to you because you are subscribed to
>  the mailing list Lasso
> [hidden email]
> To unsubscribe, E-mail to: <[hidden email]>
> Send administrative queries to  <[hidden email]>

#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Scream for Help

Eric Browning-4
In reply to this post by Pier Kuipers
What are the log snips as it tries to start up?

--
Eric Browning
Systems Administrator
801-984-7623

Skaggs Catholic Center
Juan Diego Catholic High School
Saint John the Baptist Middle
Saint John the Baptist Elementary
  _____  

From: Pier Kuipers [mailto:[hidden email]]
To: Lasso Knitting Club [mailto:[hidden email]], Lasso Talk [mailto:[hidden email]]
Sent: Thu, 11 Oct 2012 20:02:21 -0700
Subject: Scream for Help

Hi everyone,
 
  The unthinkable has happened: Lasso on production server won's start up.
  If ther is anyone on the planet who is awake and canhelp, I would greatly
  appreciate it..!
 
  Yours in Panic,
 
  Pier
 
  --
  Pier Kuipers
  Mobile: +353 87 294 3063
  Blog: http://www.pierkuipers.com
  Facebook: http://www.facebook.com/pier.kuipers
  Twitter: @pkvisualid
  AIM: pkvisualid
  #############################################################
  This message is sent to you because you are subscribed to
    the mailing list Lasso
  [hidden email]
  To unsubscribe, E-mail to: <[hidden email]>
  Send administrative queries to  <[hidden email]>
   
#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Scream for Help

Paul Higgins-3
In reply to this post by Pier Kuipers

Hi Pier,

What version of Lasso? If you're running 8, have you tried running Lasso
in console mode?

Paul
LassoSoft Support

On 10/11/2012 10:02 PM, Pier Kuipers wrote:

> Hi everyone,
>
> The unthinkable has happened: Lasso on production server won's start up.
> If ther is anyone on the planet who is awake and canhelp, I would greatly
> appreciate it..!
>
> Yours in Panic,
>
> Pier
>

#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Scream for Help

Pier Kuipers
Thanks a million guys, got it sorted with some expert help via chat. Fyi,
it was 8.5 on Centos, plus me panicking.
Really appreciate all your responses, makes me appreciate what an amazing
community the Lasso product gives us :-D

Thanks again, must post a bit more

Bestestest regards

Pier

On 12 October 2012 03:44, Paul Higgins <[hidden email]> wrote:

>
> Hi Pier,
>
> What version of Lasso? If you're running 8, have you tried running Lasso
> in console mode?
>
> Paul
> LassoSoft Support
>
>
> On 10/11/2012 10:02 PM, Pier Kuipers wrote:
>
>> Hi everyone,
>>
>> The unthinkable has happened: Lasso on production server won's start up.
>> If ther is anyone on the planet who is awake and canhelp, I would greatly
>> appreciate it..!
>>
>> Yours in Panic,
>>
>> Pier
>>
>>
> ##############################**##############################**#
> This message is sent to you because you are subscribed to
>  the mailing list Lasso
> [hidden email]
> To unsubscribe, E-mail to: <Lasso-unsubscribe@lists.**lassosoft.com<[hidden email]>
> >
> Send administrative queries to  <Lasso-request@lists.**lassosoft.com<[hidden email]>
> >
>



--
Pier Kuipers
Mobile: +353 87 294 3063
Blog: http://www.pierkuipers.com
Facebook: http://www.facebook.com/pier.kuipers
Twitter: @pkvisualid
AIM: pkvisualid
#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Scream for Help

Jonathan Guthrie-3
So everyone knows:

CentOS 5.6, Lasso 8.5

OK, here's the rundown:

Once I determined the "standard" methods were not working (the server's been up over 1000 days - not saying Lasso has been up that long but that uptime brings it's own issues into play)...
Then because it was so locked up, brute force came into play.

ps aux | grep lasso
# determines the pid's to nuke
# then...
sudo kill -9 [pid]
# for each affected pid.

Then, started in console mode, determined no damage to SQLite tables in master or any instance, all was in good order, no issues with Java or any DB connectors.
Watched some httpd happening, all good.

Shut down console mode and started the Lasso service normally...

And all is well.

Note, brute force should only ever be applied in extreme circumstances.

It's not rocket science, but when the crap's hitting the fan it's a good thing to reach out.

I'll note, as well as hitting the list Pier reached out direct to us. Normally we would not have seen the list traffic until morning EST. Contrary to popular belief we do sleep occasionally :)
I suggest anyone requiring urgent assistance follow Pier's lead and contact [hidden email] directly.
And of course if it's *really* urgent there's the capacity to invoke "Code Red" support and pick up the phone - http://www.lassosoft.com/code-red

Thanks
Jono


On 2012-10-11, at 11:02 PM, Pier Kuipers <[hidden email]> wrote:

> Thanks a million guys, got it sorted with some expert help via chat. Fyi,
> it was 8.5 on Centos, plus me panicking.
> Really appreciate all your responses, makes me appreciate what an amazing
> community the Lasso product gives us :-D
>
> Thanks again, must post a bit more
>
> Bestestest regards
>
> Pier


----------------------------
Jonathan Guthrie
[hidden email]
LassoSoft Inc.
#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Scream for Help

gzasuwa
Thanks - I am happy to know you are there when we need you!!!


Sincerely,

Gerard Zasuwa Gary
Lead Systems Analyst
Henry Ford Health System
Greenfield Health System
(313)-916-2651 work
(313)-585-9621 NEW CELL
(248)-360-0456 home






LEGAL NOTICE: This e-mail is for the exclusive use of the intended recipient(s), and may contain privileged and confidential information. If you are not an intended recipient, please notify the sender, delete the e-mail from your computer and do not copy or disclose it to anyone else. Your receipt of this message is not intended to waive any applicable privilege.

FEDERAL TAX ADVICE DISCLAIMER: Under U. S. Treasury Regulations, we are informing you that, to the extent this message includes any federal tax advice, this message is not intended or written by the sender to be used, and cannot be used, for the purpose of avoiding federal tax penalties.









On Oct 11, 2012, at 11:19 PM, Jonathan Guthrie wrote:

> So everyone knows:
>
> CentOS 5.6, Lasso 8.5
>
> OK, here's the rundown:
>
> Once I determined the "standard" methods were not working (the server's been up over 1000 days - not saying Lasso has been up that long but that uptime brings it's own issues into play)...
> Then because it was so locked up, brute force came into play.
>
> ps aux | grep lasso
> # determines the pid's to nuke
> # then...
> sudo kill -9 [pid]
> # for each affected pid.
>
> Then, started in console mode, determined no damage to SQLite tables in master or any instance, all was in good order, no issues with Java or any DB connectors.
> Watched some httpd happening, all good.
>
> Shut down console mode and started the Lasso service normally...
>
> And all is well.
>
> Note, brute force should only ever be applied in extreme circumstances.
>
> It's not rocket science, but when the crap's hitting the fan it's a good thing to reach out.
>
> I'll note, as well as hitting the list Pier reached out direct to us. Normally we would not have seen the list traffic until morning EST. Contrary to popular belief we do sleep occasionally :)
> I suggest anyone requiring urgent assistance follow Pier's lead and contact [hidden email] directly.
> And of course if it's *really* urgent there's the capacity to invoke "Code Red" support and pick up the phone - http://www.lassosoft.com/code-red
>
> Thanks
> Jono
>
>
> On 2012-10-11, at 11:02 PM, Pier Kuipers <[hidden email]> wrote:
>
>> Thanks a million guys, got it sorted with some expert help via chat. Fyi,
>> it was 8.5 on Centos, plus me panicking.
>> Really appreciate all your responses, makes me appreciate what an amazing
>> community the Lasso product gives us :-D
>>
>> Thanks again, must post a bit more
>>
>> Bestestest regards
>>
>> Pier
>
>
> ----------------------------
> Jonathan Guthrie
> [hidden email]
> LassoSoft Inc.
> #############################################################
> This message is sent to you because you are subscribed to
>  the mailing list Lasso
> [hidden email]
> To unsubscribe, E-mail to: <[hidden email]>
> Send administrative queries to  <[hidden email]>

#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Scream for Help

stevepiercy
In reply to this post by Jonathan Guthrie-3
On 10/11/12 at 11:19 PM, [hidden email] (Jonathan Guthrie) pronounced:

>So everyone knows:
>
>CentOS 5.6, Lasso 8.5
>
>OK, here's the rundown:
>
>Once I determined the "standard" methods were not working (the
>server's been up over 1000 days - not saying Lasso has been up
>that long but that uptime brings it's own issues into play)...
>Then because it was so locked up, brute force came into play.
>
>ps aux | grep lasso
># determines the pid's to nuke

I tend to have senior moments and have a hard time remembering
UNIX commands, so I copy and paste commands from my cheat
sheet.  The following returns only the Lasso pids using a
regular expression:

     ps ax | grep [.]/Lasso8Service

This does not return an extra pid, specifically the grep in use
to get the pids.  It also removes the noise from the u option.

># then...
>sudo kill -9 [pid] # for each affected pid.

If you are user root, then sudo is not necessary (of course, one
should avoid being root unless one knows and accepts the risks
and responsibilities of being a SysAdmin).  I first try omitting
the -9 option for kill.  Also one can delimit pids with spaces.  
Thus if I am root, I do this first:

     kill pid1 pid2 pid3 ...

And if any Lasso processes won't die, then add -9:

     kill -9 pid1 pid2 ... pidN

If that doesn't do it, then I go into HULK SMASH mode, to
shutdown and reboot after shutdown:

     shutdown -r now

And if that doesn't work, I call the ISP to press the "big red button".

Anyway, once Lasso is killed, then I do what Jono describes
below.  But I also do a couple of other things to try to
determine what what went sideways in the first place.

Grab this:

     /tmp/LassoService8.out

*before* restarting Lasso.  When you restart, it gets truncated
and you've lost some forensic evidence.  That file is your best
bet at finding the culprit.  It should never get larger than a
few MB, and if it does then you should change your logging
settings.  If it is huge already, then use tail to get the last
50 or so lines:

     tail -50 /tmp/LassoService8.out

In production it is always best to minimize logging, but when it
is necessary then properly manage or rotate them so they don't
get bloated.  A LassoService8.out or LassoErrors.txt file of
several hundred MB is a really bad thing.  That goes equally for
custom log files and logging to the internal Lasso Admin SQLite databases.

Check for a lock on SQLite internal databases.  Can you delete a
record?  smtp_queue is a known suspect.

JVM and java diagnostics
http://www.ibm.com/developerworks/java/library/j-java6perfmon/#N102EC

JVM process status tool
jps

JVM statistics monitoring tool
jstat

JVM jstat daemon
jstatd

Stack trace
jstack

Some equivalents for lasso9.
http://www.lassosoft.com/Lasso-9-Server-Administration#heading47

--steve


>
>Then, started in console mode, determined no damage to SQLite
>tables in master or any instance, all was in good order, no
>issues with Java or any DB connectors.
>Watched some httpd happening, all good.
>
>Shut down console mode and started the Lasso service normally...
>
>And all is well.
>
>Note, brute force should only ever be applied in extreme circumstances.
>
>It's not rocket science, but when the crap's hitting the fan
>it's a good thing to reach out.
>
>I'll note, as well as hitting the list Pier reached out direct
>to us. Normally we would not have seen the list traffic until
>morning EST. Contrary to popular belief we do sleep
>occasionally :)
>I suggest anyone requiring urgent assistance follow Pier's lead
>and contact [hidden email] directly. And of course if
>it's *really* urgent there's the capacity to invoke "Code Red"
>support and pick up the phone - http://www.lassosoft.com/code-red
>
>Thanks
>Jono
>
>
>On 2012-10-11, at 11:02 PM, Pier Kuipers <[hidden email]> wrote:
>
>>Thanks a million guys, got it sorted with some expert help via chat. Fyi,
>>it was 8.5 on Centos, plus me panicking.
>>Really appreciate all your responses, makes me appreciate what an amazing
>>community the Lasso product gives us :-D
>>
>>Thanks again, must post a bit more
>>
>>Bestestest regards
>>
>>Pier
>
>
>----------------------------
>Jonathan Guthrie
>[hidden email]
>LassoSoft Inc.
>#############################################################
>This message is sent to you because you are subscribed to
>the mailing list Lasso
>[hidden email]
>To unsubscribe, E-mail to: <[hidden email]>
>Send administrative queries to  <[hidden email]>

-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
-- --
Steve Piercy               Web Site Builder              
Soquel, CA
<[hidden email]>                  <http://www.StevePiercy.com/>

#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Scream for Help

Daniel Parker

On 12/10/2012, at 6:45 PM, Steve Piercy - Web Site Builder <[hidden email]> wrote:

> And if that doesn't work, I call the ISP to press the "big red button".

Try this before the ISP.. Assuming Linux.

echo 1 > /proc/sys/kernel/sysrq
echo b > /proc/sysrq-trigger

Its particularly evil.. Essentially the same as pressing the reset switch.

Daniel
#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>