LP 8.5.5 Intermittent slow down

classic Classic list List threaded Threaded
22 messages Options
12
Reply | Threaded
Open this post in threaded view
|

LP 8.5.5 Intermittent slow down

Black Night Lists
Hi

Following on from my other issue with the Lasso process crashing the same
server has the following problem :

Lasso 8.5.5, Mac OS X 10.5.5, 2.8Ghz Quad-Core xServe with 6Gb ram

Lasso served sites on this server will periodically slow down severely and
may last up to 5 minutes. Note that I have never seen this happen on this
server at the same time that the customer, and his customers report but this
is hard to really be available at the exact same time. The couple of times
when I have been checking the sites appeared to be ok.

In a nutshell, static sites and php sites will display instantly and fast
with no problems at the exact time that Lasso hosted sites will slow right
down, majority of them have MySQL databases.

The text loads but the images take a long time to display. CPU usage at this
time is low, in the last seven days the highest the CPU usage has gone up to
is 30%.

I increased the memory from 2Gb to 6Gb in case it was a memory related issue
due to the fact we have 8 Lasso site processes running because of the way I
have split some hosted sites into different processes. This is to avoid all
going down if one process dies.

The network, switch, port, cable have all been verified working and nothing
has been found, though with non-lasso sites serving instantly I was not
expecting anything to be found.

During the testing of this issue, which has been ongoing for some time now
we found that one site was re-scaling images on the fly before serving them.
We think that there maybe a link between when the Lasso hosted sites start
to slow down this occasionally seems to tie in with when this specific site
is serving scaled images. Of course this may not be linked but it's worth
mentioning.

We have other servers plus co-located and dedicated servers running Lasso 6,
7, 8 and 8.5 but only this server seems to have this issue. Again, anyone
else come across this?


Regards

Stephen
www.blacknight.co.uk



--
This list is a free service of LassoSoft: http://www.LassoSoft.com/
Search the list archives: http://www.ListSearch.com/Lasso/Browse/
Manage your subscription: http://www.ListSearch.com/Lasso/

Reply | Threaded
Open this post in threaded view
|

Re: LP 8.5.5 Intermittent slow down

Ke Carlton-2
A few questions,

Are the scale images being generated on the fly?

How long on average do your pages take to load?

What's the database activity like?

Ke

>
> During the testing of this issue, which has been ongoing for some time now
> we found that one site was re-scaling images on the fly before serving them.
> We think that there maybe a link between when the Lasso hosted sites start
> to slow down this occasionally seems to tie in with when this specific site
> is serving scaled images. Of course this may not be linked but it's worth
> mentioning.

--
This list is a free service of LassoSoft: http://www.LassoSoft.com/
Search the list archives: http://www.ListSearch.com/Lasso/Browse/
Manage your subscription: http://www.ListSearch.com/Lasso/

Reply | Threaded
Open this post in threaded view
|

Re: LP 8.5.5 Intermittent slow down

Black Night Lists
On 13/11/08 11:03, "Ke Carlton" <[hidden email]> wrote:

> Are the scale images being generated on the fly?

The images are scaled on the fly when the page loads. I am not sure if they
have changed this as I asked them to try scaling the images upon upload so
that they have the specific sizes already
 
> How long on average do your pages take to load?
For me it's pretty much instant, during the time of the issue. I am not
sure.  Just looked through the some error logs and found the following of
which there are a lot of :

Message: Script execution time limit exceeded while executing
'/index.lasso'. Aborting thread B154B000

Not sure which actual web site this is as this site process serves about 20
web sites, wonder if one of the sites scale the images on the fly on the
index page, will ask the client.

> What's the database activity like?

Interesting you ask that as I can see the following :

Message: MySQLDS failed connecting to MySQL. errno=2013 "Lost connection to
MySQL server during query" MySQL is configured to have 1,000 max
connections. However, these are only shown on this specific Lasso site, it's
not reported on any other Lasso site process which also use the same MySQL
server.

Regards

Stephen



--
This list is a free service of LassoSoft: http://www.LassoSoft.com/
Search the list archives: http://www.ListSearch.com/Lasso/Browse/
Manage your subscription: http://www.ListSearch.com/Lasso/

Reply | Threaded
Open this post in threaded view
|

Re: LP 8.5.5 Intermittent slow down

Black Night Lists
On 13/11/08 11:39, "Stephen Thirlwell" <[hidden email]> wrote:

> Message: Script execution time limit exceeded while executing
> '/index.lasso'. Aborting thread B154B000

I have reported this to the client, they are going to check all sites to see
what each site index.lasso page is doing. Is there any easy way of locating
which site this error could be related to?

Regards

Stephen



--
This list is a free service of LassoSoft: http://www.LassoSoft.com/
Search the list archives: http://www.ListSearch.com/Lasso/Browse/
Manage your subscription: http://www.ListSearch.com/Lasso/

Reply | Threaded
Open this post in threaded view
|

Re: LP 8.5.5 Intermittent slow down

Ke Carlton-2
In reply to this post by Black Night Lists
2008/11/13 Stephen Thirlwell <[hidden email]>:
> On 13/11/08 11:03, "Ke Carlton" <[hidden email]> wrote:
>
> The images are scaled on the fly when the page loads. I am not sure if they
> have changed this as I asked them to try scaling the images upon upload so
> that they have the specific sizes already

Ok, if the images are scale by Lasso (not the browser) then this is
very CPU intensive.

>
>> How long on average do your pages take to load?
> For me it's pretty much instant, during the time of the issue. I am not
> sure.  Just looked through the some error logs and found the following of
> which there are a lot of :

When you say instant how many seconds / milliseconds?

>
> Message: Script execution time limit exceeded while executing
> '/index.lasso'. Aborting thread B154B000

OK, looks like one of the sites / pages is running very slow. This
would cause a snowball effect (and likely crash Lasso) if hit with
heavy traffic. Your machine has plenty of RAM, I would recommend
breaking the single Lasso site into multiple Lasso sites in
serverAdmin. This will allow you to determine which website is
snowballing and timing out and isolate working sites.

> Interesting you ask that as I can see the following :
>
> Message: MySQLDS failed connecting to MySQL. errno=2013 "Lost connection to
> MySQL server during query" MySQL is configured to have 1,000 max
> connections. However, these are only shown on this specific Lasso site, it's
> not reported on any other Lasso site process which also use the same MySQL
> server.

OK, if Lasso cannot connect to MySQL (or another database it needs to
use) then page trying to use that will timeout after 60 seconds or so.
Lasso will accumulate threads waiting for the database until it
eventually crashes. So you need to ensure that the datasource is
solid.

Ke.

--
This list is a free service of LassoSoft: http://www.LassoSoft.com/
Search the list archives: http://www.ListSearch.com/Lasso/Browse/
Manage your subscription: http://www.ListSearch.com/Lasso/

Reply | Threaded
Open this post in threaded view
|

Re: LP 8.5.5 Intermittent slow down

Ke Carlton-2
In reply to this post by Black Night Lists
2008/11/13 Stephen Thirlwell <[hidden email]>:
>
> I have reported this to the client, they are going to check all sites to see
> what each site index.lasso page is doing. Is there any easy way of locating
> which site this error could be related to?
>

Yes, you could break down the single Lasso site into multiple Lasso
sites. Or add some logging to each index.lasso file. Something like
the below:

var('_pageStart') = date_mSec;
define_atEnd:{
local('diff') = (date_mSec - $_pageStart)*0.001;
#diff > 1.0 ? log_critical('SLOW: '#diff' seconds - 'client_url);
};

This will log slow requests (based on 1 second) to the sites error log
and console. Console mode is a great way to observe what's happening
in a snowballing situation.

Ke.

--
This list is a free service of LassoSoft: http://www.LassoSoft.com/
Search the list archives: http://www.ListSearch.com/Lasso/Browse/
Manage your subscription: http://www.ListSearch.com/Lasso/

Reply | Threaded
Open this post in threaded view
|

Re: LP 8.5.5 Intermittent slow down

Black Night Lists
In reply to this post by Ke Carlton-2
On 13/11/08 12:11, "Ke Carlton" <[hidden email]> wrote:

>> The images are scaled on the fly when the page loads. I am not sure if they
>> have changed this as I asked them to try scaling the images upon upload so
>> that they have the specific sizes already
>
> Ok, if the images are scale by Lasso (not the browser) then this is
> very CPU intensive.
Indeed, though the processor usage generally hovers around the 5% mark and
rarely rises above 10%, the last time it went above 20% it was about 30% but
not sure exactly when this was and what was happening that day, could of
been a software update.
 
>>
>>> How long on average do your pages take to load?
>> For me it's pretty much instant, during the time of the issue. I am not
>> sure.  Just looked through the some error logs and found the following of
>> which there are a lot of :
>
> When you say instant how many seconds / milliseconds?
None of the sites have a timer on them so I am only visually stating how
fast it loads. For me one site loads within a second.

Part of the problem is that I have not actually witnessed the issue but I do
believe the client and of course the client's customers report the same
issue.

Here is what the client reports.

Accessing via IE the page does not load, just sits for ages
Accessing via Firefox, the text is shown (not sure if this is dynamically
loaded from the database or not) but the images are simply not displayed.
Just sits trying to load them. Stopping the page and reloading may cure it
or it may sit like that for a few minutes.

The owner of a couple of the sites use the sites pretty much all day so they
tend to notice the issues quite quickly but it's not contained to just them
but it is just lasso served sites.

>>
>> Message: Script execution time limit exceeded while executing
>> '/index.lasso'. Aborting thread B154B000
>
> OK, looks like one of the sites / pages is running very slow. This
> would cause a snowball effect (and likely crash Lasso) if hit with
> heavy traffic. Your machine has plenty of RAM, I would recommend
> breaking the single Lasso site into multiple Lasso sites in
> serverAdmin. This will allow you to determine which website is
> snowballing and timing out and isolate working sites.
Yes, we are planning on splitting the sites further. Part of the problem
though is it means more downtime whilst the site processes are split,
permissions set etc which is what we are trying to avoid. Though I think
this is inevitable. I have discussed this with the client and we have a list
of three possible sites which we think it maybe related to, the plan is to
move them to a different process and see if the errors reported move too.


>> Interesting you ask that as I can see the following :
>>
>> Message: MySQLDS failed connecting to MySQL. errno=2013 "Lost connection to
>> MySQL server during query" MySQL is configured to have 1,000 max
>> connections. However, these are only shown on this specific Lasso site, it's
>> not reported on any other Lasso site process which also use the same MySQL
>> server.
>
> OK, if Lasso cannot connect to MySQL (or another database it needs to
> use) then page trying to use that will timeout after 60 seconds or so.
> Lasso will accumulate threads waiting for the database until it
> eventually crashes. So you need to ensure that the datasource is
> solid.
Agreed, though during this time none of the other sites on other processes
have a problem accessing the same MySQL server. I suspect that it has to be
one of the sites with high traffic. I wonder if it's worth checking to
ensure all sql statements are optimised so that only what is absolutely
required to be displayed on the page is requested instead of select * - if
not already of course ;)

I definitely think it is one specific site causing the problem - the issue
is which one. The customer is checking the sites to see what each index page
is doing so we might end up finding the culprit.

Thanks for your responses, much appreciated

Regards

Stephen






--
This list is a free service of LassoSoft: http://www.LassoSoft.com/
Search the list archives: http://www.ListSearch.com/Lasso/Browse/
Manage your subscription: http://www.ListSearch.com/Lasso/

Reply | Threaded
Open this post in threaded view
|

Re: LP 8.5.5 Intermittent slow down

Ke Carlton-2
2008/11/13 Stephen Thirlwell <[hidden email]>:
> Accessing via IE the page does not load, just sits for ages
> Accessing via Firefox, the text is shown (not sure if this is dynamically
> loaded from the database or not) but the images are simply not displayed.
> Just sits trying to load them. Stopping the page and reloading may cure it
> or it may sit like that for a few minutes.

Are the images created on the fly by Lasso?

> Yes, we are planning on splitting the sites further. Part of the problem
> though is it means more downtime whilst the site processes are split,
> permissions set etc which is what we are trying to avoid. Though I think
> this is inevitable. I have discussed this with the client and we have a list
> of three possible sites which we think it maybe related to, the plan is to
> move them to a different process and see if the errors reported move too.

This can be achieved without any down time. Just to be clear - we're
talking about Lasso sites within Lasso Server Admin.

> Agreed, though during this time none of the other sites on other processes
> have a problem accessing the same MySQL server. I suspect that it has to be
> one of the sites with high traffic. I wonder if it's worth checking to
> ensure all sql statements are optimised so that only what is absolutely
> required to be displayed on the page is requested instead of select * - if
> not already of course ;)

Yes, select * from a very large table could choke Lasso without a limit clause.

> I definitely think it is one specific site causing the problem - the issue
> is which one. The customer is checking the sites to see what each index page
> is doing so we might end up finding the culprit.

Ke.

--
This list is a free service of LassoSoft: http://www.LassoSoft.com/
Search the list archives: http://www.ListSearch.com/Lasso/Browse/
Manage your subscription: http://www.ListSearch.com/Lasso/

Reply | Threaded
Open this post in threaded view
|

Re: LP 8.5.5 Intermittent slow down

Black Night Lists
On 13/11/08 12:47, "Ke Carlton" <[hidden email]> wrote:

>> Accessing via IE the page does not load, just sits for ages
>> Accessing via Firefox, the text is shown (not sure if this is dynamically
>> loaded from the database or not) but the images are simply not displayed.
>> Just sits trying to load them. Stopping the page and reloading may cure it
>> or it may sit like that for a few minutes.
>
> Are the images created on the fly by Lasso?

No, only one site does this but not on the index page. Some images are
loaded as referenced from a field in a database for most sites, I have not
asked the client if it's all images are just database referenced images.
Will ask. Though not all sites are like this but the client states it's all
sites I gave him to test that are served by Lasso across a few different
Lasso sites. Static sites (.html pages) don't have the issue.
 
>> Yes, we are planning on splitting the sites further. Part of the problem
>> though is it means more downtime whilst the site processes are split,
>> permissions set etc which is what we are trying to avoid. Though I think
>> this is inevitable. I have discussed this with the client and we have a list
>> of three possible sites which we think it maybe related to, the plan is to
>> move them to a different process and see if the errors reported move too.
>
> This can be achieved without any down time. Just to be clear - we're
> talking about Lasso sites within Lasso Server Admin.

Yeah, I guess it would be transparent if the site is set up in advance then
the url deleted from the current site. As long as all the permissions were
set up correctly ;)


Regards

Stephen



--
This list is a free service of LassoSoft: http://www.LassoSoft.com/
Search the list archives: http://www.ListSearch.com/Lasso/Browse/
Manage your subscription: http://www.ListSearch.com/Lasso/

Reply | Threaded
Open this post in threaded view
|

Re: LP 8.5.5 Intermittent slow down

Jim VH-2
In reply to this post by Black Night Lists
Have you setup a slow query log in MySQL? It may show that a specific  
query may be the issue that is called occasionally. I found this to be  
the case in one solution where the query is created dynamically from  
the search criteria the client inputs on a form. Unknown to me, they  
had the ability to do a full join query w/o any WHERE criteria. Oops.

---
Jim Van Heule
Heunox Corporation



On Nov 13, 2008, at 6:39 AM, Stephen Thirlwell wrote:

> On 13/11/08 11:03, "Ke Carlton" <[hidden email]> wrote:
>
>> Are the scale images being generated on the fly?
>
> The images are scaled on the fly when the page loads. I am not sure  
> if they
> have changed this as I asked them to try scaling the images upon  
> upload so
> that they have the specific sizes already
>
>> How long on average do your pages take to load?
> For me it's pretty much instant, during the time of the issue. I am  
> not
> sure.  Just looked through the some error logs and found the  
> following of
> which there are a lot of :
>
> Message: Script execution time limit exceeded while executing
> '/index.lasso'. Aborting thread B154B000
>
> Not sure which actual web site this is as this site process serves  
> about 20
> web sites, wonder if one of the sites scale the images on the fly on  
> the
> index page, will ask the client.
>
>> What's the database activity like?
>
> Interesting you ask that as I can see the following :
>
> Message: MySQLDS failed connecting to MySQL. errno=2013 "Lost  
> connection to
> MySQL server during query" MySQL is configured to have 1,000 max
> connections. However, these are only shown on this specific Lasso  
> site, it's
> not reported on any other Lasso site process which also use the same  
> MySQL
> server.
>
> Regards
>
> Stephen
>
>
>
> --
> This list is a free service of LassoSoft: http://www.LassoSoft.com/
> Search the list archives: http://www.ListSearch.com/Lasso/Browse/
> Manage your subscription: http://www.ListSearch.com/Lasso/
>


--
This list is a free service of LassoSoft: http://www.LassoSoft.com/
Search the list archives: http://www.ListSearch.com/Lasso/Browse/
Manage your subscription: http://www.ListSearch.com/Lasso/

Reply | Threaded
Open this post in threaded view
|

Re: LP 8.5.5 Intermittent slow down

Bil Corry-3
In reply to this post by Black Night Lists
Stephen Thirlwell wrote on 11/13/2008 4:49 AM:
> The text loads but the images take a long time to display. CPU usage at this
> time is low, in the last seven days the highest the CPU usage has gone up to
> is 30%.

If the text loads, then it means Lasso processed the page and returned it to the browser.  So your issue is strictly with Lasso having to crunch images to serve.  From reading the thread, it sounds like Lasso is used to resize the images on-the-fly, the obviously solution is to not do this, instead have Apache serve the images by resizing when they're first added to the system and get Lasso out of image-serving.

Now, it also sounds like you have all your various websites going through Site 1, which means when one website starts slowing down, it slows all of them down.  Break the websites out into multiple Lasso sites (especially since you have so much ram), then when one site slows down, the others can keep chugging along.

- Bil


--
This list is a free service of LassoSoft: http://www.LassoSoft.com/
Search the list archives: http://www.ListSearch.com/Lasso/Browse/
Manage your subscription: http://www.ListSearch.com/Lasso/

Reply | Threaded
Open this post in threaded view
|

Re: LP 8.5.5 Intermittent slow down

Black Night Lists
On 13/11/08 17:58, "Bil Corry" <[hidden email]> wrote:

> If the text loads, then it means Lasso processed the page and returned it to
> the browser.  So your issue is strictly with Lasso having to crunch images to
> serve.  From reading the thread, it sounds like Lasso is used to resize the
> images on-the-fly, the obviously solution is to not do this, instead have
> Apache serve the images by resizing when they're first added to the system and
> get Lasso out of image-serving.
One site resizes images on the fly but not on the index page and I have
asked the customer to change this process (it was doing this as the
dimensions were dynamic, they have now settled on two sizes so I believe
they are going to change it so that this resize is done on upload so that
the images are served as normal).

 
> Now, it also sounds like you have all your various websites going through Site
> 1, which means when one website starts slowing down, it slows all of them
> down.  Break the websites out into multiple Lasso sites (especially since you
> have so much ram), then when one site slows down, the others can keep chugging
> along.
Not the case, most of the sites this customer are on lasso site 7 with a
further 2/3 running on site 8 (there are 8 site processes currently). Also
the customer states that other sites running on different lasso processes
also have the problem - but I cannot back this up at the current time as I
have yet to see the issue. For example when his sites have the problem a
different customers site exhibits the same problem and that is site 3.

Site 1 serves the server IP and one other site. This site is used for
uploading an image of about 2-3Mb, it's then scaled and resized then pushed
across via ftp to a Lasso 6 server - not something I would expect to cause
the whole server to slow down. Though this is only used every now and then
which could explain the intermittent nature of this problem I guess,
clutching at straws now ;)

Regards

Stephen



--
This list is a free service of LassoSoft: http://www.LassoSoft.com/
Search the list archives: http://www.ListSearch.com/Lasso/Browse/
Manage your subscription: http://www.ListSearch.com/Lasso/

Reply | Threaded
Open this post in threaded view
|

Re: LP 8.5.5 Intermittent slow down

Bil Corry-3
Stephen Thirlwell wrote on 11/13/2008 12:25 PM:

>> Now, it also sounds like you have all your various websites going through Site
>> 1, which means when one website starts slowing down, it slows all of them
>> down.  Break the websites out into multiple Lasso sites (especially since you
>> have so much ram), then when one site slows down, the others can keep chugging
>> along.
> Not the case, most of the sites this customer are on lasso site 7 with a
> further 2/3 running on site 8 (there are 8 site processes currently). Also
> the customer states that other sites running on different lasso processes
> also have the problem - but I cannot back this up at the current time as I
> have yet to see the issue. For example when his sites have the problem a
> different customers site exhibits the same problem and that is site 3.

If all sites are affected by the slow-down, then something is affecting performance that is common to all sites, such as using excessive CPU, over-utilizing the hard drive, consuming larges swathes of RAM, a query causing MySQL to spin, etc.

So you have to do a bit of detective work, you originally said that PHP and static sites are not affected.  You also said the Lasso page is returned which means Lasso isn't affected, the only slow-down was the images, which are served by Lasso.  So that tells me that having Lasso process lots of images on your system isn't going fast enough for your users.  So the obvious fix is to not have Lasso serve those images, but instead serve them statically by Apache.

So assuming the images are served statically and are returned quickly, will that solve your issue?


- Bil


--
This list is a free service of LassoSoft: http://www.LassoSoft.com/
Search the list archives: http://www.ListSearch.com/Lasso/Browse/
Manage your subscription: http://www.ListSearch.com/Lasso/

Reply | Threaded
Open this post in threaded view
|

Re: LP 8.5.5 Intermittent slow down

Black Night Lists
On 13/11/08 19:04, "Bil Corry" <[hidden email]> wrote:

> If all sites are affected by the slow-down, then something is affecting
> performance that is common to all sites, such as using excessive CPU,
> over-utilizing the hard drive, consuming larges swathes of RAM, a query
> causing MySQL to spin, etc.
It can't be ram - I installed 4Gb of ram on the off chance it was, nor CPU
processing as this would show on the server history. MySQL is a possibility
as the site process is showing lost queries.
 

> So you have to do a bit of detective work, you originally said that PHP and
> static sites are not affected.  You also said the Lasso page is returned which
> means Lasso isn't affected, the only slow-down was the images, which are
> served by Lasso.  So that tells me that having Lasso process lots of images on
> your system isn't going fast enough for your users.  So the obvious fix is to
> not have Lasso serve those images, but instead serve them statically by
> Apache.
>
> So assuming the images are served statically and are returned quickly, will
> that solve your issue?

Reported to me is the following :

IE - Nothing is displayed, no text, no images - nothing, just sits thinking
about it for a while until you get fed up and refresh the request.
Firefox - Text is displayed but images are not.

Customer calls me saying it's slow, I check and it's fine for me.

Just to clarify, is Lasso serving these images as I am under the impression
that it's not, as all it's doing is simply providing the path.

<img src="images/[field:'thumbnail_image'].jpg" alt="" height="60"
width="80" border="0" />

I think part of the battle here is isolating which site is causing the
problem, find that and the issue is likely to go away. This is not a single
site server, it's a shared environment. Only one customer reports problems
directly to me but he has several customers reporting issues to him, none of
my other clients are reporting this issue which makes it so confusing.

I use my site frequently during the day for admin purposes and I never see
the problem - this is what is so frustrating about all this.

I think the first step is trying to find out which web sites index.lasso is
experiencing the script execution error in the logs.  Then try to get that
site optimised and moved to it's own Lasso site process. I am going to set
up slow query logging on MySQL, install some timing code as suggested by Ke
and see where we go from here.

Thanks for the reply :)

Regards

Stephen




--
This list is a free service of LassoSoft: http://www.LassoSoft.com/
Search the list archives: http://www.ListSearch.com/Lasso/Browse/
Manage your subscription: http://www.ListSearch.com/Lasso/

Reply | Threaded
Open this post in threaded view
|

Re: LP 8.5.5 Intermittent slow down

Ke Carlton-2
You find it easier to install the console logging script as a site
wide atBegin.lasso file and drop it in each of the start up folders.

define_atBegin:{
   var('_pageStart') = date_mSec;
   define_atEnd:{
      local('diff') = (date_mSec - $_pageStart)*0.001;
      #diff > 1.0 ? log_critical('SLOW: '#diff' seconds - 'client_url);
   };
};

tbh, it sounds like your getting some odd user feedback - are they on
dial up or something equally stupid?

2008/11/13 Stephen Thirlwell <[hidden email]>:
> I am going to set
> up slow query logging on MySQL, install some timing code as suggested by Ke
> and see where we go from here.
>

--
This list is a free service of LassoSoft: http://www.LassoSoft.com/
Search the list archives: http://www.ListSearch.com/Lasso/Browse/
Manage your subscription: http://www.ListSearch.com/Lasso/

Reply | Threaded
Open this post in threaded view
|

Re: LP 8.5.5 Intermittent slow down

Black Night Lists
On 13/11/08 22:24, "Ke Carlton" <[hidden email]> wrote:

> You find it easier to install the console logging script as a site
> wide atBegin.lasso file and drop it in each of the start up folders.
>
> define_atBegin:{
>    var('_pageStart') = date_mSec;
>    define_atEnd:{
>       local('diff') = (date_mSec - $_pageStart)*0.001;
>       #diff > 1.0 ? log_critical('SLOW: '#diff' seconds - 'client_url);
>    };
> };
>
> tbh, it sounds like your getting some odd user feedback - are they on
> dial up or something equally stupid?

Thanks Ke.

At first I did put it down to a connection issue on their broadband
connection or just one of those things that happens from time to time, but
when their customers started reporting the same issues too it became obvious
that their connection cannot be the cause (well unless they are all on the
same ISP but I doubt that is likely).

Also during this problem they can access other sites and servers on our
network with no issues, it simply does just appear to be this one server and
only where Lasso is involved.

I have lots to go on after the responses from you and Bil so I am sure we
will isolate the site causing the problem soon and can take steps to iron
out the issues. Maybe there is just too many sites on the same Lasso
process.

Which ever site is timing out on the index.lasso page is more than likely
the cause, I thought it was the site that was serving images on the fly but
that's not happening on index.lasso but I am going to move this specific
site off onto it's own Lasso process so that should show in the logs if it's
the culprit.

Regards

Stephen



--
This list is a free service of LassoSoft: http://www.LassoSoft.com/
Search the list archives: http://www.ListSearch.com/Lasso/Browse/
Manage your subscription: http://www.ListSearch.com/Lasso/

Reply | Threaded
Open this post in threaded view
|

Re: LP 8.5.5 Intermittent slow down

Bil Corry-3
In reply to this post by Black Night Lists
Stephen Thirlwell wrote on 11/13/2008 3:36 PM:
> IE - Nothing is displayed, no text, no images - nothing, just sits thinking
> about it for a while until you get fed up and refresh the request.
> Firefox - Text is displayed but images are not.

This may be simply a case of the browsers rendering differently, especially if you don't provide the width and height of the image in the <IMG>.  In some cases, if the browser doesn't know how much space to allocate for the image, it waits to display the page until it gets the image so it can figure out the dimensions.


> Customer calls me saying it's slow, I check and it's fine for me.

It could be a lot of things but if you're logged in as the customer and are doing the very same things they are, yet its slow for them and fast for you, it's almost always a network issue.


> Just to clarify, is Lasso serving these images as I am under the impression
> that it's not, as all it's doing is simply providing the path.
>
> <img src="images/[field:'thumbnail_image'].jpg" alt="" height="60"
> width="80" border="0" />

You tell us.  Does Lasso serve the images stored in /images?  One very real possibility is the image isn't resized to thumbnail size, so instead of serving a 80x60 image, you're serving a 3mb image that the browser is resizing using the given height and width.  If you're testing it on a local 100mb network, you won't notice the large size images.  But everyone outside the local network sure will.

Something that can immediately help is to set the Expires header to 1 year for all the images, that way the browser will download the image once, and won't do it again for a year.


- Bil


--
This list is a free service of LassoSoft: http://www.LassoSoft.com/
Search the list archives: http://www.ListSearch.com/Lasso/Browse/
Manage your subscription: http://www.ListSearch.com/Lasso/

Reply | Threaded
Open this post in threaded view
|

Re: LP 8.5.5 Intermittent slow down

Steve Upton
In reply to this post by Black Night Lists
>
>
>Customer calls me saying it's slow, I check and it's fine for me.

quick question:

Are you *certain* that it's the server slowing things down?

I have gone on such goose chases before only to find that I couldn't find a problem OR that a different problem came to light later on.

Don't forget there are a million links between the customer and your server. Problematic DNS servers (bitten me a couple of times), unreliable internet routes (show up in strange ways at ISPs that have multiple links to the rest of the 'Net), and other things can bog you down.

The fact that you can't reproduce it from your location and you can't diagnose it within Lasso implies to me that you may want to take a step back and look at the bigger picture.

Double check THEIR DNS server setting on their machine (if your TTL on DNS is small and their server is whacko it'll slow things down unpredictably). Get them to do a traceroute from their machine to your server when things are good and then when things are bad. I have seen routes grow by 6-7 hops when things slow down and routes adjust - nasty! Check with your ISP if they have been experiencing any "network events" like DoS attacks while the slowdowns occur, etc.

Our first inclination is to look at the portion of a system we are working on (and perhaps we trust the least). Don't forget about Occam's razor.

good luck

Steve Upton

--


--
This list is a free service of LassoSoft: http://www.LassoSoft.com/
Search the list archives: http://www.ListSearch.com/Lasso/Browse/
Manage your subscription: http://www.ListSearch.com/Lasso/

Reply | Threaded
Open this post in threaded view
|

Re: LP 8.5.5 Intermittent slow down

Black Night Lists
In reply to this post by Ke Carlton-2
On 13/11/08 22:24, "Ke Carlton" <[hidden email]> wrote:

> You find it easier to install the console logging script as a site
> wide atBegin.lasso file and drop it in each of the start up folders.
>
> define_atBegin:{
>    var('_pageStart') = date_mSec;
>    define_atEnd:{
>       local('diff') = (date_mSec - $_pageStart)*0.001;
>       #diff > 1.0 ? log_critical('SLOW: '#diff' seconds - 'client_url);
>    };
> };
>
> tbh, it sounds like your getting some odd user feedback - are they on
> dial up or something equally stupid?

Hi Ke/List

I implemented this on all sites and today Lasso Crashed again at 13.56 with
:

Message: StorageManager::Execute failed. errno=14 "unable to open database
file"
Message: Shutting down children...
Message: Shutting down threads...

At around the same time one customer site started showing slow index pages
using the above code so now had a site to look at.

It started from 13.55. Looking through the access_log file for this site at
13.55 showed that a request for a script that used to exist in a cgi-bin
when it was hosted elsewhere before the redesign was being accessed, at
least was trying to be accessed. A .htaccess file is used to redirect people
to the index page if folders, files etc don't exist. We suspect that it got
itself in a loop over and over as when the original redirect happened it's
trying to load images which don't exist in the path as it was leaving the
original cgi-bin directory in the redirect which eventually caused server
slow down and ultimately a server crash, there were around 30 httpd
processes running at the time of the crash and a huge traffic spike showing.

So, the customer has tweaked the .htaccess file and we are going to take it
from there and see if that is the real cause. Checking through the logs from
the same time the server crashed last time the same files were requested so
I am pretty confident that this is the cause.

Thanks Ke for the logging script which showed exactly which site we needed
to investigate. Here's hoping this is the end of it :)

Regards

Stephen




--
This list is a free service of LassoSoft: http://www.LassoSoft.com/
Search the list archives: http://www.ListSearch.com/Lasso/Browse/
Manage your subscription: http://www.ListSearch.com/Lasso/

Reply | Threaded
Open this post in threaded view
|

Re: LP 8.5.5 Intermittent slow down

Bil Corry-3
Stephen Thirlwell wrote on 11/18/2008 9:38 AM:
> A .htaccess file is used to redirect people to the index page if
> folders, files etc don't exist. We suspect that it got itself in a
> loop

Which version of Apache are you running?  As of Apache 2.0.45, it should (by default) limit the redirects to 10; more than 10 returns an ISE:

        http://httpd.apache.org/docs/2.0/mod/mod_rewrite.html#rewriteoptions


- Bil


--
This list is a free service of LassoSoft: http://www.LassoSoft.com/
Search the list archives: http://www.ListSearch.com/Lasso/Browse/
Manage your subscription: http://www.ListSearch.com/Lasso/

12