8.5.6 memory issues

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

8.5.6 memory issues

Clive Bruton
I have LP8.5.6 running on a Mac Mini, OSX 10.4.11. When running a  
large report I have intermittent problems in that I receive a blank  
page back - no error, just nothing returned at all.

It seems that if I restart Lasso I can run the report a few times,  
perhaps up to five, then I get the error. After the error occurs its  
ongoing errors after that (ie I never get a report).

If I run Lasso in console mode I see it gradually eat up about 1.8GB  
of RAM in one process when it is working on the report. After it  
delivers the report it does not let go of the RAM, so no other  
process can use it. I get the same errors in console mode, I do get  
this printed:

Lasso8Service(1602,0x1d2ee00) malloc: *** vm_allocate(size=6778880)  
failed (error code=3)
Lasso8Service(1602,0x1d2ee00) malloc: *** error: can't allocate region
Lasso8Service(1602,0x1d2ee00) malloc: *** set a breakpoint in  
szone_error to debug
Lasso8Service(1602,0x1d2ee00) malloc: *** vm_allocate(size=5423104)  
failed (error code=3)
Lasso8Service(1602,0x1d2ee00) malloc: *** error: can't allocate region
Lasso8Service(1602,0x1d2ee00) malloc: *** set a breakpoint in  
szone_error to debug

I never get any report in the errors table.

Is there any way to get Lasso to release the RAM after processing the  
report - I'm suspecting that this is the reason it fails after a few  
successful attempts (runs out of RAM)?


-- Clive

#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: 8.5.6 memory issues

stevepiercy
What does the report actually do?  What's the datasource?  PDF?

In my experience, the cause is usually something smelly in the
code—making too many connections to a datasource, abusing
FileMaker, generating huge PDFs—and not in Lasso itself.  A
good indicator of smelly code is if generating the report one
time takes more than a few seconds.

You could try searching the archive, too.
http://lasso.2283332.n4.nabble.com/template/NamlServlet.jtp?macro=search_page&node=3096189&query=malloc+lasso8service+szone_error&days=0

--steve


On 7/19/15 at 5:04 PM, [hidden email] (Clive Bruton) pronounced:

>I have LP8.5.6 running on a Mac Mini, OSX 10.4.11. When running
>a large report I have intermittent problems in that I receive a
>blank page back - no error, just nothing returned at all.
>
>It seems that if I restart Lasso I can run the report a few
>times, perhaps up to five, then I get the error. After the
>error occurs its ongoing errors after that (ie I never get a report).
>
>If I run Lasso in console mode I see it gradually eat up about
>1.8GB of RAM in one process when it is working on the report.
>After it delivers the report it does not let go of the RAM, so
>no other process can use it. I get the same errors in console
>mode, I do get this printed:
>
>Lasso8Service(1602,0x1d2ee00) malloc: *** vm_allocate(size=6778880) failed (error code=3)
>Lasso8Service(1602,0x1d2ee00) malloc: *** error: can't allocate region
>Lasso8Service(1602,0x1d2ee00) malloc: *** set a breakpoint in szone_error to debug
>Lasso8Service(1602,0x1d2ee00) malloc: *** vm_allocate(size=5423104) failed (error code=3)
>Lasso8Service(1602,0x1d2ee00) malloc: *** error: can't allocate region
>Lasso8Service(1602,0x1d2ee00) malloc: *** set a breakpoint in szone_error to debug
>
>I never get any report in the errors table.
>
>Is there any way to get Lasso to release the RAM after
>processing the report - I'm suspecting that this is the reason
>it fails after a few successful attempts (runs out of RAM)?
>
>
>-- Clive
>
>#############################################################
>
>This message is sent to you because you are subscribed to
>the mailing list Lasso [hidden email]
>Official list archives available at http://www.lassotalk.com
>To unsubscribe, E-mail to: <[hidden email]>
>Send administrative queries to  <[hidden email]>

-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
Steve Piercy              Website Builder              Soquel, CA
<[hidden email]>               <http://www.StevePiercy.com/>


#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: 8.5.6 memory issues

Clive Bruton

On 19 Jul 2015, at 18:32, Steve Piercy - Website Builder wrote:

> What does the report actually do?

In brief, it takes a lot of related data, forms it into various  
(quite large) arrays of several different types of (complex) custom  
objects and then manipulates those into various reports (I haven't  
really looked at it in that way, but it wouldn't surprise me to find  
that the reports run to 70-100 pages - Safari indicates that the  
report is 4.5MB).

> What's the datasource?

MySQL.

> PDF?

No, just very basic HTML.

> In my experience, the cause is usually something smelly in the code—
> making too many connections to a datasource, abusing FileMaker,  
> generating huge PDFs—and not in Lasso itself.  A good indicator of  
> smelly code is if generating the report one time takes more than a  
> few seconds.

I'm fairly certain that the problem lies within Lasso, because the  
MySQL query that originates the data takes a few seconds, but forming  
all the objects and manipulating them causes the reports to take  
something like 3 minutes to produce output - if I get a blank page  
that typically takes about the same amount of time as the whole report.

The database connection is only in the origin of the data, if I cut  
down the report, so that it only does the database connection, or  
just forms the base objects, it will usually do this without issue  
(but often will not if it's already gone through using up all the RAM  
and spitting out a blank page).

Probably what I should have also mentioned is that it could have been  
running out of RAM previously, but I've upgraded the RAM, and there  
could be 600MB or more free now, but it still ends up outputting a  
blank page at times (the RAM upgrade improved this, in that it will  
run several times, where it was only running if I limited the scope  
of the reports artificially).

I'm also seeing that Lasso will occasionally release the RAM (I think  
this only happens if I restart Lasso), but then there's a big chunk  
of RAM "inactive", that just keeps on growing (until there's none  
"free") if I try to run the report again.

> You could try searching the archive, too.
> http://lasso.2283332.n4.nabble.com/template/NamlServlet.jtp?
> macro=search_page&node=3096189&query=malloc+lasso8service
> +szone_error&days=0

Thanks.


-- Clive

#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: 8.5.6 memory issues

stevepiercy
On 7/19/15 at 10:20 PM, [hidden email] (Clive Bruton) pronounced:

>>In my experience, the cause is usually something smelly in the
>>code—making too many connections to a datasource, abusing
>>FileMaker, generating huge PDFs—and not in Lasso itself.  A
>>good indicator of smelly code is if generating the report one
>>time takes more than a few seconds.
>
>I'm fairly certain that the problem lies within Lasso, because
>the MySQL query that originates the data takes a few seconds,
>but forming all the objects and manipulating them causes the
>reports to take something like 3 minutes to produce output - if
>I get a blank page that typically takes about the same amount
>of time as the whole report.

To me it sounds like manipulating the objects is where your code
could use some optimization, and that Lasso itself is not the
root cause.  Perhaps you are making copies of the objects, or
your objects are deeply nested or recursive?  Hard to say
without actual code to examine.

>The database connection is only in the origin of the data, if I
>cut down the report, so that it only does the database
>connection, or just forms the base objects, it will usually do
>this without issue (but often will not if it's already gone
>through using up all the RAM and spitting out a blank page).

If I understand correctly, the db connection and creation of the
objects are not the problem, but as soon as you start
manipulating the objects, your code crashes Lasso, correct?  If
so, that supports my intuition above.

>Probably what I should have also mentioned is that it could
>have been running out of RAM previously, but I've upgraded the
>RAM, and there could be 600MB or more free now, but it still
>ends up outputting a blank page at times (the RAM upgrade
>improved this, in that it will run several times, where it was
>only running if I limited the scope of the reports artificially).
>
>I'm also seeing that Lasso will occasionally release the RAM (I
>think this only happens if I restart Lasso), but then there's a
>big chunk of RAM "inactive", that just keeps on growing (until
>there's none "free") if I try to run the report again.

It sounds like you watch top or the Activity Monitor while the
process is running, so you can get a feel for when RAM gets
consumed and released.  That's a good idea.

Although Lasso can handle large files, what you actually do with
those files in your Lasso code can cause serious problems.  For
example somebody somewhere mentioned that string manipulation
with += is grossly inefficient compared to -> append in Lasso 9,
and I assume the same is true in Lasso 8.  I can't remember the
exact reason -> append is faster, but I think it is because +=
copies the original string, then appends the part to be appended
to the copy, and finally assigns the result to the string
variable, whereas -> append just tacks the string to be appended
on to the end of the string variable without copying the data.  
En/decoding data and serialization can be killers.

If you currently dump your report in one huge file_write at the
end, I would strongly suggest instead that you use file_write to
incrementally build your report in chunks of pages.  This won't
address any problems with inefficient code, but it might at
least allow the process to complete.

Here are a couple of things you can do to help isolate where
your code is slow, which implies a lot of RAM or CPU consumption.

* Sprinkle your code with timers, at the start, end, and once
inside.  Jason Huck's timer is probably the more appropriate in
this situation.
http://www.lassosoft.com/tagswap/detail/timer
http://www.lassosoft.com/tagswap/detail/lp_page_timer
* Sprinkle your code with:
log_critical(date + ' ' + timer('did something'));
then examine the log file.  tail -f <logfile> does it in realtime.

--steve

-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
Steve Piercy              Website Builder              Soquel, CA
<[hidden email]>               <http://www.StevePiercy.com/>


#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: 8.5.6 memory issues

Clive Bruton

On 19 Jul 2015, at 23:15, Steve Piercy - Website Builder wrote:

> To me it sounds like manipulating the objects is where your code  
> could use some optimization, and that Lasso itself is not the root  
> cause.  Perhaps you are making copies of the objects, or your  
> objects are deeply nested or recursive?  Hard to say without actual  
> code to examine.

Prior to the RAM upgrade there were occasions where forming the  
objects would fail - so I'm tending to think that Lasso is just  
running out of RAM, rather than the code causing some issue somewhere  
along its path. I've tried stripping out various elements of the  
report, to see if it's problematic code in one area or another, but  
never found anything consistent.

In particular, if I artificially restrict the original data set, then  
there are levels at which the problem does not occur. I've just done  
some tests that return a report of about 10% of the size of the full  
report (that doesn't necessarily mean that it is 10% of the data).  
While an initial hit sees Lasso grab about 300MB of RAM, that it does  
not release, this does not grow, nor does the "free" RAM start  
disappearing, no matter how many times the same report is requested.

If I make simultaneous requests for the same report Lasso will grab  
and hold on to about another 200MB of RAM for each request (ie three  
requests has it just under 700MB). But again repeating the requests  
doesn't eat in to the "free" RAM any further.

Is it normal that Lasso doesn't release the RAM after processing?

My conclusion is that it's the overall size of the data that seems to  
be the issue, rather than particular pieces of code consistently  
tripping Lasso. I'm sure that there could be optimisation (who's code  
is perfect?), but it doesn't seem to be some loop or recursion that's  
going awry.

For example, the report will execute perhaps five or six times  
without an issue, but then Lasso seems to fall over.

> If I understand correctly, the db connection and creation of the  
> objects are not the problem, but as soon as you start manipulating  
> the objects, your code crashes Lasso, correct?  If so, that  
> supports my intuition above.

Not necessarily, see above.

> Although Lasso can handle large files, what you actually do with  
> those files in your Lasso code can cause serious problems.  For  
> example somebody somewhere mentioned that string manipulation with  
> += is grossly inefficient compared to -> append in Lasso 9, and I  
> assume the same is true in Lasso 8.  I can't remember the exact  
> reason -> append is faster, but I think it is because += copies the  
> original string, then appends the part to be appended to the copy,  
> and finally assigns the result to the string variable, whereas ->  
> append just tacks the string to be appended on to the end of the  
> string variable without copying the data.  En/decoding data and  
> serialization can be killers.

There is a lot of += of strings in there, so I could look at that, if  
someone can confirm that ->append is much more efficient in Lasso  
8.5. There's next to no data encoding in there, or serialisation.

>
> If you currently dump your report in one huge file_write at the  
> end, I would strongly suggest instead that you use file_write to  
> incrementally build your report in chunks of pages.  This won't  
> address any problems with inefficient code, but it might at least  
> allow the process to complete.

No, it's just delivered as HTML to a browser. I don't expect that  
this is really an issue, because data in the output stream isn't  
being manipulated. Essentially the objects are formed, and then  
passed to various tags to form the output, which is in classic Lasso  
syntax (so it looks something like):

        [$obj1 = make_obj1]
        [$obj2 = make_obj2($obj1)]
        [$obj3 = make_obj3($obj1)]

        [report1($obj1)]
        [report2($obj1)]
        [report3($obj1)]

        [report4($obj2)]
        [report5($obj2)]
        [report6($obj2)]

        [report7($obj3)]
        [report8($obj3)]
        [report9($obj3)]

etc

>
> Here are a couple of things you can do to help isolate where your  
> code is slow, which implies a lot of RAM or CPU consumption.
>
> * Sprinkle your code with timers, at the start, end, and once inside.

Yep, that's all in the code already. So, for instance I can see that  
there aren't really any particular parts that take a long time  
(forming the initial objects is the most time consuming), the reports  
are reasonably quick (given the amount of data) and essentially each  
step gets longer as the data set grows (but nothing stands out above  
any of the others).

> * Sprinkle your code with:
> log_critical(date + ' ' + timer('did something'));
> then examine the log file.  tail -f <logfile> does it in realtime.

Currently not logging anything, but that may help.

Would like to get some feedback on the unreleased RAM though.


-- Clive

#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: 8.5.6 memory issues

stevepiercy
On 7/20/15 at 12:38 AM, [hidden email] (Clive Bruton) pronounced:

>On 19 Jul 2015, at 23:15, Steve Piercy - Website Builder wrote:
>
>>To me it sounds like manipulating the objects is where your
>>code could use some optimization, and that Lasso itself is not
>>the root cause.  Perhaps you are making copies of the objects,
>>or your objects are deeply nested or recursive?  Hard to say
>>without actual code to examine.
>
>Prior to the RAM upgrade there were occasions where forming the
>objects would fail - so I'm tending to think that Lasso is just
>running out of RAM, rather than the code causing some issue
>somewhere along its path. I've tried stripping out various
>elements of the report, to see if it's problematic code in one
>area or another, but never found anything consistent.
>
>In particular, if I artificially restrict the original data
>set, then there are levels at which the problem does not occur.
>I've just done some tests that return a report of about 10% of
>the size of the full report (that doesn't necessarily mean that
>it is 10% of the data). While an initial hit sees Lasso grab
>about 300MB of RAM, that it does not release, this does not
>grow, nor does the "free" RAM start disappearing, no matter how
>many times the same report is requested.
>
>If I make simultaneous requests for the same report Lasso will
>grab and hold on to about another 200MB of RAM for each request
>(ie three requests has it just under 700MB). But again
>repeating the requests doesn't eat in to the "free" RAM any further.

Which column of RAM are you looking at?  In Activity Monitor or
top, the one that matters is Real Mem or RSIZE.

Nonetheless, if each request consumes 200MB of RAM, then that's
excessive, especially if there will be concurrent requests.  You
must find a way to reduce that consumption in your code.

>Is it normal that Lasso doesn't release the RAM after processing?

Lasso should release Real Mem/RSIZE, although sometimes it does
not.  For me it tends to gradually creep upward until monit
restarts the site every week or so.

>My conclusion is that it's the overall size of the data that
>seems to be the issue, rather than particular pieces of code
>consistently tripping Lasso. I'm sure that there could be
>optimisation (who's code is perfect?), but it doesn't seem to
>be some loop or recursion that's going awry.
>
>For example, the report will execute perhaps five or six times
>without an issue, but then Lasso seems to fall over.
>
>>If I understand correctly, the db connection and creation of
>>the objects are not the problem, but as soon as you start
>>manipulating the objects, your code crashes Lasso, correct?  
>>If so, that supports my intuition above.
>
>Not necessarily, see above.

It's inconclusive without examining code.  It's too easy to
blame Lasso, especially when your code "just works"*.

* under certain conditions, void where prohibited, must be at
least 21 years of age or older.

>>Although Lasso can handle large files, what you actually do
>>with those files in your Lasso code can cause serious
>>problems.  For example somebody somewhere mentioned that
>>string manipulation with += is grossly inefficient compared to
>>-> append in Lasso 9, and I assume the same is true in Lasso
>>8.  I can't remember the exact reason -> append is faster, but
>>I think it is because += copies the original string, then
>>appends the part to be appended to the copy, and finally
>>assigns the result to the string variable, whereas -> append
>>just tacks the string to be appended on to the end of the
>>string variable without copying the data.  En/decoding data
>>and serialization can be killers.
>
>There is a lot of += of strings in there, so I could look at
>that, if someone can confirm that ->append is much more
>efficient in Lasso 8.5. There's next to no data encoding in
>there, or serialisation.

You can test it with a loop in two separate files.  Season to
taste.  No need to output the actual strings.  Run several times
to get an adequate sample size.

addequals.lasso
-------------------------
var('debug') = true;
local('n') = 1000;
local('s') = string;
local('data') = 'd '*1000000; // make a string of significant size

timer('start +=');
loop(#n);
     #s1 += #data;
/loop;
timer('end +=');

append.lasso
-------------------------
var('debug') = true;
local('n') = 1000;
local('s') = string;
local('data') = '<insert_massive_string_of_200MB>';

timer('start append');
loop(#n);
     #s2 -> append(#data);
/loop;
timer('end append');

>>If you currently dump your report in one huge file_write at
>>the end, I would strongly suggest instead that you use
>>file_write to incrementally build your report in chunks of
>>pages.  This won't address any problems with inefficient code,
>>but it might at least allow the process to complete.
>
>No, it's just delivered as HTML to a browser. I don't expect
>that this is really an issue, because data in the output stream
>isn't being manipulated. Essentially the objects are formed,
>and then passed to various tags to form the output, which is in
>classic Lasso syntax (so it looks something like):
>
>[$obj1 = make_obj1]
>[$obj2 = make_obj2($obj1)]
>[$obj3 = make_obj3($obj1)]
>
>[report1($obj1)]
>[report2($obj1)]
>[report3($obj1)]
>
>[report4($obj2)]
>[report5($obj2)]
>[report6($obj2)]
>
>[report7($obj3)]
>[report8($obj3)]
>[report9($obj3)]
>
>etc

How big is each of the objects if you dump it to a file?

Anyway, without seeing what actually happens in your tags and
types, my only recommendation would be to dump stuff as each
step finishes to file, appending as you go, then serve the
resulting file.

--steve

-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
Steve Piercy              Website Builder              Soquel, CA
<[hidden email]>               <http://www.StevePiercy.com/>


#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: 8.5.6 memory issues

Clive Bruton

On 20 Jul 2015, at 02:02, Steve Piercy - Website Builder wrote:

> var('debug') = true;
> local('n') = 1000;
> local('s') = string;
> local('data') = 'd '*1000000; // make a string of significant size
>
> timer('start +=');
> loop(#n);
>     #s1 += #data;
> /loop;
> timer('end +=');
>
> append.lasso
> -------------------------
> var('debug') = true;
> local('n') = 1000;
> local('s') = string;
> local('data') = '<insert_massive_string_of_200MB>';
>
> timer('start append');
> loop(#n);
>     #s2 -> append(#data);
> /loop;
> timer('end append');


Well, what I discern from that is that += takes about 25% longer than  
append. But the really interesting aspect is that neither seems to be  
affected very much whether you run the loop 100 times, 1,000 times,  
10,000 times or even 100,000 times - it always takes ~10secs/8secs  
for each operation (and I even put in a #z += 1 in the loops to check  
it really was looping that many times).

> Anyway, without seeing what actually happens in your tags and types...

Not really practical, it's a whole suite of tags and types - it's at  
least 6,000 lines. So without knowing what is causing the problem I  
couldn't really ask what anyone thought of 20 particular lines.

Thanks for the feedback, at least I'll probably be using append for  
strings now.


-- Clive

#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: 8.5.6 memory issues

Clive Bruton
After experimenting for some time I haven't really come to any  
conclusion as to why Lasso grabs so much RAM when running my reports,  
but I did develop a test that illustrates the problem:

*************

<?lassoscript

        var: 'jumbleA' = map();
       
        local: 'randC' = math_random(-lower=4, -upper=8);
       
        loop: #randC;
       
                local: 'randA' = math_random(-lower=1, -upper=128);
       
                var: 'jumbleB' = array();
       
                loop: #randA;
               
                        local: 'randD' = math_random(-lower=64, -upper=256);
               
                        local: 'randB' = math_random(-lower=8, -upper=32);
                       
                        //A: This grabs loads of RAM, but gives it back
                        //$jumbleB->insert(pair(fwp_makeid(-len=20)=(fwp_makeid(-
len=#randB)*#randD)));
                       
                        //B: This grabs a bit of RAM, but doesn't let it go
                        $jumbleB->insert(pair(fwp_makeid(-len=20)=fwp_makeid(-len=#randD)));
               
                /loop;
               
                $jumbleA->insert(pair(fwp_makeid(-len=20)=$jumbleB));

       
        /loop;
       
?>

[$jumbleA]

*************

If you make multiple requests for this script running "A" to modify  
$jumbleB then Lasso can grab up to 600MB, but then lets it go again.  
But if you run "B" to modify $jumbleB Lasso grabs around 100MB, but  
doesn't let it go again.

As my reports do a lot of iteration and looping through maps and  
arrays I'm thinking this is the core of the problem.

Oh, of course you'll need fwp_makeid.


-- Clive

#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: 8.5.6 memory issues

Clive Bruton
I have just come to the conclusion that I'm not going to make any  
progress with this, at least as far as Lasso holding on to RAM goes.  
The base SQL query which kicks off the reports causes Lasso to grab  
an additional 300MB, without doing any processing at all - and it  
doesn't seem to let go of this, without a restart.

While I'd like to think I could improve my code to avoid this, I  
can't really see how I can "improve" what Lasso does if all I'm  
asking it to do is execute a single (fairly complex) MySQL query.

I'd also like to think that the 300MB it is consuming is all that  
lovely data I'm feeding it - but dumped to a text file that only  
comes to 4.5MB.


-- Clive

#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: 8.5.6 memory issues

stevepiercy
Are you saying that you created a reduced test case, something
like this, and Lasso won't release the memory?

test.lasso
---------------
[
inline(
     -host=array(<connection params>), // most efficient
connection method for Lasso 8, bypasses Lasso Admin security
     -database=d,
     -table=t,
     -sql="sql query",
     );
     found_count;
/inline;
]

--steve


On 7/25/15 at 7:04 PM, [hidden email] (Clive Bruton) pronounced:

>I have just come to the conclusion that I'm not going to make
>any progress with this, at least as far as Lasso holding on to
>RAM goes. The base SQL query which kicks off the reports causes
>Lasso to grab an additional 300MB, without doing any processing
>at all - and it doesn't seem to let go of this, without a restart.
>
>While I'd like to think I could improve my code to avoid this,
>I can't really see how I can "improve" what Lasso does if all
>I'm asking it to do is execute a single (fairly complex) MySQL query.
>
>I'd also like to think that the 300MB it is consuming is all
>that lovely data I'm feeding it - but dumped to a text file
>that only comes to 4.5MB.
>
>
>-- Clive
>
>#############################################################
>
>This message is sent to you because you are subscribed to
>the mailing list Lasso [hidden email]
>Official list archives available at http://www.lassotalk.com
>To unsubscribe, E-mail to: <[hidden email]>
>Send administrative queries to  <[hidden email]>

-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
Steve Piercy              Website Builder              Soquel, CA
<[hidden email]>               <http://www.StevePiercy.com/>


#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: 8.5.6 memory issues

Bil Corry-3
In reply to this post by Clive Bruton
Kyle has mentioned in the past that Lasso will hold onto memory and reuse it.  I guess if your test repeated always ups the memory, then it's a problem, but if it allocates memory the first time, but then reuses it on subsequent tests, then it should be fine.

- Bil

> On Jul 25, 2015, at 8:04 PM, Clive Bruton <[hidden email]> wrote:
>
> I have just come to the conclusion that I'm not going to make any progress with this, at least as far as Lasso holding on to RAM goes. The base SQL query which kicks off the reports causes Lasso to grab an additional 300MB, without doing any processing at all - and it doesn't seem to let go of this, without a restart.
>
> While I'd like to think I could improve my code to avoid this, I can't really see how I can "improve" what Lasso does if all I'm asking it to do is execute a single (fairly complex) MySQL query.
>
> I'd also like to think that the 300MB it is consuming is all that lovely data I'm feeding it - but dumped to a text file that only comes to 4.5MB.
>
>
> -- Clive
>
> #############################################################
>
> This message is sent to you because you are subscribed to
> the mailing list Lasso [hidden email]
> Official list archives available at http://www.lassotalk.com
> To unsubscribe, E-mail to: <[hidden email]>
> Send administrative queries to  <[hidden email]>

#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: 8.5.6 memory issues

Clive Bruton
In reply to this post by stevepiercy

On 25 Jul 2015, at 22:06, Steve Piercy - Website Builder wrote:

> Are you saying that you created a reduced test case, something like  
> this, and Lasso won't release the memory?

Yes.


-- Clive

#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: 8.5.6 memory issues

stevepiercy
On 7/25/15 at 11:21 PM, [hidden email] (Clive Bruton) pronounced:

>On 25 Jul 2015, at 22:06, Steve Piercy - Website Builder wrote:
>
>>Are you saying that you created a reduced test case, something
>>like this, and Lasso won't release the memory?
>
>Yes.

So this becomes a problem when there are concurrent requests.  
If the requests are queued—one at a time—then does memory
continue to grow?

Ultimately it sounds like your options are to (1) find a way to
optimize your SQL, (2) use a queue for requests, or (3) break
apart the process into digestible chunks, writing the result to
a file and appending it for each chunk.

It also just occurred to me, perhaps Lasso's internal SQLite
database has become bloated?  I wonder if the concurrent
requests hit something in there, say the errors table?  If you
use Lasso Security instead of -host in your inline, then
maybe...?  It's not a likely cause, but should be considered for
elimination as a contributing factor.

--steve

-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
Steve Piercy              Website Builder              Soquel, CA
<[hidden email]>               <http://www.StevePiercy.com/>


#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: 8.5.6 memory issues

Clive Bruton
In reply to this post by Bil Corry-3

On 25 Jul 2015, at 23:00, Bil Corry wrote:

> Kyle has mentioned in the past that Lasso will hold onto memory and  
> reuse it.  I guess if your test repeated always ups the memory,  
> then it's a problem, but if it allocates memory the first time, but  
> then reuses it on subsequent tests, then it should be fine.

When trying some tests I find that it can do almost anything: grab  
600MB and then let it go, grab 300MB and keep it (in both cases  
reporting no errors), report out of memory errors and keep the RAM  
(and a hint that it won't use it again)...

I suppose from my perspective I don't really care whether it holds on  
to the RAM and reuses it, because nothing else is trying to use it -  
but what seems to happen is that Lasso grabs the RAM, performs ok for  
a while, but then starts returning blank pages. If it was giving-up  
the RAM then I could have an idea what it was doing, but the way it  
is now is I don't know whether there's an underlying problem or not.

The only clue I get is if I run it in console mode, then I get the  
memory error I posted previously:

Lasso8Service(1602,0x1d2ee00) malloc: *** vm_allocate(size=6778880)  
failed (error code=3)
Lasso8Service(1602,0x1d2ee00) malloc: *** error: can't allocate region
Lasso8Service(1602,0x1d2ee00) malloc: *** set a breakpoint in  
szone_error to debug
Lasso8Service(1602,0x1d2ee00) malloc: *** vm_allocate(size=5423104)  
failed (error code=3)
Lasso8Service(1602,0x1d2ee00) malloc: *** error: can't allocate region
Lasso8Service(1602,0x1d2ee00) malloc: *** set a breakpoint in  
szone_error to debug

The machine isn't running out of RAM, I can see 5-600MB free, and  
Lasso has grabbed about 1.8GB.


-- Clive

#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: 8.5.6 memory issues

Wade Maxfield
> On 26/07/2015, at 10:34, Clive Bruton <[hidden email]> wrote:
>
> The machine isn't running out of RAM, I can see 5-600MB free, and Lasso has grabbed about 1.8GB.

Lasso 8.5.6 is running as a 32 bit app, and I think it will be limited to max of 2GB for a single process, even if you have more ram installed.

Just as an out there suggestion, at the very end of processing one of the reports have you tried reinitialising some of the larger variables?  For instance in your jumble example, as the very last thing on the page set jumbleA back to an empty map: var: 'jumbleA' = map();

 - Wade



#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: 8.5.6 memory issues

Clive Bruton
In reply to this post by stevepiercy

On 25 Jul 2015, at 23:31, Steve Piercy - Website Builder wrote:

> On 7/25/15 at 11:21 PM, [hidden email] (Clive Bruton) pronounced:
>
>> On 25 Jul 2015, at 22:06, Steve Piercy - Website Builder wrote:
>>
>>> Are you saying that you created a reduced test case, something  
>>> like this, and Lasso won't release the memory?
>>
>> Yes.
>
> So this becomes a problem when there are concurrent requests.  If  
> the requests are queued—one at a time—then does memory continue to  
> grow?

It seems that it, the original report, grabs about 1.8GB, but that  
this doesn't grow (or at least not noticeably) on consecutive hits.  
On concurrent hits it just pretty quickly. Even with consecutive hits  
it might operate fine half a dozen times, but then fail from that  
point on.

Just doing the SQL query alone Lasso seems to grab the RAM, and then  
not take any more, even with concurrent hits it just grabs a couple  
of MB extra (from around 300MB it already had).

> Ultimately it sounds like your options are to (1) find a way to  
> optimize your SQL, (2) use a queue for requests, or (3) break apart  
> the process into digestible chunks, writing the result to a file  
> and appending it for each chunk.

That's not really going to work, because the analysis Lasso is doing  
after the SQL requires it to be looking at the whole data set - even  
if it gets written to file, Lasso will have to reload it all to be  
able to process it.

I think what will have to happen is that the reports will have to be  
broken up, so that it does little segments of those at a time - but I  
don't really have a lot of confidence that will help, at least in the  
longer term:

     i    I think it's the complex objects that are using
          up all the RAM

     ii   It's more likely than not that the data set is
          going to get larger (in fact it will only get
          larger), so even if I do work around it this
          week it's only going to crop up again.


> It also just occurred to me, perhaps Lasso's internal SQLite  
> database has become bloated?  I wonder if the concurrent requests  
> hit something in there, say the errors table?  If you use Lasso  
> Security instead of -host in your inline, then maybe...?  It's not  
> a likely cause, but should be considered for elimination as a  
> contributing factor.

There are no new entries in the errors table, I don't know why, but  
it seems to have stopped writing new errors on this site. I doubt  
that it's really any real issue with the security settings, because  
it's only me testing it on a local desktop, and even when it's  
running on a production host it's still only been me making the hits.

-- Clive

#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: 8.5.6 memory issues

Clive Bruton
In reply to this post by Wade Maxfield

On 26 Jul 2015, at 03:13, Wade Maxfield wrote:

>> On 26/07/2015, at 10:34, Clive Bruton <[hidden email]> wrote:
>>
>> The machine isn't running out of RAM, I can see 5-600MB free, and  
>> Lasso has grabbed about 1.8GB.
>
> Lasso 8.5.6 is running as a 32 bit app, and I think it will be  
> limited to max of 2GB for a single process, even if you have more  
> ram installed.

Well, 32bits of addressing is 4.3GB of RAM, but if you're saying a  
single process is limited to 2GB, then I can see that would quickly  
be a problem - if I'm using 1.8GB, then it's only going to take a bit  
of extra processing/data to run into the limit.

>
> Just as an out there suggestion, at the very end of processing one  
> of the reports have you tried reinitialising some of the larger  
> variables?  For instance in your jumble example, as the very last  
> thing on the page set jumbleA back to an empty map: var: 'jumbleA'  
> = map();

I didn't do that with that test, but I will try it. I have tried that  
with the report suit and it didn't make any difference.

I'm probably expecting too much of Lasso here and should see if I can  
get MySQL to do more of the work, perhaps by creating temporary  
tables, etc.


-- Clive

#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>