knop_nav performance and cache

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

knop_nav performance and cache

stevepiercy
A really big, complex knop_nav object takes a long time to
render.  Time in milliseconds.

     render thisnav_https anon   4517
     render thisnav_https auth   4685
     render thisnav_http anon    4351
     render subnav_leaves        2661

Bleargh!

So after experimenting with knop_cache, [cache], and finally
[cache_*] tags, I settled on using the latter to render once and
cache it using the following sample implementation:

     if(cache_exists(-name=(server_name + '_https_anon')));
         cache_fetch(-name=(server_name + '_https_anon'), -encodenone);
     else;
         cache_store(-name=(server_name + '_https_anon'),
             -content=thisnav_https($nav->renderhtml(-expand)),
             -expires=60*60*24);
         cache_fetch(-name=(server_name + '_https_anon'), -encodenone);
     /if;

Results:

     render thisnav_https anon   3
     render thisnav_https auth   5
     render thisnav_http anon    2
     render subnav_leaves        2

One of the things I discovered was that knop_cache does not work
at all like Lasso's [cache].  knop_cache stores knop objects,
whereas [cache] stores strings and no other objects.  I tried this:

     knop_cachestore(-type='string');

but was disappointed when I tried to retrieve and display that string.

I also found that [cache] was utterly useless for controlling
cache.  I don't know wtf LassoSoft has done with that tag, but
make your life easier and just use [cache_*] control tags.  I
must have tried a dozen permutations of using plain old [cache]
to no avail.

--steve

-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
-- --
Steve Piercy               Web Site Builder              
Soquel, CA
<[hidden email]>                  <http://www.StevePiercy.com/>


--
#############################################################
This message is sent to you because you are subscribed to
the mailing list <[hidden email]>.
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
List archive http://lasso.2283332.n4.nabble.com/Knop-Framework-Discussion-f3157831.html
The Knop project and code is hosted at GitHub.
https://github.com/knop-project/knop
Reply | Threaded
Open this post in threaded view
|

Re: knop_nav performance and cache

Ric Lewis-3
Hmm, interesting.

Whilst not that big, our nav does take a while, we’re still on Knop 8, Lasso 8.6.2. (I know there’s a fast nav in K9/L9)

If the results are so dramatic, looks like it should go into the framework, no?

Any changes to nav (ie new options after logging in etc), and the whole thing is re-loaded and thus re-cached anyway.

So no downside?


I’ll try out your technique, thanks for the tip!


On 5 Jan 2014, at 13:48, Steve Piercy - Web Site Builder <[hidden email]> wrote:

> A really big, complex knop_nav object takes a long time to render.  Time in milliseconds.
>
>    render thisnav_https anon   4517
>    render thisnav_https auth   4685
>    render thisnav_http anon    4351
>    render subnav_leaves        2661
>
> Bleargh!
>
> So after experimenting with knop_cache, [cache], and finally [cache_*] tags, I settled on using the latter to render once and cache it using the following sample implementation:
>
>    if(cache_exists(-name=(server_name + '_https_anon')));
>        cache_fetch(-name=(server_name + '_https_anon'), -encodenone);
>    else;
>        cache_store(-name=(server_name + '_https_anon'),
>            -content=thisnav_https($nav->renderhtml(-expand)),
>            -expires=60*60*24);
>        cache_fetch(-name=(server_name + '_https_anon'), -encodenone);
>    /if;
>
> Results:
>
>    render thisnav_https anon   3
>    render thisnav_https auth   5
>    render thisnav_http anon    2
>    render subnav_leaves        2
>
> One of the things I discovered was that knop_cache does not work at all like Lasso's [cache].  knop_cache stores knop objects, whereas [cache] stores strings and no other objects.  I tried this:
>
>    knop_cachestore(-type='string');
>
> but was disappointed when I tried to retrieve and display that string.
>
> I also found that [cache] was utterly useless for controlling cache.  I don't know wtf LassoSoft has done with that tag, but make your life easier and just use [cache_*] control tags.  I must have tried a dozen permutations of using plain old [cache] to no avail.
>
> --steve
>
> -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
> Steve Piercy               Web Site Builder               Soquel, CA
> <[hidden email]>                  <http://www.StevePiercy.com/>
>
>
> --
> #############################################################
> This message is sent to you because you are subscribed to
> the mailing list <[hidden email]>.
> To unsubscribe, E-mail to: <[hidden email]>
> Send administrative queries to  <[hidden email]>
> List archive http://lasso.2283332.n4.nabble.com/Knop-Framework-Discussion-f3157831.html
> The Knop project and code is hosted at GitHub.
> https://github.com/knop-project/knop


--
#############################################################
This message is sent to you because you are subscribed to
the mailing list <[hidden email]>.
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
List archive http://lasso.2283332.n4.nabble.com/Knop-Framework-Discussion-f3157831.html
The Knop project and code is hosted at GitHub.
https://github.com/knop-project/knop
Reply | Threaded
Open this post in threaded view
|

Re: knop_nav performance and cache

stevepiercy
As an aside, I don't know whether my code is thread-safe.  I
haven't really delved into that, although Knop's util.inc has
some examples of how to implement thread safety.

Also to detect timings, I use this for pretty barcharts,
timings, and percent of total time.
http://www.lassosoft.com/tagswap/detail/timer

Do this (usually in cnt_*.inc or your template):

     timer('render thisnav_http anon start');
     $nav->renderhtml(-expand);
     timer('render thisnav_http anon end');

And set $debug to true in cfg__global.inc, and use timerstats in
the debug block (usually in your template footer).

I also cache the nav objects as they are created in
cfg__nav.inc, for example:

     ...
     if(!knop_cachefetch(-type='knop_nav', -name=$s_user->username));
         var('nav_admin') = knop_nav;
             $nav_admin->insert(
                 -key='users',
                 -label='Users',
                 -default='list',
                 -children=$nav_list_edit);
         ...
         knop_cachestore(-type='knop_nav',
-name=$s_user->username, -expires=$admin_session);
     /if;

My nav has 101 ->inserts, and 44 nestings with -children.  Yeah,
that's nuts.  It's a tree taxonomy.

I might migrate this app to Pyramid or SubstanceD, an
application server built using the Pyramid web framework,
because both support hierarchical data structures with no speed penalty.

Anyway, to address your comments and question, I'm using 8.6.x,
too.  AFAIK, Knop for Lasso 9 uses the same methods to build a
nav as Knop for Lasso 8: Yo Dawg, I put a map inside of your map!

There was talk about 5 years of how to create a better data
structure.  I've been looking at ZODB in Python as an example,
as well as XML, and xs_cat.  However, I can't stomach the
concept of trying to jam a tree-like data structure into a
relational database; conical peg, square hole.

As far as an addition to the framework, let's see.  I imagine it
would need a series of flags:

     $nav->renderhtml(
         ...
         -cache='cachename',
         -expires=seconds,
         -username=$s_user->username);

Comments?

--steve


On 1/5/14 at 5:54 PM, [hidden email] (Ric Lewis) pronounced:

>Hmm, interesting.
>
>Whilst not that big, our nav does take a while, we’re still
>on Knop 8, Lasso 8.6.2. (I know there’s a fast nav in K9/L9)
>
>If the results are so dramatic, looks like it should go into the framework, no?
>
>Any changes to nav (ie new options after logging in etc), and
>the whole thing is re-loaded and thus re-cached anyway.
>
>So no downside?
>
>
>I’ll try out your technique, thanks for the tip!
>
>
>On 5 Jan 2014, at 13:48, Steve Piercy - Web Site Builder <[hidden email]> wrote:
>
>>A really big, complex knop_nav object takes a long time to render.  Time in
>milliseconds.
>>
>>render thisnav_https anon   4517
>>render thisnav_https auth   4685
>>render thisnav_http anon    4351
>>render subnav_leaves        2661
>>
>>Bleargh!
>>
>>So after experimenting with knop_cache, [cache], and finally [cache_*] tags, I
>settled on using the latter to render once and cache it using
>the following sample implementation:
>>
>>if(cache_exists(-name=(server_name + '_https_anon')));
>>cache_fetch(-name=(server_name + '_https_anon'), -encodenone);
>>else;
>>cache_store(-name=(server_name + '_https_anon'),
>>-content=thisnav_https($nav->renderhtml(-expand)),
>>-expires=60*60*24);
>>cache_fetch(-name=(server_name + '_https_anon'), -encodenone);
>>/if;
>>
>>Results:
>>
>>render thisnav_https anon   3
>>render thisnav_https auth   5
>>render thisnav_http anon    2
>>render subnav_leaves        2
>>
>>One of the things I discovered was that knop_cache does not work at all like Lasso's
>[cache].  knop_cache stores knop objects, whereas [cache]
>stores strings and no other objects.  I tried this:
>>
>>knop_cachestore(-type='string');
>>
>>but was disappointed when I tried to retrieve and display that string.
>>
>>I also found that [cache] was utterly useless for controlling cache.  I don't know
>wtf LassoSoft has done with that tag, but make your life easier
>and just use [cache_*] control tags.  I must have tried a dozen
>permutations of using plain old [cache] to no avail.
>>
>>--steve
>>
>>-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
>>Steve Piercy               Web Site Builder               Soquel, CA
>><[hidden email]>                  <http://www.StevePiercy.com/>
>>
>>
>>--
>>#############################################################
>>This message is sent to you because you are subscribed to
>>the mailing list <[hidden email]>.
>>To unsubscribe, E-mail to: <[hidden email]>
>>Send administrative queries to  <[hidden email]>
>>List archive
>http://lasso.2283332.n4.nabble.com/Knop-Framework-Discussion-f3157831.html
>>The Knop project and code is hosted at GitHub.
>>https://github.com/knop-project/knop
>
>

-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
-- --
Steve Piercy               Web Site Builder              
Soquel, CA
<[hidden email]>                  <http://www.StevePiercy.com/>


--
#############################################################
This message is sent to you because you are subscribed to
the mailing list <[hidden email]>.
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
List archive http://lasso.2283332.n4.nabble.com/Knop-Framework-Discussion-f3157831.html
The Knop project and code is hosted at GitHub.
https://github.com/knop-project/knop
Reply | Threaded
Open this post in threaded view
|

Re: knop_nav performance and cache

Ric Lewis-3
When I tried out your technique on the dev site I appended a $nav->getnav to the ‘cachename', as the submenu changes depending on location (plus the current location is highlighted in the main menu). This could be done more elegantly but it works fine.

So in our case we need to cache a number of menu/submenu variations, and three sets of menus, one for each user type: regular users, admins and super users. Some apps might require individual nav html for each user but I would imagine that’s rare, presumably most systems have a finite number of user types, who are all presented with with the same set of menus/options.

knop_cachestore(-type='knop_nav', -name=$s_user->id, -expires=$sessiontime);

Yes, we do this too, however that’s for the nav object which affects all sorts of things like security etc. I’m now thinking that storing for every user is wasteful, but I believed there was little overhead in knop_caching. I’m assuming that the rendering into html is the problem in your instance, not the nav routine itself.

I haven’t done performance comparisons yet, but I’ll check out the tag you linked, thanks. 

Jono had a presentation on tree data structures in Florida a while back, didn’t he? Pretty fascinating but far removed from anything I’d encountered. Perhaps I remember wrong, you were there too I think.


On 5 Jan 2014, at 23:05, Steve Piercy - Web Site Builder <[hidden email]> wrote:

As far as an addition to the framework, let's see.  I imagine it would need a series of flags:

   $nav->renderhtml(
       ...
       -cache='cachename',
       -expires=seconds,
       -username=$s_user->username);

Comments?

--steve

Reply | Threaded
Open this post in threaded view
|

Re: knop_nav performance and cache

stevepiercy
On 1/5/14 at 11:46 PM, [hidden email] (Ric Lewis) pronounced:

>When I tried out your technique on the dev site I appended a
>$nav->getnav to the ‘cachename', as the submenu changes
>depending on location (plus the current location is highlighted
>in the main menu). This could be done more elegantly but it
>works fine.

That sounds good to me.  In fact it works great on ->renderbreadcrumb:

     timer('render breadcrumb start');
     if(cache_exists(-name=(server_name + '_bc_' + $nav->path)));
         cache_fetch(-name=(server_name + '_bc_' + $nav->path), -encodenone);
     else;
         cache_store(-name=(server_name + '_bc_' + $nav->path),
         -content=($nav->renderbreadcrumb(-home, -delimiter=' /
')->replace('<a href="/tree_type/">Overview</a>','Get Started')&),
         -expires=60*60*24);
         cache_fetch(-name=(server_name + '_bc_' + $nav->path), -encodenone);
     /if;
     timer('render breadcrumb end');

8 levels:
572 ms
  24 ms

6 levels:
360
  18

4 levels:
172
  14

2 levels:
95
  8

>So in our case we need to cache a number of menu/submenu
>variations, and three sets of menus, one for each user type:
>regular users, admins and super users. Some apps might require
>individual nav html for each user but I would imagine that’s
>rare, presumably most systems have a finite number of user
>types, who are all presented with with the same set of menus/options.

That's a good point.  It's better to use types of users, or
roles, for building and rendering nav objects that are shared.  
I made the adjustment to my cache names.

>knop_cachestore(-type='knop_nav', -name=$s_user->id, -expires=$sessiontime);
>
>Yes, we do this too, however that’s for the nav object which
>affects all sorts of things like security etc. I’m now
>thinking that storing for every user is wasteful, but I
>believed there was little overhead in knop_caching. I’m
>assuming that the rendering into html is the problem in your
>instance, not the nav routine itself.

The biggest hit was from rendering.  Building the nav objects
versus rendering was not nearly as much, 788ms vs. 158ms.

>Jono had a presentation on tree data structures in Florida a
>while back, didn’t he? Pretty fascinating but far removed
>from anything I’d encountered. Perhaps I remember wrong, you
>were there too I think.

I was there, but I don't recall it.  It may have been this:
http://www.lassosoft.com/tagswap/detail/xs_cat

Along with Brian Loomis':
http://www.lassosoft.com/tagswap/detail/ul_taxonomy/

--steve

-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
-- --
Steve Piercy               Web Site Builder              
Soquel, CA
<[hidden email]>                  <http://www.StevePiercy.com/>


--
#############################################################
This message is sent to you because you are subscribed to
the mailing list <[hidden email]>.
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
List archive http://lasso.2283332.n4.nabble.com/Knop-Framework-Discussion-f3157831.html
The Knop project and code is hosted at GitHub.
https://github.com/knop-project/knop
Reply | Threaded
Open this post in threaded view
|

Re: knop_nav performance and cache

Ric Lewis-3
Yes that was the one, I guess it was a round table session rather than a main presentation.


On 6 Jan 2014, at 02:32, Steve Piercy - Web Site Builder <[hidden email]> wrote:

Jono had a presentation on tree data structures in Florida a while back, didn’t he? Pretty fascinating but far removed from anything I’d encountered. Perhaps I remember wrong, you were there too I think.

I was there, but I don't recall it.  It may have been this:
http://www.lassosoft.com/tagswap/detail/xs_cat

Reply | Threaded
Open this post in threaded view
|

Re: knop_nav performance and cache

Ric Lewis-3
In reply to this post by stevepiercy
Great that your caching speeds nav html rendering, I think K9/L9 also sped up the whole nav object considerably.

There was talk of a 'lightweight' nav years ago, and I think the essence of that has been implemented in K9.

One of these days I'll have to move to Knop9  but currently stability is pretty good, performance is fine, so hard to prioritise that over new development (and end of year accounts!)

Perhaps we should arrange a mini conference/workshop for all the Knop8 holdouts*, in a few days we could probably get it all ported to K9/L9. 

*Mind you that's probably just you and me now :)



On 6 Jan 2014, at 02:32, Steve Piercy - Web Site Builder <[hidden email]> wrote:

The biggest hit was from rendering.  Building the nav objects versus rendering was not nearly as much, 788ms vs. 158ms.

Reply | Threaded
Open this post in threaded view
|

Re: knop_nav performance and cache

stevepiercy
After glancing over the change notes and nav->insert method,
it's essentially the same beast.  Jolle and Tim would know
better than I.
https://github.com/knop-project/knop/blob/master/knop9/knoplibs/knop_nav.lasso#L12-L26
https://github.com/knop-project/knop/blob/master/knop9/knoplibs/knop_nav.lasso#L178

There's lots of valid reasons not to move to Lasso 9.  At the
top of my list are:

* For my clients the benefits of migrating to 9 do not outweigh
or justify the cost to migrate.  They are happy with the legacy
apps.  8 is "good enough".
* There are better web frameworks and languages than Knop and
Lasso for the job.

That said, a Knop sprint would be very interesting to me.  
What's a sprint, you ask?
https://us.pycon.org/2014/community/sprints/

LDC does not have sprints, although I have lobbied for them.  
:(  Maybe this year with all the new open source projects coming out?

I've participated in three sprints for Pyramid: two immediately
following PyCon and one in Breckenridge, CO.  At BreckProg (a
combination of the town's name, programming, and progressive
rock) we rented a vacation cabin for a week, worked our tails
off, and took breaks with hikes, mountain biking, soaking in the
hot tub, cooking, jamming, gaming, and indulging.  I hope to go
to the next Pyramid sprint in New Orleans in February, but we
just refinanced our mortgage and tapped into our emergency fund
so cash is tight.

Those sprints led to landing new paying gigs, and more than paid
for themselves financially, not to mention the camaraderie.

As far as locations for a Knop (and any other Lasso open source
project) sprint, I'd nominate my locality of Santa Cruz/Monterey.

The only requirement for nominating a location is the commitment
to lead the organization of the sprint.  That means finding a
facility, collecting money a couple of weeks before the sprint
from participants to pay for the facility rental (and fronting
the deposit), providing logistic information to participants,
and after the sprint collecting money to pay for the groceries
collectively consumed (we do a big grocery run at the start) and
for any collective rental vehicles and gas.

--steve


On 1/6/14 at 8:12 AM, [hidden email] (Ric Lewis) pronounced:

>Great that your caching speeds nav html rendering, I think
>K9/L9 also sped up the whole nav object considerably.
>
>There was talk of a 'lightweight' nav years ago, and I think
>the essence of that has been implemented in K9.
>
>One of these days I'll have to move to Knop9  but currently
>stability is pretty good, performance is fine, so hard to
>prioritise that over new development (and end of year accounts!)
>
>Perhaps we should arrange a mini conference/workshop for all
>the Knop8 holdouts*, in a few days we could probably get it all
>ported to K9/L9.
>*Mind you that's probably just you and me now :)
>
>
>
>On 6 Jan 2014, at 02:32, Steve Piercy - Web Site Builder <[hidden email]> wrote:
>
>>The biggest hit was from rendering.  Building the nav objects versus rendering was
>not nearly as much, 788ms vs. 158ms.
>
>

-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
-- --
Steve Piercy               Web Site Builder              
Soquel, CA
<[hidden email]>                  <http://www.StevePiercy.com/>


--
#############################################################
This message is sent to you because you are subscribed to
the mailing list <[hidden email]>.
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
List archive http://lasso.2283332.n4.nabble.com/Knop-Framework-Discussion-f3157831.html
The Knop project and code is hosted at GitHub.
https://github.com/knop-project/knop
Reply | Threaded
Open this post in threaded view
|

Re: knop_nav performance and cache

Ric Lewis-3
Sure, it was just some earlier discussion and also the leading comment which suggests that it was a faster version of the nav object, but I haven't looked at the equivalent Knop 9 code.

\/**!
knop_nav
Experimental new leaner version of custom type to handle site navigation menu
**/

Sprint concept sounds good, I enjoy the LDCs but focussing completely on coding is also appealing. I had a stab at converting my codebase to Knop 9, no shortage of experts at Niagara, just too much going on to make any real progress.

On 6 Jan 2014, at 11:53, Steve Piercy - Web Site Builder <[hidden email]> wrote:

After glancing over the change notes and nav->insert method, it's essentially the same beast.  Jolle and Tim would know better than I.
https://github.com/knop-project/knop/blob/master/knop9/knoplibs/knop_nav.lasso#L12-L26
https://github.com/knop-project/knop/blob/master/knop9/knoplibs/knop_nav.lasso#L178

There's lots of valid reasons not to move to Lasso 9.  At the top of my list are:

* For my clients the benefits of migrating to 9 do not outweigh or justify the cost to migrate.  They are happy with the legacy apps.  8 is "good enough".
* There are better web frameworks and languages than Knop and Lasso for the job.

That said, a Knop sprint would be very interesting to me.  What's a sprint, you ask?
https://us.pycon.org/2014/community/sprints/

LDC does not have sprints, although I have lobbied for them.  :(  Maybe this year with all the new open source projects coming out?

I've participated in three sprints for Pyramid: two immediately following PyCon and one in Breckenridge, CO.  At BreckProg (a combination of the town's name, programming, and progressive rock) we rented a vacation cabin for a week, worked our tails off, and took breaks with hikes, mountain biking, soaking in the hot tub, cooking, jamming, gaming, and indulging.  I hope to go to the next Pyramid sprint in New Orleans in February, but we just refinanced our mortgage and tapped into our emergency fund so cash is tight.

Those sprints led to landing new paying gigs, and more than paid for themselves financially, not to mention the camaraderie.

As far as locations for a Knop (and any other Lasso open source project) sprint, I'd nominate my locality of Santa Cruz/Monterey.

The only requirement for nominating a location is the commitment to lead the organization of the sprint.  That means finding a facility, collecting money a couple of weeks before the sprint from participants to pay for the facility rental (and fronting the deposit), providing logistic information to participants, and after the sprint collecting money to pay for the groceries collectively consumed (we do a big grocery run at the start) and for any collective rental vehicles and gas.

--steve


On 1/6/14 at 8:12 AM, [hidden email] (Ric Lewis) pronounced:

Great that your caching speeds nav html rendering, I think K9/L9 also sped up the whole nav object considerably.

There was talk of a 'lightweight' nav years ago, and I think the essence of that has been implemented in K9.

One of these days I'll have to move to Knop9  but currently stability is pretty good, performance is fine, so hard to prioritise that over new development (and end of year accounts!)

Perhaps we should arrange a mini conference/workshop for all the Knop8 holdouts*, in a few days we could probably get it all ported to K9/L9.
*Mind you that's probably just you and me now :)



On 6 Jan 2014, at 02:32, Steve Piercy - Web Site Builder <[hidden email]> wrote:

The biggest hit was from rendering.  Building the nav objects versus rendering was
not nearly as much, 788ms vs. 158ms.



-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
Steve Piercy               Web Site Builder               Soquel, CA
<[hidden email]>                  <http://www.StevePiercy.com/>


--
#############################################################
This message is sent to you because you are subscribed to
the mailing list <[hidden email]>.
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
List archive http://lasso.2283332.n4.nabble.com/Knop-Framework-Discussion-f3157831.html
The Knop project and code is hosted at GitHub.
https://github.com/knop-project/knop

Reply | Threaded
Open this post in threaded view
|

Re: knop_nav performance and cache

Johan Solve-2
In reply to this post by stevepiercy
Caching the rendered nav is a very good idea, as well as caching the actual cache object.

I use knop_cche to cache the nav object and cache_* to cache rendered output (both nav and commonly used page content).

knop_cache should be able to cache page variables of any Lasso type, but using it to cache a common data type as string is probably not a good idea since all page vars of the specified type will be cached, and is not the purpose of it.

This is the core of kncop_cachestore:

        // store all page vars of the specified type
        iterate: vars -> keys, local: 'item';
                if: (var: #item) -> isa(#type);
                        #data -> insert(#item = (var: #item));
                /if;
        /iterate;

knop_cache should also be thread safe by using thread_locks for reads and writes.


To facilitate caching rendered content I use a [capture]...[/capture] container tag. Everything within it is captured instead of output to page.

The capture is returned with [capture_result] and stored with cache_store.


Define_Tag('capture', -container,
        -priority='replace',
        -optional='name');
        Var('__capture_result__' + local('name')) = run_children;
/Define_Tag;
Define_Tag('capture_result', -optional='xhtml',
        -priority='replace',
        -optional='name');
        if(var_defined('__capture_result__' + local('name')));
                local('result'=var('__capture_result__' + local('name')));
                if(#result -> size);
                        if(local_defined('xhtml') && #xhtml != false);
                                // fix endslash for xhtml
                                #result -> replace('<br>', '<br />') & replace('<br/>', '<br />');
                        /if;
                        Return(@#result);
                /if;
        /if;
/Define_Tag;



This is how I use it, first in a library file (a bit simplified, in reality the cached object is a map with content and timestamp):

var('content'=cache_fetch(-name=$cachename));
// validate the cache before using it
if($content -> size);

        knop_debug('Using cached data' = $cachename);

else;
        // no valid cache data, perform database queries to create new data
        ...
/if;


Then in a content file:

[if($content -> size);

        // use cached content
        knop_debug('Using cached content' = include_currentpath);
else;
        // create new content
        capture]

        Page content with markup and database output

        [/capture;
        $content = capture_result;
        // store the result in cache
        knop_debug('Storing content in cache');
        cache_store(-name=$cachename, -content=$output, -expires=$cache_expire);
/if;

]


I use a similar principle when caching the rendered nav.


At 05.48 -0800 2014-01-05, Steve Piercy - Web Site Builder wrote:

>A really big, complex knop_nav object takes a long time to render.  Time in milliseconds.
>
>    render thisnav_https anon   4517
>    render thisnav_https auth   4685
>    render thisnav_http anon    4351
>    render subnav_leaves        2661
>
>Bleargh!
>
>So after experimenting with knop_cache, [cache], and finally [cache_*] tags, I settled on using the latter to render once and cache it using the following sample implementation:
>
>    if(cache_exists(-name=(server_name + '_https_anon')));
>        cache_fetch(-name=(server_name + '_https_anon'), -encodenone);
>    else;
>        cache_store(-name=(server_name + '_https_anon'),
>            -content=thisnav_https($nav->renderhtml(-expand)),
>            -expires=60*60*24);
>        cache_fetch(-name=(server_name + '_https_anon'), -encodenone);
>    /if;
>
>Results:
>
>    render thisnav_https anon   3
>    render thisnav_https auth   5
>    render thisnav_http anon    2
>    render subnav_leaves        2
>
>One of the things I discovered was that knop_cache does not work at all like Lasso's [cache].  knop_cache stores knop objects, whereas [cache] stores strings and no other objects.  I tried this:
>
>    knop_cachestore(-type='string');
>
>but was disappointed when I tried to retrieve and display that string.
>
>I also found that [cache] was utterly useless for controlling cache.  I don't know wtf LassoSoft has done with that tag, but make your life easier and just use [cache_*] control tags.  I must have tried a dozen permutations of using plain old [cache] to no avail.
>
>--steve
>
>-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
>Steve Piercy               Web Site Builder
>Soquel, CA
><[hidden email]>                  <http://www.StevePiercy.com/>
>
>
>--
>#############################################################
>This message is sent to you because you are subscribed to
>the mailing list <[hidden email]>.
>To unsubscribe, E-mail to: <[hidden email]>
>Send administrative queries to  <[hidden email]>
>List archive http://lasso.2283332.n4.nabble.com/Knop-Framework-Discussion-f3157831.html
>The Knop project and code is hosted at GitHub.
>https://github.com/knop-project/knop


--
     Johan Sölve
     Web Developer
     Montania System AB
     http://www.montania.se

--
#############################################################
This message is sent to you because you are subscribed to
the mailing list <[hidden email]>.
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
List archive http://lasso.2283332.n4.nabble.com/Knop-Framework-Discussion-f3157831.html
The Knop project and code is hosted at GitHub.
https://github.com/knop-project/knop
Reply | Threaded
Open this post in threaded view
|

Re: knop_nav performance and cache

Johan Solve-2
In reply to this post by stevepiercy
knop_debug implements L-Debug which provides good timers with charts and all built in.


At 15.05 -0800 2014-01-05, Steve Piercy - Web Site Builder wrote:

>As an aside, I don't know whether my code is thread-safe.  I haven't really delved into that, although Knop's util.inc has some examples of how to implement thread safety.
>
>Also to detect timings, I use this for pretty barcharts, timings, and percent of total time.
>http://www.lassosoft.com/tagswap/detail/timer
>
>Do this (usually in cnt_*.inc or your template):
>
>    timer('render thisnav_http anon start');
>    $nav->renderhtml(-expand);
>    timer('render thisnav_http anon end');
>
>And set $debug to true in cfg__global.inc, and use timerstats in the debug block (usually in your template footer).
>
>I also cache the nav objects as they are created in cfg__nav.inc, for example:
>
>    ...
>    if(!knop_cachefetch(-type='knop_nav', -name=$s_user->username));
>        var('nav_admin') = knop_nav;
>            $nav_admin->insert(
>                -key='users',
>                -label='Users',
>                -default='list',
>                -children=$nav_list_edit);
>        ...
>        knop_cachestore(-type='knop_nav', -name=$s_user->username, -expires=$admin_session);
>    /if;
>
>My nav has 101 ->inserts, and 44 nestings with -children.  Yeah, that's nuts.  It's a tree taxonomy.
>
>I might migrate this app to Pyramid or SubstanceD, an application server built using the Pyramid web framework, because both support hierarchical data structures with no speed penalty.
>
>Anyway, to address your comments and question, I'm using 8.6.x, too.  AFAIK, Knop for Lasso 9 uses the same methods to build a nav as Knop for Lasso 8: Yo Dawg, I put a map inside of your map!
>
>There was talk about 5 years of how to create a better data structure.  I've been looking at ZODB in Python as an example, as well as XML, and xs_cat.  However, I can't stomach the concept of trying to jam a tree-like data structure into a relational database; conical peg, square hole.
>
>As far as an addition to the framework, let's see.  I imagine it would need a series of flags:
>
>    $nav->renderhtml(
>        ...
>        -cache='cachename',
>        -expires=seconds,
>        -username=$s_user->username);
>
>Comments?
>
>--steve
>
>
>On 1/5/14 at 5:54 PM, [hidden email] (Ric Lewis) pronounced:
>
>>Hmm, interesting.
>>
>>Whilst not that big, our nav does take a while, we're still on Knop 8, Lasso 8.6.2. (I know there's a fast nav in K9/L9)
>>
>>If the results are so dramatic, looks like it should go into the framework, no?
>>
>>Any changes to nav (ie new options after logging in etc), and the whole thing is re-loaded and thus re-cached anyway.
>>
>>So no downside?
>>
>>
>>I'll try out your technique, thanks for the tip!
>>
>>
>>On 5 Jan 2014, at 13:48, Steve Piercy - Web Site Builder <[hidden email]> wrote:
>>
>>>A really big, complex knop_nav object takes a long time to render.  Time in
>>milliseconds.
>>>
>>>render thisnav_https anon   4517
>>>render thisnav_https auth   4685
>>>render thisnav_http anon    4351
>>>render subnav_leaves        2661
>>>
>>>Bleargh!
>>>
>>>So after experimenting with knop_cache, [cache], and finally [cache_*] tags, I
>>settled on using the latter to render once and cache it using the following sample implementation:
>>>
>>>if(cache_exists(-name=(server_name + '_https_anon')));
>>>cache_fetch(-name=(server_name + '_https_anon'), -encodenone);
>>>else;
>>>cache_store(-name=(server_name + '_https_anon'),
>>>-content=thisnav_https($nav->renderhtml(-expand)),
>>>-expires=60*60*24);
>>>cache_fetch(-name=(server_name + '_https_anon'), -encodenone);
>>>/if;
>>>
>>>Results:
>>>
>>>render thisnav_https anon   3
>>>render thisnav_https auth   5
>>>render thisnav_http anon    2
>>>render subnav_leaves        2
>>>
>>>One of the things I discovered was that knop_cache does not work at all like Lasso's
>>[cache].  knop_cache stores knop objects, whereas [cache] stores strings and no other objects.  I tried this:
>>>
>>>knop_cachestore(-type='string');
>>>
>>>but was disappointed when I tried to retrieve and display that string.
>>>
>>>I also found that [cache] was utterly useless for controlling cache.  I don't know
>>wtf LassoSoft has done with that tag, but make your life easier and just use [cache_*] control tags.  I must have tried a dozen permutations of using plain old [cache] to no avail.
>>>
>>>--steve
>>>
>>>-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
>>>Steve Piercy               Web Site Builder               Soquel, CA
>>><[hidden email]>                  <http://www.StevePiercy.com/>
>>>
>>>
>>>--
>>>#############################################################
>>>This message is sent to you because you are subscribed to
>>>the mailing list <[hidden email]>.
>>>To unsubscribe, E-mail to: <[hidden email]>
>>>Send administrative queries to  <[hidden email]>
>>>List archive
>>http://lasso.2283332.n4.nabble.com/Knop-Framework-Discussion-f3157831.html
>>>The Knop project and code is hosted at GitHub.
>>>https://github.com/knop-project/knop
>>
>>
>
>-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
>Steve Piercy               Web Site Builder
>Soquel, CA
><[hidden email]>                  <http://www.StevePiercy.com/>
>
>
>--
>#############################################################
>This message is sent to you because you are subscribed to
>the mailing list <[hidden email]>.
>To unsubscribe, E-mail to: <[hidden email]>
>Send administrative queries to  <[hidden email]>
>List archive http://lasso.2283332.n4.nabble.com/Knop-Framework-Discussion-f3157831.html
>The Knop project and code is hosted at GitHub.
>https://github.com/knop-project/knop


--
     Johan Sölve
     Web Developer
     Montania System AB
     http://www.montania.se

--
#############################################################
This message is sent to you because you are subscribed to
the mailing list <[hidden email]>.
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
List archive http://lasso.2283332.n4.nabble.com/Knop-Framework-Discussion-f3157831.html
The Knop project and code is hosted at GitHub.
https://github.com/knop-project/knop
Reply | Threaded
Open this post in threaded view
|

Re: knop_nav performance and cache

Johan Solve-2
In reply to this post by Ric Lewis-3
I have quite a few K8 sites running and evolving... Migrating some of them to L9 is under consideration though.


At 08.12 +0000 2014-01-06, Ric Lewis wrote:

>Great that your caching speeds nav html rendering, I think K9/L9 also sped up the whole nav object considerably.
>
>There was talk of a 'lightweight' nav years ago, and I think the essence of that has been implemented in K9.
>
>One of these days I'll have to move to Knop9  but currently stability is pretty good, performance is fine, so hard to prioritise that over new development (and end of year accounts!)
>
>Perhaps we should arrange a mini conference/workshop for all the Knop8 holdouts*, in a few days we could probably get it all ported to K9/L9.
>
>*Mind you that's probably just you and me now :)
>
>
>
>On 6 Jan 2014, at 02:32, Steve Piercy - Web Site Builder <<mailto:[hidden email]>[hidden email]> wrote:
>
>>The biggest hit was from rendering.  Building the nav objects versus rendering was not nearly as much, 788ms vs. 158ms.


--
     Johan Sölve
     Web Developer
     Montania System AB
     http://www.montania.se

--
#############################################################
This message is sent to you because you are subscribed to
the mailing list <[hidden email]>.
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
List archive http://lasso.2283332.n4.nabble.com/Knop-Framework-Discussion-f3157831.html
The Knop project and code is hosted at GitHub.
https://github.com/knop-project/knop
Reply | Threaded
Open this post in threaded view
|

Re: knop_nav performance and cache

Johan Solve-2
In reply to this post by Johan Solve-2
Eer, I mean "as well as caching the actual NAV object"...

At 20.55 +0100 2014-01-15, Johan Solve wrote:

>Caching the rendered nav is a very good idea, as well as caching the actual cache object.
>
>I use knop_cche to cache the nav object and cache_* to cache rendered output (both nav and commonly used page content).
>
>knop_cache should be able to cache page variables of any Lasso type, but using it to cache a common data type as string is probably not a good idea since all page vars of the specified type will be cached, and is not the purpose of it.
>
>This is the core of kncop_cachestore:
>
> // store all page vars of the specified type
> iterate: vars -> keys, local: 'item';
> if: (var: #item) -> isa(#type);
> #data -> insert(#item = (var: #item));
> /if;
> /iterate;
>
>knop_cache should also be thread safe by using thread_locks for reads and writes.
>
>
>To facilitate caching rendered content I use a [capture]...[/capture] container tag. Everything within it is captured instead of output to page.
>
>The capture is returned with [capture_result] and stored with cache_store.
>
>
>Define_Tag('capture', -container,
> -priority='replace',
> -optional='name');
> Var('__capture_result__' + local('name')) = run_children;
>/Define_Tag;
>Define_Tag('capture_result', -optional='xhtml',
> -priority='replace',
> -optional='name');
> if(var_defined('__capture_result__' + local('name')));
> local('result'=var('__capture_result__' + local('name')));
> if(#result -> size);
> if(local_defined('xhtml') && #xhtml != false);
> // fix endslash for xhtml
> #result -> replace('<br>', '<br />') & replace('<br/>', '<br />');
> /if;
> Return(@#result);
> /if;
> /if;
>/Define_Tag;
>
>
>
>This is how I use it, first in a library file (a bit simplified, in reality the cached object is a map with content and timestamp):
>
>var('content'=cache_fetch(-name=$cachename));
>// validate the cache before using it
>if($content -> size);
>
> knop_debug('Using cached data' = $cachename);
>
>else;
> // no valid cache data, perform database queries to create new data
> ...
>/if;
>
>
>Then in a content file:
>
>[if($content -> size);
>
> // use cached content
> knop_debug('Using cached content' = include_currentpath);
>else;
> // create new content
> capture]
>
> Page content with markup and database output
>
> [/capture;
> $content = capture_result;
> // store the result in cache
> knop_debug('Storing content in cache');
> cache_store(-name=$cachename, -content=$output, -expires=$cache_expire);
>/if;
>
>]
>
>
>I use a similar principle when caching the rendered nav.
>
>
>At 05.48 -0800 2014-01-05, Steve Piercy - Web Site Builder wrote:
>>A really big, complex knop_nav object takes a long time to render.  Time in milliseconds.
>>
>>    render thisnav_https anon   4517
>>    render thisnav_https auth   4685
>>    render thisnav_http anon    4351
>>    render subnav_leaves        2661
>>
>>Bleargh!
>>
>>So after experimenting with knop_cache, [cache], and finally [cache_*] tags, I settled on using the latter to render once and cache it using the following sample implementation:
>>
>>    if(cache_exists(-name=(server_name + '_https_anon')));
>>        cache_fetch(-name=(server_name + '_https_anon'), -encodenone);
>>    else;
>>        cache_store(-name=(server_name + '_https_anon'),
>>            -content=thisnav_https($nav->renderhtml(-expand)),
>>            -expires=60*60*24);
>>        cache_fetch(-name=(server_name + '_https_anon'), -encodenone);
>>    /if;
>>
>>Results:
>>
>>    render thisnav_https anon   3
>>    render thisnav_https auth   5
>>    render thisnav_http anon    2
>>    render subnav_leaves        2
>>
>>One of the things I discovered was that knop_cache does not work at all like Lasso's [cache].  knop_cache stores knop objects, whereas [cache] stores strings and no other objects.  I tried this:
>>
>>    knop_cachestore(-type='string');
>>
>>but was disappointed when I tried to retrieve and display that string.
>>
>>I also found that [cache] was utterly useless for controlling cache.  I don't know wtf LassoSoft has done with that tag, but make your life easier and just use [cache_*] control tags.  I must have tried a dozen permutations of using plain old [cache] to no avail.
> >
>>--steve
>>
>>-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
>>Steve Piercy               Web Site Builder
>>Soquel, CA
>><[hidden email]>                  <http://www.StevePiercy.com/>
>>
>>
>>--
>>#############################################################
>>This message is sent to you because you are subscribed to
>>the mailing list <[hidden email]>.
>>To unsubscribe, E-mail to: <[hidden email]>
>>Send administrative queries to  <[hidden email]>
>>List archive http://lasso.2283332.n4.nabble.com/Knop-Framework-Discussion-f3157831.html
>>The Knop project and code is hosted at GitHub.
>>https://github.com/knop-project/knop
>
>
>--
>     Johan Sölve
>     Web Developer
>     Montania System AB
>     http://www.montania.se
>
>--
>#############################################################
>This message is sent to you because you are subscribed to
>the mailing list <[hidden email]>.
>To unsubscribe, E-mail to: <[hidden email]>
>Send administrative queries to  <[hidden email]>
>List archive http://lasso.2283332.n4.nabble.com/Knop-Framework-Discussion-f3157831.html
>The Knop project and code is hosted at GitHub.
>https://github.com/knop-project/knop


--
     Johan Sölve
     Web Developer
     Montania System AB
     http://www.montania.se

--
#############################################################
This message is sent to you because you are subscribed to
the mailing list <[hidden email]>.
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
List archive http://lasso.2283332.n4.nabble.com/Knop-Framework-Discussion-f3157831.html
The Knop project and code is hosted at GitHub.
https://github.com/knop-project/knop
Reply | Threaded
Open this post in threaded view
|

Re: knop_nav performance and cache

stevepiercy
In reply to this post by Johan Solve-2
Nice!  I like that capture technique.

Out of curiosity, why do you implement your cached objects as a map with timestamp?

Do you have a single cachename that is a map object, instead of many cachenames?

--steve


On 1/15/14 at 8:55 PM, [hidden email] (Johan Solve) pronounced:

> Caching the rendered nav is a very good idea, as well as caching the actual cache
> object.
>
> I use knop_cche to cache the nav object and cache_* to cache rendered output (both nav
> and commonly used page content).
>
> knop_cache should be able to cache page variables of any Lasso type, but using it to
> cache a common data type as string is probably not a good idea since all page vars of
> the specified type will be cached, and is not the purpose of it.
>
> This is the core of kncop_cachestore:
>
>   // store all page vars of the specified type
>   iterate: vars -> keys, local: 'item';
>       if: (var: #item) -> isa(#type);
>           #data -> insert(#item = (var: #item));
>       /if;
>   /iterate;
>
> knop_cache should also be thread safe by using thread_locks for reads and writes.
>
>
> To facilitate caching rendered content I use a [capture]...[/capture] container tag.
> Everything within it is captured instead of output to page.
>
> The capture is returned with [capture_result] and stored with cache_store.
>
>
> Define_Tag('capture', -container,
>   -priority='replace',
>   -optional='name');
>   Var('__capture_result__' + local('name')) = run_children;
> /Define_Tag;
> Define_Tag('capture_result', -optional='xhtml',
>   -priority='replace',
>   -optional='name');
>   if(var_defined('__capture_result__' + local('name')));
>       local('result'=var('__capture_result__' + local('name')));
>       if(#result -> size);
>           if(local_defined('xhtml') && #xhtml != false);
>               // fix endslash for xhtml
>               #result -> replace('<br>', '<br />') & replace('<br/>', '<br />');
>           /if;
>           Return(@#result);
>       /if;
>   /if;
> /Define_Tag;
>
>
>
> This is how I use it, first in a library file (a bit simplified, in reality the cached
> object is a map with content and timestamp):
>
> var('content'=cache_fetch(-name=$cachename));
> // validate the cache before using it
> if($content -> size);
>
>   knop_debug('Using cached data' = $cachename);
>
> else;
>   // no valid cache data, perform database queries to create new data
>   ...
> /if;
>
>
> Then in a content file:
>
> [if($content -> size);
>
>   // use cached content
>   knop_debug('Using cached content' = include_currentpath);
> else;
>   // create new content
>   capture]
>
>   Page content with markup and database output
>
>   [/capture;
>   $content = capture_result;
>   // store the result in cache
>   knop_debug('Storing content in cache');
>   cache_store(-name=$cachename, -content=$output, -expires=$cache_expire);
> /if;
>
> ]
>
>
> I use a similar principle when caching the rendered nav.
>
>
> At 05.48 -0800 2014-01-05, Steve Piercy - Web Site Builder wrote:
> >A really big, complex knop_nav object takes a long time to render.  Time in
> milliseconds.
> >
> >    render thisnav_https anon   4517
> >    render thisnav_https auth   4685
> >    render thisnav_http anon    4351
> >    render subnav_leaves        2661
> >
> >Bleargh!
> >
> >So after experimenting with knop_cache, [cache], and finally [cache_*] tags, I settled
> on using the latter to render once and cache it using the following sample
> implementation:
> >
> >    if(cache_exists(-name=(server_name + '_https_anon')));
> >        cache_fetch(-name=(server_name + '_https_anon'), -encodenone);
> >    else;
> >        cache_store(-name=(server_name + '_https_anon'),
> >            -content=thisnav_https($nav->renderhtml(-expand)),
> >            -expires=60*60*24);
> >        cache_fetch(-name=(server_name + '_https_anon'), -encodenone);
> >    /if;
> >
> >Results:
> >
> >    render thisnav_https anon   3
> >    render thisnav_https auth   5
> >    render thisnav_http anon    2
> >    render subnav_leaves        2
> >
> >One of the things I discovered was that knop_cache does not work at all like Lasso's
> [cache].  knop_cache stores knop objects, whereas [cache] stores strings and no other
> objects.  I tried this:
> >
> >    knop_cachestore(-type='string');
> >
> >but was disappointed when I tried to retrieve and display that string.
> >
> >I also found that [cache] was utterly useless for controlling cache.  I don't know wtf
> LassoSoft has done with that tag, but make your life easier and just use [cache_*]
> control tags.  I must have tried a dozen permutations of using plain old [cache] to no
> avail.
> >
> >--steve
> >
> >-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
> >Steve Piercy               Web Site Builder
> >Soquel, CA
> ><[hidden email]>                  <http://www.StevePiercy.com/>
> >
> >
> >--
> >#############################################################
> >This message is sent to you because you are subscribed to
> >the mailing list <[hidden email]>.
> >To unsubscribe, E-mail to: <[hidden email]>
> >Send administrative queries to  <[hidden email]>
> >List archive
> http://lasso.2283332.n4.nabble.com/Knop-Framework-Discussion-f3157831.html
> >The Knop project and code is hosted at GitHub.
> >https://github.com/knop-project/knop
>
>
> --
>      Johan Sölve
>      Web Developer
>      Montania System AB
>      http://www.montania.se
>

-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
Steve Piercy               Web Site Builder               Soquel, CA
<[hidden email]>                  <http://www.StevePiercy.com/>


--
#############################################################
This message is sent to you because you are subscribed to
the mailing list <[hidden email]>.
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
List archive http://lasso.2283332.n4.nabble.com/Knop-Framework-Discussion-f3157831.html
The Knop project and code is hosted at GitHub.
https://github.com/knop-project/knop
Reply | Threaded
Open this post in threaded view
|

Re: knop_nav performance and cache

Johan Solve-2
Many cachenames.

The timestamp in the cached data is used to invalidate the cache against the database. Any updates of the web database by the backend updates a timestamp per table in a global, so relevant caches are automatically invalidated depending on what tables the content depends on (a table dependence list is specified for each lib file) . This happens in the cache validation in the lib.

Here's the real cache validation
// validate the cache before using it

if(cachevalid($cachedata,
                -tables='artiklar,artikelprislista,artikelvarianter')
        && $cachedata -> find('content') -> size);


cahcevalid is a tag that compares the cache's timestamp with the timestamps for the specified tables.


Proper invalidation of caches is key...


At 13.07 -0800 2014-01-15, Steve Piercy - Web Site Builder wrote:

>Nice!  I like that capture technique.
>
>Out of curiosity, why do you implement your cached objects as a map with timestamp?
>
>Do you have a single cachename that is a map object, instead of many cachenames?
>
>--steve
>
>
>On 1/15/14 at 8:55 PM, [hidden email] (Johan Solve) pronounced:
>
>> Caching the rendered nav is a very good idea, as well as caching the actual cache
>> object.
>>
>> I use knop_cche to cache the nav object and cache_* to cache rendered output (both nav
>> and commonly used page content).
>>
>> knop_cache should be able to cache page variables of any Lasso type, but using it to
>> cache a common data type as string is probably not a good idea since all page vars of
>> the specified type will be cached, and is not the purpose of it.
>>
>> This is the core of kncop_cachestore:
>>
>>   // store all page vars of the specified type
>>   iterate: vars -> keys, local: 'item';
>>       if: (var: #item) -> isa(#type);
>>           #data -> insert(#item = (var: #item));
>>       /if;
>>   /iterate;
>>
>> knop_cache should also be thread safe by using thread_locks for reads and writes.
>>
>>
>> To facilitate caching rendered content I use a [capture]...[/capture] container tag.
>> Everything within it is captured instead of output to page.
>>
>> The capture is returned with [capture_result] and stored with cache_store.
>>
>>
>> Define_Tag('capture', -container,
>>   -priority='replace',
>>   -optional='name');
>>   Var('__capture_result__' + local('name')) = run_children;
>> /Define_Tag;
>> Define_Tag('capture_result', -optional='xhtml',
>>   -priority='replace',
>>   -optional='name');
>>   if(var_defined('__capture_result__' + local('name')));
>>       local('result'=var('__capture_result__' + local('name')));
>>       if(#result -> size);
>>           if(local_defined('xhtml') && #xhtml != false);
>>               // fix endslash for xhtml
>>               #result -> replace('<br>', '<br />') & replace('<br/>', '<br />');
>>           /if;
>>           Return(@#result);
>>       /if;
>>   /if;
>> /Define_Tag;
>>
>>
>>
>> This is how I use it, first in a library file (a bit simplified, in reality the cached
>> object is a map with content and timestamp):
>>
>> var('content'=cache_fetch(-name=$cachename));
>> // validate the cache before using it
>> if($content -> size);
>>
>>   knop_debug('Using cached data' = $cachename);
>>
>> else;
>>   // no valid cache data, perform database queries to create new data
>>   ...
>> /if;
>>
>>
>> Then in a content file:
>>
>> [if($content -> size);
>>
>>   // use cached content
>>   knop_debug('Using cached content' = include_currentpath);
>> else;
>>   // create new content
>>   capture]
>>
>>   Page content with markup and database output
>>
>>   [/capture;
>>   $content = capture_result;
>>   // store the result in cache
>>   knop_debug('Storing content in cache');
>>   cache_store(-name=$cachename, -content=$output, -expires=$cache_expire);
>> /if;
>>
>> ]
>>
>>
>> I use a similar principle when caching the rendered nav.
>>
>>
>> At 05.48 -0800 2014-01-05, Steve Piercy - Web Site Builder wrote:
>> >A really big, complex knop_nav object takes a long time to render.  Time in
>> milliseconds.
>> >
>> >    render thisnav_https anon   4517
> > >    render thisnav_https auth   4685
>> >    render thisnav_http anon    4351
>> >    render subnav_leaves        2661
>> >
>> >Bleargh!
>> >
>> >So after experimenting with knop_cache, [cache], and finally [cache_*] tags, I settled
>> on using the latter to render once and cache it using the following sample
>> implementation:
>> >
>> >    if(cache_exists(-name=(server_name + '_https_anon')));
>> >        cache_fetch(-name=(server_name + '_https_anon'), -encodenone);
>> >    else;
>> >        cache_store(-name=(server_name + '_https_anon'),
>> >            -content=thisnav_https($nav->renderhtml(-expand)),
>> >            -expires=60*60*24);
>> >        cache_fetch(-name=(server_name + '_https_anon'), -encodenone);
>> >    /if;
>> >
>> >Results:
>> >
>> >    render thisnav_https anon   3
>> >    render thisnav_https auth   5
>> >    render thisnav_http anon    2
>> >    render subnav_leaves        2
>> >
>> >One of the things I discovered was that knop_cache does not work at all like Lasso's
>> [cache].  knop_cache stores knop objects, whereas [cache] stores strings and no other
>> objects.  I tried this:
>> >
>> >    knop_cachestore(-type='string');
>> >
>> >but was disappointed when I tried to retrieve and display that string.
>> >
>> >I also found that [cache] was utterly useless for controlling cache.  I don't know wtf
>> LassoSoft has done with that tag, but make your life easier and just use [cache_*]
>> control tags.  I must have tried a dozen permutations of using plain old [cache] to no
>> avail.
>> >
>> >--steve
>> >
>> >-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
>> >Steve Piercy               Web Site Builder
>> >Soquel, CA
>> ><[hidden email]>                  <http://www.StevePiercy.com/>
>> >
>> >
>> >--
>> >#############################################################
>> >This message is sent to you because you are subscribed to
>> >the mailing list <[hidden email]>.
>> >To unsubscribe, E-mail to: <[hidden email]>
>> >Send administrative queries to  <[hidden email]>
>> >List archive
>> http://lasso.2283332.n4.nabble.com/Knop-Framework-Discussion-f3157831.html
>> >The Knop project and code is hosted at GitHub.
>> >https://github.com/knop-project/knop
>>
>>
>> --
>>      Johan Sölve
>>      Web Developer
>>      Montania System AB
>>      http://www.montania.se
>>
>
>-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
>Steve Piercy               Web Site Builder               Soquel, CA
><[hidden email]>                  <http://www.StevePiercy.com/>
>
>
>--
>#############################################################
>This message is sent to you because you are subscribed to
>the mailing list <[hidden email]>.
>To unsubscribe, E-mail to: <[hidden email]>
>Send administrative queries to  <[hidden email]>
>List archive http://lasso.2283332.n4.nabble.com/Knop-Framework-Discussion-f3157831.html
>The Knop project and code is hosted at GitHub.
>https://github.com/knop-project/knop


--
     Johan Sölve
     Web Developer
     Montania System AB
     http://www.montania.se

--
#############################################################
This message is sent to you because you are subscribed to
the mailing list <[hidden email]>.
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
List archive http://lasso.2283332.n4.nabble.com/Knop-Framework-Discussion-f3157831.html
The Knop project and code is hosted at GitHub.
https://github.com/knop-project/knop
Reply | Threaded
Open this post in threaded view
|

Re: knop_nav performance and cache

stevepiercy
Ah, I think I see now.  Let me try to paraphrase, and correct me
where I'm wrong.

// cache the object with a timestamp in a map:
var('output' = map(
     'content' = capture_result,
     'timestamp' = date->format('%Q %T'));

cache_store(-name=$cachename,
     -content=$output,
     -expires=$cache_expire);

// check timestamp

if(date_difference(date, date($cachedata->find('timestamp'),
-second) < $cache_expires)
     && date_difference(date,
date($cachedata->find('timestamp'), -second) > 0));
     // Lasso's cache has not expired, yet the timestamp is
recent enough that the cache should be invalidated
     // invalidate the cache, refresh it

--steve


On 1/15/14 at 10:25 PM, [hidden email] (Johan Solve) pronounced:

>Many cachenames.
>
>The timestamp in the cached data is used to invalidate the
>cache against the database. Any updates of the web database by
>the backend updates a timestamp per table in a global, so
>relevant caches are automatically invalidated depending on what
>tables the content depends on (a table dependence list is
>specified for each lib file) . This happens in the cache
>validation in the lib.
>
>Here's the real cache validation
>// validate the cache before using it
>
>if(cachevalid($cachedata,
>-tables='artiklar,artikelprislista,artikelvarianter')
>&& $cachedata -> find('content') -> size);
>
>
>cahcevalid is a tag that compares the cache's timestamp with
>the timestamps for the specified tables.
>
>
>Proper invalidation of caches is key...
>
>
>At 13.07 -0800 2014-01-15, Steve Piercy - Web Site Builder wrote:
>>Nice!  I like that capture technique.
>>
>>Out of curiosity, why do you implement your cached objects as a map with timestamp?
>>
>>Do you have a single cachename that is a map object, instead of many cachenames?
>>
>>--steve
>>
>>
>>On 1/15/14 at 8:55 PM, [hidden email] (Johan Solve) pronounced:
>>
>>> Caching the rendered nav is a very good idea, as well as caching the actual cache
>>> object.
>>>
>>> I use knop_cche to cache the nav object and cache_* to cache rendered output (both
>nav
>>> and commonly used page content).
>>>
>>> knop_cache should be able to cache page variables of any Lasso type, but using it to
>>> cache a common data type as string is probably not a good idea since all page vars
>of
>>> the specified type will be cached, and is not the purpose of it.
>>>
>>> This is the core of kncop_cachestore:
>>>
>>>   // store all page vars of the specified type
>>>   iterate: vars -> keys, local: 'item';
>>>       if: (var: #item) -> isa(#type);
>>>           #data -> insert(#item = (var: #item));
>>>       /if;
>>>   /iterate;
>>>
>>> knop_cache should also be thread safe by using thread_locks for reads and writes.
>>>
>>>
>>> To facilitate caching rendered content I use a [capture]...[/capture] container tag.
>>> Everything within it is captured instead of output to page.
>>>
>>> The capture is returned with [capture_result] and stored with cache_store.
>>>
>>>
>>> Define_Tag('capture', -container,
>>>   -priority='replace',
>>>   -optional='name');
>>>   Var('__capture_result__' + local('name')) = run_children;
>>> /Define_Tag;
>>> Define_Tag('capture_result', -optional='xhtml',
>>>   -priority='replace',
>>>   -optional='name');
>>>   if(var_defined('__capture_result__' + local('name')));
>>>       local('result'=var('__capture_result__' + local('name')));
>>>       if(#result -> size);
>>>           if(local_defined('xhtml') && #xhtml != false);
>>>               // fix endslash for xhtml
>>>               #result -> replace('<br>', '<br />') & replace('<br/>', '<br />');
>>>           /if;
>>>           Return(@#result);
>>>       /if;
>>>   /if;
>>> /Define_Tag;
>>>
>>>
>>>
>>> This is how I use it, first in a library file (a bit simplified, in reality the
>cached
>>> object is a map with content and timestamp):
>>>
>>> var('content'=cache_fetch(-name=$cachename));
>>> // validate the cache before using it
>>> if($content -> size);
>>>
>>>   knop_debug('Using cached data' = $cachename);
>>>
>>> else;
>>>   // no valid cache data, perform database queries to create new data
>>>   ...
>>> /if;
>>>
>>>
>>> Then in a content file:
>>>
>>> [if($content -> size);
>>>
>>>   // use cached content
>>>   knop_debug('Using cached content' = include_currentpath);
>>> else;
>>>   // create new content
>>>   capture]
>>>
>>>   Page content with markup and database output
>>>
>>>   [/capture;
>>>   $content = capture_result;
>>>   // store the result in cache
>>>   knop_debug('Storing content in cache');
>>>   cache_store(-name=$cachename, -content=$output, -expires=$cache_expire);
>>> /if;
>>>
>>> ]
>>>
>>>
>>> I use a similar principle when caching the rendered nav.
>>>
>>>
>>> At 05.48 -0800 2014-01-05, Steve Piercy - Web Site Builder wrote:
>>> >A really big, complex knop_nav object takes a long time to render.  Time in
>>> milliseconds.
>>> >
>>> >    render thisnav_https anon   4517
>>>>render thisnav_https auth   4685
>>> >    render thisnav_http anon    4351
>>> >    render subnav_leaves        2661
>>> >
>>> >Bleargh!
>>> >
>>> >So after experimenting with knop_cache, [cache], and finally [cache_*] tags, I
>settled
>>> on using the latter to render once and cache it using the following sample
>>> implementation:
>>> >
>>> >    if(cache_exists(-name=(server_name + '_https_anon')));
>>> >        cache_fetch(-name=(server_name + '_https_anon'), -encodenone);
>>> >    else;
>>> >        cache_store(-name=(server_name + '_https_anon'),
>>> >            -content=thisnav_https($nav->renderhtml(-expand)),
>>> >            -expires=60*60*24);
>>> >        cache_fetch(-name=(server_name + '_https_anon'), -encodenone);
>>> >    /if;
>>> >
>>> >Results:
>>> >
>>> >    render thisnav_https anon   3
>>> >    render thisnav_https auth   5
>>> >    render thisnav_http anon    2
>>> >    render subnav_leaves        2
>>> >
>>> >One of the things I discovered was that knop_cache does not work at all like
>Lasso's
>>> [cache].  knop_cache stores knop objects, whereas [cache] stores strings and no
>other
>>> objects.  I tried this:
>>> >
>>> >    knop_cachestore(-type='string');
>>> >
>>> >but was disappointed when I tried to retrieve and display that string.
>>> >
>>> >I also found that [cache] was utterly useless for controlling cache.  I don't know
>wtf
>>> LassoSoft has done with that tag, but make your life easier and just use [cache_*]
>>> control tags.  I must have tried a dozen permutations of using plain old [cache] to
>no
>>> avail.
>>> >
>>> >--steve
>>> >
>>> >-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
>>> >Steve Piercy               Web Site Builder
>>> >Soquel, CA
>>> ><[hidden email]>                  <http://www.StevePiercy.com/>
>>> >
>>> >
>>> >--
>>> >#############################################################
>>> >This message is sent to you because you are subscribed to
>>> >the mailing list <[hidden email]>.
>>> >To unsubscribe, E-mail to: <[hidden email]>
>>> >Send administrative queries to  <[hidden email]>
>>> >List archive
>>> http://lasso.2283332.n4.nabble.com/Knop-Framework-Discussion-f3157831.html
>>> >The Knop project and code is hosted at GitHub.
>>> >https://github.com/knop-project/knop
>>>
>>>
>>> --
>>>      Johan Sölve
>>>      Web Developer
>>>      Montania System AB
>>>      http://www.montania.se
>>>
>>
>>-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
>>Steve Piercy               Web Site Builder               Soquel, CA
>><[hidden email]>                  <http://www.StevePiercy.com/>
>>
>>
>>--
>>#############################################################
>>This message is sent to you because you are subscribed to
>>the mailing list <[hidden email]>.
>>To unsubscribe, E-mail to: <[hidden email]>
>>Send administrative queries to  <[hidden email]>
>>List archive
>http://lasso.2283332.n4.nabble.com/Knop-Framework-Discussion-f3157831.html
>>The Knop project and code is hosted at GitHub.
>>https://github.com/knop-project/knop
>
>
>--
>Johan Sölve
>Web Developer
>Montania System AB
>http://www.montania.se
>

-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
-- --
Steve Piercy               Web Site Builder              
Soquel, CA
<[hidden email]>                  <http://www.StevePiercy.com/>


--
#############################################################
This message is sent to you because you are subscribed to
the mailing list <[hidden email]>.
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
List archive http://lasso.2283332.n4.nabble.com/Knop-Framework-Discussion-f3157831.html
The Knop project and code is hosted at GitHub.
https://github.com/knop-project/knop
Reply | Threaded
Open this post in threaded view
|

Re: knop_nav performance and cache

stevepiercy
In reply to this post by stevepiercy
As a comparison, I asked a colleague to run a timing of a complex navigational hierarchy that had about 1500 objects as deep as 4 levels. It was using SubstanceD on top of Pyramid with ZODB as persistent storage.

The timing to build and render all those objects was ~470ms without caching.

My nav object in Knop has about 100 objects, is 8 levels deep, and takes ~4700ms to render *alone*.

The difference in speed is a compelling argument to try some new stuff.