Lots of Cache's

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Lots of Cache's

Daniel Logue
Hi list,

I'm a big fan of the Cache tag, and I'm considering using it to cache
the HTML result on thousands of pages.  This means that the cache DB in
SQLite would grow to probably around 300,000 to 500,000 records
eventually...

So are the cache tags really going to give me a performance boost at
that volume?  Can the internal SQLite DB be happy with a relatively
large table?

Thanks
Daniel

============================================
Attend the Lasso Summit
March 2-4, 2007 in Fort Lauderdale, FL
http://www.LassoSummit.com/
============================================

------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage
Reply | Threaded
Open this post in threaded view
|

Re: Lots of Cache's

Fletcher Sandbeck
> I'm a big fan of the Cache tag, and I'm considering using it to cache
> the HTML result on thousands of pages.  This means that the cache DB
> in SQLite would grow to probably around 300,000 to 500,000 records
> eventually...
>
> So are the cache tags really going to give me a performance boost at
> that volume?  Can the internal SQLite DB be happy with a relatively
> large table?

The cache tags are memory-resident unless the -Session option is used.
You will essentially be storing all of your HTML pages in RAM and
serving them from there rather than recreating them using database
calls.

Whether this will be a win depends on a few factors.  Do you have enough
RAM to store the cached pages in memory?  Does the time it takes to
retrieve a cache compare favorably with the time it takes to generate a
page?  How often is the server restarted, forcing the cache to be
regenerated?

My feeling is that hundreds of thousands of records is overkill for the
cache tags.  With that number of records you are probably better off
implementing a custom cache scheme that can take the particular details
of your site into account.

[fletcher]
--
Fletcher Sandbeck                         [hidden email]
Director of Product Development       http://www.lassostudio.com
OmniPilot Software, Inc.                http://www.omnipilot.com

============================================
Attend the Lasso Summit
March 2-4, 2007 in Fort Lauderdale, FL
http://www.LassoSummit.com/
============================================

------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage
Reply | Threaded
Open this post in threaded view
|

Re: Lots of Cache's

Marc Pope
In reply to this post by Daniel Logue
Daniel:

I did this originally for a site and it kept crashing the server. One
problem is even though the cache expires, the cached file still seems to
reside in the server memory, never getting dumped. So a script is needed to
remove those old ones.

The server in question had a problem where one news story would all of the
sudden get 1000s of pages views per minute. So I built in a counter
mechanism that if a story/page got requested more than 10 times in the last
minute, I would cache the story until the traffic went down.

Eventually, I did it my own way...actually save generated pages, you can use
scripts to pull the page into a variable and write out that file as a real
html file. You could do this with just part of the page if you wanted to.
E.g. Make a /content/ directory and store the generated content. If the page
does not exist, generate one on the fly and save it. You can then run
scripts in the unix shell to delete files older than X days or whatever.

Another thing, is it possible that you don't have the SQL queries optimized
as good as they can be? Building a cache system like this is generally only
needed for very very busy sites.

Hope that gives you some help.

Sincerely,

Marc Pope

 




On 2/21/07 1:21 PM, "Daniel Logue" <[hidden email]> wrote:

> Hi list,
>
> I'm a big fan of the Cache tag, and I'm considering using it to cache
> the HTML result on thousands of pages.  This means that the cache DB in
> SQLite would grow to probably around 300,000 to 500,000 records
> eventually...
>
> So are the cache tags really going to give me a performance boost at
> that volume?  Can the internal SQLite DB be happy with a relatively
> large table?
>
> Thanks
> Daniel



============================================
Attend the Lasso Summit
March 2-4, 2007 in Fort Lauderdale, FL
http://www.LassoSummit.com/
============================================

------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage
Reply | Threaded
Open this post in threaded view
|

Re: Lots of Cache's

Daniel Logue
In reply to this post by Fletcher Sandbeck
Thanks Fletcher,

I don't know how I got it in my head that the cached content was stored
in a SQLite db.  I guess I'm just making things up now.

I probably do not have enough memory to handle as much content as I
described so I will pursue a custom solution.

Thanks again.

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On
Behalf Of Fletcher Sandbeck
Sent: Wednesday, February 21, 2007 10:37 AM
To: [hidden email]
Subject: Re: Lots of Cache's

> I'm a big fan of the Cache tag, and I'm considering using it to cache
> the HTML result on thousands of pages.  This means that the cache DB
> in SQLite would grow to probably around 300,000 to 500,000 records
> eventually...
>
> So are the cache tags really going to give me a performance boost at
> that volume?  Can the internal SQLite DB be happy with a relatively
> large table?

The cache tags are memory-resident unless the -Session option is used.
You will essentially be storing all of your HTML pages in RAM and
serving them from there rather than recreating them using database
calls.

Whether this will be a win depends on a few factors.  Do you have enough
RAM to store the cached pages in memory?  Does the time it takes to
retrieve a cache compare favorably with the time it takes to generate a
page?  How often is the server restarted, forcing the cache to be
regenerated?

My feeling is that hundreds of thousands of records is overkill for the
cache tags.  With that number of records you are probably better off
implementing a custom cache scheme that can take the particular details
of your site into account.

[fletcher]
--
Fletcher Sandbeck                         [hidden email]
Director of Product Development       http://www.lassostudio.com
OmniPilot Software, Inc.                http://www.omnipilot.com

============================================
Attend the Lasso Summit
March 2-4, 2007 in Fort Lauderdale, FL
http://www.LassoSummit.com/
============================================

------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage

============================================
Attend the Lasso Summit
March 2-4, 2007 in Fort Lauderdale, FL
http://www.LassoSummit.com/
============================================

------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage
Reply | Threaded
Open this post in threaded view
|

Re: Lots of Cache's

Daniel Logue
In reply to this post by Marc Pope
Good tip.

I'm planning for the future here.  The SQL runs quick, and the lasso
code seems very efficient.  But I had a lesson in how this stuff really
compounds when we had our site on the front pages of del.icio.us and
digg on the same day.

Code that seems lightweight enough can suddenly put a serious load on
your server(s) when enough people show up to run it.


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On
Behalf Of Marc Pope
Sent: Wednesday, February 21, 2007 11:01 AM
To: [hidden email]
Subject: Re: Lots of Cache's

Daniel:

I did this originally for a site and it kept crashing the server. One
problem is even though the cache expires, the cached file still seems to
reside in the server memory, never getting dumped. So a script is needed
to
remove those old ones.

The server in question had a problem where one news story would all of
the
sudden get 1000s of pages views per minute. So I built in a counter
mechanism that if a story/page got requested more than 10 times in the
last
minute, I would cache the story until the traffic went down.

Eventually, I did it my own way...actually save generated pages, you can
use
scripts to pull the page into a variable and write out that file as a
real
html file. You could do this with just part of the page if you wanted
to.
E.g. Make a /content/ directory and store the generated content. If the
page
does not exist, generate one on the fly and save it. You can then run
scripts in the unix shell to delete files older than X days or whatever.

Another thing, is it possible that you don't have the SQL queries
optimized
as good as they can be? Building a cache system like this is generally
only
needed for very very busy sites.

Hope that gives you some help.

Sincerely,

Marc Pope

 




On 2/21/07 1:21 PM, "Daniel Logue" <[hidden email]> wrote:

> Hi list,
>
> I'm a big fan of the Cache tag, and I'm considering using it to cache
> the HTML result on thousands of pages.  This means that the cache DB
in
> SQLite would grow to probably around 300,000 to 500,000 records
> eventually...
>
> So are the cache tags really going to give me a performance boost at
> that volume?  Can the internal SQLite DB be happy with a relatively
> large table?
>
> Thanks
> Daniel



============================================
Attend the Lasso Summit
March 2-4, 2007 in Fort Lauderdale, FL
http://www.LassoSummit.com/
============================================

------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage

============================================
Attend the Lasso Summit
March 2-4, 2007 in Fort Lauderdale, FL
http://www.LassoSummit.com/
============================================

------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage