Lasso-based file synching?

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Lasso-based file synching?

Robert Carpenter
Hey all-

L7/OS X

I'm writing a lightweight CMS tool for a couple of my sites, mostly just
adding text content to a set of mysql db's, but it also has a component to
upload images to the filesystem.

The problem is that I'd very much prefer to have users access the tool on
our in-house development server only, and I'm wondering how I can, using
Lasso, have those image files elegantly sych up to the production box as
part of the process.

I could just cron job an rsynch of the image dirs, but I'd love to have it
function in real-time (or close to it), and I'd also like to keep the tool
Lasso-based for simplicity and portability.

Anybody doing anything like this? Seems like the Net tags or include_url
with post params might be good candidates, wondering if anyone has any
wisdom to share before I roll up my sleeves.

Thanks in advance!
-Robert-



--
------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage
Reply | Threaded
Open this post in threaded view
|

Re: Lasso-based file synching?

Marc Pope-3
I do something similar, but not with lasso. I use the unix rsync command. We
have multiple web servers load balanced and we upload to one main server and
then it replicates out. It's EXTREMELY fast. We have it check every minute.

Here's a great explanation of how to set this up on OSX:
http://www.macosxhints.com/article.php?story=20031024013757927

This might also help:
http://rsyncbackup.erlang.no/

Email me if you have questions. It's easier than it looks.

Marc


On 10/26/05 8:49 PM, "Robert Carpenter" <[hidden email]> wrote:

> Hey all-
>
> L7/OS X
>
> I'm writing a lightweight CMS tool for a couple of my sites, mostly just
> adding text content to a set of mysql db's, but it also has a component to
> upload images to the filesystem.
>
> The problem is that I'd very much prefer to have users access the tool on
> our in-house development server only, and I'm wondering how I can, using
> Lasso, have those image files elegantly sych up to the production box as
> part of the process.
>
> I could just cron job an rsynch of the image dirs, but I'd love to have it
> function in real-time (or close to it), and I'd also like to keep the tool
> Lasso-based for simplicity and portability.
>
> Anybody doing anything like this? Seems like the Net tags or include_url
> with post params might be good candidates, wondering if anyone has any
> wisdom to share before I roll up my sleeves.
>
> Thanks in advance!
> -Robert-
>
>



--
------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage
Reply | Threaded
Open this post in threaded view
|

Re: Lasso-based file synching?

Rick Draper
In reply to this post by Robert Carpenter
Hi All,

Is anyone doing this in a windoze environment?  This could be really
handy for us.


Very best regards,
 
 
Rick

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On
Behalf Of Marc Pope
Sent: Thursday, 27 October 2005 11:34 AM
To: [hidden email]
Subject: Re: Lasso-based file synching?

I do something similar, but not with lasso. I use the unix rsync
command. We have multiple web servers load balanced and we upload to one
main server and then it replicates out. It's EXTREMELY fast. We have it
check every minute.

Here's a great explanation of how to set this up on OSX:
http://www.macosxhints.com/article.php?story=20031024013757927

This might also help:
http://rsyncbackup.erlang.no/

Email me if you have questions. It's easier than it looks.

Marc


On 10/26/05 8:49 PM, "Robert Carpenter" <[hidden email]> wrote:

> Hey all-
>
> L7/OS X
>
> I'm writing a lightweight CMS tool for a couple of my sites, mostly
> just adding text content to a set of mysql db's, but it also has a
> component to upload images to the filesystem.
>
> The problem is that I'd very much prefer to have users access the tool

> on our in-house development server only, and I'm wondering how I can,
> using Lasso, have those image files elegantly sych up to the
> production box as part of the process.
>
> I could just cron job an rsynch of the image dirs, but I'd love to
> have it function in real-time (or close to it), and I'd also like to
> keep the tool Lasso-based for simplicity and portability.
>
> Anybody doing anything like this? Seems like the Net tags or
> include_url with post params might be good candidates, wondering if
> anyone has any wisdom to share before I roll up my sleeves.
>
> Thanks in advance!
> -Robert-
>
>



--
------------------------------
Lasso Support: http://support.omnipilot.com/ Search the list archives:
http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage


--
------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage
Reply | Threaded
Open this post in threaded view
|

Re: Lasso-based file synching?

Peter D Bethke
In reply to this post by Robert Carpenter
I've not done it myself, but you might consider synching your two  
MySQL servers in a master-slave configuration, and storing the images  
in a binary format, if the image issue is what you are after. Then  
with MySQL handling the synching, you would have two accessible  
sources for your binary image data.

Maybe someone who is more versant in advanced MySQL (like Olivier for  
example) might be able to speak to the technical possiblity. For  
myself I've had good success in storing images in MySQL and it would  
be good to draw on it's built in synching abilities to help your set-up.

Peter

On Oct 26, 2005, at 8:49 PM, Robert Carpenter wrote:

> Hey all-
>
> L7/OS X
>
> I'm writing a lightweight CMS tool for a couple of my sites, mostly  
> just
> adding text content to a set of mysql db's, but it also has a  
> component to
> upload images to the filesystem.
>
> The problem is that I'd very much prefer to have users access the  
> tool on
> our in-house development server only, and I'm wondering how I can,  
> using
> Lasso, have those image files elegantly sych up to the production  
> box as
> part of the process.
>
> I could just cron job an rsynch of the image dirs, but I'd love to  
> have it
> function in real-time (or close to it), and I'd also like to keep  
> the tool
> Lasso-based for simplicity and portability.
>
> Anybody doing anything like this? Seems like the Net tags or  
> include_url
> with post params might be good candidates, wondering if anyone has any
> wisdom to share before I roll up my sleeves.
>
> Thanks in advance!
> -Robert-
>
>
>
> --
> ------------------------------
> Lasso Support: http://support.omnipilot.com/
> Search the list archives: http://www.listsearch.com/lassotalk.lasso
> Manage your list subscription:
> http://www.listsearch.com/lassotalk.lasso?manage
>


--
------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage
Reply | Threaded
Open this post in threaded view
|

Re: Lasso-based file synching?

m i l e s-4
In reply to this post by Robert Carpenter
Robert,

Not that I would dispute my esteemed colleague Peter's word on this  
one...but storing images in a SQL database isn't what I'd call a real  
solution here.  Perhaps its my hold over from having it drilled into  
my head from my FMP days...no never nyet upon penalty of DEATH.  
Besides it bloats the Tables when it doesn't need to.  Store your  
images locally and let the db do its thang...serve data!  That  
said...Peter's got an idea there.  It would solve the immediate  
problem if you're needing just the images.  Ive done the master/slave  
thing (no jokes Steffan or Steve!) and it works like a chhhhAmmm!  
The set up isn't that hard.  But I'll defer to Olivier if he chooses  
to chime in on this one.  Took me about 3 hrs to get it right and  
once I got it set up I completely forgot that it was running.  
However if I suspect (as is true of all CMS's, after having built  
them for the last 7 years), its not just the images you'll want, its  
everything.  So I'd go with Marc's suggestion there...rsync!

M i l e s.

--
------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage
Reply | Threaded
Open this post in threaded view
|

Re: Lasso-based file synching?

Robert Carpenter
In reply to this post by Robert Carpenter
Thanks, Mark. I'm already using rsync in other situations (and it is great
stuff!), but I'd like to be able to keep this Lasso only if I can. If Lasso
doesn't pan out, I'll fall back to rsync, but seems like there *should* be a
way to sync file system objects from one Lasso server to another.


> I do something similar, but not with lasso. I use the unix rsync command. We
> have multiple web servers load balanced and we upload to one main server and
> then it replicates out. It's EXTREMELY fast. We have it check every minute.

>>
>> L7/OS X
>>
>> I'm writing a lightweight CMS tool for a couple of my sites, mostly just
>> adding text content to a set of mysql db's, but it also has a component to
>> upload images to the filesystem.
>>
>> The problem is that I'd very much prefer to have users access the tool on
>> our in-house development server only, and I'm wondering how I can, using
>> Lasso, have those image files elegantly sych up to the production box as
>> part of the process.
>>
>> I could just cron job an rsynch of the image dirs, but I'd love to have it
>> function in real-time (or close to it), and I'd also like to keep the tool
>> Lasso-based for simplicity and portability.
>>
>> Anybody doing anything like this? Seems like the Net tags or include_url
>> with post params might be good candidates, wondering if anyone has any
>> wisdom to share before I roll up my sleeves.
>>
>> Thanks in advance!
>> -Robert-
>>
>>
>
>
>
> --
> ------------------------------
> Lasso Support: http://support.omnipilot.com/
> Search the list archives: http://www.listsearch.com/lassotalk.lasso
> Manage your list subscription:
> http://www.listsearch.com/lassotalk.lasso?manage



--
------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage
Reply | Threaded
Open this post in threaded view
|

Re: Lasso-based file synching?

Robert Carpenter
In reply to this post by Robert Carpenter
Thanks, Peter.

So are you serving your files directly from MySQL blob fields? How is the
performance, compared to serving from the file system? Have your tables
bloated out as a result?

The images I'll be dealing with should be small (using image tags to verify
size at upload), but eventually there will be several hundred or more -
wondering how this would scale.

It occurs to me that I might be able to store the images temporarily in
MySQL, and use routines on both my dev & production servers to write from
there to the FS,  but seems that's getting a little Rube Goldberg-ian.

-rbt-


 

> I've not done it myself, but you might consider synching your two
> MySQL servers in a master-slave configuration, and storing the images
> in a binary format, if the image issue is what you are after. Then
> with MySQL handling the synching, you would have two accessible
> sources for your binary image data.
> For myself I've had good success in storing images in MySQL and it would
> be good to draw on it's built in synching abilities to help your set-up.
>
> Peter
>
> On Oct 26, 2005, at 8:49 PM, Robert Carpenter wrote:
>
>> Hey all-
>>
>> L7/OS X
>>
>> I'm writing a lightweight CMS tool for a couple of my sites, mostly
>> just
>> adding text content to a set of mysql db's, but it also has a
>> component to
>> upload images to the filesystem.
>>
>> The problem is that I'd very much prefer to have users access the
>> tool on
>> our in-house development server only, and I'm wondering how I can,
>> using
>> Lasso, have those image files elegantly sych up to the production
>> box as
>> part of the process.
>>
>> I could just cron job an rsynch of the image dirs, but I'd love to
>> have it
>> function in real-time (or close to it), and I'd also like to keep
>> the tool
>> Lasso-based for simplicity and portability.
>>
>> Anybody doing anything like this? Seems like the Net tags or
>> include_url
>> with post params might be good candidates, wondering if anyone has any
>> wisdom to share before I roll up my sleeves.
>>
>> Thanks in advance!
>> -Robert-
>>
>>
>>
>> --
>> ------------------------------
>> Lasso Support: http://support.omnipilot.com/
>> Search the list archives: http://www.listsearch.com/lassotalk.lasso
>> Manage your list subscription:
>> http://www.listsearch.com/lassotalk.lasso?manage
>>
>
>
> --
> ------------------------------
> Lasso Support: http://support.omnipilot.com/
> Search the list archives: http://www.listsearch.com/lassotalk.lasso
> Manage your list subscription:
> http://www.listsearch.com/lassotalk.lasso?manage



--
------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage
Reply | Threaded
Open this post in threaded view
|

Re: Lasso-based file synching?

Peter D Bethke
In reply to this post by Robert Carpenter
Robert,

I've become very fond of storing images in tables, for a number of  
reasons.

1) it makes portability much easier - exporting SQL from the db to  
rebuild includes the images along with it.
2) (My Main Reason) Access control. If an image is a record then it  
can be subject to any mysql query to determine whether a user can  
view it. I'll be talking more on this at the Summit.
3) Performance can be greatly enhanced through the use of the MySQL  
query cache if the images are on the smaller size. What I do is to  
use mysql longtext fields to store the images as serialized, encoded  
text. Since it's text it benefits from the query cache if it falls  
below the query cache limit, which you can adjust. Lately I've even  
been storing the images in a serialized variable Ctype that contains  
instance vars of different file sizes, ie preview images, along with  
static data like height/width etc so I can avoid using the image tags  
to grab that. It works something like $myimage->small_preview outputs  
an image type instance variable which I then present to the user  
using the images tag.

I understand Mile's point but I think, as he admits, that the belief  
that storing images in mysql tables is bad is more a product of the  
fmp era. Using current lasso techniques and leveraging mysql's built-
in abilities goes a long way to evening the playing field. If the  
table gets too big, you might consider using multiple smaller tables.  
There are always ways.

I use mysql tables to serve ad banners, which are notorious  
bottlenecks when it comes to speed. I find it works great. And hey,  
it sure is nice to store click counts in the same record as the  
actual image data. Hate to say it, but imbedded images are uses a  
whole lot in other languages -- its just a matter of proper  
implementation.

Peter




On Oct 27, 2005, at 3:42 PM, Robert Carpenter wrote:

> Thanks, Peter.
>
> So are you serving your files directly from MySQL blob fields? How  
> is the
> performance, compared to serving from the file system? Have your  
> tables
> bloated out as a result?
>
> The images I'll be dealing with should be small (using image tags  
> to verify
> size at upload), but eventually there will be several hundred or  
> more -
> wondering how this would scale.
>
> It occurs to me that I might be able to store the images  
> temporarily in
> MySQL, and use routines on both my dev & production servers to  
> write from
> there to the FS,  but seems that's getting a little Rube Goldberg-ian.
>
> -rbt-


--
------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage
Reply | Threaded
Open this post in threaded view
|

Re: Lasso-based file synching?

Olivier Miossec
In reply to this post by Robert Carpenter
> I've not done it myself, but you might consider synching your two
> MySQL servers in a master-slave configuration, and storing the images
> in a binary format, if the image issue is what you are after. Then
> with MySQL handling the synching, you would have two accessible
> sources for your binary image data.
>
> Maybe someone who is more versant in advanced MySQL (like Olivier for
> example) might be able to speak to the technical possiblity. For
> myself I've had good success in storing images in MySQL and it would
> be good to draw on it's built in synching abilities to help your
> set-up.
>
> Peter
>

Storing a large amount of binary data into a Mysql Master/Slave work but are
 and can be really efficient in some condition.

First there is a physical limitation. MyIsam table store data in one files
and file size are limited by the file system (for example 2Go on some unix
flavour, 64 Go on some windows ....)

The other limitation is the network. You will need, at least 2 network card,
one card for Master/Slave operation, and an other for other operation.

There are also some optimisation tip. For example create a table that will
only store the images and an other that will store other elements (Author,
size, data ....) In this situation Mysql will only use  the image table to
server image.

But you can have better perfomence with some other storage engine. MyIsam
and Innodb storage engine is not made to store large ammount of data they
are made to make online transaction. So you may have to choose an other
database server after some year.
There is 2 that can do it
Sap/DB MAX DB, free and have a really good storage mode to store large data
Oracle Expensive but it's the best solution for the moment
You can also check Sql Server 2000/2005.


---
Olivier Miossec
AIM/ichat omiossec
--
http://www.lasso-developpeur.net
--

Need a Lasso consultant ?
(Lasso, Java, Databases Consultant ...)
http://www.lasso-developpeur.net/oliviermiossec/

--




--
------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage