sending mail = 100 percent cpu usage

classic Classic list List threaded Threaded
34 messages Options
12
Reply | Threaded
Open this post in threaded view
|

sending mail = 100 percent cpu usage

Dan McComb
I do a weekly bulk mailing in which about 600 emails are sent out  
from a membership database. Everything works fine, but it takes a  
LOOOOONG time to send (I had to disable the script execution timeout  
for it to work without timing out partway through the mailing). But  
what's most weird, is that the CPU jumps to 100% during the mailing  
(G5 xserve). Here's the code I'm using to the do the mailing. Is  
there a way for me to streamline this so it doesn't hog all the CPU?

/dan

[var:'query'='select state, first_name, last_name, email, id,  
what_i_do, id_name, image_url from members where account_status =  
\'active\' AND published_status = \'yes\' AND weekly_emails = \'yes\';']
[Inline: $mydb,
        -Search,
        -SQL=$query,
        -MaxRecords=All]
[records]

[Email_Send:
               
                -Host='localhost',
               
                -Username=$db_username,
               
                -Password=$db_password,

                -To=(field:'email'),

                -From='[hidden email]',

                -Subject='Weekly Biznik Stats and Update for '(field:'first_name')'  
'(field:'last_name'),

                -HTML=(include:'weekly_mailing_body.html'),
               
                -Body='Your weekly Biznik stats and update are ready for your  
review at http://www.biznik.com/members/weekly_stats.html']
               
[/records]
[/inline]


------------------------------------------------------------------------
------------

Visual Contact
4318 5th Ave. NW
Seattle, WA 98107
206.228.0780

[hidden email]
http://www.visualcontact.com


------------------------------
If you missed the Summit, you can still learn from the best at a Lasso Master Class near you!
http://www.omnipilot.com/id.lasso?tid=91

------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage
Reply | Threaded
Open this post in threaded view
|

Re: sending mail = 100 percent cpu usage

decorior
you should be able to control the maximum # queues and send # of  
messages per connection to spread these out over a longer time.

Although 600 does not seem like a lot.

Does your html page that you include have a lot of processing or is  
it big? (e.g. big images).

Also I would make sure your query is optimized (right indexes).

I guess, I would try to figure out what is hogging the processor  
before I tried anything :-)

Deco
On Apr 10, 2006, at 5:31 PM, Dan McComb wrote:

> I do a weekly bulk mailing in which about 600 emails are sent out  
> from a membership database. Everything works fine, but it takes a  
> LOOOOONG time to send (I had to disable the script execution  
> timeout for it to work without timing out partway through the  
> mailing). But what's most weird, is that the CPU jumps to 100%  
> during the mailing (G5 xserve). Here's the code I'm using to the do  
> the mailing. Is there a way for me to streamline this so it doesn't  
> hog all the CPU?
>
> /dan
>
> [var:'query'='select state, first_name, last_name, email, id,  
> what_i_do, id_name, image_url from members where account_status =  
> \'active\' AND published_status = \'yes\' AND weekly_emails = \'yes
> \';']
> [Inline: $mydb,
> -Search,
> -SQL=$query,
> -MaxRecords=All]
> [records]
>
> [Email_Send:
>
> -Host='localhost',
>
> -Username=$db_username,
>
> -Password=$db_password,
>
> -To=(field:'email'),
>
> -From='[hidden email]',
>
> -Subject='Weekly Biznik Stats and Update for  
> '(field:'first_name')' '(field:'last_name'),
>
> -HTML=(include:'weekly_mailing_body.html'),
>
> -Body='Your weekly Biznik stats and update are ready for your  
> review at http://www.biznik.com/members/weekly_stats.html']
>
> [/records]
> [/inline]
>
>
> ----------------------------------------------------------------------
> --------------
>
> Visual Contact
> 4318 5th Ave. NW
> Seattle, WA 98107
> 206.228.0780
>
> [hidden email]
> http://www.visualcontact.com
>
>
> ------------------------------
> If you missed the Summit, you can still learn from the best at a  
> Lasso Master Class near you!
> http://www.omnipilot.com/id.lasso?tid=91
>
> ------------------------------
> Lasso Support: http://support.omnipilot.com/
> Search the list archives: http://www.listsearch.com/lassotalk.lasso
> Manage your list subscription: http://www.listsearch.com/ 
> lassotalk.lasso?manage


------------------------------
If you missed the Summit, you can still learn from the best at a Lasso Master Class near you!
http://www.omnipilot.com/id.lasso?tid=91

------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage
Reply | Threaded
Open this post in threaded view
|

Re: sending mail = 100 percent cpu usage

Alex Pilson
In reply to this post by Dan McComb
At 4:31 PM -0700 4/10/06, Dan McComb wrote:
>I do a weekly bulk mailing in which about 600 emails are sent out
>from a membership database. Everything works fine, but it takes a
>LOOOOONG time to send (I had to disable the script execution timeout
>for it to work without timing out partway through the mailing). But
>what's most weird, is that the CPU jumps to 100% during the mailing
>(G5 xserve). Here's the code I'm using to the do the mailing. Is
>there a way for me to streamline this so it doesn't hog all the CPU?

That is mostly my experience as well. Especially with a busy server.
Try sending over 30,000...near impossible. The last one I did, I had
to restart the "mailing process" 3 times from where it left off to
complete the mailing. I have since given up on the prospect of a
Lasso driven mass emailer.

--
<------------------------------------------------------------------------------>
     Alex Pilson          FlagShip Hosting, LLC
     800.592.6781     <http://www.flagshiphosting.com>
<------------------------------------------------------------------------------>

------------------------------
If you missed the Summit, you can still learn from the best at a Lasso Master Class near you!
http://www.omnipilot.com/id.lasso?tid=91

------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage
Reply | Threaded
Open this post in threaded view
|

Re: sending mail = 100 percent cpu usage

Viaduct Productions
In reply to this post by Dan McComb
What you can do is optimize your emails.

Instead of sending one email per person, you can bcc 100 people and
stagger your lasso server's refresh.  Your BCC parameter can be passed
to the mail server, which, if a good mail server, will optimize
connectivity based upon domain name targets.  It reduces your outgoing
emails from lasso by 100.  I'm just using 100 as an example.

Dan McComb wrote:

> I do a weekly bulk mailing in which about 600 emails are sent out from
> a membership database. Everything works fine, but it takes a LOOOOONG
> time to send (I had to disable the script execution timeout for it to
> work without timing out partway through the mailing). But what's most
> weird, is that the CPU jumps to 100% during the mailing (G5 xserve).
> Here's the code I'm using to the do the mailing. Is there a way for me
> to streamline this so it doesn't hog all the CPU?
>
> /dan
>
> [var:'query'='select state, first_name, last_name, email, id,
> what_i_do, id_name, image_url from members where account_status =
> \'active\' AND published_status = \'yes\' AND weekly_emails = \'yes\';']
> [Inline: $mydb,
>     -Search,
>     -SQL=$query,
>     -MaxRecords=All]
> [records]
>
> [Email_Send:
>        
>         -Host='localhost',
>        
>         -Username=$db_username,
>        
>         -Password=$db_password,
>
>         -To=(field:'email'),
>
>         -From='[hidden email]',
>
>         -Subject='Weekly Biznik Stats and Update for
> '(field:'first_name')' '(field:'last_name'),
>
>         -HTML=(include:'weekly_mailing_body.html'),
>        
>         -Body='Your weekly Biznik stats and update are ready for your
> review at http://www.biznik.com/members/weekly_stats.html']
>        
> [/records]
> [/inline]
>
>
> ------------------------------------------------------------------------------------
>
>
> Visual Contact
> 4318 5th Ave. NW
> Seattle, WA 98107
> 206.228.0780
>
> [hidden email]
> http://www.visualcontact.com
>
>
> ------------------------------
> If you missed the Summit, you can still learn from the best at a Lasso
> Master Class near you!
> http://www.omnipilot.com/id.lasso?tid=91
>
> ------------------------------
> Lasso Support: http://support.omnipilot.com/
> Search the list archives: http://www.listsearch.com/lassotalk.lasso
> Manage your list subscription:
> http://www.listsearch.com/lassotalk.lasso?manage
>

------------------------------
If you missed the Summit, you can still learn from the best at a Lasso Master Class near you!
http://www.omnipilot.com/id.lasso?tid=91

------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage
Reply | Threaded
Open this post in threaded view
|

Re: sending mail = 100 percent cpu usage

charles brisson
In reply to this post by Dan McComb
Alex,
I'm facing the same problem. Trying to build something that might have
to send thousands of emails at once. I clogged my Lasso Hosting service
last night and I am still receiving emails tonight. I need all the
emails to be sent at once and delivered at once. OK, sure, I could stand
a few minutes delay but not 30 hours.

So, my question to you is what are you using now for mass email-sending?

Thanks

Charles

Alex Pilson wrote:

> At 4:31 PM -0700 4/10/06, Dan McComb wrote:
>
>> I do a weekly bulk mailing in which about 600 emails are sent out
>> from a membership database. Everything works fine, but it takes a
>> LOOOOONG time to send (I had to disable the script execution timeout
>> for it to work without timing out partway through the mailing). But
>> what's most weird, is that the CPU jumps to 100% during the mailing
>> (G5 xserve). Here's the code I'm using to the do the mailing. Is
>> there a way for me to streamline this so it doesn't hog all the CPU?
>
>
> That is mostly my experience as well. Especially with a busy server.
> Try sending over 30,000...near impossible. The last one I did, I had
> to restart the "mailing process" 3 times from where it left off to
> complete the mailing. I have since given up on the prospect of a Lasso
> driven mass emailer.
>

------------------------------
If you missed the Summit, you can still learn from the best at a Lasso Master Class near you!
http://www.omnipilot.com/id.lasso?tid=91

------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage
Reply | Threaded
Open this post in threaded view
|

Re: sending mail = 100 percent cpu usage

decorior
In reply to this post by Dan McComb
It seems like a general problem!

Although 600 emails is not that much to send, and easily broken down.

However, sending 30,000 emails is another problem!

The first problem is finding a hosting service that does not mind one  
of its servers sending 30,000 emails.

Charles, is your hosting service doing any throttling?

Two other approaches...

1. Push the email send off to a secondary mail server...that way it  
does not clog things up on the main site. (We use event_schedule to  
launch a job on another server).
2. We are investigating using a mailing service like Campaign monitor  
or other service with a decent api (we already use vertical  
response). Our customers are willing to pay for the "permission"  
marketing. We use lasso to "tighten" the list, etc.

ONE BIG QUESTION

The -body= (include: etc..) tag has to be slow...since includes are  
slow anyway.

I am ASSUMING that is is better to cast the file as a variable and do  
-body = $var

OK it is not a personalized message at that point, but has to be  
faster. Anybody know this?

Deco

On Apr 10, 2006, at 9:28 PM, charles brisson wrote:

> Alex,
> I'm facing the same problem. Trying to build something that might  
> have to send thousands of emails at once. I clogged my Lasso  
> Hosting service last night and I am still receiving emails tonight.  
> I need all the emails to be sent at once and delivered at once. OK,  
> sure, I could stand a few minutes delay but not 30 hours.
>
> So, my question to you is what are you using now for mass email-
> sending?
>
> Thanks
>
> Charles
>
> Alex Pilson wrote:
>
>> At 4:31 PM -0700 4/10/06, Dan McComb wrote:
>>
>>> I do a weekly bulk mailing in which about 600 emails are sent out  
>>> from a membership database. Everything works fine, but it takes a  
>>> LOOOOONG time to send (I had to disable the script execution  
>>> timeout for it to work without timing out partway through the  
>>> mailing). But what's most weird, is that the CPU jumps to 100%  
>>> during the mailing (G5 xserve). Here's the code I'm using to the  
>>> do the mailing. Is there a way for me to streamline this so it  
>>> doesn't hog all the CPU?
>>
>>
>> That is mostly my experience as well. Especially with a busy  
>> server. Try sending over 30,000...near impossible. The last one I  
>> did, I had to restart the "mailing process" 3 times from where it  
>> left off to complete the mailing. I have since given up on the  
>> prospect of a Lasso driven mass emailer.
>>
>
> ------------------------------
> If you missed the Summit, you can still learn from the best at a  
> Lasso Master Class near you!
> http://www.omnipilot.com/id.lasso?tid=91
>
> ------------------------------
> Lasso Support: http://support.omnipilot.com/
> Search the list archives: http://www.listsearch.com/lassotalk.lasso
> Manage your list subscription:  http://www.listsearch.com/ 
> lassotalk.lasso?manage


------------------------------
If you missed the Summit, you can still learn from the best at a Lasso Master Class near you!
http://www.omnipilot.com/id.lasso?tid=91

------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage
Reply | Threaded
Open this post in threaded view
|

Re: sending mail = 100 percent cpu usage

Fabrizio Carioni
In reply to this post by Dan McComb
Ciao, I've been facing the same problem after passing to Lasso 8.
With Lasso 6 things where much much faster, talking about mailing
issues.

Anyway, we send out a newsletter to 50.000 email adresses every day +
a personalised newsletter (with unique/different content) to 2000
people daily + a newsletter to 180.000 twice a week. All this happens
on a G5 with Lasso 8. This is to say that with Lasso 8 you can do
this things, even though it's much harder than with previous versions
of Lasso. Maybe future versions of Lasso 8.5 will speed things up.

Things you might do are:

- use the bcc field to send out emails when the email content is
identical. 10 emails at 100 persons go out much faster than 1000
single emails. Just pay attention to not put 2 much email adresses in
the bcc field. We use max 1000.
Validate the emails. A malformed/invalid email adress can block a
message holding other hundreds of email adresses in the bcc. Check
the email error logs.

- If the main script sending out the emails takes too long you can
split it up (maybe using a skiprecords parameter). This way, other
tasks can occur between one call of the script and the other. What we
do is to use scheduled events that during a period of one hour send
out 200 emails every 5 minutes or so.

HTH

Ciao

--
======================================================================
Fabrizio Carioni - Golem100 S.r.l.
Via Gran Sasso, 50 - 20090 - Segrate (Mi) - Italy
Voice +39-02-2133402  -  Fax +39-02-2132105  - Mobile 3356463448
Email [hidden email] - URL http://www.golem100.com/
======================================================================

------------------------------
If you missed the Summit, you can still learn from the best at a Lasso Master Class near you!
http://www.omnipilot.com/id.lasso?tid=91

------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage
Reply | Threaded
Open this post in threaded view
|

Re: sending mail = 100 percent cpu usage

Dan McComb
In reply to this post by Dan McComb

> - use the bcc field to send out emails when the email content is
> identical. 10 emails at 100 persons go out much faster than 1000
> single emails. Just pay attention to not put 2 much email adresses in
> the bcc field. We use max 1000.

So, correct me if I'm wrong, but what I'm hearing is that basically  
Lasso is a poor choice for sending bulk custom email. I can't use the  
BCC field, because each of the 600 emails is personalized, not only  
by name but also the data that each contains.

I use IntelliMerge Pro to send out bulk mail that doesn't require  
Lasso processed messages, and it flat flies, without hogging all the  
cpu. So it really floors me that it takes Lasso 50 minutes at 100  
percent CPU on an Xserve G5 to send 600 messages. I guess the  
solution for the short term is to use a secondary mail server, so it  
doesn't bring my main machine to a crawl whenever I send mail using  
Lasso.

/dan

------------------------------------------------------------------------
------------

Visual Contact
4318 5th Ave. NW
Seattle, WA 98107
206.228.0780

[hidden email]
http://www.visualcontact.com


------------------------------
If you missed the Summit, you can still learn from the best at a Lasso Master Class near you!
http://www.omnipilot.com/id.lasso?tid=91

------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage
Reply | Threaded
Open this post in threaded view
|

Re: sending mail = 100 percent cpu usage

Jon Harris
In reply to this post by Dan McComb
Hi Dan

We had this issue with Lasso on our windows server.

We created a page with an iframe in it.

In the iframe there was a go button, to send 100 at a time. It uses a
meta refresh to set itself up for the next "run". Something like this
(shortened version).

var: 'throttle' = 100;
var: 'nextpage' = (math_add: (action_param: 'startpos', $throttle) ;
var: 'refresh'= http://www.domain.com/emailiframe.lasso?startpos=' +
$nextpage  + 'process=true'

if: $nexpage < $maxrecords
        output: '<META HTTP-EQUIV=Refresh CONTENT="10; URL=' + $refresh
+ '">' :
if ;


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On
Behalf Of Dan McComb
Sent: 11 April 2006 15:36
To: [hidden email]
Subject: Re: sending mail = 100 percent cpu usage


> - use the bcc field to send out emails when the email content is
> identical. 10 emails at 100 persons go out much faster than 1000
> single emails. Just pay attention to not put 2 much email adresses in
> the bcc field. We use max 1000.

So, correct me if I'm wrong, but what I'm hearing is that basically
Lasso is a poor choice for sending bulk custom email. I can't use the
BCC field, because each of the 600 emails is personalized, not only by
name but also the data that each contains.

I use IntelliMerge Pro to send out bulk mail that doesn't require Lasso
processed messages, and it flat flies, without hogging all the cpu. So
it really floors me that it takes Lasso 50 minutes at 100 percent CPU on
an Xserve G5 to send 600 messages. I guess the solution for the short
term is to use a secondary mail server, so it doesn't bring my main
machine to a crawl whenever I send mail using Lasso.

/dan

------------------------------------------------------------------------
------------

Visual Contact
4318 5th Ave. NW
Seattle, WA 98107
206.228.0780

[hidden email]
http://www.visualcontact.com


------------------------------
If you missed the Summit, you can still learn from the best at a Lasso
Master Class near you!
http://www.omnipilot.com/id.lasso?tid=91

------------------------------
Lasso Support: http://support.omnipilot.com/ Search the list archives:
http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage

------------------------------
If you missed the Summit, you can still learn from the best at a Lasso Master Class near you!
http://www.omnipilot.com/id.lasso?tid=91

------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage
Reply | Threaded
Open this post in threaded view
|

Re: sending mail = 100 percent cpu usage

Fletcher Sandbeck
In reply to this post by Dan McComb
On 4/11/06 at 7:35 AM by [hidden email] (Dan McComb):

>So, correct me if I'm wrong, but what I'm hearing is that basically  
>Lasso is a poor choice for sending bulk custom email. I can't use the  
>BCC field, because each of the 600 emails is personalized, not only  
>by name but also the data that each contains.

Lasso was not designed to send bulk email.  The email features are designed to be complementary to what is required for communication with Web site visitors so are optimized for quickly sending password reminders, order confirmations, etc.

In your case, the quoted-printable encoding of the message body is probably what is causing the main slow down.  The messages generally send very quickly once they are queued, but can take a while to compose.  You can turn off encoding of the message by adding -TransferEncoding='8bit' to the [Email_Send] tag.  This will send the message as raw UTF-8 characters.

We are working on improving the speed of message encoding and on other enhancements that will make sending many customized messages through Lasso easier.

[fletcher]
--
Fletcher Sandbeck                         [hidden email]
Director of Product Development       http://www.lassostudio.com
OmniPilot Software, Inc.                http://www.omnipilot.com

------------------------------
If you missed the Summit, you can still learn from the best at a Lasso Master Class near you!
http://www.omnipilot.com/id.lasso?tid=91

------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage
Reply | Threaded
Open this post in threaded view
|

Re: sending mail = 100 percent cpu usage

Brett Circe
In reply to this post by Dan McComb
Dan, we send our monthly newsletter each month to about 19,000  
people, each email customized with their name and email address  
through Lasso. We build the emails and store them in a table, then  
sort them by domain and send them in batched as Jon pointed out in  
his email. This takes several hours but is stable, reliable, and does  
not consume a ton of CPU. We delete each batch as it is sent just in  
case we need to restart the queue.

And, we are looking forward to re-writing this routing using the new  
email tags in 8.5 to make it more efficient and faster.


On Apr 11, 2006, at 10:35 AM, Dan McComb wrote:

>
>> - use the bcc field to send out emails when the email content is
>> identical. 10 emails at 100 persons go out much faster than 1000
>> single emails. Just pay attention to not put 2 much email adresses in
>> the bcc field. We use max 1000.
>
> So, correct me if I'm wrong, but what I'm hearing is that basically  
> Lasso is a poor choice for sending bulk custom email. I can't use  
> the BCC field, because each of the 600 emails is personalized, not  
> only by name but also the data that each contains.
>
> I use IntelliMerge Pro to send out bulk mail that doesn't require  
> Lasso processed messages, and it flat flies, without hogging all  
> the cpu. So it really floors me that it takes Lasso 50 minutes at  
> 100 percent CPU on an Xserve G5 to send 600 messages. I guess the  
> solution for the short term is to use a secondary mail server, so  
> it doesn't bring my main machine to a crawl whenever I send mail  
> using Lasso.
>
> /dan
>
> ----------------------------------------------------------------------
> --------------
>
> Visual Contact
> 4318 5th Ave. NW
> Seattle, WA 98107
> 206.228.0780
>
> [hidden email]
> http://www.visualcontact.com
>
>
> ------------------------------
> If you missed the Summit, you can still learn from the best at a  
> Lasso Master Class near you!
> http://www.omnipilot.com/id.lasso?tid=91
>
> ------------------------------
> Lasso Support: http://support.omnipilot.com/
> Search the list archives: http://www.listsearch.com/lassotalk.lasso
> Manage your list subscription: http://www.listsearch.com/ 
> lassotalk.lasso?manage


Brett Circe
[hidden email]




------------------------------
If you missed the Summit, you can still learn from the best at a Lasso Master Class near you!
http://www.omnipilot.com/id.lasso?tid=91

------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage
Reply | Threaded
Open this post in threaded view
|

Re: sending mail = 100 percent cpu usage

Viaduct Productions
In reply to this post by Dan McComb
Hi Brett.  Any chance you could keep some metrics on improvement with
8.5 on that project, and post here?

Cheers

Brett Circe wrote:

> Dan, we send our monthly newsletter each month to about 19,000 people,
> each email customized with their name and email address through Lasso.
> We build the emails and store them in a table, then sort them by
> domain and send them in batched as Jon pointed out in his email. This
> takes several hours but is stable, reliable, and does not consume a
> ton of CPU. We delete each batch as it is sent just in case we need to
> restart the queue.
>
> And, we are looking forward to re-writing this routing using the new
> email tags in 8.5 to make it more efficient and faster.
>
>
> On Apr 11, 2006, at 10:35 AM, Dan McComb wrote:
>
>>
>>> - use the bcc field to send out emails when the email content is
>>> identical. 10 emails at 100 persons go out much faster than 1000
>>> single emails. Just pay attention to not put 2 much email adresses in
>>> the bcc field. We use max 1000.
>>
>> So, correct me if I'm wrong, but what I'm hearing is that basically
>> Lasso is a poor choice for sending bulk custom email. I can't use the
>> BCC field, because each of the 600 emails is personalized, not only
>> by name but also the data that each contains.
>>
>> I use IntelliMerge Pro to send out bulk mail that doesn't require
>> Lasso processed messages, and it flat flies, without hogging all the
>> cpu. So it really floors me that it takes Lasso 50 minutes at 100
>> percent CPU on an Xserve G5 to send 600 messages. I guess the
>> solution for the short term is to use a secondary mail server, so it
>> doesn't bring my main machine to a crawl whenever I send mail using
>> Lasso.
>>
>> /dan
>>
>> ------------------------------------------------------------------------------------
>>
>>
>> Visual Contact
>> 4318 5th Ave. NW
>> Seattle, WA 98107
>> 206.228.0780
>>
>> [hidden email]
>> http://www.visualcontact.com
>>
>>
>> ------------------------------
>> If you missed the Summit, you can still learn from the best at a
>> Lasso Master Class near you!
>> http://www.omnipilot.com/id.lasso?tid=91
>>
>> ------------------------------
>> Lasso Support: http://support.omnipilot.com/
>> Search the list archives: http://www.listsearch.com/lassotalk.lasso
>> Manage your list subscription:
>> http://www.listsearch.com/lassotalk.lasso?manage
>
>
> Brett Circe
> [hidden email]
>
>
>
>
> ------------------------------
> If you missed the Summit, you can still learn from the best at a Lasso
> Master Class near you!
> http://www.omnipilot.com/id.lasso?tid=91
>
> ------------------------------
> Lasso Support: http://support.omnipilot.com/
> Search the list archives: http://www.listsearch.com/lassotalk.lasso
> Manage your list subscription:
> http://www.listsearch.com/lassotalk.lasso?manage
>

------------------------------
If you missed the Summit, you can still learn from the best at a Lasso Master Class near you!
http://www.omnipilot.com/id.lasso?tid=91

------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage
Reply | Threaded
Open this post in threaded view
|

Re: sending mail = 100 percent cpu usage

Alex Pilson
In reply to this post by Dan McComb
At 7:35 AM -0700 4/11/06, Dan McComb wrote:

>>- use the bcc field to send out emails when the email content is
>>identical. 10 emails at 100 persons go out much faster than 1000
>>single emails. Just pay attention to not put 2 much email adresses in
>>the bcc field. We use max 1000.
>
>So, correct me if I'm wrong, but what I'm hearing is that basically
>Lasso is a poor choice for sending bulk custom email. I can't use
>the BCC field, because each of the 600 emails is personalized, not
>only by name but also the data that each contains.
>
>I use IntelliMerge Pro to send out bulk mail that doesn't require
>Lasso processed messages, and it flat flies, without hogging all the
>cpu. So it really floors me that it takes Lasso 50 minutes at 100
>percent CPU on an Xserve G5 to send 600 messages. I guess the
>solution for the short term is to use a secondary mail server, so it
>doesn't bring my main machine to a crawl whenever I send mail using
>Lasso.

The mail server has little to do with Lasso's inability to send email
efficiently. It the process of  building the message, encoding (if
any), queuing, then sending, all the while you are telling Lasso to
pull emails and data in a records loop. I have tried several tricks
in the books, and all-in-all the only way I have successfully sent
mass email at high speed at no CPU cost to Lasso, is using a custom
LJAPI module that sends the mail for you. I had this running under 6,
but haven't tried under 8. I was hopeful the new stuff in 8.1 as
going to work, I was wrong.

--
<------------------------------------------------------------------------------>
     Alex Pilson          FlagShip Hosting, LLC
     800.592.6781     <http://www.flagshiphosting.com>
<------------------------------------------------------------------------------>

------------------------------
If you missed the Summit, you can still learn from the best at a Lasso Master Class near you!
http://www.omnipilot.com/id.lasso?tid=91

------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage
Reply | Threaded
Open this post in threaded view
|

Re: sending mail = 100 percent cpu usage

Fabrizio Carioni
In reply to this post by Dan McComb
Ciao Fletcher, just out of curiosity, why was Lasso 6 so fast then?
I never had a problem sending out lots and lots of emails (with
exactly the same content i'm doing now) with Lasso 6.


Ciao


>On 4/11/06 at 7:35 AM by [hidden email] (Dan McComb):
>
>>So, correct me if I'm wrong, but what I'm hearing is that basically
>>Lasso is a poor choice for sending bulk custom email. I can't use the
>>BCC field, because each of the 600 emails is personalized, not only
>>by name but also the data that each contains.
>
>Lasso was not designed to send bulk email.  The email features are
>designed to be complementary to what is required for communication
>with Web site visitors so are optimized for quickly sending password
>reminders, order confirmations, etc.
>
>In your case, the quoted-printable encoding of the message body is
>probably what is causing the main slow down.  The messages generally
>send very quickly once they are queued, but can take a while to
>compose.  You can turn off encoding of the message by adding
>-TransferEncoding='8bit' to the [Email_Send] tag.  This will send
>the message as raw UTF-8 characters.
>
>We are working on improving the speed of message encoding and on
>other enhancements that will make sending many customized messages
>through Lasso easier.
>
>[fletcher]
>--
>Fletcher Sandbeck                         [hidden email]
>Director of Product Development       http://www.lassostudio.com
>OmniPilot Software, Inc.                http://www.omnipilot.com
>
>------------------------------
>If you missed the Summit, you can still learn from the best at a
>Lasso Master Class near you!
>http://www.omnipilot.com/id.lasso?tid=91
>
>------------------------------
>Lasso Support: http://support.omnipilot.com/
>Search the list archives: http://www.listsearch.com/lassotalk.lasso
>Manage your list subscription:
>http://www.listsearch.com/lassotalk.lasso?manage


--
======================================================================
Fabrizio Carioni -
Golem100                                                                                  
Via Gran Sasso, 50 - 20090 - Segrate (Mi) - Italy
Voice +39-02-2133402  -  3356463448
Email [hidden email] - URL http://www.golem100.com/
======================================================================

------------------------------
If you missed the Summit, you can still learn from the best at a Lasso Master Class near you!
http://www.omnipilot.com/id.lasso?tid=91

------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage
Reply | Threaded
Open this post in threaded view
|

Re: sending mail = 100 percent cpu usage

Fabrizio Carioni
In reply to this post by Dan McComb
It's probably a bad choiche if you are only looking at speed.

The sending of personalised emails involves in our case mantaining
the users list, creating dynamically the content of the emails and
finally sending it out through a mail server.
I think Lasso handles the first two parts really well, it's just the
sending that is real bad (in terms of speed).

The numbers you are giving are a bit too bad though. On a normal G5
(1.6 ghz) we send out approx. 1 email per second.
This involves looking up the email addresses on a table, fetching the
correct content for every mail on a second table, composing the mail
and queuing.
This means 2000 emails in less than 40 minutes. During this time, the
machine slows down a bit but not too much.

This is just to give you some info that I hope can help you find a solution.

HTH




>>- use the bcc field to send out emails when the email content is
>>identical. 10 emails at 100 persons go out much faster than 1000
>>single emails. Just pay attention to not put 2 much email adresses in
>>the bcc field. We use max 1000.
>
>So, correct me if I'm wrong, but what I'm hearing is that basically
>Lasso is a poor choice for sending bulk custom email. I can't use
>the BCC field, because each of the 600 emails is personalized, not
>only by name but also the data that each contains.
>
>I use IntelliMerge Pro to send out bulk mail that doesn't require
>Lasso processed messages, and it flat flies, without hogging all the
>cpu. So it really floors me that it takes Lasso 50 minutes at 100
>percent CPU on an Xserve G5 to send 600 messages. I guess the
>solution for the short term is to use a secondary mail server, so it
>doesn't bring my main machine to a crawl whenever I send mail using
>Lasso.
>
>/dan
>
>------------------------------------------------------------------------------------
>
>Visual Contact
>4318 5th Ave. NW
>Seattle, WA 98107
>206.228.0780
>
>[hidden email]
>http://www.visualcontact.com
>
>
>------------------------------
>If you missed the Summit, you can still learn from the best at a
>Lasso Master Class near you!
>http://www.omnipilot.com/id.lasso?tid=91
>
>------------------------------
>Lasso Support: http://support.omnipilot.com/
>Search the list archives: http://www.listsearch.com/lassotalk.lasso
>Manage your list subscription:
>http://www.listsearch.com/lassotalk.lasso?manage


--
======================================================================
Fabrizio Carioni -
Golem100                                                                                  
Via Gran Sasso, 50 - 20090 - Segrate (Mi) - Italy
Voice +39-02-2133402  -  3356463448
Email [hidden email] - URL http://www.golem100.com/
======================================================================

------------------------------
If you missed the Summit, you can still learn from the best at a Lasso Master Class near you!
http://www.omnipilot.com/id.lasso?tid=91

------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage
Reply | Threaded
Open this post in threaded view
|

Re: sending mail = 100 percent cpu usage

Fletcher Sandbeck
In reply to this post by Dan McComb
On 4/11/06 at 10:24 PM by [hidden email] (Fabrizio Carioni):

>Ciao Fletcher, just out of curiosity, why was Lasso 6 so fast then?
>I never had a problem sending out lots and lots of emails (with
>exactly the same content i'm doing now) with Lasso 6.

In Lasso 6 the encoding was applied to the body after the message was queued.  Messages were encoded just before being sent and the time for encoding was absorbed into a delay in the email queue.  However, Lasso 6 did not directly support HTML messages or multi-part messages and only had basic support for attachments.

Lasso 7 added HTML and multi-part message support.  As the messages became more complex it is necessary to perform the encoding of each part as the message was queued.  Lasso 8.1 has significantly reworked encoding algorithms, but there's still a lot of room for improvement.

In Lasso 8.5 we're working on several improvements to the email sending system.  Lasso 8.5 will send email directly to target SMTP servers, will have much faster encoding algorithms, will support programmatically created attachments, will support embedded HTML images, and will have an email merge feature which allows you to queue up one message and have tokens within the message replaced with personalized information within the email queue rather than when the message is sent.

[fletcher]
--
Fletcher Sandbeck                         [hidden email]
Director of Product Development       http://www.lassostudio.com
OmniPilot Software, Inc.                http://www.omnipilot.com

------------------------------
If you missed the Summit, you can still learn from the best at a Lasso Master Class near you!
http://www.omnipilot.com/id.lasso?tid=91

------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage
Reply | Threaded
Open this post in threaded view
|

Re: sending mail = 100 percent cpu usage

Fabrizio Carioni
In reply to this post by Dan McComb
Thanks for the explanation Fletcher.
These improvements will be really useful for our projects. The email
issue is the only reason why 2 of our servers are still using Lasso 6.

Regards



>On 4/11/06 at 10:24 PM by [hidden email] (Fabrizio Carioni):
>
>>Ciao Fletcher, just out of curiosity, why was Lasso 6 so fast then?
>>I never had a problem sending out lots and lots of emails (with
>>exactly the same content i'm doing now) with Lasso 6.
>
>In Lasso 6 the encoding was applied to the body after the message
>was queued.  Messages were encoded just before being sent and the
>time for encoding was absorbed into a delay in the email queue.
>However, Lasso 6 did not directly support HTML messages or
>multi-part messages and only had basic support for attachments.
>
>Lasso 7 added HTML and multi-part message support.  As the messages
>became more complex it is necessary to perform the encoding of each
>part as the message was queued.  Lasso 8.1 has significantly
>reworked encoding algorithms, but there's still a lot of room for
>improvement.
>
>In Lasso 8.5 we're working on several improvements to the email
>sending system.  Lasso 8.5 will send email directly to target SMTP
>servers, will have much faster encoding algorithms, will support
>programmatically created attachments, will support embedded HTML
>images, and will have an email merge feature which allows you to
>queue up one message and have tokens within the message replaced
>with personalized information within the email queue rather than
>when the message is sent.
>
>[fletcher]
>--
>Fletcher Sandbeck                         [hidden email]
>Director of Product Development       http://www.lassostudio.com
>OmniPilot Software, Inc.                http://www.omnipilot.com
>
>------------------------------
>If you missed the Summit, you can still learn from the best at a
>Lasso Master Class near you!
>http://www.omnipilot.com/id.lasso?tid=91
>
>------------------------------
>Lasso Support: http://support.omnipilot.com/
>Search the list archives: http://www.listsearch.com/lassotalk.lasso
>Manage your list subscription:
>http://www.listsearch.com/lassotalk.lasso?manage


--
======================================================================
Fabrizio Carioni -
Golem100                                                                                  
Via Gran Sasso, 50 - 20090 - Segrate (Mi) - Italy
Voice +39-02-2133402  -  3356463448
Email [hidden email] - URL http://www.golem100.com/
======================================================================

------------------------------
If you missed the Summit, you can still learn from the best at a Lasso Master Class near you!
http://www.omnipilot.com/id.lasso?tid=91

------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage
Reply | Threaded
Open this post in threaded view
|

Re: sending mail = 100 percent cpu usage

Ke Carlton
In reply to this post by Dan McComb
Hello All,

We currently use Lasso 8 for our large mailing lists(30k+) with no problem.
Our setup processes 20,000 customized emails an hour based on a 15-30kb
email. It deals with 100k+ emails fine, although it things slow down
respectively.

It took some time and a lot of frustration to get to this point, the L8
site's sqlite database could not keep pace, would bloat to 500-800mb before
falling over. After which the site would either not restart or would take
some time to do so (I believe the site was having a good think about what it
should try and do with the 500mb smtp_queue table).

In the end the trick was to bypass the L8 smtp queue and connect directly to
the smtp server. The speed improvement is huge and there are advantages
using this method - although it does required some extra leg work.

The core of the code is below, you should be able to build everything else
needed around it - I hope some people with find this useful.

Regards,

Ke


var: 'SMTP' = (Email_SMTP);
$SMTP->(Open: -Host=host,-port=25);
                               
if:(error_currenterror:-errorcode) != 0;
        //SMTP OFFLINE
        abort;
/if;

Records;
        //SET VARS ETC

        var:'msg' = (Email_compose:
                -To=$email,
                -From=$from,
                -Subject=$subject,
                -port=port,-HTML=$email,-ExtraMIMEHeaders=$MIME_Headers);

        protect;
        var:'ok' = $SMTP->(Send:-From=$msg->From,
                                -Recipients=$msg->Recipients,
                                -Message=$msg->Data);
        /protect;

        If:$ok;
                //MSG SENT
        Else;
                //MSG FAILED
        /if;
/Records;

$SMTP->Close;





-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf
Of Alex Pilson
Sent: 11 April 2006 02:52
To: [hidden email]
Subject: Re: sending mail = 100 percent cpu usage

At 4:31 PM -0700 4/10/06, Dan McComb wrote:
>I do a weekly bulk mailing in which about 600 emails are sent out
>from a membership database. Everything works fine, but it takes a
>LOOOOONG time to send (I had to disable the script execution timeout
>for it to work without timing out partway through the mailing). But
>what's most weird, is that the CPU jumps to 100% during the mailing
>(G5 xserve). Here's the code I'm using to the do the mailing. Is
>there a way for me to streamline this so it doesn't hog all the CPU?

That is mostly my experience as well. Especially with a busy server.
Try sending over 30,000...near impossible. The last one I did, I had
to restart the "mailing process" 3 times from where it left off to
complete the mailing. I have since given up on the prospect of a
Lasso driven mass emailer.

--
<---------------------------------------------------------------------------
--->
     Alex Pilson          FlagShip Hosting, LLC
     800.592.6781     <http://www.flagshiphosting.com>
<---------------------------------------------------------------------------
--->

------------------------------
If you missed the Summit, you can still learn from the best at a Lasso
Master Class near you!
http://www.omnipilot.com/id.lasso?tid=91

------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage

------------------------------
If you missed the Summit, you can still learn from the best at a Lasso Master Class near you!
http://www.omnipilot.com/id.lasso?tid=91

------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage
Reply | Threaded
Open this post in threaded view
|

Re: sending mail = 100 percent cpu usage

decorior
In reply to this post by Dan McComb
Hi, Ke

Did the memory associated with LP8 also increase?

Thanks for the code

Deco
On Apr 12, 2006, at 3:43 AM, K Carlton wrote:

> Hello All,
>
> We currently use Lasso 8 for our large mailing lists(30k+) with no  
> problem.
> Our setup processes 20,000 customized emails an hour based on a  
> 15-30kb
> email. It deals with 100k+ emails fine, although it things slow down
> respectively.
>
> It took some time and a lot of frustration to get to this point,  
> the L8
> site's sqlite database could not keep pace, would bloat to  
> 500-800mb before
> falling over. After which the site would either not restart or  
> would take
> some time to do so (I believe the site was having a good think  
> about what it
> should try and do with the 500mb smtp_queue table).
>
> In the end the trick was to bypass the L8 smtp queue and connect  
> directly to
> the smtp server. The speed improvement is huge and there are  
> advantages
> using this method - although it does required some extra leg work.
>
> The core of the code is below, you should be able to build  
> everything else
> needed around it - I hope some people with find this useful.
>
> Regards,
>
> Ke
>
>
> var: 'SMTP' = (Email_SMTP);
> $SMTP->(Open: -Host=host,-port=25);
>
> if:(error_currenterror:-errorcode) != 0;
> //SMTP OFFLINE
> abort;
> /if;
>
> Records;
> //SET VARS ETC
>
> var:'msg' = (Email_compose:
> -To=$email,
> -From=$from,
> -Subject=$subject,
> -port=port,-HTML=$email,-ExtraMIMEHeaders=$MIME_Headers);
>
> protect;
> var:'ok' = $SMTP->(Send:-From=$msg->From,
> -Recipients=$msg->Recipients,
> -Message=$msg->Data);
> /protect;
>
> If:$ok;
> //MSG SENT
> Else;
> //MSG FAILED
> /if;
> /Records;
>
> $SMTP->Close;
>
>
>
>
>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]]  
> On Behalf
> Of Alex Pilson
> Sent: 11 April 2006 02:52
> To: [hidden email]
> Subject: Re: sending mail = 100 percent cpu usage
>
> At 4:31 PM -0700 4/10/06, Dan McComb wrote:
>> I do a weekly bulk mailing in which about 600 emails are sent out
>> from a membership database. Everything works fine, but it takes a
>> LOOOOONG time to send (I had to disable the script execution timeout
>> for it to work without timing out partway through the mailing). But
>> what's most weird, is that the CPU jumps to 100% during the mailing
>> (G5 xserve). Here's the code I'm using to the do the mailing. Is
>> there a way for me to streamline this so it doesn't hog all the CPU?
>
> That is mostly my experience as well. Especially with a busy server.
> Try sending over 30,000...near impossible. The last one I did, I had
> to restart the "mailing process" 3 times from where it left off to
> complete the mailing. I have since given up on the prospect of a
> Lasso driven mass emailer.
>
> --
> <---------------------------------------------------------------------
> ------
> --->
>      Alex Pilson          FlagShip Hosting, LLC
>      800.592.6781     <http://www.flagshiphosting.com>
> <---------------------------------------------------------------------
> ------
> --->
>
> ------------------------------
> If you missed the Summit, you can still learn from the best at a Lasso
> Master Class near you!
> http://www.omnipilot.com/id.lasso?tid=91
>
> ------------------------------
> Lasso Support: http://support.omnipilot.com/
> Search the list archives: http://www.listsearch.com/lassotalk.lasso
> Manage your list subscription:
> http://www.listsearch.com/lassotalk.lasso?manage
>
> ------------------------------
> If you missed the Summit, you can still learn from the best at a  
> Lasso Master Class near you!
> http://www.omnipilot.com/id.lasso?tid=91
>
> ------------------------------
> Lasso Support: http://support.omnipilot.com/
> Search the list archives: http://www.listsearch.com/lassotalk.lasso
> Manage your list subscription:
> http://www.listsearch.com/lassotalk.lasso?manage


------------------------------
If you missed the Summit, you can still learn from the best at a Lasso Master Class near you!
http://www.omnipilot.com/id.lasso?tid=91

------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage
Reply | Threaded
Open this post in threaded view
|

Re: sending mail = 100 percent cpu usage

Alex Pilson
In reply to this post by Dan McComb
At 10:43 AM +0100 4/12/06, K Carlton wrote:

>Hello All,
>
>
>var: 'SMTP' = (Email_SMTP);
>$SMTP->(Open: -Host=host,-port=25);
>
>if:(error_currenterror:-errorcode) != 0;
> //SMTP OFFLINE
> abort;
>/if;
>
>Records;
> //SET VARS ETC
>
> var:'msg' = (Email_compose:
> -To=$email,
> -From=$from,
> -Subject=$subject,
> -port=port,-HTML=$email,-ExtraMIMEHeaders=$MIME_Headers);
>
> protect;
> var:'ok' = $SMTP->(Send:-From=$msg->From,
> -Recipients=$msg->Recipients,
> -Message=$msg->Data);
> /protect;
>
> If:$ok;
> //MSG SENT
> Else;
> //MSG FAILED
> /if;
>/Records;
>
>$SMTP->Close;

This is quite interesting. I didn't try this technique, though the
formula is very close to mine. I did try email_immediate, I wonder
how much different this is than email_immediate? Maybe Fletcher can
let us know. Maybe one open SMTP connection versus constant
open/close of tcp connections?

--
<------------------------------------------------------------------------------>
     Alex Pilson          FlagShip Hosting, LLC
     800.592.6781     <http://www.flagshiphosting.com>
<------------------------------------------------------------------------------>

------------------------------
If you missed the Summit, you can still learn from the best at a Lasso Master Class near you!
http://www.omnipilot.com/id.lasso?tid=91

------------------------------
Lasso Support: http://support.omnipilot.com/
Search the list archives: http://www.listsearch.com/lassotalk.lasso
Manage your list subscription:  
http://www.listsearch.com/lassotalk.lasso?manage
12