Mail Queue Replacement for 8.6?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Mail Queue Replacement for 8.6?

Mason Miller
A couple of times over the past year or so, people have mentioned replacements for Lasso's mail queue. Does anyone have one I can start with, even if it is not well documented, etc?

Thanks,

Mason


#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Mail Queue Replacement for 8.6?

Lars A. Gundersen-2

On 04 Feb 2013, at 19:01, Mason Miller <[hidden email]> wrote:

> A couple of times over the past year or so, people have mentioned replacements for Lasso's mail queue. Does anyone have one I can start with, even if it is not well documented, etc?
> Thanks,
> Mason

Hi,

I think many have ended up rolling their own custom solution, but maybe those, like mine, are to closely tied into the workflow of their own code and setup to be easy to package as a stand-alone email queue replacement.
However, having complained about it for so long, I feel a certain responsibility to share info about how i did it, maybe it'll help you and others along.

The built-in queue basically maintains copies of any email you send in a separate table, with fields for ID, counter with number of sending attempts, status, etc, as well as the message itself. Then there is a script that runs periodically and try to send the messages. If it manages, good, if not, the message is left in queue and the counter increases, until it is either sent or the counter gets to a predetermined number, whereupon it is marked as failed.
This is the baseline, it is what I had to replicate.

So, turns out, I and maybe you, have lots of the stuff that gets shoveled into this queue elsewhere already. I have a customer table where the email message text gets stored anyway, togheter with other data for the customers. And I maintain records of recipients of emails for other reasons than the email. Add an integer counter column to that table, and a varchar status column and a timestamp, and I had what I needed - no need to shuffle the outgoing emails into its own table.

When customers press send email, I use the SMTP tags to send. No email_send at all. Just construct the email pieces from the info in the customer's record with

Var: 'msg' = (Email_Compose: -To=$to_emailaddress, -From=$sender, -Subject=$thisSubject, -Body=$textBody, -Html=(Include: '/public/resources/email_template.lasso'), -HTMLImages=(Array: '/author/files/' + ($user) + '/' + $headerimage));

You can disregard the HTML parts, I use an hetml template as indicated. This template can contan Lasso code and can use variables form your current page, so it can produce nice emails.

Then I send to a custom tag:

bt_send_email:  -emailaddress=$thisEmail, -messagepart=$msg;

and here is bt_send_email:

<?LassoScript
        define_tag: 'bt_send_email',
                -required='emailaddress',
                -required='messagepart';
               
                Local: 'willsend' = true;
                Local: 'hassent' = false;
                Local: 'errormsg' = '';
                Local: 'domain' = #emailaddress->(Split: '@')->Last;
                local: 'SMTP' = (Email_SMTP);
               
                #domain = Email_MXLookup(#domain);
                If: !(#domain->(isa('map')));
                        #errormsg = 'MX LOOKUP ERROR: No SMTP host found for direct send to ' + #emailaddress->(Split: '@')->Last;
                        #willsend = false;
                Else;
                        #domain = (#domain)->Find('host');
                        If: #domain == '';
                                #errormsg = 'MX LOOKUP ERROR: No SMTP host found for direct send to ' + #emailaddress->(Split: '@')->Last;
                                #willsend = false;
                        /If;
                /If;
                If: #willsend;
                        Protect;
                                #SMTP->(Open: -Host=#domain,-port=25);
                                handle_error;
                                        #willsend = false;
                                        #errormsg = 'SMTP OPEN ERROR: ' + Error_CurrentError;
                                /handle_error;
                        /Protect;
                        If: #willsend;
                                Protect;
                                        #hassent = #SMTP->(Send: -Message=#messagepart->data, -From='[hidden email]', -Recipients=#messagepart->recipients);
                                        handle_error;
                                                #willsend = false;
                                                #errormsg = 'SMTP SEND ERROR: ' + Error_CurrentError;
                                        /handle_error;
                                /Protect;
                        /If;
                        #SMTP->(Close);
                /If;
                Local: 'sentTime' = (date)->(Format: '%Q %T');
                If: #hassent;
                        // write to the recipient table with status of sent and a timestamp
                Else if: #errormsg;
                        //Write to the recipient table with status of the error messages, a timestamp and increase the value of the counter that counts sending attempts, and write that too.
                Else;
                        //Should never end up here....
                /If;
        /define_tag;
?>

That's the sending and the structures needed for storing all necessary bits of info about the email. The only thing left is the script that periodically tries to send emails that don't get sucessfully sent the first time. I have this Lasso code excecuted in the event system every 15 minutes. Notice that this isn't just for "error recovery", as it were. A lot of mail servers are sat up to bounce messages first time they come from an unknown server/sender. They will subsequently accept the same message a few minutes later. This is known as greylisting, and 15 minutes seemed like a good compromise.

<?Lassoscript
var: 'user' = (Action_Param: 'username'); // 'xxx';
var: 'passw' = (Action_Param: 'pass'); //'xxx';
var: 'failed' = false;
var: 'cutoff_date' = Date_Format( Date_Subtract( Date, -Month=6), -Format='%Q %T');
var: 'cutoff_time' = Date_Format( Date_Subtract( Date, -Day=6), -Format='%Q %T');


// I have a structure for surveys, you might not, but it could be 'member lists' or whatever group the customer wants to send mass emails to.

// Do this survey by survey, far all surveys started the last 1/2 year

Inline: -username='xxx', -password='xxx"', -database='surveys', -table='survey_defs', -Search, -maxrecords='all', -KeyField='ID', ...... , -op='gt', 'invitationTime'=$cutoff_date;
 Records;
       
        //Then we iterate over the survey's participants, set all to queued again and send them off to the sending system.
        //The sending system will attempt to send and incerase the count, so that we only try maximum 5 times.
       
                Inline: -username='xxx', -password='xxx"', -database='parts', -table='participants', -Search, -MaxRecords='all', -KeyField='ID', -op='eq', 'survey_FK1'=(KeyField_Value),
                -op='cn', 'emailStatus'= 'error',
                -op='gt', 'attempts'=0,
                -op='lt', 'attempts'=5,
                -op='gt', 'timestamp'=$cutoff_time,
                -returnField='ID';
                Var: 'shouldtry' = Found_Count;
                        Records;
                                Inline: -username=$user, -password=$passw, -database='parts', -table='participants', -Update, -KeyField='ID', -KeyValue=(KeyField_Value), 'emailStatus'='queued';
                                /Inline;
                        /Records;
                /Inline;
                If: $shouldtry;
                        Event_Schedule: -URL='http://' + $servername + '/author/process/sendEmail.lasso?user=' + $user + '&passw=' + $passw + '&survey_id=' + (KeyField_Value) + '&action=requeAll;
                /If;
/Records;
/Inline;
?>

The event scheduled at the end of the code above basically calls the email sending routine at the top again, which attempts to send the messages, and increase the counter again...
Well, the event calls sendEmail.lasso, which does more: it recreates the email by getting the message text itself from the customer record, and recipient's data from its record, etc, but it *does* contain the first two lines of Lasso code at the start.

And that's it. Not too complex, and remarkably robust. The main thing to remember, on the logic side, is to write a 'queued' status to the recipient record as soon as possible, when an email for that recipient is picked up by some code that will try to send an email, to never let the code create a new email for a recipient that have a 'queued' status, and to only write another status once sending has been attempted.

Lars
#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Mail Queue Replacement for 8.6?

Carl Ketterling
In reply to this post by Mason Miller
Why would you want to not use Lasso's mail queue?


On 2/4/13 12:01 PM, "Mason Miller" <[hidden email]> wrote:

> A couple of times over the past year or so, people have mentioned replacements
> for Lasso's mail queue. Does anyone have one I can start with, even if it is
> not well documented, etc?
>
> Thanks,
>
> Mason


#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Mail Queue Replacement for 8.6?

Marc Pope-2
Large volumes of email Lasso chokes...

I use BulkMail Pro and PowerMTA, it can send 5mil per hour +

PowerMta accepts various methods of email injection to speed up email creation.

Marc

On Feb 6, 2013, at 5:52 PM, Carl Ketterling <[hidden email]> wrote:

> Why would you want to not use Lasso's mail queue?
>
>
> On 2/4/13 12:01 PM, "Mason Miller" <[hidden email]> wrote:
>
>> A couple of times over the past year or so, people have mentioned replacements
>> for Lasso's mail queue. Does anyone have one I can start with, even if it is
>> not well documented, etc?
>>
>> Thanks,
>>
>> Mason
>
>
> #############################################################
> This message is sent to you because you are subscribed to
>  the mailing list Lasso
> [hidden email]
> To unsubscribe, E-mail to: <[hidden email]>
> Send administrative queries to  <[hidden email]>
#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Mail Queue Replacement for 8.6?

Marc Vos
In reply to this post by Lars A. Gundersen-2
Wow! Thanks for the insight! This gives me new ideas!

- -
Marc

Sent from my iPhone

On 6 feb. 2013, at 23:38, "Lars A. Gundersen" <[hidden email]> wrote:

>
> On 04 Feb 2013, at 19:01, Mason Miller <[hidden email]> wrote:
>
>> A couple of times over the past year or so, people have mentioned replacements for Lasso's mail queue. Does anyone have one I can start with, even if it is not well documented, etc?
>> Thanks,
>> Mason
>
> Hi,
>
> I think many have ended up rolling their own custom solution, but maybe those, like mine, are to closely tied into the workflow of their own code and setup to be easy to package as a stand-alone email queue replacement.
> However, having complained about it for so long, I feel a certain responsibility to share info about how i did it, maybe it'll help you and others along.
>
> The built-in queue basically maintains copies of any email you send in a separate table, with fields for ID, counter with number of sending attempts, status, etc, as well as the message itself. Then there is a script that runs periodically and try to send the messages. If it manages, good, if not, the message is left in queue and the counter increases, until it is either sent or the counter gets to a predetermined number, whereupon it is marked as failed.
> This is the baseline, it is what I had to replicate.
>
> So, turns out, I and maybe you, have lots of the stuff that gets shoveled into this queue elsewhere already. I have a customer table where the email message text gets stored anyway, togheter with other data for the customers. And I maintain records of recipients of emails for other reasons than the email. Add an integer counter column to that table, and a varchar status column and a timestamp, and I had what I needed - no need to shuffle the outgoing emails into its own table.
>
> When customers press send email, I use the SMTP tags to send. No email_send at all. Just construct the email pieces from the info in the customer's record with
>
> Var: 'msg' = (Email_Compose: -To=$to_emailaddress, -From=$sender, -Subject=$thisSubject, -Body=$textBody, -Html=(Include: '/public/resources/email_template.lasso'), -HTMLImages=(Array: '/author/files/' + ($user) + '/' + $headerimage));
>
> You can disregard the HTML parts, I use an hetml template as indicated. This template can contan Lasso code and can use variables form your current page, so it can produce nice emails.
>
> Then I send to a custom tag:
>
> bt_send_email:  -emailaddress=$thisEmail, -messagepart=$msg;
>
> and here is bt_send_email:
>
> <?LassoScript
>    define_tag: 'bt_send_email',
>        -required='emailaddress',
>        -required='messagepart';
>        
>        Local: 'willsend' = true;
>        Local: 'hassent' = false;
>        Local: 'errormsg' = '';
>        Local: 'domain' = #emailaddress->(Split: '@')->Last;
>        local: 'SMTP' = (Email_SMTP);
>        
>        #domain = Email_MXLookup(#domain);
>        If: !(#domain->(isa('map')));
>            #errormsg = 'MX LOOKUP ERROR: No SMTP host found for direct send to ' + #emailaddress->(Split: '@')->Last;
>            #willsend = false;
>        Else;
>            #domain = (#domain)->Find('host');
>            If: #domain == '';
>                #errormsg = 'MX LOOKUP ERROR: No SMTP host found for direct send to ' + #emailaddress->(Split: '@')->Last;
>                #willsend = false;
>            /If;
>        /If;
>        If: #willsend;
>            Protect;
>                #SMTP->(Open: -Host=#domain,-port=25);
>                handle_error;
>                    #willsend = false;
>                    #errormsg = 'SMTP OPEN ERROR: ' + Error_CurrentError;
>                /handle_error;
>            /Protect;
>            If: #willsend;
>                Protect;
>                    #hassent = #SMTP->(Send: -Message=#messagepart->data, -From='[hidden email]', -Recipients=#messagepart->recipients);
>                    handle_error;
>                        #willsend = false;
>                        #errormsg = 'SMTP SEND ERROR: ' + Error_CurrentError;
>                    /handle_error;
>                /Protect;
>            /If;
>            #SMTP->(Close);
>        /If;
>        Local: 'sentTime' = (date)->(Format: '%Q %T');
>        If: #hassent;
>            // write to the recipient table with status of sent and a timestamp
>        Else if: #errormsg;
>            //Write to the recipient table with status of the error messages, a timestamp and increase the value of the counter that counts sending attempts, and write that too.
>        Else;
>            //Should never end up here....
>        /If;
>    /define_tag;
> ?>
>
> That's the sending and the structures needed for storing all necessary bits of info about the email. The only thing left is the script that periodically tries to send emails that don't get sucessfully sent the first time. I have this Lasso code excecuted in the event system every 15 minutes. Notice that this isn't just for "error recovery", as it were. A lot of mail servers are sat up to bounce messages first time they come from an unknown server/sender. They will subsequently accept the same message a few minutes later. This is known as greylisting, and 15 minutes seemed like a good compromise.
>
> <?Lassoscript
> var: 'user' = (Action_Param: 'username'); // 'xxx';
> var: 'passw' = (Action_Param: 'pass'); //'xxx';
> var: 'failed' = false;
> var: 'cutoff_date' = Date_Format( Date_Subtract( Date, -Month=6), -Format='%Q %T');
> var: 'cutoff_time' = Date_Format( Date_Subtract( Date, -Day=6), -Format='%Q %T');
>
>
> // I have a structure for surveys, you might not, but it could be 'member lists' or whatever group the customer wants to send mass emails to.
>
> // Do this survey by survey, far all surveys started the last 1/2 year
>
> Inline: -username='xxx', -password='xxx"', -database='surveys', -table='survey_defs', -Search, -maxrecords='all', -KeyField='ID', ...... , -op='gt', 'invitationTime'=$cutoff_date;
> Records;
>    
>    //Then we iterate over the survey's participants, set all to queued again and send them off to the sending system.
>    //The sending system will attempt to send and incerase the count, so that we only try maximum 5 times.
>    
>        Inline: -username='xxx', -password='xxx"', -database='parts', -table='participants', -Search, -MaxRecords='all', -KeyField='ID', -op='eq', 'survey_FK1'=(KeyField_Value),                        
>        -op='cn', 'emailStatus'= 'error',
>        -op='gt', 'attempts'=0,
>        -op='lt', 'attempts'=5,
>        -op='gt', 'timestamp'=$cutoff_time,
>        -returnField='ID';
>        Var: 'shouldtry' = Found_Count;
>            Records;
>                Inline: -username=$user, -password=$passw, -database='parts', -table='participants', -Update, -KeyField='ID', -KeyValue=(KeyField_Value), 'emailStatus'='queued';
>                /Inline;
>            /Records;
>        /Inline;
>        If: $shouldtry;
>            Event_Schedule: -URL='http://' + $servername + '/author/process/sendEmail.lasso?user=' + $user + '&passw=' + $passw + '&survey_id=' + (KeyField_Value) + '&action=requeAll;
>        /If;
> /Records;
> /Inline;
> ?>
>
> The event scheduled at the end of the code above basically calls the email sending routine at the top again, which attempts to send the messages, and increase the counter again...
> Well, the event calls sendEmail.lasso, which does more: it recreates the email by getting the message text itself from the customer record, and recipient's data from its record, etc, but it *does* contain the first two lines of Lasso code at the start.
>
> And that's it. Not too complex, and remarkably robust. The main thing to remember, on the logic side, is to write a 'queued' status to the recipient record as soon as possible, when an email for that recipient is picked up by some code that will try to send an email, to never let the code create a new email for a recipient that have a 'queued' status, and to only write another status once sending has been attempted.
>
> Lars
> #############################################################
> This message is sent to you because you are subscribed to
>  the mailing list Lasso
> [hidden email]
> To unsubscribe, E-mail to: <[hidden email]>
> Send administrative queries to  <[hidden email]>
#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>