There's something funky with the SMTP tags

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

There's something funky with the SMTP tags

Lars A. Gundersen-2
I have made a ticket with this on Lasso's bug tracker, but I'm hoping to maybe get some help and tips here, since the SMTP tags actually are open source and available in the
<Lasso App folder>/Documentation/3- Language Guide/LassoApps/Startup foleder for all to see and mess with.

I'm using this with Lasso 8.6 on Mac OS X 10.8.5 and have used the SMTP tags for quite a while now, seemingly with success. Nothing fancy, and basically you just do

local: 'SMTP' = (Email_SMTP);
#SMTP->(Open: -Host=#domain,-port=25, -timeout=40);
#hassent = #SMTP->(Send: -Message=#messagepart->data, -From='[hidden email]', -Recipients=#messagepart->recipients);

Something like that. pluss error checking and what not, it's not important here. My users use this to send emails, and the webapp that does it logs error messages that the SMTP tags report, when the sending fails.
I noticed not long ago that all the emails to a large University here failed, and the error message was the for me unusual
        [Email_SMTP] Server Error: 550 5.5.1 Protocol error
I contacted them and got a this reply:
"Your SMTP client does not follow protocol and starts the SMTP-dialogue before having recieved an 'ok' signal back. Specifically, after a 'connect' the client should wait for a '220 ...' back before issuing any more commands. You don't, your first command ('EHLO...') comes immediately, you dont wait for an answer. http://tools.ietf.org/html/rfc5321#section-4.3.1 "
I thought, 'huh', and started digging. The obvious place to look is the aforementioned SMTP tag, the file smtp.inc in that folder.

I am not quite good enough to grok everything that's going on with the self references and things like that, but it seems to me that this has to be the result of things that happen from line 121 and onwards, inside the smtp->Open tag.
The SMTP code uses the net tags, and on line 121 you have the net->connect command followed by a smtp->command call that simply waits for the 220 back, AFAICT. Then if it gets that within the timeout limit, it will send the 'EHLO' command.
The SMTP->Command tag is defined from line 262 onwards. When you do a command without any other argument, you just wait for a signal back (line 301) then you read it (line 308) into the local variable #result.

This you test for the expected return code. In the case of a connect, that would be a 220 followed by some welcome message by the mail server. If you don't see it, you set the current error message to '[Email_SMTP] Server Error: ' #result

The net tags are supposed to be blocking, i.e. they wait for a result before proceeding, unless you state non_blocking, which I don't see. Given that, it looks ok to me, I don't see how the SMTP Open tag can just continue straight from a connect to sending EHLO without waiting. But I trust the mail guys at this Uni, it's a well respected technical one.

Also, with this in mind, I went back to our databases to check if I could find cases where the SMTP Open tag reported an error, but where the recipient still obvioulsy got the email anyway, and I found many cases. Most where logged with the error message from line 304, "timed out waiting for read". I can't really parse the exact reason, but it too suggests that there's something funky with the timing inside the SMTP code, it doesn't seem to do it quite correctly.

If anyone can shed some light on this issue somehow, it would be great.

Lars

#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: There's something funky with the SMTP tags

Ke Carlton-3
Hello,

It seems to me that you're connecting directly to the destination server?
if so, I wouldn't recommend this — too many variables involved.

Send via an intermediary smtp server and let that deal with the oddities /
unavailability — something like sendmail or postfix hosted on the local
machine.

Ke




On 24 October 2013 18:01, Lars A. Gundersen <[hidden email]> wrote:

> I have made a ticket with this on Lasso's bug tracker, but I'm hoping to
> maybe get some help and tips here, since the SMTP tags actually are open
> source and available in the
> <Lasso App folder>/Documentation/3- Language Guide/LassoApps/Startup
> foleder for all to see and mess with.
>
> I'm using this with Lasso 8.6 on Mac OS X 10.8.5 and have used the SMTP
> tags for quite a while now, seemingly with success. Nothing fancy, and
> basically you just do
>
> local: 'SMTP' = (Email_SMTP);
> #SMTP->(Open: -Host=#domain,-port=25, -timeout=40);
> #hassent = #SMTP->(Send: -Message=#messagepart->data, -From='
> [hidden email]', -Recipients=#messagepart->recipients);
>
> Something like that. pluss error checking and what not, it's not important
> here. My users use this to send emails, and the webapp that does it logs
> error messages that the SMTP tags report, when the sending fails.
> I noticed not long ago that all the emails to a large University here
> failed, and the error message was the for me unusual
>         [Email_SMTP] Server Error: 550 5.5.1 Protocol error
> I contacted them and got a this reply:
> "Your SMTP client does not follow protocol and starts the SMTP-dialogue
> before having recieved an 'ok' signal back. Specifically, after a 'connect'
> the client should wait for a '220 ...' back before issuing any more
> commands. You don't, your first command ('EHLO...') comes immediately, you
> dont wait for an answer. http://tools.ietf.org/html/rfc5321#section-4.3.1"
> I thought, 'huh', and started digging. The obvious place to look is the
> aforementioned SMTP tag, the file smtp.inc in that folder.
>
> I am not quite good enough to grok everything that's going on with the
> self references and things like that, but it seems to me that this has to
> be the result of things that happen from line 121 and onwards, inside the
> smtp->Open tag.
> The SMTP code uses the net tags, and on line 121 you have the net->connect
> command followed by a smtp->command call that simply waits for the 220
> back, AFAICT. Then if it gets that within the timeout limit, it will send
> the 'EHLO' command.
> The SMTP->Command tag is defined from line 262 onwards. When you do a
> command without any other argument, you just wait for a signal back (line
> 301) then you read it (line 308) into the local variable #result.
>
> This you test for the expected return code. In the case of a connect, that
> would be a 220 followed by some welcome message by the mail server. If you
> don't see it, you set the current error message to '[Email_SMTP] Server
> Error: ' #result
>
> The net tags are supposed to be blocking, i.e. they wait for a result
> before proceeding, unless you state non_blocking, which I don't see. Given
> that, it looks ok to me, I don't see how the SMTP Open tag can just
> continue straight from a connect to sending EHLO without waiting. But I
> trust the mail guys at this Uni, it's a well respected technical one.
>
> Also, with this in mind, I went back to our databases to check if I could
> find cases where the SMTP Open tag reported an error, but where the
> recipient still obvioulsy got the email anyway, and I found many cases.
> Most where logged with the error message from line 304, "timed out waiting
> for read". I can't really parse the exact reason, but it too suggests that
> there's something funky with the timing inside the SMTP code, it doesn't
> seem to do it quite correctly.
>
> If anyone can shed some light on this issue somehow, it would be great.
>
> Lars
>
> #############################################################
> This message is sent to you because you are subscribed to
>   the mailing list Lasso
> [hidden email]
> To unsubscribe, E-mail to: <[hidden email]>
> Send administrative queries to  <[hidden email]>
>

#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: There's something funky with the SMTP tags

Lars A. Gundersen-2
Hi,

I've had that discussion many times and the fact is I do, and I must, if I use Lasso for sending at all. There are indeed a alot else to be said about sending email from Lasso, but this was about the SMTP tags and the possibility that they might not adhere to the SMTP protocol. It's an issue that I hope it's interesting to discuss - there's nothing magical about sending email directly from Lasso after all. It works great - when it works.

Lars

On 24. okt. 2013, at 19:40, Ke Carlton <[hidden email]> wrote:

> Hello,
>
> It seems to me that you're connecting directly to the destination server?
> if so, I wouldn't recommend this — too many variables involved.
>
> Send via an intermediary smtp server and let that deal with the oddities /
> unavailability — something like sendmail or postfix hosted on the local
> machine.
>
> Ke
>
>
>
>
> On 24 October 2013 18:01, Lars A. Gundersen <[hidden email]> wrote:
>
>> I have made a ticket with this on Lasso's bug tracker, but I'm hoping to
>> maybe get some help and tips here, since the SMTP tags actually are open
>> source and available in the
>> <Lasso App folder>/Documentation/3- Language Guide/LassoApps/Startup
>> foleder for all to see and mess with.
>>
>> I'm using this with Lasso 8.6 on Mac OS X 10.8.5 and have used the SMTP
>> tags for quite a while now, seemingly with success. Nothing fancy, and
>> basically you just do
>>
>> local: 'SMTP' = (Email_SMTP);
>> #SMTP->(Open: -Host=#domain,-port=25, -timeout=40);
>> #hassent = #SMTP->(Send: -Message=#messagepart->data, -From='
>> [hidden email]', -Recipients=#messagepart->recipients);
>>
>> Something like that. pluss error checking and what not, it's not important
>> here. My users use this to send emails, and the webapp that does it logs
>> error messages that the SMTP tags report, when the sending fails.
>> I noticed not long ago that all the emails to a large University here
>> failed, and the error message was the for me unusual
>>       [Email_SMTP] Server Error: 550 5.5.1 Protocol error
>> I contacted them and got a this reply:
>> "Your SMTP client does not follow protocol and starts the SMTP-dialogue
>> before having recieved an 'ok' signal back. Specifically, after a 'connect'
>> the client should wait for a '220 ...' back before issuing any more
>> commands. You don't, your first command ('EHLO...') comes immediately, you
>> dont wait for an answer. http://tools.ietf.org/html/rfc5321#section-4.3.1"
>> I thought, 'huh', and started digging. The obvious place to look is the
>> aforementioned SMTP tag, the file smtp.inc in that folder.
>>
>> I am not quite good enough to grok everything that's going on with the
>> self references and things like that, but it seems to me that this has to
>> be the result of things that happen from line 121 and onwards, inside the
>> smtp->Open tag.
>> The SMTP code uses the net tags, and on line 121 you have the net->connect
>> command followed by a smtp->command call that simply waits for the 220
>> back, AFAICT. Then if it gets that within the timeout limit, it will send
>> the 'EHLO' command.
>> The SMTP->Command tag is defined from line 262 onwards. When you do a
>> command without any other argument, you just wait for a signal back (line
>> 301) then you read it (line 308) into the local variable #result.
>>
>> This you test for the expected return code. In the case of a connect, that
>> would be a 220 followed by some welcome message by the mail server. If you
>> don't see it, you set the current error message to '[Email_SMTP] Server
>> Error: ' #result
>>
>> The net tags are supposed to be blocking, i.e. they wait for a result
>> before proceeding, unless you state non_blocking, which I don't see. Given
>> that, it looks ok to me, I don't see how the SMTP Open tag can just
>> continue straight from a connect to sending EHLO without waiting. But I
>> trust the mail guys at this Uni, it's a well respected technical one.
>>
>> Also, with this in mind, I went back to our databases to check if I could
>> find cases where the SMTP Open tag reported an error, but where the
>> recipient still obvioulsy got the email anyway, and I found many cases.
>> Most where logged with the error message from line 304, "timed out waiting
>> for read". I can't really parse the exact reason, but it too suggests that
>> there's something funky with the timing inside the SMTP code, it doesn't
>> seem to do it quite correctly.
>>
>> If anyone can shed some light on this issue somehow, it would be great.
>>
>> Lars
>>
>> #############################################################
>> This message is sent to you because you are subscribed to
>> the mailing list Lasso
>> [hidden email]
>> To unsubscribe, E-mail to: <[hidden email]>
>> Send administrative queries to  <[hidden email]>
>>
>
> #############################################################
> This message is sent to you because you are subscribed to
> the mailing list Lasso
> [hidden email]
> To unsubscribe, E-mail to: <[hidden email]>
> Send administrative queries to  <[hidden email]>
>


#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: There's something funky with the SMTP tags

Seth Ganahl
In reply to this post by Lars A. Gundersen-2
I think you may be looking for something like this:

local: 'Result'=#myConnection->(Wait: 60, Net_WaitWrite);
        local: 'Capabilities' = #myConnection->(Read: 32768);
        Fail_If: #Capabilities !>> '220', -1, 'Timed Out Waiting for
Connection';

Disclaimer...this is taken from my send_ssl code, written before Lasso had
the ability to send email via ssl, and I have not looked at the SMTP tags in
quite a while.


On 10/24/13 12:01 PM, "Lars A. Gundersen" <[hidden email]> did
quoth:

> I have made a ticket with this on Lasso's bug tracker, but I'm hoping to maybe
> get some help and tips here, since the SMTP tags actually are open source and
> available in the
> <Lasso App folder>/Documentation/3- Language Guide/LassoApps/Startup foleder
> for all to see and mess with.
[snip]
-------------------------------------------------------
Seth C Ganahl (501) 282-4867
Ganahl Consulting ­ Web Applications
http://www.ganahlconsulting.com/
[hidden email]
-------------------------------------------------------



#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: There's something funky with the SMTP tags

Steve Upton
In reply to this post by Lars A. Gundersen-2

On Oct 24, 2013, at 12:17 PM, Lars A. Gundersen <[hidden email]> wrote:

> I've had that discussion many times and the fact is I do, and I must, if I use Lasso for sending at all. There are indeed a alot else to be said about sending email from Lasso, but this was about the SMTP tags and the possibility that they might not adhere to the SMTP protocol. It's an issue that I hope it's interesting to discuss - there's nothing magical about sending email directly from Lasso after all. It works great - when it works.

I think the problem lies in the "implementation" of the SMTP protocol.

I'm not an authority on this but I recall reading on another list somewhere a response to someone wanting to implement their own email client.

The response was "good luck", The UI part would be trivial compared to the myriad implementation flavors of SMTP on different email servers.

just my 2 bits

It's why we forward all outgoing SMTP mail to a local server. But you do lose some of the ability to detect mail delivery problems within your app (actual errors returned from servers). You have to deal with the resulting bounces which can be a pain as *they* also never adhere to a standard error-reporting format.

regards,

Steve


#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: There's something funky with the SMTP tags

fletcher sandbeck-2
On Oct 28, 2013, at 12:24 PM, Steve Upton <[hidden email]> wrote:

> It's why we forward all outgoing SMTP mail to a local server. But you do lose some of the ability to detect mail delivery problems within your app (actual errors returned from servers). You have to deal with the resulting bounces which can be a pain as *they* also never adhere to a standard error-reporting format.

This should always be the preferred configuration for Lasso.  Sending direct to SMTP servers allows email to be sent prior to Lasso be completely configured and in situations where it is not possible to setup a local SMTP server for relay.

[fletcher]


#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: There's something funky with the SMTP tags

Lars A. Gundersen-2
In reply to this post by Steve Upton
On 28. okt. 2013, at 20:24, Steve Upton <[hidden email]> wrote:

> I think the problem lies in the "implementation" of the SMTP protocol.

Yes, that is what I'm suggesting! And Lasso's (8.6) implementation of the SMTP protocoll is open source and can be found in one file, the one I referenced in my email, therefore I and anybody else can read it and I was hoping someone could see if what I reported could be the case and if so how it might be mitigated.

> I'm not an authority on this but I recall reading on another list somewhere a response to someone wanting to implement their own email client.
>
> The response was "good luck", The UI part would be trivial compared to the myriad implementation flavors of SMTP on different email servers.

Yes, but I'm not trying to implement an email client. I'm just trying to find out if there could be a discrepancy between the SMTP standard and Lasso's implementation of it. A discrepancy that, while part of the protocol, many servers ignore and therefore is not widely known.
An email client has to try to make sense of POP and IMAP, text encoding and rich text and HTML formatting. THAT must be a pain.
The SMTP protocoll is small and neat in comparison. Lasso's implementation is one document of 400 lines of Lassoscript. I was simply hoping if someone could look at that specific code and see if and how it could be that the SMTP->Open tag would do a connect and then immediately send 'EHLO...' without waiting for the '220...' acknowledgement, and if there was any way around it.

Lars

#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: There's something funky with the SMTP tags

Lars A. Gundersen-2
In reply to this post by fletcher sandbeck-2
On 28. okt. 2013, at 21:13, Fletcher Sandbeck <[hidden email]> wrote:

> On Oct 28, 2013, at 12:24 PM, Steve Upton <[hidden email]> wrote:
>
>> It's why we forward all outgoing SMTP mail to a local server. But you do lose some of the ability to detect mail delivery problems within your app (actual errors returned from servers). You have to deal with the resulting bounces which can be a pain as *they* also never adhere to a standard error-reporting format.
>
> This should always be the preferred configuration for Lasso.  Sending direct to SMTP servers allows email to be sent prior to Lasso be completely configured and in situations where it is not possible to setup a local SMTP server for relay.

Why? I don't want to come off as cranky, but I've implemented an outgoing email sending/queue system based on the SMTP tags that works great! The ONLY problems I have are the stuck email queue and now this new kink, that affects just a small percentage of sends.

Yet, when I try to ask about a possible fix for this, which also has the potential to make Lasso's SMTP implementation better, the majority of responses are basically that I should run screaming from Lasso when it comes to actually using one of its features.
Yes, I could use Mandrill or something similar, and I might do that going forward, but for now this is what I do, and I know why I'm doing it, it's an informed choice.

Lars

#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: There's something funky with the SMTP tags

Steve Upton
In reply to this post by Lars A. Gundersen-2

On Oct 28, 2013, at 1:24 PM, Lars A. Gundersen <[hidden email]> wrote:

>> I think the problem lies in the "implementation" of the SMTP protocol.
>
> Yes, that is what I'm suggesting! And Lasso's (8.6) implementation of the SMTP protocoll is open source and can be found in one file, the one I referenced in my email, therefore I and anybody else can read it and I was hoping someone could see if what I reported could be the case and if so how it might be mitigated.

I'm sorry, I guess I was unclear in my response.

I didn't mean Lasso's implementation of SMTP. I meant the implementation of SMTP in the many different SMTP servers out there.

I'm not trying to defend Lasso's implementation, just saying that the world's SMTP implementations are a moving target.

Steve



#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>