Lasso 8 - optimised encode_json tag

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Lasso 8 - optimised encode_json tag

James Harvard
... or 'optimized', I suppose, if you're West of the Atlantic :-)

Anyway, I found encode_json was really slow when encoding a decent sized array of maps (output JSON aprox 125k), so I did some work on it and my modified version is >2x faster for the data I was using it with. I've submitted it as a feature request to Rhinotrac: http://www.lassosoft.com/rhinotrac?id=7750

Warning: on my test data my modified tag produced the same result as the LassoSoft one, but I haven't tested this all that extensively. Use at your own risk! I plan to replace the built-in encode_json, but if you want to run it as encode_json2 for example, don't forget to find & replace all the internal, recursive calls to encode_json.

Any further ideas for optimisation (within LassoScript, ie without using some external C or Java library) would be welcome, and could perhaps be submitted as comments to the Rhinotrac article.

James
#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Lasso 8 - optimised encode_json tag

Steve Upton

On Apr 8, 2014, at 9:01 AM, James Harvard <[hidden email]> wrote:

> ... or 'optimized', I suppose, if you're West of the Atlantic :-)
>
> Anyway, I found encode_json was really slow when encoding a decent sized array of maps (output JSON aprox 125k), so I did some work on it and my modified version is >2x faster for the data I was using it with. I've submitted it as a feature request to Rhinotrac: http://www.lassosoft.com/rhinotrac?id=7750
>
> Warning: on my test data my modified tag produced the same result as the LassoSoft one, but I haven't tested this all that extensively. Use at your own risk! I plan to replace the built-in encode_json, but if you want to run it as encode_json2 for example, don't forget to find & replace all the internal, recursive calls to encode_json.
>
> Any further ideas for optimisation (within LassoScript, ie without using some external C or Java library) would be welcome, and could perhaps be submitted as comments to the Rhinotrac article.

This is great.

Please report back how it goes in production use. We’re on Lasso 8.x and could use a JSON encoder with better performance!

regards,

Steve


#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Lasso 8 - optimised encode_json tag

fletcher sandbeck-2
I wonder if it would be faster yet if you changed #output += ‘…’ to #output->append(‘…’).

[fletcher]

On Apr 8, 2014, at 11:27 AM, Steve Upton <[hidden email]> wrote:

>
> On Apr 8, 2014, at 9:01 AM, James Harvard <[hidden email]> wrote:
>
>> ... or 'optimized', I suppose, if you're West of the Atlantic :-)
>>
>> Anyway, I found encode_json was really slow when encoding a decent sized array of maps (output JSON aprox 125k), so I did some work on it and my modified version is >2x faster for the data I was using it with. I've submitted it as a feature request to Rhinotrac: http://www.lassosoft.com/rhinotrac?id=7750
>>
>> Warning: on my test data my modified tag produced the same result as the LassoSoft one, but I haven't tested this all that extensively. Use at your own risk! I plan to replace the built-in encode_json, but if you want to run it as encode_json2 for example, don't forget to find & replace all the internal, recursive calls to encode_json.
>>
>> Any further ideas for optimisation (within LassoScript, ie without using some external C or Java library) would be welcome, and could perhaps be submitted as comments to the Rhinotrac article.
>
> This is great.
>
> Please report back how it goes in production use. We’re on Lasso 8.x and could use a JSON encoder with better performance!
>
> regards,
>
> Steve
>
>
> #############################################################
> This message is sent to you because you are subscribed to
>  the mailing list Lasso [hidden email]
> Official list archives available at http://www.lassotalk.com
> To unsubscribe, E-mail to: <[hidden email]>
> Send administrative queries to  <[hidden email]>

#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Lasso 8 - optimised encode_json tag

James Harvard
I haven't tested $out->append within the context of the tag, but I did test string->append versus string+= and the latter came out faster. (I've got a little test page thingy with four inputs: init code (define vars etc); code variant #1; code variant #2; number of iterations to perform. It then reports the number of ms each code variant takes to execute N times.)

<?lassoscript

var('out');
var('i');
var('alphabet') = 'abcdefghijklmnopqrstuvwxyz';

$out = string;
iterate($alphabet,$i);
        $out += $i; // seems aprox one third slower to use $out->append($i);
/iterate;

?>

I did test pretty much every coding syntax choice possible (albeit in isolation, rather than in the tag) :-)

By far the biggest optimisation though, and the reason I started on it, was compiling the regexp pattern once only (ie outside the iteration). Interestingly match_regexp was noticeably slower than regexp->matches when iterating; my guess is that match_regexp compiles its regexp for each comparison, or something like that.

I just took it as read that iterate($thing) was faster than loop($thing->size). Because references, and we've always known that's a better way to do it. Uuh, well, I just tested it. It's not. *confused face* *goes off to dig deeper*

James


On 8 Apr 2014, at 19:49, Fletcher Sandbeck wrote:

> I wonder if it would be faster yet if you changed #output += ‘…’ to #output->append(‘…’).
>
> [fletcher]
>
> On Apr 8, 2014, at 11:27 AM, Steve Upton <[hidden email]> wrote:
>
>>
>> On Apr 8, 2014, at 9:01 AM, James Harvard <[hidden email]> wrote:
>>
>>> ... or 'optimized', I suppose, if you're West of the Atlantic :-)
>>>
>>> Anyway, I found encode_json was really slow when encoding a decent sized array of maps (output JSON aprox 125k), so I did some work on it and my modified version is >2x faster for the data I was using it with. I've submitted it as a feature request to Rhinotrac: http://www.lassosoft.com/rhinotrac?id=7750
>>>
>>> Warning: on my test data my modified tag produced the same result as the LassoSoft one, but I haven't tested this all that extensively. Use at your own risk! I plan to replace the built-in encode_json, but if you want to run it as encode_json2 for example, don't forget to find & replace all the internal, recursive calls to encode_json.
>>>
>>> Any further ideas for optimisation (within LassoScript, ie without using some external C or Java library) would be welcome, and could perhaps be submitted as comments to the Rhinotrac article.
>>
>> This is great.
>>
>> Please report back how it goes in production use. We’re on Lasso 8.x and could use a JSON encoder with better performance!
>>
>> regards,
>>
>> Steve
>>
>>
>> #############################################################
>> This message is sent to you because you are subscribed to
>> the mailing list Lasso [hidden email]
>> Official list archives available at http://www.lassotalk.com
>> To unsubscribe, E-mail to: <[hidden email]>
>> Send administrative queries to  <[hidden email]>
>
> #############################################################
> This message is sent to you because you are subscribed to
>  the mailing list Lasso [hidden email]
> Official list archives available at http://www.lassotalk.com
> To unsubscribe, E-mail to: <[hidden email]>
> Send administrative queries to  <[hidden email]>

#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

RE: Lasso 8 - optimised encode_json tag

Rick Draper-2
>  seems aprox one third slower to use $out->append($i);

That is interesting - in Lasso 9 it is quite the opposite.  ->append is
much, much faster.

VBR

Rick


#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Lasso 8 - optimised encode_json tag

Ke Carlton-3
In reply to this post by James Harvard
Might I suggest dumping it on http://www.lassosoft.com/tagswap or better
yet; Github as a gist.

I also roll my own for Lasso 9; however the optimisations are completely
different compared to L8x; as Rick pointed out append is much faster than
+= in L9.

Ke

On 9 April 2014 04:01, James Harvard <[hidden email]
> wrote:

> ... or 'optimized', I suppose, if you're West of the Atlantic :-)
>
> Anyway, I found encode_json was really slow when encoding a decent sized
> array of maps (output JSON aprox 125k), so I did some work on it and my
> modified version is >2x faster for the data I was using it with. I've
> submitted it as a feature request to Rhinotrac:
> http://www.lassosoft.com/rhinotrac?id=7750
>
> Warning: on my test data my modified tag produced the same result as the
> LassoSoft one, but I haven't tested this all that extensively. Use at your
> own risk! I plan to replace the built-in encode_json, but if you want to
> run it as encode_json2 for example, don't forget to find & replace all the
> internal, recursive calls to encode_json.
>
> Any further ideas for optimisation (within LassoScript, ie without using
> some external C or Java library) would be welcome, and could perhaps be
> submitted as comments to the Rhinotrac article.
>
> James
> #############################################################
> This message is sent to you because you are subscribed to
>   the mailing list Lasso [hidden email]
> Official list archives available at http://www.lassotalk.com
> To unsubscribe, E-mail to: <[hidden email]>
> Send administrative queries to  <[hidden email]>
>
#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Lasso 8 - optimised encode_json tag

James Harvard
Yes, Jono e-mailed me to say the same and I will but … *cough* … er, I kinda don't use any version control / source code management at the moment. *blushes*

So any recommendations for 'get up-to-speed with GitHub in 15 minutes' tutorials would be gratefully received.

Anyway, moving VERY swiftly on, what's with loop() ->get(loop_count) seeming faster than iterate nowadays (albeit with Lasso 8.6.3)? Seriously: I'd always thought that iterate gave a significant performance advantage because it used references etc etc.

James

On 8 Apr 2014, at 23:59, Ke Carlton wrote:

> Might I suggest dumping it on http://www.lassosoft.com/tagswap or better
> yet; Github as a gist.
>
> I also roll my own for Lasso 9; however the optimisations are completely
> different compared to L8x; as Rick pointed out append is much faster than
> += in L9.
>
> Ke
>
> On 9 April 2014 04:01, James Harvard <[hidden email]
>> wrote:
>
>> ... or 'optimized', I suppose, if you're West of the Atlantic :-)
>>
>> Anyway, I found encode_json was really slow when encoding a decent sized
>> array of maps (output JSON aprox 125k), so I did some work on it and my
>> modified version is >2x faster for the data I was using it with. I've
>> submitted it as a feature request to Rhinotrac:
>> http://www.lassosoft.com/rhinotrac?id=7750
>>
>> Warning: on my test data my modified tag produced the same result as the
>> LassoSoft one, but I haven't tested this all that extensively. Use at your
>> own risk! I plan to replace the built-in encode_json, but if you want to
>> run it as encode_json2 for example, don't forget to find & replace all the
>> internal, recursive calls to encode_json.
>>
>> Any further ideas for optimisation (within LassoScript, ie without using
>> some external C or Java library) would be welcome, and could perhaps be
>> submitted as comments to the Rhinotrac article.
>>
>> James
>> #############################################################
>> This message is sent to you because you are subscribed to
>>  the mailing list Lasso [hidden email]
>> Official list archives available at http://www.lassotalk.com
>> To unsubscribe, E-mail to: <[hidden email]>
>> Send administrative queries to  <[hidden email]>
>>
> #############################################################
> This message is sent to you because you are subscribed to
>  the mailing list Lasso [hidden email]
> Official list archives available at http://www.lassotalk.com
> To unsubscribe, E-mail to: <[hidden email]>
> Send administrative queries to  <[hidden email]>

#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Lasso 8 - optimised encode_json tag

Brad Lindsay
Iterate can make code easier to read and understand. Depending on what
you're doing, that benefit outweighs slight performance hits.

Brad

On 4/8/14, 8:33 PM, James Harvard wrote:

> Yes, Jono e-mailed me to say the same and I will but … *cough* … er, I kinda don't use any version control / source code management at the moment. *blushes*
>
> So any recommendations for 'get up-to-speed with GitHub in 15 minutes' tutorials would be gratefully received.
>
> Anyway, moving VERY swiftly on, what's with loop() ->get(loop_count) seeming faster than iterate nowadays (albeit with Lasso 8.6.3)? Seriously: I'd always thought that iterate gave a significant performance advantage because it used references etc etc.
>
> James
>
> On 8 Apr 2014, at 23:59, Ke Carlton wrote:
>
>> Might I suggest dumping it on http://www.lassosoft.com/tagswap or better
>> yet; Github as a gist.
>>
>> I also roll my own for Lasso 9; however the optimisations are completely
>> different compared to L8x; as Rick pointed out append is much faster than
>> += in L9.
>>
>> Ke
>>
>> On 9 April 2014 04:01, James Harvard<[hidden email]
>>> wrote:
>>> ... or 'optimized', I suppose, if you're West of the Atlantic :-)
>>>
>>> Anyway, I found encode_json was really slow when encoding a decent sized
>>> array of maps (output JSON aprox 125k), so I did some work on it and my
>>> modified version is>2x faster for the data I was using it with. I've
>>> submitted it as a feature request to Rhinotrac:
>>> http://www.lassosoft.com/rhinotrac?id=7750
>>>
>>> Warning: on my test data my modified tag produced the same result as the
>>> LassoSoft one, but I haven't tested this all that extensively. Use at your
>>> own risk! I plan to replace the built-in encode_json, but if you want to
>>> run it as encode_json2 for example, don't forget to find&  replace all the
>>> internal, recursive calls to encode_json.
>>>
>>> Any further ideas for optimisation (within LassoScript, ie without using
>>> some external C or Java library) would be welcome, and could perhaps be
>>> submitted as comments to the Rhinotrac article.
>>>
>>> James
>>> #############################################################
>>> This message is sent to you because you are subscribed to
>>>   the mailing list Lasso [hidden email]
>>> Official list archives available at http://www.lassotalk.com
>>> To unsubscribe, E-mail to:<[hidden email]>
>>> Send administrative queries to<[hidden email]>
>>>
>> #############################################################
>> This message is sent to you because you are subscribed to
>>   the mailing list Lasso [hidden email]
>> Official list archives available at http://www.lassotalk.com
>> To unsubscribe, E-mail to:<[hidden email]>
>> Send administrative queries to<[hidden email]>
>
> #############################################################
> This message is sent to you because you are subscribed to
>    the mailing list Lasso [hidden email]
> Official list archives available at http://www.lassotalk.com
> To unsubscribe, E-mail to:<[hidden email]>
> Send administrative queries to<[hidden email]>
#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Lasso 8 - optimised encode_json tag

Ke Carlton-3
In reply to this post by James Harvard
For gists, you just need a Github account (sign up for free). Then use the
online editor here: https://gist.github.com/

For git / svn in general there's loads of tutorials online, you can learn
about it here: http://git-scm.com/about in terms of Github you can use
there client or another one or use cli:
http://try.github.io/levels/1/challenges/1 -- no excuses get your code in
some form of version control.

I can no longer commit on L8.x as it's been so long since I've used it, but
in L9 both loop and iterate are slower than ->foreach due to the extra
overhead incurred by the niceties.

Ke

On 9 April 2014 12:33, James Harvard <[hidden email]
> wrote:

> Yes, Jono e-mailed me to say the same and I will but ... *cough* ... er, I
> kinda don't use any version control / source code management at the moment.
> *blushes*
>
> So any recommendations for 'get up-to-speed with GitHub in 15 minutes'
> tutorials would be gratefully received.
>
> Anyway, moving VERY swiftly on, what's with loop() ->get(loop_count)
> seeming faster than iterate nowadays (albeit with Lasso 8.6.3)? Seriously:
> I'd always thought that iterate gave a significant performance advantage
> because it used references etc etc.
>
> James
>
>
#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Lasso 8 - optimised encode_json tag

James Harvard
In reply to this post by Brad Lindsay
Fair point, and normally I would value readability, but in this case fractions of milliseconds count, where the code might be recursively called thousands of times over a complex data structure.

To answer my own question: a bit of testing suggests that if you only want to access each component element of the data type you're iterating just once, then you are better off with loop($thing->size) instead of iterate(). If you want to access the each element twice it doesn't make any significant difference which method you use, but for three or more uses of each element it is faster to to use iterate().

I suppose that makes sense when one thinks about it! For just one use you have a small but still unnecessary overhead of assigning the object to the iteration variable. Presumably what's happening internally is the equivalent of [$iteration = @ $thing->get(loop_count)] anyway, so quicker to cut out the middleman!

James

On 9 Apr 2014, at 01:35, Brad Lindsay wrote:

> Iterate can make code easier to read and understand. Depending on what you're doing, that benefit outweighs slight performance hits.
>
> Brad
>
> On 4/8/14, 8:33 PM, James Harvard wrote:
>> Yes, Jono e-mailed me to say the same and I will but … *cough* … er, I kinda don't use any version control / source code management at the moment. *blushes*
>>
>> So any recommendations for 'get up-to-speed with GitHub in 15 minutes' tutorials would be gratefully received.
>>
>> Anyway, moving VERY swiftly on, what's with loop() ->get(loop_count) seeming faster than iterate nowadays (albeit with Lasso 8.6.3)? Seriously: I'd always thought that iterate gave a significant performance advantage because it used references etc etc.
>>
>> James

#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>