keeping a process running in Lasso 8.5

classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

keeping a process running in Lasso 8.5

John Morris-3
Hey All,

I was chatting w/ Bil at the conference, and he shared with me the way he would do this, I wrote down the tags he mentioned, and I can't find them now!

So, I'll ask again :)

I have a task that takes a while to run.
Every set of 50 records I need to process can take anywhere from 10 seconds to over 2 minutes.

I want to schedule this process, but timing is bad.
Scheduling it every 3 minutes, is a waist of time, but some times it needs that much time.


Bill mentioned making my page run indefinitely in a loop.

With the combination of Lasso_ExecutionTimeLimit, and Thread_Priority_High, and ….. some_other_tag_I_cant_remember

And code example would also be helpful.

Thanks

 - John Morris


#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: keeping a process running in Lasso 8.5

John Morris-3
Make that Lasso 9, not 8.5
Any code example in L9 would be great :)

 - John Morris

On Sep 30, 2013, at 2:28 PM, John Morris <[hidden email]> wrote:

> Hey All,
>
> I was chatting w/ Bil at the conference, and he shared with me the way he would do this, I wrote down the tags he mentioned, and I can't find them now!
>
> So, I'll ask again :)
>
> I have a task that takes a while to run.
> Every set of 50 records I need to process can take anywhere from 10 seconds to over 2 minutes.
>
> I want to schedule this process, but timing is bad.
> Scheduling it every 3 minutes, is a waist of time, but some times it needs that much time.
>
>
> Bill mentioned making my page run indefinitely in a loop.
>
> With the combination of Lasso_ExecutionTimeLimit, and Thread_Priority_High, and ….. some_other_tag_I_cant_remember
>
> And code example would also be helpful.
>
> Thanks
>
> - John Morris
>
>
> #############################################################
> This message is sent to you because you are subscribed to
>  the mailing list Lasso
> [hidden email]
> To unsubscribe, E-mail to: <[hidden email]>
> Send administrative queries to  <[hidden email]>


#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: keeping a process running in Lasso 8.5

Brad Lindsay
Not sure I understand - when do you need to schedule it to run? Is it
that a page takes a long time to load and you'd like to just let it do
its task in the background? Or is there some sort of queue that you have
that needs to be checked periodically for work to do? A little more
details on the setup would be helpful

Thanks,
Brad

On 9/30/13, 3:33 PM, John Morris wrote:

> Make that Lasso 9, not 8.5
> Any code example in L9 would be great :)
>
>   - John Morris
>
> On Sep 30, 2013, at 2:28 PM, John Morris<[hidden email]>  wrote:
>
>> Hey All,
>>
>> I was chatting w/ Bil at the conference, and he shared with me the way he would do this, I wrote down the tags he mentioned, and I can't find them now!
>>
>> So, I'll ask again :)
>>
>> I have a task that takes a while to run.
>> Every set of 50 records I need to process can take anywhere from 10 seconds to over 2 minutes.
>>
>> I want to schedule this process, but timing is bad.
>> Scheduling it every 3 minutes, is a waist of time, but some times it needs that much time.
>>
>>
>> Bill mentioned making my page run indefinitely in a loop.
>>
>> With the combination of Lasso_ExecutionTimeLimit, and Thread_Priority_High, and ….. some_other_tag_I_cant_remember
>>
>> And code example would also be helpful.
>>
>> Thanks
>>
>> - John Morris
>>
>>
>> #############################################################
>> This message is sent to you because you are subscribed to
>>   the mailing list Lasso
>> [hidden email]
>> To unsubscribe, E-mail to:<[hidden email]>
>> Send administrative queries to<[hidden email]>
>
>
> #############################################################
> This message is sent to you because you are subscribed to
>    the mailing list Lasso
> [hidden email]
> To unsubscribe, E-mail to:<[hidden email]>
> Send administrative queries to<[hidden email]>

#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: keeping a process running in Lasso 8.5

John Morris-3
I need to run a section of code on 25 million records.

So, I currently have a page that finds the 25 million records, and then 50 at a time, I run some code, preforming more searched and updates and other stuff.
Depending on other data for the found set, this set of 50 records could be done in 10 seconds, or it could be done in 2 minutes.

I currently have it scheduled to run my page of 50 records every 3 minutes.
What I want to change is to make it run, and it will "Crunch" that set of 50, then once it's done, not sit still for 2 minutes & 50 seconds, but to start on the next set.

Does that explain it a bit better?

 - John Morris

On Sep 30, 2013, at 2:37 PM, Brad Lindsay <[hidden email]> wrote:

> Not sure I understand - when do you need to schedule it to run? Is it that a page takes a long time to load and you'd like to just let it do its task in the background? Or is there some sort of queue that you have that needs to be checked periodically for work to do? A little more details on the setup would be helpful
>
> Thanks,
> Brad
>
> On 9/30/13, 3:33 PM, John Morris wrote:
>> Make that Lasso 9, not 8.5
>> Any code example in L9 would be great :)
>>
>>  - John Morris
>>
>> On Sep 30, 2013, at 2:28 PM, John Morris<[hidden email]>  wrote:
>>
>>> Hey All,
>>>
>>> I was chatting w/ Bil at the conference, and he shared with me the way he would do this, I wrote down the tags he mentioned, and I can't find them now!
>>>
>>> So, I'll ask again :)
>>>
>>> I have a task that takes a while to run.
>>> Every set of 50 records I need to process can take anywhere from 10 seconds to over 2 minutes.
>>>
>>> I want to schedule this process, but timing is bad.
>>> Scheduling it every 3 minutes, is a waist of time, but some times it needs that much time.
>>>
>>>
>>> Bill mentioned making my page run indefinitely in a loop.
>>>
>>> With the combination of Lasso_ExecutionTimeLimit, and Thread_Priority_High, and ….. some_other_tag_I_cant_remember
>>>
>>> And code example would also be helpful.
>>>
>>> Thanks
>>>
>>> - John Morris
>>>
>>>
>>> #############################################################
>>> This message is sent to you because you are subscribed to
>>>  the mailing list Lasso
>>> [hidden email]
>>> To unsubscribe, E-mail to:<[hidden email]>
>>> Send administrative queries to<[hidden email]>
>>
>>
>> #############################################################
>> This message is sent to you because you are subscribed to
>>   the mailing list Lasso
>> [hidden email]
>> To unsubscribe, E-mail to:<[hidden email]>
>> Send administrative queries to<[hidden email]>
>
> #############################################################
> This message is sent to you because you are subscribed to
> the mailing list Lasso
> [hidden email]
> To unsubscribe, E-mail to: <[hidden email]>
> Send administrative queries to  <[hidden email]>


#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: keeping a process running in Lasso 8.5

fletcher sandbeck-2

On Sep 30, 2013, at 12:46 PM, John Morris <[hidden email]> wrote:

> I need to run a section of code on 25 million records.
>
> So, I currently have a page that finds the 25 million records, and then 50 at a time, I run some code, preforming more searched and updates and other stuff.
> Depending on other data for the found set, this set of 50 records could be done in 10 seconds, or it could be done in 2 minutes.
>
> I currently have it scheduled to run my page of 50 records every 3 minutes.
> What I want to change is to make it run, and it will "Crunch" that set of 50, then once it's done, not sit still for 2 minutes & 50 seconds, but to start on the next set.
>
> Does that explain it a bit better?

One pattern I've used is to call a page from a shell script running curl.  The page returns a 302 redirect to the next batch so that each time it finishes it gets redirected to the next part of the job.

Here's the curl call.  The -L flag tells curl to follow redirects if they occur, -s makes curl silent, and -S tells it report errors, -o throws away the output, and --max-redirs sets the maximum number of redirects to follow.  -I (capital eye) tells curl to ask for the HEAD only, but for these purposes is probably not important.

curl --max-redirs 1000 -ILsS -o /dev/null "http://www.example.com/scripts/myscript.lasso?batch=true"

The Lasso code at the bottom of the page looks like this.  If the script page sees the batch parameter then it returns a 302 redirect to the next batch.  I give it a random value as a cache buster.

if(action_param('batch', -count) > 0);
  sleep(50);
  redirect_url('http://wwwsrc.thriftyfun.com/scripts/myscript.lasso?batch=' + math_random(-lower=100000,-upper=999999));
/if;

I think in practice a thousand is about the maximum number of redirects curl will follow.  You can set the --max-redirs higher, but it seems to fail out somewhere around there no matter what it is set to.  And of course the script will fail altogether if one of the pages has a Lasso error and can't issue the 302 redirect.  Cron can help with that.

[fletcher]


#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: keeping a process running in Lasso 9

Brad Lindsay
In reply to this post by John Morris-3
Make sense, but still not sure exactly what's going on. Do you want it
to never stop crunching 50 records at a time? Once it finishes all 25
million, start again? Or is it a one and done deal?

I would have it run in it's own thread. For Lasso 9, I would either use
a thread object
(http://lassoguide.com/language/threading.html#thread-objects) to force
the code to only be run in one thread, or use [split_thread]
(http://lassoguide.com/language/threading.html#splitting-threads) to
spawn off as many concurrent threads of the code as you need.

The basic concept is to call something that sets up the work to be run
in the background. The trick is to recover gracefully from crashes.

Hopefully this points you in the right direction, but I'm sure it raises
some questions too.

Brad


On 9/30/13, 3:46 PM, John Morris wrote:

> I need to run a section of code on 25 million records.
>
> So, I currently have a page that finds the 25 million records, and then 50 at a time, I run some code, preforming more searched and updates and other stuff.
> Depending on other data for the found set, this set of 50 records could be done in 10 seconds, or it could be done in 2 minutes.
>
> I currently have it scheduled to run my page of 50 records every 3 minutes.
> What I want to change is to make it run, and it will "Crunch" that set of 50, then once it's done, not sit still for 2 minutes&  50 seconds, but to start on the next set.
>
> Does that explain it a bit better?
>
>   - John Morris
>
> On Sep 30, 2013, at 2:37 PM, Brad Lindsay<[hidden email]>  wrote:
>
>> Not sure I understand - when do you need to schedule it to run? Is it that a page takes a long time to load and you'd like to just let it do its task in the background? Or is there some sort of queue that you have that needs to be checked periodically for work to do? A little more details on the setup would be helpful
>>
>> Thanks,
>> Brad
>>
>> On 9/30/13, 3:33 PM, John Morris wrote:
>>> Make that Lasso 9, not 8.5
>>> Any code example in L9 would be great :)
>>>
>>>   - John Morris
>>>
>>> On Sep 30, 2013, at 2:28 PM, John Morris<[hidden email]>   wrote:
>>>
>>>> Hey All,
>>>>
>>>> I was chatting w/ Bil at the conference, and he shared with me the way he would do this, I wrote down the tags he mentioned, and I can't find them now!
>>>>
>>>> So, I'll ask again :)
>>>>
>>>> I have a task that takes a while to run.
>>>> Every set of 50 records I need to process can take anywhere from 10 seconds to over 2 minutes.
>>>>
>>>> I want to schedule this process, but timing is bad.
>>>> Scheduling it every 3 minutes, is a waist of time, but some times it needs that much time.
>>>>
>>>>
>>>> Bill mentioned making my page run indefinitely in a loop.
>>>>
>>>> With the combination of Lasso_ExecutionTimeLimit, and Thread_Priority_High, and ….. some_other_tag_I_cant_remember
>>>>
>>>> And code example would also be helpful.
>>>>
>>>> Thanks
>>>>
>>>> - John Morris
>>>>
>>>>
>>>> #############################################################
>>>> This message is sent to you because you are subscribed to
>>>>   the mailing list Lasso
>>>> [hidden email]
>>>> To unsubscribe, E-mail to:<[hidden email]>
>>>> Send administrative queries to<[hidden email]>
>>>
>>> #############################################################
>>> This message is sent to you because you are subscribed to
>>>    the mailing list Lasso
>>> [hidden email]
>>> To unsubscribe, E-mail to:<[hidden email]>
>>> Send administrative queries to<[hidden email]>
>> #############################################################
>> This message is sent to you because you are subscribed to
>> the mailing list Lasso
>> [hidden email]
>> To unsubscribe, E-mail to:<[hidden email]>
>> Send administrative queries to<[hidden email]>
>
>
> #############################################################
> This message is sent to you because you are subscribed to
>    the mailing list Lasso
> [hidden email]
> To unsubscribe, E-mail to:<[hidden email]>
> Send administrative queries to<[hidden email]>

#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: keeping a process running in Lasso 9

John Morris-3
Once I'm done with the original 25 million que, I'll have nightly updates to say anywhere from 1,000 - 10,000 to run
So I'll run those until done. then sit and wait till the next night.

I've been reading up on threads, and an a bit lost.


Lets say I want to do this:

        Server A.

                inline( -sql='mySearch',
                                -max-records=50)
       
                        // finds 25 million, grabs the 1st set of 50 to work on
       
                        do the rest of my stuff here….

                /inline;


        Server B.

                inline( -sql='mySearch',
                                -max-records=50)
       
                        // finds (25 million - the 50 Comp A grabbed), grabs the 1st set of 50 to work on
       
                        do the rest of my stuff here….

                /inline;



        Server A's set of 50 takes it 15 seconds.
        Server B's set of 50 takes it 2 minutes.

        I want Server to run again instantly, getting another set of 50 records which would be the next set of 50 after Server B.



Currently I set up a event that is ran every 3 minutes, but their is a lot of waisted time.

So how would I keep this running until the found count of the inline was 0?

Am I over thinking this?
Should I just place this code in a loop(found_count / 50 ) tag and then do the search again within the loop to see if I should about the loop?

And, how can I check to see if this code is currently running? so if it isn't I can start it again.

Thanks for all the advice
 - John Morris






On Sep 30, 2013, at 4:11 PM, Brad Lindsay <[hidden email]> wrote:

> Make sense, but still not sure exactly what's going on. Do you want it to never stop crunching 50 records at a time? Once it finishes all 25 million, start again? Or is it a one and done deal?
>
> I would have it run in it's own thread. For Lasso 9, I would either use a thread object (http://lassoguide.com/language/threading.html#thread-objects) to force the code to only be run in one thread, or use [split_thread] (http://lassoguide.com/language/threading.html#splitting-threads) to spawn off as many concurrent threads of the code as you need.
>
> The basic concept is to call something that sets up the work to be run in the background. The trick is to recover gracefully from crashes.
>
> Hopefully this points you in the right direction, but I'm sure it raises some questions too.
>
> Brad
>
>
> On 9/30/13, 3:46 PM, John Morris wrote:
>> I need to run a section of code on 25 million records.
>>
>> So, I currently have a page that finds the 25 million records, and then 50 at a time, I run some code, preforming more searched and updates and other stuff.
>> Depending on other data for the found set, this set of 50 records could be done in 10 seconds, or it could be done in 2 minutes.
>>
>> I currently have it scheduled to run my page of 50 records every 3 minutes.
>> What I want to change is to make it run, and it will "Crunch" that set of 50, then once it's done, not sit still for 2 minutes&  50 seconds, but to start on the next set.
>>
>> Does that explain it a bit better?
>>
>>  - John Morris
>>
>> On Sep 30, 2013, at 2:37 PM, Brad Lindsay<[hidden email]>  wrote:
>>
>>> Not sure I understand - when do you need to schedule it to run? Is it that a page takes a long time to load and you'd like to just let it do its task in the background? Or is there some sort of queue that you have that needs to be checked periodically for work to do? A little more details on the setup would be helpful
>>>
>>> Thanks,
>>> Brad
>>>
>>> On 9/30/13, 3:33 PM, John Morris wrote:
>>>> Make that Lasso 9, not 8.5
>>>> Any code example in L9 would be great :)
>>>>
>>>>  - John Morris
>>>>
>>>> On Sep 30, 2013, at 2:28 PM, John Morris<[hidden email]>   wrote:
>>>>
>>>>> Hey All,
>>>>>
>>>>> I was chatting w/ Bil at the conference, and he shared with me the way he would do this, I wrote down the tags he mentioned, and I can't find them now!
>>>>>
>>>>> So, I'll ask again :)
>>>>>
>>>>> I have a task that takes a while to run.
>>>>> Every set of 50 records I need to process can take anywhere from 10 seconds to over 2 minutes.
>>>>>
>>>>> I want to schedule this process, but timing is bad.
>>>>> Scheduling it every 3 minutes, is a waist of time, but some times it needs that much time.
>>>>>
>>>>>
>>>>> Bill mentioned making my page run indefinitely in a loop.
>>>>>
>>>>> With the combination of Lasso_ExecutionTimeLimit, and Thread_Priority_High, and ….. some_other_tag_I_cant_remember
>>>>>
>>>>> And code example would also be helpful.
>>>>>
>>>>> Thanks
>>>>>
>>>>> - John Morris


#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: keeping a process running in Lasso 9

Brad Lindsay
On 10/1/13, 9:02 AM, John Morris wrote:

> Once I'm done with the original 25 million que, I'll have nightly updates to say anywhere from 1,000 - 10,000 to run
> So I'll run those until done. then sit and wait till the next night.
>
> I've been reading up on threads, and an a bit lost.
>
>
> Lets say I want to do this:
>
> Server A.
>
> inline( -sql='mySearch',
> -max-records=50)
>
> // finds 25 million, grabs the 1st set of 50 to work on
>
> do the rest of my stuff here….
>
> /inline;
>
>
> Server B.
>
> inline( -sql='mySearch',
> -max-records=50)
>
> // finds (25 million - the 50 Comp A grabbed), grabs the 1st set of 50 to work on
>
> do the rest of my stuff here….
>
> /inline;
>
>
>
> Server A's set of 50 takes it 15 seconds.
> Server B's set of 50 takes it 2 minutes.
>
> I want Server to run again instantly, getting another set of 50 records which would be the next set of 50 after Server B.
>
>

Adding multiple servers into the mix complicates things. Essentially you
need a centralized way to track which tasks have been done or which are
left to do. And you're probably going to need to investigate MySQL's
locking mechanisms to do it.



> So how would I keep this running until the found count of the inline was 0?
>
> Am I over thinking this?
> Should I just place this code in a loop(found_count / 50 ) tag and then do the search again within the loop to see if I should about the loop?

The other alternative is to stick it in a [while(true) => {}] and then
use [loop_abort] when it meets the conditions to abort. Be careful to
actually have it abort!


>
> And, how can I check to see if this code is currently running? so if it isn't I can start it again.

It depends on how you're doing what you're doing. If you use
[split_thread], it returns a set of file descriptors that can be used to
communicate with the thread. I would stash those in a thread object and
use that thread object to determine if it's running or not.



Do you really need to be running this from a web page at all? Would it
be simpler / easier to write a command-line script rather than dealing
with threads? You could run the script as a normal background process
that you'd be able to see if it exited. If you need to run the script on
multiple servers, that's fine - you would then just need to worry about
making sure they didn't do work that was already being done.


Brad

#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: keeping a process running in Lasso 9

John Morris-3
I've been in Lasso 8 land for so long, I forgot about Lasso 9 in the command line.
That MIGHT be the route to go.

Lasso is really only used to run the MySQL statement.
The multiple server's is really needed for the MySQL part.

Do you have those examples of Lasso 9 in the command line that you and Jolle did at the LDC?
Those would really come in handy in playing with that route.

I could write my script in lasso then have cron keep it running.

 - John Morris

On Oct 1, 2013, at 8:34 AM, Brad Lindsay <[hidden email]> wrote:

> On 10/1/13, 9:02 AM, John Morris wrote:
>> Once I'm done with the original 25 million que, I'll have nightly updates to say anywhere from 1,000 - 10,000 to run
>> So I'll run those until done. then sit and wait till the next night.
>>
>> I've been reading up on threads, and an a bit lost.
>>
>>
>> Lets say I want to do this:
>>
>> Server A.
>>
>> inline( -sql='mySearch',
>> -max-records=50)
>>
>> // finds 25 million, grabs the 1st set of 50 to work on
>>
>> do the rest of my stuff here….
>>
>> /inline;
>>
>>
>> Server B.
>>
>> inline( -sql='mySearch',
>> -max-records=50)
>>
>> // finds (25 million - the 50 Comp A grabbed), grabs the 1st set of 50 to work on
>>
>> do the rest of my stuff here….
>>
>> /inline;
>>
>>
>>
>> Server A's set of 50 takes it 15 seconds.
>> Server B's set of 50 takes it 2 minutes.
>>
>> I want Server to run again instantly, getting another set of 50 records which would be the next set of 50 after Server B.
>>
>>
>
> Adding multiple servers into the mix complicates things. Essentially you need a centralized way to track which tasks have been done or which are left to do. And you're probably going to need to investigate MySQL's locking mechanisms to do it.
>
>
>
>> So how would I keep this running until the found count of the inline was 0?
>>
>> Am I over thinking this?
>> Should I just place this code in a loop(found_count / 50 ) tag and then do the search again within the loop to see if I should about the loop?
>
> The other alternative is to stick it in a [while(true) => {}] and then use [loop_abort] when it meets the conditions to abort. Be careful to actually have it abort!
>
>
>>
>> And, how can I check to see if this code is currently running? so if it isn't I can start it again.
>
> It depends on how you're doing what you're doing. If you use [split_thread], it returns a set of file descriptors that can be used to communicate with the thread. I would stash those in a thread object and use that thread object to determine if it's running or not.
>
>
>
> Do you really need to be running this from a web page at all? Would it be simpler / easier to write a command-line script rather than dealing with threads? You could run the script as a normal background process that you'd be able to see if it exited. If you need to run the script on multiple servers, that's fine - you would then just need to worry about making sure they didn't do work that was already being done.


#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: keeping a process running in Lasso 9

Bil Corry-3
My understanding is that there is a single MySQL server, and connecting to
it are multiple Lasso servers.  If so, this is how I would do it:

On each server, you drop the following script into LassoStartup (script is
pseudo-code because my Lasso is rusty - some method names may be horribly
wrong, sorry):

// --------------------------
local(threadID) = 'server4'; // make sure this value is unique per server

while(true);

// mark the 50 to process
inline(-sql='UPDATE myTable SET threadID = "'+#threadID+'", processStart =
NOW() WHERE threadID IS NULL LIMIT 50');
/inline;

// grab our 50, might be more if the process was interrupted and never
completed
inline(-sql='SELECT * FROM myTable WHERE threadID = '+#threadID);
  if(found_count > 0); // if found some to process, do it!
    rows;
      // process each row
      inline(-sql='UPDATE myTable SET threadID = "done" WHERE rowKey =
'+#rowKeyValue);
    /rows;
  else; // everything looks like it was completed, so sleep 8 hours (or can
sleep until midnight, etc)
    sleep(28800000); // 8 hours
  /if;
/inline;

/while;
// --------------------------


Some other thoughts:

You may want to create a helper script that looks at the processStart time
and if it's been longer than the max time you expect, you may want to
consider rebooting the server as the process has probably hung.

In Lasso 8, there was a Lasso_ExecutionTimeLimit tag, not sure if it exists
in Lasso 9, but the idea is you want to prevent Lasso from killing this
thread, which will run forever (hopefully).

You may want to add in sleep() after each round of 50 and/or between
processing each row.  With multiple servers all hitting the same MySQL
table, there's a chance Lasso could overwhelm it.  Even adding in a small
delay, such as 50 ms could be the difference between it running and it
bombing.  You'll have to test it.

The beauty of the above is that even if you reboot the Lasso server, it
will continue where it left off, and will continue to process until all
records are done, so even if a server blows up, the other ones will finish
it off (except for the few records that were reserved by the downed server).

Hope that clarifies what I had mentioned at LDC.

- Bil


On Tue, Oct 1, 2013 at 3:46 PM, John Morris <[hidden email]>wrote:

> I've been in Lasso 8 land for so long, I forgot about Lasso 9 in the
> command line.
> That MIGHT be the route to go.
>
> Lasso is really only used to run the MySQL statement.
> The multiple server's is really needed for the MySQL part.
>
> Do you have those examples of Lasso 9 in the command line that you and
> Jolle did at the LDC?
> Those would really come in handy in playing with that route.
>
> I could write my script in lasso then have cron keep it running.
>
>  - John Morris
>
> On Oct 1, 2013, at 8:34 AM, Brad Lindsay <[hidden email]> wrote:
>
> > On 10/1/13, 9:02 AM, John Morris wrote:
> >> Once I'm done with the original 25 million que, I'll have nightly
> updates to say anywhere from 1,000 - 10,000 to run
> >> So I'll run those until done. then sit and wait till the next night.
> >>
> >> I've been reading up on threads, and an a bit lost.
> >>
> >>
> >> Lets say I want to do this:
> >>
> >>      Server A.
> >>
> >>              inline( -sql='mySearch',
> >>                              -max-records=50)
> >>
> >>                      //      finds 25 million, grabs the 1st set of 50
> to work on
> >>
> >>                      do the rest of my stuff here….
> >>
> >>              /inline;
> >>
> >>
> >>      Server B.
> >>
> >>              inline( -sql='mySearch',
> >>                              -max-records=50)
> >>
> >>                      //      finds (25 million - the 50 Comp A
> grabbed), grabs the 1st set of 50 to work on
> >>
> >>                      do the rest of my stuff here….
> >>
> >>              /inline;
> >>
> >>
> >>
> >>      Server A's set of 50 takes it 15 seconds.
> >>      Server B's set of 50 takes it 2 minutes.
> >>
> >>      I want Server to run again instantly, getting another set of 50
> records which would be the next set of 50 after Server B.
> >>
> >>
> >
> > Adding multiple servers into the mix complicates things. Essentially you
> need a centralized way to track which tasks have been done or which are
> left to do. And you're probably going to need to investigate MySQL's
> locking mechanisms to do it.
> >
> >
> >
> >> So how would I keep this running until the found count of the inline
> was 0?
> >>
> >> Am I over thinking this?
> >> Should I just place this code in a loop(found_count / 50 ) tag and then
> do the search again within the loop to see if I should about the loop?
> >
> > The other alternative is to stick it in a [while(true) => {}] and then
> use [loop_abort] when it meets the conditions to abort. Be careful to
> actually have it abort!
> >
> >
> >>
> >> And, how can I check to see if this code is currently running? so if it
> isn't I can start it again.
> >
> > It depends on how you're doing what you're doing. If you use
> [split_thread], it returns a set of file descriptors that can be used to
> communicate with the thread. I would stash those in a thread object and use
> that thread object to determine if it's running or not.
> >
> >
> >
> > Do you really need to be running this from a web page at all? Would it
> be simpler / easier to write a command-line script rather than dealing with
> threads? You could run the script as a normal background process that you'd
> be able to see if it exited. If you need to run the script on multiple
> servers, that's fine - you would then just need to worry about making sure
> they didn't do work that was already being done.
>
>
> #############################################################
> This message is sent to you because you are subscribed to
>   the mailing list Lasso
> [hidden email]
> To unsubscribe, E-mail to: <[hidden email]>
> Send administrative queries to  <[hidden email]>
>

#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: keeping a process running in Lasso 8.5

codylentz
In reply to this post by John Morris-3
John,
This may be a relatively simple solution and doesn't account for error handling and such but why not just set a variable = 50 and then each time it processes a record decrement the variable each time the process completes. Add a conditional and when it hits 0 add 50 new records and reset the variable to 50.

On Sep 30, 2013, at 12:28 PM, John Morris <[hidden email]> wrote:

> Hey All,
>
> I was chatting w/ Bil at the conference, and he shared with me the way he would do this, I wrote down the tags he mentioned, and I can't find them now!
>
> So, I'll ask again :)
>
> I have a task that takes a while to run.
> Every set of 50 records I need to process can take anywhere from 10 seconds to over 2 minutes.
>
> I want to schedule this process, but timing is bad.
> Scheduling it every 3 minutes, is a waist of time, but some times it needs that much time.
>
>
> Bill mentioned making my page run indefinitely in a loop.
>
> With the combination of Lasso_ExecutionTimeLimit, and Thread_Priority_High, and ….. some_other_tag_I_cant_remember
>
> And code example would also be helpful.
>
> Thanks
>
> - John Morris
>
>
> #############################################################
> This message is sent to you because you are subscribed to
>  the mailing list Lasso
> [hidden email]
> To unsubscribe, E-mail to: <[hidden email]>
> Send administrative queries to  <[hidden email]>


#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: keeping a process running in Lasso 9

Brad Lindsay
In reply to this post by John Morris-3
On 10/1/13, 9:46 AM, John Morris wrote:
> I've been in Lasso 8 land for so long, I forgot about Lasso 9 in the command line.
> That MIGHT be the route to go.
>
> Lasso is really only used to run the MySQL statement.
> The multiple server's is really needed for the MySQL part.
>
> Do you have those examples of Lasso 9 in the command line that you and Jolle did at the LDC?
> Those would really come in handy in playing with that route.

I know Jolle has them. The biggest thing you're going to need is this
line of code to load the MySQL data source connector:


lcapi_loadModule((sys_masterHomePath || sys_homePath) +
'/LassoModules/MySQLConnector.' + sys_dll_ext)



Brad

#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: keeping a process running in Lasso 9

Jolle Carlestam-3
1 okt 2013 kl. 19:56 skrev Brad Lindsay <[hidden email]>:

> On 10/1/13, 9:46 AM, John Morris wrote:
>> I've been in Lasso 8 land for so long, I forgot about Lasso 9 in the command line.
>> That MIGHT be the route to go.
>>
>> Lasso is really only used to run the MySQL statement.
>> The multiple server's is really needed for the MySQL part.
>>
>> Do you have those examples of Lasso 9 in the command line that you and Jolle did at the LDC?
>> Those would really come in handy in playing with that route.
>
> I know Jolle has them.

I sent all material used for our presentation and the one I did by myself to Lassosoft for publishing in a proper LDC related place. Both the slides and the examples.
So far seen no reaction from their behalf. A pity, I'm longing for the other speakers slides to refresh what I learned.

I can send the zipped archive Lassosoft got to anyone who requests it. It is a tad bit big though, ca 22 MB.

HDB
Jolle

#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: keeping a process running in Lasso 9

Ke Carlton-3
In reply to this post by John Morris-3
Have been loosely following this. Thought it may be worth chiming in.

25 Million records at 50 rows per 15 - 120 seconds will take between 86 and
694 days to complete.

Another way to look at it is 25,000,000 / 50 = 500,000 iterations, so in
order to achieve that per day you will need each 50 row iteration to take a
maximum of 0.172 seconds (60 * 60 * 24 / 500000).

As soon as you start using multiple servers to work on such a task you need
a central way of tracking there progress and ensure that they continue
where they left off on restart:

// global progress table
id taskname position


Essentially each server increments the position by the number of rows it
intends to process. After it's completed a cycle it gets current and
increments the position in the table. You can add a second table to ensure
each 50 row block gets completed if there's an error midway through 50 rows:

// server progress table
id server_ip taskname position


From there it reverts to the primary position — although it's unlikely you
will need this depending on the sensitivity of the data / critical nature
of the iteration.

define mybigtask => thread {
public doit => {
/*
SELECT position FROM task_progress WHERE name = 'example';
UPDATE task_progress SET position = position + 50;
*/
 // do your thing
}
 public active_tick => {
.doit
return 1
}
}


Ke

On 1 October 2013 14:02, John Morris <[hidden email]> wrote:

> Once I'm done with the original 25 million que, I'll have nightly updates
> to say anywhere from 1,000 - 10,000 to run
> So I'll run those until done. then sit and wait till the next night.
>
> I've been reading up on threads, and an a bit lost.
>
>
> Lets say I want to do this:
>
>         Server A.
>
>                 inline( -sql='mySearch',
>                                 -max-records=50)
>
>                         //      finds 25 million, grabs the 1st set of 50
> to work on
>
>                         do the rest of my stuff here….
>
>                 /inline;
>
>
>         Server B.
>
>                 inline( -sql='mySearch',
>                                 -max-records=50)
>
>                         //      finds (25 million - the 50 Comp A
> grabbed), grabs the 1st set of 50 to work on
>
>                         do the rest of my stuff here….
>
>                 /inline;
>
>
>
>         Server A's set of 50 takes it 15 seconds.
>         Server B's set of 50 takes it 2 minutes.
>
>         I want Server to run again instantly, getting another set of 50
> records which would be the next set of 50 after Server B.
>
> Currently I set up a event that is ran every 3 minutes, but their is a lot
> of waisted time.
>
> So how would I keep this running until the found count of the inline was 0?
>
> Am I over thinking this?
> Should I just place this code in a loop(found_count / 50 ) tag and then do
> the search again within the loop to see if I should about the loop?
>
> And, how can I check to see if this code is currently running? so if it
> isn't I can start it again.
>
> Thanks for all the advice
>  - John Morris

#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: keeping a process running in Lasso 9

John Morris-3
Yes, I definitely need to get multiple server's running on this because of the time.

Thanks for all the great ideas and info.

I'll post again once I get it all worked out.

 - John Morris

On Oct 2, 2013, at 4:12 AM, Ke Carlton <[hidden email]> wrote:

> Have been loosely following this. Thought it may be worth chiming in.
>
> 25 Million records at 50 rows per 15 - 120 seconds will take between 86 and
> 694 days to complete.
>
> Another way to look at it is 25,000,000 / 50 = 500,000 iterations, so in
> order to achieve that per day you will need each 50 row iteration to take a
> maximum of 0.172 seconds (60 * 60 * 24 / 500000).
>
> As soon as you start using multiple servers to work on such a task you need
> a central way of tracking there progress and ensure that they continue
> where they left off on restart:
>
> // global progress table
> id taskname position
>
>
> Essentially each server increments the position by the number of rows it
> intends to process. After it's completed a cycle it gets current and
> increments the position in the table. You can add a second table to ensure
> each 50 row block gets completed if there's an error midway through 50 rows:
>
> // server progress table
> id server_ip taskname position
>
>
> From there it reverts to the primary position — although it's unlikely you
> will need this depending on the sensitivity of the data / critical nature
> of the iteration.
>
> define mybigtask => thread {
> public doit => {
> /*
> SELECT position FROM task_progress WHERE name = 'example';
> UPDATE task_progress SET position = position + 50;
> */
> // do your thing
> }
> public active_tick => {
> .doit
> return 1
> }
> }
>
>
> Ke
>
> On 1 October 2013 14:02, John Morris <[hidden email]> wrote:
>
>> Once I'm done with the original 25 million que, I'll have nightly updates
>> to say anywhere from 1,000 - 10,000 to run
>> So I'll run those until done. then sit and wait till the next night.
>>
>> I've been reading up on threads, and an a bit lost.
>>
>>
>> Lets say I want to do this:
>>
>>        Server A.
>>
>>                inline( -sql='mySearch',
>>                                -max-records=50)
>>
>>                        //      finds 25 million, grabs the 1st set of 50
>> to work on
>>
>>                        do the rest of my stuff here….
>>
>>                /inline;
>>
>>
>>        Server B.
>>
>>                inline( -sql='mySearch',
>>                                -max-records=50)
>>
>>                        //      finds (25 million - the 50 Comp A
>> grabbed), grabs the 1st set of 50 to work on
>>
>>                        do the rest of my stuff here….
>>
>>                /inline;
>>
>>
>>
>>        Server A's set of 50 takes it 15 seconds.
>>        Server B's set of 50 takes it 2 minutes.
>>
>>        I want Server to run again instantly, getting another set of 50
>> records which would be the next set of 50 after Server B.
>>
>> Currently I set up a event that is ran every 3 minutes, but their is a lot
>> of waisted time.
>>
>> So how would I keep this running until the found count of the inline was 0?
>>
>> Am I over thinking this?
>> Should I just place this code in a loop(found_count / 50 ) tag and then do
>> the search again within the loop to see if I should about the loop?
>>
>> And, how can I check to see if this code is currently running? so if it
>> isn't I can start it again.
>>
>> Thanks for all the advice
>> - John Morris
>
> #############################################################
> This message is sent to you because you are subscribed to
>  the mailing list Lasso
> [hidden email]
> To unsubscribe, E-mail to: <[hidden email]>
> Send administrative queries to  <[hidden email]>


#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: keeping a process running in Lasso 9

John Morris-3
In reply to this post by Jolle Carlestam-3
I'd LOVE a copy Jolle.

I plan on starting w/ Bil's route, but moving it to Lasso9 command line like Brad suggested.
Your examples would save me tons of time.

Do you have a dropbox account you could upload them to, so we could download them?

 - John Morris

On Oct 2, 2013, at 12:16 AM, Jolle Carlestam <[hidden email]> wrote:

> 1 okt 2013 kl. 19:56 skrev Brad Lindsay <[hidden email]>:
>
>> On 10/1/13, 9:46 AM, John Morris wrote:
>>> I've been in Lasso 8 land for so long, I forgot about Lasso 9 in the command line.
>>> That MIGHT be the route to go.
>>>
>>> Lasso is really only used to run the MySQL statement.
>>> The multiple server's is really needed for the MySQL part.
>>>
>>> Do you have those examples of Lasso 9 in the command line that you and Jolle did at the LDC?
>>> Those would really come in handy in playing with that route.
>>
>> I know Jolle has them.
>
> I sent all material used for our presentation and the one I did by myself to Lassosoft for publishing in a proper LDC related place. Both the slides and the examples.
> So far seen no reaction from their behalf. A pity, I'm longing for the other speakers slides to refresh what I learned.
>
> I can send the zipped archive Lassosoft got to anyone who requests it. It is a tad bit big though, ca 22 MB.
>
> HDB
> Jolle
>
> #############################################################
> This message is sent to you because you are subscribed to
>  the mailing list Lasso
> [hidden email]
> To unsubscribe, E-mail to: <[hidden email]>
> Send administrative queries to  <[hidden email]>


#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: keeping a process running in Lasso 9

John Morris-3
In reply to this post by Bil Corry-3
Thanks again for this example Bil.

Like I said @ the conference, you gave me a better path, and a much easier one than I had originally conceived.

 - John Morris

On Oct 1, 2013, at 10:23 AM, Bil Corry <[hidden email]> wrote:

> My understanding is that there is a single MySQL server, and connecting to
> it are multiple Lasso servers.  If so, this is how I would do it:
>
> On each server, you drop the following script into LassoStartup (script is
> pseudo-code because my Lasso is rusty - some method names may be horribly
> wrong, sorry):
>
> // --------------------------
> local(threadID) = 'server4'; // make sure this value is unique per server
>
> while(true);
>
> // mark the 50 to process
> inline(-sql='UPDATE myTable SET threadID = "'+#threadID+'", processStart =
> NOW() WHERE threadID IS NULL LIMIT 50');
> /inline;
>
> // grab our 50, might be more if the process was interrupted and never
> completed
> inline(-sql='SELECT * FROM myTable WHERE threadID = '+#threadID);
>  if(found_count > 0); // if found some to process, do it!
>    rows;
>      // process each row
>      inline(-sql='UPDATE myTable SET threadID = "done" WHERE rowKey =
> '+#rowKeyValue);
>    /rows;
>  else; // everything looks like it was completed, so sleep 8 hours (or can
> sleep until midnight, etc)
>    sleep(28800000); // 8 hours
>  /if;
> /inline;
>
> /while;
> // --------------------------
>
>
> Some other thoughts:
>
> You may want to create a helper script that looks at the processStart time
> and if it's been longer than the max time you expect, you may want to
> consider rebooting the server as the process has probably hung.
>
> In Lasso 8, there was a Lasso_ExecutionTimeLimit tag, not sure if it exists
> in Lasso 9, but the idea is you want to prevent Lasso from killing this
> thread, which will run forever (hopefully).
>
> You may want to add in sleep() after each round of 50 and/or between
> processing each row.  With multiple servers all hitting the same MySQL
> table, there's a chance Lasso could overwhelm it.  Even adding in a small
> delay, such as 50 ms could be the difference between it running and it
> bombing.  You'll have to test it.
>
> The beauty of the above is that even if you reboot the Lasso server, it
> will continue where it left off, and will continue to process until all
> records are done, so even if a server blows up, the other ones will finish
> it off (except for the few records that were reserved by the downed server).
>
> Hope that clarifies what I had mentioned at LDC.
>
> - Bil
>
>
> On Tue, Oct 1, 2013 at 3:46 PM, John Morris <[hidden email]>wrote:
>
>> I've been in Lasso 8 land for so long, I forgot about Lasso 9 in the
>> command line.
>> That MIGHT be the route to go.
>>
>> Lasso is really only used to run the MySQL statement.
>> The multiple server's is really needed for the MySQL part.
>>
>> Do you have those examples of Lasso 9 in the command line that you and
>> Jolle did at the LDC?
>> Those would really come in handy in playing with that route.
>>
>> I could write my script in lasso then have cron keep it running.
>>
>> - John Morris
>>
>> On Oct 1, 2013, at 8:34 AM, Brad Lindsay <[hidden email]> wrote:
>>
>>> On 10/1/13, 9:02 AM, John Morris wrote:
>>>> Once I'm done with the original 25 million que, I'll have nightly
>> updates to say anywhere from 1,000 - 10,000 to run
>>>> So I'll run those until done. then sit and wait till the next night.
>>>>
>>>> I've been reading up on threads, and an a bit lost.
>>>>
>>>>
>>>> Lets say I want to do this:
>>>>
>>>>     Server A.
>>>>
>>>>             inline( -sql='mySearch',
>>>>                             -max-records=50)
>>>>
>>>>                     //      finds 25 million, grabs the 1st set of 50
>> to work on
>>>>
>>>>                     do the rest of my stuff here….
>>>>
>>>>             /inline;
>>>>
>>>>
>>>>     Server B.
>>>>
>>>>             inline( -sql='mySearch',
>>>>                             -max-records=50)
>>>>
>>>>                     //      finds (25 million - the 50 Comp A
>> grabbed), grabs the 1st set of 50 to work on
>>>>
>>>>                     do the rest of my stuff here….
>>>>
>>>>             /inline;
>>>>
>>>>
>>>>
>>>>     Server A's set of 50 takes it 15 seconds.
>>>>     Server B's set of 50 takes it 2 minutes.
>>>>
>>>>     I want Server to run again instantly, getting another set of 50
>> records which would be the next set of 50 after Server B.
>>>>
>>>>
>>>
>>> Adding multiple servers into the mix complicates things. Essentially you
>> need a centralized way to track which tasks have been done or which are
>> left to do. And you're probably going to need to investigate MySQL's
>> locking mechanisms to do it.
>>>
>>>
>>>
>>>> So how would I keep this running until the found count of the inline
>> was 0?
>>>>
>>>> Am I over thinking this?
>>>> Should I just place this code in a loop(found_count / 50 ) tag and then
>> do the search again within the loop to see if I should about the loop?
>>>
>>> The other alternative is to stick it in a [while(true) => {}] and then
>> use [loop_abort] when it meets the conditions to abort. Be careful to
>> actually have it abort!
>>>
>>>
>>>>
>>>> And, how can I check to see if this code is currently running? so if it
>> isn't I can start it again.
>>>
>>> It depends on how you're doing what you're doing. If you use
>> [split_thread], it returns a set of file descriptors that can be used to
>> communicate with the thread. I would stash those in a thread object and use
>> that thread object to determine if it's running or not.
>>>
>>>
>>>
>>> Do you really need to be running this from a web page at all? Would it
>> be simpler / easier to write a command-line script rather than dealing with
>> threads? You could run the script as a normal background process that you'd
>> be able to see if it exited. If you need to run the script on multiple
>> servers, that's fine - you would then just need to worry about making sure
>> they didn't do work that was already being done.
>>
>>
>> #############################################################
>> This message is sent to you because you are subscribed to
>>  the mailing list Lasso
>> [hidden email]
>> To unsubscribe, E-mail to: <[hidden email]>
>> Send administrative queries to  <[hidden email]>
>>
>
> #############################################################
> This message is sent to you because you are subscribed to
>  the mailing list Lasso
> [hidden email]
> To unsubscribe, E-mail to: <[hidden email]>
> Send administrative queries to  <[hidden email]>


#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: keeping a process running in Lasso 9

Rachel Guthrie
In reply to this post by Jolle Carlestam-3

On 2013-10-02, at 1:16 AM, Jolle Carlestam <[hidden email]> wrote:

> I sent all material used for our presentation and the one I did by myself to Lassosoft for publishing in a proper LDC related place. Both the slides and the examples.
> So far seen no reaction from their behalf. A pity, I'm longing for the other speakers slides to refresh what I learned.

Hi Jolle, thanks, and apologies for not getting back to you sooner.

We are still waiting on a number of presentations... particuarly the speakers notes that fill in the concepts between the bullet points and code examples.

We are working on getting them up and ready for public viewing...

Cheers
Rachel

#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: keeping a process running in Lasso 9

Jolle Carlestam-3
In reply to this post by John Morris-3
2 okt 2013 kl. 14:57 skrev John Morris <[hidden email]>:

> I'd LOVE a copy Jolle.

This link should work:
https://docs.google.com/file/d/0BwtFtDG8_cIeZ3JVd2ZsLXVYeFU/

HDB
Jolle

#############################################################
This message is sent to you because you are subscribed to
  the mailing list Lasso
[hidden email]
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>