Performance difference between 2 servers, SQLite issue maybe?

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Performance difference between 2 servers, SQLite issue maybe?

Wade Maxfield
Hey List,

I’m after a little bit of help to diagnose a very particular performance issue with Lasso 8.5.6.

I have 2 servers, and the difference in performance between them for some actions has me stumped.

Server 1 (Dev) is a Mac Mini 2.6GHz Quad Core i7, OS X 10.10, Lasso 8.5.6.
Server 2 (Live) is Xserve 2.26GHz 2x Quad Core, OS X 10.10, Lasso 8.5.6.

Machines perform similarly with most Lasso pages (straight forward code, grabbing data from MySQL, etc).

But, server 2 is particularly slow when accessing some pages in ServerAdmin.lassoapp and SiteAdmin.lassoapp, for instance editing the path permissions in Setup > Security > Files > Path Permissions.  It can take 3-4s to load the page when you click on the path to edit, 4-6s when you hit assign to submit the form.  The same action on server 1 takes 200-400ms to load, 500-650ms to assign.

The bigger issue is with Email_Send.  Both servers are set up the same in Utility > Email > Setup, with Ignore -Host Param set to yes, Send Directly to SMTP Servers set to no.   I’ve wrapped the Email_Send with a quick timer using _date_msec.  Queuing the email with Email_Send on server 2 takes 1500-1900ms, exact same code/email on server 1 takes 90-110ms.  More than a factor of 10 difference.

Starting Lasso from terminal via startLassoService.command is also noticeably slower on server 2.

All signs seem to be pointing to slow performance with the SQLite tables that Lasso uses for settings and email.  I thought there might be some sort of SQLite corruption so I’ve clean installed Lasso from scratch on server 2 and manually entered all the settings again, and it’s still just as slow.

CPU usage via top for any Lasso process on the machine doesn’t go above 5%, but something is causing the delays.

What else can I try or where else can I look to sort this out?


 - Wade



#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Performance difference between 2 servers, SQLite issue maybe?

Carl Ketterling
Wade,

Could the difference simply be the machines?  The Xserve is likely several years older than the mini, and the processors improve every year.  Also, the newer Macs can come with a fusion disk that makes a drastic difference in some situations.

Have you checked the Lasso databases?  I use cron to check the integrity of my databases every week.  I must have had a problem once, because I set up the script. But that was 2015 or earlier, so I don’t remember the specifics.  Here’s what I run in my script:
echo "pragma integrity_check;" | sqlite3 /var/lasso/instances/instance_one/SQLiteDBs/database_registry
I have one line for each database.

Have you rebooted recently?  When I believe a machine is running slow, one troubleshooting step is to reboot and retest the times.  That helps to determine if the problem is persistent (like a database problem) or something that gets worse with time (like a memory problem).

Carl


> On Jun 17, 2017, at 2:28 AM, Wade Maxfield <[hidden email]> wrote:
>
> Hey List,
>
> I’m after a little bit of help to diagnose a very particular performance issue with Lasso 8.5.6.
>
> I have 2 servers, and the difference in performance between them for some actions has me stumped.
>
> Server 1 (Dev) is a Mac Mini 2.6GHz Quad Core i7, OS X 10.10, Lasso 8.5.6.
> Server 2 (Live) is Xserve 2.26GHz 2x Quad Core, OS X 10.10, Lasso 8.5.6.
>
> Machines perform similarly with most Lasso pages (straight forward code, grabbing data from MySQL, etc).
>
> But, server 2 is particularly slow when accessing some pages in ServerAdmin.lassoapp and SiteAdmin.lassoapp, for instance editing the path permissions in Setup > Security > Files > Path Permissions.  It can take 3-4s to load the page when you click on the path to edit, 4-6s when you hit assign to submit the form.  The same action on server 1 takes 200-400ms to load, 500-650ms to assign.
>
> The bigger issue is with Email_Send.  Both servers are set up the same in Utility > Email > Setup, with Ignore -Host Param set to yes, Send Directly to SMTP Servers set to no.   I’ve wrapped the Email_Send with a quick timer using _date_msec.  Queuing the email with Email_Send on server 2 takes 1500-1900ms, exact same code/email on server 1 takes 90-110ms.  More than a factor of 10 difference.
>
> Starting Lasso from terminal via startLassoService.command is also noticeably slower on server 2.
>
> All signs seem to be pointing to slow performance with the SQLite tables that Lasso uses for settings and email.  I thought there might be some sort of SQLite corruption so I’ve clean installed Lasso from scratch on server 2 and manually entered all the settings again, and it’s still just as slow.
>
> CPU usage via top for any Lasso process on the machine doesn’t go above 5%, but something is causing the delays.
>
> What else can I try or where else can I look to sort this out?
>
>
> - Wade
>
>
>
> #############################################################
>
> This message is sent to you because you are subscribed to
>  the mailing list Lasso [hidden email]
> Official list archives available at http://www.lassotalk.com
> To unsubscribe, E-mail to: <[hidden email]>
> Send administrative queries to  <[hidden email]>


#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Performance difference between 2 servers, SQLite issue maybe?

Wade Maxfield
> On 18/06/2017, at 02:50, Carl Ketterling <[hidden email]> wrote:
>
> Wade,
>
> Could the difference simply be the machines?  The Xserve is likely several years older than the mini, and the processors improve every year.  Also, the newer Macs can come with a fusion disk that makes a drastic difference in some situations.
>

Yes, there could be some speed difference due to machine architecture (2009 Xserve vs 2012 Mini), but shouldn’t that be noticeable across all types of lasso processing?  An example page I have calls 3 different MySQL inlines, and sends 2 emails.  Putting timers around 8 different parts of the page shows only the email_send runs slower on the xserve (by a factor of 10 or more) everything else on the page runs as fast or faster than the mini.

Both machines are using Intel 53x SSDs for boot drive, and separate SSDs for the data drives. I’ve also reformatted the boot drive on the xserve and speed tested it in case it was failing (no issues reported, speed as expected). Then I restored the drive from backup and then reinstalled Lasso from scratch.


> Have you checked the Lasso databases?  I use cron to check the integrity of my databases every week.  I must have had a problem once, because I set up the script. But that was 2015 or earlier, so I don’t remember the specifics.  Here’s what I run in my script:
> echo "pragma integrity_check;" | sqlite3 /var/lasso/instances/instance_one/SQLiteDBs/database_registry
> I have one line for each database.
>

I haven’t tried that. Those paths don’t look anything like what I have installed on my machine. Is that Lasso 9?  And how safe is it to open a SQLite file that might already be in use by Lasso?


> Have you rebooted recently?  When I believe a machine is running slow, one troubleshooting step is to reboot and retest the times.  That helps to determine if the problem is persistent (like a database problem) or something that gets worse with time (like a memory problem).

Rebooted less that 24 hours ago, and this is with a clean install of Lasso 8.5.6 and entering the ServerAdmin and SiteAdmin Lasso settings by hand (having 2 windows open side by side using the working server as a reference).  I figure that should minimise the chance of database corruption. Plus machine is receiving no traffic, apart from my testing, so that’s another variable removed.

 - Wade


#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Performance difference between 2 servers, SQLite issue maybe?

Carl Ketterling
Yes, I use Lasso 9.

It sounds like you’ve tested everything I could think of. Hopefully someone else can provide better help.

Carl


> On Jun 17, 2017, at 5:05 PM, Wade Maxfield <[hidden email]> wrote:
>
>> On 18/06/2017, at 02:50, Carl Ketterling <[hidden email]> wrote:
>>
>> Wade,
>>
>> Could the difference simply be the machines?  The Xserve is likely several years older than the mini, and the processors improve every year.  Also, the newer Macs can come with a fusion disk that makes a drastic difference in some situations.
>>
>
> Yes, there could be some speed difference due to machine architecture (2009 Xserve vs 2012 Mini), but shouldn’t that be noticeable across all types of lasso processing?  An example page I have calls 3 different MySQL inlines, and sends 2 emails.  Putting timers around 8 different parts of the page shows only the email_send runs slower on the xserve (by a factor of 10 or more) everything else on the page runs as fast or faster than the mini.
>
> Both machines are using Intel 53x SSDs for boot drive, and separate SSDs for the data drives. I’ve also reformatted the boot drive on the xserve and speed tested it in case it was failing (no issues reported, speed as expected). Then I restored the drive from backup and then reinstalled Lasso from scratch.
>
>
>> Have you checked the Lasso databases?  I use cron to check the integrity of my databases every week.  I must have had a problem once, because I set up the script. But that was 2015 or earlier, so I don’t remember the specifics.  Here’s what I run in my script:
>> echo "pragma integrity_check;" | sqlite3 /var/lasso/instances/instance_one/SQLiteDBs/database_registry
>> I have one line for each database.
>>
>
> I haven’t tried that. Those paths don’t look anything like what I have installed on my machine. Is that Lasso 9?  And how safe is it to open a SQLite file that might already be in use by Lasso?
>
>
>> Have you rebooted recently?  When I believe a machine is running slow, one troubleshooting step is to reboot and retest the times.  That helps to determine if the problem is persistent (like a database problem) or something that gets worse with time (like a memory problem).
>
> Rebooted less that 24 hours ago, and this is with a clean install of Lasso 8.5.6 and entering the ServerAdmin and SiteAdmin Lasso settings by hand (having 2 windows open side by side using the working server as a reference).  I figure that should minimise the chance of database corruption. Plus machine is receiving no traffic, apart from my testing, so that’s another variable removed.
>
> - Wade
>
>
> #############################################################
>
> This message is sent to you because you are subscribed to
>  the mailing list Lasso [hidden email]
> Official list archives available at http://www.lassotalk.com
> To unsubscribe, E-mail to: <[hidden email]>
> Send administrative queries to  <[hidden email]>


#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Performance difference between 2 servers, SQLite issue maybe?

Marc Pinnell-3
On the mail issue, my first thought when I read your comments was DNS...

====================
Marc Pinnell
Director of Web Services
[hidden email]
www.shasta.com                                    
833 Mistletoe Lane, Suite A1
Redding, CA  96002
530-224-6866  Ext. 100
530-224-2035 Fax

> On Jun 18, 2017, at 11:35 AM, Carl Ketterling <[hidden email]> wrote:
>
> Yes, I use Lasso 9.
>
> It sounds like you’ve tested everything I could think of. Hopefully someone else can provide better help.
>
> Carl
>
>
>> On Jun 17, 2017, at 5:05 PM, Wade Maxfield <[hidden email]> wrote:
>>
>>> On 18/06/2017, at 02:50, Carl Ketterling <[hidden email]> wrote:
>>>
>>> Wade,
>>>
>>> Could the difference simply be the machines?  The Xserve is likely several years older than the mini, and the processors improve every year.  Also, the newer Macs can come with a fusion disk that makes a drastic difference in some situations.
>>>
>>
>> Yes, there could be some speed difference due to machine architecture (2009 Xserve vs 2012 Mini), but shouldn’t that be noticeable across all types of lasso processing?  An example page I have calls 3 different MySQL inlines, and sends 2 emails.  Putting timers around 8 different parts of the page shows only the email_send runs slower on the xserve (by a factor of 10 or more) everything else on the page runs as fast or faster than the mini.
>>
>> Both machines are using Intel 53x SSDs for boot drive, and separate SSDs for the data drives. I’ve also reformatted the boot drive on the xserve and speed tested it in case it was failing (no issues reported, speed as expected). Then I restored the drive from backup and then reinstalled Lasso from scratch.
>>
>>
>>> Have you checked the Lasso databases?  I use cron to check the integrity of my databases every week.  I must have had a problem once, because I set up the script. But that was 2015 or earlier, so I don’t remember the specifics.  Here’s what I run in my script:
>>> echo "pragma integrity_check;" | sqlite3 /var/lasso/instances/instance_one/SQLiteDBs/database_registry
>>> I have one line for each database.
>>>
>>
>> I haven’t tried that. Those paths don’t look anything like what I have installed on my machine. Is that Lasso 9?  And how safe is it to open a SQLite file that might already be in use by Lasso?
>>
>>
>>> Have you rebooted recently?  When I believe a machine is running slow, one troubleshooting step is to reboot and retest the times.  That helps to determine if the problem is persistent (like a database problem) or something that gets worse with time (like a memory problem).
>>
>> Rebooted less that 24 hours ago, and this is with a clean install of Lasso 8.5.6 and entering the ServerAdmin and SiteAdmin Lasso settings by hand (having 2 windows open side by side using the working server as a reference).  I figure that should minimise the chance of database corruption. Plus machine is receiving no traffic, apart from my testing, so that’s another variable removed.
>>
>> - Wade
>>
>>
>> #############################################################
>>
>> This message is sent to you because you are subscribed to
>> the mailing list Lasso [hidden email]
>> Official list archives available at http://www.lassotalk.com
>> To unsubscribe, E-mail to: <[hidden email]>
>> Send administrative queries to  <[hidden email]>
>
>
> #############################################################
>
> This message is sent to you because you are subscribed to
>  the mailing list Lasso [hidden email]
> Official list archives available at http://www.lassotalk.com
> To unsubscribe, E-mail to: <[hidden email]>
> Send administrative queries to  <[hidden email]>


#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Performance difference between 2 servers, SQLite issue maybe?

Wade Maxfield
> On 19/06/2017, at 08:31, Marc Pinnell <[hidden email]> wrote:
>
> On the mail issue, my first thought when I read your comments was DNS...

For queuing the email?  Send Directly to SMTP is off, and Ignore -Host is on.   I thought DNS would come into play when it was trying to send emails from the queue to the mail server.

I might try setting the default SMTP has to an ip address and see if that makes a difference.  I’m assuming that using an IP address would work around the need to do any DNS lookups.

 - Wade



#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Performance difference between 2 servers, SQLite issue maybe?

Wade Maxfield
In reply to this post by Carl Ketterling
> On 19/06/2017, at 06:35, Carl Ketterling <[hidden email]> wrote:
>
> It sounds like you’ve tested everything I could think of. Hopefully someone else can provide better help.

Thanks Carl.  I think I’m going to have to go to the datacentre and try isolating hardware, like cloning the boot drive to a firewire drive and booting off that, just to try and eliminate as many options as possible.

 - Wade


#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Performance difference between 2 servers, SQLite issue maybe?

Wade Maxfield
In reply to this post by Wade Maxfield
> On 17/06/2017, at 19:28, Wade Maxfield <[hidden email]> wrote:
>
> I have 2 servers, and the difference in performance between them for some actions has me stumped.
>
> Server 1 (Dev) is a Mac Mini 2.6GHz Quad Core i7, OS X 10.10, Lasso 8.5.6.
> Server 2 (Live) is Xserve 2.26GHz 2x Quad Core, OS X 10.10, Lasso 8.5.6.
>
> Machines perform similarly with most Lasso pages (straight forward code, grabbing data from MySQL, etc).
>
> But, server 2 is particularly slow when accessing some pages in ServerAdmin.lassoapp and SiteAdmin.lassoapp, for instance editing the path permissions in Setup > Security > Files > Path Permissions.  It can take 3-4s to load the page when you click on the path to edit, 4-6s when you hit assign to submit the form.  The same action on server 1 takes 200-400ms to load, 500-650ms to assign.
>
> The bigger issue is with Email_Send.  Both servers are set up the same in Utility > Email > Setup, with Ignore -Host Param set to yes, Send Directly to SMTP Servers set to no.   I’ve wrapped the Email_Send with a quick timer using _date_msec.  Queuing the email with Email_Send on server 2 takes 1500-1900ms, exact same code/email on server 1 takes 90-110ms.  More than a factor of 10 difference.
>
> Starting Lasso from terminal via startLassoService.command is also noticeably slower on server 2.
>
> All signs seem to be pointing to slow performance with the SQLite tables that Lasso uses for settings and email.  I thought there might be some sort of SQLite corruption so I’ve clean installed Lasso from scratch on server 2 and manually entered all the settings again, and it’s still just as slow.
>
> CPU usage via top for any Lasso process on the machine doesn’t go above 5%, but something is causing the delays.


Ok, just for completeness, delays resolved but not fully understood as to why/how it happened.

Sever 2 now processes the email_send in 15-20ms, as opposed to the 1500-1900ms or even 23-32s it took sometimes. ServerAdmin pages load as quickly as expected, and Lasso start up via startLassoService.command is back to 5-6 seconds to complete, rather than 20s+


Gory details:
It looks like some form of corruption to OS X system software/drivers/kext.  The boot drive is an Intel SSD connected via the Apple RAID card.  I also had an emergency boot drive SSD connected via USB that was cloned from the boot drive when the machine was first setup with OS X 10.10.  Cloning the bad OS to a Samsung SSD connected via the RAID card had the same random slow downs, cloning the good OS to the same Samsung SSD connected via RAID card showed no issues.
Xserve 1 with RAID card
Good OS (email_send 15ms+)
Bad OS (email_send 1500ms+)

Taking both drives to an Xserve with the normal SATA card (same model/CPU, non RAID card) showed no issues with either. Both drives/OS were slower than with the RAID card but consistent with each other.
Xserve 2 with SATA card
Good OS (email_send 45-50ms+)
Bad OS (email_send 45-50ms+)

Installing the SATA card in the original Xserve also showed the same slightly slower but consistent speeds
Xserve 1 with SATA card
Good OS (email_send 45-50ms+)
Bad OS (email_send 45-50ms+)

And for completeness, installing the RAID card in Xserve 2 showed the same delay issues.

Up shot is, something on the bad OS doesn’t like the RAID card.

Both OS had full complement of OS and Security updates. Zapping PRAM, resetting PMU, reinstalling Security Updates, running Disk Utility, Disk Warrior, etc. found no issues, and made no difference.

So I’ve wiped the bad OS, cloned from the good OS and returned it to the datacentre.


 - Wade


#############################################################

This message is sent to you because you are subscribed to
  the mailing list Lasso [hidden email]
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <[hidden email]>
Send administrative queries to  <[hidden email]>