Getting hammered

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Getting hammered

Steve Upton

I knew that would get your attention... almost as good as "Free Beer!"

No, this is not about drinking, this is about our websites getting thrashed by:

- rogue crawlers - eg Searchme.com
- hackers & other malicious types - trying to break in, DoS attacks, etc
- over-zealous users
- API-hammering client apps

So, are there techniques that work well to avoid this kind of problem?

Are there Lasso-specific weaknesses and/or strengths that we should be aware of or can exploit?

It seemed like a good time to start this discussion. I'm hoping we can start a thread and get some solutions going that will work for all of us. If they don't find us handsome they should at least find us handy....

Regards,

Steve Upton


--


--
This list is a free service of LassoSoft: http://www.LassoSoft.com/
Search the list archives: http://www.ListSearch.com/Lasso/Browse/
Manage your subscription: http://www.ListSearch.com/Lasso/

Reply | Threaded
Open this post in threaded view
|

Re: Getting hammered

Marc Pinnell-3
I learned (the hard way earlier this year) that encode_sql is my  
friend and should be applied liberally.

Marc


On Apr 16, 2008, at 1:34 PM, Steve Upton wrote:

>
> I knew that would get your attention... almost as good as "Free Beer!"
>
> No, this is not about drinking, this is about our websites getting  
> thrashed by:
>
> - rogue crawlers - eg Searchme.com
> - hackers & other malicious types - trying to break in, DoS attacks,  
> etc
> - over-zealous users
> - API-hammering client apps
>
> So, are there techniques that work well to avoid this kind of problem?
>
> Are there Lasso-specific weaknesses and/or strengths that we should  
> be aware of or can exploit?
>
> It seemed like a good time to start this discussion. I'm hoping we  
> can start a thread and get some solutions going that will work for  
> all of us. If they don't find us handsome they should at least find  
> us handy....
>
> Regards,
>
> Steve Upton
>
>
> --
>
>
> --
> This list is a free service of LassoSoft: http://www.LassoSoft.com/
> Search the list archives: http://www.ListSearch.com/Lasso/Browse/
> Manage your subscription: http://www.ListSearch.com/Lasso/
>

--
Marc Pinnell
1027 Design
PO Box 990872
Redding, CA 96099-0872
530.941.4706
www.1027Design.com




--
This list is a free service of LassoSoft: http://www.LassoSoft.com/
Search the list archives: http://www.ListSearch.com/Lasso/Browse/
Manage your subscription: http://www.ListSearch.com/Lasso/

Reply | Threaded
Open this post in threaded view
|

Re: Getting hammered

Steve Upton
At 1:46 PM -0700 4/16/08, Marc Pinnell wrote:
>I learned (the hard way earlier this year) that encode_sql is my friend and should be applied liberally.

indeed. It's an important lock to have on the door.

I'm looking more for techniques to counter a site getting hit many times. It seems like there's always a trade-off between allowing a user or API client a certain number of hits or new records per xxx and locking them out because they've exceeded their quota, their software is malfunctioning or it looks like they're just trying to cause trouble.

We are also dealing with API clients where the API-accessing software is created (& registered) by one company and the data it handles (and sends to us) is created by another account holder...

So, as you can imagine, handling quotas in a logical and efficient way is important. I'm just wondering if other Lasso'ers (ee's?) have dealt with such issues before.

Regards,

Steve Upton


--


--
This list is a free service of LassoSoft: http://www.LassoSoft.com/
Search the list archives: http://www.ListSearch.com/Lasso/Browse/
Manage your subscription: http://www.ListSearch.com/Lasso/

Reply | Threaded
Open this post in threaded view
|

Re: Getting hammered

stevepiercy
In reply to this post by Steve Upton
On Wednesday, April 16, 2008, [hidden email] (Steve Upton) pronounced:

>- rogue crawlers - eg Searchme.com
>- hackers & other malicious types - trying to break in, DoS attacks, etc
>- over-zealous users
>- API-hammering client apps

Firewalls are pretty handy for preventing malicious attacks.  Here's one:
<http://www.sonicwall.com/us/products/NSA_3500.html>

There are less pricey products with fewer options.  Devices with automated intrusion detection can help prevent service outages or performance hits that would otherwise extend until you happen to notice it or a user complains.

--steve

-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
Steve Piercy               Web Site Builder               Soquel, CA
<[hidden email]>                  <http://www.StevePiercy.com/>

--
This list is a free service of LassoSoft: http://www.LassoSoft.com/
Search the list archives: http://www.ListSearch.com/Lasso/Browse/
Manage your subscription: http://www.ListSearch.com/Lasso/

Reply | Threaded
Open this post in threaded view
|

Re: Getting hammered

Adam Randall-2
On Wed, 16 Apr 2008 21:44:20 -0700, Steve Piercy - Web Site Builder wrote:

> On Wednesday, April 16, 2008, [hidden email] (Steve Upton) pronounced:
>
>> - rogue crawlers - eg Searchme.com
>> - hackers & other malicious types - trying to break in, DoS attacks, etc
>> - over-zealous users
>> - API-hammering client apps
>
> Firewalls are pretty handy for preventing malicious attacks.  Here's one:
> <http://www.sonicwall.com/us/products/NSA_3500.html>
>
> There are less pricey products with fewer options.  Devices with
> automated intrusion detection can help prevent service outages or
> performance hits that would otherwise extend until you happen to
> notice it or a user complains.

If you are not using SSH, turn it off, or restrict who can access SSH by using either ipfw or tcpwrappers (drop me a line if you want to know more about tcp wrappers, and lasso integration into it).

In regards to searchme, I had to ban their IP addresses as they were relentlessly hitting the servers, and actually causing lasso to puke all over itself. Not the best situation (got the machine up to a load average of 400. I was impressed, heh).

Regards,

Adam.

--
-----------------------------------------------------------------------
Adam Randall                                       http://www.xaren.net
[hidden email]                                   AIM/iChat:  blitz574

"Macintosh users are a special case. They care passionately about the
Mac OS and would rewire their own bodies to run on Mac OS X if such a
thing were possible." -- Peter H. Lewis

--
This list is a free service of LassoSoft: http://www.LassoSoft.com/
Search the list archives: http://www.ListSearch.com/Lasso/Browse/
Manage your subscription: http://www.ListSearch.com/Lasso/

Reply | Threaded
Open this post in threaded view
|

Re: Getting hammered

Viaduct Productions
Port knocking around on OS X?


On Apr 17, 2008, at 12:52 AM, Adam Randall wrote:
> If you are not using SSH, turn it off, or restrict who can access  
> SSH by using either ipfw or tcpwrappers (drop me a line if you want  
> to know more about tcp wrappers, and lasso integration into it).


Rich in Toronto
...now go get on your bike


--
This list is a free service of LassoSoft: http://www.LassoSoft.com/
Search the list archives: http://www.ListSearch.com/Lasso/Browse/
Manage your subscription: http://www.ListSearch.com/Lasso/

Reply | Threaded
Open this post in threaded view
|

Re: Getting hammered

Adam Randall-2
On Thu, 17 Apr 2008 03:47:31 -0400, Rich wrote:
> Port knocking around on OS X?

I've never played with that actually, and I haven't seen any mention of it on OS X. I bet that third-party routers could implement that though, which could be interesting.

Adam.

--
-----------------------------------------------------------------------
Adam Randall                                       http://www.xaren.net
[hidden email]                                   AIM/iChat:  blitz574

"Macintosh users are a special case. They care passionately about the
Mac OS and would rewire their own bodies to run on Mac OS X if such a
thing were possible." -- Peter H. Lewis

--
This list is a free service of LassoSoft: http://www.LassoSoft.com/
Search the list archives: http://www.ListSearch.com/Lasso/Browse/
Manage your subscription: http://www.ListSearch.com/Lasso/

Reply | Threaded
Open this post in threaded view
|

Re: Getting hammered

Lars A. Gundersen-2
In reply to this post by Marc Pinnell-3

Den 16. april. 2008 kl. 22.46 skrev Marc Pinnell:

> I learned (the hard way earlier this year) that encode_sql is my  
> friend and should be applied liberally.

Yes, I've seen this mentioned many times on the list. I have to ask:  
Doesn't Lasso normally do this automatically when Inline statements  
are being sent to SQL data sources?
Is it just an issue when you use -sql in Inlines, or always?
If it just an issue with -sql then why use direct sql statements  
instead of the normal Lasso Inline syntax?
I am just asking, because I've never had to use -sql in Inlines. Is it  
very much faster to use -sql, or is it just necessary in complex cases?

I sleep well at night, knowing I could, if I had to, swap mySQL for  
another data storage engine, because Lasso has abstracted the queries  
for me.

Just asking.

Lars
--
Lars A. Gundersen
http://www.larsagundersen.no/ • +47 91 64 46 10


--
This list is a free service of LassoSoft: http://www.LassoSoft.com/
Search the list archives: http://www.ListSearch.com/Lasso/Browse/
Manage your subscription: http://www.ListSearch.com/Lasso/

Reply | Threaded
Open this post in threaded view
|

Re: Getting hammered

Simon Forster

On 17 Apr 2008, at 11:05, Lars A. Gundersen wrote:
> Den 16. april. 2008 kl. 22.46 skrev Marc Pinnell:
>
>> I learned (the hard way earlier this year) that encode_sql is my  
>> friend and should be applied liberally.
>
> Yes, I've seen this mentioned many times on the list. I have to ask:  
> Doesn't Lasso normally do this automatically when Inline statements  
> are being sent to SQL data sources?

If you use Lasso's query structure, it encodes the data for you. If  
you hand craft your SQL statements (i.e. use -sql='xyz'), it doesn't.

> Is it just an issue when you use -sql in Inlines, or always?

-sql='xyz' only.

> If it just an issue with -sql then why use direct sql statements  
> instead of the normal Lasso Inline syntax?
> I am just asking, because I've never had to use -sql in Inlines. Is  
> it very much faster to use -sql, or is it just necessary in complex  
> cases?

Hand crafting your SQL statements allows you to do things which just  
aren't possible with Lasso's more generic query structure.

> I sleep well at night, knowing I could, if I had to, swap mySQL for  
> another data storage engine, because Lasso has abstracted the  
> queries for me.

Hmm. Certainly I've read several cogent articles pointing our the  
pitfalls of database abstraction layers - some of which suggest that  
as an aim, it's deeply flawed. For all but the simplest use, I tend to  
agree - which to my mind suggests that Lasso's got it just about right.

> Just asking.

Just telling.

;-)

Simon Forster

   LDML Ltd
   62 Pall Mall
   London
   SW1Y 5HZ
   United Kingdom
   t: +44 20 7993 8813
   f: +44 70 9230 5247




--
This list is a free service of LassoSoft: http://www.LassoSoft.com/
Search the list archives: http://www.ListSearch.com/Lasso/Browse/
Manage your subscription: http://www.ListSearch.com/Lasso/

Reply | Threaded
Open this post in threaded view
|

Re: Getting hammered

Jolle Carlestam-2
In reply to this post by Lars A. Gundersen-2
17 apr 2008 kl. 12.05 skrev Lars A. Gundersen:

>
>
> Den 16. april. 2008 kl. 22.46 skrev Marc Pinnell:
>
>> I learned (the hard way earlier this year) that encode_sql is my  
>> friend and should be applied liberally.
>
> Yes, I've seen this mentioned many times on the list. I have to ask:  
> Doesn't Lasso normally do this automatically when Inline statements  
> are being sent to SQL data sources?

Lasso does it for you when you use regular Inline statements.

> Is it just an issue when you use -sql in Inlines, or always?

It's only an issue when you write your own sql queries.

> If it just an issue with -sql then why use direct sql statements  
> instead of the normal Lasso Inline syntax?

When you want something from your DB that a normal inline can't  
handle. Like JOIN, GROUP, calculations etc.

The result of a JOIN can be accomplished by using several inline  
statements nested inside each other. But that is a lot slower than  
building your own query. A lot of other specific sql action can only  
be done by building your own statement.

> I am just asking, because I've never had to use -sql in Inlines. Is  
> it very much faster to use -sql, or is it just necessary in complex  
> cases?

It's not faster for normal actions. If you can use a regular inline by  
all means do so. But if you start tweaking them, by for example using  
several inlines directed at different tables or post processing the  
results record by record instead of letting Mysql do the processing  
for you or for other purposes then building your own sql is much faster.

> I sleep well at night, knowing I could, if I had to, swap mySQL for  
> another data storage engine, because Lasso has abstracted the  
> queries for me.

If you take special care building your sql you can switch between  
different sql application. That give you all the freedom you need.  
True you can't use Filemaker. But if you've started using Mysql  
there's probably little reason to switch to Filemaker anyhow.

HDB
Jolle

--
This list is a free service of LassoSoft: http://www.LassoSoft.com/
Search the list archives: http://www.ListSearch.com/Lasso/Browse/
Manage your subscription: http://www.ListSearch.com/Lasso/