[WEB SECURITY] Appropriate platform for testing/evaluating Web-Security-Scanners on?

[Yeager, Joe]

J,

I understand your feeling of use it or lose it, and you are correct that you need to keep your pen testing skills up.  There is hardly anything that can beat a really good security engineer, except a really good security engineer with a really good web application scanner.  Tools only help someone, not hinder them.  Adrian is completely right, the human eyeballs get tired.  You start to forget to test various attack vectors and you miss things.  In addition, I certainly don't want to waste my time throwing a XSS or SQLi attack into every single parameter; it just isn't worth paying me to do.  A security expert is supposed to know how to run their tools to the fullest extent, know the limitations of their tools, and then fill in the gaps to complete the assessment.

I don't believe your thought process scales up to a large or even a medium-sized assessment.  If you think it does, you're most likely not doing a complete assessment of the applications you're up against.  99% of the corporations out there paying for these assessments want them done as quickly as possible while also being as thorough as possible.  I know, it's a confusing game... but the only way to solve it is with tools.  As we all know, time is money.

As far as writing your own tool, it might be trickier than you think.  A lot of the tools in the industry have had people hacking on them for 7-10 years, especially the corporate tools which have had teams of 40+ developers and security experts pouring technology and code into them.  Now, if you're talking about a quick Perl script to run an exploit, that's a different story... but if you're telling everyone to go out and design their own scanner from scratch, you might consider rethinking your logic.

And lastly, PACHers (Point and Click Hackers) are running tools against the very same sites that you are tasked with assessing.  The last thing you want is for that 10-year-old to find something like a blatant XSS because you forgot that one input buried deep somewhere in the application... then the company makes the front page and you have some angry eyes turned your way.  Add a sprinkling of PCI / SOX for some measure and you have a problem.

Joe Yeager
Security Engineer
HP Software, formally SPI Dynamics
London, UK

-----Original Message-----
From: J. Oquendo [mailto:sil at infiltrated.net]
Sent: 25 September 2007 9:51 PM
To: Arian J. Evans
Cc: Marco; websecurity at webappsec.org
Subject: Re: [WEB SECURITY] Appropriate platform for testing/evaluating Web-Security-Scanners on?

Arian J. Evans wrote:

> #1 -- You have to test on your own software. There's really no substitute.

I've a friend who I've known for about 9 years now. Really talented guy
who knows his stuff. From time to time we speak about tools, what
methods he uses, what methods I use and I've always respected his stance
on not using any tools but relying on common knowledge of security from
the ground up.

> #2 If you don't know what your vulns are ahead of time, from manually
> testing and from source, you are only benchmarking noise ratios between
> the evaluated scanners.

Not only what is mentioned here but I'll answer this from a pentesting
scenario - not a "so how do I secure my servers." From a pentesting
scenario ask yourself if you believe what it is you seeing based on the
output of product X. For the older crowd who'd remember tools like
Deception Tool Kit, Portsentry, etc., I don't believe tools always tell
the whole picture let alone a truthful picture. Sure most make things
easier and quicker, but relying on a tool for security is pointless. How
many of us read about million dollar networks being compromised while
having the latest million dollar firewalls, IDS'/IPS', etc.

> Personally I do not believe most scanner benchmarks and reviews are
> functional. I believe they are dysfunctional in the sense that what folks
> *intend to measure* is "how well does this scanner work" but what they
> really measure is "scanner signal to noise ration in relation to each
other".

I think most scanners give nothing more than a benchmark if anything.
Too many up and coming security engineers are relying on running NMAP,
Ethereal, followed by Nessus, Wikto and others. Far too many of these
guys have in my perspective little talents. I know 12 year olds that can
do this. Security is becoming such a PACH (Point And Click Hacker)
industry and certfactory.

> bias disclaimer: I work on a "webappsec scanner" for my employer.
> Since we are a managed service for hundreds of websites, if anything,
> this makes me feel even more reality-grounded.

You sell out you ;)

As for an opinion on the subject... Make your own scanner, make your own
tools. Understand what it is you're attacking and tailor it. Once you
have an in-depth understanding of things, you'd be surprised to see
you'll quite often discover holes that weren't detected by someone else
tools. Outside of this one can develop 1) greater skills to sell
themselves with (I detected X while 50 other scanners saw nothing) 2) a
faster mechanism to target specifics...

E.g. A huge portion of current web-pentesting I do, involves me using
text files, NC, POST and HEAD. Sure I use Wikto from time to time,
Cenzic, etc... I use them when I'm sort of lazy and want to hurry and
shut up management who are rushing to see results... Its often when I'm
picking my nose out of boredom using insane awk/sed/perl/ruby commands
my mind wanders and I try obscure things. This comes a lot from time
spent configuring and breaking things as a system engineer, reading one
too many mailing list and blogs, drinking too much caffeine and too much
time on my hands.



--
====================================================
J. Oquendo
"Excusatio non petita, accusatio manifesta"

http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xF684C42E
sil . infiltrated @ net http://www.infiltrated.net


----------------------------------------------------------------------------
Join us on IRC: irc.freenode.net #webappsec

Have a question? Search The Web Security Mailing List Archives: 
http://www.webappsec.org/lists/websecurity/

Subscribe via RSS: 
http://www.webappsec.org/rss/websecurity.rss [RSS Feed]