[WEB SECURITY] RE: 02/2009 WASC WAF thread

[Arian J. Evans]

I am still not getting any direct answers to my WAF
questions with concrete examples to back them up.
(Except for offline emails.)

There are a lot of important questions we need to
answer regarding WAFs, but pragmatically, I think
the questions I posed are where we need to start.

<inline>

On Sun, Feb 8, 2009 at 10:59 AM, Martin O'Neal
<martin.oneal at corsaire.com> wrote:
>
>> 1) Have you ever dealt with a compromise incident
>> on a web app at an organization with a WAF, that
>> could not have been stopped by a WAF?
>
> We do the occasional cleanup, but not that many (it isn't core business
> for us).  But of the last three we have been called in to help on, all
> were relatively complex, and a WAF wouldn't have been able to help.

Would you be wiling to provide specifics?

Otherwise, I have to say, I am skeptical that they could not be solved. :)

I have solved attack mitigation on some fairly complicated
business logic issues with WAFs before, and sometimes using
simple techniques like parameter checksums and replacement,
or parameter correlation in a back-end database (for business
logic/semantic flaws that flow across multiple applications
plugged together).

I believe the top-tier providers (F5, Breach, Imperva, and
maybe the Citrix solution) are *capable* of preventing most
successful compromises I have seen of a web applications
(and web servers, for that matter).

Putting aside discussions about deployablity, stability, and
maintainability (which are equally important, but not the point
of this question)...I would like to see real-world examples of
exploitable flaws that mainstream WAFs cannot protect.

Obviously most WAFs will not do much out-of-the box.

Besides the above vendors, there are tier two and three
vendors of WAFs I am excluding in this discussion. Some of
the software widgets I've seen over the years don't even inspect
POST requests so they are necessarily limited given today's
threat landscape (e.g. LHF Bots can POST forms).

Fact: WAFs can solve most issues that lead to compromise
if they are properly configured.

I am painfully well aware of limitations in WAFs, particularly
in areas like dealing with canonicalization. I ran a lab with 9
WAFs in it at one point and my colleagues and I could virtually
always find a way through the WAFs using encoding and charset
tricks like have been discussed. (Heck, as late as 2005 a
few main WAFs did not support UTF-8 :). Yet...


Mitigation Fact: I have found very few classes of real-world
compromises of web applications that could not have been
trivially blocked by a WAF with simple configuration, if the
issues were known & understood by the WAF operator.


So...

If anyone has examples or is willing to provide factual evidence
of compromises that cannot be blocked by a WAF -- please share.

Otherwise I think our discussion is very valuable, but our
priorities are not in order.



>> 2) If #1, should it have been stopped by a WAF,
>> and do you have any idea why it was not? (e.g.-
>> product limitation, lack of configuration, etc.)


Again, most attacks will not be stopped by WAFs in
MagicElfInside auto-protection mode.

That problem is an artifact of marketing and sales pitches.

This is not what we are debating.


>> 3) What do you think the #1 reasons WAFs are not
>> helping your customers is?
>
> Mostly because they don't work very well. :)  Like I have said before,
> they try and fix problems by looking at the data validation of the user
> input as it enters the web tier.  I would suggest that the vast majority
> of vulns in a web app aren't caused by validation at all, but by
> failures to encode appropriately when the data leaves the web tier.  So
> even if you use a WAF to fix the problem, it hasn't (as it were).


Is this a fair summation of your words?:

"The WAFs you see (which ones?) do not help your customers because
of technical limitations, namely focusing on HTTP requests and not
inspecting/ filtering HTTP responses."

I have seen where this is an issue in some cases. I also believe that most
cases I have seen can be addressed on input if the problem is properly
described to the WAF.

Let's extend this discussion beyond "WAF vendors". (I am doing this
for a reason :)

IDS and IPS also commonly have this limitation. Ignoring responses is
a trick used to increase the performance numbers of IDS/IPS widgets.
So as IDS and IPS vendors dig into this realm, we will probably have
to accept that as a limitation.

I have said before and will say again: if IPS vendors decide to come
after the WAF market, I think they can and will crush it through
marketing simply by saying "we do that too".

I know folks still waiting on their IPS vendor for "a signature" to
detect/block their most recent webapp compromise.

The IPS market is not taking webappsec seriously today; at least
the vendors I have discussed this with so far. I have been told this
is because they do not see enough potential market growth and
value to invest resources in the problem. The perceived market
cap is largely due to the unspectacular success of WAFs in the
marketplace (at least, perhaps, up until now).

IBM's ISS Proventia product is the only IPS vendor I can find that
still claims to block web application attacks openly on their website.
Other vendors over the years have claimed this, but all of them
appear to have removed the claims except for IBM.

I do know several Proventia customers whom had their web
applications compromised. When they call IBM the response
is always "we don't have a signature for that yet".

Here is a snip of Proventia's marketing page:

http://www-935.ibm.com/services/us/index.wss/offerfamily/iss/a1030570

    * Application attacks
    * Attack obfuscation
    * Cross-site scripting attacks
    * Drive-by downloads
    * SQL injection attacks
    * Web browser attacks


XSS and SQLi? Technically they *could* but I think most of
us on the list know they don't do this in an effective enough
manner to stop the attacks we see today IRL.

C-levels remain confused on this front, especially when their
network engineers call IBM and hear they are "waiting on the
signature" that may never arrive. And WRT to malware injecting
SQL injection bots -- if and when they do provide signatures it
will probably be a case of "the kid has already pee'd in the pool"
and the IPS is not likely to scrub that out.

This said: the IPS vendors could play here if they wanted to.

2010 prediction: They will.

WAF market is blossoming. I would like to see Sourcefire
and Palo Alto networks step into this area. They appear to
be the two fastest growing IDS/IPS and UTM players.


>> 4) If your customers have a problem (or three) as
>> you stated below in using those WAFs, why do you not
>> solve your customers' problems with using their WAFs?
>
> For a couple of reasons.  Firstly we are usually assessing the apps; if
> we started configuring them too, then this would be a conflict of
> interest.  Secondly, as above, WAFs don't actually work well.  :)

Really? Perhaps business across the pond is a bit different than
what I have experienced.

The "Conflict of interest" card sounds like an excuse. :)

Seriously, though, I have never found a customer that did not
want help resolving challenging issues involving mitigation
and remediation. As my cell phone bill can attest -- I find that
I am usually one of the first people called in the event of a compromise
for mitigation and containment strategies, even when I have
only been playing the role of "auditor". I find it valuable to be
able to offer people a wide range of tactical and strategic steps
to contain their issues, from permissions auditing to WAF configuration
log analysis to developer training and SDL suggestions. etc.

Sometimes, rarely, folks would bring up "conflict of interest" when
discussing products vs. "auditing/pen-testing/VA" type services,
but at the end of the day people almost always want solutions
and I find tend to go with people they trust. If those people also
understand the problem deeply and have experience with a
customer's organizational realities, that often leads to a better
solution for the customer. IMHO.

Also -- an entire network VA + IDS/IPS market was built
around this concept by ISS. :)

Thanks for answering me Martin; I always appreciate your
views and the experience you bring to the discussion.


For concluding thought:


Firewalls didn't work so well in the early days, and neither did IDS.

Pen testers were often inexperienced or clueless about the
the "issues" they would find and report as exploitable.

Marcus Ranum preached for years that penetrate and patch
(or block) would never work. He advocated that the answer
was to "build secure networks". It is really easy, Marcus would
explain, to contain your threat landscape and remove your
attack surface if you rebuild your network from the ground up
to be secure, and train your developers to program more securely.

While that may be the right answer --  it does not appear to
be an answer that has found widespread acceptance, or success.

---
Arian Evans
Solipsistic Software Security Sophist

----------------------------------------------------------------------------
Join us on IRC: irc.freenode.net #webappsec

Have a question? Search The Web Security Mailing List Archives: 
http://www.webappsec.org/lists/websecurity/archive/

Subscribe via RSS: 
http://www.webappsec.org/rss/websecurity.rss [RSS Feed]

Join WASC on LinkedIn
http://www.linkedin.com/e/gis/83336/4B20E4374DBA