Tony Turner tony.turner at owasp.org
Fri Oct 9 08:20:09 EDT 2015

Hi Christian, thanks for your question. One of the things I'd like want to
get away from in a future version of WAFEC, not sure if it will make the
next release, is the yes/no responses. Ideally I'd like to identify a
scoring mechanism, say on a scale of 1-10 how a product meets a specific
criteria. So 2 WAF vendors may both have signatures for XSS, but one may be
much better at detecting evasion attempts, maybe one doesn't normalize,
etc. or 2 vendors may have mechanisms for mitigating that vary in
effectiveness. Otherwise with a binary approach for core capabilities, you
are correct there will likely be very little deviation from many WAF
vendors until we start hitting the extrinsic criteria. There's likely a ton
of research and testing as well as some tool development efforts to support
more granular evaluations such as this, which is why I think the next
release may be too soon for this.

The additional qualifiers you mention could be covered by that scaled
approach. Some of these qualifiers will also be called out in a separate
set of extrinsic criteria. For instance, ease of use might be a category on
its own that includes criteria for specific items related to ease of use
such as GUI functionality as well as criteria that allows the evaluator to
judge how intuitive the interface is, tuning violations and policies, how
difficult to generate or customize reports, etc. Additional criteria like
architectural change categories to clearly spell out where the solution
requires things like additional ports to be opened, changing IP schemes, OS
level changes like agent installation or authentication changes, etc. Other
extrinsic criteria such as user communities, vendor training and
documentation, product certification, sales process and more could be
evaluated in the same fashion, but I don't intend to go beyond technical
criteria, feature sets and usage for the purposes of WAFEC documentation.
I'm not sure I'm prepared to provide guidance in WAFEC documentation, for
example, identifying how vendors should be working with customers. Some of
these extrinsic categories may never find their way into the core WAFEC
document, but might still be included in a response matrix.

The biggest problem with those kinds of evaluations is they tend to be very
subjective and don't align well to a mature and repeatable process with
multiple end-users comparing results. When I've created similar matrixes in
the past, (namely for vulnerability management and SIEM products) I've
always done something similar here and admittedly done a poor job of
clearly identifying what is the difference between a 8 and a 6 as I went
with a gut feel and have typically been the only user (except when a
certain VM vendor got a copy of my matrix I used for a bake-off and thought
it would be a great sales tool). That's a maturity consideration that will
be planned for before we include that capability in a future Response

One thing I've done in the past that I intend to bring over, however, is
the concept of weighting. For instance I've typically used a weight of 1-5
so that I could assign different weights to criteria. For instance, maybe
you don't care as much about how impactful the deployment will be so you
assign those criteria a weight of 1, while the ability to have robust
support for policy tuning might be a 5 and associated scoring would have a
much greater impact on overall score.  This way, once a WAF has been
evaluated, even if the requirements change the scores can be easily
recalculated based on the new set of weighted requirements.

On Fri, Oct 9, 2015 at 3:15 AM, Christian Strache <cs at strache-it.de> wrote:

>  Dear Tony,
> your explanations sound reasonable.
> But, when you created your requirement catalogue and try to compare
> different products, many 'big players' (i.e. Gartner listed WAFs) offer the
> same set of features.
> So, for example using the WAFEC to find a suitable product for a customer
> with few requirements often ends up with a drawn in the evaluation.
> I guess if try to find out what features an AWS WAF offers, Blacklisting,
> Whitelisting, Signatures, regular updates etcetera the AWS WAF will fulfill
> those features. The quality and the customization features however may
> differ from the once you get with other 'on premise' products.
> Do you think it may be possible to add a criteria to the WAFEC to
> distinguish between:
> - yes, product offers feature XYZ
> and
> - yes, product offers feature XYZ based on a GUI/requires manual adaption
> or
> - yes, product offers features XYZ but only allows restricted adaption
> requiering operating system level changes.
> Kind regards,
> Christian
> *Von:* wasc-wafec [ <wasc-wafec-bounces at lists.webappsec.org>
> mailto:wasc-wafec-bounces at lists.webappsec.org
> <wasc-wafec-bounces at lists.webappsec.org>] *Im Auftrag von *Tony Turner
> *Gesendet:* Donnerstag, 8. Oktober 2015 21:20
> *An:* Anton Chuvakin <anton at chuvakin.org> <anton at chuvakin.org>
> *Cc:* wasc-wafec at lists.webappsec.org
> *Betreff:* Re: [WASC-WAFEC] AWS WAF
> Anton, A WAF is just a tool, it's all in how it's used. How can the tool
> be extended through additional extrinsic features? This is where the top
> players in the space (no I won't name names, we are all friends here)
> really start to shine and where they differentiate themselves from the
> competition. Ease of use, centralized management, threat feeds, load
> balancing, etc. Does that mean if you don't have those things you can't use
> the tools effectively? Does it make it an inferior product? What if I don't
> need an ADC-based WAF because I'm happy with my existing load balancer?
> Does that mean my WAF can't still perform its core functions to detect and
> mitigate web app attacks?
> I do understand the direction you are going in here, and yes this will
> likely be misused by compliance minded entities. I think it's important to
> remember that any 2 WAF consumers may have vastly different requirements.
> The objective here is to define the core of what a WAF is and should be
> capable of, but what exactly are we talking about here? Many (but not all)
> of these extrinsic criteria are dependent upon deployment architectures. We
> aren't always comparing reverse proxy to reverse proxy here, and in fact
> the customer may not even want to architect the solution that way. This
> makes WAF a more complex product to develop criteria for because
> implementations are frequently not very cookie-cutter. Then when you start
> talking about framework filters and RASP, the conversation gets even more
> confusing for a lot of people.
> I think when you take the typical Gartner-centric view of a solution space
> you are looking at a product category very generically. How can we evaluate
> vendors in a way that fits as many scenarios as possible with the greatest
> amount of options for consumers? That's great, and has substantial value.
> But that doesn't always map to real world requirements.
> Instead I think it makes sense to create a core framework for WAF criteria
> and provide a modular approach we can leverage when we develop the WAFEC
> Response Matrix that identifies those extrinsic criteria only when they are
> mapped to actual customer requirements. The criteria will be broken up into
> categories such that if you need SSL offload, or a cloud deployment we will
> have a set of criteria that map to that requirement. If you don't, why
> would you care about those capabilities?
> Lastly, there's really 2 audiences for WAFEC. Firstly, are the vendors
> that make WAFs and organizations like Gartner seeking to evaluate the
> product space. Secondly, are consumers who likely have very specific needs.
> They may not care about cloud deployments or DLP capabilities or how
> friendly the support staff are that they never need to call or <insert
> extrinsic capability>. I honestly think with this approach you can be as
> inclusive or exclusive as you want to when utilizing the framework for
> evaluation. So if you are a vendor or Gartner, or you have not identified
> all your requirements and you want to see who has the shiniest toys, you
> just check all the boxes. If you have more selective requirements, you can
> just build your menu and evaluate based on those criteria.
> -Tony
> On Thu, Oct 8, 2015 at 2:37 PM, Anton Chuvakin < <anton at chuvakin.org>
> <anton at chuvakin.org>anton at chuvakin.org> wrote:
> So, assuming this WAF would be "a basic WAF", do you guys think that a
> basic WAF is actually useful for anything apart from compliance? When was
> the last time somebody used a generic SQLi (like so
> <https://xkcd.com/327/> <https://xkcd.com/327/>https://xkcd.com/327/) to
> attack an app?    #just_wondering
> On Wed, Oct 7, 2015 at 9:23 PM, Christian Heinrich <
> <christian.heinrich at cmlh.id.au> <christian.heinrich at cmlh.id.au>
> christian.heinrich at cmlh.id.au> wrote:
> Tony,
> <http://www.slideshare.net/AmazonWebServices/sec323-new-securing-web-applications-with-aws-waf>
> <http://www.slideshare.net/AmazonWebServices/sec323-new-securing-web-applications-with-aws-waf>
> http://www.slideshare.net/AmazonWebServices/sec323-new-securing-web-applications-with-aws-waf
> are the slides of the WAF Product from Amazon Web Services:
> I would assume that the AWS partner/vendor integration is via
> <http://docs.aws.amazon.com/waf/latest/APIReference/Welcome.html>
> <http://docs.aws.amazon.com/waf/latest/APIReference/Welcome.html>
> http://docs.aws.amazon.com/waf/latest/APIReference/Welcome.html
> --
> Regards,
> Christian Heinrich
> http://cmlh.id.au/contact
> _______________________________________________
> wasc-wafec mailing list
> wasc-wafec at lists.webappsec.org
> <http://lists.webappsec.org/mailman/listinfo/wasc-wafec_lists.webappsec.org>
> <http://lists.webappsec.org/mailman/listinfo/wasc-wafec_lists.webappsec.org>
> http://lists.webappsec.org/mailman/listinfo/wasc-wafec_lists.webappsec.org
> --
> Dr. Anton Chuvakin
> Site: http://www.chuvakin.org
> Twitter: @anton_chuvakin <https://twitter.com/anton_chuvakin>
> Work: <http://www.linkedin.com/in/chuvakin>
> <http://www.linkedin.com/in/chuvakin>http://www.linkedin.com/in/chuvakin
> --
> Tony Turner
> OWASP Orlando Chapter Founder/Co-Leader
> WAFEC Project Leader
> STING Game Project Leader
> tony.turner at owasp.org
> <https://www.owasp.org/index.php/Orlando>
> <https://www.owasp.org/index.php/Orlando>
> https://www.owasp.org/index.php/Orlando

Tony Turner
OWASP Orlando Chapter Founder/Co-Leader
WAFEC Project Leader
STING Game Project Leader
tony.turner at owasp.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webappsec.org/pipermail/wasc-wafec_lists.webappsec.org/attachments/20151009/b18a3e07/attachment-0003.html>

More information about the wasc-wafec mailing list