Dear Tony,

your explanations sound reasonable.
But, when you created your requirement catalogue and try to compare
different products, many 'big players' (i.e. Gartner listed WAFs) offer the
same set of features.
So, for example using the WAFEC to find a suitable product for a customer
with few requirements often ends up with a drawn in the evaluation.
I guess if try to find out what features an AWS WAF offers, Blacklisting,
Whitelisting, Signatures, regular updates etcetera the AWS WAF will fulfill
those features. The quality and the customization features however may
differ from the once you get with other 'on premise' products.

Do you think it may be possible to add a criteria to the WAFEC to
distinguish between:

• yes, product offers feature XYZ
  and
• yes, product offers feature XYZ based on a GUI/requires manual adaption
  or
• yes, product offers features XYZ but only allows restricted adaption
  requiering operating system level changes.

Kind regards,
Christian

Von: wasc-wafec [ wasc-wafec-bounces@lists.webappsec.org
mailto:wasc-wafec-bounces@lists.webappsec.org
wasc-wafec-bounces@lists.webappsec.org] *Im Auftrag von *Tony Turner
Gesendet: Donnerstag, 8. Oktober 2015 21:20
An: Anton Chuvakin anton@chuvakin.org anton@chuvakin.org
Cc: wasc-wafec@lists.webappsec.org
Betreff: Re: [WASC-WAFEC] AWS WAF

Anton, A WAF is just a tool, it's all in how it's used. How can the tool
be extended through additional extrinsic features? This is where the top
players in the space (no I won't name names, we are all friends here)
really start to shine and where they differentiate themselves from the
competition. Ease of use, centralized management, threat feeds, load
balancing, etc. Does that mean if you don't have those things you can't use
the tools effectively? Does it make it an inferior product? What if I don't
need an ADC-based WAF because I'm happy with my existing load balancer?
Does that mean my WAF can't still perform its core functions to detect and
mitigate web app attacks?

I do understand the direction you are going in here, and yes this will
likely be misused by compliance minded entities. I think it's important to
remember that any 2 WAF consumers may have vastly different requirements.
The objective here is to define the core of what a WAF is and should be
capable of, but what exactly are we talking about here? Many (but not all)
of these extrinsic criteria are dependent upon deployment architectures. We
aren't always comparing reverse proxy to reverse proxy here, and in fact
the customer may not even want to architect the solution that way. This
makes WAF a more complex product to develop criteria for because
implementations are frequently not very cookie-cutter. Then when you start
talking about framework filters and RASP, the conversation gets even more
confusing for a lot of people.

I think when you take the typical Gartner-centric view of a solution space
you are looking at a product category very generically. How can we evaluate
vendors in a way that fits as many scenarios as possible with the greatest
amount of options for consumers? That's great, and has substantial value.
But that doesn't always map to real world requirements.

Instead I think it makes sense to create a core framework for WAF criteria
and provide a modular approach we can leverage when we develop the WAFEC
Response Matrix that identifies those extrinsic criteria only when they are
mapped to actual customer requirements. The criteria will be broken up into
categories such that if you need SSL offload, or a cloud deployment we will
have a set of criteria that map to that requirement. If you don't, why
would you care about those capabilities?

Lastly, there's really 2 audiences for WAFEC. Firstly, are the vendors
that make WAFs and organizations like Gartner seeking to evaluate the
product space. Secondly, are consumers who likely have very specific needs.
They may not care about cloud deployments or DLP capabilities or how
friendly the support staff are that they never need to call or <insert extrinsic capability>. I honestly think with this approach you can be as
inclusive or exclusive as you want to when utilizing the framework for
evaluation. So if you are a vendor or Gartner, or you have not identified
all your requirements and you want to see who has the shiniest toys, you
just check all the boxes. If you have more selective requirements, you can
just build your menu and evaluate based on those criteria.

-Tony

On Thu, Oct 8, 2015 at 2:37 PM, Anton Chuvakin < anton@chuvakin.org
anton@chuvakin.organton@chuvakin.org> wrote:

So, assuming this WAF would be "a basic WAF", do you guys think that a
basic WAF is actually useful for anything apart from compliance? When was
the last time somebody used a generic SQLi (like so
https://xkcd.com/327/ https://xkcd.com/327/https://xkcd.com/327/) to
attack an app?    #just_wondering

On Wed, Oct 7, 2015 at 9:23 PM, Christian Heinrich <
christian.heinrich@cmlh.id.au christian.heinrich@cmlh.id.au
christian.heinrich@cmlh.id.au> wrote:

Tony,

http://www.slideshare.net/AmazonWebServices/sec323-new-securing-web-applications-with-aws-waf
http://www.slideshare.net/AmazonWebServices/sec323-new-securing-web-applications-with-aws-waf
http://www.slideshare.net/AmazonWebServices/sec323-new-securing-web-applications-with-aws-waf
are the slides of the WAF Product from Amazon Web Services:

I would assume that the AWS partner/vendor integration is via
http://docs.aws.amazon.com/waf/latest/APIReference/Welcome.html
http://docs.aws.amazon.com/waf/latest/APIReference/Welcome.html
http://docs.aws.amazon.com/waf/latest/APIReference/Welcome.html

--
Regards,
Christian Heinrich

http://cmlh.id.au/contact

wasc-wafec mailing list
wasc-wafec@lists.webappsec.org

http://lists.webappsec.org/mailman/listinfo/wasc-wafec_lists.webappsec.org
http://lists.webappsec.org/mailman/listinfo/wasc-wafec_lists.webappsec.org
http://lists.webappsec.org/mailman/listinfo/wasc-wafec_lists.webappsec.org

--

Dr. Anton Chuvakin
Site: http://www.chuvakin.org
Twitter: @anton_chuvakin https://twitter.com/anton_chuvakin
Work: http://www.linkedin.com/in/chuvakin
http://www.linkedin.com/in/chuvakinhttp://www.linkedin.com/in/chuvakin

--

Tony Turner
OWASP Orlando Chapter Founder/Co-Leader

WAFEC Project Leader

STING Game Project Leader
tony.turner@owasp.org
https://www.owasp.org/index.php/Orlando
https://www.owasp.org/index.php/Orlando
https://www.owasp.org/index.php/Orlando

Hi Christian, thanks for your question. One of the things I'd like want to
get away from in a future version of WAFEC, not sure if it will make the
next release, is the yes/no responses. Ideally I'd like to identify a
scoring mechanism, say on a scale of 1-10 how a product meets a specific
criteria. So 2 WAF vendors may both have signatures for XSS, but one may be
much better at detecting evasion attempts, maybe one doesn't normalize, etc.
or 2 vendors may have mechanisms for mitigating that vary in effectiveness.
Otherwise with a binary approach for core capabilities, you are correct
there will likely be very little deviation from many WAF vendors until we
start hitting the extrinsic criteria. There's likely a ton of research and
testing as well as some tool development efforts to support more granular
evaluations such as this, which is why I think the next release may be too
soon for this.

The additional qualifiers you mention could be covered by that scaled
approach. Some of these qualifiers will also be called out in a separate set
of extrinsic criteria. For instance, ease of use might be a category on its
own that includes criteria for specific items related to ease of use such as
GUI functionality as well as criteria that allows the evaluator to judge how
intuitive the interface is, tuning violations and policies, how difficult to
generate or customize reports, etc. Additional criteria like architectural
change categories to clearly spell out where the solution requires things
like additional ports to be opened, changing IP schemes, OS level changes
like agent installation or authentication changes, etc. Other extrinsic
criteria such as user communities, vendor training and documentation,
product certification, sales process and more could be evaluated in the same
fashion, but I don't intend to go beyond technical criteria, feature sets
and usage for the purposes of WAFEC documentation. I'm not sure I'm prepared
to provide guidance in WAFEC documentation, for example, identifying how
vendors should be working with customers. Some of these extrinsic categories
may never find their way into the core WAFEC document, but might still be
included in a response matrix.

The biggest problem with those kinds of evaluations is they tend to be very
subjective and don't align well to a mature and repeatable process with
multiple end-users comparing results. When I've created similar matrixes in
the past, (namely for vulnerability management and SIEM products) I've
always done something similar here and admittedly done a poor job of clearly
identifying what is the difference between a 8 and a 6 as I went with a gut
feel and have typically been the only user (except when a certain VM vendor
got a copy of my matrix I used for a bake-off and thought it would be a
great sales tool). That's a maturity consideration that will be planned for
before we include that capability in a future Response Matrix.

One thing I've done in the past that I intend to bring over, however, is the
concept of weighting. For instance I've typically used a weight of 1-5 so
that I could assign different weights to criteria. For instance, maybe you
don't care as much about how impactful the deployment will be so you assign
those criteria a weight of 1, while the ability to have robust support for
policy tuning might be a 5 and associated scoring would have a much greater
impact on overall score.  This way, once a WAF has been evaluated, even if
the requirements change the scores can be easily recalculated based on the
new set of weighted requirements.

--
Regards,
Christian Heinrich

http://cmlh.id.au/contact

wasc-wafec mailing list
wasc-wafec@lists.webappsec.org
http://lists.webappsec.org/mailman/listinfo/wasc-wafec_lists.webappsec.org

My responses inline below Christian, I don't necessarily agree with your
assessment
This distrust has to do with lack of transparency and use of a benchmark by
a commercial entity before it is a mature framework. Don't confuse the
issues with your bias against OWASP by making inflammatory statements out of
context. I'm very aware of the issues and do not intend to repeat those
mistakes.

I'm not quite sure what issue you are having here. Please elaborate. I do
not intend for WAFEC to perform evaluations. Its simply a framework and set
of tools for vendors, independent testing entities and consumers to conduct
their own evaluations. It should be designed for consistency and flexibility
to map as closely as possible to the unique scenario it is being used to
evaluate.

Lack of weighting makes all criteria equally important. For the purpose of a
generic evaluation, that's fine but that does not map to how consumers need
to use evaluation tools. Not all implementations are the same, not all
requirements are the same and not all consumers will consider criteria with
the exact same weight. I'm sorry but I strongly disagree with you here. I'm
not talking about setting weights, I'm talking about providing the
flexibility for the user of the tool to set their own weights. Please
provide concrete examples of how you find this to be problematic.

Secondly, yes/no answers are very easy to game unless we get very very
specific with the questions, far more specific than the previous version of
the Response Matrix. For example "Does the WAF support signature based
detection?" is a terrible binary question. There may be varying degrees of
how comprehensive default signatures are, how easy to create new, modify
existing, differences in regex that can impact performance, denial of
service conditions for overly greedy regex, etc.

I do not intend to provide a graduated scoring mechanism unless those
scores can be clearly defined.

Thanks Christian(s). It occurs to me that there is little difference between
a set of binary questions that produce a score and a graduated score that is
produced by using the well defined criteria I mentioned. I'm fine with
keeping it binary, and expanding the criteria as it makes sense to do so.

That being said, I'm very interested to hear from any metrics nerds on the
list as we could use a good statistician on the core team.

Thanks Christian(s). It occurs to me that there is little difference
between a set of binary questions that produce a score and a graduated
score that is produced by using the well defined criteria I mentioned. I'm
fine with keeping it binary, and expanding the criteria as it makes sense
to do so.

That being said, I'm very interested to hear from any metrics nerds on the
list as we could use a good statistician on the core team.