[WEB SECURITY] Experience with using HTTP proxy tools for QA testers

Andre Gironda andreg at gmail.com
Mon Feb 28 17:59:24 EST 2011

On Mon, Feb 28, 2011 at 2:00 PM, Rohit Sethi <rklists at gmail.com> wrote:
> I think relying soly on QA for security testing is a slippery slope. We've

I don't think I've ever seen anyone propose that, but I'd fight it ;>

> heard interest from our clients about reducing the number of vulnerabilities
> that get caught in pen testing, rather than using pen testing as the only
> testing method. Rather than making QA comprehensive, we're trying to see
> which low-hanging fruit can be caught with minimal effort - both by those
> who use automated run-time assessment scanners and those who do not.

More on this below in psiinon's comments.

> Interestingly, we're not seeing many of our clients make use of QA
> automation tools. While developers may make extensive integration testing
> suites in JUnit/NUnit we've found less evidence of QTP / Selnium being used
> in enterprise QA. What have your experiences been? Clearly automation is
> important but if it requires training in a new tool then it's less likely to
> be adopted.

I think it's more common to see testing automation in place where
there is also a focus on application lifecycle management. If a CxO or
R&D director/manager has focused on hygiene to solve some systemic
project issues, then you'll probably find testing automation.

Because many development projects, including Enterprise, involve only
one developer, you will only find JUnit/NUnit where the app developer
has done their own work (usually because they were given the time by
management to focus on these efforts), and usually they have had past
experience with unit testing frameworks. QTP/Selenium style testing
automation, as you suggest, is even rarer to find -- possibly because
app developers do not have the experience or exposure to these at all.
I haven't seen training budget for test automation frameworks and we
do not see these classes being offered -- especially not at the same
pace as app development or even app penetration-testing.

When I see application developers get to a major iteration for a
project/product that is absolutely going to be released -- and they
know that the app is going to be around for 5+ years or decades...
then I always question them: Why haven't you put more rigor into unit
and component test automation when you know that somebody (even if
it's not you) is going to be refactoring this project/product?

An even more important question would be about keeping the test
automation up-to-date. I have seen many projects implement JUnit with
either HtmlUnit or Selenium to later abandon the test automation,
leaving it several revisions out of date compared to the latest
iteration of the system build. When performing continuous integration,
particularly nightly builds -- all build code and test code should
remain in working order on the build server(s).

Rohit -- one interesting point here is that you and your company may
have had experience with many development shops that lack hygiene in
both test automation and appsec because the two concepts are
intrinsically linked. It's a lot less likely that you'll walk into a
dev shop that recently had an appsec-related incident (or proactively
hires consultants because they are worried) to find that the
documentation and test automation is world-class.

On Mon, Feb 28, 2011 at 2:30 PM, psiinon <psiinon at gmail.com> wrote:
> As I try to make clear in the preso here:
> http://www.owasp.org/images/c/c8/Conference_Style_slides_for_ZAP.ppt I
> definitely dont think we should just rely on QA for security testing.
> There is no silver bullet, and so this isnt it ;)
> However professional pentesting is expensive, and often gets done late
> in the product cycle.

Why is penetration-testing performed late in the product cycle? Why
not put pen-testers and app developers together during a
project/product's first iteration? If there is a "system" (anything
past the point of wireframes to include deployable instances or even
mocks) to test, I say test it when it's considered a system, not when
it's considered "ready" by some other standards. An afternoon spent at
an early iteration would easily find and fix more issues than a more
formal pen-test right before an app is flagged for push into
production. Better, using continuous deployments could allow
pen-testers and app developers to work together during blue
deployments (where the code push/release is active for some) instead
of green deployments (active for everyone) if a project/product is
already in production and this new iteration is actually a new code

> If you can catch even a few security vulnerabilities earlier on then
> it will save time, effort and therefore money.
> Its not cost effective to pay pen testers to find the simple issues -
> pick those up in QA and then get value for money from the
> professionals by clearing the way for them to look for the hard ones

I don't agree with this. Training dev-testers or quality testers to
understand HTTP/TLS requests/responses or other nuances (for appsec!)
is too difficult. It takes time, research, dedication, and focus to
become fluent with web application vulnerability
testing/exploitation/risk-analysis. I would rather that quality
testers understand the architectural appsec issues facing
in-development apps: e.g. they should know that Classic ASP and PHP
are ripe for problems as compared to a standardized ESAPI Java -- and
grab an appsec expert to discuss the ramifications coming down the
pipeline. In other words, QA should be trained as whistleblowers and
get the experts involved. They can easily be rewarded as such -- just
give bonuses to any QA teams or individuals that report risky
applications that include sensitive data to the InfoSec team.

Another thing here: There are no easy problems. You don't see
eradication of SQLi or XSS anywhere. QA is not going to add any value,
even when armed with the "right" tools (note: I don't believe any
commercial appsec tools today work in the ways that they are
marketed). You can take a QA person and turn him or her into an app
penetration-tester, but again -- this takes time, research,
dedication, and focus on appsec and only appsec. Bring this person
onto an appsec team instead of leaving them behind in QA. Best of all
-- they probably already know the app developers and have a better
working relationship with them than the appsec team does. Now put them
in between QA and development -- they should be on the development
team as a dev-tester, not at a quality/security gate.

In many ways, forcing or helping QA to add in some low-hanging fruit
vulnerability-finding techniques only placates the notion that appsec
is not being taken seriously.

On a completely separate topic point, I'd say that appsec teams and
app penetration-testers can learn more from test automation experts
and quality testers rather than vice-versa. I have never seen or heard
of an app pen-tester who utilizes equivalence classification wisely,
nor combinatorial/pairwise testing, exploratory testing, or
domain-specific knowledge. At the very least -- they do not call these
concepts out directly/correctly, or identify their own "ad-hoc"
techniques (even if similar, and even if formalized). If app
pen-testers aren't using the correct language, then who are they to
say that they know anything about testing software at all?


More information about the websecurity mailing list