[WEB SECURITY] Need a vulnerable XML Web Service

Arian J. Evans arian.evans at anachronic.com
Sat Jun 5 16:16:21 EDT 2010


Ory,

Actually, the 85% was somewhat anecdotal. The 1% to 15% was not anecdotal,
and based upon vuln count, pulse and frequency, discovered per input, based
upon expected issues compared to per-input web application discover, and
issues uncovered from human review. We are extremely quantitative and stats
driven at our shop - you should know that. :)

Given the state of WS scanning, simply the act of parsing and manipulating
complex SOAP messages is an accomplishment in and of itself. Like I said -
you may be "state of the art" here, but I don't think that deserves bragging
rights in terms of the results folks get.

I don't tend to agree that it beats manual fuzzing. My experience here is
that scanners have limited value at best.

I have not seen your scanner in a few years, but I have seen the results,
and I have seen a pretty sizable sample of web services from which I base my
opinion.

You have probably put a lot of hard work over the years to get your scanner
to the point it is at today, and it may beat all the other desktop scanners
hands-down. Which naturally is going to make you proud of it, and proud of
the work you've done, and there's nothing wrong with that.

But in terms of my thoughts on functionality utility and results - I stated
my case in my previous message.

If you want to improve the discussion and correct any errors in my
statements - start publishing analysis details and data. What vulns do you
find, how many, where? What type of web services do you see, how many can
you actually work on, who really uses SOAP, etc. etc.?

We have shied away from default BB scanning of web services for these
reasons:

1) A significant number were bespoke -e.g.-followed no standard known to
man, or loosely followed some standard(s) at best. Only a subset use SOAP.

2) Little to now value from default (lame) scanning

3) Fairly significant cost to customize messaging and communication for each
bespoke web service to effectively "scan" for still relatively minimal
results gain

4) Inability to find some of the biggest, glaring holes (AuthC/Z) which are
in comparison relatively easy to find in a web app via automation (even
though I don't think the desktop webapps scanners do this well at all,
today)

Well, my list could go on, but I think it is all more of the same initial
point I made.

If I'm not thinking about this correctly, missing something obvious, please
don't hesitate to correct me. Large samples of data help too. :)

One thing I have to compliment IBM on - they have at least been honest when
the marketing runs out of control in webappsec. What was it, a year or two
ago, that the Proventia device claimed to "block" a bunch of the WASC TC/24
items, and even prevent "phishing"?

When we questioned these "features" their product management folks came
forward and apologized for marketing zeal, and reality-grounded the message.
Very impressive,

---
Arian Evans
Software Security Scanner Sophisticate


On Sat, Jun 5, 2010 at 11:38 AM, Ory Segal <SEGALORY at il.ibm.com> wrote:

> Arian,
>
> Are you sure it's 85% and not 87.5%? you sure know how to make things sound
> scientific ;-)
>
> Hey - we (IBM) invest a lot of effort in AppScan's SOAP testing
> capabilities. Yes it is limited, but it beats the hell out of doing a lot of
> manual SOAP fuzzing, and it does take care of most WS-Security requirements,
> which removes the burden of doing things yourself.
>
> Naturally, we can't validate the results of the testing on the back-end
> systems, but that's the nature of blackbox scanning.
>
> As I've said - unlike other scanners, that simply read WSDLs and send some
> lame SOAP messages, we do invest a lot of effort in being able to create,
> analyze and submit more complex SOAP messages (WS-Addressing, WS-Security,
> certificates, etc.), you should really give it a try sometime.
>
> -Ory
>
>
> *-------------------------------------------------------------**
> Ory Segal
> Security Products Architect*
> *AppScan Product Manager*
> Rational, Application Security
> IBM Corporation
> Tel: +972-9-962-9836
> Mobile: +972-54-773-9359
> e-mail: *segalory at il.ibm.com* <segalory at il.ibm.com>
>
>
>
>
> From:        "Arian J. Evans" <arian.evans at anachronic.com>
> To:        Arshan Dabirsiaghi <arshan.dabirsiaghi at aspectsecurity.com>
> Cc:        Ory Segal/Haifa/IBM at IBMIL, Tom Stripling <
> tstripling at appsecconsulting.com>, 7Lyrix <7lyrix at gmail.com>, Nilesh
> Bhosale <nilesh at gslab.com>, websecurity at webappsec.org
> Date:        05-06-2010 01:18 AM
> Subject:        Re: [WEB SECURITY] Need a vulnerable XML Web Service
> Sent by:        arian.evans at gmail.com
> ------------------------------
>
>
>
> For once I have absolutely agree with Arshanti from Aspect. Except for his
> unfounded praise of Black Box scanners on web services.
>
> Let's be blunt here gentlemen: scanners are between 85% ineffective to
> utterly useless for testing web services.
>
> Scanner Success Scenario: The only scenario where scanners work reasonably
> well is when a RESTful web service is bound to an identical UI, as some
> modern frameworks automatically provide (say, Drupal or Ruby on Rails). This
> way you can trace data, output, and behavioral changes via monitoring the
> UI. However - this does not ensure that the controls (like AuthC/Z) test are
> the same between the API and the UI.
>
> Web Service Test Requirements: In the vast majority of test-cases for web
> services -- there are important unidirectional functions whose security
> implications are not visible or measurable through a response from the web
> service (which BB scanners ultimately rely on).
>
> So, sure, your scanner might "hack the crap out of it" but you and your
> scanner will never know, unless you go look in the async MQs, the DBs, the
> logs, etc. etc. and see what is there, and what transactions actually
> occurred. Scanners by and large cannot provide this insight, and in my
> experience most scanner jockeys rarely go and perform deeper analysis, nor
> know how to.
>
> Web Service Scanning Results: With BB scanning you can find a small subset
> of verbose SQLi and directory traversal, and that's about it. There are edge
> cases where you can find more, but by and large you will find little to
> nothing. That's a fact.
>
> Web Service "Standards": Not to mention that many, many web services do not
> follow standards, so you have to customize your BB scanner for each one,
> which most BB scanners aren't very good at handling.
>
> Web Service Authentication Holes: The biggest issues with web services are
> AuthC/Z issues. Syntax injection issues are there too, but nothing spells
> Security-Fail like AuthC issues which are fairly common in WS from what I've
> seen. But don't take my word for it - talk to the folks on WASC and OWASP
> who specialize in testing web services and they will tell you the same
> thing.
>
> Source Code Scanners on Web Services: Source code scanners are fairly
> limited here for different reasons, but the outcome is usually the same.
> Source code scanners are better at finding potential SQLi than BB/runtime
> scanners, but they also tend to fail at finding AuthC/Z issues and design
> pattern problems out of the box. (some source code scanners can be
> customized to do this).
>
> Binary Analysis on Web Services: I would expect binary analysis to be
> similarly, but more-so, limited than source code scanners, due to
> limitations in tracing regression and such. Basically the way most web
> services are used are as a transaction and message broker that is part of a
> larger, distributed, and complex system, and analyzing these for security
> implications effectively requires the sum of the parts, which is not
> something binary analysis is very good at/for.
>
>
> Conclusion: So what are we left with?
>
> I recommend people perform the following on web services:
>
> + Threat modeling, especially on B2B or WS to WS, to make sure you
> understand your attack surface
>
> + Human source-code review, to review your controls implementation
> particularly AuthC/Z
>
> + Run-time analysis to see how the running engine handles input thrown into
> it
>
> + A customized source-code scanner might be useful here for repeated
> testing over time
>
>
> Deep Thoughts from a Black Box Scanner Builder: At my work we have to say
> that we "scan web services" for business reasons. Basically to check off the
> check-box on RFPs that we respond to. We have to do this thanks to all the
> Scanner-Spin-Doctors out there inflating claims about what value they
> provide when scanning web services with their Windows-based desktop scanning
> widgets. (Or the hosted "SaaS" versions of the the same widgets.)
>
> However - while we say "yes we scan web services" - We make sure we are
> honest to our customers about what value we provide.
>
> Web Service Scanning Value: We tell people that they will get between 1%
> and 15% of the assessment depth & coverage (value) using our scanning engine
> on a web service, that they would expect to get from using it on a website.
> And we even customize the web service scanning for their unique WS messaging
> types, and know what we are doing, and even then that is about the best you
> can do with a scanner.
>
> Most people appreciate our honestly, and that we discourage our customers
> from using our scanning engine on web services unless they really want/need
> to.
>
> The quality of our black-box web scanning engine is excellent, and yes --
> it supports "SOAP messages", and more importantly -- also supports and the
> even more-common "SOAP-LIKE Messages" :). It also provides coverage features
> that no desktop widget provides today - and we are very proud of the quality
> of coverage it provides on Websites. Because of this we feel that it is a
> bit of  a sham to provide exceptionally high-quality coverage scanning web
> applications and then turn around and provide unequally low-quality coverage
> for web services under the same brand name.
>
> Most people appreciate honesty from what we see. We've only had a few folks
> wander away to marketing misdirection from magic web service scanning
> products.
>
> Since all scanners suck at testing web services, it's easy to spin this and
> for a given scanner make to say they have "industry leading security
> scanning coverage for web services" and actually be honest about it. Because
> they all suck pretty equally.
>
> But it doesn't change that fact it is utterly lame and misleading, and a
> bit manipulative of consumer expectations.
>
> Happy Friday!
>
> ---
> Arian Evans
> Software Security Scanner Sycophant
>
>
>
> On Fri, Jun 4, 2010 at 11:31 AM, Arshan Dabirsiaghi <*
> arshan.dabirsiaghi at aspectsecurity.com*<arshan.dabirsiaghi at aspectsecurity.com>>
> wrote:
> AppScan does a great job, I’m sure. Recently, though, I’ve had success
> using SoapUI (free version) for testing complex, WS-Security-protected
> endpoints. It’s got Groovy scripting capability that I’ve used to turn a few
> man days of scripting into good, automated, repeatable
> authentication/authorization tests for a whole suite of Web Services.
> Assertions are a big part of their framework and they make the tests easy to
> write. Naturally, injection testing is a little different.
>
>
>
> Sure, you could hook it up to a fuzz database pretty easily to launch
> injection attacks along the same line. Unfortunately, verifying the results
> of those tests is something at which AppScan will (probably) be much better.
> Can’t say Web Service injection testing has every been very fruitful for me,
> though. Karaoke-quality assessors will look for simple input reflected in
> output, but that is meaningless 99% of the time, since the response data is
> usually serialized reliably and not hand built, and couldn’t be exploited
> even if it was. SQL injection is pretty much the only hope you have for
> meaningful results at the time of the test.
>
>
>
> Assessments are usually scoped in such a way that the app that consumes
> your Web Service input is not being tested at the same time as the Web
> Service itself. So, your opportunity to observe evidence of a successful
> attack is slim. Said attacks are probably way downstream but probably have a
> better than average chance at success, if I had to guess.
>
>
>
> Arshan
>
>
>
> *From:* Ory Segal [mailto:*SEGALORY at il.ibm.com* <SEGALORY at il.ibm.com>]
> *Sent:* Friday, June 04, 2010 6:11 AM*
> To:* Tom Stripling*
> Cc:* '7Lyrix'; 'Nilesh Bhosale'; *websecurity at webappsec.org*<websecurity at webappsec.org>
>
> *Subject:* RE: [WEB SECURITY] Need a vulnerable XML Web Service
>
>
>
> AFAIK, You have two options, one is a product that my company sells -
> [ALERT!!! - propaganda, if you don't want to read, skip to part (B)]
>
> (A) Luckily for me, [*sorry about the shameless propaganda*] I am using
> IBM Rational AppScan, which comes with a SOAP tool called GSC (Generic SOAP
> Client). GSC supports WS-Security 1.1, SOAP attachments, WS-Addressing,
> certificates, etc.
>
> In addition, AppScan itself is capable of testing SOAP messages.  *
> http://www-01.ibm.com/software/awdtools/appscan/*<http://www-01.ibm.com/software/awdtools/appscan/>
>
>
> (B) As far as I know, the only other tool that is capable of using
> WS-Security and other WS-* is SOAPUI ( *http://www.soapui.org/*<http://www.soapui.org/>), but it doesn't have the same security testing capabilities as AppScan
>
> -Ory
> *
> -------------------------------------------------------------
> Ory Segal
> Security Products Architect* *
> AppScan Product Manager*
> Rational, Application Security
> IBM Corporation
> Tel: +972-9-962-9836
> Mobile: +972-54-773-9359
> e-mail: *segalory at il.ibm.com* <segalory at il.ibm.com>
>
>
>
>
> From:        "Tom Stripling" <*tstripling at appsecconsulting.com*<tstripling at appsecconsulting.com>
> >
> To:        Ory Segal/Haifa/IBM at IBMIL, "'7Lyrix'" <*7lyrix at gmail.com*<7lyrix at gmail.com>>,
> <*websecurity at webappsec.org* <websecurity at webappsec.org>>
> Cc:        "'Nilesh Bhosale'" <*nilesh at gslab.com* <nilesh at gslab.com>>
> Date:        04-06-2010 01:02 AM
> Subject:        RE: [WEB SECURITY] Need a vulnerable XML Web Service
>
> ------------------------------
>
>
>
>
> Along that vein, most penetration testing tools I’ve used are awful at
> constructing WS-Security headers for manual testing.  Excluding high-dollar
> automated scanners (I’m not trying to start a vendor war here), what tools
> do you all use for testing web services when WS-Security, SAML, etc. are
> being used?  I have yet to find a free or cheap tool that allows me to test
> these effectively.  I sometimes have to resort to building a testing client
> myself just so I can interact with the web service manually.
>
>   *
> From:* Ory Segal [*mailto:SEGALORY at il.ibm.com* <SEGALORY at il.ibm.com>] *
> Sent:* Thursday, June 03, 2010 1:39 PM*
> To:* 7Lyrix; *websecurity at webappsec.org* <websecurity at webappsec.org>*
> Cc:* Nilesh Bhosale*
> Subject:* Re: [WEB SECURITY] Need a vulnerable XML Web Service
>
> Has anyone noticed how all these "theoretical / tutorial" Web Services,
> never use any WS-Security?
>
> a) Real world SOAP web services, in real SOA environments, usually come
> with plenty of WS-Security
>
> b) There are plenty of things that can go wrong when implementing
> WS-Security, yet most security experts and most demo web sites, tend to talk
> about SQL-Injection over SOAP. I've seen some External entities here and
> there, but that's as deep as it goes most times.
>
> -Ory *
>
> -------------------------------------------------------------
> Ory Segal
> Security Products Architect* *
> AppScan Product Manager*
> Rational, Application Security
> IBM Corporation
> Tel: +972-9-962-9836
> Mobile: +972-54-773-9359
>
>
>
>
> From:        7Lyrix <*7lyrix at gmail.com* <7lyrix at gmail.com>>
> To:        Nilesh Bhosale <*nilesh at gslab.com* <nilesh at gslab.com>>
> Cc:        *websecurity at webappsec.org* <websecurity at webappsec.org>
> Date:        03-06-2010 09:17 PM
> Subject:        Re: [WEB SECURITY] Need a vulnerable XML Web Service
>
>
>
> ------------------------------
>
>
>
>
>
> A few.
> See:*
> **http://www.yehg.net/lab/pr0js/training/webgoat.php#Web_Services*<http://www.yehg.net/lab/pr0js/training/webgoat.php#Web_Services>
>
>
> On Thu, Jun 3, 2010 at 3:17 PM, Nilesh Bhosale <*nilesh at gslab.com*<nilesh at gslab.com>>
> wrote:
> > Thanks for all the responses.
> >
> > Does Webgoat has a vulnerable XML Webservice?
> >
> > Thanks,
> > Nilesh
> >
> > On Thursday 03 June 2010 11:36 AM, 7Lyrix wrote:
> >
> > Try webgoat:
> > *http://www.owasp.org/index.php/Category:OWASP_WebGoat_Project*<http://www.owasp.org/index.php/Category:OWASP_WebGoat_Project>
> >
> > Ivan Buetler, thank you for *http://www.hacking-lab.com*<http://www.hacking-lab.com/>.
> It's very great.
> >
> > On Wed, Jun 2, 2010 at 3:23 PM, Nilesh Bhosale <*nilesh at gslab.com*<nilesh at gslab.com>>
> wrote:
> >
> >
> > I need a vulnerable Web Service which I can use to try out and learn some
> of
> > the XML web service attacks.
> >
> > Thanks,
> > Nilesh
> >
> >
> ----------------------------------------------------------------------------
> > Join us on IRC: *irc.freenode.net* <http://irc.freenode.net/> #webappsec
> >
> > Have a question? Search The Web Security Mailing List Archives:
> > *http://www.webappsec.org/lists/websecurity/archive/*<http://www.webappsec.org/lists/websecurity/archive/>
> >
> > Subscribe via RSS: *http://www.webappsec.org/rss/websecurity.rss*<http://www.webappsec.org/rss/websecurity.rss>[RSS Feed]
> >
> > Join WASC on LinkedIn
> > *http://www.linkedin.com/e/gis/83336/4B20E4374DBA*<http://www.linkedin.com/e/gis/83336/4B20E4374DBA>
> >
> >
> >
> >
> >
> >
>
>
> ----------------------------------------------------------------------------
> Join us on IRC: *irc.freenode.net* <http://irc.freenode.net/> #webappsec
>
> Have a question? Search The Web Security Mailing List Archives: *
> **http://www.webappsec.org/lists/websecurity/archive/*<http://www.webappsec.org/lists/websecurity/archive/>
>
> Subscribe via RSS: *
> **http://www.webappsec.org/rss/websecurity.rss*<http://www.webappsec.org/rss/websecurity.rss>[RSS Feed]
>
> To unsubscribe email *websecurity-unsubscribe at webappsec.org*<websecurity-unsubscribe at webappsec.org>and reply to
> the confirmation email
>
> Join WASC on LinkedIn*
> **http://www.linkedin.com/e/gis/83336/4B20E4374DBA*<http://www.linkedin.com/e/gis/83336/4B20E4374DBA>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webappsec.org/pipermail/websecurity_lists.webappsec.org/attachments/20100605/b0f9d2bc/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 2359 bytes
Desc: not available
URL: <http://lists.webappsec.org/pipermail/websecurity_lists.webappsec.org/attachments/20100605/b0f9d2bc/attachment.gif>


More information about the websecurity mailing list