websecurity@lists.webappsec.org

The Web Security Mailing List

View all threads

program to crawl website looking for string patterns

YJ
Youngquist, Jason R.
Fri, Sep 16, 2011 1:55 PM

We are looking for a tool that can be configured to crawl for string patterns (ie. SSNs, credit card numbers, etc).  Cornell's Spider 2008 beta has this capability, but every time we used it, it crashed on us.

We also found a program called webshag, but it would only look for pre-defined stuff like email addresses or external links.

Did some googling, but haven't really found anything.  Thoughts?

Thanks.
Jason Youngquist, CISSP
Information Technology Security Engineer
Technology Services
Columbia College
1001 Rogers Street, Columbia, MO  65216
(573) 875-7334
jryoungquist@ccis.edu
http://www.ccis.edu

We are looking for a tool that can be configured to crawl for string patterns (ie. SSNs, credit card numbers, etc). Cornell's Spider 2008 beta has this capability, but every time we used it, it crashed on us. We also found a program called webshag, but it would only look for pre-defined stuff like email addresses or external links. Did some googling, but haven't really found anything. Thoughts? Thanks. Jason Youngquist, CISSP Information Technology Security Engineer Technology Services Columbia College 1001 Rogers Street, Columbia, MO  65216 (573) 875-7334 jryoungquist@ccis.edu http://www.ccis.edu
RD
Ryan Dewhurst
Fri, Sep 16, 2011 4:51 PM

w3af [0] has lot's of grepping plugins which can easily be expanded.
Should do what you want.

[0] http://w3af.sourceforge.net/

Ryan Dewhurst

blog www.ethicalhack3r.co.uk
projects www.dvwa.co.uk | www.webwordcount.com
twitter www.twitter.com/ethicalhack3r

On Fri, Sep 16, 2011 at 2:55 PM, Youngquist, Jason R.
jryoungquist@ccis.edu wrote:

We are looking for a tool that can be configured to crawl for string patterns (ie. SSNs, credit card numbers, etc).  Cornell's Spider 2008 beta has this capability, but every time we used it, it crashed on us.

We also found a program called webshag, but it would only look for pre-defined stuff like email addresses or external links.

Did some googling, but haven't really found anything.  Thoughts?

Thanks.
Jason Youngquist, CISSP
Information Technology Security Engineer
Technology Services
Columbia College
1001 Rogers Street, Columbia, MO  65216
(573) 875-7334
jryoungquist@ccis.edu
http://www.ccis.edu


The Web Security Mailing List

WebSecurity RSS Feed
http://www.webappsec.org/rss/websecurity.rss

Join WASC on LinkedIn http://www.linkedin.com/e/gis/83336/4B20E4374DBA

WASC on Twitter
http://twitter.com/wascupdates

websecurity@lists.webappsec.org
http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org

w3af [0] has lot's of grepping plugins which can easily be expanded. Should do what you want. [0] http://w3af.sourceforge.net/ Ryan Dewhurst blog www.ethicalhack3r.co.uk projects www.dvwa.co.uk | www.webwordcount.com twitter www.twitter.com/ethicalhack3r On Fri, Sep 16, 2011 at 2:55 PM, Youngquist, Jason R. <jryoungquist@ccis.edu> wrote: > We are looking for a tool that can be configured to crawl for string patterns (ie. SSNs, credit card numbers, etc).  Cornell's Spider 2008 beta has this capability, but every time we used it, it crashed on us. > > We also found a program called webshag, but it would only look for pre-defined stuff like email addresses or external links. > > Did some googling, but haven't really found anything.  Thoughts? > > > > Thanks. > Jason Youngquist, CISSP > Information Technology Security Engineer > Technology Services > Columbia College > 1001 Rogers Street, Columbia, MO  65216 > (573) 875-7334 > jryoungquist@ccis.edu > http://www.ccis.edu > > > _______________________________________________ > The Web Security Mailing List > > WebSecurity RSS Feed > http://www.webappsec.org/rss/websecurity.rss > > Join WASC on LinkedIn http://www.linkedin.com/e/gis/83336/4B20E4374DBA > > WASC on Twitter > http://twitter.com/wascupdates > > websecurity@lists.webappsec.org > http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org >
AM
Adam Muntner
Fri, Sep 16, 2011 5:23 PM

FuzzDB has a set of regex patterns

http://code.google.com/p/fuzzdb/source/browse/trunk/regex/pii.readme.txt
http://code.google.com/p/fuzzdb/source/browse/trunk/regex/pii.fuzz.txt

for the docs and set of test cases, respectively

Depending on your purpose and the value of depth and completeness to
your test, as well as the structure of the sites to be evaluated
themselves, you may not want to entirely rely on an automated crawler.
Many websites have entire sections that automated web crawlers can
never reach. The crawler may also not handle authenticated session
state well, without some effort. Solution: If depth is critical, make
sure to do a manual crawl, as well. You can load the PII regex
patterns into something like Burp, in order to monitor the traffic
passively for them. Then just have your automated spider tools also
run through burp. You may want to have Burp log the full request and
response to a file, for later, deeper analysis.

One thing you want to do post-crawl analysis for is that there may be
PII in the results, encoded in a way which your regex would miss.
Examples: SSN in URL encoding, base-64 encoded, unicode encoded, etc.

Sounds like an interesting project. Good luck!
Adam

On Fri, Sep 16, 2011 at 9:55 AM, Youngquist, Jason R.
jryoungquist@ccis.edu wrote:

We are looking for a tool that can be configured to crawl for string patterns (ie. SSNs, credit card numbers, etc).  Cornell's Spider 2008 beta has this capability, but every time we used it, it crashed on us.

We also found a program called webshag, but it would only look for pre-defined stuff like email addresses or external links.

Did some googling, but haven't really found anything.  Thoughts?

Thanks.
Jason Youngquist, CISSP
Information Technology Security Engineer
Technology Services
Columbia College
1001 Rogers Street, Columbia, MO  65216
(573) 875-7334
jryoungquist@ccis.edu
http://www.ccis.edu


The Web Security Mailing List

WebSecurity RSS Feed
http://www.webappsec.org/rss/websecurity.rss

Join WASC on LinkedIn http://www.linkedin.com/e/gis/83336/4B20E4374DBA

WASC on Twitter
http://twitter.com/wascupdates

websecurity@lists.webappsec.org
http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org

FuzzDB has a set of regex patterns http://code.google.com/p/fuzzdb/source/browse/trunk/regex/pii.readme.txt http://code.google.com/p/fuzzdb/source/browse/trunk/regex/pii.fuzz.txt for the docs and set of test cases, respectively Depending on your purpose and the value of depth and completeness to your test, as well as the structure of the sites to be evaluated themselves, you may not want to entirely rely on an automated crawler. Many websites have entire sections that automated web crawlers can never reach. The crawler may also not handle authenticated session state well, without some effort. Solution: If depth is critical, make sure to do a manual crawl, as well. You can load the PII regex patterns into something like Burp, in order to monitor the traffic passively for them. Then just have your automated spider tools also run through burp. You may want to have Burp log the full request and response to a file, for later, deeper analysis. One thing you want to do post-crawl analysis for is that there may be PII in the results, encoded in a way which your regex would miss. Examples: SSN in URL encoding, base-64 encoded, unicode encoded, etc. Sounds like an interesting project. Good luck! Adam On Fri, Sep 16, 2011 at 9:55 AM, Youngquist, Jason R. <jryoungquist@ccis.edu> wrote: > We are looking for a tool that can be configured to crawl for string patterns (ie. SSNs, credit card numbers, etc).  Cornell's Spider 2008 beta has this capability, but every time we used it, it crashed on us. > > We also found a program called webshag, but it would only look for pre-defined stuff like email addresses or external links. > > Did some googling, but haven't really found anything.  Thoughts? > > > > Thanks. > Jason Youngquist, CISSP > Information Technology Security Engineer > Technology Services > Columbia College > 1001 Rogers Street, Columbia, MO  65216 > (573) 875-7334 > jryoungquist@ccis.edu > http://www.ccis.edu > > > _______________________________________________ > The Web Security Mailing List > > WebSecurity RSS Feed > http://www.webappsec.org/rss/websecurity.rss > > Join WASC on LinkedIn http://www.linkedin.com/e/gis/83336/4B20E4374DBA > > WASC on Twitter > http://twitter.com/wascupdates > > websecurity@lists.webappsec.org > http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org >
TL
Tasos Laskos
Fri, Sep 16, 2011 6:25 PM
<shameless selfpromotion>

Or you could use arachni[1]:

  1. Create a module like:

module Arachni
module Modules
class MyModule < Arachni::Module::Base

 def initialize( page )
     @page = page
 end

 def run( )
     match_and_log( "the string you're looking for" )
 end

 def self.info
     {
         :name           => 'My module',
         :description    => %q{Greps pages for a string.},
         :author         => 'Your name',
         :version        => '0.1',
         :targets        => { 'Generic' => 'all' },
         :issue   => {
             :name        => %q{Found my string},
             :description => %q{some description},
             :cwe         => '',
             :severity    => Issue::Severity::LOW,
             :cvssv2      => '0',
             :remedy_guidance    => %q{Remode the damn thing.},
             :remedy_code => '',
         }
     }
 end

end
end
end

  1. Save it as "my_module.rb" and put it under "modules/recon/grep/"
  2. run arachni like so:
    arachni -m my_module <site url>

And you're good to go. :)

[1] http://arachni.segfault.gr/

</shameless selfpromotion>

On 09/16/2011 07:51 PM, Ryan Dewhurst wrote:

w3af [0] has lot's of grepping plugins which can easily be expanded.
Should do what you want.

[0] http://w3af.sourceforge.net/

Ryan Dewhurst

blog www.ethicalhack3r.co.uk
projects www.dvwa.co.uk | www.webwordcount.com
twitter www.twitter.com/ethicalhack3r

On Fri, Sep 16, 2011 at 2:55 PM, Youngquist, Jason R.
jryoungquist@ccis.edu  wrote:

We are looking for a tool that can be configured to crawl for string patterns (ie. SSNs, credit card numbers, etc).  Cornell's Spider 2008 beta has this capability, but every time we used it, it crashed on us.

We also found a program called webshag, but it would only look for pre-defined stuff like email addresses or external links.

Did some googling, but haven't really found anything.  Thoughts?

Thanks.
Jason Youngquist, CISSP
Information Technology Security Engineer
Technology Services
Columbia College
1001 Rogers Street, Columbia, MO  65216
(573) 875-7334
jryoungquist@ccis.edu
http://www.ccis.edu


The Web Security Mailing List

WebSecurity RSS Feed
http://www.webappsec.org/rss/websecurity.rss

Join WASC on LinkedIn http://www.linkedin.com/e/gis/83336/4B20E4374DBA

WASC on Twitter
http://twitter.com/wascupdates

websecurity@lists.webappsec.org
http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org

<shameless selfpromotion> Or you could use arachni[1]: 1) Create a module like: ---------------- module Arachni module Modules class MyModule < Arachni::Module::Base def initialize( page ) @page = page end def run( ) match_and_log( "the string you're looking for" ) end def self.info { :name => 'My module', :description => %q{Greps pages for a string.}, :author => 'Your name', :version => '0.1', :targets => { 'Generic' => 'all' }, :issue => { :name => %q{Found my string}, :description => %q{some description}, :cwe => '', :severity => Issue::Severity::LOW, :cvssv2 => '0', :remedy_guidance => %q{Remode the damn thing.}, :remedy_code => '', } } end end end end ---------------- 2) Save it as "my_module.rb" and put it under "modules/recon/grep/" 3) run arachni like so: arachni -m my_module <site url> And you're good to go. :) [1] http://arachni.segfault.gr/ </shameless selfpromotion> On 09/16/2011 07:51 PM, Ryan Dewhurst wrote: > w3af [0] has lot's of grepping plugins which can easily be expanded. > Should do what you want. > > [0] http://w3af.sourceforge.net/ > > Ryan Dewhurst > > blog www.ethicalhack3r.co.uk > projects www.dvwa.co.uk | www.webwordcount.com > twitter www.twitter.com/ethicalhack3r > > > > On Fri, Sep 16, 2011 at 2:55 PM, Youngquist, Jason R. > <jryoungquist@ccis.edu> wrote: >> We are looking for a tool that can be configured to crawl for string patterns (ie. SSNs, credit card numbers, etc). Cornell's Spider 2008 beta has this capability, but every time we used it, it crashed on us. >> >> We also found a program called webshag, but it would only look for pre-defined stuff like email addresses or external links. >> >> Did some googling, but haven't really found anything. Thoughts? >> >> >> >> Thanks. >> Jason Youngquist, CISSP >> Information Technology Security Engineer >> Technology Services >> Columbia College >> 1001 Rogers Street, Columbia, MO 65216 >> (573) 875-7334 >> jryoungquist@ccis.edu >> http://www.ccis.edu >> >> >> _______________________________________________ >> The Web Security Mailing List >> >> WebSecurity RSS Feed >> http://www.webappsec.org/rss/websecurity.rss >> >> Join WASC on LinkedIn http://www.linkedin.com/e/gis/83336/4B20E4374DBA >> >> WASC on Twitter >> http://twitter.com/wascupdates >> >> websecurity@lists.webappsec.org >> http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org >> > > _______________________________________________ > The Web Security Mailing List > > WebSecurity RSS Feed > http://www.webappsec.org/rss/websecurity.rss > > Join WASC on LinkedIn http://www.linkedin.com/e/gis/83336/4B20E4374DBA > > WASC on Twitter > http://twitter.com/wascupdates > > websecurity@lists.webappsec.org > http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org >
I
illumina7i
Mon, Sep 19, 2011 10:34 AM

This may also work:
https://github.com/urbanadventurer/WhatWeb/

On Fri, Sep 16, 2011 at 2:25 PM, Tasos Laskos tasos.laskos@gmail.com wrote:

<shameless selfpromotion>

Or you could use arachni[1]:

  1. Create a module like:

module Arachni
module Modules
class MyModule < Arachni::Module::Base

   def initialize( page )
       @page = page
   end

   def run( )
       match_and_log( "the string you're looking for" )
   end

   def self.info
       {
           :name           => 'My module',
           :description    => %q{Greps pages for a string.},
           :author         => 'Your name',
           :version        => '0.1',
           :targets        => { 'Generic' => 'all' },
           :issue   => {
               :name        => %q{Found my string},
               :description => %q{some description},
               :cwe         => '',
               :severity    => Issue::Severity::LOW,
               :cvssv2      => '0',
               :remedy_guidance    => %q{Remode the damn thing.},
               :remedy_code => '',
           }
       }
   end

end
end
end

  1. Save it as "my_module.rb" and put it under "modules/recon/grep/"
  2. run arachni like so:
           arachni -m my_module <site url>

And you're good to go. :)

[1] http://arachni.segfault.gr/

</shameless selfpromotion>

On 09/16/2011 07:51 PM, Ryan Dewhurst wrote:

w3af [0] has lot's of grepping plugins which can easily be expanded.
Should do what you want.

[0] http://w3af.sourceforge.net/

Ryan Dewhurst

blog www.ethicalhack3r.co.uk
projects www.dvwa.co.uk | www.webwordcount.com
twitter www.twitter.com/ethicalhack3r

On Fri, Sep 16, 2011 at 2:55 PM, Youngquist, Jason R.
jryoungquist@ccis.edu  wrote:

We are looking for a tool that can be configured to crawl for string
patterns (ie. SSNs, credit card numbers, etc).  Cornell's Spider 2008 beta
has this capability, but every time we used it, it crashed on us.

We also found a program called webshag, but it would only look for
pre-defined stuff like email addresses or external links.

Did some googling, but haven't really found anything.  Thoughts?

Thanks.
Jason Youngquist, CISSP
Information Technology Security Engineer
Technology Services
Columbia College
1001 Rogers Street, Columbia, MO  65216
(573) 875-7334
jryoungquist@ccis.edu
http://www.ccis.edu


The Web Security Mailing List

WebSecurity RSS Feed
http://www.webappsec.org/rss/websecurity.rss

Join WASC on LinkedIn http://www.linkedin.com/e/gis/83336/4B20E4374DBA

WASC on Twitter
http://twitter.com/wascupdates

websecurity@lists.webappsec.org

http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org

This may also work: https://github.com/urbanadventurer/WhatWeb/ On Fri, Sep 16, 2011 at 2:25 PM, Tasos Laskos <tasos.laskos@gmail.com> wrote: > <shameless selfpromotion> > > Or you could use arachni[1]: > > 1) Create a module like: > ---------------- > module Arachni > module Modules > class MyModule < Arachni::Module::Base > >    def initialize( page ) >        @page = page >    end > >    def run( ) >        match_and_log( "the string you're looking for" ) >    end > >    def self.info >        { >            :name           => 'My module', >            :description    => %q{Greps pages for a string.}, >            :author         => 'Your name', >            :version        => '0.1', >            :targets        => { 'Generic' => 'all' }, >            :issue   => { >                :name        => %q{Found my string}, >                :description => %q{some description}, >                :cwe         => '', >                :severity    => Issue::Severity::LOW, >                :cvssv2      => '0', >                :remedy_guidance    => %q{Remode the damn thing.}, >                :remedy_code => '', >            } >        } >    end > > end > end > end > ---------------- > 2) Save it as "my_module.rb" and put it under "modules/recon/grep/" > 3) run arachni like so: >        arachni -m my_module <site url> > > > And you're good to go. :) > > [1] http://arachni.segfault.gr/ > > </shameless selfpromotion> > > On 09/16/2011 07:51 PM, Ryan Dewhurst wrote: >> >> w3af [0] has lot's of grepping plugins which can easily be expanded. >> Should do what you want. >> >> [0] http://w3af.sourceforge.net/ >> >> Ryan Dewhurst >> >> blog www.ethicalhack3r.co.uk >> projects www.dvwa.co.uk | www.webwordcount.com >> twitter www.twitter.com/ethicalhack3r >> >> >> >> On Fri, Sep 16, 2011 at 2:55 PM, Youngquist, Jason R. >> <jryoungquist@ccis.edu>  wrote: >>> >>> We are looking for a tool that can be configured to crawl for string >>> patterns (ie. SSNs, credit card numbers, etc).  Cornell's Spider 2008 beta >>> has this capability, but every time we used it, it crashed on us. >>> >>> We also found a program called webshag, but it would only look for >>> pre-defined stuff like email addresses or external links. >>> >>> Did some googling, but haven't really found anything.  Thoughts? >>> >>> >>> >>> Thanks. >>> Jason Youngquist, CISSP >>> Information Technology Security Engineer >>> Technology Services >>> Columbia College >>> 1001 Rogers Street, Columbia, MO  65216 >>> (573) 875-7334 >>> jryoungquist@ccis.edu >>> http://www.ccis.edu >>> >>> >>> _______________________________________________ >>> The Web Security Mailing List >>> >>> WebSecurity RSS Feed >>> http://www.webappsec.org/rss/websecurity.rss >>> >>> Join WASC on LinkedIn http://www.linkedin.com/e/gis/83336/4B20E4374DBA >>> >>> WASC on Twitter >>> http://twitter.com/wascupdates >>> >>> websecurity@lists.webappsec.org >>> >>> http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org >>> >> >> _______________________________________________ >> The Web Security Mailing List >> >> WebSecurity RSS Feed >> http://www.webappsec.org/rss/websecurity.rss >> >> Join WASC on LinkedIn http://www.linkedin.com/e/gis/83336/4B20E4374DBA >> >> WASC on Twitter >> http://twitter.com/wascupdates >> >> websecurity@lists.webappsec.org >> >> http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org >> > > > _______________________________________________ > The Web Security Mailing List > > WebSecurity RSS Feed > http://www.webappsec.org/rss/websecurity.rss > > Join WASC on LinkedIn http://www.linkedin.com/e/gis/83336/4B20E4374DBA > > WASC on Twitter > http://twitter.com/wascupdates > > websecurity@lists.webappsec.org > http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org >
M
MaXe
Mon, Sep 19, 2011 5:22 PM

And if the website is public, some Google Kung Fu:
site:domain-name.tld keyword

That is of course, only what Google has seen and it will also only work if Google is not disabled from viewing the site.

~ MaXe
----- Original meddelelse -----

This may also work:
https://github.com/urbanadventurer/WhatWeb/

On Fri, Sep 16, 2011 at 2:25 PM, Tasos Laskos tasos.laskos@gmail.com
wrote:

<shameless selfpromotion>

Or you could use arachni[1]:

  1. Create a module like:

module Arachni
module Modules
class MyModule < Arachni::Module::Base

   def initialize( page )
       @page = page
   end

   def run( )
       match_and_log( "the string you're looking for" )
   end

   def self.info
       {
           :name           => 'My module',
           :description    => %q{Greps pages for a string.},
           :author         => 'Your name',
           :version        => '0.1',
           :targets        => { 'Generic' => 'all' },
           :issue   => {
               :name        => %q{Found my string},
               :description => %q{some description},
               :cwe         => '',
               :severity    => Issue::Severity::LOW,
               :cvssv2      => '0',
               :remedy_guidance    => %q{Remode the damn thing.},
               :remedy_code => '',
           }
       }
   end

end
end
end

  1. Save it as "my_module.rb" and put it under "modules/recon/grep/"
  2. run arachni like so:
           arachni -m my_module <site url>

And you're good to go. :)

[1] http://arachni.segfault.gr/

</shameless selfpromotion>

On 09/16/2011 07:51 PM, Ryan Dewhurst wrote:

w3af [0] has lot's of grepping plugins which can easily be expanded.
Should do what you want.

[0] http://w3af.sourceforge.net/

Ryan Dewhurst

blog www.ethicalhack3r.co.uk
projects www.dvwa.co.uk | www.webwordcount.com
twitter www.twitter.com/ethicalhack3r

On Fri, Sep 16, 2011 at 2:55 PM, Youngquist, Jason R.
jryoungquist@ccis.edu  wrote:

We are looking for a tool that can be configured to crawl for
string patterns (ie. SSNs, credit card numbers, etc).  Cornell's
Spider 2008 beta has this capability, but every time we used it,
it crashed on us.

We also found a program called webshag, but it would only look for
pre-defined stuff like email addresses or external links.

Did some googling, but haven't really found anything.  Thoughts?

Thanks.
Jason Youngquist, CISSP
Information Technology Security Engineer
Technology Services
Columbia College
1001 Rogers Street, Columbia, MO  65216
(573) 875-7334
jryoungquist@ccis.edu
http://www.ccis.edu


The Web Security Mailing List

WebSecurity RSS Feed
http://www.webappsec.org/rss/websecurity.rss

Join WASC on LinkedIn
http://www.linkedin.com/e/gis/83336/4B20E4374DBA

WASC on Twitter
http://twitter.com/wascupdates

websecurity@lists.webappsec.org

http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org

And if the website is public, some Google Kung Fu: site:domain-name.tld keyword That is of course, only what Google has seen and it will also only work if Google is not disabled from viewing the site. ~ MaXe ----- Original meddelelse ----- > This may also work: > https://github.com/urbanadventurer/WhatWeb/ > > On Fri, Sep 16, 2011 at 2:25 PM, Tasos Laskos <tasos.laskos@gmail.com> > wrote: > > <shameless selfpromotion> > > > > Or you could use arachni[1]: > > > > 1) Create a module like: > > ---------------- > > module Arachni > > module Modules > > class MyModule < Arachni::Module::Base > > > >    def initialize( page ) > >        @page = page > >    end > > > >    def run( ) > >        match_and_log( "the string you're looking for" ) > >    end > > > >    def self.info > >        { > >            :name           => 'My module', > >            :description    => %q{Greps pages for a string.}, > >            :author         => 'Your name', > >            :version        => '0.1', > >            :targets        => { 'Generic' => 'all' }, > >            :issue   => { > >                :name        => %q{Found my string}, > >                :description => %q{some description}, > >                :cwe         => '', > >                :severity    => Issue::Severity::LOW, > >                :cvssv2      => '0', > >                :remedy_guidance    => %q{Remode the damn thing.}, > >                :remedy_code => '', > >            } > >        } > >    end > > > > end > > end > > end > > ---------------- > > 2) Save it as "my_module.rb" and put it under "modules/recon/grep/" > > 3) run arachni like so: > >        arachni -m my_module <site url> > > > > > > And you're good to go. :) > > > > [1] http://arachni.segfault.gr/ > > > > </shameless selfpromotion> > > > > On 09/16/2011 07:51 PM, Ryan Dewhurst wrote: > > > > > > w3af [0] has lot's of grepping plugins which can easily be expanded. > > > Should do what you want. > > > > > > [0] http://w3af.sourceforge.net/ > > > > > > Ryan Dewhurst > > > > > > blog www.ethicalhack3r.co.uk > > > projects www.dvwa.co.uk | www.webwordcount.com > > > twitter www.twitter.com/ethicalhack3r > > > > > > > > > > > > On Fri, Sep 16, 2011 at 2:55 PM, Youngquist, Jason R. > > > <jryoungquist@ccis.edu>  wrote: > > > > > > > > We are looking for a tool that can be configured to crawl for > > > > string patterns (ie. SSNs, credit card numbers, etc).  Cornell's > > > > Spider 2008 beta has this capability, but every time we used it, > > > > it crashed on us. > > > > > > > > We also found a program called webshag, but it would only look for > > > > pre-defined stuff like email addresses or external links. > > > > > > > > Did some googling, but haven't really found anything.  Thoughts? > > > > > > > > > > > > > > > > Thanks. > > > > Jason Youngquist, CISSP > > > > Information Technology Security Engineer > > > > Technology Services > > > > Columbia College > > > > 1001 Rogers Street, Columbia, MO  65216 > > > > (573) 875-7334 > > > > jryoungquist@ccis.edu > > > > http://www.ccis.edu > > > > > > > > > > > > _______________________________________________ > > > > The Web Security Mailing List > > > > > > > > WebSecurity RSS Feed > > > > http://www.webappsec.org/rss/websecurity.rss > > > > > > > > Join WASC on LinkedIn > > > > http://www.linkedin.com/e/gis/83336/4B20E4374DBA > > > > > > > > WASC on Twitter > > > > http://twitter.com/wascupdates > > > > > > > > websecurity@lists.webappsec.org > > > > > > > > http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org > > > > > > > > > > _______________________________________________ > > > The Web Security Mailing List > > > > > > WebSecurity RSS Feed > > > http://www.webappsec.org/rss/websecurity.rss > > > > > > Join WASC on LinkedIn > > > http://www.linkedin.com/e/gis/83336/4B20E4374DBA > > > > > > WASC on Twitter > > > http://twitter.com/wascupdates > > > > > > websecurity@lists.webappsec.org > > > > > > http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org > > > > > > > > > _______________________________________________ > > The Web Security Mailing List > > > > WebSecurity RSS Feed > > http://www.webappsec.org/rss/websecurity.rss > > > > Join WASC on LinkedIn http://www.linkedin.com/e/gis/83336/4B20E4374DBA > > > > WASC on Twitter > > http://twitter.com/wascupdates > > > > websecurity@lists.webappsec.org > > http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org > > > > _______________________________________________ > The Web Security Mailing List > > WebSecurity RSS Feed > http://www.webappsec.org/rss/websecurity.rss > > Join WASC on LinkedIn http://www.linkedin.com/e/gis/83336/4B20E4374DBA > > WASC on Twitter > http://twitter.com/wascupdates > > websecurity@lists.webappsec.org > http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org
MJ
Menerick, John
Mon, Sep 19, 2011 6:14 PM

If bandwidth and the size of the site are not large concerns, a simple:  wget <parameters> www.example.com | grep -r 'regex for string' *  should serve as a simple sanity check.

-----Original Message-----
From: websecurity-bounces@lists.webappsec.org [mailto:websecurity-bounces@lists.webappsec.org] On Behalf Of MaXe
Sent: Monday, September 19, 2011 10:22 AM
To: illumina7i; Tasos Laskos
Cc: websecurity@webappsec.org
Subject: Re: [WEB SECURITY] program to crawl website looking for string patterns

And if the website is public, some Google Kung Fu:
site:domain-name.tld keyword

That is of course, only what Google has seen and it will also only work if Google is not disabled from viewing the site.

~ MaXe
----- Original meddelelse -----

This may also work:
https://github.com/urbanadventurer/WhatWeb/

On Fri, Sep 16, 2011 at 2:25 PM, Tasos Laskos tasos.laskos@gmail.com
wrote:

<shameless selfpromotion>

Or you could use arachni[1]:

  1. Create a module like:

module Arachni
module Modules
class MyModule < Arachni::Module::Base

   def initialize( page )
       @page = page
   end

   def run( )
       match_and_log( "the string you're looking for" )
   end

   def self.info
       {
           :name           => 'My module',
           :description    => %q{Greps pages for a string.},
           :author         => 'Your name',
           :version        => '0.1',
           :targets        => { 'Generic' => 'all' },
           :issue   => {
               :name        => %q{Found my string},
               :description => %q{some description},
               :cwe         => '',
               :severity    => Issue::Severity::LOW,
               :cvssv2      => '0',
               :remedy_guidance    => %q{Remode the damn thing.},
               :remedy_code => '',
           }
       }
   end

end
end
end

  1. Save it as "my_module.rb" and put it under "modules/recon/grep/"
  2. run arachni like so:
           arachni -m my_module <site url>

And you're good to go. :)

[1] http://arachni.segfault.gr/

</shameless selfpromotion>

On 09/16/2011 07:51 PM, Ryan Dewhurst wrote:

w3af [0] has lot's of grepping plugins which can easily be expanded.
Should do what you want.

[0] http://w3af.sourceforge.net/

Ryan Dewhurst

blog www.ethicalhack3r.co.uk
projects www.dvwa.co.uk | www.webwordcount.com
twitter www.twitter.com/ethicalhack3r

On Fri, Sep 16, 2011 at 2:55 PM, Youngquist, Jason R.
jryoungquist@ccis.edu  wrote:

We are looking for a tool that can be configured to crawl for
string patterns (ie. SSNs, credit card numbers, etc).  Cornell's
Spider 2008 beta has this capability, but every time we used it,
it crashed on us.

We also found a program called webshag, but it would only look for
pre-defined stuff like email addresses or external links.

Did some googling, but haven't really found anything.  Thoughts?

Thanks.
Jason Youngquist, CISSP
Information Technology Security Engineer
Technology Services
Columbia College
1001 Rogers Street, Columbia, MO  65216
(573) 875-7334
jryoungquist@ccis.edu
http://www.ccis.edu


The Web Security Mailing List

WebSecurity RSS Feed
http://www.webappsec.org/rss/websecurity.rss

Join WASC on LinkedIn
http://www.linkedin.com/e/gis/83336/4B20E4374DBA

WASC on Twitter
http://twitter.com/wascupdates

websecurity@lists.webappsec.org

http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org


The Web Security Mailing List

WebSecurity RSS Feed
http://www.webappsec.org/rss/websecurity.rss

Join WASC on LinkedIn http://www.linkedin.com/e/gis/83336/4B20E4374DBA

WASC on Twitter
http://twitter.com/wascupdates

websecurity@lists.webappsec.org
http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org
NOTICE: This email and any attachments may contain confidential and proprietary information of NetSuite Inc. and is for the sole use of the intended recipient for the stated purpose.  Any improper use or distribution is prohibited.  If you are not the intended recipient, please notify the sender; do not review, copy or distribute; and promptly delete or destroy all transmitted information.  Please note that all communications and information transmitted through this email system may be monitored by NetSuite or its agents and that all incoming email is automatically scanned by a third party spam and filtering service.

If bandwidth and the size of the site are not large concerns, a simple: wget <parameters> www.example.com | grep -r 'regex for string' * should serve as a simple sanity check. -----Original Message----- From: websecurity-bounces@lists.webappsec.org [mailto:websecurity-bounces@lists.webappsec.org] On Behalf Of MaXe Sent: Monday, September 19, 2011 10:22 AM To: illumina7i; Tasos Laskos Cc: websecurity@webappsec.org Subject: Re: [WEB SECURITY] program to crawl website looking for string patterns And if the website is public, some Google Kung Fu: site:domain-name.tld keyword That is of course, only what Google has seen and it will also only work if Google is not disabled from viewing the site. ~ MaXe ----- Original meddelelse ----- > This may also work: > https://github.com/urbanadventurer/WhatWeb/ > > On Fri, Sep 16, 2011 at 2:25 PM, Tasos Laskos <tasos.laskos@gmail.com> > wrote: > > <shameless selfpromotion> > > > > Or you could use arachni[1]: > > > > 1) Create a module like: > > ---------------- > > module Arachni > > module Modules > > class MyModule < Arachni::Module::Base > > > >    def initialize( page ) > >        @page = page > >    end > > > >    def run( ) > >        match_and_log( "the string you're looking for" ) > >    end > > > >    def self.info > >        { > >            :name           => 'My module', > >            :description    => %q{Greps pages for a string.}, > >            :author         => 'Your name', > >            :version        => '0.1', > >            :targets        => { 'Generic' => 'all' }, > >            :issue   => { > >                :name        => %q{Found my string}, > >                :description => %q{some description}, > >                :cwe         => '', > >                :severity    => Issue::Severity::LOW, > >                :cvssv2      => '0', > >                :remedy_guidance    => %q{Remode the damn thing.}, > >                :remedy_code => '', > >            } > >        } > >    end > > > > end > > end > > end > > ---------------- > > 2) Save it as "my_module.rb" and put it under "modules/recon/grep/" > > 3) run arachni like so: > >        arachni -m my_module <site url> > > > > > > And you're good to go. :) > > > > [1] http://arachni.segfault.gr/ > > > > </shameless selfpromotion> > > > > On 09/16/2011 07:51 PM, Ryan Dewhurst wrote: > > > > > > w3af [0] has lot's of grepping plugins which can easily be expanded. > > > Should do what you want. > > > > > > [0] http://w3af.sourceforge.net/ > > > > > > Ryan Dewhurst > > > > > > blog www.ethicalhack3r.co.uk > > > projects www.dvwa.co.uk | www.webwordcount.com > > > twitter www.twitter.com/ethicalhack3r > > > > > > > > > > > > On Fri, Sep 16, 2011 at 2:55 PM, Youngquist, Jason R. > > > <jryoungquist@ccis.edu>  wrote: > > > > > > > > We are looking for a tool that can be configured to crawl for > > > > string patterns (ie. SSNs, credit card numbers, etc).  Cornell's > > > > Spider 2008 beta has this capability, but every time we used it, > > > > it crashed on us. > > > > > > > > We also found a program called webshag, but it would only look for > > > > pre-defined stuff like email addresses or external links. > > > > > > > > Did some googling, but haven't really found anything.  Thoughts? > > > > > > > > > > > > > > > > Thanks. > > > > Jason Youngquist, CISSP > > > > Information Technology Security Engineer > > > > Technology Services > > > > Columbia College > > > > 1001 Rogers Street, Columbia, MO  65216 > > > > (573) 875-7334 > > > > jryoungquist@ccis.edu > > > > http://www.ccis.edu > > > > > > > > > > > > _______________________________________________ > > > > The Web Security Mailing List > > > > > > > > WebSecurity RSS Feed > > > > http://www.webappsec.org/rss/websecurity.rss > > > > > > > > Join WASC on LinkedIn > > > > http://www.linkedin.com/e/gis/83336/4B20E4374DBA > > > > > > > > WASC on Twitter > > > > http://twitter.com/wascupdates > > > > > > > > websecurity@lists.webappsec.org > > > > > > > > http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org > > > > > > > > > > _______________________________________________ > > > The Web Security Mailing List > > > > > > WebSecurity RSS Feed > > > http://www.webappsec.org/rss/websecurity.rss > > > > > > Join WASC on LinkedIn > > > http://www.linkedin.com/e/gis/83336/4B20E4374DBA > > > > > > WASC on Twitter > > > http://twitter.com/wascupdates > > > > > > websecurity@lists.webappsec.org > > > > > > http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org > > > > > > > > > _______________________________________________ > > The Web Security Mailing List > > > > WebSecurity RSS Feed > > http://www.webappsec.org/rss/websecurity.rss > > > > Join WASC on LinkedIn http://www.linkedin.com/e/gis/83336/4B20E4374DBA > > > > WASC on Twitter > > http://twitter.com/wascupdates > > > > websecurity@lists.webappsec.org > > http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org > > > > _______________________________________________ > The Web Security Mailing List > > WebSecurity RSS Feed > http://www.webappsec.org/rss/websecurity.rss > > Join WASC on LinkedIn http://www.linkedin.com/e/gis/83336/4B20E4374DBA > > WASC on Twitter > http://twitter.com/wascupdates > > websecurity@lists.webappsec.org > http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org _______________________________________________ The Web Security Mailing List WebSecurity RSS Feed http://www.webappsec.org/rss/websecurity.rss Join WASC on LinkedIn http://www.linkedin.com/e/gis/83336/4B20E4374DBA WASC on Twitter http://twitter.com/wascupdates websecurity@lists.webappsec.org http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org NOTICE: This email and any attachments may contain confidential and proprietary information of NetSuite Inc. and is for the sole use of the intended recipient for the stated purpose. Any improper use or distribution is prohibited. If you are not the intended recipient, please notify the sender; do not review, copy or distribute; and promptly delete or destroy all transmitted information. Please note that all communications and information transmitted through this email system may be monitored by NetSuite or its agents and that all incoming email is automatically scanned by a third party spam and filtering service.
HW
Hao Wang (haowa2)
Sat, Oct 8, 2011 8:43 AM

Hi All,

Do you know some material about how to test a WAF and write a testing report? I wish there are some examples, could you help?

Regards,
-Hao

Hi All, Do you know some material about how to test a WAF and write a testing report? I wish there are some examples, could you help? Regards, -Hao
SO
Simone Onofri
Sat, Oct 8, 2011 8:05 PM

Hi Hao,

IMHO  as for methodology You can use the OSSTMM [1] which have modules
such as Active Detection Verification, task for it and tool [2] (this
is used prior testing because a filtering device can corrupts the test
result, but I think it's a good start). Please correct me if I'm wrong
with this mapping.

So how to run the tasks depends on the vector and test type and your
reason to test, even if OSSTMM said "It may be necessary to coordinate
these tests with the appropriate persons within the scope.".

If you have access to Web Application Firewall/Webserver logs you can
check for requests passed or not, otherwise if You have no access to
logs You may "tune" the script in order to recognize if a request is
blocked or not (for example, dropped packets, specific or generic HTTP
errors, WSOD... depends on WAF used). Testing in a "lab" attacker ->
WAF -> test-webserver is the best.

Depending on WAF type and techniques used (regular expression and so
on...) it's also possible to use a custom script in order to:

  • crawl the application, looking for pages and parameters
  • tune the script
  • send sample attacks, looking for reactions
  • you may also elaborate attacks with evasion techniques (there are a
    lot of papers... from packets fragmentation to encoding)

Now You have "raw" output with requests passed and not. This output
can be analyzed (on parameters, attack type and/or evasion techniques)
in order to get some conclusion about how and when WAF works.

Note depending on WAF type you may found a filtering on a layer 7,
also with custom rules for each parameter or in a lower layer, with
different characteristics. Also note a WAF can be configured to
proactively blocks the attacker, compromising Your test (reactions
must be deactivated for test).

Cheers,

Simone

[1] http://www.isecom.org/osstmm/
[2] http://www.purehacking.com/news/afd-technical-details

On Sat, Oct 8, 2011 at 10:43 AM, Hao Wang (haowa2) haowa2@cisco.com wrote:

Hi All,

Do you know some material about how to test a WAF and write a testing report? I wish there are some examples, could you help?

Regards,
-Hao


The Web Security Mailing List

WebSecurity RSS Feed
http://www.webappsec.org/rss/websecurity.rss

Join WASC on LinkedIn http://www.linkedin.com/e/gis/83336/4B20E4374DBA

WASC on Twitter
http://twitter.com/wascupdates

websecurity@lists.webappsec.org
http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org

Hi Hao, IMHO as for methodology You can use the OSSTMM [1] which have modules such as Active Detection Verification, task for it and tool [2] (this is used prior testing because a filtering device can corrupts the test result, but I think it's a good start). Please correct me if I'm wrong with this mapping. So how to run the tasks depends on the vector and test type and your reason to test, even if OSSTMM said "It may be necessary to coordinate these tests with the appropriate persons within the scope.". If you have access to Web Application Firewall/Webserver logs you can check for requests passed or not, otherwise if You have no access to logs You may "tune" the script in order to recognize if a request is blocked or not (for example, dropped packets, specific or generic HTTP errors, WSOD... depends on WAF used). Testing in a "lab" attacker -> WAF -> test-webserver is the best. Depending on WAF type and techniques used (regular expression and so on...) it's also possible to use a custom script in order to: - crawl the application, looking for pages and parameters - tune the script - send sample attacks, looking for reactions - you may also elaborate attacks with evasion techniques (there are a lot of papers... from packets fragmentation to encoding) Now You have "raw" output with requests passed and not. This output can be analyzed (on parameters, attack type and/or evasion techniques) in order to get some conclusion about how and when WAF works. Note depending on WAF type you may found a filtering on a layer 7, also with custom rules for each parameter or in a lower layer, with different characteristics. Also note a WAF can be configured to proactively blocks the attacker, compromising Your test (reactions must be deactivated for test). Cheers, Simone [1] http://www.isecom.org/osstmm/ [2] http://www.purehacking.com/news/afd-technical-details On Sat, Oct 8, 2011 at 10:43 AM, Hao Wang (haowa2) <haowa2@cisco.com> wrote: > Hi All, > > Do you know some material about how to test a WAF and write a testing report? I wish there are some examples, could you help? > > Regards, > -Hao > _______________________________________________ > The Web Security Mailing List > > WebSecurity RSS Feed > http://www.webappsec.org/rss/websecurity.rss > > Join WASC on LinkedIn http://www.linkedin.com/e/gis/83336/4B20E4374DBA > > WASC on Twitter > http://twitter.com/wascupdates > > websecurity@lists.webappsec.org > http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org >
CH
Christian Heinrich
Sat, Oct 8, 2011 11:11 PM

Hao,

On Sat, Oct 8, 2011 at 7:43 PM, Hao Wang (haowa2) haowa2@cisco.com wrote:

Do you know some material about how to test a WAF and write a testing report? I wish there are some examples, could you help?

Have you considered the Web Application Firewall Evaluation Criteria?
i.e. http://projects.webappsec.org/w/page/13246985/Web%20Application%20Firewall%20Evaluation%20Criteria

--
Regards,
Christian Heinrich

http://cmlh.id.au/contact

Hao, On Sat, Oct 8, 2011 at 7:43 PM, Hao Wang (haowa2) <haowa2@cisco.com> wrote: > Do you know some material about how to test a WAF and write a testing report? I wish there are some examples, could you help? Have you considered the Web Application Firewall Evaluation Criteria? i.e. http://projects.webappsec.org/w/page/13246985/Web%20Application%20Firewall%20Evaluation%20Criteria -- Regards, Christian Heinrich http://cmlh.id.au/contact