We are looking for a tool that can be configured to crawl for string patterns (ie. SSNs, credit card numbers, etc). Cornell's Spider 2008 beta has this capability, but every time we used it, it crashed on us.
We also found a program called webshag, but it would only look for pre-defined stuff like email addresses or external links.
Did some googling, but haven't really found anything. Thoughts?
Thanks.
Jason Youngquist, CISSP
Information Technology Security Engineer
Technology Services
Columbia College
1001 Rogers Street, Columbia, MO 65216
(573) 875-7334
jryoungquist@ccis.edu
http://www.ccis.edu
w3af [0] has lot's of grepping plugins which can easily be expanded.
Should do what you want.
[0] http://w3af.sourceforge.net/
Ryan Dewhurst
blog www.ethicalhack3r.co.uk
projects www.dvwa.co.uk | www.webwordcount.com
twitter www.twitter.com/ethicalhack3r
On Fri, Sep 16, 2011 at 2:55 PM, Youngquist, Jason R.
jryoungquist@ccis.edu wrote:
We are looking for a tool that can be configured to crawl for string patterns (ie. SSNs, credit card numbers, etc). Cornell's Spider 2008 beta has this capability, but every time we used it, it crashed on us.
We also found a program called webshag, but it would only look for pre-defined stuff like email addresses or external links.
Did some googling, but haven't really found anything. Thoughts?
Thanks.
Jason Youngquist, CISSP
Information Technology Security Engineer
Technology Services
Columbia College
1001 Rogers Street, Columbia, MO 65216
(573) 875-7334
jryoungquist@ccis.edu
http://www.ccis.edu
The Web Security Mailing List
WebSecurity RSS Feed
http://www.webappsec.org/rss/websecurity.rss
Join WASC on LinkedIn http://www.linkedin.com/e/gis/83336/4B20E4374DBA
WASC on Twitter
http://twitter.com/wascupdates
websecurity@lists.webappsec.org
http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org
FuzzDB has a set of regex patterns
http://code.google.com/p/fuzzdb/source/browse/trunk/regex/pii.readme.txt
http://code.google.com/p/fuzzdb/source/browse/trunk/regex/pii.fuzz.txt
for the docs and set of test cases, respectively
Depending on your purpose and the value of depth and completeness to
your test, as well as the structure of the sites to be evaluated
themselves, you may not want to entirely rely on an automated crawler.
Many websites have entire sections that automated web crawlers can
never reach. The crawler may also not handle authenticated session
state well, without some effort. Solution: If depth is critical, make
sure to do a manual crawl, as well. You can load the PII regex
patterns into something like Burp, in order to monitor the traffic
passively for them. Then just have your automated spider tools also
run through burp. You may want to have Burp log the full request and
response to a file, for later, deeper analysis.
One thing you want to do post-crawl analysis for is that there may be
PII in the results, encoded in a way which your regex would miss.
Examples: SSN in URL encoding, base-64 encoded, unicode encoded, etc.
Sounds like an interesting project. Good luck!
Adam
On Fri, Sep 16, 2011 at 9:55 AM, Youngquist, Jason R.
jryoungquist@ccis.edu wrote:
We are looking for a tool that can be configured to crawl for string patterns (ie. SSNs, credit card numbers, etc). Cornell's Spider 2008 beta has this capability, but every time we used it, it crashed on us.
We also found a program called webshag, but it would only look for pre-defined stuff like email addresses or external links.
Did some googling, but haven't really found anything. Thoughts?
Thanks.
Jason Youngquist, CISSP
Information Technology Security Engineer
Technology Services
Columbia College
1001 Rogers Street, Columbia, MO 65216
(573) 875-7334
jryoungquist@ccis.edu
http://www.ccis.edu
The Web Security Mailing List
WebSecurity RSS Feed
http://www.webappsec.org/rss/websecurity.rss
Join WASC on LinkedIn http://www.linkedin.com/e/gis/83336/4B20E4374DBA
WASC on Twitter
http://twitter.com/wascupdates
websecurity@lists.webappsec.org
http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org
Or you could use arachni[1]:
module Arachni
module Modules
class MyModule < Arachni::Module::Base
def initialize( page )
@page = page
end
def run( )
match_and_log( "the string you're looking for" )
end
def self.info
{
:name => 'My module',
:description => %q{Greps pages for a string.},
:author => 'Your name',
:version => '0.1',
:targets => { 'Generic' => 'all' },
:issue => {
:name => %q{Found my string},
:description => %q{some description},
:cwe => '',
:severity => Issue::Severity::LOW,
:cvssv2 => '0',
:remedy_guidance => %q{Remode the damn thing.},
:remedy_code => '',
}
}
end
And you're good to go. :)
[1] http://arachni.segfault.gr/
</shameless selfpromotion>
On 09/16/2011 07:51 PM, Ryan Dewhurst wrote:
w3af [0] has lot's of grepping plugins which can easily be expanded.
Should do what you want.
[0] http://w3af.sourceforge.net/
Ryan Dewhurst
blog www.ethicalhack3r.co.uk
projects www.dvwa.co.uk | www.webwordcount.com
twitter www.twitter.com/ethicalhack3r
On Fri, Sep 16, 2011 at 2:55 PM, Youngquist, Jason R.
jryoungquist@ccis.edu wrote:
We are looking for a tool that can be configured to crawl for string patterns (ie. SSNs, credit card numbers, etc). Cornell's Spider 2008 beta has this capability, but every time we used it, it crashed on us.
We also found a program called webshag, but it would only look for pre-defined stuff like email addresses or external links.
Did some googling, but haven't really found anything. Thoughts?
Thanks.
Jason Youngquist, CISSP
Information Technology Security Engineer
Technology Services
Columbia College
1001 Rogers Street, Columbia, MO 65216
(573) 875-7334
jryoungquist@ccis.edu
http://www.ccis.edu
The Web Security Mailing List
WebSecurity RSS Feed
http://www.webappsec.org/rss/websecurity.rss
Join WASC on LinkedIn http://www.linkedin.com/e/gis/83336/4B20E4374DBA
WASC on Twitter
http://twitter.com/wascupdates
websecurity@lists.webappsec.org
http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org
The Web Security Mailing List
WebSecurity RSS Feed
http://www.webappsec.org/rss/websecurity.rss
Join WASC on LinkedIn http://www.linkedin.com/e/gis/83336/4B20E4374DBA
WASC on Twitter
http://twitter.com/wascupdates
websecurity@lists.webappsec.org
http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org
This may also work:
https://github.com/urbanadventurer/WhatWeb/
On Fri, Sep 16, 2011 at 2:25 PM, Tasos Laskos tasos.laskos@gmail.com wrote:
Or you could use arachni[1]:
module Arachni
module Modules
class MyModule < Arachni::Module::Base
def initialize( page )
@page = page
end
def run( )
match_and_log( "the string you're looking for" )
end
def self.info
{
:name => 'My module',
:description => %q{Greps pages for a string.},
:author => 'Your name',
:version => '0.1',
:targets => { 'Generic' => 'all' },
:issue => {
:name => %q{Found my string},
:description => %q{some description},
:cwe => '',
:severity => Issue::Severity::LOW,
:cvssv2 => '0',
:remedy_guidance => %q{Remode the damn thing.},
:remedy_code => '',
}
}
end
And you're good to go. :)
[1] http://arachni.segfault.gr/
</shameless selfpromotion>
On 09/16/2011 07:51 PM, Ryan Dewhurst wrote:
w3af [0] has lot's of grepping plugins which can easily be expanded.
Should do what you want.
[0] http://w3af.sourceforge.net/
Ryan Dewhurst
blog www.ethicalhack3r.co.uk
projects www.dvwa.co.uk | www.webwordcount.com
twitter www.twitter.com/ethicalhack3r
On Fri, Sep 16, 2011 at 2:55 PM, Youngquist, Jason R.
jryoungquist@ccis.edu wrote:
We are looking for a tool that can be configured to crawl for string
patterns (ie. SSNs, credit card numbers, etc). Cornell's Spider 2008 beta
has this capability, but every time we used it, it crashed on us.
We also found a program called webshag, but it would only look for
pre-defined stuff like email addresses or external links.
Did some googling, but haven't really found anything. Thoughts?
Thanks.
Jason Youngquist, CISSP
Information Technology Security Engineer
Technology Services
Columbia College
1001 Rogers Street, Columbia, MO 65216
(573) 875-7334
jryoungquist@ccis.edu
http://www.ccis.edu
The Web Security Mailing List
WebSecurity RSS Feed
http://www.webappsec.org/rss/websecurity.rss
Join WASC on LinkedIn http://www.linkedin.com/e/gis/83336/4B20E4374DBA
WASC on Twitter
http://twitter.com/wascupdates
websecurity@lists.webappsec.org
http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org
The Web Security Mailing List
WebSecurity RSS Feed
http://www.webappsec.org/rss/websecurity.rss
Join WASC on LinkedIn http://www.linkedin.com/e/gis/83336/4B20E4374DBA
WASC on Twitter
http://twitter.com/wascupdates
websecurity@lists.webappsec.org
http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org
The Web Security Mailing List
WebSecurity RSS Feed
http://www.webappsec.org/rss/websecurity.rss
Join WASC on LinkedIn http://www.linkedin.com/e/gis/83336/4B20E4374DBA
WASC on Twitter
http://twitter.com/wascupdates
websecurity@lists.webappsec.org
http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org
And if the website is public, some Google Kung Fu:
site:domain-name.tld keyword
That is of course, only what Google has seen and it will also only work if Google is not disabled from viewing the site.
~ MaXe
----- Original meddelelse -----
This may also work:
https://github.com/urbanadventurer/WhatWeb/
On Fri, Sep 16, 2011 at 2:25 PM, Tasos Laskos tasos.laskos@gmail.com
wrote:
Or you could use arachni[1]:
module Arachni
module Modules
class MyModule < Arachni::Module::Base
def initialize( page )
@page = page
end
def run( )
match_and_log( "the string you're looking for" )
end
def self.info
{
:name => 'My module',
:description => %q{Greps pages for a string.},
:author => 'Your name',
:version => '0.1',
:targets => { 'Generic' => 'all' },
:issue => {
:name => %q{Found my string},
:description => %q{some description},
:cwe => '',
:severity => Issue::Severity::LOW,
:cvssv2 => '0',
:remedy_guidance => %q{Remode the damn thing.},
:remedy_code => '',
}
}
end
And you're good to go. :)
[1] http://arachni.segfault.gr/
</shameless selfpromotion>
On 09/16/2011 07:51 PM, Ryan Dewhurst wrote:
w3af [0] has lot's of grepping plugins which can easily be expanded.
Should do what you want.
[0] http://w3af.sourceforge.net/
Ryan Dewhurst
blog www.ethicalhack3r.co.uk
projects www.dvwa.co.uk | www.webwordcount.com
twitter www.twitter.com/ethicalhack3r
On Fri, Sep 16, 2011 at 2:55 PM, Youngquist, Jason R.
jryoungquist@ccis.edu wrote:
We are looking for a tool that can be configured to crawl for
string patterns (ie. SSNs, credit card numbers, etc). Cornell's
Spider 2008 beta has this capability, but every time we used it,
it crashed on us.
We also found a program called webshag, but it would only look for
pre-defined stuff like email addresses or external links.
Did some googling, but haven't really found anything. Thoughts?
Thanks.
Jason Youngquist, CISSP
Information Technology Security Engineer
Technology Services
Columbia College
1001 Rogers Street, Columbia, MO 65216
(573) 875-7334
jryoungquist@ccis.edu
http://www.ccis.edu
The Web Security Mailing List
WebSecurity RSS Feed
http://www.webappsec.org/rss/websecurity.rss
Join WASC on LinkedIn
http://www.linkedin.com/e/gis/83336/4B20E4374DBA
WASC on Twitter
http://twitter.com/wascupdates
websecurity@lists.webappsec.org
http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org
The Web Security Mailing List
WebSecurity RSS Feed
http://www.webappsec.org/rss/websecurity.rss
Join WASC on LinkedIn
http://www.linkedin.com/e/gis/83336/4B20E4374DBA
WASC on Twitter
http://twitter.com/wascupdates
websecurity@lists.webappsec.org
http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org
The Web Security Mailing List
WebSecurity RSS Feed
http://www.webappsec.org/rss/websecurity.rss
Join WASC on LinkedIn http://www.linkedin.com/e/gis/83336/4B20E4374DBA
WASC on Twitter
http://twitter.com/wascupdates
websecurity@lists.webappsec.org
http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org
The Web Security Mailing List
WebSecurity RSS Feed
http://www.webappsec.org/rss/websecurity.rss
Join WASC on LinkedIn http://www.linkedin.com/e/gis/83336/4B20E4374DBA
WASC on Twitter
http://twitter.com/wascupdates
websecurity@lists.webappsec.org
http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org
If bandwidth and the size of the site are not large concerns, a simple: wget <parameters> www.example.com | grep -r 'regex for string' * should serve as a simple sanity check.
-----Original Message-----
From: websecurity-bounces@lists.webappsec.org [mailto:websecurity-bounces@lists.webappsec.org] On Behalf Of MaXe
Sent: Monday, September 19, 2011 10:22 AM
To: illumina7i; Tasos Laskos
Cc: websecurity@webappsec.org
Subject: Re: [WEB SECURITY] program to crawl website looking for string patterns
And if the website is public, some Google Kung Fu:
site:domain-name.tld keyword
That is of course, only what Google has seen and it will also only work if Google is not disabled from viewing the site.
~ MaXe
----- Original meddelelse -----
This may also work:
https://github.com/urbanadventurer/WhatWeb/
On Fri, Sep 16, 2011 at 2:25 PM, Tasos Laskos tasos.laskos@gmail.com
wrote:
Or you could use arachni[1]:
module Arachni
module Modules
class MyModule < Arachni::Module::Base
def initialize( page )
@page = page
end
def run( )
match_and_log( "the string you're looking for" )
end
def self.info
{
:name => 'My module',
:description => %q{Greps pages for a string.},
:author => 'Your name',
:version => '0.1',
:targets => { 'Generic' => 'all' },
:issue => {
:name => %q{Found my string},
:description => %q{some description},
:cwe => '',
:severity => Issue::Severity::LOW,
:cvssv2 => '0',
:remedy_guidance => %q{Remode the damn thing.},
:remedy_code => '',
}
}
end
And you're good to go. :)
[1] http://arachni.segfault.gr/
</shameless selfpromotion>
On 09/16/2011 07:51 PM, Ryan Dewhurst wrote:
w3af [0] has lot's of grepping plugins which can easily be expanded.
Should do what you want.
[0] http://w3af.sourceforge.net/
Ryan Dewhurst
blog www.ethicalhack3r.co.uk
projects www.dvwa.co.uk | www.webwordcount.com
twitter www.twitter.com/ethicalhack3r
On Fri, Sep 16, 2011 at 2:55 PM, Youngquist, Jason R.
jryoungquist@ccis.edu wrote:
We are looking for a tool that can be configured to crawl for
string patterns (ie. SSNs, credit card numbers, etc). Cornell's
Spider 2008 beta has this capability, but every time we used it,
it crashed on us.
We also found a program called webshag, but it would only look for
pre-defined stuff like email addresses or external links.
Did some googling, but haven't really found anything. Thoughts?
Thanks.
Jason Youngquist, CISSP
Information Technology Security Engineer
Technology Services
Columbia College
1001 Rogers Street, Columbia, MO 65216
(573) 875-7334
jryoungquist@ccis.edu
http://www.ccis.edu
The Web Security Mailing List
WebSecurity RSS Feed
http://www.webappsec.org/rss/websecurity.rss
Join WASC on LinkedIn
http://www.linkedin.com/e/gis/83336/4B20E4374DBA
WASC on Twitter
http://twitter.com/wascupdates
websecurity@lists.webappsec.org
http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org
The Web Security Mailing List
WebSecurity RSS Feed
http://www.webappsec.org/rss/websecurity.rss
Join WASC on LinkedIn
http://www.linkedin.com/e/gis/83336/4B20E4374DBA
WASC on Twitter
http://twitter.com/wascupdates
websecurity@lists.webappsec.org
http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org
The Web Security Mailing List
WebSecurity RSS Feed
http://www.webappsec.org/rss/websecurity.rss
Join WASC on LinkedIn http://www.linkedin.com/e/gis/83336/4B20E4374DBA
WASC on Twitter
http://twitter.com/wascupdates
websecurity@lists.webappsec.org
http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org
The Web Security Mailing List
WebSecurity RSS Feed
http://www.webappsec.org/rss/websecurity.rss
Join WASC on LinkedIn http://www.linkedin.com/e/gis/83336/4B20E4374DBA
WASC on Twitter
http://twitter.com/wascupdates
websecurity@lists.webappsec.org
http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org
The Web Security Mailing List
WebSecurity RSS Feed
http://www.webappsec.org/rss/websecurity.rss
Join WASC on LinkedIn http://www.linkedin.com/e/gis/83336/4B20E4374DBA
WASC on Twitter
http://twitter.com/wascupdates
websecurity@lists.webappsec.org
http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org
NOTICE: This email and any attachments may contain confidential and proprietary information of NetSuite Inc. and is for the sole use of the intended recipient for the stated purpose. Any improper use or distribution is prohibited. If you are not the intended recipient, please notify the sender; do not review, copy or distribute; and promptly delete or destroy all transmitted information. Please note that all communications and information transmitted through this email system may be monitored by NetSuite or its agents and that all incoming email is automatically scanned by a third party spam and filtering service.
Hi All,
Do you know some material about how to test a WAF and write a testing report? I wish there are some examples, could you help?
Regards,
-Hao
Hi Hao,
IMHO as for methodology You can use the OSSTMM [1] which have modules
such as Active Detection Verification, task for it and tool [2] (this
is used prior testing because a filtering device can corrupts the test
result, but I think it's a good start). Please correct me if I'm wrong
with this mapping.
So how to run the tasks depends on the vector and test type and your
reason to test, even if OSSTMM said "It may be necessary to coordinate
these tests with the appropriate persons within the scope.".
If you have access to Web Application Firewall/Webserver logs you can
check for requests passed or not, otherwise if You have no access to
logs You may "tune" the script in order to recognize if a request is
blocked or not (for example, dropped packets, specific or generic HTTP
errors, WSOD... depends on WAF used). Testing in a "lab" attacker ->
WAF -> test-webserver is the best.
Depending on WAF type and techniques used (regular expression and so
on...) it's also possible to use a custom script in order to:
Now You have "raw" output with requests passed and not. This output
can be analyzed (on parameters, attack type and/or evasion techniques)
in order to get some conclusion about how and when WAF works.
Note depending on WAF type you may found a filtering on a layer 7,
also with custom rules for each parameter or in a lower layer, with
different characteristics. Also note a WAF can be configured to
proactively blocks the attacker, compromising Your test (reactions
must be deactivated for test).
Cheers,
Simone
[1] http://www.isecom.org/osstmm/
[2] http://www.purehacking.com/news/afd-technical-details
On Sat, Oct 8, 2011 at 10:43 AM, Hao Wang (haowa2) haowa2@cisco.com wrote:
Hi All,
Do you know some material about how to test a WAF and write a testing report? I wish there are some examples, could you help?
Regards,
-Hao
The Web Security Mailing List
WebSecurity RSS Feed
http://www.webappsec.org/rss/websecurity.rss
Join WASC on LinkedIn http://www.linkedin.com/e/gis/83336/4B20E4374DBA
WASC on Twitter
http://twitter.com/wascupdates
websecurity@lists.webappsec.org
http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org
Hao,
On Sat, Oct 8, 2011 at 7:43 PM, Hao Wang (haowa2) haowa2@cisco.com wrote:
Do you know some material about how to test a WAF and write a testing report? I wish there are some examples, could you help?
Have you considered the Web Application Firewall Evaluation Criteria?
i.e. http://projects.webappsec.org/w/page/13246985/Web%20Application%20Firewall%20Evaluation%20Criteria
--
Regards,
Christian Heinrich