[WEB SECURITY] program to crawl website looking for string patterns

Menerick, John jmenerick at netsuite.com
Mon Sep 19 14:14:58 EDT 2011


If bandwidth and the size of the site are not large concerns, a simple:  wget <parameters> www.example.com | grep -r 'regex for string' *  should serve as a simple sanity check.

-----Original Message-----
From: websecurity-bounces at lists.webappsec.org [mailto:websecurity-bounces at lists.webappsec.org] On Behalf Of MaXe
Sent: Monday, September 19, 2011 10:22 AM
To: illumina7i; Tasos Laskos
Cc: websecurity at webappsec.org
Subject: Re: [WEB SECURITY] program to crawl website looking for string patterns

And if the website is public, some Google Kung Fu:
site:domain-name.tld keyword

That is of course, only what Google has seen and it will also only work if Google is not disabled from viewing the site.



~ MaXe
----- Original meddelelse -----
> This may also work:
> https://github.com/urbanadventurer/WhatWeb/
> 
> On Fri, Sep 16, 2011 at 2:25 PM, Tasos Laskos <tasos.laskos at gmail.com>
> wrote:
> > <shameless selfpromotion>
> > 
> > Or you could use arachni[1]:
> > 
> > 1) Create a module like:
> > ----------------
> > module Arachni
> > module Modules
> > class MyModule < Arachni::Module::Base
> > 
> >    def initialize( page )
> >        @page = page
> >    end
> > 
> >    def run( )
> >        match_and_log( "the string you're looking for" )
> >    end
> > 
> >    def self.info
> >        {
> >            :name           => 'My module',
> >            :description    => %q{Greps pages for a string.},
> >            :author         => 'Your name',
> >            :version        => '0.1',
> >            :targets        => { 'Generic' => 'all' },
> >            :issue   => {
> >                :name        => %q{Found my string},
> >                :description => %q{some description},
> >                :cwe         => '',
> >                :severity    => Issue::Severity::LOW,
> >                :cvssv2      => '0',
> >                :remedy_guidance    => %q{Remode the damn thing.},
> >                :remedy_code => '',
> >            }
> >        }
> >    end
> > 
> > end
> > end
> > end
> > ----------------
> > 2) Save it as "my_module.rb" and put it under "modules/recon/grep/"
> > 3) run arachni like so:
> >        arachni -m my_module <site url>
> > 
> > 
> > And you're good to go. :)
> > 
> > [1] http://arachni.segfault.gr/
> > 
> > </shameless selfpromotion>
> > 
> > On 09/16/2011 07:51 PM, Ryan Dewhurst wrote:
> > > 
> > > w3af [0] has lot's of grepping plugins which can easily be expanded.
> > > Should do what you want.
> > > 
> > > [0] http://w3af.sourceforge.net/
> > > 
> > > Ryan Dewhurst
> > > 
> > > blog www.ethicalhack3r.co.uk
> > > projects www.dvwa.co.uk | www.webwordcount.com
> > > twitter www.twitter.com/ethicalhack3r
> > > 
> > > 
> > > 
> > > On Fri, Sep 16, 2011 at 2:55 PM, Youngquist, Jason R.
> > > <jryoungquist at ccis.edu>  wrote:
> > > > 
> > > > We are looking for a tool that can be configured to crawl for
> > > > string patterns (ie. SSNs, credit card numbers, etc).  Cornell's
> > > > Spider 2008 beta has this capability, but every time we used it,
> > > > it crashed on us.
> > > > 
> > > > We also found a program called webshag, but it would only look for
> > > > pre-defined stuff like email addresses or external links.
> > > > 
> > > > Did some googling, but haven't really found anything.  Thoughts?
> > > > 
> > > > 
> > > > 
> > > > Thanks.
> > > > Jason Youngquist, CISSP
> > > > Information Technology Security Engineer
> > > > Technology Services
> > > > Columbia College
> > > > 1001 Rogers Street, Columbia, MO  65216
> > > > (573) 875-7334
> > > > jryoungquist at ccis.edu
> > > > http://www.ccis.edu
> > > > 
> > > > 
> > > > _______________________________________________
> > > > The Web Security Mailing List
> > > > 
> > > > WebSecurity RSS Feed
> > > > http://www.webappsec.org/rss/websecurity.rss
> > > > 
> > > > Join WASC on LinkedIn
> > > > http://www.linkedin.com/e/gis/83336/4B20E4374DBA
> > > > 
> > > > WASC on Twitter
> > > > http://twitter.com/wascupdates
> > > > 
> > > > websecurity at lists.webappsec.org
> > > > 
> > > > http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org
> > > > 
> > > 
> > > _______________________________________________
> > > The Web Security Mailing List
> > > 
> > > WebSecurity RSS Feed
> > > http://www.webappsec.org/rss/websecurity.rss
> > > 
> > > Join WASC on LinkedIn
> > > http://www.linkedin.com/e/gis/83336/4B20E4374DBA
> > > 
> > > WASC on Twitter
> > > http://twitter.com/wascupdates
> > > 
> > > websecurity at lists.webappsec.org
> > > 
> > > http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org
> > > 
> > 
> > 
> > _______________________________________________
> > The Web Security Mailing List
> > 
> > WebSecurity RSS Feed
> > http://www.webappsec.org/rss/websecurity.rss
> > 
> > Join WASC on LinkedIn http://www.linkedin.com/e/gis/83336/4B20E4374DBA
> > 
> > WASC on Twitter
> > http://twitter.com/wascupdates
> > 
> > websecurity at lists.webappsec.org
> > http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org
> > 
> 
> _______________________________________________
> The Web Security Mailing List
> 
> WebSecurity RSS Feed
> http://www.webappsec.org/rss/websecurity.rss
> 
> Join WASC on LinkedIn http://www.linkedin.com/e/gis/83336/4B20E4374DBA
> 
> WASC on Twitter
> http://twitter.com/wascupdates
> 
> websecurity at lists.webappsec.org
> http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org


_______________________________________________
The Web Security Mailing List

WebSecurity RSS Feed
http://www.webappsec.org/rss/websecurity.rss

Join WASC on LinkedIn http://www.linkedin.com/e/gis/83336/4B20E4374DBA

WASC on Twitter
http://twitter.com/wascupdates

websecurity at lists.webappsec.org
http://lists.webappsec.org/mailman/listinfo/websecurity_lists.webappsec.org
NOTICE: This email and any attachments may contain confidential and proprietary information of NetSuite Inc. and is for the sole use of the intended recipient for the stated purpose.  Any improper use or distribution is prohibited.  If you are not the intended recipient, please notify the sender; do not review, copy or distribute; and promptly delete or destroy all transmitted information.  Please note that all communications and information transmitted through this email system may be monitored by NetSuite or its agents and that all incoming email is automatically scanned by a third party spam and filtering service.



More information about the websecurity mailing list