User: Password:
|
|
Subscribe / Log in / New account

dictionaries/README-FIRST

This directory contains four alternative, hand-picked Skipfish dictionaries.

Before you pick one, you should understand several basic concepts related to
dictionary management in this scanner, as this topic is of critical importance
to the quality of your scans.

-----------------------------
Dictionary management basics:
-----------------------------

1) Each dictionary may consist of a number of extensions, and a number of
   "regular" keywords. Extensions are considered just a special subset of
   the keyword list.

2) Use -W to specify the dictionary file to use. The dictionary may be
   custom, but must conform to the following format:

   type hits total_age last_age keyword

   ...where 'type' is either 'e' or 'w' (extension or wordlist); 'hits'
   is the total number of times this keyword resulted in a non-404 hit
   in all previous scans; 'total_age' is the number of scan cycles this
   word is in the dictionary; 'last_age' is the number of scan cycles
   since the last 'hit'; and 'keyword' is the actual keyword.

   Do not duplicate extensions as keywords - if you already have 'html' as
   an 'e' entry, there is no need to also create a 'w' one.

   There must be no empty or malformed lines, comments in the wordlist
   file. Extension keywords must have no leading dot (e.g., 'exe', not '.exe'),
   and all keywords should be NOT url-encoded (e.g., 'Program Files', not
   'Program%20Files'). No keyword should exceed 64 characters.

   If you omit -W in the command line, 'skipfish.wl' is assumed. This
   file does not exist by default; this is by design.

3) The scanner will automatically learn new keywords and extensions based on
   any links discovered during the scan; and will also analyze pages and
   extract words to use as keyword candidates.

   A capped number of candidates is kept in memory (you can set the jar size
   with the -G option) in FIFO mode, and are used for brute-force attacks.
   When a particular candidate results in a non-404 hit, it is promoted to
   the "real" dictionary; other candidates are discarded at the end of the
   scan.

   You can inhibit this auto-learning behavior by specifying -L in the
   command line.

4) Keyword hit counts and age information will be updated at the end of the
   scan. This can be prevented with -V.

5) Old dictionary entries with no hits for a specified number of scans can
   be purged by specifying the -R <cnt> option.

----------------------------------------------
Dictionaries are used for the following tasks:
----------------------------------------------

1) When a new directory, or a file-like query or POST parameter is discovered,
   the scanner attempts passing all possible <keyword> values to discover new
   files, directories, etc.

2) The scanner also tests all possible <keyword>.<extension> pairs. Note that
   this results in several orders of magnitude more requests, but is the only
   way to discover files such as 'backup.tar.gz', 'database.csv', etc. 

   In some cases, you might want to inhibit this step. This can be achieved
   with the -Y switch.

3) For any non-404 file or directory discovered by any other means, the scanner
   also attempts all <node_filename>.<extension> combinations, to discover,
   for example, entries such as 'index.php.old'. This behavior is independent
   of the -Y option, since it is much less request-intensive.

----------------------
Supplied dictionaries:
----------------------

1) Empty dictionary (-).

   Simply create an empty file, then load it via -W. If you use this option
   in conjunction with -L, this essentially inhibits all brute-force testing,
   and results in an orderly, link-based crawl.

   If -L is not used, the crawler will still attempt brute-force, but only
   based on the keywords and extensions discovered when crawling the site.
   This means it will likely learn keywords such as 'index' or extensions
   such as 'html' - but may never attempt probing for 'log', 'old', 'bak', etc.

   Both these variants are very useful for lightweight scans, but are not
   particularly exhaustive.

2) Extension-only dictionary (extensions-only.wl).

   This dictionary contains about 90 common file extensions, and no other
   keywords. It must be used in conjunction with -Y (otherwise, it will not
   behave as expected).

   This is often a better alternative to a null dictionary: the scanner will
   still limit brute-force primarily to file names learned on the site, but
   will know about extensions such as 'log' or 'old', and will test for them
   accordingly.

3) Basic extensions dictionary (minimal.wl).

   This dictionary contains about 25 extensions, focusing on common entries
   most likely to spell trouble (.bak, .old, .conf, .zip, etc); and about 1,700
   hand-picked keywords.

   This is useful for quick assessments where no obscure technologies are used.
   The principal scan cost is about 42,000 requests per each fuzzed directory.

   Using it without -L is recommended, as the list of extensions does not
   include standard framework-specific cases (.asp, .jsp, .php, etc), and
   these are best learned on the fly.

   ** This dictionary is strongly recommended for your first experiments with
   ** skipfish, as it's reasonably lightweight.

   You can also use this dictionary with -Y option enabled, approximating the
   behavior of most other security scanners; in this case, it will send only
   about 1,700 requests per directory, and will look for 25 secondary extensions
   only on otherwise discovered resources.

3) Standard extensions dictionary (default.wl).

   This dictionary contains about 60 common extensions, plus the same set of
   1,700 keywords. The extensions cover most of the common, interesting web
   resources.

   This is a good starting point for assessments where scan times are not
   a critical factor; the cost is about 100,000 requests per each fuzzed
   directory.

   In -Y mode, it behaves nearly identical to minimal.wl, but will test a
   greater set of extensions on otherwise discovered resources at a relatively
   minor expense.

4) Complete extensions dictionary (complete.wl).

   Contains about 90 common extensions and 1,700 keywords. These extensions
   cover a broader range of media types, including some less common programming
   languages, image and video formats, etc.

   Useful for comprehensive assessments, over 150,000 requests per each fuzzed
   directory.

   In -Y mode, this dictionary offers the best coverage of all three wordlists
   at a relatively low cost.

Of course, you can customize these dictionaries as seen fit. It might be, for
example, a good idea to downgrade file extensions not likely to occur given
the technologies used by your target host to regular 'w' records.

Whichever option you choose, be sure to make a *copy* of this dictionary, and
load that copy, not the original, via -W. The specified file will be overwritten
with site-specific information unless -V used - and you probably want to keep
the original around.

----------------------------------
Bah, these dictionaries are small!
----------------------------------

Keep in mind that web crawling is not password guessing; it is exceedingly
unlikely for web servers to have directories or files named 'henceforth',
'abating', or 'witlessly'. Because of this, using 200,000+ entry English
wordlists, or similar data sets, is largely pointless.

More importantly, doing so often leads to reduced coverage or unacceptable
scan times; with a 200k wordlist and 80 extensions, trying all combinations
for a single directory would take 30-40 hours against a slow server; and even
with a fast one, at least 5 hours is to be expected.

DirBuster uses a unique approach that seems promising at first sight - to
base their wordlists on how often a particular keyword appeared in URLs seen on
the Internet. This is interesting, but comes with two gotchas:

  - Keywords related to popular websites and brands are heavily
    overrepresented; DirBuster wordlists have 'bbc_news_24', 'beebie_bunny',
    and 'koalabrothers' near the top of their list, but it is pretty unlikely
    these keywords would be of any use in real-world assessments of a typical
    site, unless it happens to be BBC or Disney.

  - Some of the most interesting security-related keywords are not commonly
    indexed, and may appear, say, on no more than few dozen or few thousand
    crawled websites in Google index. But, that does not make 'AggreSpy' or
    '.ssh/authorized_keys' any less interesting - in fact, you might care
    about them a whole lot more.

Bottom line is, tread carefully; poor wordlists are one of the reasons why some
web security scanners perform worse than expected. You will almost always be
better off narrowing down or selectively extending the supplied set (and
possibly contributing back your changes upstream!), than importing a giant
wordlist scored elsewhere.

(Log in to post comments)


Copyright © 2010, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds