# # This is the robots.txt file. It should be in the root (/) # of your web site. # # This file is used to tell search engine crawlers what # directories they should ignore when indexing. # # Lines beginning with # are comments, and are not processed # by crawlers. # # The format is a series of records. Each record begins with # a User-agent: statement, and is followed by at least one # Disallow: statement. The preset records below demonstrate # this format. # # Records are separated by blank lines # # For further information, visit # http://info.webcrawler.com/mak/projects/robots/norobots.html # # Leave these records intact: User-agent: stress-agent Disallow: / # Edit this record as appropriate for your site. Add any # other directories you want crawlers to ignore. User-agent: * Disallow: / Disallow: /images/ Disallow: /scripts/ Disallow: /styles/ Disallow: /animations/ Disallow: /text/ Disallow: /spacer.htm Disallow: /bottom.htm Disallow: /mainPage.asp Disallow: /audio/ Disallow: /test/ Disallow: /mail/ Disallow: /_