Ping IP / IP Checker Traceroute DNS Checker WHOIS Lookup What is My IP PageRank Checker Reverse DNS

Robot.txt Generator

A robots.txt file is a short text file that resides in your home directory. Before search engines spider your site, they look into this file to see which files/file types and/or directories they are not allowed to see. This tool will make your robots.txt file. As a little 'extra' it also includes a block to many unwanted spiders that only crawl your site in order to collect the email addresses stored on your pages.(SPAM-bots)

1.

Enter the directories of your site that you don't want to be seen by search engines: 1 directory per line.

Leave blank if you don't want to block anything.
For www.yoursite.com/private/ enter private
For www.yoursite.com/mp3/A/ enter mp3/A
For www.yoursite.com/mp3/ and its subfolders enter mp3

2.

Click this box if you want to disallow 'nasty' robots, like email-extractors and robots that copy your whole site to someone's hard drive. (e.g. WGET)

Block nasty bots

3.

Related Questions

Bots!! Follow & Revisit?
Where can I find a spider emulator?

Related Tools

Spider Emulator