Printable version SEO Toolkit 3 Overview Features Versions Licence information System requirements Before Starting Registering Entering account details Engine subscription wizard Settings Program interface and customization Exporting data Upgrading Engine subscription renewal Support Keyword Research The Keyword Manager Searching for keywords Displaying the seasonal trends graph Keyword analysis Deleting columns Keyword Management Exporting Word Density Industry Keywords Finding keywords Keyword popularity Exporting Seasonal trends graph Competitor Research Viewing domain data Displaying a seasonal trends graph Site optimization Meta Tags Meta browser Meta Editor Using Meta Editor Source code Importing URL and Contact details Saving meta tags Setting default tags Global meta tag editor Page content Keyword Density Analyzer Analyzing a page Filters Results Keyword Popularity Exporting keywords Ranking Advisor Selecting a page Analyzing a page Viewing the page The robots.txt file Server header check Search engine submission SubmitWolf Submission Rules Icon legend Category descriptions Importing previous SubmitWolf projects Submission Wizard Entering URL Details Category & Contact Details Engine Selection Entering a URL Entering URL details Alternate URL titles Importing URL and contact details Reciprocal link page builder Confirm Email Keywords Editor Search engine selection Submission confirmation Connection status Submission status Manual submissions Batch submissions Reporting and maintenance Reports Submission history Reporting non functioning engines Suggesting new engines Enterprise features Importing data Setting up an online submission form Exporting submission data Reports Site Maps Crawling a URL Specifying URL information Selecting URLs Building the Site Map Reporting and revision Search engine rankings Entering a URL Entering keywords Selecting engines Starting the ranking check Ranking results Exporting ranking results Ranking reports Ranking history Reporting non functioning engines Customizing the ranking tab Internet Presence Entering a URL Selecting engines Starting the popularity check Popularity results Popularity report Popularity history Customizing the presence tab ROI Calculators CPM to CPC ROI calculator CPC calculator Reciprocal link checker Checking for reciprocal links Link check results Exporting link check results Domain Manager Entering a domain Importing multiple domains Custom import Obtaining domain information Whois Registering a domain Editing domain information Sorting domains Exporting domain information Marking domains
SEO Toolkit 3

The robots.txt file



A spider is an automated program that is used by search engines to find and index the contents of a website.

Spiders will look in a site's root domain for a special file named "robots.txt". The file provides information to the robot (spider) regarding the files and directories that it may or may not index or crawl.

The format for the robots.txt file consists of multiple records. Each record contains two fields, a user-agent specification and one or more Disallow statements. The format is:

User-agent:<robot name>
Disallow:<statements>


User-agent

The User-agent line specifies the robot name. For example:

User-agent: googlebot

You may also use the wildcard character "*" to specify all robots. For example

User-agent: *

You can find user agent names in your site's logs by checking for requests to the robots.txt file.



Disallow:

The second part of a record consists of Disallow: statements. These statements specify files and/or directories. For example, the following line instructs spiders that they cannot index the email.htm file:

Disallow: email.htm


You may also specify directories:
Disallow: /cgi-bin/

Leaving the Disallow statement blank implies that the robot can index any files without restriction. At least one disallow line must be present for each User-agent directive to be valid. A blank robots.txt file is treated as if it were non existent.



Examples

The following example allows all robots to index all files:

User-agent: *
Disallow:



The following example denies access to all robots:

User-agent: *
Disallow: /



The following example denies all robots access to the cgi-bin and images directories:

User-agent: *
Disallow: /cgi-bin/
Disallow: /images/



The following example denies googlebot access to all files:

User-agent: googlebot
Disallow: /



For more complex examples, view the robots.txt files from other websites.

The robots.txt file should be created in a plain text editor. htm, .html, .rtf or any file format other than .txt is not acceptable.