Printable version SEO Toolkit 3 Overview Features Versions Licence information System requirements Before Starting Registering Entering account details Engine subscription wizard Settings Program interface and customization Exporting data Upgrading Engine subscription renewal Support Keyword Research The Keyword Manager Searching for keywords Displaying the seasonal trends graph Keyword analysis Deleting columns Keyword Management Exporting Word Density Industry Keywords Finding keywords Keyword popularity Exporting Seasonal trends graph Competitor Research Viewing domain data Displaying a seasonal trends graph Site optimization Meta Tags Meta browser Meta Editor Using Meta Editor Source code Importing URL and Contact details Saving meta tags Setting default tags Global meta tag editor Page content Keyword Density Analyzer Analyzing a page Filters Results Keyword Popularity Exporting keywords Ranking Advisor Selecting a page Analyzing a page Viewing the page The robots.txt file Server header check Search engine submission SubmitWolf Submission Rules Icon legend Category descriptions Importing previous SubmitWolf projects Submission Wizard Entering URL Details Category & Contact Details Engine Selection Entering a URL Entering URL details Alternate URL titles Importing URL and contact details Reciprocal link page builder Confirm Email Keywords Editor Search engine selection Submission confirmation Connection status Submission status Manual submissions Batch submissions Reporting and maintenance Reports Submission history Reporting non functioning engines Suggesting new engines Enterprise features Importing data Setting up an online submission form Exporting submission data Reports Site Maps Crawling a URL Specifying URL information Selecting URLs Building the Site Map Reporting and revision Search engine rankings Entering a URL Entering keywords Selecting engines Starting the ranking check Ranking results Exporting ranking results Ranking reports Ranking history Reporting non functioning engines Customizing the ranking tab Internet Presence Entering a URL Selecting engines Starting the popularity check Popularity results Popularity report Popularity history Customizing the presence tab ROI Calculators CPM to CPC ROI calculator CPC calculator Reciprocal link checker Checking for reciprocal links Link check results Exporting link check results Domain Manager Entering a domain Importing multiple domains Custom import Obtaining domain information Whois Registering a domain Editing domain information Sorting domains Exporting domain information Marking domains
SEO Toolkit 3

Crawling a URL

The next step is to crawl the URL.

  1. Click on the Crawl button to crawl all URLs listed.

    Crawling a URL

    The Site Maps tool will prompt for any exclusion options to be entered. Excluded URLs are not included in the site map.

    Crawling exclusions

    To exclude specific URLs or to specify a wildcard to exclude, enter the URL in the "Exclude URLs" section. eg. If you don't want to include the page 404.htm then you would enter:

    If you wanted to exclude all shtml files, then you would enter:*.shtml

    Ignore dynamic links
    This option will ignore dynamic links i.e. links containing the ? character.

    Strip URL parameters
    Strips parameters from URLs.

    Do not Cache files locally
    Files will not be cached locally and will be fetched from the Internet as required. If left unchecked, the Toolkit will download the files once, and all tools will use the cached files.

  2. Click on the start button to begin the crawl. When finished, the URLs from the site will be listed, with their last modified date and title.

    Results of a crawl

To view a URL, right click the URL and click on the Open button. The URL will be opened in a browser window. Click the Refresh button to refresh URL information.