Error ErrCrawlBlockedByRobotsStatus

Description

The site could not be scanned because requests for ROBOTS.TXT return ‘401 Unauthorized’ or ‘403 Forbidden’.

Help

Please contact the site administrator or developer to resolve this issue.

Applicable Standards

• Blocked Links
  

Change history

• 3.4 Sep 2009 Added.

This page describes a web site issue detected by SortSite Desktop and OnDemand Suite.

Rule ID: ErrCrawlBlockedByRobotsStatus