Error ErrCrawlBlockedByRobotsStatus

Description

The site could not be scanned because requests for ROBOTS.TXT return ‘401 Unauthorized’ or ‘403 Forbidden’.

Help

Please contact the site administrator or developer to resolve this issue.

Applicable Standards

Change history

This page describes a web site issue detected by SortSite Desktop and OnDemand Suite.

Rule ID: ErrCrawlBlockedByRobotsStatus