robots.txt
returns 401 or 403 status code
Error
Description
The site could not be scanned because requests for ROBOTS.TXT return 401 Unauthorized
or 403 Forbidden
.
Help
Please contact the site administrator or developer to resolve this issue.
Applicable standards
- Blocked Links
Change history
- 5.3 Sep 2013 Removed - the original robots.txt draft treated HTTP 401/403 responses for robots.txt as site blocked, but Googlebot treats this like a 404.
- 3.4 Sep 2009 Added.
This page describes a web site issue detected by SortSite Desktop and OnDemand Suite.
Rule ID: ErrCrawlBlockedByRobotsStatus