robots.txt returns 401 or 403 status code Error

Description

The site could not be scanned because requests for ROBOTS.TXT return 401 Unauthorized or 403 Forbidden.

Help

Please contact the site administrator or developer to resolve this issue.

Applicable standards

  • Blocked Links

Change history

  • 5.3 Sep 2013 Removed - the original robots.txt draft treated HTTP 401/403 responses for robots.txt as site blocked, but Googlebot treats this like a 404.
  • 3.4 Sep 2009 Added.

This page describes a web site issue detected by SortSite Desktop and OnDemand Suite.

Rule ID: ErrCrawlBlockedByRobotsStatus