Robots.txt Standard Search Engine Guidelines

Robots.txt is a standard that controls how search robots access websites. Mistakes in robots.txt files can prevent a site being indexed by search engines.

• Robots.txt Standard

Rules

• Page has noindex meta tag
• Page has nofollow meta tag