Search Engine Guidelines: Robots.txt Standard

Robots.txt is a standard used to control how search robots should access websites. Mistakes in use robots.txt can prevent a site appearing in search engines.

Rules