[Skip navigation links]

Blog » Web Crawler

Search 101 - how search engines work - part 1

Posted by Mark Rogers on Jan 30, 2008 | 


Having some background on how search engines work is very useful when you're trying to optimize your site. We hope we've a bit of perspective on this, having spent the best part of a decade implementing search engines and web crawlers. How do words on a web page end up searchable? This happens in three phases: Crawling (or spidering) the web, finding pages people want to search Indexing words on web pages Searching the index (i.