Blog » Web Crawler
Search 101 - how search engines work - part 1
Having some background on how search engines work is very useful when you're trying to optimize your site. We hope we've a bit of perspective on this, having spent the best part of a decade implementing search engines and web crawlers. How do words on a web page end up searchable? This happens in three phases: Crawling (or spidering) the web, finding pages people want to search Indexing words on web pages Searching the index (i.