I, robot: How do search engine spiders and robots work? - SEO Consulting