Crawler
Also known as: Bot, Spider, Web CrawlerA program search engines use to crawl the web. Bots visit webpages to collect information and add or update a search engine’s index.
FAQ
What do your call crawlers from different search engines?
Google’s crawlers are called googlebots, while those of Microsoft Bing are called bingbots, to name a few.