EverylotBot

EverylotBot is the crawler behind Everylot. It reads public used-vehicle pages on dealer websites, keeps the facts, and links every result back to the dealer. This page is what a webmaster needs.

What it fetches

robots.txt, your sitemap, and used-vehicle detail pages that carry schema.org vehicle data (the same structured data you publish for Google's vehicle listings). It does not log in, does not run scripts, does not bypass bot challenges, does not fetch new-vehicle or lead-form pages, and stops on a host that answers with a challenge or a 429.

How often

robots.txt daily, sitemaps twice a day, each vehicle page about once a day with If-Modified-Since and ETag so unchanged pages cost a 304. At least two seconds between requests to one host, or your Crawl-delay if larger, and at most a few pages per host per minute.

Identify it

Mozilla/5.0 (compatible; EverylotBot/0.1; +https://everylot.io/bot)

Control it

Standard robots.txt rules apply and are re-read daily. Block everything:

User-agent: EverylotBot
Disallow: /

Slow it down:

User-agent: EverylotBot
Crawl-delay: 10

A page with <meta name="robots" content="noindex"> or an X-Robots-Tag: noindex header is never listed. Prefer email? See removal requests.

What we store

Vehicle facts (year, make, model, trim, price, mileage, colors, VIN), dealer name and address, the listing URL, and the URLs of listing photos. Listing descriptions and page text are not stored. Photos are fetched only to make reduced previews that link back to the source, and those previews expire when a listing disappears.