dealer-crawler
A crawler that reads car dealers' public inventory pages.
What it does
dealer-crawler visits the inventory pages of car dealers' own websites, usually once a night, and records each listed vehicle's advertised price, mileage, VIN and details. Comparing nights shows how long a car has been listed and when its price changed. The data is offered to subscribers of a vehicle-inventory API, and every record links back to the dealer's own page.
What it reads
- Inventory pages only: the same vehicle listings any shopper sees. It also reads
robots.txt, and at most one terms-of-use page per dealer group. - It never signs in, submits forms, or collects anything about shoppers or buyers.
How it behaves
- It identifies itself honestly. Its User-Agent is
dealer-crawler/<version> (+<contact address>). - It obeys
robots.txtrules for bothdealer-crawlerand*, includingCrawl-delay. - It sends one request at a time to a site, at least 2 seconds apart, with an overall cap per website platform.
- If a site blocks or challenges it, it stops and records the dealer as blocked. It never solves challenges, spoofs another client, or routes around a block through proxies.
How to verify it
Requests may be signed with Web Bot Auth (HTTP Message Signatures). Signed requests carry Signature-Agent: "https://crawler.spurauto.com". The public key directory is https://crawler.spurauto.com/.well-known/http-message-signatures-directory, key id ZQPEny1b23tTwHWRVtlmcauvWfg6htOdbFKOOzoHyuM.
Opting out
To stop dealer-crawler reading your site, add to your robots.txt:
User-agent: dealer-crawlerDisallow: /
To have your inventory removed from the data as well, email hacksforlife14@gmail.com with your website address. It is removed from the next nightly update and the site is no longer read.
Contact
Questions or problems with the crawler: hacksforlife14@gmail.com.