dealer-crawler

A crawler that reads car dealers' public inventory pages.

What it does

dealer-crawler visits the inventory pages of car dealers' own websites, usually once a night, and records each listed vehicle's advertised price, mileage, VIN and details. Comparing nights shows how long a car has been listed and when its price changed. The data is offered to subscribers of a vehicle-inventory API, and every record links back to the dealer's own page.

What it reads

How it behaves

How to verify it

Requests may be signed with Web Bot Auth (HTTP Message Signatures). Signed requests carry Signature-Agent: "https://crawler.spurauto.com". The public key directory is https://crawler.spurauto.com/.well-known/http-message-signatures-directory, key id ZQPEny1b23tTwHWRVtlmcauvWfg6htOdbFKOOzoHyuM.

Opting out

To stop dealer-crawler reading your site, add to your robots.txt:

User-agent: dealer-crawler
Disallow: /

To have your inventory removed from the data as well, email hacksforlife14@gmail.com with your website address. It is removed from the next nightly update and the site is no longer read.

Contact

Questions or problems with the crawler: hacksforlife14@gmail.com.