About our crawler
InTheAnswerBot reads the public websites of Ontario clinics to prepare a report for that clinic about its own site.
- It follows robots.txt. To keep it out, add
User-agent: InTheAnswerBotandDisallow: /. - It makes one request every few seconds and reads at most 40 pages per site.
- It reads only public pages. It does not log in, submit forms or book appointments.
Questions: hello@intheanswer.ca.