FinchleyBot

FinchleyBot is the crawler Finchley uses to read a website an agency has added as a knowledge source for one of its clients. It only visits sites that a Finchley customer has asked it to read, and it only reads pages a normal visitor could open. It identifies itself as:

FinchleyBot/1.0 (+https://finchley.ai/bot)

It fetches a page's HTML and follows links on the same site, a few hundred pages at most, with a short pause between requests. It does not submit forms, sign in, or read anything behind a login. The text it collects is used only to answer questions on the customer's own site, and it is refreshed daily so answers stay current.

Keeping it out

FinchleyBot honours robots.txt. To block it from your whole site:

User-agent: FinchleyBot
Disallow: /

Or block particular paths with the usual Disallow lines. Changes take effect the next time the site is read.

Questions

If you see FinchleyBot on a site you did not expect, or want a site removed from Finchley entirely, email support@finchley.ai and a person will sort it out.