About gaybooklistbot
If you found this page from your server logs: hello! Here's exactly what our crawler does.
gaybooklistbot/1.0 is the fetcher for Gay Book List (https://www.gaybooklist.com), a community-run LGBTQ+ book catalog maintained by one person.
- What it fetches: published RSS feeds of a small, hand-picked list of LGBTQ+ book sites (once per day, with conditional requests so unchanged feeds cost you nothing), and occasionally individual public pages (publisher product pages, review pages) while researching a specific book.
- What we keep from feeds: post title, link, date, and the short excerpt your feed itself provides. We never store or republish full article text.
- robots.txt: honored on every fetch, including re-checking your file daily. Blocking
gaybooklistbot(or all agents) stops us within a day. We never bypass paywalls, challenges, or blocks — a block is treated as permanent. - To block us:
User-agent: gaybooklistbot · Disallow: /— or just email us and we'll remove your site immediately.
Questions, requests, or a “please stop”: contact@gaybooklist.com. See also where our data comes from.