Skip to content

Implement bot detection challenge#360

Closed
jazairi wants to merge 1 commit intomainfrom
use-410
Closed

Implement bot detection challenge#360
jazairi wants to merge 1 commit intomainfrom
use-410

Conversation

@jazairi
Copy link
Contributor

@jazairi jazairi commented Feb 24, 2026

Why these changes are being introduced:

We need a preliminary solution to manage bot traffic in production.

Relevant ticket(s):

How this addresses that need:

  • Implements a Bot Detector service that uses Crawler Detect to identify bots to refer to Cloudflare Turnstile.
  • Implements a Turnstile Controller (and view) to manage routing for verification challenges.

Side effects of this change:

  • I chose not to pass the remoteip param to CloudFlare, because I
    wanted to avoid potentially passing a non-bot user's IP to a third
    party.
  • I had some uncertainty about logging, particularly around failed
    CrawlerDetect evaluations. It seems like it might be useful to send
    those to Sentry, but I was wary of doing so until we learn how much
    noise they introduce.
  • The new form may need additional styling. For now, I just added our
    button styles.

Developer

Accessibility
  • ANDI or WAVE has been run in accordance to our guide.
  • This PR contains no changes to the view layer.
  • New issues flagged by ANDI or WAVE have been resolved.
  • New issues flagged by ANDI or WAVE have been ticketed (link in the Pull Request details above).
  • No new accessibility issues have been flagged.
New ENV
  • All new ENV is documented in README.
  • All new ENV has been added to Heroku Pipeline, Staging and Prod.
  • ENV has not changed.
Approval beyond code review
  • UXWS/stakeholder approval has been confirmed.
  • UXWS/stakeholder review will be completed retroactively.
  • UXWS/stakeholder review is not needed.
Additional context needed to review

This adds the crawler_detect gem.

The best way to see the bot detection page is to flip the boolean in BotDetector#should_challenge? locally. However, the challenge will only render for registered domains in our CloudFlare account.

Code Reviewer

Code
  • I have confirmed that the code works as intended.
  • Any CodeClimate issues have been fixed or confirmed as
    added technical debt.
Documentation
  • The commit message is clear and follows our guidelines
    (not just this pull request message).
  • The documentation has been updated or is unnecessary.
  • New dependencies are appropriate or there were no changes.
Testing
  • There are appropriate tests covering any new functionality.
  • No additional test coverage is required.

@mitlib mitlib temporarily deployed to timdex-ui-pi-use-410-4s9zkq8hy February 24, 2026 23:25 Inactive
@jazairi jazairi temporarily deployed to timdex-ui-pi-use-410-4s9zkq8hy February 25, 2026 18:42 Inactive
@jazairi jazairi temporarily deployed to timdex-ui-pi-use-410-4s9zkq8hy February 25, 2026 18:42 Inactive
@jazairi jazairi temporarily deployed to timdex-ui-pi-use-410-4s9zkq8hy February 25, 2026 18:46 Inactive
Why these changes are being introduced:

We need a preliminary solution to manage bot traffic in production.

Relevant ticket(s):

- https://mitlibraries.atlassian.net/browse/USE-410

How this addresses that need:

- Implements a Bot Detector service that uses Crawler Detect to identify
bots to refer to Cloudflare Turnstile.
- Implements a Turnstile Controller (and view) to manage routing for
verification challenges.

Side effects of this change:

- I chose not to pass the `remoteip` param to CloudFlare, because I
wanted to avoid potentially passing a non-bot user's IP to a third
party.
- I had some uncertainty about logging, particularly around failed
CrawlerDetect evaluations. It seems like it *might* be useful to send
those to Sentry, but I was wary of doing so until we learn how much
noise they introduce.
- The new form may need additional styling. For now, I just added our
button styles.
@jazairi jazairi temporarily deployed to timdex-ui-pi-use-410-4s9zkq8hy February 25, 2026 19:05 Inactive
@jazairi jazairi marked this pull request as ready for review February 25, 2026 19:11
@jazairi jazairi closed this Feb 25, 2026
@jazairi jazairi deleted the use-410 branch February 25, 2026 20:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants