

If nginx, here’s an open-source blocker/honeypot: https://github.com/raminf/RoboNope-nginx
If you have it set up to be proxied or hosted by Cloudflare, they have their own solution: https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/
Totally understandable.
If scanning to help send traffic to your website, that’s cool. If scanning to generate summaries that won’t send any traffic your way. No bueno.
Ultimately, it should be whatever most benefits users.