They follow robots.txt

There are different kinds of AI scraper defenses.

This one is an active strategy. No shit people know that this costs them resources. The point is that they want to punish the owners of bad-behaved scrapers.

There is also another kind which just blocks anything that tries to follow an invisible link that goes to a resource forbidden by robots.txt