Any good web crawler has limits.

Yeah. Like, literally just:

• Keep track of which URLs you’ve been to
• Avoid going back to the same URL
• Set a soft limit, once you’ve hit it, start comparing the contents of the page with the previous one (to avoid things like dynamic URLs taking you to the same content)
• Set a hard limit, once you hit it, leave the domain altogether

What kind of lazy-ass crawler doesn’t even do that?