Pasta Dental@sh.itjust.workstoTechnology@lemmy.world•AI haters build tarpits to trap and trick AI scrapers that ignore robots.txtEnglish
34·
3 days agoThe big search engine crawlers like googles or Microsoft’s should respect your robots.txt file. This trick affects those who don’t honor the file and just scrape your website even if you told it not to
I mean… Yeah, it pretty obviously was a mistake from the beginning. Anyone who thinks otherwise is just as bad as the conspiracy theorists who think there’s a deep state controlling us with the 5g in the vaccines. Facebook don’t have any incentive to ban the word Linux, even they use it. It’s not like it’s a subject that would make their customers go away (like NSFW stuff for example). The shit I’ve seen people say “it’s probably Microsoft paying them to censor Linux users” as if 1. Linux is any threat to windows’s market share and 2. Microsoft didn’t intentionally keep Linux alive as a way to avoid antitrust lawsuits. Microsoft is big on Linux too with azure.