You are viewing a single thread.
View all comments
24 points

I’ve kinda said this before, but I expect shit like this to feed a growing trend of treating all crawlers as malicious until proven otherwise, and blocking them accordingly - even if it causes serious problems, its arguably better than the risk of getting fucked by an AI crawler.

I also expect robots.txt to fall out of use as a consequence of this plague of AI crawlers. Between OpenAI openly ignoring it, Perplexity openly deceiving it, and Anthropic spamming crawlers to get past it, the trust necessary for robots’ “voluntary compliance” model to work has been thoroughly fucking shredded.

permalink
report
reply

TechTakes

!techtakes@awful.systems

Create post

Big brain tech dude got yet another clueless take over at HackerNews etc? Here’s the place to vent. Orange site, VC foolishness, all welcome.

This is not debate club. Unless it’s amusing debate.

For actually-good tech, you want our NotAwfulTech community

Community stats

  • 1.5K

    Monthly active users

  • 417

    Posts

  • 11K

    Comments

Community moderators