Parts of the Internet now only searchable on specific sites now? What next - charging a monthly subscription to use Google?
This needs to be regulated before the Internet becomes like streaming TV.
Robots.txt has been around for a long time, and all the major search engines will honor it. Not having a full index of the Web is the norm.
That isn’t to say that the practice of signing agreements isn’t potentially a concern. Not sure that I like the idea of search engines paying sites money to degrade search results of competitors.
These two shitty companies deserve each other.
Excellent! Now I won’t get reddit results and then have to filter them out!
Kagi has a “search lens” specifically to search the Threadiverse. Like, they track lemmy/kbin/etc sites and you can specifically include them in their own results section, and can also have “!threadiverse” or whatever you want specifically search that.
They do the same for Usenet.
I suppose, given this new robots.txt Reddit development, that they’ll probably never have a Reddit lens, though.
Thank you Lemmy, for making it so much easier to walk away from that dumpster fire!
This seems illegal to me 😮
