This is a proposal by some AI bro to add a file called llms.txt that contains a version of your websites text that is easier to process for LLMs. Its a similar idea to the robots.txt file for webcrawlers.
Wouldn’t it be a real shame if everyone added this file to their websites and filled them with complete nonsense. Apparently you only need to poison 0.1% of the training data to get an effect.
So AI should get the most relevant info, while we (humans) have to fight through ads, and popups and shit… At this point, I feel discriminated
Theoretically speaking, what level of nonsense are we talking about in order to really mess up the training model?
a) Something that doesn’t represent the actual contents of the website (like posting “The Odyssey” to the llms.txt of a software documentation site),
b) a randomly generated wall of real words out of context, or
c) just straight lorem ipsum filler?
Place output from another LLM in there that has thematically the same content as what’s on the website, but full of absolutely wrong information. Straight up hallucinations.
It would be incredibly funny wrong if this was adopted and used to poison LLMs.
We could respect this convention the same way the IA webcrawlers respect robot.txt 🤷♂️
Do webcrawlers from places other than Iowa respect that file differently?
I’m sure it will totally be respected and used correctly.
