Google’s John Mueller stated it could make sense to noindex your LLMs.txt file. This fashion, the file does not get listed by search engines like google, after which one way or the other a consumer lands on it and is confused.
John wrote on Bluesky, “That stated, utilizing noindex for it might make sense, as websites would possibly hyperlink to it and it might in any other case develop into listed, which might be bizarre for customers.”
The query was:
Will Google view LLMs.txt recordsdata as duplicate content material? It appears stiff necked to take action, on condition that they know that it’s not, and what it’s actually for.
Ought to I add a “noindex” header for llms.txt for Googlebot?
John Mueller replied in full:
It will solely be duplicate content material if the content material have been the identical as a HTML web page, which would not make sense (assuming the file itself have been helpful). That stated, utilizing noindex for it might make sense, as websites would possibly hyperlink to it and it might in any other case develop into listed, which might be bizarre for customers.
John beforehand stated that so far as he is aware of no AI system at the moment makes use of llms.txt.
In any occasion, quite a lot of websites are adopting it, will you noindex it?
Discussion board dialogue at Bluesky.
