OpenAI has up to date its spider/crawler it makes use of for ChatGPT, OAI-SearchBot. The AI firm up to date the outline and particulars of the crawler info eradicating that it’s used for hyperlinks and to coach OpenAI’s generative AI basis fashions.
Pieter Serraris noticed this variation and posted about it on LinkedIn – he wrote, “OpenAI has made some huge adjustments within the description of the roles of their crawlers and consumer brokers.”
Listed here are the adjustments, the phrases highlighted in pink have been eliminated and the phrases highlighted in inexperienced have been added:
Pieter summarized the adjustments:
- It appears to be like like ChatGPT Consumer (the consumer motion bot) will not comply to robots.txt guidelines (Open modified the wording from “the next robots.txt tags”, referring to all 3 consumer brokers to “OAI SearchBot and GPTBot robots.txt tags”)
- OAISearchBot is not used to feed the navigational hyperlinks in ChatGPT solutions (blocking this bot doesn’t imply your is not going to seem within the hyperlinks)
- ChatGPT Consumer is now additionally mentioned for use for Customized GPT requests and GPT Actions.
- OAISearchBot and GPTBot share info with one another (“In case your website has allowed each bots, we could use the outcomes from only one crawl for each use circumstances to keep away from duplicate crawling.”) -> this confirms what we’ve been seeing recently within the log recordsdata: for some prompts we see 2 or all 3 OAI bots visiting the positioning.
For most individuals, OpenAI crawls your website far more than it sends you any visitors – so maintain that in thoughts.
Discussion board dialogue at LinkedIn.
