Understanding it
The file is made of rule groups: a User-agent line names a bot (or all of them with *), then Disallow and Allow lines list the forbidden or permitted paths. It can also point to the sitemap.
AI has given the file a new role. OpenAI, for example, separates GPTBot, which collects content for training, from OAI-SearchBot, which powers ChatGPT search. Google offers Google-Extended to opt out of Gemini training without affecting Google Search. You can say yes to one and no to the other.
It is an instruction, not a lock. Reputable bots follow it, others do not, and a sensitive page must be protected another way, with a password.
A concrete example
Illustrative example: opting out of training, staying citable
A specialist publisher does not want its guides used to train models, but wants to be cited in ChatGPT’s answers. Its robots.txt blocks GPTBot and allows OAI-SearchBot.
It then checks its server logs to make sure each bot behaves as expected, and records the decision so it can be reviewed: the list of AI bots keeps changing, and each vendor documents its own.
The common mistake
Blocking every AI bot without realizing it
A rule that is too broad, a copied template or a hosting option switched on by default can shut out every AI bot, including the ones that get you cited. Review your robots.txt bot by bot, and decide each access deliberately.