LLMs.txt might become the new robots.txt
I recently read a tweet by Immad Akhund (CEO of Mercury) in which he claimed an exponential rise of signups on Mercury via AI search.
He said, they didn't do much to optimize for AI search, but just created an LLMs.txt file which is robots.txt equivalent for AI search engines.
I dug deeper and found that LLMs.txt is an emerging standard used by websites to expose information in a structured format to AI developer tools.
LLMs.txt file ➡️ follows the Markdown format ➡️ is quick to load and easy to read for LLMs ➡️ each page of the website is summarized in a one-sentence description with its URL ➡️ structured for AI so it's easy to understand for LLMs
I've seen Mercury, Cohere, Vapi, beehiiv, ElevenLabs, LaunchDarkly, and many more companies starting to use LLMs.txt on their website, knowledge base, and docs.