Free Robots.txt Generator

A plain text file at the root of a site that tells crawlers which paths they may fetch. It is the first thing a well-behaved crawler reads, and the rules in it decide what can appear in a search engine at all.

For most sites, yes. Without it every path is crawlable, which means admin endpoints and internal search results absorb crawl budget that should have gone to your real pages. Even a file that does nothing but name your sitemap is better than none.

It depends what you want. Blocking the training crawlers keeps your pages out of future models. Blocking the search-side ones also stops those assistants citing you, which is traffic you may want. The switch is off by default because there is no answer that is right for everyone.

At the root of the domain, so it answers at your-site.com/robots.txt. The path is fixed by the standard. A robots.txt anywhere else is read by nothing.

Publish it and then run the Robots.txt Analyzer against the live address: it fetches the real file and grades it. To see which AI crawlers your rules actually let through, run the AI Crawler Access Checker on the same site.

Disallow tells a crawler not to fetch a path. Allow carves an exception out of a broader Disallow, which is why the WordPress template blocks wp-admin but allows admin-ajax.php inside it.

Because there is nothing there for a reader. It holds the internal libraries WordPress renders pages with, so a crawler that visits only finds errors and directory listings, and the version information those leak is what tells an attacker which installs are worth trying.

It stops the crawlers that honour it, which includes Google, Bing and the named AI vendors. It stops nothing else: a scraper that ignores the file is not affected by anything you write in it. For those, the answer is a block at the server, not a line in a text file.