Robots.txt Generator

Robots.txt is a small plain-text file that lives in the root of a website and tells visiting crawlers - search engine bots such as Googlebot and Bingbot - which URLs they may and may not request. Each rule names a user-agent and lists allowed or disallowed paths. Well-behaved crawlers read it before crawling. It has been a web standard since 1994 and almost every site, large or small, has one.

Not entirely. A disallowed page cannot be crawled, but if other sites link to it, search engines may still list its URL in results - shown without a description, because they could not read the content. For a page that must truly stay out of search results, use a noindex tag after allowing the crawl, or protect it with a login. Never use robots.txt to hide sensitive data such as private documents or personal information.

Disallow forbids crawling for everything under a path; Allow carves out an exception inside a blocked area. For example, Disallow: /admin/ with Allow: /admin/public/ blocks the whole admin folder except its public subfolder. When rules overlap, crawlers generally apply the most specific matching rule - the one with the longest path - which is what makes an Allow exception work even though the broader Disallow also matches.

Crawl-delay tells a crawler how many seconds to wait between requests, which can protect a small or slow server from being overwhelmed by rapid crawling. Some crawlers, notably Bing's, honour it. Googlebot ignores the line completely - Google asks you to manage crawl rate in Search Console instead. Include it if your host struggles under crawl load, but do not expect it to change how often Google visits.

It must sit in the top-level directory of a host and be named exactly robots.txt, all lowercase, so it is reachable at yoursite.com/robots.txt. Each subdomain needs its own file - the robots.txt on www.example.com does not cover blog.example.com - and the same applies to non-standard ports. This generator gives you the file ready-named when you download it; you just upload it to that root location.

Yes. A file can hold several groups, each starting with its own User-agent line followed by that crawler's rules, and crawlers only obey their own group (plus the wildcard group). This generator builds one group at a time, which covers the most common case: identical rules for everyone. For a special case, generate a second file for the named crawler and merge the two blocks together in a text editor.

No. Robots.txt is a voluntary standard - a set of requests, not a lock. Reputable search engines obey it, but spam bots, scrapers and attackers routinely ignore it, and some actively read it to find the paths you hoped to hide. Protect private areas with passwords and proper access controls, and treat robots.txt purely as traffic guidance for cooperative crawlers.