Web / SEO

robots.txt Generator

Create a clean robots.txt file for search engines and other crawlers. The tool generates user-agent rules, allowed and blocked paths, sitemap entries and an optional crawl-delay hint.

Create robots.txt

Choose a template, add your own paths and copy the generated file into the root directory of your website. A robots.txt file is a crawler instruction and does not replace real access protection.

Generator

The output is created locally in your browser. No data is sent to a server.

Example: *, Googlebot or Bingbot.

One path per line. Empty lines are ignored.

Do not block CSS or JavaScript files that are required to render your pages.

Optional, but very useful for search engines.

Optional. Not all crawlers use crawl-delay.

Comments are written with # in the robots.txt file.

Output

What is robots.txt used for?

The robots.txt file usually lives in the root directory of a website, for example at /robots.txt. It tells crawlers which areas they may visit and which areas should not be crawled.

User-agent

Defines which crawler the following rules apply to. * means all crawlers.

Disallow

Blocks paths for crawlers, such as internal system folders or search result pages.

Sitemap

Points to your XML sitemap so search engines can discover important URLs faster.

Common examples

Allow everything

Useful for public websites without sensitive areas. The robots.txt file then contains an empty Disallow rule.

Block private areas

Common for websites with internal areas such as /config/, /system/ or /admin/.

Important notes

robots.txt is not password protection

A robots.txt file does not prevent direct access to files. Sensitive areas must be protected server-side, for example with authentication, file permissions or web server configuration.

Check the generated file before uploading it. Incorrect rules can prevent important pages from being crawled or expose paths that should not be listed publicly.