Skip to content
cubdomaincubdomain

robots.txt Generator

Create a robots.txt file that tells search engines and AI crawlers what they can visit.

Default for all robots
Rules
Block AI crawlers
Start from an existing robots.txt
robots.txt
User-agent: *
Allow: /
Disallow: /admin/

Upload it as robots.txt to your site's root folder: https://yoursite.com/robots.txt.

How to use it

  1. 01

    Choose a default

    Allow or block all robots.

  2. 02

    Add rules

    Block folders for specific crawlers, and choose which AI crawlers to block.

  3. 03

    Download the file

    Upload robots.txt to your site's root folder.

Questions

Does robots.txt hide pages from Google?

It stops crawling, but a blocked page can still appear in results if other sites link to it. Use a noindex tag to keep it out.

Do all crawlers follow robots.txt?

Reputable ones do. It is a request, not a lock, so don't rely on it to protect private content.

Can I block AI crawlers?

Yes — add rules for crawlers like GPTBot or CCBot. Blocking them doesn't affect normal search indexing.