What Is Robots.txt? A Plain-English Guide With Practical Examples

A robots.txt file is a small plain-text file that lives at the root of a website — like yoursite.com/robots.txt — and tells search engine crawlers which pages and folders they’re allowed to visit. It’s a set of polite instructions, not a security lock. Well-behaved crawlers follow it, but it won’t hide pages from people.

If you’ve ever run an SEO audit or poked around your website’s settings, you’ve probably seen robots.txt mentioned. It sounds technical, but the idea is simple. Think of it as a “do not enter” sign you hang on certain doors of your website for search engine bots.

Most small websites don’t need a complicated one. But a single wrong line can accidentally block your entire site from Google — and that happens more often than you’d think. This guide explains what robots.txt does, shows you practical sample examples you can adapt, and points out the mistakes that hurt the most.

How Does Robots.txt Actually Work?

When a search engine crawler (like Googlebot) visits your site, one of the first things it does is check yoursite.com/robots.txt. It reads the file, looks for rules aimed at it, and then crawls your site accordingly.

Here’s the key distinction most people miss: robots.txt controls crawling, not indexing. Crawling means the bot visits the page. Indexing means Google adds the page to its search results. Blocking a page in robots.txt stops the bot from visiting — but if Google already knows the page exists because other sites link to it, the URL can still show up in search results (usually without a title or description). So robots.txt is the wrong tool if your goal is removing a page from Google. More on that in the FAQ.

A few more basics worth knowing:

• The file must live at the root of your domain. Your site’s crawlers only look there.
• It has to be named exactly “robots.txt” — all lowercase, no variations.
• It’s public. Anyone can type your domain plus /robots.txt into a browser and read it, so never put anything sensitive in it.
• The major search engines respect it, but shady scrapers and spam bots often ignore it completely.

What Do the Commands in a Robots.txt File Mean?

A robots.txt file is built from short, simple rules. Each rule names a “user-agent” (the bot the rule applies to) and then gives instructions. Here are the commands you’ll actually run into:

• User-agent: — names the crawler the rules below it apply to. User-agent: * means “every crawler.” User-agent: Googlebot means only Google’s main crawler. You can write separate rule blocks for different bots.
• Disallow: — lists paths the bot may not crawl. Disallow: /admin/ blocks everything inside the /admin/ folder. Disallow: /private-page blocks one specific page. And Disallow: / — with nothing after the slash — blocks the entire site.
• Allow: — carves out an exception inside a disallowed area. For example, you might block /downloads/ but allow one public file inside it.
• Sitemap: — points crawlers to your XML sitemap, like Sitemap: https://yoursite.com/sitemap.xml. This one is optional, but it helps crawlers discover your pages faster.

You might also see Crawl-delay in older guides. It was meant to tell bots to slow down between requests, but Google never supported it, so it’s ignored by the crawler that matters most. If bots are overloading your server, that’s a hosting or server-configuration issue — robots.txt can’t fix it.

Rules are matched from the most specific to the least specific, and a bot follows the single most specific rule that matches the page it’s about to visit. That’s why Allow exceptions work even inside Disallowed folders.

Where Do You Put a Robots.txt File?

At the root of your domain. The very top level. That’s the only place crawlers ever look.

• Works: https://yoursite.com/robots.txt
• Does nothing: https://yoursite.com/blog/robots.txt

If you run WordPress, most SEO plugins (like Yoast SEO or Rank Math) give you a built-in robots.txt editor, so you never have to touch server files. If your site has no robots.txt file at all, that’s completely fine — crawlers simply treat it as “crawl everything you can reach.” A missing file is not an error and won’t hurt you.

The file needs to be served as plain text and return a 200 status code when requested. If it returns a 404, crawlers treat that the same as having no file and crawl freely.

What Are Some Practical Robots.txt Examples?

Seeing sample files makes everything click. Here are three common setups you can adapt.

1. The default: let everything crawl

User-agent: *
Disallow:

That empty Disallow means “block nothing.” Every crawler may visit every page. This is a common choice for small websites, and it’s perfectly fine. If you don’t have specific pages to hide from bots, this (or no file at all) is the right choice.

2. Block low-value areas, keep the rest open

User-agent: *
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /?s=
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yoursite.com/sitemap.xml

This is a typical WordPress setup. It keeps bots out of the login area, the shopping cart, checkout pages, and internal search-result URLs — pages with zero SEO value that just waste your crawl budget (the limited attention Google gives your site per visit). The Allow line makes one exception so the admin-ajax file, which themes and plugins need to function, stays reachable. The Sitemap line points crawlers to the XML sitemap.

3. Block everything — for staging sites only

User-agent: *
Disallow: /

One line, total block. This belongs on a staging or development copy of your site so Google doesn’t index your unfinished work. It should never appear on a live site — yet this is one of the most damaging robots.txt mistakes in SEO. Developers forget to remove it at launch, and the brand-new site is invisible in Google for weeks before anyone notices.

What Should You Never Block in Robots.txt?

Some blocks sound smart but backfire badly. Keep these off your Disallow list:

• Your CSS and JavaScript files. Google needs these to render your pages the way real visitors see them. Block them and Google sees a broken, unstyled version of your page, which can drag down rankings.
• Your entire site by accident. A stray Disallow: / on a live site is an SEO emergency. Always re-read the file after editing it.
• Pages you want removed from search results. This is the big misunderstanding. If you disallow a page, Googlebot can’t visit it — which means it also can’t see a “noindex” tag on it. The page can then linger in search results indefinitely. To remove a page from Google, use a noindex tag (or password-protect the page) and let crawlers visit it.
• Image or media folders if your images bring traffic through Google Image search.
• Your XML sitemap. There’s no reason to block the very file that helps crawlers find your content.

When in doubt, block less. A minimal robots.txt is far safer than an aggressive one.

What Are the Most Common Robots.txt Mistakes?

The same errors show up again and again:

• Blocking the staging file on the live site. A developer copies the staging robots.txt (with Disallow: /) to production during launch and nobody checks. Weeks of invisible-in-Google follow.
• Using robots.txt to “delete” pages from Google. As explained above, this does the opposite of what people want — it freezes the page in the index instead of removing it.
• Blocking query parameters too broadly. A rule like Disallow: /*? can accidentally block legitimate pages that happen to use parameters, like filtered product listings or tracking URLs you actually want indexed.
• Typos in paths. Disallow: /wp-admin (no trailing slash) also blocks a hypothetical page called /wp-administration-guide. Trailing slashes matter.
• Forgetting the file is public. Some site owners disallow paths like /secret-launch-page/ — and then wonder how competitors found it. Anyone can read your robots.txt.

How Do You Check Your Robots.txt File?

Checking takes about ten seconds, and you should do it after every site migration, redesign, or developer handoff:

• Type your domain plus /robots.txt into your browser and read it. If you see Disallow: / on a live site, fix it immediately.
• In Google Search Console, open Settings and find the robots.txt report. It shows the exact file Google last fetched and flags any syntax problems. If it reports issues, see how to fix crawl errors in Search Console.
• After making changes, use Search Console’s URL Inspection tool on a few important pages to confirm Google can still crawl them.

If you’re not comfortable editing the file yourself, hand it to your developer or SEO person. It’s a two-minute job for someone who knows what they’re doing — and a very expensive mistake for someone who doesn’t.

Frequently Asked Questions

Is robots.txt a Google ranking factor?

No. Having a robots.txt file doesn’t boost your rankings, and not having one doesn’t hurt them. It’s purely a set of crawling instructions. The only thing that matters for SEO is that it doesn’t accidentally block pages you want ranked.

Does robots.txt hide a page from visitors?

No. It only talks to crawlers. Any person who knows (or guesses) the URL can still visit a disallowed page directly. If a page truly needs to stay private, protect it with a login or password — robots.txt is not security.

What’s the difference between robots.txt and a noindex tag?

They do opposite jobs. Robots.txt tells crawlers “don’t visit this page.” A noindex tag, placed in the page’s HTML, tells search engines “don’t show this page in results.” If you want a page out of Google, use noindex and allow crawlers to visit the page so they can see the tag. Blocking the page in robots.txt prevents exactly that.

Can a bad robots.txt file break my whole site?

It can wipe out your whole site’s visibility in search engines, yes. One line — Disallow: / — tells every crawler to stay away, and your pages will gradually disappear from Google. The site itself still works fine for visitors; it just becomes invisible in search. Always verify the file after editing.

Do all search engines follow robots.txt?

The major ones do — Google, Bing, and others treat it as a standard. But compliance is voluntary. Aggressive scrapers, spam bots, and some data-harvesting crawlers ignore it entirely. Think of robots.txt as a “please,” not a lock.

Related Guides

• How to Allow AI Bots in robots.txt (GPTBot, PerplexityBot, and More)
• What Is SEO? Search Engine Optimization Explained
• Why Is Technical SEO Important for Websites
• The Ultimate SEO Audit Checklist for 2026
• 301 vs 302 Redirect: Which One Should You Use?
• How to Fix 404 Errors for SEO: Find, Redirect, and Prevent

Technical SEO details like this are easy to get wrong and painful to fix after the fact. If you’d rather have an expert handle your crawl settings, site audits, and the rest of the technical side, take a look at our digital marketing services.