Robots.txt Explained: What It Does and How to Use It

Robots.txt is a plain-text file that sits at the root of your website and tells search engine crawlers which parts of the site they are allowed to request. It is one of the oldest tools in technical SEO, and also one of the most misunderstood. Used well, it keeps bots away from pages that waste crawl effort. Used carelessly, a single line can hide your entire site from Google.

What robots.txt does

When a crawler visits your site, it first looks for https://yourdomain.com/robots.txt. The file lists rules that say which user agents (crawlers) may access which URL paths. A basic file looks like this:

• User-agent: * — the rules below apply to all crawlers.
• Disallow: /admin/ — do not request anything under /admin/.
• Allow: /admin/public/ — but this subfolder is fine.
• Sitemap: https://yourdomain.com/sitemap.xml — where to find the sitemap.

The Sitemap line is a useful pointer that helps crawlers discover your XML sitemap, even though it is not a crawl rule.

Rules are grouped by user agent, so you can set different instructions for different crawlers. A block beginning User-agent: Googlebot applies only to Google, while User-agent: * covers everyone not named specifically. Within a block, Disallow lines close off paths and Allow lines carve out exceptions. Most sites need only a single, simple block — complexity here tends to create bugs rather than solve them.

Crawling versus indexing

This is the distinction that trips people up. Robots.txt controls crawling — whether a bot may fetch a URL. It does not control indexing — whether a URL appears in search results. If a blocked page has links pointing to it, Google can still list the URL in results without a description, because it was told not to read the page. To keep a page out of the index, you must let Google crawl it and use a noindex tag, or protect it behind a login.

The practical takeaway is to match the tool to the goal. Use robots.txt to stop crawlers from wasting effort on low-value URLs, and use a noindex directive to keep pages out of search results. Confusing the two is behind a surprising share of indexing problems, because the two controls pull in different directions and blocking a page prevents Google from ever seeing the noindex instruction on it.

How to use robots.txt correctly

Reach for robots.txt when you want to save crawl effort on URLs that have no business in search: faceted-search filter combinations, internal search results, cart and checkout paths, or staging directories. Keep the file small and specific. Test every change in Google Search Console before you trust it, and remember that paths are case-sensitive and matched from the start of the URL path. When your goal is to consolidate duplicate URLs rather than block them, a canonical tag is usually the better tool, because it keeps the pages crawlable while pointing ranking signals at one master URL.

On very large sites, robots.txt also helps manage crawl budget — the finite attention Google gives your site. Steering crawlers away from thousands of parameter-generated URLs leaves more room for them to reach your genuinely important pages. On a small site, crawl budget is rarely a concern, so the main value of robots.txt is simply keeping obviously private or pointless paths out of the crawl. Either way, change it deliberately and document why each rule exists.

Common robots.txt mistakes

The most damaging mistake is a stray Disallow: /, which blocks the entire site — this often survives a launch when a staging file is copied to production. Another is blocking CSS or JavaScript that the page needs to render, which can stop Google from seeing your layout correctly. A third is using robots.txt to try to hide sensitive files: the file is public, so listing a secret folder simply advertises it. Finally, blocking a page you also want deindexed backfires, because Google can never read the noindex tag on a page it is not allowed to crawl.

Can robots.txt keep a page out of Google?

Not reliably. Blocking a URL in robots.txt stops Google from reading its content, but the URL itself can still appear in results if other pages link to it. If your real goal is to keep something out of search entirely, allow crawling and add a noindex meta tag or X-Robots-Tag header, or require authentication. Think of robots.txt as a way to manage crawler traffic, not as a privacy or security control. Google has been explicit on this point: a page disallowed in robots.txt can still be indexed as a bare URL, and the only dependable ways to keep something out of results are a crawlable noindex directive or a login wall.

Do all crawlers obey robots.txt?

No. Robots.txt is a voluntary standard. Reputable search engines such as Google and Bing follow it, but malicious scrapers and many aggressive bots ignore it completely. Because the file is publicly readable and merely advisory, you should never rely on it to protect confidential URLs. Anything that must stay private belongs behind authentication, not behind a Disallow line.

How do I check whether my robots.txt is working?

Start by visiting yourdomain.com/robots.txt in a browser to confirm the file loads and shows the rules you expect. Then use the robots.txt report and the URL Inspection tool in Google Search Console to see how Google interprets specific URLs and whether any important page is blocked. After any launch or migration, re-check the file first — it is one of the quickest checks on a technical SEO review and one of the easiest things to get wrong, which is why it belongs alongside your on-page SEO checklist.