Free Robots.txt Tester, Checker & Validator

This is a free robots.txt tester and checker. You paste a website URL, and the tool fetches that site's robots.txt file from its root, validates every line, and summarizes what the file actually does. It then lets you test a specific URL path against a specific crawler so you can see, in plain language, whether Googlebot is allowed to index a page or whether an AI training bot is blocked.

Most webmasters never read their own robots.txt until something goes wrong. That is usually a page falling out of the index or a whole section going dark after a migration. This tool turns a normally invisible file into something you can read, understand, and fix in one sitting, without logging into Search Console.

The role of robots.txt in SEO is narrower than its reputation suggests, and knowing the boundary saves a lot of wasted effort. The file governs crawling, not ranking and not indexing. It cannot push a page up the results, and it cannot reliably pull one out of them. What it does control is where crawlers spend their time on your site, which matters enormously once you are large enough for crawl budget to bite, and it is the fastest way to lose everything if you block the wrong path. Treat it as a traffic director for bots, not as an SEO lever, and you will use it correctly.

Testing and validating are two different jobs, and this tool does both. The tester answers a behavioural question, can this crawler reach this URL. The validator answers a structural one, is this file written correctly in the first place. A file can be perfectly valid and still block your whole site, and it can contain a broken line that silently voids the rule you were counting on.

Every time you check your robots.txt here, the built-in robots.txt file validator walks it one line at a time and reports what it finds. These are the conditions it looks for:

One point worth stressing, because it accounts for a surprising share of failed checks: the file is named robots.txt, plural, with a dot. A file uploaded as robot.txt, robots.text, or robots txt is not read by any crawler. The name is part of the standard, so a typo there means your rules simply never take effect, and nothing in your analytics will tell you.

Where the Google robots.txt checker went

If you came here looking for the Google Search Console robots.txt tester, there is something you should know first: Google retired it. The old robots.txt Tester inside Search Console, the one with the editable text box and the red highlight on the blocking line, was shut down in December 2023 after more than a decade of service. Plenty of tutorials still link to it, which is why so many people searching for a Google robots.txt checker arrive at a dead URL.

The Search Console robots.txt report

What replaced it is a report rather than a tester. To check robots.txt in Google Search Console today, open Settings, then robots.txt, and Google shows you the last file it fetched for each host, when it fetched it, and any parsing problems it hit. It is genuinely useful for confirming what Google is currently working from, but it is a read-only view. There is no box to paste a URL into and no way to try a rule change before you ship it.

Using this tool as a Googlebot checker

Select Googlebot in the crawler list and the access checker behaves as a Googlebot checker, resolving a path exactly the way Google's parser would, including the longest-match and Allow-over-Disallow precedence written into RFC 9309. Switch the selection to Googlebot-Image, Bingbot, or GPTBot and you can verify the same path against each crawler in turn, which the Search Console report cannot do at all.

The practical workflow is to use both. Test and iterate here, where you can check any path against any crawler without signing in, then confirm in Search Console that Google has picked up the file you deployed. If you need Google's own verdict on one specific live URL, the URL Inspection tool still reports crawl-blocked status directly from Google's index.

Google recrawls robots.txt roughly once every 24 hours, so a fix you deploy today may not show in the Search Console report until tomorrow. This tool reads the file live, which is why the two can disagree for a day after a change.

A robots.txt error is one of the few SEO problems that can erase months of work overnight, and it is almost always self-inflicted. These are the concrete reasons to test your file before and after any change:

A good robots.txt is boring. It is short, lives at the right path, and only contains directives the crawler can actually understand. Here are the qualities that separate a healthy file from a risky one:

• Located at the root. The file must be at yourdomain.com/robots.txt. A file anywhere else is ignored.
• Only valid directives. Stick to User-agent, Allow, Disallow, Sitemap, and optionally Crawl-delay. Unknown fields are skipped or flagged.
• Googlebot is never blocked. Unless the whole site is intentionally offline, blocking Googlebot removes you from search.
• No blocking of CSS and JS. Google needs to render your pages, so assets like CSS and JavaScript must stay crawlable.
• Sitemap is declared. A working Sitemap: line speeds up discovery of new and updated pages.
• Rules are specific. Target folders you actually want hidden, not broad patterns that accidentally cover important pages.
• No sensitive paths listed. Never use robots.txt to hide private URLs, the file is public and tells attackers exactly where to look.

When a crawler like Googlebot hits your site, it evaluates your robots.txt against the exact URL it wants to fetch. The logic comes from the Robots Exclusion Protocol, the robot exclusion standard first agreed between webmasters and crawler operators in 1994 and finally published as RFC 9309 in 2022. This tester follows the same rules, so its answer matches what a real crawler would do.

This matches Google's documented behaviour. For the official answer on a production site, the URL Inspection tool in Google Search Console applies the same logic against Google's own parser.

Finding the file is the easy part, because the standard fixes its address. It always sits at the root of the host, at yourdomain.com/robots.txt, and it is always public. That is true for your own site and for any site you want to check, which is what makes competitor files worth a look. Here is how to check robots.txt on any domain, in rough order of speed:

Two traps catch people here. Subdomains need their own file, so blog.example.com is not covered by the rules at example.com, and each protocol and port is treated as a separate host. And a file that only exists at www.example.com does nothing for visitors who land on example.com without the www, unless one redirects to the other.

Once you have the file open, reading it takes about a minute. The syntax is small on purpose. Every meaningful line is a field name, a colon, and a value, and the file is organised into groups that each start with a User-agent line. Here is an annotated file with every directive you are likely to meet:

# Group 1: rules for every crawler without its own group
User-agent: *
Disallow: /cart/
Disallow: /search
Disallow: /*?sort=
Allow: /cart/shipping-info

# Group 2: rules that apply to Googlebot only
User-agent: Googlebot
Disallow: /internal/

# Group 3: an AI training crawler, blocked entirely
User-agent: GPTBot
Disallow: /

# Applies to the whole file, not to any one group
Sitemap: https://example.com/sitemap.xml

Reading it line by line, that file says the following:

• A hash starts a comment. Everything after # on a line is ignored by crawlers, so comments cost nothing and are worth writing.
• User-agent names the crawler. An asterisk is the catch-all. A crawler that finds a group with its own name ignores the asterisk group completely, so Googlebot above obeys only Group 2 and is not blocked from /cart/.
• Disallow blocks a path prefix. Disallow: /search blocks /search, /search/, and /search-results, because the match is on the prefix and not the whole path.
• An asterisk inside a path is a wildcard. Disallow: /*?sort= catches every URL containing that query string, which is the standard way to stop faceted navigation from eating crawl budget.
• Allow carves out an exception. /cart/shipping-info stays crawlable even though the rest of /cart/ is blocked, because Allow wins over Disallow at equal specificity.
• An empty Disallow means allow everything. Disallow: with no value is the opposite of Disallow: /, and mixing the two up is a genuinely common error.
• Sitemap is file-wide. It is not tied to the group above it and can appear anywhere in the file, though the end is conventional.

What the file does not do is control indexing, and this is the single most misread part of the standard. A blocked URL can still appear in search results if other sites link to it, because Google can index a URL it has never crawled. If your goal is to keep a page out of results, use a noindex tag and leave the page crawlable so Google can actually see that tag.

Most sites need something close to one of these. Take the sample robots.txt file that matches your setup, adjust the paths and the sitemap URL to your own domain, then paste your site into the tester above to confirm the result before you deploy it. Every block below is complete robots.txt code you can copy as it stands, with nothing left out.

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap_index.xml

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /*?filter=
Disallow: /*?sort=
Disallow: /*&page=

Sitemap: https://example.com/sitemap.xml

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

User-agent: *
Disallow: /

Whichever one you start from, change example.com to your own domain and confirm the Sitemap line resolves to a live XML file. A sitemap pointing at a 404 is the most common defect in an otherwise healthy file.

robots.txt is not just a search-engine file. Every major social platform runs its own crawler to read your pages and build link previews, and those crawlers honour the same file. Get the rules wrong and your links share as bare URLs instead of rich cards.

How social crawlers read robots.txt

When someone pastes your link into a post, Facebook's external hit, X's Twitterbot, and LinkedIn's bot all fetch the page to pull the title, description, and og:image. If your robots.txt blocks those user-agents, the platform falls back to a plain text link with no preview, and your post performs measurably worse.

Why blocking them breaks link previews

A preview is built from the same page the crawler tries to fetch. Blocking the social bot is the technical equivalent of turning off your open graph tags. The fix is a small Allow rule for each social user-agent, which this tester can verify in the same pass it checks Googlebot.

Cross-platform consistency

The same og:title, og:description, and og:image tags power previews on Facebook, X, LinkedIn, and messaging apps. A robots.txt that allows the social crawlers on one platform but not another produces inconsistent, broken shares that undermine your technical SEO work across the board.

What this means for marketers

Before you run a campaign, test your robots.txt for the social bots as well as the search bots. The same migration that breaks Googlebot will often break link previews too. If you are managing AI crawler rules, pair this tester with the AI robots.txt generator to set GPTBot and Google-Extended rules correctly, then run a full site audit to confirm nothing else regressed.

A one-off check tells you the file is fine today. It says nothing about tomorrow. robots.txt is edited by developers, overwritten by plugin updates, regenerated by platform migrations, and reset by hosting changes, usually without anyone announcing it. Sites that get burned are rarely the ones that never checked. They are the ones that checked once, six months ago.

This is where a robots.txt analyzer earns its keep over a one-line verdict. Proper robots.txt analysis goes past pass or fail and asks what the file is actually doing to your crawl. When you analyze a robots.txt file properly, these are the questions worth answering:

• Rule coverage. How many URLs on your site does each Disallow rule really match? A rule written to block one folder often catches far more than intended once wildcards are involved.
• Crawl budget impact. Cross-reference the blocked patterns against your server logs. If crawlers are still hammering paths you meant to block, the rule is not matching the way you think it is.
• Orphaned rules. Disallow lines pointing at folders that were removed years ago add noise and hide the rules that matter.
• Sitemap health. The declared sitemap should return a 200 and list live URLs, not redirect or 404.
• Bot coverage drift. New AI crawlers appear every few months. A file written two years ago has no opinion about the crawlers hitting you now.

For robots.txt monitoring, pick a cadence and stick to it. Once a month is enough for a stable site. What matters more is testing at the moments when the file is most likely to change without warning: after a platform migration, a theme or plugin update, a hosting move, a CDN or edge-config change, and any deployment that touched routing or redirects. Save your current file somewhere outside the server, and you can diff it against the live version in seconds.

Because this robots.txt testing tool runs online with nothing to install and no account to create, re-verifying costs you about ten seconds. That is the whole argument for doing it often. Bookmark the page, verify your robots.txt after every deployment that touched infrastructure, and you will catch the accidental block in the window where it is still cheap to fix rather than a month later in a traffic report.

If the test came back with a 404, your site has no robots.txt at all. That is valid and not an emergency, since a missing file means everything is crawlable, but you are giving up the sitemap declaration and any control over AI training bots. Creating one takes a few minutes.

There are two routes. If you want to know how to write a robots.txt file by hand, start from one of the samples above, save it as a plain text file named robots.txt, and upload it to your web root. If you would rather make a robots.txt file online without touching the syntax, use an online robots.txt generator instead. Our AI robots.txt generator builds one from your CMS and your choices about which crawlers to allow, then hands you something ready to paste. Either way, come back and run it through the tester before you consider the job finished. Creating the file and verifying the file are two separate steps, and skipping the second is how a blocking rule slips through.

Three things to get right when you create the file. Save it as plain UTF-8 text, not a Word document or a file your editor has quietly given a .txt.txt extension. Upload it to the web root so it answers at yourdomain.com/robots.txt and not a subfolder. And confirm it returns a 200 with a content type of text/plain, because a platform that serves your 404 page in its place has effectively given you no file at all.

There is no shortage of robots checker tools out there, but most are either dated, locked behind a login, or give you a pass or fail with no explanation. This one is built by the SEOWebster team to be fast, transparent, and free, with the same access logic Google uses so you can trust the answer.

If a test turns up more than a simple fix, SEOWebster's technical SEO services cover crawl budget, indexing, and migration support end to end.