Technical SEO
Indexability Checker
Enter a URL. We check the meta robots, the response headers, the canonical, and robots.txt, then tell you plainly whether the page can be indexed and cited.
The five gates a page passes before it can be cited
The check runs five tests and reports each one: the page returns a 200, there is no meta robots noindex, there is no noindex in the X-Robots-Tag response header, robots.txt allows the path, and the canonical points at this URL. Failing any of the first four means the page cannot be indexed. The canonical is a separate case: a page whose canonical names another URL can be indexed perfectly well, you have told the engine to credit somewhere else.
The header test is the one that catches what nobody sees. X-Robots-Tag lives in the HTTP response rather than the HTML, so it does not appear in view source and it survives every content edit. A noindex set at the CDN or in a framework config can sit on a page for months while the team rewrites copy trying to work out why it never ranked.
Indexable is not indexed. Passing all five means nothing on your side is stopping the engine, and whether it indexes the page is still its decision. Crawled and currently not indexed in Search Console is that decision, and it is a judgement about the page being thin or near-duplicate rather than a technical block.
The five checks, what failure looks like, and the fix
| Check | What a failure looks like | The fix |
|---|---|---|
| Page responds with 200 | Any other status. A 404 or 500 has nothing to index, and a 301 means the URL you tested is not the one that gets indexed. | Test the final URL in the chain. If the page should exist, fix the status before touching anything else on this list. |
| No meta robots noindex | A noindex tag in the head, usually left over from staging or switched on by a CMS toggle nobody remembers. | Remove the tag or the setting behind it, then request indexing so you are not waiting for the next natural crawl. |
| No X-Robots-Tag noindex | A noindex in the response header, invisible in the page source. | Check your CDN rules, hosting config, and framework headers. Run curl -I on the URL to see what the server sends. |
| Allowed in robots.txt | The path matches a Disallow rule, so crawlers never fetch the content. | Remove the rule if the page should be public. If it should stay out of the index, allow the crawl and serve a noindex instead. |
| Canonical points to this page | The canonical names a different URL, so this one gets folded into that one. | Fix it when the two pages are genuinely different. Leave it when it is intentional, as with a paginated or parameterised copy. |
A page can pass all five and still be invisible to an answer engine, because robots.txt can allow Googlebot and block GPTBot in the same file. The result panel counts how many AI crawlers are blocked for that reason, and the AI Crawler Checker names them.
Indexability questions
The page is indexable but Search Console says crawled and currently not indexed. Why?
Google fetched the page and decided it was not worth storing. That is almost always about value rather than technology: the page is thin, it repeats another page of yours closely, or nothing internal links to it so it reads as unimportant. Adding substance and internal links moves it. Pressing request indexing again on an unchanged page rarely does.
Should I use noindex or a robots.txt disallow to keep a page out?
noindex, and the crawler has to be allowed to fetch the page to read it. Doing both at once is the classic mistake: robots.txt blocks the fetch, the noindex is never seen, and the URL can still be listed from links pointing at it with no description. Allow the crawl, serve the noindex, and remove the disallow.
Does a canonical guarantee which URL gets indexed?
No, it is a strong hint rather than an instruction. Engines can pick a different URL, particularly if the two pages differ in content or if your internal links and sitemap point somewhere other than the canonical. Keep all three signals consistent and the hint is usually honoured.
Does this check cover AI engines as well as search?
Indexability is the shared precondition for both, so a page failing here is invisible everywhere. The second gate is specific to AI: an engine can only cite what its own crawler was allowed to fetch, which is a different set of rules in the same robots.txt file. The blocked crawler count in the result flags that, and the AI Crawler Checker breaks it down.
Previews the Technical Agent
Check one page here. Rankfastt watches every page for a stray noindex and warns you before it quietly drops you from results.