Knowledge base

AI crawler access checklist

Separate retrieval crawlers from training crawlers, then audit robots.txt, status codes and fetch results for pages that should be citable.

Start with access, not copy. Retrieval crawlers fetch pages while an answer is being written, while training crawlers collect text for future models. They follow separate rules. A robots.txt that blocks every bot with a wildcard removes your pages from the retrieval pool and leaves a competitor untouched. The first question is whether a citable page can be fetched at all.

Audit the declared rules line by line. Confirm that the homepage, the main service pages and the policy pages are allowed. Then fetch those URLs as a crawler would and record status codes, redirect chains and the text visible before any interaction. A login wall, a full-screen consent dialog or client-only rendering each empties the extractor hands.

In this product, access problems sit inside AI citability, which carries 25 of 100 points, alongside AI visibility at 30, search visibility at 20, technical foundation at 15 and brand trust at 10. The eleven information domains only organise the report; they never change the weights. The evidence table records each fact with its value and status.

Coverage is weight-based. At 0.80 or above the total score is published normally; between 0.60 and 0.79 it is published with a note that some dimensions lack data; below 0.60 only dimension scores and a gap list are shown. Fix access before rewriting copy, otherwise the rewrite cannot be verified.

Run the single-signal check first, then read an example report to see how the evidence table separates a pass from a gap. After each change, write down the date and what was edited, so the next run can explain which fix moved the result instead of guessing which change helped.