Alp Korpe Berlin · Marketing Ops
Web Tools · Selected work

IsVisible.ai

Free AI crawler visibility checker — tests whether GPTBot, ClaudeBot, and 11 others can reach your site.

Stack
  • Crawler Simulation
  • robots.txt & llms.txt Parsing
  • REST API
Links
Impact
  • Scores 100/100, Grade A, when run on this site
IsVisible.ai — Free AI crawler visibility checker — tests whether GPTBot, ClaudeBot, and 11 others can reach your site.

IsVisible.ai checks whether AI crawlers can actually reach a website. Paste in a URL and it tests against 13 crawlers and search bots: search engines (Googlebot, Bingbot), AI training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, meta-externalagent), and AI answer engines (ChatGPT-User, OAI-SearchBot, Claude-User, PerplexityBot). It reads the site’s robots.txt and llms.txt, sends live requests using each crawler’s real user agent, and merges the declared rules with what actually happens to produce a 0-100 visibility score and an agent-by-agent access map.

Why sites block AI crawlers without knowing it

The Hardal team has been building an AI visibility reporting product, one that shows companies which AI crawlers are actually visiting their site. Working on that, the same gap kept turning up: sites expected to show up when someone asked ChatGPT or Claude about them, with no idea they were blocking the exact crawlers that make that possible. Old robots.txt rules written for a pre-LLM web, a missing llms.txt, a WAF setting nobody had touched in years. None of it shows up unless someone actually simulates the crawlers and checks. IsVisible.ai is the free, standalone check for that specific gap.

Seeing it work

Run against this site, yigitalpkorpe.com scores 100 out of 100, Grade A: fully accessible to Google Search, Bing/Copilot, Google Gemini and AI Overviews training, GPTBot training, and the rest of the 13 crawlers it checks.

IsVisible.ai report for yigitalpkorpe.com: 100/100, Grade A

How the crawler simulation works

Getting the crawler simulation right took more tweaking than expected. Different bots honor different rules: some respect a wildcard Disallow, some only match their exact user-agent string, some ignore robots.txt for training purposes but respect it for live retrieval. Redirects add another layer: a crawler blocked on the final URL but allowed on a redirecting one produces a different result than checking either URL in isolation. How a site responds to a user agent it’s never seen before turned into its own category of edge case, since some servers default to blocking unfamiliar bots rather than allowing them.

API access

The tool is free with no signup for the standard check, and it also exposes an unauthenticated REST API for checking a site programmatically, useful for teams that want the same crawler-access check as part of a CI pipeline or a recurring monitor, rather than a one-off manual lookup.