https://spider.es/

{
  "url": "https://spider.es/",
  "timestamp": 1771155843241,
  "fetch": {
    "mode": "simple",
    "timeMs": 97,
    "htmlSizeBytes": 39107,
    "supportsMarkdown": false,
    "statusCode": 200
  },
  "extraction": {
    "title": "Spider.es · Lightning-fast crawler & bot access checker",
    "excerpt": "Audit, in seconds, whether search engines, AI crawlers, SEO tools and scrapers can reach your pages—and pinpoint the directive that stops them.",
    "byline": null,
    "siteName": "spider.es",
    "lang": "en",
    "contentLength": 1866,
    "metadata": {
      "description": "Audit, in seconds, whether search engines, AI crawlers, SEO tools and scrapers can reach your pages—and pinpoint the directive that stops them.",
      "ogTitle": "Spider.es · Lightning-fast crawler & bot access checker",
      "ogDescription": "Audit, in seconds, whether search engines, AI crawlers, SEO tools and scrapers can reach your pages—and pinpoint the directive that stops them.",
      "ogImage": null,
      "ogType": "website",
      "canonical": "https://spider.es/",
      "lang": "en",
      "schemas": [],
      "robotsMeta": null,
      "author": null,
      "generator": null
    }
  },
  "markdown": "Lightning-fast crawler visibility assistant for technical SEOs.\n\n## 🤖 Instant Crawler Checker\n\nPaste any URL and get an immediate verdict on flagship search, AI, SEO and monitoring bots—from Googlebot and Bingbot to GPTBot, Ahrefs and beyond—so you know exactly who can reach your pages. [Explore supported crawlers & user agents](https://spider.es/faq/#supported).\n\n## 💸 Avoid Costly SEO Mistakes\n\nMisconfigured directives drain organic reach. Verify your crawl rules, keep mission-critical assets open and fence off unwanted scrapers. [Boost SEO visibility](https://spider.es/faq/#seo-visibility) • [Troubleshoot common problems](https://spider.es/faq/#issues).\n\n## 🧩 How Spider Works\n\nSpider cross-references robots.txt directives, meta robots tags and X-Robots-Tag headers to produce a per-bot decision log you can action immediately. [See Spider's methodology](https://spider.es/faq/#how-it-works).\n\n## Why This Report Matters\n\nThe report confirms whether search engines, AI services and scrapers can reach your content—or if something is unintentionally blocked.\n\n-   **Protect visibility:** verify Google, Bing and other engines aren't excluded by stray robots, meta or header rules.\n-   **Control AI usage:** check that ChatGPT, Claude, Perplexity and fellow LLM crawlers respect your boundaries.\n-   **Demonstrate enforcement:** explicit blocks document your policy for compliance, licensing or legal discussions.\n-   **Spend crawl budget wisely:** trim noisy bots so search engines focus on revenue-driving pages.\n\nWhether you welcome or reject bots, Spider.es keeps your crawl setup predictable.\n",
  "fullPageMarkdown": "Spider.es · Lightning-fast crawler & bot access checker\n\n# [spider.es](https://spider.es/)\n\n☀️ 🌙 Language 🇬🇧 English 🇪🇸 Español 🇫🇷 Français 🇵🇹 Português 🇮🇹 Italiano 🇩🇪 Deutsch 🇰🇷 한국어 🇯🇵 日本語\n\n URL to analyze\n\n Analyze\n\nResults Crawler report Technical details\n\n### Domain overview\n\n### robots.txt\n\n**Status** \\-\n\n**Origin** \\-\n\n**Sitemaps** \\-\n\n#### Additional files\n\n### Meta robots\n\n### Headers\n\nLightning-fast crawler visibility assistant for technical SEOs.\n\n## 🤖 Instant Crawler Checker\n\nPaste any URL and get an immediate verdict on flagship search, AI, SEO and monitoring bots—from Googlebot and Bingbot to GPTBot, Ahrefs and beyond—so you know exactly who can reach your pages. [Explore supported crawlers & user agents](https://spider.es/faq/#supported).\n\n## 💸 Avoid Costly SEO Mistakes\n\nMisconfigured directives drain organic reach. Verify your crawl rules, keep mission-critical assets open and fence off unwanted scrapers. [Boost SEO visibility](https://spider.es/faq/#seo-visibility) • [Troubleshoot common problems](https://spider.es/faq/#issues).\n\n## 🧩 How Spider Works\n\nSpider cross-references robots.txt directives, meta robots tags and X-Robots-Tag headers to produce a per-bot decision log you can action immediately. [See Spider's methodology](https://spider.es/faq/#how-it-works).\n\n## Why This Report Matters\n\nThe report confirms whether search engines, AI services and scrapers can reach your content—or if something is unintentionally blocked.\n\n-   **Protect visibility:** verify Google, Bing and other engines aren't excluded by stray robots, meta or header rules.\n-   **Control AI usage:** check that ChatGPT, Claude, Perplexity and fellow LLM crawlers respect your boundaries.\n-   **Demonstrate enforcement:** explicit blocks document your policy for compliance, licensing or legal discussions.\n-   **Spend crawl budget wisely:** trim noisy bots so search engines focus on revenue-driving pages.\n\nWhether you welcome or reject bots, Spider.es keeps your crawl setup predictable.\n",
  "markdownStats": {
    "images": 0,
    "links": 4,
    "tables": 0,
    "codeBlocks": 0,
    "headings": 4
  },
  "tokens": {
    "htmlTokens": 9599,
    "markdownTokens": 351,
    "reduction": 9248,
    "reductionPercent": 96
  },
  "score": {
    "score": 77,
    "grade": "B",
    "dimensions": {
      "semanticHtml": {
        "score": 96,
        "weight": 20,
        "grade": "A",
        "checks": {
          "uses_article_or_main": {
            "score": 100,
            "weight": 20,
            "details": "Has <main>"
          },
          "proper_heading_hierarchy": {
            "score": 85,
            "weight": 25,
            "details": "1 heading level skip(s)"
          },
          "semantic_elements": {
            "score": 100,
            "weight": 20,
            "details": "10 semantic elements, 21 divs (ratio: 32%)"
          },
          "meaningful_alt_texts": {
            "score": 100,
            "weight": 15,
            "details": "No images found"
          },
          "low_div_nesting": {
            "score": 100,
            "weight": 20,
            "details": "Avg div depth: 1.8, max: 3"
          }
        }
      },
      "contentEfficiency": {
        "score": 70,
        "weight": 25,
        "grade": "C",
        "checks": {
          "token_reduction_ratio": {
            "score": 100,
            "weight": 40,
            "details": "96% token reduction (HTML→Markdown)"
          },
          "content_to_noise_ratio": {
            "score": 0,
            "weight": 30,
            "details": "Content ratio: 4.8% (1866 content chars / 39107 HTML bytes)"
          },
          "minimal_inline_styles": {
            "score": 100,
            "weight": 15,
            "details": "0/144 elements with inline styles (0.0%)"
          },
          "reasonable_page_weight": {
            "score": 100,
            "weight": 15,
            "details": "HTML size: 38KB"
          }
        }
      },
      "aiDiscoverability": {
        "score": 63,
        "weight": 25,
        "grade": "C",
        "checks": {
          "has_llms_txt": {
            "score": 50,
            "weight": 25,
            "details": "llms.txt exists but appears empty or invalid"
          },
          "has_robots_txt": {
            "score": 100,
            "weight": 15,
            "details": "robots.txt exists"
          },
          "robots_allows_ai_bots": {
            "score": 100,
            "weight": 20,
            "details": "All major AI bots allowed"
          },
          "has_sitemap": {
            "score": 100,
            "weight": 15,
            "details": "Sitemap found"
          },
          "supports_markdown_negotiation": {
            "score": 0,
            "weight": 15,
            "details": "No markdown content negotiation"
          },
          "has_content_signals": {
            "score": 0,
            "weight": 10,
            "details": "No Content-Signal header"
          }
        }
      },
      "structuredData": {
        "score": 62,
        "weight": 15,
        "grade": "C",
        "checks": {
          "has_schema_org": {
            "score": 0,
            "weight": 30,
            "details": "No JSON-LD / Schema.org found"
          },
          "has_open_graph": {
            "score": 67,
            "weight": 25,
            "details": "2/3 OG tags present"
          },
          "has_meta_description": {
            "score": 100,
            "weight": 20,
            "details": "Meta description: 143 chars"
          },
          "has_canonical_url": {
            "score": 100,
            "weight": 15,
            "details": "Canonical URL present"
          },
          "has_lang_attribute": {
            "score": 100,
            "weight": 10,
            "details": "lang=\"en\""
          }
        }
      },
      "accessibility": {
        "score": 100,
        "weight": 15,
        "grade": "A",
        "checks": {
          "content_without_js": {
            "score": 100,
            "weight": 40,
            "details": "Content available without JavaScript"
          },
          "reasonable_page_size": {
            "score": 100,
            "weight": 30,
            "details": "Page size: 38KB"
          },
          "fast_content_position": {
            "score": 100,
            "weight": 30,
            "details": "Main content starts at 12% of HTML"
          }
        }
      }
    }
  },
  "recommendations": [
    {
      "id": "improve_content_ratio",
      "priority": "critical",
      "category": "contentEfficiency",
      "titleKey": "rec.improve_content_ratio.title",
      "descriptionKey": "rec.improve_content_ratio.description",
      "howToKey": "rec.improve_content_ratio.howto",
      "effort": "moderate",
      "estimatedImpact": 6,
      "checkScore": 0,
      "checkDetails": "Content ratio: 4.8% (1866 content chars / 39107 HTML bytes)"
    },
    {
      "id": "add_markdown_negotiation",
      "priority": "critical",
      "category": "aiDiscoverability",
      "titleKey": "rec.add_markdown_negotiation.title",
      "descriptionKey": "rec.add_markdown_negotiation.description",
      "howToKey": "rec.add_markdown_negotiation.howto",
      "effort": "significant",
      "estimatedImpact": 4,
      "checkScore": 0,
      "checkDetails": "No markdown content negotiation"
    },
    {
      "id": "add_content_signals",
      "priority": "critical",
      "category": "aiDiscoverability",
      "titleKey": "rec.add_content_signals.title",
      "descriptionKey": "rec.add_content_signals.description",
      "howToKey": "rec.add_content_signals.howto",
      "effort": "moderate",
      "estimatedImpact": 3,
      "checkScore": 0,
      "checkDetails": "No Content-Signal header"
    },
    {
      "id": "add_schema_org",
      "priority": "high",
      "category": "structuredData",
      "titleKey": "rec.add_schema_org.title",
      "descriptionKey": "rec.add_schema_org.description",
      "howToKey": "rec.add_schema_org.howto",
      "effort": "moderate",
      "estimatedImpact": 6,
      "checkScore": 0,
      "checkDetails": "No JSON-LD / Schema.org found"
    },
    {
      "id": "add_open_graph",
      "priority": "medium",
      "category": "structuredData",
      "titleKey": "rec.add_open_graph.title",
      "descriptionKey": "rec.add_open_graph.description",
      "howToKey": "rec.add_open_graph.howto",
      "effort": "quick-win",
      "estimatedImpact": 4,
      "checkScore": 67,
      "checkDetails": "2/3 OG tags present"
    }
  ],
  "llmsTxtPreview": "# spider.es\n\n> Audit, in seconds, whether search engines, AI crawlers, SEO tools and scrapers can reach your pages—and pinpoint the directive that stops them.\n\n## Documentation\n- [FAQ](https://spider.es/faq)\n- [Explore supported crawlers & user agents](https://spider.es/faq/)\n\n## Main\n- [Spider.es · Lightning-fast crawler & bot access checker](https://spider.es/): Audit, in seconds, whether search engines, AI crawlers, SEO tools and scrapers can reach your pages—and pinpoint the di…\n- [About us](https://spider.es/about)\n- [History](https://spider.es/history)\n\n## Blog\n- [Blog](https://spider.es/blog)\n\n## Support\n- [FAQ](https://spider.es/faq)\n\n",
  "llmsTxtExisting": "<!DOCTYPE html>\n<html lang=\"en\">\n<head>\n    <meta charset=\"UTF-8\">\n    <meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0\">\n    <title>Spider.es · Lightning-fast crawler &amp; bot access checker</title>\n        <link rel=\"preload\" href=\"https://spider.es/css/styles.min.css?v=1758571656\" as=\"style\" fetchpriority=\"high\">\n    <link rel=\"stylesheet\" href=\"https://spider.es/css/styles.min.css?v=1758571656\">\n    <noscript><link rel=\"stylesheet\" href=\"https://spider.es/css/styles.min.css?v=1758571656\"></noscript>\n        <link rel=\"canonical\" href=\"https://spider.es/\">\n                    <link rel=\"alternate\" hreflang=\"en\" href=\"https://spider.es/\">\n                    <link rel=\"alternate\" hreflang=\"es\" href=\"https://spider.es/es\">\n                    <link rel=\"alternate\" hreflang=\"fr\" href=\"https://spider.es/fr\">\n                    <link rel=\"alternate\" hreflang=\"pt\" href=\"https://spider.es/pt\">\n                    <link rel=\"alternate\" hreflang=\"it\" href=\"https://spider.es/it\">\n                    <link rel=\"alternate\" hreflang=\"de\" href=\"https://spider.es/de\">\n                    <link rel=\"alternate\" hreflang=\"ko\" href=\"https://spider.es/ko\">\n                    <link rel=\"alternate\" hreflang=\"ja\" href=\"https://spider.es/ja\">\n                <meta name=\"description\" content=\"Audit, in seconds, whether search engines, AI crawlers, SEO tools and scrapers can reach your pages—and pinpoint the directive that stops them.\">\n        <meta property=\"og:type\" content=\"website\">\n    <meta property=\"og:title\" content=\"Spider.es · Lightning-fast crawler &amp; bot access checker\">\n            <meta property=\"og:description\" content=\"Audit, in seconds, whether search engines, AI crawlers, SEO tools and scrapers can reach your pages—and pinpoint the directive that stops them.\">\n        <meta property=\"og:url\" content=\"https://spider.es/\">\n    <meta property=\"og:site_name\" content=\"spider.es\">\n        <meta name=\"twitter:card\" content=\"summary\">\n    <meta name=\"twitter:title\" content=\"Spider.es · Lightning-fast crawler &amp; bot access checker\">\n            <meta name=\"twitter:description\" content=\"Audit, in seconds, whether search engines, AI crawlers, SEO tools and scrapers can reach your pages—and pinpoint the directive that stops them.\">\n                    <link rel=\"preconnect\" href=\"https://a.colorvivo.com\" crossorigin>\n<link rel=\"preconnect\" href=\"https://pagead2.googlesyndication.com\" crossorigin>\n<link rel=\"dns-prefetch\" href=\"//pagead2.googlesyndication.com\">\n<link rel=\"preload\" href=\"https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js?client=ca-pub-9314112849612183\" as=\"script\" crossorigin=\"anonymous\" fetchpriority=\"low\">\n<script>\n(function () {\n    const loadAsyncScript = function (src, attributes) {\n        const el = document.createElement('script');\n        el.src = src;\n        el.async = true;\n        if (attributes && typeof attributes === 'object') {\n            Object.keys(attributes).forEach(function (key) {\n                if (attributes[key] === true) {\n                    el.setAttribute(key, '');\n                } else if (attributes[key] !== false && attributes[key] !== null && attributes[key] !== undefined) {\n                    el.setAttribute(key, attributes[key]);\n                }\n            });\n        }\n        (document.head || document.body || document.documentElement).appendChild(el);\n    };\n\n    const triggerLazyScripts = function () {\n        if (triggerLazyScripts.loaded) {\n            return;\n        }\n        triggerLazyScripts.loaded = true;\n        loadAsyncScript('https://a.colorvivo.com/pixel/UflN9Ti7BxkissoV');\n        loadAsyncScript('https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js?client=ca-pub-9314112849612183', {\n            crossorigin: 'anonymous'\n        });\n    };\n\n    const scheduleLazyScripts = function () {\n        if ('requestIdleCallback' in window) {\n            window.requestIdleCallback(triggerLazyScripts, { timeout: 2000 });\n        } else {\n            window.setTimeout(triggerLazyScripts, 1500);\n        }\n    };\n\n    const readyState = document.readyState;\n    if (readyState === 'complete') {\n        window.setTimeout(triggerLazyScripts, 100);\n    } else if (readyState === 'interactive') {\n        scheduleLazyScripts();\n    } else {\n        document.addEventListener('DOMContentLoaded', scheduleLazyScripts, { once: true });\n    }\n\n    window.addEventListener('scroll', triggerLazyScripts, { once: true, passive: true });\n    window.addEventListener('pointerdown', triggerLazyScripts, { once: true });\n    scheduleLazyScripts();\n})();\n</script>\n    </head>\n<body data-theme=\"light\">\n<div class=\"page\">\n    <header class=\"header\">\n        <div class=\"header-top\">\n            <div class=\"header-brand\">\n                                <h1>\n                    <a class=\"brand-link\" href=\"https://spider.es/\">\n                                                <span class=\"brand-text\">spider.es</span>\n                    </a>\n                </h1>\n            </div>\n            <nav id=\"siteNav\" class=\"main-nav\" aria-label=\"Site\">\n                <a href=\"https://spider.es/\" class=\"nav-link active\" data-i18n=\"nav_analyzer\">Analyzer</a>\n                <a href=\"https://spider.es/history\" class=\"nav-link\" data-i18n=\"nav_history\">History</a>\n                <a href=\"https://spider.es/blog\" class=\"nav-link\" data-i18n=\"nav_blog\">Blog</a>\n                <a href=\"https://spider.es/about\" class=\"nav-link\" data-i18n=\"nav_about\">About us</a>\n                <a href=\"https://spider.es/faq\" class=\"nav-link\" data-i18n=\"nav_faq\">FAQ</a>\n            </nav>\n            <div class=\"header-actions\">\n                <button type=\"button\" id=\"themeToggle\" class=\"theme-toggle-btn\" data-mode=\"light\" aria-pressed=\"false\" aria-label=\"Toggle dark mode\">\n                    <span class=\"theme-icon theme-icon-sun\" aria-hidden=\"true\">☀️</span>\n                    <span class=\"theme-icon theme-icon-moon\" aria-hidden=\"true\">🌙</span>\n                    <span class=\"theme-toggle-thumb\" aria-hidden=\"true\"></span>\n                </button>\n                <label for=\"localeSelect\" class=\"sr-only\">Language</label>\n                <select id=\"localeSelect\" class=\"locale-select\">\n                                                                    <option value=\"https://spider.es/\"\n                            data-label-full=\"🇬🇧 English\"\n                            data-label-short=\"🇬🇧 EN\"\n                             selected>🇬🇧 English</option>\n                                                                    <option value=\"https://spider.es/es\"\n                            data-label-full=\"🇪🇸 Español\"\n                            data-label-short=\"🇪🇸 ES\"\n                            >🇪🇸 Español</option>\n                                                                    <option value=\"https://spider.es/fr\"\n                            data-label-full=\"🇫🇷 Français\"\n                            data-label-short=\"🇫🇷 FR\"\n                            >🇫🇷 Français</option>\n                                                                    <option value=\"https://spider.es/pt\"\n                            data-label-full=\"🇵🇹 Português\"\n                            data-label-short=\"🇵🇹 PT\"\n                            >🇵🇹 Português</option>\n                                                                    <option value=\"https://spider.es/it\"\n                            data-label-full=\"🇮🇹 Italiano\"\n                            data-label-short=\"🇮🇹 IT\"\n                            >🇮🇹 Italiano</option>\n                                                                    <option value=\"https://spider.es/de\"\n                            data-label-full=\"🇩🇪 Deutsch\"\n                            data-label-short=\"🇩🇪 DE\"\n                            >🇩🇪 Deutsch</option>\n                                                                    <option value=\"https://spider.es/ko\"\n                            data-label-full=\"🇰🇷 한국어\"\n                            data-label-short=\"🇰🇷 KO\"\n                            >🇰🇷 한국어</option>\n                                                                    <option value=\"https://spider.es/ja\"\n                            data-label-full=\"🇯🇵 日本語\"\n                            data-label-short=\"🇯🇵 JA\"\n                            >🇯🇵 日本語</option>\n                                    </select>\n            </div>\n            <button type=\"button\" class=\"menu-toggle\" id=\"menuToggle\" aria-label=\"Toggle navigation\" aria-controls=\"siteNav\" aria-expanded=\"false\">\n                <span class=\"menu-icon\" aria-hidden=\"true\">\n                    <span></span>\n                    <span></span>\n                    <span></span>\n                </span>\n            </button>\n        </div>\n    </header>\n\n    <main>\n        <section class=\"card\">\n            <form id=\"analyzeForm\" class=\"analyze-form\" novalidate>\n                <input type=\"hidden\" name=\"lang\" value=\"en\">\n                <label for=\"urlInput\" class=\"form-label\">URL to analyze</label>\n                <div class=\"form-inline\">\n                    <input type=\"text\" id=\"urlInput\" name=\"url\" required placeholder=\"Enter the domain (e.g. example.com)\" value=\"\" autocomplete=\"off\" inputmode=\"url\" autocapitalize=\"none\" spellcheck=\"false\">\n                    <button type=\"submit\" id=\"analyzeBtn\" data-i18n=\"analyze\" disabled>Analyze</button>\n                </div>\n            </form>\n            <div id=\"formFeedback\" role=\"status\" aria-live=\"polite\"></div>\n        </section>\n\n        <section class=\"card tabs\" id=\"resultsSection\" hidden>\n            <div class=\"tabs-nav\" role=\"tablist\">\n                <button class=\"tab-button active\" data-tab=\"results\" role=\"tab\" data-i18n=\"tab_results\">Results</button>\n                <button class=\"tab-button\" data-tab=\"crawlers\" role=\"tab\" data-i18n=\"tab_crawlers\">Crawler report</button>\n                <button class=\"tab-button\" data-tab=\"technical\" role=\"tab\" data-i18n=\"tab_technical\">Technical details</button>\n                            </div>\n            <div class=\"tab-content active\" id=\"tab-results\" role=\"tabpanel\">\n                <div id=\"resultsContainer\">\n                    <div id=\"resultsMeta\"></div>\n                    <div class=\"summary-cards\" id=\"resultsSummaryCards\"></div>\n                </div>\n            </div>\n            <div class=\"tab-content\" id=\"tab-crawlers\" role=\"tabpanel\">\n                <div id=\"crawlersContainer\"></div>\n            </div>\n            <div class=\"tab-content\" id=\"tab-technical\" role=\"tabpanel\">\n                <h3 data-i18n=\"technical_domain_heading\">Domain overview</h3>\n                <div class=\"technical-summary\" id=\"technicalSummaryCards\"></div>\n                <h3 data-i18n=\"technical_robots\">robots.txt</h3>\n                <pre id=\"robotsRaw\" class=\"code-block\"></pre>\n                <div class=\"inline-meta\">\n                    <div>\n                        <strong data-i18n=\"technical_status\">Status</strong>\n                        <span id=\"robotsStatus\">-</span>\n                    </div>\n                    <div>\n                        <strong data-i18n=\"technical_origin\">Origin</strong>\n                        <span id=\"robotsUrl\">-</span>\n                    </div>\n                    <div>\n                        <strong data-i18n=\"technical_sitemaps\">Sitemaps</strong>\n                        <span id=\"sitemapsList\">-</span>\n                    </div>\n                </div>\n                <div class=\"technical-assets\">\n                    <h4 data-i18n=\"technical_additional_files\">Additional files</h4>\n                    <ul id=\"technicalFilesList\" class=\"technical-files-list\"></ul>\n                </div>\n                <h3 data-i18n=\"technical_meta\">Meta robots</h3>\n                <pre id=\"metaInfo\" class=\"code-block\"></pre>\n                <h3 data-i18n=\"technical_headers\">Headers</h3>\n                <pre id=\"headersInfo\" class=\"code-block\"></pre>\n            </div>\n                    </section>\n\n                    <div id=\"homeIntro\">\n                <p class=\"home-tagline\" data-i18n=\"home_tagline\">Lightning-fast crawler visibility assistant for technical SEOs.</p>\n\n                <section class=\"card home-section\">\n                    <h2 data-i18n=\"home_section_instant_title\">🤖 Instant Crawler Checker</h2>\n                    <p>Paste any URL and get an immediate verdict on flagship search, AI, SEO and monitoring bots—from Googlebot and Bingbot to GPTBot, Ahrefs and beyond—so you know exactly who can reach your pages. <a href=\"/faq/#supported\">Explore supported crawlers &amp; user agents</a>.</p>\n                </section>\n\n                <section class=\"card home-section\">\n                    <h2 data-i18n=\"home_section_seo_title\">💸 Avoid Costly SEO Mistakes</h2>\n                    <p>Misconfigured directives drain organic reach. Verify your crawl rules, keep mission-critical assets open and fence off unwanted scrapers. <a href=\"/faq/#seo-visibility\">Boost SEO visibility</a> • <a href=\"/faq/#issues\">Troubleshoot common problems</a>.</p>\n                </section>\n\n                <section class=\"card home-section\" id=\"how-it-works\">\n                    <h2 data-i18n=\"home_section_how_title\">🧩 How Spider Works</h2>\n                    <p>Spider cross-references robots.txt directives, meta robots tags and X-Robots-Tag headers to produce a per-bot decision log you can action immediately. <a href=\"/faq/#how-it-works\">See Spider's methodology</a>.</p>\n                </section>\n\n                <section class=\"card home-section\">\n                    <h2 data-i18n=\"home_section_why_title\">Why This Report Matters</h2>\n                    <p>The report confirms whether search engines, AI services and scrapers can reach your content—or if something is unintentionally blocked.</p>\n                    <ul><li><strong>Protect visibility:</strong> verify Google, Bing and other engines aren't excluded by stray robots, meta or header rules.</li><li><strong>Control AI usage:</strong> check that ChatGPT, Claude, Perplexity and fellow LLM crawlers respect your boundaries.</li><li><strong>Demonstrate enforcement:</strong> explicit blocks document your policy for compliance, licensing or legal discussions.</li><li><strong>Spend crawl budget wisely:</strong> trim noisy bots so search engines focus on revenue-driving pages.</li></ul>                    <p>Whether you welcome or reject bots, Spider.es keeps your crawl setup predictable.</p>\n                </section>\n            </div>\n        \n        \n            </main>\n\n        <footer class=\"footer\">\n        <p>© 1995-2025 Spider.es by <a href=\"https://colorvivo.com\" target=\"_blank\" rel=\"noopener noreferrer\">Color Vivo Internet</a> - Learn more about the Spider service.</p>\n        <p>Project in development, API activation pending. Hosted on <a href=\"https://www.stackscale.com\" target=\"_blank\" rel=\"noopener noreferrer\">Stackscale</a>'s cloud infrastructure.</p>\n        <p>Made with ❤ from Madrid and Herencia (Ciudad Real) - Spain.</p>\n    </footer>\n</div>\n\n<script>\n    window.__APP__ = {\n        lang: \"en\",\n        translations: {\"app_title\":\"Search Engines Checker\",\"brand_name\":\"Spider.es\",\"seo_service_suffix\":\"Spider.es · Lightning-fast crawler access checker\",\"seo_home_title\":\"Spider.es · Lightning-fast crawler \\u0026 bot access checker\",\"seo_home_description\":\"Audit, in seconds, whether search engines, AI crawlers, SEO tools and scrapers can reach your pages—and pinpoint the directive that stops them.\",\"seo_history_title\":\"Recent crawler access checks · Spider.es\",\"seo_history_description\":\"Browse the latest domains analyzed and reopen their crawler access reports in one click.\",\"seo_domain_description\":\"Instant crawler access report for {domain}. See which search engines, AI bots and scrapers are allowed or blocked with {service}.\",\"seo_faq_title\":\"Spider.es crawler access FAQ\",\"seo_faq_description\":\"Answers to frequent questions about Googlebot, AI crawlers, robots.txt and technical SEO using Spider.es reports.\",\"language_self\":\"English\",\"language_flag\":\"🇬🇧\",\"dark_mode\":\"Dark mode\",\"url_label\":\"URL to analyze\",\"url_placeholder\":\"Enter the domain (e.g. example.com)\",\"bots_legend\":\"Bots to check\",\"bot_type_unknown\":\"n/a\",\"select_all\":\"Select all\",\"select_none\":\"Deselect all\",\"analyze\":\"Analyze\",\"scope_site\":\"Evaluate entire site (/)\",\"scope_path\":\"Analyze only this path\",\"tab_results\":\"Results\",\"tab_technical\":\"Technical details\",\"tab_export\":\"Export\",\"tab_crawlers\":\"Crawler report\",\"tab_history\":\"History\",\"technical_robots\":\"robots.txt\",\"technical_status\":\"Status\",\"technical_origin\":\"Origin\",\"technical_sitemaps\":\"Sitemaps\",\"technical_favicon_heading\":\"Favicon\",\"technical_additional_files\":\"Additional files\",\"technical_no_favicon\":\"No favicon detected.\",\"technical_special_files_none\":\"No additional files detected.\",\"technical_favicon_alt\":\"Site favicon\",\"technical_domain_heading\":\"Domain overview\",\"technical_domain_name\":\"Domain\",\"technical_domain_ips\":\"IP addresses\",\"technical_domain_nameservers\":\"Name servers\",\"technical_domain_registered\":\"Registered on {date} ({years} years)\",\"technical_domain_registered_unknown\":\"Registration date unavailable.\",\"technical_domain_registrar\":\"Registrar\",\"technical_domain_updated\":\"Last updated\",\"technical_domain_expires\":\"Expires on\",\"summary_show_details\":\"Show details\",\"file_llms\":\"llms.txt\",\"file_humans\":\"humans.txt\",\"file_security\":\"security.txt\",\"file_ads\":\"ads.txt\",\"file_manifest\":\"manifest.json\",\"technical_meta\":\"Meta robots\",\"technical_headers\":\"Headers\",\"download_json\":\"Download JSON\",\"copy_markdown\":\"Copy as Markdown\",\"nav_analyzer\":\"Analyzer\",\"nav_history\":\"History\",\"nav_about\":\"About us\",\"nav_faq\":\"FAQ\",\"nav_blog\":\"Blog\",\"nav_menu_toggle\":\"Toggle navigation\",\"toggle_dark_mode\":\"Toggle dark mode\",\"locale_label\":\"Language\",\"footer_notice\":\"Configurable by editing config/bots.json. MIT License.\",\"history_slug\":\"history\",\"blog_slug\":\"blog\",\"analysis_enter_url\":\"Enter a valid URL.\",\"analysis_in_progress\":\"Analyzing...\",\"analysis_complete\":\"Analysis complete.\",\"analysis_network_error\":\"Network error during analysis.\",\"analysis_timeout\":\"Analysis timed out. Please try again.\",\"mechanisms_none\":\"No specific mechanisms\",\"notes_default\":\"No additional notes.\",\"documentation_link\":\"Documentation\",\"history_heading\":\"Recent searches\",\"history_empty\":\"No searches recorded yet.\",\"history_prev\":\"Previous\",\"history_next\":\"Next\",\"history_page\":\"Page {page} of {total}\",\"history_load_error\":\"Unable to load history.\",\"markdown_table_header\":\"| Bot | Result | Mechanisms | Notes |\",\"markdown_table_separator\":\"| --- | --- | --- | --- |\",\"robots_no_content\":\"(no content)\",\"robots_no_headers\":\"No headers fetched.\",\"sitemaps_not_declared\":\"Not declared\",\"export_copy_success\":\"Copied\",\"export_copy_action\":\"Copy as Markdown\",\"export_copy_fail\":\"Copy failed\",\"server_error_method_not_allowed\":\"Method not allowed\",\"server_error_rate_limit\":\"Too many requests. Please try again later.\",\"server_error_rate_internal\":\"Internal rate limit error\",\"server_error_invalid_payload\":\"Invalid payload received.\",\"server_error_invalid_url\":\"Invalid URL. Must start with http:// or https://\",\"server_error_invalid_domain\":\"Invalid domain name. Please use a valid domain with an extension.\",\"server_error_no_bots\":\"No valid bots selected.\",\"server_error_processing\":\"Unable to process the request\",\"robots_note_empty\":\"robots.txt is empty: treated as allowed.\",\"robots_note_inaccessible\":\"robots.txt unreachable: {error}\",\"robots_note_server_error\":\"robots.txt returned a server error. Treated as allowed unless meta/headers override.\",\"robots_note_not_found\":\"robots.txt not found (404). Treated as allowed unless meta/headers override.\",\"robots_note_client_error\":\"robots.txt inaccessible (HTTP {status}). Treated as allowed unless meta/headers override.\",\"note_noindex_source\":\"Noindex detected in {source}.\",\"note_rule_applies\":\"{type} rule applies: {line}\",\"history_api_error\":\"History service error\",\"rule_show\":\"Show rule\",\"rule_hide\":\"Hide rule\",\"results_empty\":\"No bots returned in the report.\",\"matched_rule_explanation\":\"Matches {path} (rule {type})\",\"rule_type_allow\":\"Allow\",\"rule_type_disallow\":\"Disallow\",\"mechanism_header\":\"Header\",\"mechanism_meta\":\"Meta\",\"mechanism_robots\":\"robots.txt\",\"history_allowed\":\"Allowed\",\"history_disallowed\":\"Disallowed\",\"history_noindexed\":\"Noindexed\",\"history_scope_site\":\"Site\",\"history_scope_path\":\"Path-specific\",\"category_search_engines\":\"Search Engines\",\"category_ai_bots\":\"AI Bots\",\"category_social_bots\":\"Social Bots\",\"category_seo_tools\":\"SEO Tools\",\"category_scrapers\":\"Scrapers\",\"category_cloud_services\":\"Cloud Services\",\"category_google_bots\":\"Google Specialized Bots\",\"category_other_agents\":\"Other Agents\",\"category_summary_full\":\"{category} are fully allowed to access the website URL.\",\"category_summary_partial\":\"{category} have restrictions: {disallowed} disallowed, {noindexed} noindexed.\",\"category_percentage\":\"{percentage}%\",\"report_title\":\"Quick crawler access report for {url}\",\"report_overview_heading\":\"Category overview\",\"report_details_heading\":\"Detailed crawlers and user-agents access report:\",\"report_checked_url\":\"Checked URL\",\"report_robots\":\"Robots.txt\",\"report_sitemaps\":\"Sitemap\",\"report_cached_at\":\"Cached at\",\"category_allowed_label\":\"Allowed\",\"category_disallowed_label\":\"Disallowed\",\"category_noindexed_label\":\"Noindexed\",\"col_user_agent\":\"User-Agent\",\"col_status\":\"Status\",\"status_allowed\":\"Allowed\",\"status_disallowed\":\"Disallowed\",\"status_noindexed\":\"Noindexed\",\"report_no_sitemaps\":\"Not provided\",\"category_summary_empty\":\"{category} have no configured bots.\",\"analysis_hint\":\"All supported bots will be analysed automatically.\",\"category_no_bots\":\"No crawlers defined for this category.\",\"category_toggle_show\":\"Show\",\"category_toggle_hide\":\"Hide\",\"report_snapshot\":\"Snapshot\",\"domain_slug\":\"domain\",\"cached_stale_notice\":\"Cached report is older than the refresh interval. Run a new analysis if needed.\",\"footer_line_intro\":\"© 1995-2025 Spider.es by {color_vivo} - Learn more about the Spider service.\",\"footer_line_status\":\"Project in development, API activation pending. Hosted on {stackscale}\\u0027s cloud infrastructure.\",\"footer_line_made\":\"Made with ❤ from Madrid and Herencia (Ciudad Real) - Spain.\",\"category_security_bots\":\"Security Bots\",\"category_monitoring_bots\":\"Monitoring Bots\",\"category_academic_bots\":\"Academic \\u0026 Research Bots\",\"faq_page_heading\":\"Frequently Asked Questions\",\"faq_intro\":\"Discover how Spider.es helps you audit crawler access, diagnose technical SEO issues and manage the new wave of AI bots.\",\"about_page_heading\":\"About Spider.es\",\"about_intro_title\":\"Spider.es: essential insights for SEO professionals \\u0026 webmasters\",\"about_intro_body\":\"Spider.es maintains a curated, categorised directory of crawlers. From headline search engines and AI LLM bots to SEO auditors, social platforms, security services and research scrapers, you know exactly who is hitting your site and why that matters.\",\"about_supported_title\":\"Supported crawlers and user-agents\",\"about_supported_intro\":\"Here\\u0027s a snapshot of the ecosystems Spider.es monitors to help you stay in control of crawlability, security and performance.\",\"about_supported_list\":\"\\u003Cul\\u003E\\u003Cli\\u003E\\u003Cstrong\\u003ESearch engines:\\u003C/strong\\u003E Googlebot, Bingbot, YandexBot, Baiduspider, DuckDuckBot, Applebot, Qwantbot, SeznamBot, Sogou.\\u003C/li\\u003E\\u003Cli\\u003E\\u003Cstrong\\u003EAI \\u0026amp; LLM crawlers:\\u003C/strong\\u003E ChatGPT-User, GPTBot, Google-Extended, ClaudeBot, Claude-Web, PerplexityBot, Cohere, Anthropics, OAI-SearchBot, Quillbot, YouBot, MyCentralAIScraperBot.\\u003C/li\\u003E\\u003Cli\\u003E\\u003Cstrong\\u003ESEO tools:\\u003C/strong\\u003E AhrefsBot, SemrushBot, MJ12bot, DotBot, DataForSeoBot, Awario bots, SEOkicks, Botify, Jetslide, peer39.\\u003C/li\\u003E\\u003Cli\\u003E\\u003Cstrong\\u003ESocial \\u0026amp; sharing:\\u003C/strong\\u003E facebookexternalhit, FacebookBot, Twitterbot (X), Pinterestbot, Slackbot, Meta external fetchers.\\u003C/li\\u003E\\u003Cli\\u003E\\u003Cstrong\\u003ESecurity \\u0026amp; cloud:\\u003C/strong\\u003E AliyunSecBot, Amazonbot, Google-CloudVertexBot and more.\\u003C/li\\u003E\\u003Cli\\u003E\\u003Cstrong\\u003EScrapers \\u0026amp; research:\\u003C/strong\\u003E BLEXBot, Bytespider, CCBot, Diffbot, DuckAssistBot, EchoboxBot, FriendlyCrawler, ImagesiftBot, magpie-crawler, NewsNow, news-please, omgili, Poseidon Research Crawler, Quora-Bot, Scrapy, SeekrBot, SeznamHomepageCrawler, TaraGroup, Timpibot, TurnitinBot, ViennaTinyBot, ZoomBot, ZoominfoBot.\\u003C/li\\u003E\\u003C/ul\\u003E\",\"faq_index_heading\":\"Jump to a question\",\"faq_index_intro\":\"Pick a topic to scroll straight to the answer.\",\"about_seo_title\":\"About Spider.es · Who we monitor\",\"about_seo_description\":\"Learn how Spider.es tracks search, AI, SEO, social, cloud and scraper crawlers so you stay in control of who reaches your site.\",\"faq_q_googlebot\":\"How can I check if Googlebot is blocked by my site?\",\"faq_a_googlebot\":\"Run any URL through Spider.es and, within seconds, you\\u0027ll see the robots.txt rule, meta directive or X-Robots-Tag header that affects Googlebot, together with the exact allow or disallow that fired.\",\"faq_q_bingbot\":\"How do I test Bingbot vs. Googlebot access?\",\"faq_a_bingbot\":\"Compare the Bingbot and Googlebot rows in the decision table to spot differences in permissions, crawl delays or overrides for each engine.\",\"faq_q_ai\":\"Can I see if AI crawlers like ChatGPT or Perplexity can crawl my site?\",\"faq_a_ai\":\"Spider.es keeps an eye on GPTBot, ChatGPT-User, Claude, Perplexity, Google-Extended and many other AI user agents, flagging whether they are blocked and which directive enforces it.\",\"faq_q_indexing\":\"Why isn\\u0027t Google indexing all my sitemap pages?\",\"faq_a_indexing\":\"If strategic URLs are disallowed or tagged noindex, they won\\u0027t be indexed even if the sitemap references them. Use the report to ensure key sections are crawlable, then resubmit the sitemap in Search Console.\",\"faq_q_robots\":\"What\\u0027s an easy way to understand robots.txt?\",\"faq_a_robots\":\"Robots.txt is a site-wide manifest of crawl rules. Spider.es highlights the directive that matched your URL so you understand the impact without parsing the file line by line.\",\"faq_q_specific\":\"Can I test specific pages, not just the homepage?\",\"faq_a_specific\":\"Submit the full URL of any product page, article or resource—Spider.es checks robots.txt, meta tags and headers for that specific path so you can validate granular directives.\",\"faq_permalink_label\":\"Permalink to this answer\",\"faq_section_insights_title\":\"Spider.es: essential insights for SEO professionals \\u0026 webmasters\",\"faq_section_insights_body\":\"Spider.es maintains a curated, categorised directory of crawlers. From headline search engines and AI LLM bots to SEO auditors, social platforms, security services and research scrapers, you know exactly who is hitting your site and why that matters.\",\"faq_section_supported_title\":\"Supported crawlers and user-agents\",\"faq_section_supported_intro\":\"Here\\u0027s a snapshot of the ecosystems Spider.es monitors to help you stay in control of crawlability, security and performance.\",\"faq_section_supported_list\":\"\\u003Cul\\u003E\\u003Cli\\u003E\\u003Cstrong\\u003ESearch engines:\\u003C/strong\\u003E Googlebot, Bingbot, YandexBot, Baiduspider, DuckDuckBot, Applebot, Qwantbot, SeznamBot, Sogou.\\u003C/li\\u003E\\u003Cli\\u003E\\u003Cstrong\\u003EAI \\u0026amp; LLM crawlers:\\u003C/strong\\u003E ChatGPT-User, GPTBot, Google-Extended, ClaudeBot, Claude-Web, PerplexityBot, Cohere, Anthropics, OAI-SearchBot, Quillbot, YouBot, MyCentralAIScraperBot.\\u003C/li\\u003E\\u003Cli\\u003E\\u003Cstrong\\u003ESEO tools:\\u003C/strong\\u003E AhrefsBot, SemrushBot, MJ12bot, DotBot, DataForSeoBot, Awario bots, SEOkicks, Botify, Jetslide, peer39.\\u003C/li\\u003E\\u003Cli\\u003E\\u003Cstrong\\u003ESocial \\u0026amp; sharing:\\u003C/strong\\u003E facebookexternalhit, FacebookBot, Twitterbot (X), Pinterestbot, Slackbot, Meta external fetchers.\\u003C/li\\u003E\\u003Cli\\u003E\\u003Cstrong\\u003ESecurity \\u0026amp; cloud:\\u003C/strong\\u003E AliyunSecBot, Amazonbot, Google-CloudVertexBot and more.\\u003C/li\\u003E\\u003Cli\\u003E\\u003Cstrong\\u003EScrapers \\u0026amp; research:\\u003C/strong\\u003E BLEXBot, Bytespider, CCBot, Diffbot, DuckAssistBot, EchoboxBot, FriendlyCrawler, ImagesiftBot, magpie-crawler, NewsNow, news-please, omgili, Poseidon Research Crawler, Quora-Bot, Scrapy, SeekrBot, SeznamHomepageCrawler, TaraGroup, Timpibot, TurnitinBot, ViennaTinyBot, ZoomBot, ZoominfoBot.\\u003C/li\\u003E\\u003C/ul\\u003E\",\"faq_section_visibility_title\":\"How to improve SEO visibility with Spider.es reports\",\"faq_section_visibility_intro\":\"Turn every report into a checklist that keeps search engines focused on your most valuable content.\",\"faq_section_visibility_list\":\"\\u003Cul\\u003E\\u003Cli\\u003E\\u003Cstrong\\u003EOptimise crawl budget:\\u003C/strong\\u003E retire low-value or duplicate areas so Google spends time on strategic URLs.\\u003C/li\\u003E\\u003Cli\\u003E\\u003Cstrong\\u003EExpose critical resources:\\u003C/strong\\u003E make sure CSS, JavaScript and imagery remain crawlable for full rendering.\\u003C/li\\u003E\\u003Cli\\u003E\\u003Cstrong\\u003EReference sitemaps:\\u003C/strong\\u003E declare or refresh XML sitemaps in robots.txt to guide discovery.\\u003C/li\\u003E\\u003Cli\\u003E\\u003Cstrong\\u003ERefine directives:\\u003C/strong\\u003E catch accidental blocks or redundant allows and align them with your SEO strategy.\\u003C/li\\u003E\\u003C/ul\\u003E\",\"faq_section_issues_title\":\"Common crawler access issues \\u0026amp; fixes\",\"faq_section_issues_intro\":\"Watch for these warning signs before they erode organic traffic:\",\"faq_section_issues_list\":\"\\u003Cul\\u003E\\u003Cli\\u003E\\u003Cstrong\\u003EUnintentional disallows:\\u003C/strong\\u003E prune legacy robots.txt rules that now block important sections.\\u003C/li\\u003E\\u003Cli\\u003E\\u003Cstrong\\u003EServer errors \\u0026amp; dead pages:\\u003C/strong\\u003E resolve 5xx responses and 404s that waste crawl budget.\\u003C/li\\u003E\\u003Cli\\u003E\\u003Cstrong\\u003EParameter chaos:\\u003C/strong\\u003E consolidate variants with clean URLs and canonical tags.\\u003C/li\\u003E\\u003Cli\\u003E\\u003Cstrong\\u003EJavaScript-only delivery:\\u003C/strong\\u003E provide server-side rendering or fallback links for vital content.\\u003C/li\\u003E\\u003Cli\\u003E\\u003Cstrong\\u003EWeak internal linking:\\u003C/strong\\u003E surface orphan pages so crawlers can discover them.\\u003C/li\\u003E\\u003Cli\\u003E\\u003Cstrong\\u003EUser-agent or IP blocks:\\u003C/strong\\u003E ensure firewalls allow legitimate bots while filtering abuse.\\u003C/li\\u003E\\u003Cli\\u003E\\u003Cstrong\\u003EMobile mismatches:\\u003C/strong\\u003E align mobile and desktop experiences for Google\\u0027s mobile-first index.\\u003C/li\\u003E\\u003C/ul\\u003E\",\"faq_section_analyzes_title\":\"What does Spider.es analyse?\",\"faq_section_analyzes_body\":\"Spider.es inspects robots.txt, meta robots tags and X-Robots-Tag headers side by side to show which bots can crawl, who is blocked and the reason behind each outcome.\",\"faq_section_goodtoknow_title\":\"SEO essentials worth remembering\",\"faq_section_robots_title\":\"Robots.txt overview\",\"faq_section_robots_body\":\"Robots.txt stops compliant bots before a URL is fetched. Because it is public, treat it as guidance for well-behaved crawlers, not a security barrier, and pair it with meta and header directives for finer control.\",\"faq_section_meta_title\":\"Meta robots vs. X-Robots-Tag\",\"faq_section_meta_body\":\"Meta robots tags live in HTML, while X-Robots-Tag headers apply to any file type. Combined, they control indexing behaviour for pages and assets that make it past the crawl gate.\",\"faq_section_ai_title\":\"Why AI bots might be blocked\",\"faq_section_ai_body\":\"AI crawlers can consume bandwidth, reuse proprietary content or spark legal debates. Blocking them in robots.txt or response headers makes your policy explicit and protects your data.\",\"faq_section_block_title\":\"When it\\u0027s okay to block bots\",\"faq_section_block_body\":\"It\\u0027s appropriate to block private areas, staging sites, duplicate content or aggressive scrapers. Pair disallow rules with noindex where necessary and maintain a whitelist for the bots you rely on.\",\"home_tagline\":\"Lightning-fast crawler visibility assistant for technical SEOs.\",\"blog_heading\":\"Blog\",\"blog_intro\":\"Fresh updates, tutorials and product notes from the Spider.es team.\",\"blog_empty\":\"No posts yet. Check back soon.\",\"blog_read_more\":\"Read more\",\"blog_published_on\":\"Published on\",\"blog_not_found_title\":\"Post not found\",\"blog_not_found_message\":\"The article you were looking for is no longer available.\",\"blog_back_to_list\":\"Back to the blog\",\"seo_blog_title\":\"Spider.es Blog · Updates \\u0026 guides\",\"seo_blog_description\":\"Explore the latest Spider.es product updates, technical SEO workflows and tips for managing crawler visibility.\",\"home_section_instant_title\":\"🤖 Instant Crawler Checker\",\"home_section_instant_body\":\"Paste any URL and get an immediate verdict on flagship search, AI, SEO and monitoring bots—from Googlebot and Bingbot to GPTBot, Ahrefs and beyond—so you know exactly who can reach your pages. \\u003Ca href=\\u0022/faq/#supported\\u0022\\u003EExplore supported crawlers \\u0026amp; user agents\\u003C/a\\u003E.\",\"home_section_seo_title\":\"💸 Avoid Costly SEO Mistakes\",\"home_section_seo_body\":\"Misconfigured directives drain organic reach. Verify your crawl rules, keep mission-critical assets open and fence off unwanted scrapers. \\u003Ca href=\\u0022/faq/#seo-visibility\\u0022\\u003EBoost SEO visibility\\u003C/a\\u003E • \\u003Ca href=\\u0022/faq/#issues\\u0022\\u003ETroubleshoot common problems\\u003C/a\\u003E.\",\"home_section_how_title\":\"🧩 How Spider Works\",\"home_section_how_body\":\"Spider cross-references robots.txt directives, meta robots tags and X-Robots-Tag headers to produce a per-bot decision log you can action immediately. \\u003Ca href=\\u0022/faq/#how-it-works\\u0022\\u003ESee Spider\\u0027s methodology\\u003C/a\\u003E.\",\"home_section_why_title\":\"Why This Report Matters\",\"home_section_why_intro\":\"The report confirms whether search engines, AI services and scrapers can reach your content—or if something is unintentionally blocked.\",\"home_section_why_list\":\"\\u003Cul\\u003E\\u003Cli\\u003E\\u003Cstrong\\u003EProtect visibility:\\u003C/strong\\u003E verify Google, Bing and other engines aren\\u0027t excluded by stray robots, meta or header rules.\\u003C/li\\u003E\\u003Cli\\u003E\\u003Cstrong\\u003EControl AI usage:\\u003C/strong\\u003E check that ChatGPT, Claude, Perplexity and fellow LLM crawlers respect your boundaries.\\u003C/li\\u003E\\u003Cli\\u003E\\u003Cstrong\\u003EDemonstrate enforcement:\\u003C/strong\\u003E explicit blocks document your policy for compliance, licensing or legal discussions.\\u003C/li\\u003E\\u003Cli\\u003E\\u003Cstrong\\u003ESpend crawl budget wisely:\\u003C/strong\\u003E trim noisy bots so search engines focus on revenue-driving pages.\\u003C/li\\u003E\\u003C/ul\\u003E\",\"home_section_why_summary\":\"Whether you welcome or reject bots, Spider.es keeps your crawl setup predictable.\"},\n        supportedLangs: [\"en\",\"es\",\"fr\",\"pt\",\"it\",\"de\",\"ko\",\"ja\"],\n        endpoints: {\n            analyze: \"/analyze.php\",\n            history: \"/history.php\"        },\n        history: {\n            pageSize: 12,\n            maxPages: 10        },\n        siteUrl: \"https://spider.es\",\n        serviceUrl: \"https://spider.es\",\n        domainSlug: \"domain\",\n        langBasePath: \"\",\n        prefetchedReport: null,\n        prefillUrl: \"\",\n        domainHost: \"\",\n        autoAnalyze: false,\n        snapshotStale: false,\n        historyOnly: false,\n        prefetchedHistory: {\"items\":[{\"url\":\"https://www.elmundo.es/\",\"host\":\"www.elmundo.es\",\"scope\":\"site\",\"lang\":\"ko\",\"timestamp\":\"2026-02-10T07:42:45+00:00\",\"counts\":{\"Allowed\":97,\"Disallowed\":32,\"Noindexed\":0},\"total_bots\":129},{\"url\":\"https://interscope-wp1-elb-k.umg-wp.com/\",\"host\":\"interscope-wp1-elb-k.umg-wp.com\",\"scope\":\"site\",\"lang\":\"en\",\"timestamp\":\"2026-02-10T03:08:18+00:00\",\"counts\":{\"Allowed\":129,\"Disallowed\":0,\"Noindexed\":0},\"total_bots\":129},{\"url\":\"https://anw.es/\",\"host\":\"anw.es\",\"scope\":\"site\",\"lang\":\"en\",\"timestamp\":\"2026-02-05T18:58:52+00:00\",\"counts\":{\"Allowed\":129,\"Disallowed\":0,\"Noindexed\":0},\"total_bots\":129},{\"url\":\"https://elmundo.es/\",\"host\":\"elmundo.es\",\"scope\":\"site\",\"lang\":\"en\",\"timestamp\":\"2026-01-28T21:12:47+00:00\",\"counts\":{\"Allowed\":97,\"Disallowed\":32,\"Noindexed\":0},\"total_bots\":129},{\"url\":\"https://www.supermercadosmas.com/\",\"host\":\"www.supermercadosmas.com\",\"scope\":\"site\",\"lang\":\"en\",\"timestamp\":\"2026-01-21T20:27:11+00:00\",\"counts\":{\"Allowed\":129,\"Disallowed\":0,\"Noindexed\":0},\"total_bots\":129},{\"url\":\"https://life.ca/\",\"host\":\"life.ca\",\"scope\":\"site\",\"lang\":\"en\",\"timestamp\":\"2026-01-18T06:07:27+00:00\",\"counts\":{\"Allowed\":129,\"Disallowed\":0,\"Noindexed\":0},\"total_bots\":129},{\"url\":\"https://www.travellerbusjobs.xyz/\",\"host\":\"www.travellerbusjobs.xyz\",\"scope\":\"site\",\"lang\":\"en\",\"timestamp\":\"2026-01-17T18:19:58+00:00\",\"counts\":{\"Allowed\":129,\"Disallowed\":0,\"Noindexed\":0},\"total_bots\":129},{\"url\":\"https://sweetlad.xyz/\",\"host\":\"sweetlad.xyz\",\"scope\":\"site\",\"lang\":\"de\",\"timestamp\":\"2026-01-17T09:06:21+00:00\",\"counts\":{\"Allowed\":129,\"Disallowed\":0,\"Noindexed\":0},\"total_bots\":129},{\"url\":\"https://elmundo.es/\",\"host\":\"elmundo.es\",\"scope\":\"site\",\"lang\":\"de\",\"timestamp\":\"2026-01-13T13:02:46+00:00\",\"counts\":{\"Allowed\":97,\"Disallowed\":32,\"Noindexed\":0},\"total_bots\":129},{\"url\":\"https://elmundo.es/\",\"host\":\"elmundo.es\",\"scope\":\"site\",\"lang\":\"de\",\"timestamp\":\"2026-01-12T06:29:29+00:00\",\"counts\":{\"Allowed\":97,\"Disallowed\":32,\"Noindexed\":0},\"total_bots\":129},{\"url\":\"https://www.elmundo.es/\",\"host\":\"www.elmundo.es\",\"scope\":\"site\",\"lang\":\"es\",\"timestamp\":\"2026-01-11T21:25:41+00:00\",\"counts\":{\"Allowed\":97,\"Disallowed\":32,\"Noindexed\":0},\"total_bots\":129},{\"url\":\"https://elmundo.es/\",\"host\":\"elmundo.es\",\"scope\":\"site\",\"lang\":\"es\",\"timestamp\":\"2026-01-11T16:02:51+00:00\",\"counts\":{\"Allowed\":97,\"Disallowed\":32,\"Noindexed\":0},\"total_bots\":129}],\"total\":120,\"page\":1,\"page_size\":12,\"total_pages\":10},\n        historyPath: \"/history\",\n        historySlug: \"history\",\n        analyzeTimeoutMs: 120000,\n        analyzeTimeoutBufferMs: 8000,\n    };\n</script>\n    <script src=\"https://spider.es/js/app.min.js?v=1758572911\" defer></script>\n</body>\n</html>",
  "snippets": [
    {
      "id": "add_open_graph",
      "title": "Add missing Open Graph tags",
      "description": "These tags control how your page looks when shared on social media and some AI platforms.",
      "language": "html",
      "code": "<meta property=\"og:image\" content=\"https://yoursite.com/og-image.jpg\">\n<meta property=\"og:url\" content=\"https://spider.es/\">\n<meta property=\"og:type\" content=\"website\">",
      "filename": "<head>"
    },
    {
      "id": "add_schema_org",
      "title": "Add Schema.org JSON-LD",
      "description": "Structured data helps AI agents understand the type, author, and purpose of your content.",
      "language": "html",
      "code": "<script type=\"application/ld+json\">\n{\n  \"@context\": \"https://schema.org\",\n  \"@type\": \"WebPage\",\n  \"name\": \"Spider.es · Lightning-fast crawler & bot access checker\",\n  \"description\": \"Audit, in seconds, whether search engines, AI crawlers, SEO tools and scrapers can reach your pages—and pinpoint the directive that stops them.\",\n  \"url\": \"https://spider.es/\",\n  \"inLanguage\": \"en\",\n  \"isPartOf\": {\n    \"@type\": \"WebSite\",\n    \"name\": \"spider.es\"\n  }\n}\n</script>",
      "filename": "<head>"
    },
    {
      "id": "add_content_signals",
      "title": "Add Content-Signal HTTP header",
      "description": "The Content-Signal header tells AI agents about the nature of your content. Add it via your web server or CDN.",
      "language": "nginx",
      "code": "# Nginx — add to your server block:\nadd_header Content-Signal \"type=website; lang=en\" always;\n\n# Apache — add to .htaccess:\n# Header set Content-Signal \"type=website; lang=en\"",
      "filename": "nginx.conf or .htaccess"
    },
    {
      "id": "add_markdown_negotiation",
      "title": "Support Accept: text/markdown",
      "description": "When a client sends Accept: text/markdown, respond with a Markdown version of the page. This is the gold standard for AI-readiness.",
      "language": "nginx",
      "code": "# Nginx — serve .md files when client requests Markdown:\n# Option 1: Serve pre-generated .md files\nmap $http_accept $markdown_suffix {\n  default \"\";\n  \"~text/markdown\" \".md\";\n}\n\n# Then in your location block:\ntry_files $uri$markdown_suffix $uri =404;\n\n# Option 2: Use your app framework to check the Accept header\n# and return Markdown content with Content-Type: text/markdown",
      "filename": "nginx.conf or application code"
    }
  ]
}

Use our API to get this programmatically (coming soon)