Start crawling to detect SEO issues and problems on your website.
URL
Title
H1
Words
Claims
Body Text
No Content Data
Start crawling to extract page content (title, H1, body text).
PageSpeed Insights Results
Performance analysis for homepage and category pages (powered by Google PageSpeed Insights)
đ PageSpeed analysis will appear here after crawl completion
Enable PageSpeed Analysis in settings to get Core Web Vitals and performance scores
Site Structure Visualization
No Data Yet
Start crawling to visualize your site structure
2xx Success
3xx Redirect
4xx Error
5xx Error
Other
Embedded Pages
URL
Title
Tokens
No embeddings yet. Enable Content Vectorization Mode and run a crawl.
Crawl History
Loading crawls...
Settings
Crawler Configuration
Extracts page content and generates embeddings. Faster than full crawl â skips SEO analysis.
Checking API key...
Content vectorization is optimized for up to 5,000 pages. Larger crawls may use significant storage.
Extracts all internal links with rich context + page content split by sections. Skips SEO analysis. Outputs a JSON for internal linking analysis & vectorization.
After crawl completes, extract factual claims from page content using AI.
How deep to crawl from the starting URL
Stop crawling after this many URLs
Delay between requests to be respectful
Follow 3xx redirects automatically
Include external domains in crawl
â External domains will be crawled â this increases crawl time and cost
HTTP Request Configuration
User agent string sent with requests
How long to wait for a response
Number of retries for failed requests
Preferred language for responses
Check robots.txt before crawling
Accept and send cookies
Automatically find and parse sitemap.xml files (including nested sitemaps)
Run Google PageSpeed Insights on homepage and 2 category pages after crawl completes
API key for higher rate limits (25k requests/day vs limited without key).
Get API key here
Content Filters
Comma-separated list of extensions to crawl
Comma-separated list of extensions to skip
Only crawl URLs matching these patterns. Leave empty to crawl all pages on the domain.If set from a previous crawl, these patterns will block discovery on a different domain. Clear this field when switching sites.
When to enable JS rendering
Required for Shopify, React, Next.js, Angular, Vue, and other SPA/JS-heavy sites.
Without it, the crawler only sees the HTML shell and misses content, links, and product data.
Not needed for WordPress, static sites, or server-rendered pages.
Render pages with JavaScript for dynamic content (slower but more accurate)
Time to wait for JavaScript to render after page load
Maximum time to wait for page to load
Browser engine for JavaScript rendering
Run browser without visible window (recommended for performance)
User agent string for JavaScript-rendered pages
Ã
Browser window size for rendering (affects responsive layouts)
Number of browser pages for parallel JavaScript rendering (higher = faster but more memory)
â ī¸ Performance Impact
JavaScript rendering is significantly slower than HTTP-only crawling. Use for sites with dynamic content or when static crawling produces incomplete results.
Crawl Strategy
How LibreCrawl handles sites that block crawlers
Proxy:Not configured Set in Advanced tab. Used automatically when stealth is needed.
Issue Exclusion Patterns
URLs matching these patterns will be excluded from issue detection. These are typically admin areas, development files, and framework internals that should never be indexed by search engines.
Use * for wildcards. Examples: /admin/*, *.json, /test/*