What is Crawler.sh?

Crawler.sh is a fast, local-first web crawler and SEO & AEO analysis tool that runs from your terminal or a native desktop app. It crawls entire websites in seconds, performs 24 automated SEO checks on every page, extracts readable content as clean Markdown, and exports results to JSON, CSV, or Sitemap XML — all without sending your data to any external server.

Who it's for

• SEO specialists who need to audit sites for missing titles, duplicate descriptions, thin content, and broken links before rankings suffer
• Content managers who want to extract article content as clean Markdown for backups, migrations, or feeding into other tools
• Developers who prefer a CLI-driven workflow for integrating crawling into automated pipelines or CI/CD processes

Key features

Content extraction as clean Markdown

Every page's main article content is automatically extracted and converted to clean Markdown. The tool includes word count, author byline, and excerpt for each page, making it easy to repurpose or archive web content.

Site crawling with JavaScript rendering

Crawl any website while staying within the same domain. Built-in JavaScript rendering auto-detects and handles JS-heavy pages, and you can configure concurrency, depth limits, and polite delays between requests.

24 automated SEO checks

Each page is scanned for missing titles, duplicate meta descriptions, noindex directives, thin content, broken links, long URLs, and content freshness signals. Issues can be exported as CSV or human-readable TXT for your team.

Multiple output formats

Stream results as NDJSON during the crawl, or export to JSON arrays and W3C-compliant Sitemap XML. SEO reports export as CSV or TXT, giving you flexibility for any downstream workflow.

What stands out

Crawl any website, find every issue, and export the data you need — all from your own machine.

This local-first approach means no data ever leaves your computer, making it ideal for sensitive or internal sites. The combination of fast crawling, automated SEO analysis, and clean Markdown extraction in a single tool eliminates the need to juggle multiple services. You get a complete pipeline from crawl to export without recurring subscription costs or API rate limits.

Worth checking out if…

You need a privacy-respecting, all-in-one tool for crawling websites, running SEO audits, and extracting content — whether you're a solo developer automating workflows or an SEO team catching issues before they impact rankings. It's especially useful if you prefer working from the terminal or want a desktop app that keeps everything local.