Loading comments…
Achievement
Project Info
Product Keywords
Crawler.sh is a fast, local-first web crawler and SEO & AEO analysis tool that runs from your terminal or a native desktop app. It crawls entire websites in seconds, performs 24 automated SEO checks on every page, extracts readable content as clean Markdown, and exports results to JSON, CSV, or Sitemap XML — all without sending your data to any external server.
Every page's main article content is automatically extracted and converted to clean Markdown. The tool includes word count, author byline, and excerpt for each page, making it easy to repurpose or archive web content.
Crawl any website while staying within the same domain. Built-in JavaScript rendering auto-detects and handles JS-heavy pages, and you can configure concurrency, depth limits, and polite delays between requests.
Each page is scanned for missing titles, duplicate meta descriptions, noindex directives, thin content, broken links, long URLs, and content freshness signals. Issues can be exported as CSV or human-readable TXT for your team.
Stream results as NDJSON during the crawl, or export to JSON arrays and W3C-compliant Sitemap XML. SEO reports export as CSV or TXT, giving you flexibility for any downstream workflow.
Crawl any website, find every issue, and export the data you need — all from your own machine.
This local-first approach means no data ever leaves your computer, making it ideal for sensitive or internal sites. The combination of fast crawling, automated SEO analysis, and clean Markdown extraction in a single tool eliminates the need to juggle multiple services. You get a complete pipeline from crawl to export without recurring subscription costs or API rate limits.
You need a privacy-respecting, all-in-one tool for crawling websites, running SEO audits, and extracting content — whether you're a solo developer automating workflows or an SEO team catching issues before they impact rankings. It's especially useful if you prefer working from the terminal or want a desktop app that keeps everything local.
Other tools you might consider
Opttab is an all-in-one AI visibility and GEO (Generative Engine Optimization) platform designed to help brands, individuals, and businesses measure, control, and grow their presence across AI-powered search engines and LLMs such as ChatGPT, Gemini, Claude, Perplexity and other AI models. Unlike traditional SEO tools, Opttab focuses on how AI models discover, interpret, rank, and reuse your content—giving you full transparency and control over your AI-facing digital footprint.
A content platform where yacht businesses share expertise via a quiz, and AI creates SEO-optimised articles that bring direct booking inquiries.
Create beautiful, always-current product documentation with AI. A built-in AI agent helps you write and maintain content and connects to dev and support tools for context. Edit via web, AI agent, or docs-as-code. A built-in assistant gives users instant, cited answers, reducing support load.
Wall Street pays up to $30K/year for their edge. That's how they stay ahead — better data, faster intel, institutional-grade tools. We're here to change that. We built a Wall Street terminal for independent traders and investors. Real-time breakouts, smart money tracking, insider & Congress trades, AI-powered analysis — all in one place, at a fraction of the cost.
Maker
moonbyte
Loading comments…