What is /agent by Firecrawl?

/agent by Firecrawl is a powerful API that transforms natural language prompts into structured data from even the most complex websites. Instead of manually scraping pages or writing brittle extraction rules, you simply describe what data you want—whether it's a single data point or an entire dataset—and the agent handles the searching, navigation, and gathering automatically. It's currently available in Research Preview and represents the next evolution of Firecrawl's extraction capabilities, making URLs optional and the entire process prompt-driven.

Who it's for

• Developers and engineers who need to extract structured data from websites without writing custom scraping logic or maintaining complex parsing pipelines.
• Data analysts and researchers who want to build datasets from sources like Y Combinator companies, arXiv papers, or real estate listings with minimal manual effort.
• Product and growth teams who require lead generation, market data, or competitive intelligence from hard-to-reach web pages at scale.

Key features

Prompt-driven data extraction

Describe what you want in plain English—like "Get all YC W24 companies with founders"—and /agent automatically searches, navigates, and extracts the relevant information. No need to specify URLs or write selectors; the agent figures out where to look and how to gather the data.

Structured JSON output with schema validation

Define your desired output schema using Pydantic (Python), Zod (JavaScript), or raw JSON, and /agent returns clean, typed data. The example output shows company names, founders, funding, and websites neatly organized in a JSON array, making it ready for immediate use in applications or databases.

Multi-source aggregation

The agent can pull data from multiple sources in a single run. As shown in the live demo, it simultaneously extracts from ycombinator.com, firecrawl.dev, crunchbase, GitHub, and Product Hunt, consolidating results into one structured response. This enables comprehensive research without manual tab-switching.

SDK and CLI support

/agent integrates seamlessly through Python and JavaScript SDKs, a REST API, and even a CLI tool. The code examples demonstrate how to get started with just a few lines of code, making it accessible whether you're building a web app, running a script, or working from the terminal.

What stands out

"Describe what data you want and /agent handles the rest."

This one-liner captures the product's core value: it eliminates the gap between intent and execution. Traditional web scraping requires you to know exactly which pages to visit, how to navigate them, and how to parse the HTML. /agent flips that model entirely—you focus on what you need, and the agent figures out how to get it. The fact that URLs are now optional, combined with the ability to extract 247 records in a single call, makes it feel less like a tool and more like a data assistant.

Worth checking out if…

You're tired of writing and maintaining custom scrapers for every new data source, or you need to collect structured information from websites that are difficult to navigate programmatically. /agent is especially valuable if you work with dynamic datasets that change frequently—like startup directories, e-commerce listings, or research paper repositories—and want a single API that adapts to your prompts rather than your code. It's also a strong fit if you're building AI-powered applications that require real-time web data without the overhead of manual extraction pipelines.