AI Investigator: Automated Research Pipeline

Overview

On November 3rd, 2024, Arvid Kahl tweeted: “I want to run AI agents to scrape URLs and do data extraction until they find specific information. What’s the framework for this?”

I built and open-sourced it over the weekend. It picked up traction the following week.

The Problem

Arvid’s need was specific: automated research that could scrape links, determine relevance, and generate structured reports. He wanted to find enterprise AI case studies without manually reviewing hundreds of pages.

I had the same pain point from years of researching case studies for marketing campaigns.

What It Does

A modular Python system that autonomously researches enterprise AI case studies:

• Input: CSV of URLs or a company website to crawl
• Processing: Firecrawl scrapes content, Claude 3.5 Sonnet qualifies and analyzes
• Output: Three structured report types (individual case analysis, cross-case patterns, executive dashboard)

The architecture is intentionally simple: Discovery → Qualification → Analysis → Synthesis. Async processing handles 50+ URLs in ~10 minutes.

Key Technical Choices

• Firecrawl API: Handles both link discovery (/v1/map) and content extraction (/v1/scrape)
• Two-stage Claude analysis: First qualifies enterprise relevance, then generates deep reports if passed
• Modular design: Scrapers, processors, and orchestration are cleanly separated
• State management: Idempotent processing with detailed logs for debugging

Distribution

Shipping inside the window of Arvid’s tweet mattered. The repo picked up a few hundred stars over the following weeks and developers started adapting it for market research and academic literature reviews.

What It Became

This was a prototype for what later became context orchestration: specialized agents discovering, qualifying, and synthesizing information in parallel. The same pattern, productized for strategic marketing rather than case study research, is what I built at 99Ravens.

Building in public reinforced one habit: ship the smallest workable system, then iterate. Solo weekend development forces prioritization. No complex infrastructure, no UI — just the core workflow, error handling, and docs. That produced cleaner code than my over-engineered side projects.

The most useful outcome was seeing developers adapt the codebase. One extended it into a literature review synthesis tool for academic papers.

The repository remains open but is unmaintained. It served its purpose as a reference implementation for autonomous research agents.