An AI-Powered Deep Research Engine with Real-Time Web Analysis

Transform any query into comprehensive, citation-backed research summaries using advanced LLM-powered agents

Python Next.js FastAPI Ollama

• Overview

• Key Features

• Architecture

• How It Works

• Tech Stack

• Getting Started

• Usage

• API Reference

• Project Structure

• Future Roadmap

• Contributing

• License

Research Agent is a sophisticated, open-source research automation platform that combines the power of Large Language Models (LLMs) with real-time web scraping to produce comprehensive, well-cited research summaries. Think of it as your personal AI research assistant that can explore any topic in depth, synthesize information from multiple sources, and deliver professional-grade research reports—all in real-time with a beautiful, Perplexity-inspired user interface.

The system employs a multi-agent architecture where specialized AI agents handle different aspects of the research pipeline:

• Query Expansion Agent - Generates semantically diverse search queries

• Search Agent - Aggregates results from multiple web sources

• Ranker Agent - Uses LLM to select the most relevant articles

• Analyst Agent - Synthesizes findings into citation-backed summaries

This project was built to democratize access to powerful research capabilities, enabling anyone to conduct thorough research on any topic without needing specialized skills or expensive tools.

The system doesn't just search for what you type—it understands the intent behind your query and generates multiple related search terms to ensure comprehensive coverage of the topic.

Searches across the web using DuckDuckGo's API, collecting results from diverse sources to avoid bias and ensure well-rounded research.

Automatically detects and removes duplicate content using MD5 fingerprinting, ensuring you get unique, valuable information without redundancy.

Uses AI to analyze and rank articles based on their relevance to your query, filtering out low-quality or tangential content.

Generates detailed summaries with inline citations [1], [2], etc., linking every claim back to its source for verification and credibility.

Experience research in real-time with Server-Sent Events (SSE) streaming. Watch as the AI expands queries, fetches pages, and generates summaries—all with live progress updates.

A sleek, dark-mode interface inspired by Perplexity AI, featuring:

• Animated thinking process visualization
• Interactive source cards with favicons
• Markdown-rendered responses with syntax highlighting
• Responsive design for all devices

• CLI Mode: Perfect for quick terminal-based research and automation
• API Server: Production-ready FastAPI server for integration into other applications

The Research Agent operates through a sophisticated multi-stage pipeline that transforms user queries into comprehensive research reports:

When you submit a query like "Future of AI agents in 2026", the system doesn't just perform a single search. Instead, it uses an LLM to generate semantically related queries:

Original: "Future of AI agents in 2026"
Expanded:
→ "AI agent technology trends 2026"
→ "Autonomous AI systems development roadmap"
→ "Machine learning agents next generation"
→ "Artificial intelligence automation future"

This ensures broader coverage and discovers information that a single query might miss.

The expanded queries are sent to DuckDuckGo's search API. Results are:

• Aggregated from all queries

• Deduplicated by URL to avoid fetching the same page multiple times

• Typically yields 15-30 unique sources per research session

Using httpx with HTTP/2 support, pages are fetched concurrently with:

• Connection pooling for efficiency

• Semaphore-controlled concurrency (default: 5 simultaneous requests)

• Configurable timeouts to handle slow servers gracefully

• Automatic redirect following

Raw HTML is processed using:

• Readability-lxml: Mozilla's Readability algorithm to extract main article content

• BeautifulSoup: For clean text extraction from HTML

• Automatic cleaning of NULL bytes and control characters

• Articles under 100 characters are filtered out as low-quality

Content is fingerprinted using MD5 hashing to identify and remove duplicates that might have different URLs but identical content.

The Ranker Agent evaluates each article's relevance to the original query:

• Sends article titles and snippets to the LLM

• LLM selects the top N most relevant articles (default: 5)

• Ensures only the most pertinent information makes it to the final summary

The Analyst Agent synthesizes all gathered information:

• Creates a comprehensive prompt with all source content

• Generates a detailed summary with inline citations [1], [2], etc.

• Streams tokens in real-time for immediate feedback

• Includes a references section at the end

| Technology | Purpose |

|------------|---------|

| Python 3.9+ | Core runtime |

| FastAPI | High-performance async web framework |

| Uvicorn | ASGI server for production deployment |

| Ollama | Local LLM inference (nemotron-3-nano:30b-cloud) |

| HTTPX | Async HTTP client with HTTP/2 support |

| BeautifulSoup | HTML parsing and content extraction |

| Readability-lxml | Main content extraction using Mozilla's algorithm |

| DuckDuckGo Search | Web search API integration |

| python-dotenv | Environment configuration |

| uv | Fast Python package manager |

| Technology | Purpose |

|------------|---------|

| Next.js 16.1.1 | React framework with App Router |

| React 19 | UI component library |

| TypeScript | Type-safe JavaScript |

| Tailwind CSS 4 | Utility-first CSS framework |

| Radix UI | Accessible component primitives |

| Lucide React | Beautiful icon library |

| React Markdown | Markdown rendering with plugins |

| pnpm | Fast, disk-efficient package manager |

• Python 3.9+ installed

• Node.js 18+ and pnpm for the frontend

• Ollama installed and running locally (Download Ollama)

git clone https://github.com/iamdanwi/research-agent-project.git
cd research-agent-project

cd research_engine
 
# Install dependencies using uv (recommended)
uv sync
 
# OR using pip
pip install -r requirements.txt

Create a .env file in the research_engine directory:

OLLAMA_API_KEY=your_key_if_needed

ollama pull nemotron-3-nano:30b-cloud

cd ../frontend
 
# Install dependencies
pnpm install

Terminal 1 - Start the Research Engine:

cd research_engine
python server.py
# Server runs on http://localhost:8000

Terminal 2 - Start the Frontend:

cd frontend
pnpm dev
# Frontend runs on http://localhost:3000

Navigate to http://localhost:3000 and start researching!

1. Open the application in your browser

2. Type your research query in the search box

3. Press Enter or click the submit button

4. Watch the real-time research process:

• Query expansion

• Web searching

• Page fetching

• Content processing

• Summary generation

5. Review the sources and citation-backed summary

For quick terminal-based research:

cd research_engine
python main.py

Follow the prompts to enter your query. The summary can be saved to a text file.

Initiates a research session and returns a Server-Sent Events stream.

Request Body:

{
"query": "Your research topic",
"max_results": 5,
"expand_count": 3
}

Parameters:

| Parameter | Type | Default | Description |

|-----------|------|---------|-------------|

| query | string | required | The research topic to explore |

| max_results | integer | 5 | Number of search results per query |

| expand_count | integer | 3 | Number of query expansions to generate |

Response (SSE Stream):

event: status
data: "Expanding queries..."
 
event: log
data: "Expansions: [\"query1\", \"query2\", \"query3\"]"
 
event: status
data: "Searching web for 4 queries..."
 
event: sources
data: [{"title": "...", "url": "...", "text": "..."}]
 
event: status
data: "Generating summary..."
 
event: token
data: "The future of AI agents..."
 
event: done
data: "Research complete."

Event Types:

| Event | Description |

|-------|-------------|

| status | Current pipeline stage (for UI progress indicators) |

| log | Detailed debug information (counts, expansions, etc.) |

| sources | Array of selected articles with metadata |

| token | Individual tokens for streaming summary display |

| error | Error message if pipeline fails |

| done | Completion signal |

research-agent-project/
├── README.md # This file
├── .gitignore
│
├── frontend/ # Next.js Frontend Application
│ ├── src/
│ │ ├── app/
│ │ │ ├── api/ # API routes (proxy to backend)
│ │ │ ├── layout.tsx # Root layout
│ │ │ ├── page.tsx # Home page
│ │ │ └── globals.css # Global styles
│ │ ├── components/
│ │ │ ├── ResearchChat.tsx # Main chat interface
│ │ │ └── ui/ # Reusable UI components
│ │ └── lib/
│ │ └── utils.ts # Utility functions
│ ├── public/ # Static assets
│ ├── package.json
│ └── tailwind.config.ts
│
└── research_engine/ # Python Backend (FastAPI)
├── agents/ # AI Agent implementations
│ ├── __init__.py
│ ├── search_agent.py # Query expansion & web search
│ ├── ranker_agent.py # LLM-based relevance ranking
│ ├── analyst_agent.py # Summary generation
│ └── workflow.py # Pipeline orchestration
├── utils/ # Helper utilities
│ ├── scraper.py # Concurrent page fetching
│ └── text_process.py # HTML extraction & deduplication
├── config.py # Configuration & settings
├── server.py # FastAPI application
├── main.py # CLI entry point
├── pyproject.toml # Python dependencies
└── .env # Environment variables

We have exciting plans to enhance Research Agent with powerful new capabilities:

• Multi-Model Support: Switch between different LLMs (GPT-4, Claude, Gemini, local models)

• Conversation History: Persistent chat history with follow-up questions

• Export Options: Export research as PDF, Markdown, or DOCX

• Source Caching: Cache fetched pages to speed up repeated queries

• User Authentication: Secure user accounts with research history

• Advanced Web Scraping: Selenium integration for JavaScript-heavy sites

• Knowledge Graph: Visual representation of research connections

• Collaborative Research: Share research sessions with team members

• Custom Source Domains: Restrict or prioritize specific websites

• Academic Paper Integration: ArXiv, Google Scholar, PubMed integration

• Browser Extension: Research any page directly from your browser

• Autonomous Research Agents: Agents that proactively update research over time

• Multi-Modal Research: Image, video, and audio content analysis

• Custom Agent Training: Fine-tune agents for specific domains (medical, legal, technical)

• Real-Time Collaboration: Multiple users researching simultaneously

• API Marketplace: Community-built research plugins and integrations

• Enterprise Features: SSO, audit logs, compliance tools

• Voice-based research queries

• Mobile native applications (iOS/Android)

• Integration with note-taking apps (Notion, Obsidian)

• Research scheduling and alerts

• Fact-checking and bias detection

• Multi-language support

We welcome contributions from the community! Here's how you can help:

1. Report Bugs: Open an issue describing the bug and steps to reproduce

2. Suggest Features: Share ideas for new features or improvements

3. Submit Pull Requests: Fix bugs or implement new features

4. Improve Documentation: Help make our docs clearer and more comprehensive

5. Share Feedback: Tell us about your experience using Research Agent

1. Fork the repository

2. Create a feature branch: git checkout -b feature/amazing-feature

3. Make your changes

4. Run tests (coming soon)

5. Commit your changes: git commit -m 'Add amazing feature'

6. Push to the branch: git push origin feature/amazing-feature

7. Open a Pull Request

• Python: Follow PEP 8 guidelines

• TypeScript/React: Use ESLint and Prettier configurations provided

• Commits: Use conventional commit messages

This project is licensed under the MIT License - see the LICENSE file for details.

• Ollama for making local LLM inference accessible

• DuckDuckGo for their free search API

• Mozilla Readability for the content extraction algorithm

• Perplexity AI for UI/UX inspiration

• The open-source community for the amazing tools and libraries

Built with ❤️ for researchers, developers, and curious minds everywhere

⭐ Star us on GitHub •

🐛 Report Bug •

✨ Request Feature