AI-Assisted Research Workflows: Best Practices

Executive Summary

Background

This report surveys best practices for AI-assisted research workflows in 2025-2026, drawing from:

• Anthropic's Claude Code best practices
• Multi-agent orchestration frameworks (LangChain, CrewAI)
• Deep research API providers (Perplexity, OpenAI, Google, xAI)
• Academic work on autonomous research agents (Agent Laboratory)

The Research Pipeline Problem

Why Single-Shot Prompts Fail

A typical approach:

"Write a comprehensive article about compute governance"

This fails because:

1. No context gathering - AI uses only training data, misses recent developments
2. No strategic planning - AI doesn't think about what actually matters
3. Premature writing - Starts generating prose before understanding the topic
4. No validation - Errors compound without feedback loops

The Better Architecture

Context Assembly → Strategic Planning → Targeted Research → Drafting → Validation → Grading
      (local)          (Opus)            (Perplexity)       (Sonnet)    (scripts)   (Haiku)

Phase 1: Context Assembly

Goal: Gather everything relevant before invoking expensive AI reasoning.

What to Gather

Why This Matters

Implementation Pattern

// 1. Find related pages (free)
const relatedPages = await searchWiki(topic);

// 2. Find existing resources (free)
const resources = await queryResourcesDB(topic);

// 3. Summarize with Haiku ($0.02)
const contextBundle = await summarizeContext({
  model: 'haiku',
  pages: relatedPages,
  resources: resources
});

Phase 2: Strategic Planning (Opus)

Goal: Figure out what this article should actually cover and why.

What Opus Should Decide

Prompt Pattern

Given this context bundle about [TOPIC]:

[CONTEXT_BUNDLE]

You are planning a wiki article. Before any writing, think through:

1. **Cruxes**: What are the 2-3 key debates or uncertainties about this topic?
2. **Framing**: What's the most useful frame for readers? (risk? opportunity? tradeoff?)
3. **Gap analysis**: What does existing coverage miss?
4. **Research questions**: What specific questions need external research?
5. **Structure**: What sections would best serve readers?

Do NOT write the article. Output a structured plan in JSON.

Budget

Phase 3: Targeted Research

Goal: Fill specific gaps identified in the plan—not open-ended browsing.

Option A: Claude Code WebSearch

Uses Claude's built-in web search. Good integration but limited depth.

// Directed by the plan
for (const question of plan.researchQuestions) {
  await webSearch(question);
}

Cost: Included in Claude API pricing

Depth: Moderate (single search per query)

Option B: Perplexity Sonar via OpenRouter

Perplexity Sonar Deep Research is purpose-built for comprehensive research. Available via OpenRouter API.

Integration Example:

const openrouter = new OpenAI({
  baseURL: 'https://openrouter.ai/api/v1',
  apiKey: process.env.OPENROUTER_API_KEY,
});

const research = await openrouter.chat.completions.create({
  model: 'perplexity/sonar-deep-research',
  messages: [{
    role: 'user',
    content: `Research: ${plan.researchQuestions.join('\n')}`
  }],
});

Option C: Other Deep Research APIs

Option D: Open Source

Open Deep Research (HuggingFace) provides an open-source implementation with 10K+ GitHub stars.

Phase 4: Drafting (Sonnet)

Goal: Execute the plan with research in hand.

Prompt Pattern

You are writing a wiki article based on this plan:

[OPUS_PLAN]

Using these research findings:

[CURATED_RESEARCH]

Following this style guide:

[STYLE_GUIDE_EXCERPT]

Write the article. Use tables over bullet lists. Include citations.
Escape all dollar signs (\\$100 not $100).

Why Sonnet, Not Opus?

Drafting is execution, not strategy. Sonnet:

• Follows instructions well
• Costs 1/10th of Opus
• Produces similar prose quality when given a good plan

Cost: $0.50-1.50 per article

Phase 5: Validation

Goal: Catch errors before they compound.

Automated Checks (Free)

npm run crux -- validate compile        # Syntax errors
npm run crux -- validate unified --rules=dollar-signs,comparison-operators
npm run crux -- validate entity-links   # Broken links

Fix Loop (Haiku)

If validation fails, use Haiku to fix mechanical issues:

if (validationErrors.length > 0) {
  await fixWithHaiku(draft, validationErrors);
  // Re-validate
}

Cost: $0.02-0.10 per fix cycle

Phase 6: Grading

Goal: Ensure quality meets threshold before accepting.

Use existing grading infrastructure:

node scripts/content/grade-by-template.mjs --page new-article

Quality Gates

Cost Comparison

Old Approach: Single-Shot Opus

New Pipeline Approach

Multi-Agent Architectures

For complex articles, consider specialized agents:

Agent Laboratory Pattern

Agent Laboratory (arXiv 2025) achieves 84% cost reduction using three stages:

1. Literature review agent - Gathers sources
2. Experimentation agent - Tests claims
3. Report writing agent - Produces output

CrewAI Pattern

from crewai import Agent, Task, Crew

researcher = Agent(role='Researcher', goal='Find authoritative sources')
analyst = Agent(role='Analyst', goal='Identify key insights')
writer = Agent(role='Writer', goal='Produce clear prose')

crew = Crew(agents=[researcher, analyst, writer], tasks=[...])

Master-Planner-Executor-Writer

Multi-agent search architecture:

• Master: Coordinates overall workflow
• Planner: Decomposes tasks into DAG
• Executor: Runs tool calls
• Writer: Synthesizes into prose

Implementation Recommendations

For LongtermWiki Wiki

Environment Setup

# .env additions
ANTHROPIC_API_KEY=sk-ant-...      # For Claude Code SDK
OPENROUTER_API_KEY=sk-or-...      # For Perplexity access

Budget Guidelines

Products & APIs Landscape

This section catalogs tools available for AI-assisted research workflows as of early 2026.

Web Search / Grounding APIs

These provide real-time web search for grounding LLM responses with citations.

Academic Literature Search

Specialized tools for searching and analyzing scientific papers.

Specialized Corpus Search

Scry Key Features:

• SQL + vector search with arbitrary query composition
• Semantic operations: mixing concepts, debiasing ("X but not Y"), contrastive axes
• Curated sources: arXiv, bioRxiv, PhilPapers, LessWrong, EA Forum, HN, Twitter/X, Bluesky
• Reproducibility: visible SQL queries, structured metadata, iterative refinement
• Custom embeddings: store named concept vectors for reuse

Scry vs ChatGPT Deep Research: Scry emphasizes control and reproducibility (you write SQL, see exactly what matched), while Deep Research is opaque but broader. Scry is better for iterative exploration of a fixed corpus; Deep Research for one-shot web synthesis.

Scientific Research Agents

Full agentic systems for autonomous research.

Web Scraping / Content Extraction

For when you need to extract content from specific URLs.

MCP Servers for Claude Code

Model Context Protocol servers enable direct integration with Claude Code.

Cost Comparison Matrix

Estimated costs for a typical research task (finding 20 sources with summaries):

Open Questions

Sources

Best Practices

• Claude Code: Best practices for agentic coding — Anthropic Engineering
• Multi-Step LLM Chains: Best Practices — Deepchecks
• 20 Agentic AI Workflow Patterns — Skywork AI

Deep Research APIs

• Perplexity Sonar Deep Research — OpenRouter
• OpenRouter Web Search — Real-time web grounding for any model
• Introducing Deep Research — OpenAI
• Deep Research AI Tools Comparison — Bright Inventions

Search APIs

• Exa AI — Semantic search API for AI applications
• Tavily — Web API for AI agents, $25M raise
• You.com APIs — 93% SimpleQA, Deep Search API
• Brave AI Grounding — 94.1% F1 score on SimpleQA
• Complete Guide to Web Search APIs 2025 — Firecrawl

Academic Research Tools

• Elicit — AI research assistant, 200M papers
• Semantic Scholar — Free academic search API
• Scry — SQL+vector search over 72M docs (LessWrong, EA Forum, arXiv, etc.)
• 8 Best AI Research Assistant Tools — Documind
• Best AI Research Tools for Literature Review — Medium

Scientific Research Agents

• FutureHouse Platform — Superintelligent AI agents for science
• FutureHouse customer story — Claude
• Agent Laboratory: Using LLM Agents as Research Assistants — arXiv 2025

Research Frameworks

• Autonomous Agents papers — GitHub (updated daily)
• LLM Agents Explained: Complete Guide — Dynamiq
• Top 5 MCP Search Tools 2025 — Oreate AI

Model Comparisons

• AI Deep Research: Claude vs ChatGPT vs Grok — AIMultiple
• Agentic LLMs in 2025 — Data Science Dojo
• Deep Research Survey — HuggingFace