Design a Citation-First AI Search Engine Like Perplexity

Design a Citation-First AI Search Engine (like Perplexity)

1. Clarifying Requirements

1.1 Functional Requirements

• Users type a natural language question and receive a synthesised answer with inline citations pointing to real web sources

• Every factual claim in the answer must be backed by a [1], [2]... citation that links to the original web page

• Below the answer, show a list of source cards — title, URL, favicon, and a short excerpt from the page

• Support follow-up questions in the same thread (multi-turn conversation with context carry-over)

• Support query modes: quick answer (fast, fewer sources) and deep research (slower, more sources, longer answer)

• Answers must reflect the current state of the web — news from the last hour must be findable

• Users can click any citation and see the original source highlighted

• Support query suggestions and related questions at the bottom of each answer

1.2 Non-Functional Requirements

• Answer latency: p50 < 3s, p99 < 8s end-to-end (web crawl is pre-done; latency = retrieval + LLM generation)

• Search index freshness: breaking news indexed within 15 minutes of publication

• System availability: 99.9% uptime

• Scale: 50M DAU, peak 100K QPS on the search/answer API

• Citation accuracy: every cited URL must be live and the cited claim must be traceable to that page — no hallucinated sources

• The system must never fabricate a URL — if no source supports a claim, the claim is either omitted or flagged as unverified

• Support multiple languages — queries and answers in English, Hindi, Spanish, French, Arabic, and more

2. Back-of-the-Envelope Estimation

Web index scale:

This reveals the core architectural insight: we cannot embed all 50B pages. Instead we use a two-stage retrieval — a traditional keyword/BM25 index (like a search engine) narrows candidates to the top ~20 pages, then we embed only those 20 pages' extracted text at query time for reranking and answer generation. This is fundamentally different from the RAG-over-private-docs design.

Query traffic:

Crawl and index:

3. Core APIs

Answer API

POST /v1/answer
Body: {
  query: string,
  conversation_id: string,       // for multi-turn
  mode: "quick" | "deep",        // quick=top5 sources, deep=top20
  language: "en",
  location: "IN"                 // for localised results
}
Response: {
  answer_id: string,
  answer: string,                // markdown with inline [1][2] citations
  citations: [
    {
      index: 1,
      url: "https://...",
      title: "BBC News — Budget 2024",
      favicon_url: "https://...",
      excerpt: "...relevant sentence from page...",
      published_at: "2024-06-01T10:00:00Z"
    }
  ],
  related_questions: ["What is...", "How does..."],
  mode: "quick",
  latency_ms: 2340
}

Follow-up API

POST /v1/answer/{conversation_id}/followup
Body: { query: string, mode: "quick" }
Response: { same as /v1/answer, with prior context injected }

Source preview API

GET /v1/source?url=https://bbc.com/news/...
Response: {
  url: string,
  title: string,
  clean_text: string,            // extracted article text, cached
  highlighted_passage: string,  // the passage that was cited
  snapshot_at: timestamp
}

Suggest API

GET /v1/suggest?q=what+is
Response: { suggestions: ["what is quantum computing", "what is inflation", ...] }

4. High-Level Design

4.1 Requirement 1: Web Crawling and Index Building

Components needed:

The foundation of the system is a large-scale web crawler and index that must be kept perpetually fresh, especially for news.

Crawler scheduler — maintains a priority queue of URLs to crawl, ordered by estimated freshness value. News sites and high-traffic domains are in the "hot tier" (recrawl every 15 minutes). Mid-tier domains (Wikipedia, docs, blogs) recrawl every 24 hours. Long-tail domains recrawl weekly. The scheduler tracks last_crawled_at, change_frequency (estimated from historical diffs), and PageRank score to decide priority. It publishes crawl jobs to a Kafka topic consumed by crawler workers.

Crawler workers — a distributed fleet (Scrapy-based or custom) that fetches raw HTML, respects robots.txt and crawl-delay, handles JavaScript-rendered pages (headless Chrome for SPAs), and deduplicates URLs via a distributed Bloom filter (saves ~40% of crawl bandwidth on near-duplicate URLs). Each fetched page is published as (url, raw_html, crawled_at) to the raw-pages Kafka topic.

Content extractor — consumes from raw-pages. Runs Trafilatura or Mozilla Readability to strip nav/footer/ads and extract clean article text, title, author, and publication date. Outputs (url, clean_text, metadata) to the extracted-pages topic. Also computes a content hash — if the hash matches the previous crawl, the page is skipped (no update needed).

BM25 indexer — consumes from extracted-pages and upserts into the distributed inverted index (Elasticsearch cluster). Indexes title, clean text, domain, publication date. Each page entry also stores the clean text itself in the index (not just term frequencies) so we can retrieve the full text at query time without a second fetch.

Web graph processor — separately processes outbound links from each page and updates a PageRank-like authority score. High-authority pages are prioritised in retrieval and their citations carry more weight in the answer synthesis prompt.

4.2 Requirement 2: Answering a Query in Real Time

Components needed:

This is the critical path — every millisecond matters. The pipeline must finish within 3s for the quick mode.

Query understanding — the raw query goes through three lightweight transforms in parallel (~30ms total):

Query rewriting: expands abbreviations, fixes typos, adds temporal signals ("budget 2024" → "Union Budget India 2024" if location=IN). Uses a small fine-tuned model, not a full LLM call.

Intent classification: classifies the query as factual (single correct answer), comparison (A vs B), how-to (procedural), news (recency-critical), or opinion (no single answer). Intent drives downstream decisions — news queries skip the cache entirely and force a fresh crawl check; opinion queries get an explicit disclaimer added to the answer.

Temporal extraction: detects time references ("yesterday", "last week", "in 2023") and converts them to absolute date filters applied to the search index.

Web retrieval (BM25 over inverted index) — fires a BM25 query against the Elasticsearch index with the rewritten query + metadata filters (date range from temporal extraction, language filter, safe-search flag). Returns top-20 results with their pre-stored clean text. This step runs in ~50–100ms on a well-tuned ES cluster. For news intent queries, the query additionally hits a separate real-time news index (Kafka-backed, updated every 15 minutes) to surface breaking stories.

Parallel page fetching and reranking — the top-20 BM25 results have their clean text retrieved (already stored in ES, no second HTTP fetch needed). A cross-encoder reranker scores all 20 (query, page_text_excerpt) pairs and selects the top 5 (quick mode) or top 10 (deep mode). The cross-encoder is the same bge-reranker-v2-m3 class of model, run on GPU in ~150ms.

Context assembler — takes the top-5/10 ranked pages and assembles the LLM context:

1. For each page, extracts the most relevant 2–3 paragraphs (using BM25 passage scoring within the page — no per-page embedding needed)

2. Truncates each to 300 tokens to fit within the context window budget

3. Numbers each source: [1] <title> — <URL>\n<excerpt>

4. Checks for source diversity — avoids 3 results from the same domain (avoids monopolisation by a single publisher)

5. Verifies each URL is live (cached liveness check, refreshed every 5 minutes) — dead links are dropped

LLM generator — receives a carefully structured prompt:

System: You are a citation-first search assistant. Rules:
1. Answer using ONLY the sources provided below.
2. After every factual claim, add the source number in brackets: [1], [2].
3. Never invent URLs or facts not present in the sources.
4. If sources conflict, cite both and note the disagreement.
5. If no source answers the question, say "I couldn't find a reliable answer."

Sources:
[1] BBC News — https://bbc.com/news/...
"The Reserve Bank of India raised interest rates by 25 basis points..."

[2] Reuters — https://reuters.com/...
"India's central bank governor stated that inflation..."

...

User: What did the RBI do in June 2024?

The LLM streams the answer token-by-token so the user sees text appearing immediately — perceived latency is the time to first token (~400ms), not total generation time (~2s).

Citation validator (post-generation, async) — after the answer is streamed, a background job verifies every [N] citation: does the cited URL exist in the sources provided? Is the cited claim lexically traceable to that source's excerpt (BERTScore > 0.80)? Any citation that fails is flagged and the answer is annotated with a ⚠ unverified badge on that citation. This runs async and updates the answer record within 500ms of generation completing — the user rarely notices the brief delay.

4.3 Requirement 3: Multi-Turn Conversations

Components needed:

Unlike a one-shot search, Perplexity-style conversations carry context across turns. "What is the GDP of India?" followed by "How does it compare to China?" requires the system to understand that "it" refers to India's GDP.

Conversation store — a conversations table in PostgreSQL and a Redis session cache storing the last N turns. Each turn stores {question, answer, cited_urls, cited_excerpts}. The Redis cache is keyed by conversation_id with a 1-hour TTL (evicts idle conversations to save memory).

Context injection — on each follow-up, the query understanding module is given the last 3 turns as context. The query rewriter resolves coreferences ("it", "that policy", "the company") against prior turns before firing retrieval. For example, "what is its capital?" after "tell me about France" is rewritten to "what is the capital of France?" — a fully self-contained query that retrieval can handle without needing conversational context.

Source continuity — for follow-up questions, the system first tries to answer from the sources already cited in the current conversation (avoiding re-retrieval latency). Only if the follow-up requires new information does it fire a fresh retrieval call.

4.4 Putting It All Together

The system has two long-running background processes (crawler and indexer) and one real-time critical path (query → retrieval → LLM → stream answer). They share only the Elasticsearch index as a boundary. The real-time path never waits for a crawl to complete — it reads from whatever is in the index at that moment. Freshness is a property of the index, not of the query pipeline.

Here is the complete high-level architecture:

5. Database Design

5.1 SQL vs NoSQL

This system needs five specialised stores, each chosen for a distinct access pattern:

Elasticsearch is the central store — it serves as both the inverted index for BM25 retrieval and the document store for pre-extracted clean text. At 50B pages, Elasticsearch is sharded across hundreds of nodes. We store clean text inside ES (not just term frequencies) so retrieval returns the full page content without a second fetch — critical for hitting p99 < 8s. Elasticsearch is chosen over a custom BM25 engine because it supports real-time upserts (pages recrawled every 15 minutes show up immediately), multi-language analyzers (Arabic, Hindi, French built-in), and horizontal scaling to petabyte-range indices.

PostgreSQL handles structured metadata — conversations, answer records, citation validation logs. It is not on the query critical path — it writes happen async after the answer is already streamed. ACID guarantees matter here: if an answer is stored but its citation record is not, that creates a consistency problem for the citation validator.

Redis serves two roles: the semantic answer cache (ANN over query embeddings, cosine > 0.95 threshold, 1-hour TTL for news, 24-hour TTL for evergreen queries) and the conversation session store (last 3 turns per conversation_id, 1-hour TTL). Redis Cluster handles the 100K QPS cache check at sub-millisecond latency.

Apache Kafka is the backbone of the crawl pipeline — raw HTML, extracted text, and index update events all flow through Kafka topics. Kafka gives durability (replay failed index jobs), backpressure (slow ES upserts don't block the crawler), and fan-out (the same extracted-page event feeds both the BM25 indexer and the real-time news index).

Object Storage (S3) stores raw crawled HTML as a cold archive. This is not on any query path — it is a compliance and reprocessing store. If the extraction logic improves (better noise removal, new language support), we can reprocess historical crawls without re-fetching the web.

5.2 Database Schema

-- PostgreSQL

conversations (
  conversation_id   UUID PRIMARY KEY,
  user_id           UUID,
  created_at        TIMESTAMP DEFAULT now(),
  last_active_at    TIMESTAMP
)

turns (
  turn_id           UUID PRIMARY KEY,
  conversation_id   UUID REFERENCES conversations(conversation_id),
  turn_index        INT,
  question          TEXT NOT NULL,
  rewritten_query   TEXT,
  answer            TEXT,
  mode              TEXT,          -- 'quick' | 'deep'
  latency_ms        INT,
  created_at        TIMESTAMP DEFAULT now()
)

citations (
  citation_id       UUID PRIMARY KEY,
  turn_id           UUID REFERENCES turns(turn_id),
  citation_index    INT,           -- [1], [2], [3]...
  url               TEXT NOT NULL,
  title             TEXT,
  excerpt           TEXT,          -- passage cited from the page
  published_at      TIMESTAMP,
  bertscore         FLOAT,         -- from async citation validator
  is_verified       BOOLEAN DEFAULT false,
  verified_at       TIMESTAMP
)

pages (
  url               TEXT PRIMARY KEY,
  domain            TEXT,
  title             TEXT,
  clean_text        TEXT,          -- cached for source preview API
  content_hash      TEXT,          -- skip re-index if unchanged
  authority_score   FLOAT,         -- PageRank-derived
  last_crawled_at   TIMESTAMP,
  language          TEXT,
  tier              TEXT           -- 'hot' | 'warm' | 'cold'
)

Elasticsearch document schema (one doc per web page):

{
  "_id": "sha256(url)",
  "url": "https://bbc.com/news/...",
  "domain": "bbc.com",
  "title": "RBI raises rates by 25 bps",
  "clean_text": "The Reserve Bank of India...",
  "language": "en",
  "published_at": "2024-06-01T10:00:00Z",
  "authority_score": 0.87,
  "tier": "hot",
  "last_crawled_at": "2024-06-01T10:12:00Z"
}

All fields except clean_text are indexed for filtering. clean_text is stored but not analysed for BM25 (too slow) — BM25 scoring runs over title and a tokenised body_tokens field that is a cleaned/stemmed version of the text.

6. Design Deep Dive

6.1 The Core Problem: Grounding Every Claim in a Live URL

Unlike the private-doc RAG system, this system faces an additional challenge — it is making claims about the live, changing web. A cited URL that returns 404 is worse than no citation. A citation whose page no longer contains the cited claim (it was edited after the crawl) is a silent trust violation.

URL liveness cache — a background job pings every URL that has been cited in the last 30 days once per hour. Results are stored in Redis with a 5-minute TTL. At context assembly time, any URL whose liveness check returned non-200 in the last 5 minutes is dropped from the source list. This adds 0ms to the critical path (the check is pre-computed) and prevents dead-link citations almost entirely.

Passage anchoring — when a page is indexed, we store not just the full text but also its paragraph-level "fingerprints" (SHA256 of each 100-word sliding window). When the citation validator runs, it checks whether the cited excerpt's fingerprint still exists in the current indexed version of the page. If the page was updated after the crawl and the passage is gone, the citation is flagged is_verified = false and shown to the user with a ⚠ passage may have changed warning rather than silently displaying an incorrect citation.

URL fabrication prevention — the LLM is never given a list of URLs directly in the prompt. It only receives numbered source indices [1], [2]... The actual URLs are stored server-side and injected into the response after generation. This makes it structurally impossible for the LLM to hallucinate a URL — it can only reference index numbers, and we map those back to the pre-verified URLs ourselves.

6.2 Freshness — Tiered Crawl Strategy

A uniform crawl strategy is wasteful. The BBC homepage changes every few minutes; a university department's "About Us" page changes annually. Crawling both at the same frequency wastes bandwidth on the stable page and misses breaking news on the fresh one.

Three tiers:

Hot tier — top 10M highest-traffic news and social domains (BBC, Reuters, TechCrunch, Wikipedia, Reddit top posts). Recrawled every 15 minutes via a priority queue in the crawler scheduler. These pages use a conditional HTTP GET with If-None-Match and If-Modified-Since headers — if the server returns 304 Not Modified, we skip re-extraction. ~60% of hot-tier fetches return 304, saving enormous bandwidth.

Warm tier — 500M mid-traffic pages (technical blogs, documentation sites, government sites). Recrawled every 24 hours. Change detection via content hash comparison — if the hash matches the previous crawl, no re-indexing occurs.

Cold tier — the remaining ~49.5B long-tail pages. Recrawled weekly or triggered by a significant inbound link increase (a cold-tier page that suddenly gets many hot-tier inbound links gets promoted to warm tier automatically).

Breaking news fast path — for queries classified as news intent, the system additionally queries a dedicated real-time news index fed by RSS feeds and news APIs (NewsAPI, GDELT) rather than just the crawled ES index. RSS feeds update within minutes of publication, far faster than the 15-minute hot-tier crawl cycle. This ensures "what happened in the last 30 minutes" queries are answerable.

6.3 Handling 100K QPS — Query Fan-Out and Caching

At 100K QPS with p99 < 8s, the system cannot afford any serial blocking step on the critical path. Every stage runs in parallel where possible, and the semantic cache is the primary cost and latency lever.

Two-tier cache:

Exact cache — SHA256 hash of the normalised query string maps to a stored answer. Hit in <1ms. Hit rate ~5% (only truly repeated identical queries). TTL: 1 hour for news queries, 24 hours for evergreen.

Semantic cache — the query is embedded (30ms) and ANN-searched against cached query embeddings in Redis (cosine > 0.95). This catches paraphrased repeats: "what is inflation?" and "explain what inflation means" share a cached answer. Hit rate ~30%. TTL: same as exact cache. Combined hit rate: ~35%.

ES query fan-out — the Elasticsearch cluster is sharded across 200 nodes. A single BM25 query fans out to all 200 shards in parallel, each returning their local top-5, and the ES coordinator merges 1,000 candidates to return the global top-20. This parallelism keeps BM25 retrieval latency flat at ~60ms regardless of index growth.

LLM inference scaling — the LLM layer uses a vLLM cluster with continuous batching. Multiple queries in flight at the same time share KV cache across their common prompt prefix (the system prompt is identical for all queries). This reduces per-query GPU memory footprint by ~40% and increases effective throughput. Horizontal scaling adds GPU nodes — the load balancer distributes based on current batch occupancy, not round-robin.

6.4 Multi-Language Support

The system targets English, Hindi, Spanish, French, and Arabic at launch. This creates challenges at every layer.

Crawling — the crawler identifies page language from the Content-Language header or <html lang="..."> attribute and routes pages to language-specific Elasticsearch indices. Each index uses a language-specific analyzer (Hindi uses the hindi analyzer with indic_normalization; Arabic uses arabic with right-to-left tokenisation). Keeping separate indices per language avoids cross-language scoring distortions.

Query understanding — the rewriter detects query language using langdetect and routes to the corresponding language-specific index. For cross-language queries ("tell me about Indian politics" in English, searching pages in both English and Hindi), the query is translated to the target language using a small translation model before firing BM25.

LLM generation — the LLM is prompted to answer in the same language as the user's query. The citation format [1][2] is language-neutral — it works in RTL (Arabic) and multi-script (Hindi/Devanagari) answers without modification.

6.5 Source Diversity and Anti-Monopolisation

Without diversity controls, a search for "climate change" might return 8 results from a single publication that publishes prolifically on the topic. The user sees a one-sided perspective, and the LLM synthesises from a narrow source pool.

Domain deduplication — the context assembler enforces a maximum of 2 results per domain in the final source set. If Reuters appears 4 times in the top-10 reranked results, only the top 2 Reuters results are kept and the next non-Reuters result is promoted.

Authority score blending — the reranker score is blended with the page's authority score: final_score = 0.7 × reranker_score + 0.3 × authority_score. This prevents very recent, low-authority pages from dominating (they can have high BM25 scores on breaking news but low credibility) while still surfacing them if the reranker strongly favours them.

Opinion query disclaimer — for queries classified as opinion intent ("is X better than Y?", "should I..."), the system adds a standard disclaimer to the answer: "This question has multiple perspectives. The answer below reflects the sources retrieved and may not represent all viewpoints." The LLM is additionally instructed to present multiple perspectives rather than a single synthesised conclusion.

6.6 Streaming Answers and Perceived Latency

Total answer latency is ~3s but users experience it as ~400ms because the answer streams token by token. The UX is: source cards appear first (populated from the reranker output, before LLM generation begins), then the answer streams in progressively. This is the same pattern Perplexity uses — by the time the LLM has generated 50 tokens, the user has already read the source cards and is engaged.

Streaming architecture — the API uses Server-Sent Events (SSE). The LLM generator calls vLLM's streaming API and forwards each token chunk directly to the SSE stream. There is no buffering — the first token is forwarded to the client within ~400ms of the query arriving. Citation numbers like [1] are streamed inline as they appear; the source card data is sent as a separate event: sources SSE message before the answer stream begins so the UI can render them immediately.

Progressive citation resolution — as citation indices [1], [2] appear in the stream, the client resolves them against the pre-sent source list and renders inline hyperlinks in real time. The user sees a live-linking answer being written, not a wall of text appearing all at once.

Summary