fix: wire EmbeddingService to simple orchestrator + improve search quality

Major fixes:
- Wire EmbeddingService to simple orchestrator for semantic deduplication
(was built but not connected - see docs/bugs/005)
- Expand BioRxiv stop words (~100) and require minimum 2 term matches
to filter out irrelevant papers
- Fix MockJudgeHandler to return honest message instead of garbage
drug candidates extracted via broken heuristics

The simple orchestrator now uses local sentence-transformers for
semantic deduplication without requiring any API keys.

Bug documentation added in docs/bugs/005_services_not_integrated.md

docs/bugs/004_gradio_intermittent_loading.md

@@ -1,44 +0,0 @@
-# Bug Report: Intermittent Gradio UI Loading (Hydration/Timeout)
-
-## 1. Symptoms
-- **Intermittent Loading**: The UI sometimes fails to load, showing a blank screen or a "Connection Error" toast.
-- **Refresh Required**: Users often have to hard refresh the page (Ctrl+Shift+R) multiple times to get the UI to appear.
-- **Mobile vs. Desktop**: The issue appears to be more prevalent or noticeable on Desktop Web than on Mobile Web (possibly due to network conditions, caching, or layout differences).
-- **Environment**: HuggingFace Spaces (Docker SDK).
-
-## 2. Root Cause Analysis
-
-Based on research into Gradio 5.x/6.x behavior on HuggingFace Spaces, this is likely due to a combination of:
-
-### A. SSR (Server-Side Rendering) Hydration Mismatch
-Gradio 5+ introduced Server-Side Rendering (SSR) to improve initial load performance. However, on HuggingFace Spaces (which uses an iframe), there can be race conditions where the server-rendered HTML doesn't match what the client-side JavaScript expects, causing a "Hydration Error". When this happens, the React/Svelte frontend crashes silently or enters an inconsistent state, requiring a full refresh.
-
-### B. WebSocket Timeouts
-HuggingFace Spaces enforces strict timeouts for WebSocket connections. If the app takes too long to initialize (e.g., loading heavy libraries or models), the initial handshake may fail.
-- *Mitigation*: Our app is relatively lightweight on startup (lazy loading models), so this is secondary, but network latency can trigger it.
-
-### C. Browser Caching
-Aggressive browser caching of the main bundle can sometimes cause version mismatches if the Space was recently rebuilt/redeployed.
-
-## 3. Proposed Solution
-
-### Immediate Fix: Disable SSR
-Forcing Client-Side Rendering (CSR) eliminates the hydration mismatch entirely. While this theoretically slightly slows down the "First Contentful Paint", it is much more robust for dynamic apps inside iframes.
-
-**Change in `src/app.py`:**
-```python
-demo.launch(
-    # ... other args ...
-    ssr_mode=False,  # Force Client-Side Rendering to fix hydration issues
-)
-```
-
-### Secondary Fixes (If needed)
-- **Increase Concurrency Limits**: Ensure `max_threads` is sufficient if many users connect at once.
-- **Health Check**: Add a simple lightweight endpoint to keep the Space "warm" if it sleeps aggressively.
-
-## 4. Verification Plan
-1. Apply `ssr_mode=False` to `src/app.py`.
-2. Deploy to HuggingFace Spaces (`fix/gradio-ui-final` branch).
-3. Test on Desktop (Chrome Incognito, Firefox) and Mobile.
-4. Verify no "Connection Error" toasts appear on initial load.

docs/bugs/005_services_not_integrated.md

@@ -0,0 +1,142 @@
+# Bug 005: Embedding Services Built But Not Wired to Default Orchestrator
+
+**Date:** November 26, 2025
+**Severity:** CRITICAL
+**Status:** Open
+
+## 1. The Problem
+
+Two complete semantic search services exist but are **NOT USED** by the default orchestrator:
+
+| Service | Location | Status |
+| ------- | -------- | ------ |
+| EmbeddingService | `src/services/embeddings.py` | BUILT, not wired to simple mode |
+| LlamaIndexRAGService | `src/services/llamaindex_rag.py` | BUILT, not wired to simple mode |
+
+## 2. Root Cause: Two Orchestrators
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│ orchestrator.py (SIMPLE MODE - DEFAULT)                         │
+│ - Basic search → judge → loop                                   │
+│ - NO embeddings                                                 │
+│ - NO semantic search                                            │
+│ - Hand-rolled keyword matching                                  │
+└─────────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────────┐
+│ orchestrator_magentic.py (MAGENTIC MODE)                        │
+│ - Multi-agent architecture                                      │
+│ - USES EmbeddingService                                         │
+│ - USES semantic search                                          │
+│ - Requires agent-framework (optional dep)                       │
+│ - OpenAI only                                                   │
+└─────────────────────────────────────────────────────────────────┘
+```
+
+**The UI defaults to simple mode**, which bypasses all the semantic search infrastructure.
+
+## 3. What's Built (Not Wired)
+
+### EmbeddingService (NO API KEY NEEDED)
+
+```python
+# src/services/embeddings.py
+class EmbeddingService:
+    async def embed(text) -> list[float]
+    async def search_similar(query) -> list[dict]  # SEMANTIC SEARCH
+    async def deduplicate(evidence) -> list        # DEDUPLICATION
+```
+
+- Uses local sentence-transformers
+- ChromaDB vector store
+- **Works without API keys**
+
+### LlamaIndexRAGService
+
+```python
+# src/services/llamaindex_rag.py
+class LlamaIndexRAGService:
+    def ingest_evidence(evidence_list)
+    def retrieve(query) -> list[dict]  # Semantic retrieval
+    def query(query_str) -> str        # Synthesized response
+```
+
+## 4. Where Services ARE Used
+
+```
+src/orchestrator_magentic.py    ← Uses EmbeddingService
+src/agents/search_agent.py      ← Uses EmbeddingService
+src/agents/report_agent.py      ← Uses EmbeddingService
+src/agents/hypothesis_agent.py  ← Uses EmbeddingService
+src/agents/analysis_agent.py    ← Uses EmbeddingService
+```
+
+All in magentic mode agents, NOT in simple orchestrator.
+
+## 5. The Fix Options
+
+### Option A: Add Embeddings to Simple Orchestrator (RECOMMENDED)
+
+Modify `src/orchestrator.py` to optionally use EmbeddingService:
+
+```python
+class Orchestrator:
+    def __init__(self, ..., use_embeddings: bool = True):
+        if use_embeddings:
+            from src.services.embeddings import get_embedding_service
+            self.embeddings = get_embedding_service()
+        else:
+            self.embeddings = None
+
+    async def run(self, query):
+        # ... search phase ...
+
+        if self.embeddings:
+            # Semantic ranking
+            all_evidence = await self._rank_by_relevance(all_evidence, query)
+            # Deduplication
+            all_evidence = await self.embeddings.deduplicate(all_evidence)
+```
+
+### Option B: Make Magentic Mode Default
+
+Change app.py to default to "magentic" mode when deps available.
+
+### Option C: Merge Best of Both
+
+Create a new orchestrator that:
+- Has the simplicity of simple mode
+- Uses embeddings for ranking/dedup
+- Doesn't require agent-framework
+
+## 6. Implementation Plan
+
+### Phase 1: Wire EmbeddingService to Simple Orchestrator
+
+1. Import EmbeddingService in orchestrator.py
+2. Add semantic ranking after search
+3. Add deduplication before judge
+4. Test end-to-end
+
+### Phase 2: Add Relevance to Evidence
+
+1. Use embedding similarity as relevance score
+2. Sort evidence by relevance
+3. Only send top-K to judge
+
+## 7. Files to Modify
+
+```
+src/orchestrator.py           ← Add embedding integration
+src/orchestrator_factory.py   ← Pass embeddings flag
+src/app.py                    ← Enable embeddings by default
+```
+
+## 8. Success Criteria
+
+- [ ] Default mode uses semantic search
+- [ ] Evidence ranked by relevance
+- [ ] Duplicates removed
+- [ ] No new API keys required (sentence-transformers is local)
+- [ ] Magentic mode still works as before

src/agent_factory/judges.py

@@ -178,38 +178,13 @@ class MockJudgeHandler:
         return findings if findings else ["No specific findings extracted (demo mode)"]
 
     def _extract_drug_candidates(self, question: str, evidence: list[Evidence]) -> list[str]:
-        """Extract potential drug names from question and evidence."""
-        # Common drug-related keywords to look for
-        candidates = set()
-
-        # Extract from question (simple heuristic)
-        question_words = question.lower().split()
-        for word in question_words:
-            # Skip common words, keep potential drug names
-            if len(word) > 3 and word not in {
-                "what", "which", "could", "drugs", "drug", "medications",
-                "medicine", "treat", "treatment", "help", "best", "effective",
-                "repurposed", "repurposing", "disease", "condition", "therapy",
-            }:
-                # Capitalize as potential drug name
-                candidates.add(word.capitalize())
-
-        # Extract from evidence titles (look for capitalized terms)
-        for e in evidence[:10]:
-            words = e.citation.title.split()
-            for word in words:
-                # Look for capitalized words that might be drug names
-                cleaned = word.strip(".,;:()[]")
-                if (
-                    len(cleaned) > 3
-                    and cleaned[0].isupper()
-                    and cleaned.lower() not in {"the", "and", "for", "with", "from"}
-                ):
-                    candidates.add(cleaned)
-
-        # Return top candidates or placeholder
-        candidate_list = list(candidates)[:5]
-        return candidate_list if candidate_list else ["See evidence below for potential candidates"]
+        """Extract drug candidates - demo mode returns honest message."""
+        # Don't attempt heuristic extraction - it produces garbage like "Oral", "Kidney"
+        # Real drug extraction requires LLM analysis
+        return [
+            "Drug identification requires AI analysis",
+            "Enter API key above for full results",
+        ]
 
     async def assess(
         self,

src/orchestrator.py

@@ -43,6 +43,7 @@ class Orchestrator:
         judge_handler: JudgeHandlerProtocol,
         config: OrchestratorConfig | None = None,
         enable_analysis: bool = False,
+        enable_embeddings: bool = True,
     ):
         """
         Initialize the orchestrator.
@@ -52,15 +53,18 @@ class Orchestrator:
             judge_handler: Handler for assessing evidence
             config: Optional configuration (uses defaults if not provided)
             enable_analysis: Whether to perform statistical analysis (if Modal available)
+            enable_embeddings: Whether to use semantic search for ranking/dedup
         """
         self.search = search_handler
         self.judge = judge_handler
         self.config = config or OrchestratorConfig()
         self.history: list[dict[str, Any]] = []
         self._enable_analysis = enable_analysis and settings.modal_available
+        self._enable_embeddings = enable_embeddings
 
-        # Lazy-load analysis (NO agent_framework dependency!)
+        # Lazy-load services
         self._analyzer: Any = None
+        self._embeddings: Any = None
 
     def _get_analyzer(self) -> Any:
         """Lazy initialization of StatisticalAnalyzer.
@@ -74,6 +78,41 @@ class Orchestrator:
             self._analyzer = get_statistical_analyzer()
         return self._analyzer
 
+    def _get_embeddings(self) -> Any:
+        """Lazy initialization of EmbeddingService.
+
+        Uses local sentence-transformers - NO API key required.
+        """
+        if self._embeddings is None and self._enable_embeddings:
+            try:
+                from src.services.embeddings import get_embedding_service
+
+                self._embeddings = get_embedding_service()
+                logger.info("Embedding service enabled for semantic ranking")
+            except Exception as e:
+                logger.warning("Embeddings unavailable, using basic ranking", error=str(e))
+                self._enable_embeddings = False
+        return self._embeddings
+
+    async def _deduplicate_and_rank(self, evidence: list[Evidence], query: str) -> list[Evidence]:
+        """Use embeddings to deduplicate and rank evidence by relevance."""
+        embeddings = self._get_embeddings()
+        if not embeddings or not evidence:
+            return evidence
+
+        try:
+            # Deduplicate using semantic similarity
+            unique_evidence: list[Evidence] = await embeddings.deduplicate(evidence, threshold=0.85)
+            logger.info(
+                "Deduplicated evidence",
+                before=len(evidence),
+                after=len(unique_evidence),
+            )
+            return unique_evidence
+        except Exception as e:
+            logger.warning("Deduplication failed, using original", error=str(e))
+            return evidence
+
     async def _run_analysis_phase(
         self, query: str, evidence: list[Evidence], iteration: int
     ) -> AsyncGenerator[AgentEvent, None]:
@@ -114,7 +153,7 @@ class Orchestrator:
                 iteration=iteration,
             )
 
-    async def run(self, query: str) -> AsyncGenerator[AgentEvent, None]:
+    async def run(self, query: str) -> AsyncGenerator[AgentEvent, None]:  # noqa: PLR0915
         """
         Run the agent loop for a query.
 
@@ -171,11 +210,14 @@ class Orchestrator:
                         # Should not happen with return_exceptions=True but safe fallback
                         errors.append(f"Unknown result type for '{q}': {type(result)}")
 
-                # Deduplicate evidence by URL
+                # Deduplicate evidence by URL (fast, basic)
                 seen_urls = {e.citation.url for e in all_evidence}
                 unique_new = [e for e in new_evidence if e.citation.url not in seen_urls]
                 all_evidence.extend(unique_new)
 
+                # Semantic deduplication and ranking (if embeddings available)
+                all_evidence = await self._deduplicate_and_rank(all_evidence, query)
+
                 yield AgentEvent(
                     type="search_complete",
                     message=f"Found {len(unique_new)} new sources ({len(all_evidence)} total)",

src/tools/biorxiv.py

@@ -2,7 +2,7 @@
 
 import re
 from datetime import datetime, timedelta
-from typing import Any
+from typing import Any, ClassVar
 
 import httpx
 from tenacity import retry, stop_after_attempt, wait_exponential
@@ -20,6 +20,211 @@ class BioRxivTool:
     # Fetch papers from last N days
     DEFAULT_DAYS = 90
 
+    # Comprehensive stop words list - these are too common to be useful for filtering
+    STOP_WORDS: ClassVar[set[str]] = {
+        # Articles and prepositions
+        "the",
+        "a",
+        "an",
+        "in",
+        "on",
+        "at",
+        "to",
+        "for",
+        "of",
+        "with",
+        "by",
+        "from",
+        "as",
+        "into",
+        "through",
+        "during",
+        "before",
+        "after",
+        "above",
+        "below",
+        "between",
+        "under",
+        "about",
+        "against",
+        "among",
+        # Conjunctions
+        "and",
+        "or",
+        "but",
+        "nor",
+        "so",
+        "yet",
+        "both",
+        "either",
+        "neither",
+        # Pronouns
+        "i",
+        "you",
+        "he",
+        "she",
+        "it",
+        "we",
+        "they",
+        "me",
+        "him",
+        "her",
+        "us",
+        "them",
+        "my",
+        "your",
+        "his",
+        "its",
+        "our",
+        "their",
+        "this",
+        "that",
+        "these",
+        "those",
+        "which",
+        "who",
+        "whom",
+        "whose",
+        "what",
+        "whatever",
+        # Question words
+        "when",
+        "where",
+        "why",
+        "how",
+        # Modal and auxiliary verbs
+        "is",
+        "are",
+        "was",
+        "were",
+        "be",
+        "been",
+        "being",
+        "am",
+        "have",
+        "has",
+        "had",
+        "having",
+        "do",
+        "does",
+        "did",
+        "doing",
+        "will",
+        "would",
+        "shall",
+        "should",
+        "can",
+        "could",
+        "may",
+        "might",
+        "must",
+        "need",
+        "ought",
+        # Common verbs
+        "get",
+        "got",
+        "make",
+        "made",
+        "take",
+        "taken",
+        "give",
+        "given",
+        "go",
+        "went",
+        "gone",
+        "come",
+        "came",
+        "see",
+        "saw",
+        "seen",
+        "know",
+        "knew",
+        "known",
+        "think",
+        "thought",
+        "find",
+        "found",
+        "show",
+        "shown",
+        "showed",
+        "use",
+        "used",
+        "using",
+        # Generic scientific terms (too common to filter on)
+        # Note: Keep medical terms like treatment, disease, drug - meaningful for queries
+        "study",
+        "studies",
+        "studied",
+        "result",
+        "results",
+        "method",
+        "methods",
+        "analysis",
+        "data",
+        "group",
+        "groups",
+        "research",
+        "findings",
+        "significant",
+        "associated",
+        "compared",
+        "observed",
+        "reported",
+        "participants",
+        "sample",
+        "samples",
+        # Other common words
+        "also",
+        "however",
+        "therefore",
+        "thus",
+        "although",
+        "because",
+        "since",
+        "while",
+        "if",
+        "then",
+        "than",
+        "such",
+        "same",
+        "different",
+        "other",
+        "another",
+        "each",
+        "every",
+        "all",
+        "any",
+        "some",
+        "no",
+        "not",
+        "only",
+        "just",
+        "more",
+        "most",
+        "less",
+        "least",
+        "very",
+        "much",
+        "many",
+        "few",
+        "new",
+        "old",
+        "first",
+        "last",
+        "next",
+        "previous",
+        "high",
+        "low",
+        "large",
+        "small",
+        "long",
+        "short",
+        "good",
+        "well",
+        "better",
+        "best",
+    }
+
     def __init__(self, server: str = DEFAULT_SERVER, days: int = DEFAULT_DAYS) -> None:
         """
         Initialize bioRxiv tool.
@@ -81,12 +286,11 @@ class BioRxivTool:
             return [self._paper_to_evidence(paper) for paper in matching]
 
     def _extract_terms(self, query: str) -> list[str]:
-        """Extract search terms from query."""
+        """Extract meaningful search terms from query."""
         # Simple tokenization, lowercase
         terms = re.findall(r"\b\w+\b", query.lower())
-        # Filter out common stop words
-        stop_words = {"the", "a", "an", "in", "on", "for", "and", "or", "of", "to"}
-        return [t for t in terms if t not in stop_words and len(t) > 2]
+        # Filter out stop words and short terms
+        return [t for t in terms if t not in self.STOP_WORDS and len(t) > 2]
 
     def _filter_by_keywords(
         self, papers: list[dict[str, Any]], terms: list[str], max_results: int
@@ -94,6 +298,9 @@ class BioRxivTool:
         """Filter papers that contain query terms in title or abstract."""
         scored_papers = []
 
+        # Require at least 2 matching terms, or all terms if fewer than 2
+        min_matches = min(2, len(terms)) if terms else 1
+
         for paper in papers:
             title = paper.get("title", "").lower()
             abstract = paper.get("abstract", "").lower()
@@ -102,7 +309,8 @@ class BioRxivTool:
             # Count matching terms
             matches = sum(1 for term in terms if term in text)
 
-            if matches > 0:
+            # Only include papers meeting minimum match threshold
+            if matches >= min_matches:
                 scored_papers.append((matches, paper))
 
         # Sort by match count (descending)