|
Download eval/README.md from sai-Rohan/Legora: direct link, hf CLI and curl.
- Browser
- Download file 27.3 kB
-
https://huggingface.co/spaces/sai-Rohan/Legora/resolve/main/eval/README.md
- Command line
-
hf download hf://spaces/sai-Rohan/Legora/eval/README.md
-
curl -L -o README.md https://huggingface.co/spaces/sai-Rohan/Legora/resolve/main/eval/README.md
27.3 kB
| # Gold-set workflow | |
| The default target is **145 items**: 30 each from BNS, BNSS, BSA and the | |
| Constitution (120), plus 15 multi-provision questions and 10 out-of-scope questions. | |
| All Gemini output is draft material with `reviewed=false`, not verified gold. | |
| Current checked-in `gold.jsonl`: **10 Gemini-generated out-of-scope drafts**. | |
| The 135 provision-backed items have not been generated because the stored corpus | |
| is unavailable following deletion of the database containers. No provision labels | |
| or metrics were invented to fill that gap. Restore/re-ingest the corpus, or use | |
| an existing stored-chunk export, before completing the remaining items. | |
| ## 1. Capture stored chunks | |
| The tools use Python 3.11+ and the project's existing dependencies. Run from the | |
| repository root. Evaluation settings are read centrally from `.env`/environment | |
| variables and do not require Neo4j or Indian Kanoon credentials. | |
| For the unified Qdrant collection: | |
| ```bash | |
| python -m eval.sample --collection legal_rag | |
| ``` | |
| For separate collections: | |
| ```bash | |
| python -m eval.sample --collection bns --collection bnss --collection bsa --collection constitution | |
| ``` | |
| For an existing JSON array, Qdrant `{"points": [...]}` object, or JSONL export: | |
| ```bash | |
| python -m eval.sample --chunks /path/to/stored-chunks.jsonl | |
| ``` | |
| Export records may be payload dictionaries (`chunk_id`, `text`, ...) or objects | |
| containing a `payload` dictionary. This must be real stored corpus data; the | |
| parser test fixtures are not a substitute for the corpus. `QDRANT_URL` and an | |
| optional `QDRANT_API_KEY` are needed only for live sampling. | |
| The snapshot `eval/provisions.jsonl` retains every recognized source chunk for | |
| review, including child chunks. Sampling uses 30 distinct section/article roots | |
| per act, preferring full parent provisions to avoid overweighting sections with | |
| many children. It is uniform within each act, reproducible with `--seed 42`, and | |
| fails if any act has fewer than 30 distinct provisions. Chapter/preamble/schedule | |
| records are not sampled. Existing snapshots are not silently overwritten. | |
| ## 2. Draft questions with Gemini | |
| ```bash | |
| python -m eval.generate | |
| # Optional: cap new items in a run, then resume with the same command/settings. | |
| python -m eval.generate --limit 10 | |
| ``` | |
| Requires `GEMINI_API_KEY`; uses `GEMINI_MODEL` and `EXTERNAL_TIMEOUT_SECONDS`. | |
| No embedder, reranker, graph, or API server is needed to draft from the snapshot. | |
| The output defaults to `eval/gold.jsonl`. Successful drafts are saved atomically | |
| after each item. Existing items, including human edits/approvals, are preserved | |
| on resume. `gold.manifest.json` binds full generation to the snapshot hash, seed, | |
| and target counts. Use a new output path to build a different dataset. | |
| Each source has a preferred intent cycling through punishment, definition, | |
| procedure, evidence and rights. Gemini must choose a supported intent rather than | |
| invent a punishment or right absent from the provision. Intent coverage therefore | |
| needs human review; no unsupported per-act intent quota is forced. | |
| Multi-provision prompts receive multiple sampled sources and must identify at | |
| least two distinct supporting provisions. Unsupported combinations and unknown | |
| IDs are rejected and retried up to three times. Generated notes explain source | |
| support and ambiguity for the reviewer. Individual source text in prompts is | |
| limited to 12,000 characters, while the review snapshot retains the full text. | |
| Review long/truncated provisions carefully. Gemini grounding is not a legal | |
| correctness guarantee. | |
| The independent refusal items can be drafted without a corpus: | |
| ```bash | |
| python -m eval.generate --out-of-scope-only | |
| ``` | |
| They use ordinary nonlegal topics (cooking, programming, astronomy, etc.), empty | |
| supporting ID lists and `answerable=false`. They also require human review. | |
| ## 3. Review each item | |
| Generation can resume after an interrupted request without replacing earlier | |
| drafts or approvals. `python -m eval.generate --limit 10 --request-interval 10` | |
| adds at most ten new drafts with requests spaced at least ten seconds apart. | |
| Pacing cannot override a provider's project/daily quota. Provider failures now | |
| report the HTTP code with fixed guidance rather than just `ClientError`; raw | |
| provider responses, keys and prompts are never printed. Rate/quota and permanent | |
| client errors stop immediately; transient server/timeouts use bounded backoff. | |
| Resolve the reported provider issue before resuming. | |
| Use `python -m eval.review --answerable-only` to review legal questions before | |
| out-of-scope items. Press `s` to inspect source text, then `a` only after checking | |
| the question/support. `x` exits without approving. Generation never marks drafts | |
| reviewed, and evaluations intentionally stop until eligible approvals exist. | |
| ```bash | |
| python -m eval.review --list | |
| python -m eval.review | |
| # Inspect a specific item: | |
| python -m eval.review --id oos-001 | |
| ``` | |
| Interactive commands display sources, edit the question, intent, expected IDs, | |
| acceptable alternate IDs or notes, skip an item, and approve it. Approval is an | |
| explicit human action. Any subsequent edit resets `reviewed=false`. Answerable | |
| items cannot be approved unless their expected and alternate IDs exist in the | |
| snapshot. Single-act items must reference that act. | |
| Noninteractive review is also supported (replace the example ID with an actual | |
| stored ID supporting the displayed question): | |
| ```bash | |
| python -m eval.review --id single-bns-001 --expected 'BNS-303(2)' --notes 'Reviewer explanation' | |
| python -m eval.review --id single-bns-001 --approve | |
| ``` | |
| Every JSONL record has `id`, `question`, `intent`, `act`, `expected_chunk_ids`, | |
| `acceptable_alternates`, `answerable`, `notes`, and `reviewed`. Acts are BNS, | |
| BNSS, BSA, Constitution, MULTI, or OUT_OF_SCOPE. Questions with `act=MULTI` | |
| need at least two expected IDs. Out-of-scope records must have no supporting IDs. | |
| ## 4. Predict and measure reviewed items only | |
| Once the API and its dependencies are running: | |
| ```bash | |
| python -m eval.predict | |
| python -m eval.metrics --output eval/metrics.json | |
| ``` | |
| `predict` submits **only reviewed questions** to `API_BASE_URL`, with | |
| `API_TIMEOUT_SECONDS`. It saves `eval/predictions.jsonl` only after all requests | |
| succeed; transport failures do not become fabricated results. Predictions have | |
| `id`, `retrieved_chunk_ids`, `score` (0–100), and `refused` (boolean). | |
| You may supply an existing prediction file with `--predictions PATH` for offline | |
| metrics. Recollect predictions after editing questions. | |
| The API has no structured refusal field. `predict` recognizes only the pipeline's | |
| canonical “The provided legal corpus does not contain sufficient information to | |
| answer this question” sentence; other refusal wording needs an explicit human | |
| `refused=true` label. Low confidence is independently measured using | |
| `PASS_THRESHOLD` (override with `metrics --threshold`). | |
| Metrics reject a dataset with zero reviewed records. Unreviewed predictions are | |
| ignored before validation/scoring. Missing predictions for reviewed records count | |
| as unsuccessful, not silently omitted. Reported metrics are retrieval hit rate, | |
| all-required-provisions rate, mean provision recall, mean citation precision and | |
| out-of-scope refusal/low-confidence success rate. Absent categories produce null. | |
| They measure retrieval and scope handling, not legal answer correctness. | |
| For single-provision questions, a reviewed alternate can satisfy the expected | |
| source. For multi-provision questions, all expected IDs remain required for full | |
| coverage; flat alternates count toward hit rate and precision but cannot represent | |
| complete alternative multi-source sets. Edit the expected set during review if | |
| the canonical multi-provision support changes. | |
| ```bash | |
| python -m pytest tests/test_eval.py | |
| ``` | |
| These tests use synthetic sources only in temporary test files and mock the | |
| Gemini boundary. Sampling, draft validation, resume, review and metric logic run | |
| without network calls. | |
| ## Retrieval ablation benchmark | |
| ```bash | |
| python -m eval.run_retrieval_eval | |
| # Evaluate a subset without loading unused dependencies: | |
| python -m eval.run_retrieval_eval --configs A B C --seed 42 | |
| # Reuse cached analysis and forbid Gemini calls: | |
| python -m eval.run_retrieval_eval --offline-analysis | |
| ``` | |
| `eval/run_retrieval_eval.py` accepts only `reviewed=true, answerable=true` items. | |
| It refuses an empty eligible set before initializing clients or loading weights. | |
| The current gold file has no eligible items, so the checked-in | |
| `results/retrieval_ablation.csv` is header-only and the Markdown report says | |
| “not run.” They contain no benchmark scores. An empty eligible set or a failed retrieval call does not | |
| overwrite earlier completed reports. | |
| Every preset uses the same retrieval path and feature switches: | |
| | Preset | Features | | |
| |---|---| | |
| | A | Original question, dense search, no act filter or query analysis | | |
| | B | A plus the primary-act filter from query analysis | | |
| | C | B plus `build_retrieval_queries` expansion, retaining the original question | | |
| | D | C plus BAAI/bge-reranker-large, without additive legal boosts | | |
| | E | D plus the existing legal-boost heuristics | | |
| | F | E plus graph expansion from the top 10 ranked results | | |
| | G | E plus BM25/dense RRF (k=60) and explicit provision-ID lookup | | |
| D and E use identical candidate pools and the existing analysis-enriched rerank | |
| query; only the additive legal-boost switch differs. F preserves E's ranked list. | |
| G keeps graph expansion off to isolate the new retrieval path against E. | |
| Analysis receives only the question, never the expected IDs, gold act or intent. | |
| An unknown primary act stays unfiltered, as in the production helper. | |
| `--configs A B ...` selects presets. Individual switches can be overridden with | |
| `--act-filter/--no-act-filter`, `--multi-query/--no-multi-query`, | |
| `--rerank/--no-rerank`, `--legal-boost/--no-legal-boost` and | |
| `--graph/--no-graph`. Legal boost requires reranking. For example: | |
| ```bash | |
| python -m eval.run_retrieval_eval --configs E --no-legal-boost --no-graph | |
| ``` | |
| `--bm25/--no-bm25` and `--exact-id-lookup/--no-exact-id-lookup` control the | |
| new stages independently. Compare E and G, or disable lookup to isolate BM25/RRF: | |
| ```bash | |
| python -m eval.run_retrieval_eval --configs E G --offline-analysis | |
| python -m eval.run_retrieval_eval --configs G --no-exact-id-lookup | |
| ``` | |
| G builds one BM25Okapi index by scrolling payloads from the same Qdrant collection, | |
| without vectors. It searches the original question, applies the inferred act | |
| filter, and fuses its ranked hits with the merged dense ranking using | |
| `sum(1 / (60 + rank))`. Each ID receives at most one contribution per stream. | |
| Only chunks with an overlapping query token enter the lexical ranking; negative | |
| BM25 scores remain valid matches. Both streams are capped to the common candidate | |
| budget, and fusion is capped again before reranking. The run manifest records | |
| the RRF constant, lexical query choice, flags and rank-bm25 package version. | |
| Explicit references such as `section 303 BNS`, `BNSS section 35`, and | |
| `article 21 Constitution` trigger payload `chunk_id` equality lookups. A bare | |
| `section 303` does not identify an act and causes no exact lookup. Qualified | |
| multi-act references and existing legacy Constitution IDs are supported. Only | |
| actual stored hits are prepended; nonexistent IDs are not synthesized. Exact | |
| hits bypass a potentially wrong inferred act filter and are pinned after | |
| reranking as well as before it, within the same candidate/output limits. | |
| BM25 is **not enabled by default in production**. The current table marks G as | |
| not evaluated because no reviewed, answerable gold items are available and the | |
| database corpus was removed. There is no measured Recall@5/MRR improvement to | |
| justify changing the default. Compare E/G on reviewed labels after restoring the | |
| corpus; the script never edits production defaults automatically. | |
| Overrides are recorded in the run manifest and the report's feature table, so a | |
| customized preset is not mistaken for the default configuration. | |
| `--candidates 50` is both the per-query search limit and the merged candidate | |
| budget, with a minimum of 10. Multi-query hits are deduplicated by exact chunk ID, | |
| keeping the highest dense score, then sorted by descending score and chunk ID and | |
| capped to that common budget before reranking. Thus expansion does not also give | |
| the reranker an unbounded larger budget. Dense search defaults to Qdrant exact | |
| search for reproducibility; `--approximate-search` uses production ANN search. | |
| Production retrieval defaults remain unchanged. | |
| All models are loaded once per run. A needs only the embedder and Qdrant. B–G | |
| need Gemini credentials only on a cache miss; only D–G load the reranker, and | |
| graph-enabled configurations (F by default) need `NEO4J_*`. No Indian Kanoon | |
| requests or answer-generation calls are made. | |
| Use `--collection`, `--embedder-model` and `--reranker-model` to select a frozen | |
| corpus and pinned/local model snapshots. | |
| ### Measures and confidence intervals | |
| The report provides two independent scoring modes: | |
| - **exact**: literal stored chunk ID equality; | |
| - **root**: section/article identity, e.g. `BSA-115(2)(a)` → `BSA-115`, | |
| `BNSS-45-EXPL-1` → `BNSS-45`, and `article_21_clause_1` → `CONST-21`. | |
| Recall@1/5/10 is the fraction of required targets recovered by that rank, macro | |
| averaged over questions. MRR uses the first relevant result anywhere in the | |
| retained candidate budget (default depth 50). nDCG@10 uses binary relevance and | |
| the ideal ranking of the required targets. Repeated exact/root hits retain their | |
| rank positions but receive credit only once. Root scoring collapses expected | |
| chunks from the same provision to one target. | |
| A reviewed single-provision alternate may satisfy that target. Flat alternates | |
| do not specify which required source they replace in a multi-provision question, | |
| so multi-provision ranking metrics use the canonical expected set only. | |
| For F, **context recall** measures target coverage in actual returned graph nodes. | |
| **combined context recall** measures coverage in graph nodes plus the top-10 | |
| retrieved chunks. Graph lookup keys, requested seed IDs and IDs merely mentioned | |
| in source text do not count as returned graph evidence. Exact graph coverage uses | |
| literal node/chunk IDs; root coverage handles graph/stored-ID format differences. | |
| Graph columns are blank/dashes for presets that do not run graph expansion. | |
| The default confidence intervals use 2,000 query-level bootstrap resamples and | |
| the 2.5th/97.5th percentiles. Resample indices are seeded and paired across | |
| configurations/scoring modes for the same question group. Per-intent intervals | |
| resample within each observed intent. Singleton groups have degenerate intervals. | |
| These intervals describe query-sample variation, not model/corpus drift or legal | |
| answer correctness. Override the count with `--bootstrap-samples`. | |
| ### Caching, reproducibility and output | |
| Query analyses are cached under `results/analysis_cache/` (ignored by Git). Keys | |
| include the exact question, Gemini model, seed, analyser code/prompt fingerprint | |
| and analysis schema. Cached records are schema-validated; a corrupt cache fails | |
| instead of silently producing a new analysis. `--refresh-analysis` explicitly | |
| replaces cached outputs, while `--offline-analysis` requires cache hits. Gold act, | |
| intent and support labels are never sent to the analyser. | |
| `--seed` seeds Python, NumPy, Torch and CUDA; query analysis uses temperature zero | |
| and the seed, and its first response is then frozen in the cache. Torch inference | |
| uses evaluation mode and deterministic algorithms; unsupported deterministic | |
| operations fail rather than silently falling back. Stable tie-breaking and exact | |
| dense search avoid dependence on set iteration or ANN ties. Identical reruns still | |
| require unchanged corpus contents, model revisions, package versions and hardware. | |
| Freeze the Qdrant/Neo4j data during evaluation. Cached Gemini output cannot make a | |
| changed corpus or model checkpoint equivalent to an earlier run. | |
| Completed runs write these files under `--output-dir` (default `results`): | |
| - `retrieval_ablation.csv`: macro estimates and separate 95% interval bounds; | |
| - `retrieval_ablation.md`: the table, feature flags, intent breakdown and worst 10; | |
| - `retrieval_ablation_by_intent.csv`: the same metrics grouped by observed intent; | |
| - `retrieval_ablation_worst10.json`: the worst query/configuration pairs with | |
| questions, expected/alternate IDs, ranked IDs and graph IDs; | |
| - `retrieval_ablation_details.jsonl`: per-question retrieval outputs for auditing; | |
| - `retrieval_ablation_run.json`: seed, gold-file hash, candidate/depth settings, | |
| model revisions when available, package versions, feature flags and cache usage. | |
| Worst cases are sorted by ascending root Recall@10, root nDCG@10, exact Recall@10 | |
| and exact nDCG@10, then configuration and item ID. The same question can appear | |
| under multiple failing configurations. Full Recall@10 with an imperfect nDCG@10 | |
| ranking is also included among failures. No incomplete query run is summarized as | |
| a completed benchmark. | |
| ## Full-pipeline answer evaluation | |
| ```bash | |
| python -m eval.run_answer_eval | |
| python -m eval.run_answer_eval --gold eval/gold.jsonl --output-dir results | |
| ``` | |
| This runner loads one pipeline, runs all reviewed questions (including | |
| out-of-scope items), and closes its clients after the batch. Unreviewed items | |
| are excluded. An empty reviewed set stops before loading clients or weights; | |
| failed batches produce no completed metrics. The current dataset has no reviewed | |
| items, so `results/answer_eval.md` records **not run** without numerical scores. | |
| Completed runs write `answer_eval.json`, `answer_eval.md`, and | |
| `answer_eval_details.jsonl`. Details retain answers, expected IDs, citation checks, | |
| attempts, evaluation scores, and refusal decisions for inspection. | |
| The deterministic verifier recognizes section/article labels, common abbreviations, | |
| act-first forms such as `BNSS 45(2)`, canonical chunk IDs, and legacy Constitution | |
| IDs. It expands bounded numeric ranges. Unknown acts, ambiguous bare sections, | |
| and invalid or excessively large ranges remain unsupported. A bare article maps | |
| to the Constitution; a bare section may use the unique matching retrieved act. | |
| Root citations can match descendant chunks. Specific subsections require an | |
| appropriate chunk or subsection markers inside the retrieved parent passage. | |
| Requested graph IDs, queries and discarded vector candidates are not evidence. | |
| These checks establish citation presence, not legal-claim entailment. | |
| Metrics use these denominators: | |
| - Citation precision: supported citation occurrences / all citation occurrences. | |
| Repeated mentions count separately. Macro precision averages citation-bearing | |
| answers; answers without citations are counted separately, with null precision. | |
| - Unsupported-citation rate: unsupported occurrences / all occurrences. | |
| Unsupported-answer rate is the fraction of answers with any unsupported citation. | |
| - Answer-contains-expected-provision rate: answerable questions whose answers | |
| mention all expected root provisions / answerable questions. A reviewed alternate | |
| may substitute for a single expected ID. The supported counterpart additionally | |
| requires those mentions to pass the context check. | |
| - Out-of-scope refusal rate: recognized refusals / reviewed out-of-scope questions. | |
| Canonical abstention and explicit inability/out-of-scope wording count; low | |
| confidence alone does not. Inspect the saved decisions for wording missed by | |
| this deterministic heuristic. Absent categories have null metrics. | |
| In production, any unsupported citation prevents PASS. Its adjusted score is | |
| the raw judge score multiplied by citation precision, capped at | |
| `max(0, PASS_THRESHOLD - 1)`; the original score remains `llm_overall_score`. | |
| The failure stage becomes `CITATION` and uses the existing retry budget. On | |
| exhaustion, the best attempt retains its own citation report and warnings. | |
| The `citation_check` stage is logged, and its duration contributes to API | |
| `latency_ms.judge`. | |
| ## Validate the LLM judge against manual labels | |
| ```bash | |
| # Requires at least 50 reviewed questions and a running full pipeline: | |
| python -m eval.judge_validation export --seed 42 | |
| # Fill only my_label and my_notes in results/judge_validation.csv. | |
| # my_label must be correct, partially, or incorrect; leave uncertain rows blank. | |
| python -m eval.judge_validation consistency | |
| python -m eval.judge_validation report | |
| ``` | |
| Export samples 50 reviewed questions without replacement (including reviewed | |
| out-of-scope items) and runs the full pipeline once per selected question. The | |
| seed controls question selection, not model generation. `--count` changes the | |
| sample size. Insufficient reviewed questions stop before model loading; no | |
| synthetic answers or labels are substituted. Existing export paths are protected; | |
| use new `--csv` and `--samples` paths for another sample. | |
| The CSV columns are `id, question, answer, context_snippet, my_label, my_notes`. | |
| Judge scores are hidden from this labeling file. Snippets contain up to 4,000 | |
| context characters with an explicit truncation notice. Inspect the full context | |
| in `results/judge_validation_samples.jsonl` when needed. That snapshot retains | |
| the final attempt's original question, expanded judge question, answer, formatted | |
| judge context, raw overall score, correctness score and model. These files contain | |
| question/answer/source text; they are local evaluation artifacts ignored by Git. | |
| Agreement uses the saved original judge, with a fixed mapping declared before | |
| labeling: correctness 0–1 = incorrect, 2 = partially, 3–4 = correct. It reports | |
| accuracy, unweighted Cohen's kappa, and a confusion matrix (rows human, columns | |
| judge). Pearson and Spearman correlations compare original **raw** overall_score | |
| with manual labels encoded incorrect=0, partially=1, correct=2. Citation penalties | |
| are excluded from that raw score. Blank labels are excluded and counted; unknown | |
| labels, changed source columns, duplicate IDs and missing rows are rejected. | |
| Constant inputs or insufficient observations produce undefined correlations; | |
| kappa is undefined when chance agreement is one. | |
| Consistency makes exactly three fresh judge calls per frozen answer at temperature | |
| zero, through the production prompt and saved full context. It does not rerun | |
| retrieval/generation or send human labels to Gemini. This step needs only Gemini | |
| configuration, not Qdrant/Neo4j/Kanoon credentials. Results are saved in | |
| `results/judge_consistency.json` and bound to the sample hash. Re-running the | |
| consistency command replaces those measurements with three new calls per answer. | |
| The report shows the current judge model, per-answer scores/sample variance | |
| (ddof=1), mean variance and maximum variance over all sampled answers. Model | |
| changes may affect comparisons; temperature zero does not guarantee identical | |
| outputs. | |
| `report` runs offline and writes `results/judge_validation.md`. It can report | |
| agreement before consistency has run, explicitly marking consistency unmeasured. | |
| Use `--consistency`, `--output`, `--csv`, and `--samples` to select other paths. | |
| The current gold file has no reviewed questions, so the checked-in report records | |
| **not run**; manual agreement and live variance have not been measured. | |
| ## Paired pipeline latency and quality benchmark | |
| ```bash | |
| python -m eval.profile_pipeline --count 50 --seed 42 --workers 4 | |
| ``` | |
| This samples 50 reviewed gold questions without replacement and runs the same | |
| questions through three switch configurations on one loaded pipeline: | |
| | Mode | Vector I/O | Graph | Analysis/embedding caches | | |
| |---|---|---|---| | |
| | before | Serial | Three calls per root | Disabled | | |
| | after_cold | Concurrent (4 workers by default) | One batched query | Empty at batch start, then filled | | |
| | after_warm | Concurrent | One batched query | Retained from after_cold | | |
| The script writes `results/latency.md` and measured rows to `results/latency.json` | |
| only after all three batches complete. The Markdown table reports p50/p95 for | |
| analysis, embedding, vector search, reranking, Neo4j, generation, judge and total. | |
| Embedding is subtracted from the enclosing vector stage to report exclusive I/O | |
| wall time. Timings sum retries per question; startup is measured separately. | |
| Other work (judgments, citation checks and orchestration) remains in total time. | |
| This is an in-process switch baseline with the same retry policy, not a saved | |
| historical deployment. The fixed mode order and shared server/model caches can | |
| affect differences; warm-cache latency is for repeated questions, not new traffic. | |
| Quality is measured on each batch: macro root Recall@5 over answerable questions, | |
| using retained reranked sources, and micro citation precision over all citation | |
| occurrences. Single-provision approved alternates may substitute; all roots of | |
| multi-provision questions remain required. Undefined denominators cannot pass a | |
| quality gate. A lower Recall@5 or citation precision is explicitly flagged | |
| `REGRESSION`; the report does not recommend that configuration as a speedup. | |
| Model generation can vary, so this sample is not a statistical equivalence proof. | |
| Freeze corpus/model versions before comparing runs. | |
| Runtime switches are `VECTOR_SEARCH_WORKERS` (1–16, default 1), | |
| `BATCH_GRAPH_EXPANSION` (default false), and `ENABLE_QUERY_CACHE` (default false). | |
| Concurrent workers perform Qdrant I/O only; embedding remains sequential to avoid | |
| concurrent access to the model. Results merge in original query order to preserve | |
| tie behavior. Any failed search fails the query rather than dropping candidates. | |
| The graph query uses three UNION branches to retain the legacy result structure | |
| and traversal directions without a Cartesian product. Live graph equivalence | |
| still needs verification on the restored corpus. | |
| Caches are bounded, thread-safe, instance-local LRUs (512 entries each). They | |
| store analysis and query embeddings in memory, keyed by whitespace-normalized | |
| queries while preserving case and punctuation. Analysis keys also include model, | |
| prompt and seed; embedding keys include the model. Returned values are copied, | |
| failed computations are not stored, and process restart clears caches. No disk | |
| query cache or new secret-bearing artifact is created. | |
| Pipeline Gemini calls, Qdrant searches/scrolls and Neo4j `run_query` calls retry | |
| transient connectivity, throttling and server errors with tenacity: three total | |
| attempts, 0.25 then 0.5 seconds exponential backoff. Authentication/validation | |
| failures do not retry. Existing `EXTERNAL_TIMEOUT_SECONDS` SDK connection/query | |
| deadlines still apply **per attempt**; retries can therefore take roughly three | |
| times that budget plus backoff. Generation retries may incur additional provider | |
| usage if a response is lost. Graph writes and collection mutations are not given | |
| automatic retries. Health probes retain their own bounded, immediate checks. | |
| `SKIP_JUDGE_HIGH_CONFIDENCE=false` remains mandatory. Setting it true fails before | |
| loading models: retrieval confidence has not yet been calibrated and no reviewed | |
| comparison establishes unchanged citation precision/answer quality. The benchmark | |
| marks judge skipping **not measured** and always runs the judge. The current gold | |
| file has zero reviewed questions and the database corpus was removed, so the | |
| checked-in latency report contains no invented p50/p95 or quality results. | |