Download eval/README.md from sai-Rohan/Legora: direct link, hf CLI and curl.
- Browser
- Download file 27.3 kB
-
https://huggingface.co/spaces/sai-Rohan/Legora/resolve/main/eval/README.md
- Command line
-
hf download hf://spaces/sai-Rohan/Legora/eval/README.md
-
curl -L -o README.md https://huggingface.co/spaces/sai-Rohan/Legora/resolve/main/eval/README.md
Gold-set workflow
The default target is 145 items: 30 each from BNS, BNSS, BSA and the
Constitution (120), plus 15 multi-provision questions and 10 out-of-scope questions.
All Gemini output is draft material with reviewed=false, not verified gold.
Current checked-in gold.jsonl: 10 Gemini-generated out-of-scope drafts.
The 135 provision-backed items have not been generated because the stored corpus
is unavailable following deletion of the database containers. No provision labels
or metrics were invented to fill that gap. Restore/re-ingest the corpus, or use
an existing stored-chunk export, before completing the remaining items.
1. Capture stored chunks
The tools use Python 3.11+ and the project's existing dependencies. Run from the
repository root. Evaluation settings are read centrally from .env/environment
variables and do not require Neo4j or Indian Kanoon credentials.
For the unified Qdrant collection:
python -m eval.sample --collection legal_rag
For separate collections:
python -m eval.sample --collection bns --collection bnss --collection bsa --collection constitution
For an existing JSON array, Qdrant {"points": [...]} object, or JSONL export:
python -m eval.sample --chunks /path/to/stored-chunks.jsonl
Export records may be payload dictionaries (chunk_id, text, ...) or objects
containing a payload dictionary. This must be real stored corpus data; the
parser test fixtures are not a substitute for the corpus. QDRANT_URL and an
optional QDRANT_API_KEY are needed only for live sampling.
The snapshot eval/provisions.jsonl retains every recognized source chunk for
review, including child chunks. Sampling uses 30 distinct section/article roots
per act, preferring full parent provisions to avoid overweighting sections with
many children. It is uniform within each act, reproducible with --seed 42, and
fails if any act has fewer than 30 distinct provisions. Chapter/preamble/schedule
records are not sampled. Existing snapshots are not silently overwritten.
2. Draft questions with Gemini
python -m eval.generate
# Optional: cap new items in a run, then resume with the same command/settings.
python -m eval.generate --limit 10
Requires GEMINI_API_KEY; uses GEMINI_MODEL and EXTERNAL_TIMEOUT_SECONDS.
No embedder, reranker, graph, or API server is needed to draft from the snapshot.
The output defaults to eval/gold.jsonl. Successful drafts are saved atomically
after each item. Existing items, including human edits/approvals, are preserved
on resume. gold.manifest.json binds full generation to the snapshot hash, seed,
and target counts. Use a new output path to build a different dataset.
Each source has a preferred intent cycling through punishment, definition, procedure, evidence and rights. Gemini must choose a supported intent rather than invent a punishment or right absent from the provision. Intent coverage therefore needs human review; no unsupported per-act intent quota is forced.
Multi-provision prompts receive multiple sampled sources and must identify at least two distinct supporting provisions. Unsupported combinations and unknown IDs are rejected and retried up to three times. Generated notes explain source support and ambiguity for the reviewer. Individual source text in prompts is limited to 12,000 characters, while the review snapshot retains the full text. Review long/truncated provisions carefully. Gemini grounding is not a legal correctness guarantee.
The independent refusal items can be drafted without a corpus:
python -m eval.generate --out-of-scope-only
They use ordinary nonlegal topics (cooking, programming, astronomy, etc.), empty
supporting ID lists and answerable=false. They also require human review.
3. Review each item
Generation can resume after an interrupted request without replacing earlier
drafts or approvals. python -m eval.generate --limit 10 --request-interval 10
adds at most ten new drafts with requests spaced at least ten seconds apart.
Pacing cannot override a provider's project/daily quota. Provider failures now
report the HTTP code with fixed guidance rather than just ClientError; raw
provider responses, keys and prompts are never printed. Rate/quota and permanent
client errors stop immediately; transient server/timeouts use bounded backoff.
Resolve the reported provider issue before resuming.
Use python -m eval.review --answerable-only to review legal questions before
out-of-scope items. Press s to inspect source text, then a only after checking
the question/support. x exits without approving. Generation never marks drafts
reviewed, and evaluations intentionally stop until eligible approvals exist.
python -m eval.review --list
python -m eval.review
# Inspect a specific item:
python -m eval.review --id oos-001
Interactive commands display sources, edit the question, intent, expected IDs,
acceptable alternate IDs or notes, skip an item, and approve it. Approval is an
explicit human action. Any subsequent edit resets reviewed=false. Answerable
items cannot be approved unless their expected and alternate IDs exist in the
snapshot. Single-act items must reference that act.
Noninteractive review is also supported (replace the example ID with an actual stored ID supporting the displayed question):
python -m eval.review --id single-bns-001 --expected 'BNS-303(2)' --notes 'Reviewer explanation'
python -m eval.review --id single-bns-001 --approve
Every JSONL record has id, question, intent, act, expected_chunk_ids,
acceptable_alternates, answerable, notes, and reviewed. Acts are BNS,
BNSS, BSA, Constitution, MULTI, or OUT_OF_SCOPE. Questions with act=MULTI
need at least two expected IDs. Out-of-scope records must have no supporting IDs.
4. Predict and measure reviewed items only
Once the API and its dependencies are running:
python -m eval.predict
python -m eval.metrics --output eval/metrics.json
predict submits only reviewed questions to API_BASE_URL, with
API_TIMEOUT_SECONDS. It saves eval/predictions.jsonl only after all requests
succeed; transport failures do not become fabricated results. Predictions have
id, retrieved_chunk_ids, score (0–100), and refused (boolean).
You may supply an existing prediction file with --predictions PATH for offline
metrics. Recollect predictions after editing questions.
The API has no structured refusal field. predict recognizes only the pipeline's
canonical “The provided legal corpus does not contain sufficient information to
answer this question” sentence; other refusal wording needs an explicit human
refused=true label. Low confidence is independently measured using
PASS_THRESHOLD (override with metrics --threshold).
Metrics reject a dataset with zero reviewed records. Unreviewed predictions are ignored before validation/scoring. Missing predictions for reviewed records count as unsuccessful, not silently omitted. Reported metrics are retrieval hit rate, all-required-provisions rate, mean provision recall, mean citation precision and out-of-scope refusal/low-confidence success rate. Absent categories produce null. They measure retrieval and scope handling, not legal answer correctness.
For single-provision questions, a reviewed alternate can satisfy the expected source. For multi-provision questions, all expected IDs remain required for full coverage; flat alternates count toward hit rate and precision but cannot represent complete alternative multi-source sets. Edit the expected set during review if the canonical multi-provision support changes.
python -m pytest tests/test_eval.py
These tests use synthetic sources only in temporary test files and mock the Gemini boundary. Sampling, draft validation, resume, review and metric logic run without network calls.
Retrieval ablation benchmark
python -m eval.run_retrieval_eval
# Evaluate a subset without loading unused dependencies:
python -m eval.run_retrieval_eval --configs A B C --seed 42
# Reuse cached analysis and forbid Gemini calls:
python -m eval.run_retrieval_eval --offline-analysis
eval/run_retrieval_eval.py accepts only reviewed=true, answerable=true items.
It refuses an empty eligible set before initializing clients or loading weights.
The current gold file has no eligible items, so the checked-in
results/retrieval_ablation.csv is header-only and the Markdown report says
“not run.” They contain no benchmark scores. An empty eligible set or a failed retrieval call does not
overwrite earlier completed reports.
Every preset uses the same retrieval path and feature switches:
| Preset | Features |
|---|---|
| A | Original question, dense search, no act filter or query analysis |
| B | A plus the primary-act filter from query analysis |
| C | B plus build_retrieval_queries expansion, retaining the original question |
| D | C plus BAAI/bge-reranker-large, without additive legal boosts |
| E | D plus the existing legal-boost heuristics |
| F | E plus graph expansion from the top 10 ranked results |
| G | E plus BM25/dense RRF (k=60) and explicit provision-ID lookup |
D and E use identical candidate pools and the existing analysis-enriched rerank query; only the additive legal-boost switch differs. F preserves E's ranked list. G keeps graph expansion off to isolate the new retrieval path against E. Analysis receives only the question, never the expected IDs, gold act or intent. An unknown primary act stays unfiltered, as in the production helper.
--configs A B ... selects presets. Individual switches can be overridden with
--act-filter/--no-act-filter, --multi-query/--no-multi-query,
--rerank/--no-rerank, --legal-boost/--no-legal-boost and
--graph/--no-graph. Legal boost requires reranking. For example:
python -m eval.run_retrieval_eval --configs E --no-legal-boost --no-graph
--bm25/--no-bm25 and --exact-id-lookup/--no-exact-id-lookup control the
new stages independently. Compare E and G, or disable lookup to isolate BM25/RRF:
python -m eval.run_retrieval_eval --configs E G --offline-analysis
python -m eval.run_retrieval_eval --configs G --no-exact-id-lookup
G builds one BM25Okapi index by scrolling payloads from the same Qdrant collection,
without vectors. It searches the original question, applies the inferred act
filter, and fuses its ranked hits with the merged dense ranking using
sum(1 / (60 + rank)). Each ID receives at most one contribution per stream.
Only chunks with an overlapping query token enter the lexical ranking; negative
BM25 scores remain valid matches. Both streams are capped to the common candidate
budget, and fusion is capped again before reranking. The run manifest records
the RRF constant, lexical query choice, flags and rank-bm25 package version.
Explicit references such as section 303 BNS, BNSS section 35, and
article 21 Constitution trigger payload chunk_id equality lookups. A bare
section 303 does not identify an act and causes no exact lookup. Qualified
multi-act references and existing legacy Constitution IDs are supported. Only
actual stored hits are prepended; nonexistent IDs are not synthesized. Exact
hits bypass a potentially wrong inferred act filter and are pinned after
reranking as well as before it, within the same candidate/output limits.
BM25 is not enabled by default in production. The current table marks G as not evaluated because no reviewed, answerable gold items are available and the database corpus was removed. There is no measured Recall@5/MRR improvement to justify changing the default. Compare E/G on reviewed labels after restoring the corpus; the script never edits production defaults automatically.
Overrides are recorded in the run manifest and the report's feature table, so a customized preset is not mistaken for the default configuration.
--candidates 50 is both the per-query search limit and the merged candidate
budget, with a minimum of 10. Multi-query hits are deduplicated by exact chunk ID,
keeping the highest dense score, then sorted by descending score and chunk ID and
capped to that common budget before reranking. Thus expansion does not also give
the reranker an unbounded larger budget. Dense search defaults to Qdrant exact
search for reproducibility; --approximate-search uses production ANN search.
Production retrieval defaults remain unchanged.
All models are loaded once per run. A needs only the embedder and Qdrant. B–G
need Gemini credentials only on a cache miss; only D–G load the reranker, and
graph-enabled configurations (F by default) need NEO4J_*. No Indian Kanoon
requests or answer-generation calls are made.
Use --collection, --embedder-model and --reranker-model to select a frozen
corpus and pinned/local model snapshots.
Measures and confidence intervals
The report provides two independent scoring modes:
- exact: literal stored chunk ID equality;
- root: section/article identity, e.g.
BSA-115(2)(a)→BSA-115,BNSS-45-EXPL-1→BNSS-45, andarticle_21_clause_1→CONST-21.
Recall@1/5/10 is the fraction of required targets recovered by that rank, macro averaged over questions. MRR uses the first relevant result anywhere in the retained candidate budget (default depth 50). nDCG@10 uses binary relevance and the ideal ranking of the required targets. Repeated exact/root hits retain their rank positions but receive credit only once. Root scoring collapses expected chunks from the same provision to one target.
A reviewed single-provision alternate may satisfy that target. Flat alternates do not specify which required source they replace in a multi-provision question, so multi-provision ranking metrics use the canonical expected set only.
For F, context recall measures target coverage in actual returned graph nodes. combined context recall measures coverage in graph nodes plus the top-10 retrieved chunks. Graph lookup keys, requested seed IDs and IDs merely mentioned in source text do not count as returned graph evidence. Exact graph coverage uses literal node/chunk IDs; root coverage handles graph/stored-ID format differences. Graph columns are blank/dashes for presets that do not run graph expansion.
The default confidence intervals use 2,000 query-level bootstrap resamples and
the 2.5th/97.5th percentiles. Resample indices are seeded and paired across
configurations/scoring modes for the same question group. Per-intent intervals
resample within each observed intent. Singleton groups have degenerate intervals.
These intervals describe query-sample variation, not model/corpus drift or legal
answer correctness. Override the count with --bootstrap-samples.
Caching, reproducibility and output
Query analyses are cached under results/analysis_cache/ (ignored by Git). Keys
include the exact question, Gemini model, seed, analyser code/prompt fingerprint
and analysis schema. Cached records are schema-validated; a corrupt cache fails
instead of silently producing a new analysis. --refresh-analysis explicitly
replaces cached outputs, while --offline-analysis requires cache hits. Gold act,
intent and support labels are never sent to the analyser.
--seed seeds Python, NumPy, Torch and CUDA; query analysis uses temperature zero
and the seed, and its first response is then frozen in the cache. Torch inference
uses evaluation mode and deterministic algorithms; unsupported deterministic
operations fail rather than silently falling back. Stable tie-breaking and exact
dense search avoid dependence on set iteration or ANN ties. Identical reruns still
require unchanged corpus contents, model revisions, package versions and hardware.
Freeze the Qdrant/Neo4j data during evaluation. Cached Gemini output cannot make a
changed corpus or model checkpoint equivalent to an earlier run.
Completed runs write these files under --output-dir (default results):
retrieval_ablation.csv: macro estimates and separate 95% interval bounds;retrieval_ablation.md: the table, feature flags, intent breakdown and worst 10;retrieval_ablation_by_intent.csv: the same metrics grouped by observed intent;retrieval_ablation_worst10.json: the worst query/configuration pairs with questions, expected/alternate IDs, ranked IDs and graph IDs;retrieval_ablation_details.jsonl: per-question retrieval outputs for auditing;retrieval_ablation_run.json: seed, gold-file hash, candidate/depth settings, model revisions when available, package versions, feature flags and cache usage.
Worst cases are sorted by ascending root Recall@10, root nDCG@10, exact Recall@10 and exact nDCG@10, then configuration and item ID. The same question can appear under multiple failing configurations. Full Recall@10 with an imperfect nDCG@10 ranking is also included among failures. No incomplete query run is summarized as a completed benchmark.
Full-pipeline answer evaluation
python -m eval.run_answer_eval
python -m eval.run_answer_eval --gold eval/gold.jsonl --output-dir results
This runner loads one pipeline, runs all reviewed questions (including
out-of-scope items), and closes its clients after the batch. Unreviewed items
are excluded. An empty reviewed set stops before loading clients or weights;
failed batches produce no completed metrics. The current dataset has no reviewed
items, so results/answer_eval.md records not run without numerical scores.
Completed runs write answer_eval.json, answer_eval.md, and
answer_eval_details.jsonl. Details retain answers, expected IDs, citation checks,
attempts, evaluation scores, and refusal decisions for inspection.
The deterministic verifier recognizes section/article labels, common abbreviations,
act-first forms such as BNSS 45(2), canonical chunk IDs, and legacy Constitution
IDs. It expands bounded numeric ranges. Unknown acts, ambiguous bare sections,
and invalid or excessively large ranges remain unsupported. A bare article maps
to the Constitution; a bare section may use the unique matching retrieved act.
Root citations can match descendant chunks. Specific subsections require an
appropriate chunk or subsection markers inside the retrieved parent passage.
Requested graph IDs, queries and discarded vector candidates are not evidence.
These checks establish citation presence, not legal-claim entailment.
Metrics use these denominators:
- Citation precision: supported citation occurrences / all citation occurrences. Repeated mentions count separately. Macro precision averages citation-bearing answers; answers without citations are counted separately, with null precision.
- Unsupported-citation rate: unsupported occurrences / all occurrences. Unsupported-answer rate is the fraction of answers with any unsupported citation.
- Answer-contains-expected-provision rate: answerable questions whose answers mention all expected root provisions / answerable questions. A reviewed alternate may substitute for a single expected ID. The supported counterpart additionally requires those mentions to pass the context check.
- Out-of-scope refusal rate: recognized refusals / reviewed out-of-scope questions. Canonical abstention and explicit inability/out-of-scope wording count; low confidence alone does not. Inspect the saved decisions for wording missed by this deterministic heuristic. Absent categories have null metrics.
In production, any unsupported citation prevents PASS. Its adjusted score is
the raw judge score multiplied by citation precision, capped at
max(0, PASS_THRESHOLD - 1); the original score remains llm_overall_score.
The failure stage becomes CITATION and uses the existing retry budget. On
exhaustion, the best attempt retains its own citation report and warnings.
The citation_check stage is logged, and its duration contributes to API
latency_ms.judge.
Validate the LLM judge against manual labels
# Requires at least 50 reviewed questions and a running full pipeline:
python -m eval.judge_validation export --seed 42
# Fill only my_label and my_notes in results/judge_validation.csv.
# my_label must be correct, partially, or incorrect; leave uncertain rows blank.
python -m eval.judge_validation consistency
python -m eval.judge_validation report
Export samples 50 reviewed questions without replacement (including reviewed
out-of-scope items) and runs the full pipeline once per selected question. The
seed controls question selection, not model generation. --count changes the
sample size. Insufficient reviewed questions stop before model loading; no
synthetic answers or labels are substituted. Existing export paths are protected;
use new --csv and --samples paths for another sample.
The CSV columns are id, question, answer, context_snippet, my_label, my_notes.
Judge scores are hidden from this labeling file. Snippets contain up to 4,000
context characters with an explicit truncation notice. Inspect the full context
in results/judge_validation_samples.jsonl when needed. That snapshot retains
the final attempt's original question, expanded judge question, answer, formatted
judge context, raw overall score, correctness score and model. These files contain
question/answer/source text; they are local evaluation artifacts ignored by Git.
Agreement uses the saved original judge, with a fixed mapping declared before labeling: correctness 0–1 = incorrect, 2 = partially, 3–4 = correct. It reports accuracy, unweighted Cohen's kappa, and a confusion matrix (rows human, columns judge). Pearson and Spearman correlations compare original raw overall_score with manual labels encoded incorrect=0, partially=1, correct=2. Citation penalties are excluded from that raw score. Blank labels are excluded and counted; unknown labels, changed source columns, duplicate IDs and missing rows are rejected. Constant inputs or insufficient observations produce undefined correlations; kappa is undefined when chance agreement is one.
Consistency makes exactly three fresh judge calls per frozen answer at temperature
zero, through the production prompt and saved full context. It does not rerun
retrieval/generation or send human labels to Gemini. This step needs only Gemini
configuration, not Qdrant/Neo4j/Kanoon credentials. Results are saved in
results/judge_consistency.json and bound to the sample hash. Re-running the
consistency command replaces those measurements with three new calls per answer.
The report shows the current judge model, per-answer scores/sample variance
(ddof=1), mean variance and maximum variance over all sampled answers. Model
changes may affect comparisons; temperature zero does not guarantee identical
outputs.
report runs offline and writes results/judge_validation.md. It can report
agreement before consistency has run, explicitly marking consistency unmeasured.
Use --consistency, --output, --csv, and --samples to select other paths.
The current gold file has no reviewed questions, so the checked-in report records
not run; manual agreement and live variance have not been measured.
Paired pipeline latency and quality benchmark
python -m eval.profile_pipeline --count 50 --seed 42 --workers 4
This samples 50 reviewed gold questions without replacement and runs the same questions through three switch configurations on one loaded pipeline:
| Mode | Vector I/O | Graph | Analysis/embedding caches |
|---|---|---|---|
| before | Serial | Three calls per root | Disabled |
| after_cold | Concurrent (4 workers by default) | One batched query | Empty at batch start, then filled |
| after_warm | Concurrent | One batched query | Retained from after_cold |
The script writes results/latency.md and measured rows to results/latency.json
only after all three batches complete. The Markdown table reports p50/p95 for
analysis, embedding, vector search, reranking, Neo4j, generation, judge and total.
Embedding is subtracted from the enclosing vector stage to report exclusive I/O
wall time. Timings sum retries per question; startup is measured separately.
Other work (judgments, citation checks and orchestration) remains in total time.
This is an in-process switch baseline with the same retry policy, not a saved
historical deployment. The fixed mode order and shared server/model caches can
affect differences; warm-cache latency is for repeated questions, not new traffic.
Quality is measured on each batch: macro root Recall@5 over answerable questions,
using retained reranked sources, and micro citation precision over all citation
occurrences. Single-provision approved alternates may substitute; all roots of
multi-provision questions remain required. Undefined denominators cannot pass a
quality gate. A lower Recall@5 or citation precision is explicitly flagged
REGRESSION; the report does not recommend that configuration as a speedup.
Model generation can vary, so this sample is not a statistical equivalence proof.
Freeze corpus/model versions before comparing runs.
Runtime switches are VECTOR_SEARCH_WORKERS (1–16, default 1),
BATCH_GRAPH_EXPANSION (default false), and ENABLE_QUERY_CACHE (default false).
Concurrent workers perform Qdrant I/O only; embedding remains sequential to avoid
concurrent access to the model. Results merge in original query order to preserve
tie behavior. Any failed search fails the query rather than dropping candidates.
The graph query uses three UNION branches to retain the legacy result structure
and traversal directions without a Cartesian product. Live graph equivalence
still needs verification on the restored corpus.
Caches are bounded, thread-safe, instance-local LRUs (512 entries each). They store analysis and query embeddings in memory, keyed by whitespace-normalized queries while preserving case and punctuation. Analysis keys also include model, prompt and seed; embedding keys include the model. Returned values are copied, failed computations are not stored, and process restart clears caches. No disk query cache or new secret-bearing artifact is created.
Pipeline Gemini calls, Qdrant searches/scrolls and Neo4j run_query calls retry
transient connectivity, throttling and server errors with tenacity: three total
attempts, 0.25 then 0.5 seconds exponential backoff. Authentication/validation
failures do not retry. Existing EXTERNAL_TIMEOUT_SECONDS SDK connection/query
deadlines still apply per attempt; retries can therefore take roughly three
times that budget plus backoff. Generation retries may incur additional provider
usage if a response is lost. Graph writes and collection mutations are not given
automatic retries. Health probes retain their own bounded, immediate checks.
SKIP_JUDGE_HIGH_CONFIDENCE=false remains mandatory. Setting it true fails before
loading models: retrieval confidence has not yet been calibrated and no reviewed
comparison establishes unchanged citation precision/answer quality. The benchmark
marks judge skipping not measured and always runs the judge. The current gold
file has zero reviewed questions and the database corpus was removed, so the
checked-in latency report contains no invented p50/p95 or quality results.