docs/validation.md

# Validation checklist

Release checks prevent software, deployment, and provenance mistakes; they do not pretend the analytical profile has measured-hardware fidelity.

## Automated software checks

- deterministic constant/Poisson/bursty workload generation
- exact trace replay and trace validation
- prefill latency monotonicity
- quantization footprint ordering
- latency sensitivity-scale behavior
- colocated end-to-end completion
- static vs continuous behavioral difference
- component TTFT/E2E SLO accounting
- robust repetition-aware capacity search
- bottleneck-diagnosis provenance
- prefix hits do not alter the underlying generated request trace
- prefix reuse reduces modeled prefill work
- P/D pipeline completion and transfer telemetry
- configurable P/D worker counts
- four-scenario topology/cache comparison
- bounded design-space sweep and two Pareto objectives
- paired common-seed research study
- bootstrap paired-effect intervals
- analytical-profile perturbation study
- external-measurement import, normalization, and validation report generation
- train-only calibration with held-out evaluation when enough cases exist
- repeated-seed robust policy ranking and bootstrap intervals
- Pareto-stability accounting across matched seeds
- bounded full-trace oracle candidate search and policy-regret accounting
- Markdown research-report generation with interpretation guardrails
- deterministic stateful agent-session trace generation
- zero cross-turn hits under immediate eviction
- KV reuse and reduced recomputation under retention + affinity
- common-trace four-policy agent comparison
- TTL latency/memory Pareto frontier
- ASCII-only public UI labels
- chart export controls present
- explicit planner worst-repetition and target columns
- no product-style footer
- Hugging Face `short_description` <= 60 characters
- `sdk: static` metadata
- canonical Python source equals browser mirror
- every browser Python module is included by the worker
- provenance remains `analytical-reference`
- JavaScript syntax parse
- Python compilation

## Empirical validation protocol

A future hardware run should be stored as external validation cases, not copied into analytical profiles without provenance. At minimum each case should record:

- model / revision
- serving topology and scheduler
- accelerator / count
- precision
- workload arrival process or trace ID
- prompt/output distribution or exact trace
- SLO definition
- p95 TTFT / p95 E2E / goodput
- runtime/software versions

For already-normalized cases run:

```bash
python scripts/validate_measurements.py measured_cases.json --output validation_report.json
```

For serving-tool artifacts run:

```bash
python scripts/import_measurements.py result.json --source auto --output measurement_cases.json
python scripts/calibrate_profiles.py measurement_cases.json --holdout 0.33 --output calibration_result.json
```

The calibration script fits only on the training cases and reports baseline/calibrated residuals on held-out cases when at least three cases are available. A smaller case set is explicitly marked as resubstitution rather than held-out validation.

## Not claimed

- empirical L4/A10G/A100 latency accuracy
- exact vLLM/SGLang scheduler equivalence
- CUDA-kernel modeling
- real network-protocol fidelity
- radix-tree prefix-cache eviction/scheduling
- speculative decoding
- Attention-FFN disaggregation
- production-fidelity agentic serving or dynamic batching inside Stateful Sessions
- real tool execution or production agent-runtime integration
- claims that the bounded offline oracle is globally optimal
- claims that simulator bootstrap intervals quantify real-hardware error

These are explicit scope boundaries, not hidden assumptions.