Spaces:
Sleeping
Sleeping
| # Hospital Copilot — Development Log | |
| **Hackathon:** Gemma 4 for Good | |
| **Team:** Ricky (fredrickandoh17@gmail.com) | |
| **Stack:** Python · Gradio · Gemma 4 · faster-whisper · ChromaDB · SQLite | |
| **Started:** 2026-05-16 | |
| --- | |
| ## Project Goal | |
| Build an AI clinical assistant that listens to doctor-patient consultations and automatically produces: | |
| - Live transcription of the conversation | |
| - Structured symptom extraction (symptoms, medications, duration, allergies, follow-up actions) | |
| - SOAP notes grounded with real ICD-10 codes and drug dosages | |
| - Plain-language patient summary | |
| - Structured patient records saved to a local database | |
| **Why:** Reduce doctor burnout from paperwork, improve care quality, and support healthcare workers in low-resource settings like Ghana. | |
| --- | |
| ## Architecture Overview | |
| ``` | |
| Microphone | |
| └─► faster-whisper (STT, local CPU) → raw transcript | |
| └─► Gemma 4 26B cloud (speaker labelling) → Doctor:/Patient: transcript | |
| ├─► Gemma 4 E2B via Ollama (symptom JSON) → local CPU | |
| └─► ChromaDB + MiniLM (RAG retrieval) → ICD-10 codes + drug info | |
| └─► Gemma 4 26B cloud (SOAP note, patient summary) | |
| └─► SQLite (patients, sessions, notes, symptoms) | |
| └─► Gradio UI | |
| ``` | |
| --- | |
| ## Features Implemented | |
| ### Core Pipeline | |
| | Feature | Status | Implementation | | |
| |---|---|---| | |
| | Live mic transcription | ✅ | faster-whisper `small` model, 3s chunks, VAD filter | | |
| | Speaker diarization | ✅ | Gemma 4 post-hoc Doctor:/Patient: labelling | | |
| | Symptom extraction | ✅ | Gemma 4 E2B via Ollama — JSON: chief complaint, symptoms, duration, severity, medications, allergies, vitals, history, follow-up actions | | |
| | RAG ICD-10 retrieval | ✅ | ChromaDB + all-MiniLM-L6-v2, 90+ Ghana-relevant codes | | |
| | RAG drug grounding | ✅ | ChromaDB, 40+ WHO Essential Medicines with dosages | | |
| | SOAP note generation | ✅ | Gemma 4 26B cloud, RAG context injected into prompt | | |
| | Patient summary | ✅ | Gemma 4 26B cloud, plain English | | |
| | Patient records (SQLite) | ✅ | patients, sessions, notes, symptoms tables | | |
| | Patient registration | ✅ | Name, DOB, gender, phone | | |
| | Records viewer | ✅ | Load any patient's most recent session | | |
| ### Translation (Twi/Akan) | |
| | Status | Note | | |
| |---|---| | |
| | ⏸️ Paused | Gemma 4 returned 500 INTERNAL errors on Twi translation. Identified root cause: Twi is a low-resource language and Gemma 4 is not purpose-built for it. Decision: implement NLLB-200 (Meta's No Language Left Behind model) which was specifically trained on Akan/Twi. Deferred until core pipeline is stable. | | |
| ### Gemma 4 Advanced Features (Added 2026-05-18) | |
| | Feature | Status | Implementation | | |
| |---|---|---| | |
| | **Reasoning mode (thinking)** | ✅ | `ThinkingConfig(thinking_budget=2048, include_thoughts=False)` on SOAP generation — Gemma 4 reasons step-by-step internally before writing the note | | |
| | **Function calling (symptom extraction)** | ✅ | `FunctionDeclaration` schema with `FunctionCallingMode.ANY` — guaranteed valid structured output, no JSON parsing | | |
| | **Multimodal image/document analysis** | ✅ | `Part.from_bytes()` with lab result / prescription images — extracted findings injected into SOAP context | | |
| --- | |
| ## Technical Decisions | |
| ### 1. Multi-agent Gemma 4 architecture | |
| **Decision:** Use multiple specialised Gemma 4 instances rather than one large model for everything. | |
| **Reasoning:** Different tasks have different speed/accuracy requirements: | |
| - Symptom extraction: needs to be fast, structured JSON → small local model (E2B) | |
| - SOAP notes: needs medical reasoning and long output → large cloud model (26B) | |
| - Speaker labelling: needs language understanding → cloud model | |
| - Embeddings: needs speed, runs every session → lightweight MiniLM locally | |
| ### 2. Local vs cloud split | |
| **Decision:** Run small models locally (Ollama E2B, Whisper, MiniLM, ChromaDB), large inference on cloud API. | |
| **Reasoning:** User has no GPU. CPU-only local inference is viable for small quantised models (Q4_K_M gemma4:e2b runs at ~5-10 tok/s). Large models (26B+) are impractical on CPU — cloud API provides them at acceptable latency. | |
| ### 3. RAG with ChromaDB + MiniLM | |
| **Decision:** Use local vector store over calling the cloud model with full knowledge base in prompt. | |
| **Reasoning:** | |
| - Injecting 70k ICD-10 codes into every prompt would exceed context limits and cost tokens | |
| - Local ChromaDB persists to disk, zero latency after first build | |
| - MiniLM-L6-v2 (~80MB) gives good semantic similarity for medical terms on CPU | |
| - Retrieves top-5 most relevant codes per consultation — keeps prompt tight and accurate | |
| ### 4. Gradio over Streamlit | |
| **Decision:** Use Gradio for the UI. | |
| **Reasoning:** Gradio has better support for streaming, audio, and timer-based polling. Streamlit's re-run model makes real-time transcript updates difficult. Gradio's `gr.Timer` makes 2-second polling trivial. | |
| ### 5. Gemma 4 reasoning mode — temperature requirement | |
| **Decision:** Set `temperature=1.0` when `thinking_config` is enabled, not `0.3`. | |
| **Reasoning:** Google's API requires temperature=1.0 when using ThinkingConfig — lower values raise an error. The thinking process itself introduces determinism so output quality is not degraded. Added graceful fallback: if the model doesn't support thinking (e.g. older model version), retry without `thinking_config`. | |
| ### 6. Function calling mode = ANY | |
| **Decision:** Use `FunctionCallingMode.ANY` (force the model to always call the function) rather than `AUTO`. | |
| **Reasoning:** `AUTO` mode allows the model to optionally use the function or just return text — unreliable for extraction tasks. `ANY` mode guarantees the model returns a structured function call every time, eliminating the JSON parse errors we had with the prompt-based approach. | |
| ### 7. Symptom extraction: local first, cloud fallback | |
| **Decision:** Keep Gemma 4 E2B (Ollama, local) as primary for symptom extraction, cloud function calling as fallback. | |
| **Reasoning:** Preserves the "local AI, privacy-preserving" story for the hackathon. Cloud fallback ensures reliability when Ollama returns malformed JSON or fails. Both paths return the same dict structure. | |
| ### 8. Transcript repair before downstream processing | |
| **Problem:** faster-whisper `small` on CPU makes errors — mishears medical terms, missing punctuation, run-on sentences. Downstream models (symptom extraction, SOAP generation) produce lower quality output when given a garbled transcript. | |
| **Decision:** Add a `clean_and_label_transcript()` step using Gemma 4 cloud that simultaneously repairs ASR errors AND labels speakers in one API call. This runs after `stop_consultation()` before any downstream processing. | |
| **What it fixes:** Incorrect drug names, missing punctuation, filler words (um/uh), run-on sentences, garbled medical terminology. | |
| **What it preserves:** All clinical facts — symptoms, medications, durations, dosages. Never adds or invents information. | |
| **Why one call:** Combining repair + labelling saves one API round-trip and is cheaper than two separate calls. | |
| ### 9. Speaker diarization: Gemma 4 post-hoc vs pyannote-audio | |
| **Decision:** Use Gemma 4 cloud to infer Doctor/Patient labels from transcript text. | |
| **Reasoning:** | |
| - `pyannote-audio` requires HuggingFace account, model license acceptance, and token setup | |
| - For a hackathon demo, Gemma 4 inference from linguistic context is good enough | |
| - Doctors and patients have very different speech patterns (questions vs symptom descriptions) that Gemma 4 reliably distinguishes | |
| - Can always upgrade to pyannote later | |
| ### 6. SQLite for storage | |
| **Decision:** Local SQLite over PostgreSQL or cloud database. | |
| **Reasoning:** Desktop app, no server, no network dependency. SQLite is reliable, zero-config, and sufficient for demo-scale data. Schema: patients → sessions → notes + symptoms. | |
| ### 7. Whisper model: small over base | |
| **Decision:** Upgrade from `base` to `small` Whisper model. | |
| **Reasoning:** `base` had poor accuracy on real speech, especially medical terminology. `small` is ~4x more accurate on medical vocabulary and still runs acceptably on CPU (~2-3x slower than base but real-time viable with 3-second chunking). `medium` was considered but too slow for live demo. | |
| --- | |
| ## Issues Encountered & Resolutions | |
| ### Issue 1: `google-generativeai` deprecated | |
| **Error:** `FutureWarning: All support for the google.generativeai package has ended` | |
| **Root cause:** Google deprecated the old `google-generativeai` SDK in favour of `google-genai` | |
| **Resolution:** Replaced `google-generativeai` with `google-genai>=1.0.0` in requirements. Updated `cloud_agents.py` to use `from google import genai` and `genai.Client()` pattern. | |
| ### Issue 2: Wrong Gemma 4 cloud model name | |
| **Error:** `404 NOT_FOUND: models/gemma-4-27b-it is not found` | |
| **Root cause:** Model name `gemma-4-27b-it` does not exist on Google AI Studio API. | |
| **Resolution:** Listed available models via API (`client.models.list()`). Correct names are: | |
| - `gemma-4-26b-a4b-it` (26B MoE, faster) | |
| - `gemma-4-31b-it` (31B dense, most capable) | |
| Updated default in `cloud_agents.py` and `.env`. | |
| ### Issue 3: Twi translation 500 INTERNAL error | |
| **Error:** `500 INTERNAL: Internal error encountered` on `translate_to_twi()` | |
| **Root cause:** Gemma 4 struggles with Twi (Akan) — a low-resource language with limited training data. The model likely has insufficient Twi coverage to translate medical content reliably, causing server-side failures. | |
| **Resolution (temporary):** Removed Twi translation from the pipeline. Added try/except guards around all cloud agent calls so one failure doesn't break the entire `generate_notes()` flow. | |
| **Planned fix:** Integrate NLLB-200 (`facebook/nllb-200-distilled-600M`) — Meta's purpose-built model for 200 low-resource languages including Akan/Twi. | |
| ### Issue 4: Ollama version too old for Gemma 4 | |
| **Error:** `Error: pull model manifest: 412: The model you are attempting to pull requires a newer version of Ollama` | |
| **Root cause:** System Ollama was v0.19.0. Gemma 4 requires a newer version. | |
| **Resolution:** Reinstall Ollama via the official install script: `curl -fsSL https://ollama.com/install.sh | sh` then `sudo systemctl restart ollama`. Note: Linux package managers (snap, apt) ship outdated Ollama versions — always use the curl script. | |
| ### Issue 5: `chromadb.PersistentClient | None` TypeError | |
| **Error:** `TypeError: unsupported operand type(s) for |: 'function' and 'NoneType'` | |
| **Root cause:** `chromadb.PersistentClient` is a factory function, not a class. Using it in a `X | None` type annotation evaluates at runtime and fails. | |
| **Resolution:** Added `from __future__ import annotations` to `rag/retriever.py` — this makes all annotations lazy (strings at runtime), bypassing the evaluation issue. | |
| ### Issue 6: White empty boxes in UI (RAG panels) | |
| **Issue:** `gr.Markdown` components rendered as white boxes on dark Gradio theme, even when empty. | |
| **Root cause:** Gradio's default light background on Markdown components clashes with the dark theme. Empty panels had no content but still showed as white rectangles. | |
| **Resolution:** Moved RAG panels (ICD-10, Drug Reference, Symptoms) into `gr.Accordion` components. Accordions collapse when not needed and have theme-consistent styling. Also added CSS `background: transparent` for markdown panels. | |
| ### Issue 9: Gemma 4 image input — wrong contents structure | |
| **Error:** `500 INTERNAL` then `Part.from_text() takes 1 positional argument but 2 were given` | |
| **Root cause:** Two sequential mistakes in the multimodal contents format: | |
| 1. First attempt wrapped parts in `types.Content(role="user", parts=[...])` — not needed | |
| 2. Used `types.Part.from_text(IMAGE_PROMPT)` — this method does not exist in the SDK | |
| **Resolution:** Per official Gemma 4 docs (philschmid.de/gemma-4-gemini-api), the correct format is a plain list mixing `Part.from_bytes()` and a raw string: | |
| ```python | |
| contents=[ | |
| types.Part.from_bytes(data=file_bytes, mime_type=mime_type), | |
| IMAGE_PROMPT, # plain string, not Part.from_text() | |
| ] | |
| ``` | |
| All Gemma 4 models (including 26B and 31B) are fully multimodal. The initial 500 error was caused by the wrong content structure, not a model limitation. | |
| ### Issue 10: pyannote-audio abandoned in favour of Gemma 4 | |
| **Decision made:** Started implementing pyannote-audio for speaker diarization, then stopped. | |
| **Reason:** User confirmed Gemma 4 post-hoc labelling is sufficient for the demo. pyannote requires HuggingFace account, model license acceptance, and heavy torch dependency. Gemma 4 language-based inference is actually more reliable for medical conversations because it uses *context* (doctors ask questions, patients describe symptoms) rather than raw audio signal (which can fail when two speakers have similar voices). | |
| ### Issue 10: Gradio CSS parameter deprecation warning | |
| **Warning:** `UserWarning: The parameters have been moved from the Blocks constructor to the launch() method` | |
| **Root cause:** Gradio 6.0 moved `css` parameter from `gr.Blocks(css=...)` to `demo.launch(css=...)`. | |
| **Resolution:** Moved `css=CSS` to `demo.launch(...)`. | |
| ### Issue 8: uv installing to wrong Python version | |
| **Issue:** `chromadb` and `sentence-transformers` installed but not importable from venv. | |
| **Root cause:** The venv was created with Python 3.11 (via uv) but system also has Python 3.12. Running `uv pip install` without specifying the environment installed to the wrong location. | |
| **Resolution:** Used `VIRTUAL_ENV=/path/to/.venv uv pip install ...` to target the correct venv, or used `/path/to/.venv/bin/python -m pip install ...`. | |
| --- | |
| ## What Was Considered and Rejected | |
| | Option | Rejected because | | |
| |---|---| | |
| | Streamlit UI | Real-time transcript polling is awkward in Streamlit's re-run model | | |
| | PostgreSQL storage | Overkill for desktop demo; SQLite is zero-config | | |
| | pyannote-audio diarization | Requires HF account + model license; too much setup for hackathon timeline | | |
| | Full 70k ICD-10 dataset | Too large to embed in demo time; curated Ghana-relevant subset is more impactful | | |
| | Running everything on cloud API | Wanted to demonstrate hybrid local+cloud multi-agent architecture | | |
| | Whisper `large-v3` | Too slow on CPU for real-time; `small` is the sweet spot | | |
| | Gemma 4 for Twi translation | Low-resource language; model returned 500 errors. NLLB-200 is the right tool | | |
| --- | |
| ## Remaining Work / Roadmap | |
| - [ ] **Twi translation via NLLB-200** — integrate `facebook/nllb-200-distilled-600M` locally | |
| - [ ] **PDF export** — export SOAP note + patient summary as printable PDF (fpdf2 already in deps) | |
| - [ ] **Multi-session history** — view all past sessions for a patient, not just the most recent | |
| - [ ] **Upgrade to Whisper `medium`** if demo machine is fast enough | |
| - [ ] **ICD-10 code expansion** — add full 70k code dataset for production use | |
| - [ ] **MedGemma** — self-host `medgemma-4b-it` or `medgemma-27b-it` for higher-accuracy medical image analysis | |
| - [ ] **Long-context patient history** — load all previous session notes into SOAP prompt for longitudinal care reasoning | |
| --- | |
| ## File Structure | |
| ``` | |
| hosptial_copilot/ | |
| ├── app.py Main Gradio app + UI | |
| ├── agents/ | |
| │ ├── cloud_agents.py Gemma 4 cloud: SOAP, summary, speaker labelling | |
| │ └── symptom_agent.py Gemma 4 E2B local: symptom JSON extraction | |
| ├── transcription/ | |
| │ └── transcriber.py faster-whisper live mic streaming | |
| ├── rag/ | |
| │ ├── retriever.py ChromaDB + MiniLM embedding + retrieval | |
| │ └── data/ | |
| │ ├── icd10_common.json 90+ ICD-10 codes (Ghana-relevant) | |
| │ └── essential_medicines.json 40+ WHO Essential Medicines | |
| ├── database/ | |
| │ └── db.py SQLite schema + helpers | |
| ├── requirements.txt | |
| ├── .env.example | |
| ├── .gitignore | |
| ├── README.md | |
| └── DEVLOG.md This file | |
| ``` | |
| --- | |
| ## Environment Variables | |
| | Variable | Default | Description | | |
| |---|---|---| | |
| | `GEMINI_API_KEY` | — | Google AI Studio API key (required) | | |
| | `WHISPER_MODEL` | `small` | Whisper model size: tiny/base/small/medium/large-v3 | | |
| | `OLLAMA_MODEL` | `gemma4:e2b` | Local Ollama model for symptom extraction | | |
| | `CLOUD_MODEL` | `gemma-4-26b-a4b-it` | Google AI Studio model name | | |