visurg/SurgLIME

Can LLM-Generated Text Empower Surgical Vision-Language Pre-training? (CVPRW 2026 Oral)

This is the official repository for the CVPRW 2026 Oral paper: Can LLM-Generated Text Empower Surgical Vision-Language Pre-training?

Star ⭐ the repository if you find this work useful.

Abstract

Recent advancements in self-supervised learning have led to powerful surgical vision encoders capable of spatiotemporal understanding. However, extending these visual foundations to multimodal reasoning tasks is severely bottlenecked by the prohibitive cost of expert textual annotations.

To overcome this scalability limitation, we introduce LIME, a large-scale multimodal dataset derived from open-access surgical videos using human-free, Large Language Model (LLM)-generated narratives. While LIME offers immense scalability, unverified generated texts may contain errors, including hallucinations, that could potentially lead to catastrophically degraded pretrained medical priors in standard contrastive pipelines.

To mitigate this issue, we propose SurgLIME, a parameter-efficient Vision-Language Pre-training (VLP) framework designed to learn reliable cross-modal alignments from noisy narratives. SurgLIME preserves foundational medical priors using a LoRA-adapted dual-encoder architecture and introduces an automated confidence estimation mechanism that dynamically down-weights uncertain text during contrastive alignment.

Evaluations on the AutoLaparo and Cholec80 benchmarks show that SurgLIME achieves competitive zero-shot cross-modal alignment while preserving the robust linear-probing performance of the visual foundation model.

Installation

Clone the repository and install the required dependencies:

git clone git@github.com:visurg-ai/SurgLIME.git
cd SurgLIME
pip install -r requirements.txt