DermFM-Zero: A Vision-Language Foundation Model for Zero-Shot Clinical Collaboration and Automated Concept Discovery in Dermatology

📢 This is the open-source release checkpoint of DermFM-Zero. The weights, evaluation pipelines, and downstream task code are publicly available. Access is collected through the form above, so please review the usage terms before requesting. Contact: siyuan.yan@monash.edu.

Model Description

DermFM-Zero is a dermatology vision-language foundation model that pairs the PanDerm vision encoder with a knowledge-enhanced PubMedBERT text encoder. It supports zero-shot diagnosis, cross-modal retrieval, and automated concept discovery via sparse autoencoders, all without task-specific fine-tuning.

Across 20 benchmarks and three multinational reader studies (753 clinicians, 1,285 sessions, 115,331 case-level observations), DermFM-Zero achieves state-of-the-art zero-shot performance while improving clinician decision-making in primary care and specialist settings.

Model Details

Model Type: Pretrained Vision-Language Foundation Model (knowledge-enhanced contrastive alignment)
Architecture:
- Vision encoder (PanDerm-Large, ViT-L/16): Initialized from the PanDerm dermatology backbone and further pretrained on the image-text corpus below
- Text encoder: PubMedBERT-256, a biomedical-domain encoder with an extended token window for detailed clinical descriptions. We initialize it by pretraining on Derm1M following KEP.
Resolution: 224 × 224 pixels
Manuscript: Currently under review at Nature Biomedical Engineering (ID: nBME-26-0776)
Code repository: https://github.com/SiyuanYan1/DermFM-Zero
License: CC-BY-NC-ND 4.0 (non-commercial academic research use only; see access conditions above)

Training Details

Pretraining corpus

517,455 dermatological image-text pairs covering 400+ skin conditions. By modality the corpus is 406,831 clinical (79%) and 110,624 dermoscopic (21%) images.

Public dermatology datasets. ISIC Archive, BCN20000, MSKCC, DermNet, Fitzpatrick17K, Derm12345, and HIBA. The 38,404 ISIC Archive images we used are listed in pretrain_image_lists/isic_image_ids.txt in this repository, so that anyone evaluating on ISIC-derived benchmarks can check for overlap.

Web and educational sources. The remainder is internet data collected from Derm1M and educational resources, spanning medical literature, textbooks, clinical forums, and video lectures.

Pretraining curriculum

Stage 1a — Masked Latent Modelling (Visual Representation Learning)

Training data: 3M+ unlabeled multimodal dermatological images from 15+ sources across 4 imaging modalities (dermoscopy, clinical photography, mobile photography)
Objective: Visible + masked latent alignment between student and teacher vision encoders (CAE-style)
Initialization: ImageNet-1K pretrained weights
Hardware: 4 × NVIDIA H100 80GB
Training time: ~7 days

Stage 1b — Knowledge-Enhanced Pretraining (Textual Representation Learning)

Training data: 556,372 attribute text instances came from Derm1M organized as a dermatological knowledge tree, where each skin-condition entity is populated with six attribute types: raw captions, ontology captions (synonyms and taxonomic ancestors), visual-concept captions (canonical morphological descriptors), and sentence-level sub-captions
Objective: AdaSP metric-learning loss, pulling attribute texts of the same condition together and pushing apart those of different conditions, following KEP
Initialization: PubMedBERT-256
Output: 768-dimensional ontology-aligned embeddings, reused both as the initialization for the trainable text tower and as a frozen knowledge encoder in Stage 2
Hardware: 1 × NVIDIA RTX 6000
Training time: 4 d 9 h

Stage 2 — Multi-Aspect Knowledge Alignment (Vision-Language Alignment)

Training data: 517,455 curated image-text pairs (see corpus above)
Coverage: 400+ skin conditions across multinational sources
Objective: Multi-aspect contrastive alignment between each image and several complementary views of its caption (raw caption, LLM-extracted disease aspect, concept aspect, and sentence-level sub-captions), plus a knowledge-distillation term anchoring the trainable text tower to the frozen knowledge encoder
Initialization: Stage 1a vision encoder + Stage 1b text encoder
Key hyper-parameters:
- Resolution: 224 x 224
- Optimizer: AdamW (lr 1x10-4, weight decay 0.1)
- Batch size: 512 per device (effective 2,048 via 2 devices x gradient accumulation 2)
- Training epochs: 15
- Warm-up steps: 200
- Image augmentation: Random resized crop (scale 0.4-1.0), color jitter, random grayscale
- Learning rate schedule: Cosine decay
- Precision: bf16 mixed precision
Hardware: 2 × NVIDIA H200
Training time: 1 d 9 h

Compute environment

Python 3.10.13, PyTorch 2.4.1, CUDA 11.8, open_clip
NumPy 2.2.6, SciPy 1.15.2, scikit-learn 1.6.1

Intended Uses

Primary Use Cases

Zero-shot dermatological diagnosis across 200+ skin conditions
Cross-modal retrieval (image ↔ clinical text)
Few-shot / label-efficient learning via linear probing
Multimodal fine-tuning with dermoscopy + clinical photography + patient metadata
Automated concept discovery via sparse autoencoders (SAE-CBM)
Artifact-aware diagnosis with concept-level intervention (ruler / pen / hair neuron suppression)

Out-of-Scope / Not Recommended

Not for clinical deployment without further validation. All clinical evaluation was retrospective and conducted in store-and-forward teledermatology settings.
Not optimized for in-person dermatology, full-body imaging, or histopathology.
Skin-tone fairness within human-AI workflows has not been formally tested.
Not for commercial use under the CC-BY-NC-ND 4.0 license.

How to Use

Installation

git clone https://github.com/SiyuanYan1/DermFM-Zero.git
cd DermFM-Zero

conda create -n dermfm-zero python=3.9.20
conda activate dermfm-zero
pip install -r requirements.txt

Quick Start: Zero-shot Classification

import open_clip
from PIL import Image
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

# Load model from HuggingFace (gated; requires access approval)
model, _, preprocess = open_clip.create_model_and_transforms(
    'hf-hub:redlessone/DermFM-Zero', device=device
)
model.eval()

# Tokenizer
tokenizer = open_clip.get_tokenizer('hf-hub:redlessone/DermFM-Zero')

# Read example image
image = preprocess(Image.open("your_skin_image.png")).unsqueeze(0).to(device)

# Define disease labels (example: PAD-UFES-20 classes)
PAD_CLASSNAMES = [
    "nevus",
    "basal cell carcinoma",
    "actinic keratosis",
    "seborrheic keratosis",
    "squamous cell carcinoma",
    "melanoma",
]

# Build text prompts
template = lambda c: f'This is a skin image of {c}.'
text = tokenizer([template(c) for c in PAD_CLASSNAMES]).to(device)

# Inference
with torch.no_grad(), torch.autocast(device):
    image_features = model.encode_image(image)
    text_features  = model.encode_text(text)

    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_features  /= text_features.norm(dim=-1, keepdim=True)

    text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)

final_prediction = PAD_CLASSNAMES[torch.argmax(text_probs[0])]
print(f"This image is diagnosed as: {final_prediction}")
print("Label probabilities:", text_probs)

Multi-prompt ensembling (recommended for best results)

DermFM-Zero was evaluated with 7 prompt templates (Extended Data Table 20). Averaging text features across templates improves robustness:

PROMPT_TEMPLATES = [
    "This is a skin image of {}.",
    "A skin image of {}.",
    "An image of {}, a skin condition.",
    "{}, a skin disorder, is shown in this image.",
    "The skin lesion depicted is {}.",
    "The skin cancer in this image is {}.",
    "This image depicts {}, a type of skin cancer.",
]

def build_classifier(class_names, templates, tokenizer, model):
    """Average text features across prompt templates for each class."""
    weights = []
    for cname in class_names:
        prompts = [t.format(cname) for t in templates]
        tokens  = tokenizer(prompts).to(device)
        with torch.no_grad():
            feats = model.encode_text(tokens)
            feats = feats / feats.norm(dim=-1, keepdim=True)
            feats = feats.mean(dim=0)
            feats = feats / feats.norm()
        weights.append(feats)
    return torch.stack(weights, dim=0)  # [num_classes, dim]

classifier = build_classifier(PAD_CLASSNAMES, PROMPT_TEMPLATES, tokenizer, model)

with torch.no_grad(), torch.autocast(device):
    image_features = model.encode_image(image)
    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_probs = (100.0 * image_features @ classifier.T).softmax(dim=-1)

final_prediction = PAD_CLASSNAMES[torch.argmax(text_probs[0])]
print(f"This image is diagnosed as: {final_prediction}")

See examples/zero-shot-classification.ipynb in the repo for a runnable demo (includes batched evaluation on a toy dataset).

Other downstream tasks

Task	Script
Cross-modal retrieval	`script/zero-shot-eval/DermFM-Zero-zs-retrieval.sh`
Linear probing (few-shot)	`script/linear-probe/DermFM-Zero-lp-eval.sh`
Multimodal fine-tuning	`script/multimodal_finetune/*.sh`
SAE concept discovery	`script/automated-concept-discovery/SAE-training/`
Concept-level artifact intervention	`script/automated-concept-discovery/ISIC-intervention/`
Reader study replication	`reader_studies/` (RS1 / RS2A / RS2B with real de-identified data)

Limitations

Disease coverage: ~400 skin conditions in pretraining; rare tropical diseases, complex systemic dermatoses, and rare genetic disorders remain underrepresented.
Source balance: Most of the corpus comes from web and literature sources rather than consecutive clinical cohorts, with curated clinical archives making up a minority of pairs. The corpus is therefore not population-representative, and performance on any specific patient population should be empirically validated before use.
Retrospective evaluation only: All clinical validation was conducted in store-and-forward teledermatology workflows, not in live patient encounters.
Skin-tone fairness in collaboration: Standalone fairness was characterized, but skin-tone-stratified fairness within human-AI workflows was not formally analyzed.
Data overlap: Despite source-level separation and SSCD image-level deduplication, some images may appear in both pretraining and downstream benchmarks via published literature in our PubMed and web corpora. Image-level overlap rates against the zero-shot benchmarks range from 0.14% (SNU) to 13.51% (Daffodil) at a cosine threshold of 0.75. That threshold is permissive: above 0.85 the rates fall sharply (e.g. ISIC2020 1.57% → 0.08%), and no zero-shot benchmark retains a near-exact duplicate (≥ 0.99). Per-dataset counts and the flagged image lists are published in the code repository under data_deduplication/results/.

Ethical Considerations

License restriction: CC-BY-NC-ND 4.0 — non-commercial academic research only. No clinical deployment.
Reader study ethics: All three reader studies were approved by relevant institutional review boards; readers participated voluntarily under signed agreements.
Redistribution: Users granted access agree not to redistribute the weights, derivative checkpoints, or extracted representations to third parties, per the access terms above.

Citation

If you use DermFM-Zero, please cite:

@article{yan2026dermfmzero,
  title   = {A Vision-Language Foundation Model for Zero-shot Clinical Collaboration and Automated Concept Discovery in Dermatology},
  author  = {Yan, Siyuan and Li, Xieji and Mo, Dan and Tschandl, Philipp and Jiang, Yiwen and Wang, Zhonghua and Hu, Ming and Ju, Lie and Vico-Alonso, Cristina and Zheng, Yizhen and Liu, Jiahe and Zhou, Juexiao and Chello, Camilla and Cheung, Jen G. and Anriot, Julien and Thomas, Luc and Primiero, Clare and Tan, Gin and Ng, Aik Beng and See, Simon and Tang, Xiaoying and Ip, Albert and Liao, Xiaoyang and Bowling, Adrian and Haskett, Martin and Zhao, Shuang and Janda, Monika and Soyer, H. Peter and Mar, Victoria and Kittler, Harald and Ge, Zongyuan},
  journal = {Nature Biomedical Engineering},
  year    = {2026},
  note    = {Under review, manuscript ID nBME-26-0776}
}

Related work:

@article{yan2025multimodal,
  title   = {A multimodal vision foundation model for clinical dermatology},
  author  = {Yan, Siyuan and Yu, Zhen and Primiero, Clare and Vico-Alonso, Cristina and Wang, Zhonghua and Yang, Litao and Tschandl, Philipp and Hu, Ming and Ju, Lie and Tan, Gin and others},
  journal = {Nature Medicine},
  year    = {2025},
  pages   = {1--12}
}

@inproceedings{yan2025derm1m,
  title     = {Derm1M: A Million-scale Vision-Language Dataset Aligned with Clinical Ontology Knowledge for Dermatology},
  author    = {Yan, Siyuan and Hu, Ming and Jiang, Yiwen and Li, Xieji and Fei, Hao and Tschandl, Philipp and Kittler, Harald and Ge, Zongyuan},
  booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  year      = {2025}
}

Contact

Siyuan Yan — Research Fellow, Monash University 📧 siyuan.yan@monash.edu

Downloads last month: -; Downloads are not tracked for this model. How to track

Paper for redlessone/DermFM-Zero

Knowledge-enhanced Visual-Language Pretraining for Computational Pathology

Paper • 2404.09942 • Published Sep 15, 2024