XinRan-Tang/MMDiff-WHU

MMDiff: Multi-modal Remote Sensing Image Generation via Cross-Modality Spatial Feature Transfer

ISPRS 2026 🔥

Haojun Tang¹ · Wenda Zhao^1,* · Hengshuai Cui¹ · Haipeng Wang²

¹ Dalian University of Technology
² Unit 92728 of PLA

^* Corresponding author:

Abstract

Collecting spatially consistent multi-modal remote sensing (MMRS) images remains challenging due to different sensors vary in the imaging principles and acquisition times. This hinders the development of data-driven MMRS technologies, which rely on large-scale training samples. This paper proposes MMDiff, the first text-driven diffusion framework explicitly designed for jointly generating structurally consistent optical (OPT), synthetic aperture radar (SAR), and infrared (IR) remote sensing images from a single text prompt via cross-modality spatial feature transfer. MMDiff first trains the OPT branch with paired optical image-text data to capture rich semantic content, and then trains the SAR/IR branches with simple modality-specific text templates to learn the corresponding style attributes, without relying on complex linguistic descriptions. Specifically, we introduce a LoRA-based modality translation adaptation mechanism to translate the style attributes of optical spatial representations to SAR and IR style attributes while preserving the underlying semantic content. The translated representations are then transferred into the SAR and IR generation branches through the proposed spatial feature transfer mechanism, enabling rich spatial details in the generated SAR/IR images while maintaining cross-modal spatial consistency. Extensive experiments demonstrate that MMDiff achieves superior image quality in terms of modality similarity and semantic consistency compared to the state-of-the-art methods. Furthermore, MMDiff benefits downstream data-driven MMRS applications, e.g., multi-modal image fusion and object classification.

For each group, from left to right: optical (OPT), synthetic aperture radar (SAR), and near-infrared (NIR) images.

Model Overview

This directory contains the MMDiff model weights fine-tuned on the WHU-OPT-SAR dataset for jointly generating spatially consistent optical (OPT), synthetic aperture radar (SAR), and near-infrared (NIR) remote sensing images from text prompts at 512×512 resolution.

Training Data

These weights were fine-tuned on the MMDiff-WHU-dataset, which is derived from the WHU-OPT-SAR dataset. The WHU-OPT-SAR imagery was first cropped into 512×512 tiles, and each tile was then assigned to the semantic class occupying the largest area in its corresponding segmentation mask (i.e., a majority vote by area). The original WHU-OPT-SAR dataset contains seven land-cover classes. We retain four classes—city, farmland, forest, and water—by merging city and village into a single category and excluding road, others, and background. Finally, every tile was captioned with GPT-4o-mini, yielding paired optical (OPT), synthetic aperture radar (SAR), near-infrared (NIR), and text data. The OPT branch was fine-tuned on all four classes, while the SAR and NIR branches were fine-tuned with class-specific LoRA adapters.

Available Checkpoints

MMDiff-WHU/
├── mmdiff-opt/   # Optical image generation branch (fine-tuned, 512×512)
├── mmdiff-sar/   # SAR image generation branch (class-specific LoRA adapters)
└── mmdiff-ir/    # Near-infrared (NIR) image generation branch (class-specific LoRA adapters)

Each SAR and NIR branch contains one LoRA adapter per class:

mmdiff-sar/
├── sd-sar-lora-city/
├── sd-sar-lora-farmland/
├── sd-sar-lora-forest/
└── sd-sar-lora-water/

mmdiff-ir/
├── sd-ir-lora-city/
├── sd-ir-lora-farmland/
├── sd-ir-lora-forest/
└── sd-ir-lora-water/

The three branches are used together by the MMDiff sampling pipeline. Refer to the main MMDiff repository for installation, sampling commands, and configuration options.

Loading the Weights

Pass the model directory to the sampling configuration or command line. For example:

python mmdiff/sampling_OPT.py \
  --model_path /path/to/MMDiff/models/MMDiff-WHU/mmdiff-opt \
  --resolution 512

The SAR and NIR branches use their corresponding class-specific LoRA checkpoint directories during the complete OPT → SAR → NIR sampling workflow.

Intended Use

These checkpoints are intended for research on text-driven multi-modal remote sensing image generation, cross-modality spatial feature transfer, multi-modal image fusion, and downstream remote sensing applications.