README.md

---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: image-text-to-text
base_model:
- MSALab/PerceptionDLM-Base
tags:
- multimodal
- diffusion-language-model
- dllm
- region-captioning
- dense-captioning
- parallel-decoding
---

# PerceptionDLM

**PerceptionDLM** is a multimodal **diffusion** language model optimized for **efficient parallel region perception**. Built upon [**PerceptionDLM-Base**](https://huggingface.co/MSALab/PerceptionDLM-Base), it fully leverages the parallel decoding nature of diffusion language models (DLMs): given an image and multiple region masks, it generates descriptions for **all regions simultaneously** within a single denoising process — avoiding the linear latency growth of autoregressive (AR) region captioners.

To the best of our knowledge, this is the first model to achieve **parallel region captioning and perception** by leveraging the advantages of diffusion language models.

<p align="center">
  📄 <a href="https://arxiv.org/abs/2606.19534">Paper</a> &nbsp;|&nbsp;
  💻 <a href="https://github.com/MSALab-PKU/PerceptionDLM">Code</a> &nbsp;|&nbsp;
  📊 <a href="https://huggingface.co/datasets/MSALab/ParaDLC-Bench">ParaDLC-Bench</a>
</p>

## Highlights

- 🧩 **Parallel region captioning.** Region prompting + structured attention masking describe many masked regions in a single denoising pass.
- ⚡ **Up to 3.44× throughput speedup** in dense multi-region scenarios, with stable per-image latency (~2.9s).
- 🎯 **Competitive quality** with strong AR region captioners while being substantially faster.

## Model Details

| | |
| :--- | :--- |
| Base model | [MSALab/PerceptionDLM-Base](https://huggingface.co/MSALab/PerceptionDLM-Base) |
| Key modules | Region prompting, RoI-aligned feature replay, structured attention masking |
| Region prompts | up to 6 per image |
| Default inference | 32 diffusion steps, generation length 32 per mask |
| Training | full ParaCaption corpus, ~2 days on 32× H100 |
| Precision | bfloat16 |

## Results (ParaDLC-Bench)

| Method | Type | Avg (%) | TPF ↑ | Time (s) ↓ |
| :--- | :--- | :---: | :---: | :---: |
| GAR-8B | AR (sequential) | 69.5 | 1.0 | 479 |
| LLaDA-V-8B | Diffusion | 35.2 | 1.0 | 3241 |
| **PerceptionDLM** | **Diffusion (parallel)** | **62.4** | **2.9** | **276** |

`TPF` = Tokens Per Forward (higher = more parallel). PerceptionDLM nearly doubles the accuracy of prior diffusion VLMs while drastically reducing inference time.

## Usage

Full inference scripts are provided in the [GitHub repository](https://github.com/MSALab-PKU/PerceptionDLM).

```bash
python demo/infer_pdmllm.py \
  --model-path MSALab/PerceptionDLM \
  --image assets/demo.jpg \
  --masks assets/demo_mask_0.jpg \
          assets/demo_mask_1.jpg \
          assets/demo_mask_2.jpg \
  --gen-length 32 --steps 32 --temperature 0.0 --top-p 1.0
```

The model takes an RGB image plus one or more binary masks, and returns one caption per region — all generated in parallel.

## Citation

```bibtex
@article{sun2026perceptiondlm,
  title   = {PerceptionDLM: Parallel Region Perception with Multimodal Diffusion Language Models},
  author  = {Sun, Yueyi and Wang, Yuhao and Li, Jason and Tian, Ye and Zhang, Tao and Mai, Jacky and Wang, Yihan and Wang, Haochen and Bai, Jinbin and Yang, Ling and Tong, Yunhai},
  journal = {arXiv preprint arXiv:2606.19534},
  year    = {2026}
}
```

## License

Released under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).