IOSD: Improved Open-vocabulary Segmentation with Diffusion Models

Enhanced Grounded Diffusion for open-vocabulary segmentation using Stable Diffusion XL.

Left: The visual encoder extracts multi-scale latent features from all U-Net layers. Right: The transformer-based fusion module aligns visual and textual representations for open-vocabulary segmentation.

Project Overview

IOSD (Improved Open-vocabulary Segmentation with Diffusion Models) introduces a novel diffusion-based framework for open-vocabulary semantic segmentation.
Built upon Grounded Diffusion and Stable Diffusion XL, IOSD enhances segmentation quality and text–image alignment by:

  • Extracting multi-scale latent features from all U-Net layers
  • Introducing a transformer-based visual–text fusion module
  • Using prompt engineering and mask filtering for better generalization

IOSD achieves state-of-the-art zero-shot segmentation performance on COCO and PASCAL VOC datasets, enabling models to understand and segment arbitrary visual concepts described in text.

Example segmentation results. IOSD accurately detects unseen categories such as “red envelope” and “lion dance,” demonstrating strong generalization in open-vocabulary scenarios.

Implementation Details

Overall architecture of IOSD. The model integrates multi-layer U-Net features with a transformer-based fusion head to enable fine-grained open-vocabulary semantic segmentation.
  • Backbone: Stable Diffusion XL-Turbo (base-1.0)
  • Training framework: MMDetection + Grounded-SAM integration
  • Datasets: PASCAL VOC 2012, COCO 2017
  • Metrics: mIoU, PQ, ZS-mIoU (zero-shot mean IoU)

Quantitative Results

We evaluate models using mIoU scores on three class splits. For each class, 50 images are generated and evaluated using mmdetection for ground-truth segmentation masks. Training follows the DSF setting (see Ablation Table for details).

Model Class Split PASCAL (mIoU ↑) COCO (mIoU ↑)
Seen Unseen Seen Unseen
DAAM[1] Split 1 61.66 75.63 62.25 55.56
Split 2 65.75 59.25 60.08 65.55
Split 3 67.11 53.82 62.81 52.48
Average 64.84 62.90 61.71 57.76
GD[2] Split 1 90.16 83.19 83.35 76.81
Split 2 90.08 86.19 82.83 74.93
Split 3 90.67 79.86 84.85 67.89
Average 90.30 83.08 83.68 73.21
IOSD (Ours) Split 1 92.54 84.76 90.79 88.04
Split 2 92.18 88.18 90.81 79.34
Split 3 93.29 80.71 91.49 83.03
Average 92.67 84.55 91.03 83.47

[1] Tang et al., DAAM: Diffusion Attention Attribution Maps, 2022. [2] Li et al., Grounded Diffusion for Open-Vocabulary Segmentation, 2023.