README.md

---
license: other
language:
- en
tags:
- OneScience
- Earth Science
- Multi-Modal Remote Sensing
- Semantic Segmentation
frameworks: PyTorch
datasets: []
---

<p align="center">
  <strong>
    <span style="font-size: 30px;">SkySense</span>
  </strong>
</p>

# Model Introduction

SkySense is a multimodal remote sensing foundation model for general Earth observation interpretation. It jointly represents high-resolution RGB, Sentinel-1, and Sentinel-2 data through independent spatial encoding, temporal aggregation, cross-modal fusion, and geographic prototype modeling.

Paper: SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagery  
https://arxiv.org/abs/2312.10115

# Model Description

SkySense was proposed by a research team from Wuhan University, Huazhong University of Science and Technology, and other institutions. The model is trained with large-scale multimodal remote sensing time-series data and fuses high-resolution RGB, Sentinel-1 radar, and Sentinel-2 multispectral observations. It is suitable for multimodal remote sensing representation learning, land-cover semantic segmentation, and other Earth observation interpretation tasks.

# Use Cases

| Scenario | Description |
| :---: | :--- |
| Multimodal remote sensing fusion | Fuse multi-resolution observations from HR, Sentinel-1, and Sentinel-2. |
| Multi-temporal modeling | Process satellite sequences using date encoding and intra-modality temporal aggregation. |
| Semantic segmentation | Produce high-resolution land-cover class maps. |
| Geographic object detection | Transfer and fine-tune multimodal representations for detecting remote sensing objects such as aircraft, ships, and vehicles. |
| Land-surface change detection | Adapt multi-temporal observations to downstream tasks to identify changes in buildings and land-cover regions. |
| Local engineering validation | Use a small amount of synthetic data to check the training, inference, and evaluation workflows. |

# Usage Guide

## 1. OneCode

Experience intelligent one-click AI4S programming through the OneCode online environment:

[Click to Experience Intelligent One-Click AI4S Programming](https://web-2069360198568017922-iaaj.ksai.scnet.cn:58043/home)

## 2. Download and Installation

```bash
hf download OneScience-Group/SkySense --local-dir ./SkySense
cd SkySense
```

### Environment Dependencies

**Hardware Requirements**

- A GPU or DCU is recommended.
- CPU can be used for small-configuration connectivity validation; full training and inference will be slow.
- DCU users must install DTK in advance. DTK 25.04.2 or above, or the OneScience recommended version matching the current cluster, is recommended.

**DCU Environment**

```bash
# Please activate DTK and CONDA first
conda create -n onescience311 python=3.11 -y
conda activate onescience311
# uv installation is supported
pip install onescience[earth-dcu] -i http://mirrors.onescience.ai:3141/pypi/simple/ --trusted-host mirrors.onescience.ai
```

**GPU Environment**

```bash
# Please activate CONDA first
conda create -n onescience311 python=3.11 -y libstdcxx-ng=12 libgcc-ng=12 gcc_linux-64=12 gxx_linux-64=12
conda activate onescience311
# uv installation is supported
pip install onescience[earth-gpu] -i http://mirrors.onescience.ai:3141/pypi/simple/ --trusted-host mirrors.onescience.ai
```

### Training Data Introduction

By default, 2 training and 1 test synthetic samples are used to validate the engineering workflow. They are saved as `data/train.npz` and `data/test.npz`, respectively.

The synthetic data preserves the specifications from the paper and the official backbone: static HR, 20 Sentinel-2 time steps, 10 Sentinel-1 time steps, and the channel counts and spatial sizes of each modality.

Real data must be preprocessed and converted to the following NPZ training protocol. This protocol is consistent with the model input specification but is not the download format of the original datasets.

```text
hr: float32 [N,1,3,224,224]
s2: float32 [N,20,10,64,64]
s1: float32 [N,10,2,64,64]
dates_hr: int64 [N,1]
dates_s2: int64 [N,20]
dates_s1: int64 [N,10]
region: int64 [N]
labels: int64 [N,224,224]
band_order_hr: string [3]
band_order_s2: string [10]
band_order_s1: string [2]
```

`fake_data.py` automatically writes the `protocol` and `data_source` protocol metadata. These fields must be retained when using real data.

```bash
python scripts/fake_data.py
```

### Training

```bash
python scripts/train.py
```

For multi-GPU training, use:

```bash
torchrun --nproc_per_node=8 scripts/train.py
```

Training jointly optimizes semantic segmentation and cross-modal representation alignment objectives and saves a checkpoint and aggregate training metrics. The default configuration is intended for quick workflow validation. Formal experiments should use the multimodal time-series data, model configuration, and training duration corresponding to the paper.

```text
result/checkpoints/skysense.pt
result/training/metrics.json
```

### Training Weights

This repository will provide SkySense training weights in the `weight/` folder. The weight files will be uploaded soon and are expected to be available in the near future.

### Inference

```bash
python scripts/inference.py
```

Inference loads the training checkpoint, generates high-resolution semantic segmentation results in batches, and saves them to:

```text
result/output/
```

### Evaluation and Visualization

```bash
python scripts/result.py
```

Evaluation reports pixel accuracy, per-class IoU, and mean IoU, and generates a comparison figure of input, label, and prediction. Results on synthetic data are only for engineering workflow validation and do not represent full-paper performance.

```text
result/evaluation/metrics.json
result/evaluation/comparison.png
```

# Official OneScience Resources

| Platform | OneScience Main Repository | Skills Repository |
| --- | --- | --- |
| Gitee | https://gitee.com/onescience-ai/onescience | https://gitee.com/onescience-ai/oneskills |
| GitHub | https://github.com/onescience-ai/OneScience | https://github.com/onescience-ai/oneskills |

# Citation and License

This repository is a reproduction of the original SkySense paper.

Use of the code and data in this repository remains subject to the licenses and terms of use of their respective projects.