Y-Research-Group/CSRv2-sts

Description

This is one CSRv2 model finetuned on MTEB sts datasets with Qwen3-Embedding-4B as backbone.

For more details, including benchmark evaluation, hardware requirements, and inference performance, please refer to our Github.

Sentence Transformer Usage

You can evaluate this model loaded by Sentence Transformers with the following code snippet (take STS12 as one example):

import mteb
from sentence_transformers import SparseEncoder
model = SparseEncoder(
    "Y-Research-Group/CSRv2-sts",  
    trust_remote_code=True
)
model.prompts = {
    "STS12": "Instruct: Retrieve semantically similar text\n Query:"  
}
task = mteb.get_tasks(tasks=["STS12"])
evaluation = mteb.MTEB(tasks=task)
evaluation.run(
    model,
    eval_splits=["test"],
    output_folder="./results/STS12",
    show_progress_bar=True
    encode_kwargs={"convert_to_sparse_tensor": False, "batch_size": 8}
)  # MTEB don't support sparse tensors yet, so we need to convert to dense tensors

It is suggested that you use our default prompts in evaluation.

Multi-TopK Support

Our model supports different sparsity levels due to the utilization of Multi-TopK loss in training. You can change sparsity model by adjusting the k parameterin the file3_SparseAutoEncoder/config.json`. We set sparsity level to 2 by default.

For instance, if you want to evaluate with sparsity level $K=8$ (which means there are 8 activated neurons in each embedding vector), the 3_SparseAutoEncoder/config.json should look like this:

{
    "input_dim": 2560,
    "hidden_dim": 10240,
    "k": 8,
    "k_aux": 1024,
    "normalize": false,
    "dead_threshold": 30
}

CSRv2 Qwen Series

We will release a series of CSRv2 models finetuned on common tasks in MTEB with Qwen3-Embedding-4B as backbone. These tasks are: 

• Classification
• Clustering
• Retrieval
• STS
• Pair_classification
• Reranking

Citation

@inproceedings{guo2026csrv2,
    title={{CSR}v2: Unlocking Ultra-sparse Embeddings},
    author={Guo, Lixuan and Wang, Yifei and Wen, Tiansheng and Wang, Yifan and Feng, Aosong and Chen, Bo and Jegelka, Stefanie and You, Chenyu},
    booktitle={International Conference on Learning Representations (ICLR)},
    year={2026}
}