Gonalb/flucold-ft-v2

SentenceTransformer based on Snowflake/snowflake-arctic-embed-l

This is a sentence-transformers model finetuned from Snowflake/snowflake-arctic-embed-l. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: Snowflake/snowflake-arctic-embed-l
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("Gonalb/flucold-ft-v2")
# Run inference
sentences = [
    'QUESTION #2: How does the sneeze centre in the brainstem coordinate the actions involved in sneezing?',
    'causes sneezing.23 The trigeminal nerves relay\ninformation to the sneeze centre in the brainstem and\ncause reflex activation of motor and parasympathetic\nbranches of the facial nerve and activate respiratory\nmuscles. A model of the sneeze reflex is illustrated in\nfigure 1. The sneeze centre coordinates the inspiratory\nand expiratory actions of sneezing via respiratory\nmuscles, and lacrimation and nasal congestion via\nparasympathetic branches of the facial nerve. The eyes\nare always closed during sneezing by activation of facial\nmuscles, indicating a close relation between the',
    'stroke, seizure disorder, dementia)\nAsthma or other chronic pulmonary disease\nChronic kidney disease\nChronic liver disease\nHeart disease (acquired or congenital)\nImmunosuppression (e.g., HIV infection, cancer, transplant \nrecipients, use of immunosuppressive medications)\nLong-term aspirin therapy in patients younger than 19 years\nMetabolic disorders (acquired [e.g., diabetes mellitus] or \ninherited [e.g., mitochondrial disorders])\nMorbid obesity\nSickle cell anemia and other hemoglobinopathies\nSpecial groups\nAdults 65 years and older\nAmerican Indians and Alaska Natives',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.6122
cosine_accuracy@3	0.8878
cosine_accuracy@5	0.9388
cosine_accuracy@10	0.9898
cosine_precision@1	0.6122
cosine_precision@3	0.2959
cosine_precision@5	0.1878
cosine_precision@10	0.099
cosine_recall@1	0.6122
cosine_recall@3	0.8878
cosine_recall@5	0.9388
cosine_recall@10	0.9898
cosine_ndcg@10	0.8165
cosine_mrr@10	0.7593
cosine_map@100	0.76

Information Retrieval

• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.61
cosine_accuracy@3	0.86
cosine_accuracy@5	0.91
cosine_accuracy@10	0.98
cosine_precision@1	0.61
cosine_precision@3	0.2867
cosine_precision@5	0.182
cosine_precision@10	0.098
cosine_recall@1	0.61
cosine_recall@3	0.86
cosine_recall@5	0.91
cosine_recall@10	0.98
cosine_ndcg@10	0.8057
cosine_mrr@10	0.749
cosine_map@100	0.7505

Training Details

Training Dataset

Unnamed Dataset

• Size: 400 training samples
• Columns: sentence_0 and sentence_1
• Approximate statistics based on the first 400 samples:

  	sentence_0	sentence_1
  type	string	string
  details	min: 2 tokens mean: 23.07 tokens max: 53 tokens	min: 25 tokens mean: 122.33 tokens max: 296 tokens

• Samples:

  sentence_0	sentence_1
  What should individuals with asthma do if they experience flu symptoms?	People with asthma who get flu symptoms should call their health care provider right away. There are antiviral drugs that can treat flu illness and help prevent serious flu complications. About asthma Asthma is a lung disease that is caused by chronic inflammation of the airways. It is one of the most common long-term diseases among children, but adults can have asthma, too. Asthma attacks occur when the lung airways tighten due to inflammation. Asthma attacks can be
  What causes asthma attacks to occur in individuals with asthma?	People with asthma who get flu symptoms should call their health care provider right away. There are antiviral drugs that can treat flu illness and help prevent serious flu complications. About asthma Asthma is a lung disease that is caused by chronic inflammation of the airways. It is one of the most common long-term diseases among children, but adults can have asthma, too. Asthma attacks occur when the lung airways tighten due to inflammation. Asthma attacks can be
  QUESTION #1: How long are people with RSV typically contagious?	second birthday. However, repeat infections may occur throughout life. People with RSV are usually contagious for 3 to 8 days and may become contagious a day or two before they start showing signs of illness. However, some infants and people with weakened immune systems can continue to spread the virus for 4 weeks or longer, even after they stop showing symptoms. Children are often exposed to and infected with RSV outside the home, such as in school or childcare centers. They can then transmit the virus to other members of the family.

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 10
• per_device_eval_batch_size: 10
• num_train_epochs: 10
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 10
• per_device_eval_batch_size: 10
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 10
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin

Training Logs

Epoch	Step	Training Loss	cosine_ndcg@10
1.0	40	-	0.8359
1.25	50	-	0.8312
2.0	80	-	0.8304
2.5	100	-	0.8156
3.0	120	-	0.8016
3.75	150	-	0.7952
4.0	160	-	0.7880
5.0	200	-	0.8021
6.0	240	-	0.8215
6.25	250	-	0.8286
7.0	280	-	0.8079
7.5	300	-	0.8043
8.0	320	-	0.8126
8.75	350	-	0.8099
9.0	360	-	0.8126
10.0	400	-	0.8165
0.6173	50	-	0.8138
1.0	81	-	0.8158
1.2346	100	-	0.7932
1.8519	150	-	0.7989
2.0	162	-	0.7866
2.4691	200	-	0.8012
3.0	243	-	0.7803
3.0864	250	-	0.7969
3.7037	300	-	0.8030
4.0	324	-	0.7993
4.3210	350	-	0.7848
4.9383	400	-	0.7852
5.0	405	-	0.7814
5.5556	450	-	0.7975
6.0	486	-	0.7846
6.1728	500	0.314	0.7925
6.7901	550	-	0.7994
7.0	567	-	0.8069
7.4074	600	-	0.8048
8.0	648	-	0.8063
8.0247	650	-	0.8062
8.6420	700	-	0.7992
9.0	729	-	0.8115
9.2593	750	-	0.8118
9.8765	800	-	0.8057
10.0	810	-	0.8057

Framework Versions

• Python: 3.11.11
• Sentence Transformers: 3.4.1
• Transformers: 4.48.3
• PyTorch: 2.5.1+cu124
• Accelerate: 1.3.0
• Datasets: 3.3.2
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}