redis/model-a-baseline

SentenceTransformer based on thenlper/gte-small

This is a sentence-transformers model finetuned from thenlper/gte-small. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: thenlper/gte-small
• Maximum Sequence Length: 128 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 128, 'do_lower_case': False, 'architecture': 'BertModel'})
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("redis/model-a-baseline")
# Run inference
sentences = [
    'what are the 5 liberties of the first amendment',
    'First Amendment to the United States Constitution The First Amendment (Amendment I) to the United States Constitution prohibits the making of any law respecting an establishment of religion, ensuring that there is no prohibition on the free exercise of religion, abridging the freedom of speech, infringing on the freedom of the press, interfering with the right to peaceably assemble, or prohibiting the petitioning for a governmental redress of grievances. It was adopted on December 15, 1791, as one of the ten amendments that constitute the Bill of Rights.',
    "Jake and the Never Land Pirates Jake and the Never Land Pirates (also known as Captain Jake and the Never Land Pirates in the fourth season and associated merchandise[1]) is an Annie Award-winning musical and interactive American children's animated television series shown on Disney Junior. It is based on Disney's Peter Pan franchise, which in turn is based on the famous book and play by British author J. M. Barrie. It is the first Disney Junior original show following the switch from Playhouse Disney. It stars Sean Ryan Fox from Henry Danger, Megan Richie, Jadon Sand, David Arquette, Corey Burton, Jeff Bennett, Loren Hoskins and Dee Bradley Baker. The title character Jake was previously voiced by Colin Ford, and then later by Cameron Boyce, while Izzy was voiced for the first three seasons by Madison Pettis and Cubby was voiced by Jonathan Morgan Heit. The series is created by Disney veteran Bobs Gannaway, whose works include another Disney Junior series, Mickey Mouse Clubhouse, and films such as Secret of the Wings, The Pirate Fairy and Planes: Fire & Rescue. The last episode aired on November 6, 2016.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 1.0000,  0.9977, -0.0309],
#         [ 0.9977,  1.0000, -0.0296],
#         [-0.0309, -0.0296,  1.0000]])

Evaluation

Metrics

Information Retrieval

• Datasets: NanoMSMARCO and NanoNQ
• Evaluated with InformationRetrievalEvaluator

Metric	NanoMSMARCO	NanoNQ
cosine_accuracy@1	0.12	0.22
cosine_accuracy@3	0.32	0.44
cosine_accuracy@5	0.48	0.5
cosine_accuracy@10	0.6	0.56
cosine_precision@1	0.12	0.22
cosine_precision@3	0.1067	0.1467
cosine_precision@5	0.096	0.1
cosine_precision@10	0.06	0.058
cosine_recall@1	0.12	0.22
cosine_recall@3	0.32	0.43
cosine_recall@5	0.48	0.49
cosine_recall@10	0.6	0.54
cosine_ndcg@10	0.3452	0.3854
cosine_mrr@10	0.265	0.3359
cosine_map@100	0.2749	0.3487

Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with NanoBEIREvaluator with these parameters:

  {
      "dataset_names": [
          "msmarco",
          "nq"
      ],
      "dataset_id": "lightonai/NanoBEIR-en"
  }
  

Metric	Value
cosine_accuracy@1	0.17
cosine_accuracy@3	0.38
cosine_accuracy@5	0.49
cosine_accuracy@10	0.58
cosine_precision@1	0.17
cosine_precision@3	0.1267
cosine_precision@5	0.098
cosine_precision@10	0.059
cosine_recall@1	0.17
cosine_recall@3	0.375
cosine_recall@5	0.485
cosine_recall@10	0.57
cosine_ndcg@10	0.3653
cosine_mrr@10	0.3004
cosine_map@100	0.3118

Training Details

Training Dataset

Unnamed Dataset

• Size: 90,000 training samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 9 tokens mean: 11.82 tokens max: 27 tokens	min: 19 tokens mean: 106.2 tokens max: 128 tokens	min: 16 tokens mean: 104.63 tokens max: 128 tokens

• Samples:

  anchor	positive	negative
  who played in the movie throw momma from the train	Anne Ramsey Angelina (Anne) Ramsey (March 27, 1929[1] – August 11, 1988) was an American stage, television, and film actress. She was best known for portraying Mama Fratelli in The Goonies (1985) and Mrs. Lift, mother of Danny DeVito's protagonist, in Throw Momma from the Train (1987). The latter film saw Ramsey nominated for a Golden Globe Award and the Academy Award for Best Supporting Actress.	Aye Mere Watan Ke Logo "Aye Mere Watan Ke Logo" (Hindi: ऐ मेरे वतन के लोगों; "O' people of my country") is a Hindi patriotic song written by Kavi Pradeep, composed by C. Ramchandra, and performed by Lata Mangeshkar. The song commemorates Indian soldiers who died during the Sino-Indian War in 1962. The song was first performed live by Mangeshkar on 27 January 1963 at the National Stadium in New Delhi in the presence of President Sarvepalli Radhakrishnan and Prime Minister Jawaharlal Nehru, on account of Republic Day (26 January) 1963, which was just two months after the end of the war.
  when was the wall in san diego built	Mexico–United States barrier In September 2017, the U.S. government announced the start of construction of eight prototype barriers made from concrete and other materials.[51][52] On June 3, 2018 the San Diego section of the US border wall construction began. [53]	The Dance of Dragons At the Wall, Jon Snow (Kit Harington) retreats from Hardhome defeated, accompanied by the surviving wildlings, much to the chagrin of some of the Night's Watch. In the North, Stannis Baratheon (Stephen Dillane) reluctantly allows Melisandre (Carice van Houten) to sacrifice his daughter Shireen (Kerry Ingram) after Ramsay Bolton (Iwan Rheon) sabotages his resources, resulting to his army's damaged morale. In Braavos, Arya Stark (Maisie Williams), detours from her mission given by Jaqen H'ghar (Tom Wlaschiha) to reconnoiter Meryn Trant (Ian Beattie) instead. In Dorne, Jaime Lannister (Nikolaj Coster-Waldau) secures Myrcella Baratheon's (Nell Tiger Free) release from Doran Martell's (Alexander Siddig) court against an indignant Ellaria Sand (Indira Varma). In Meereen, the Sons of the Harpy attack the stadium of Daznak's Pit in an attempt to assassinate Daenerys Targaryen (Emilia Clarke), who is rescued by Jorah Mormont (Iain Glen) and her firstborn dragon, Drogon. Lea...
  urology is the study of diseases of the	Urology Urology (from Greek οὖρον ouron "urine" and -λογία -logia "study of"), also known as genitourinary surgery, is the branch of medicine that focuses on surgical and medical diseases of the male and female urinary-tract system and the male reproductive organs. Organs under the domain of urology include the kidneys, adrenal glands, ureters, urinary bladder, urethra, and the male reproductive organs (testes, epididymis, vas deferens, seminal vesicles, prostate, and penis).	The Jackie Gleason Show By far the most memorable and popular of Gleason's characters was blowhard Brooklyn bus driver Ralph Kramden, featured originally in a series of Cavalcade skits known as "The Honeymooners", with Pert Kelton as his wife Alice, and Art Carney as his upstairs neighbor Ed Norton. These were so popular that in 1955 Gleason suspended the variety format and filmed The Honeymooners as a regular half-hour sitcom (television's first spin-off), co-starring Carney, Audrey Meadows (who had replaced the blacklisted Kelton after the earlier move to CBS), and Joyce Randolph. Finishing 19th in the ratings, these 39 episodes were subsequently rerun constantly in syndication, often five nights a week, with the cycle repeating every two months for decades. They are probably the most familiar body of work from 1950s television with the exception of I Love Lucy starring Lucille Ball and Desi Arnaz.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 3.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 10,000 evaluation samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 9 tokens mean: 11.76 tokens max: 24 tokens	min: 15 tokens mean: 105.95 tokens max: 128 tokens	min: 11 tokens mean: 105.69 tokens max: 128 tokens

• Samples:

  anchor	positive	negative
  what is the political system that the united states follows in regards to elections	Primary election The United States is one of few countries to select candidates through popular vote in a primary election system; most countries rely on party leaders to vote candidates, as was previously the case in the U.S.[9] In modern politics, primary elections have been described as a significant vehicle for taking decision-making from political insiders to the voters, though this is disputed by select political science research.[10] The selection of candidates for federal, state, and local general elections takes place in primary elections organized by the public administration for the general voting public to participate in for the purpose of nominating the respective parties' official candidates; state voters start the electoral process for governors and legislators through the primary process, as well as for many local officials from city councilors to county commissioners.[11] The candidate who moves from the primary to be successful in the general election takes public off...	Bob Gaudio Robert John "Bob" Gaudio (born November 17, 1942) is an American singer, songwriter, musician, and record producer, and the keyboardist/backing vocalist for The Four Seasons.
  what caused the unusual landscape at the valley of fire	Valley of Fire State Park Complex uplifting and faulting of the region, followed by extensive erosion, have created the present landscape. The rough floor and jagged walls of the park contain brilliant formations of eroded sandstone and sand dunes more than 150 million years old. Other important rock formations include limestones, shales, and conglomerates.[4]	Fundamental Constitutions of Carolina Because the Fundamental Constitutions were drafted during John Locke's service to one of Province of Carolina proprietors, Anthony Ashley Cooper, it is widely alleged that Locke had a major role in the making of the Constitutions. In the view of historian David Armitage[5] and political scientist Vicki Hsueh, the Constitutions were co-authored by Locke and his patron Cooper, known also as 1st Earl of Shaftesbury.[6] However the document was a legal document written for and signed and sealed by the eight Lord proprietors to whom Charles II had granted the colony; Locke was only a paid secretary. He wrote it much as a lawyer writes a will.[4]
  according to the guinness world records what author has the most published works	Guinness World Records The book itself holds a world record, as the best-selling copyrighted book of all time. As of the 2017 edition, it is now in its 62nd year of publication, published in 100 countries and 23 languages. The international franchise has extended beyond print to include television series and museums. The popularity of the franchise has resulted in Guinness World Records becoming the primary international authority on the cataloguing and verification of a huge number of world records; the organisation employs official record adjudicators authorised to verify the authenticity of the setting and breaking of records.[2]	The Big Chill (film) The Big Chill is a 1983 American comedy-drama film directed by Lawrence Kasdan, starring Tom Berenger, Glenn Close, Jeff Goldblum, William Hurt, Kevin Kline, Mary Kay Place, Meg Tilly, and JoBeth Williams. The plot focuses on a group of baby boomers who attended the University of Michigan, reuniting after 15 years when their friend Alex commits suicide. Kevin Costner was cast as Alex, but all scenes showing his face were cut. It was filmed in Beaufort, South Carolina.[2]

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 3.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• learning_rate: 8e-05
• weight_decay: 0.005
• max_steps: 3375
• warmup_ratio: 0.1
• fp16: True
• dataloader_drop_last: True
• dataloader_num_workers: 1
• dataloader_prefetch_factor: 1
• load_best_model_at_end: True
• optim: adamw_torch
• ddp_find_unused_parameters: False
• push_to_hub: True
• hub_model_id: redis/model-a-baseline
• eval_on_start: True

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 8e-05
• weight_decay: 0.005
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 3.0
• max_steps: 3375
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: True
• dataloader_num_workers: 1
• dataloader_prefetch_factor: 1
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: False
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: True
• resume_from_checkpoint: None
• hub_model_id: redis/model-a-baseline
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: True
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss	NanoMSMARCO_cosine_ndcg@10	NanoNQ_cosine_ndcg@10	NanoBEIR_mean_cosine_ndcg@10
0	0	-	5.0014	0.6259	0.6583	0.6421
0.3556	250	3.7345	3.0513	0.4721	0.4567	0.4644
0.7112	500	3.1165	2.9938	0.4464	0.4306	0.4385
1.0669	750	3.055	2.9656	0.4028	0.4675	0.4351
1.4225	1000	3.0018	2.9558	0.3668	0.4309	0.3989
1.7781	1250	2.988	2.9463	0.4017	0.4426	0.4221
2.1337	1500	2.9625	2.9372	0.3571	0.4003	0.3787
2.4893	1750	2.9363	2.9311	0.3729	0.4068	0.3898
2.8450	2000	2.9287	2.9274	0.3728	0.3778	0.3753
3.2006	2250	2.907	2.9254	0.3770	0.3713	0.3742
3.5562	2500	2.8979	2.9242	0.3606	0.3884	0.3745
3.9118	2750	2.8931	2.9215	0.3446	0.3955	0.3700
4.2674	3000	2.883	2.9207	0.3511	0.3777	0.3644
4.6230	3250	2.8762	2.9201	0.3452	0.3854	0.3653

Framework Versions

• Python: 3.10.18
• Sentence Transformers: 5.2.0
• Transformers: 4.57.3
• PyTorch: 2.9.1+cu128
• Accelerate: 1.12.0
• Datasets: 2.21.0
• Tokenizers: 0.22.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}