buelfhood/SOCO-Java-CodeBERTa-MNRL-Triplets

SentenceTransformer based on huggingface/CodeBERTa-small-v1

This is a sentence-transformers model finetuned from huggingface/CodeBERTa-small-v1 on the soco_train_java dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: huggingface/CodeBERTa-small-v1
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • soco_train_java

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: RobertaModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("buelfhood/SOCO-Java-CodeBERTa-MNRL-Triplets")
# Run inference
sentences = [
    'import java.net.*;\nimport java.io.*;\n\npublic class BruteForce {\n  private String strUserName;\n  private String strURL;\n  private int iAttempts;\n  \n  public BruteForce(String strURL,String strUserName) {\n    this.strURL = strURL;\n    this.strUserName = strUserName;\n    this.iAttempts = 0 ;\n\n  }\n  \n  public String getPassword(){\n      URL u;\n      String result ="";\n      PassGenBrute PG = new PassGenBrute(3);\n      URLConnection uc;\n      String strPassword = new String();\n      String strEncode;\n      try{\n        while (result.compareTo("HTTP/1.1 200 OK")!=0){\n        \n          strEncode = PG.getNewPassword();\n          u = new URL(strURL);\n          uc = u.openConnection();\n          uc.setDoInput(true);\n          uc.setDoOutput(true);\n          strPassword = strEncode;\n          strEncode =  strUserName + ":" + strEncode;\n        \n          strEncode = new String(Base64.encode(strEncode.getBytes()));\n          uc.setRequestProperty("Authorization"," " + strEncode);\n        \n          result = uc.getHeaderField(0);\n          uc = null;\n          u = null;\n          iAttempts++;\n        }\n\n      }\n      catch (Exception me) {\n      System.out.println("MalformedURLException: "+me);\n      }\n      return(strPassword);\n  }\n  \n  public int getAttempts(){\n    return (iAttempts);\n  };\n  \n  public static void main (String arg[]){\n     timeStart = 0;\n     timeEnd = 0;\n    \n    if (arg.length == 2) {\n       BruteForce BF = new BruteForce(arg[0],arg[1]);\n       System.out.println("Processing ... ");\n       timeStart = System.currentTimeMillis();\n       \n       System.out.println("Password = " + BF.getPassword());\n       timeEnd = System.currentTimeMillis();\n       System.out.println("Total Time Taken = " + (timeEnd - timeStart) + " (msec)");\n       System.out.println("Total Attempts  = " + BF.getAttempts());\n    }\n    else {\n       System.out.println("[Usage] java BruteForce <URL> <USERNAME>");\n\n    }\n\n  }\n}\n\nclass PassGenBrute {\n  private char[] password;\n  public PassGenBrute(int lenght) {\n    password = new char[lenght];\n    for (int i = 0; i < lenght; i++){\n      password[i] = 65;\n    }\n    password[0]--;\n  }\n  \n  public String getNewPassword()\n    throws PasswordFailureException{\n    password[0]++;\n\n    try {\n      for (int i=0; i<password.length ; i++){\n        if (password[i] == 90) {\n          password[i] = 97;\n        }\n        if (password[i] > 122) {\n          password[i] = 65;\n          password[i+1]++;\n        }\n      }\n    }\n    catch (RuntimeException re){\n      throw new PasswordFailureException ();\n    }\n    return new String(password);\n  }\n}\n\nclass PasswordFailureException extends RuntimeException {\n\n  public PasswordFailureException() {\n  }\n}',
    'import java.net.*;\nimport java.io.*;\n\n\npublic class Dictionary {\n  private String strUserName;\n  private String strURL;\n  private String strDictPath;\n  private int iAttempts;\n\n    \n    public Dictionary(String strURL,String strUserName,String strDictPath) {\n    this.strURL = strURL;\n    this.strUserName = strUserName;\n    this.iAttempts = 0 ;\n    this.strDictPath = strDictPath;\n    }\n  \n\n  public String getPassword(){\n      URL u;\n      String result ="";\n      PassGenDict PG = new PassGenDict(3,strDictPath);\n      URLConnection uc;\n      String strPassword = new String();\n      String strEncode;\n        try{\n        while (result.compareTo("HTTP/1.1 200 OK")!=0){\n      \n          strEncode = PG.getNewPassword();\n          u = new URL(strURL);\n          uc = u.openConnection();\n          uc.setDoInput(true);\n          uc.setDoOutput(true);\n          strPassword = strEncode;\n          strEncode =  strUserName + ":" + strEncode;\n          \n        strEncode = new String(Base64.encode(strEncode.getBytes()));\n          uc.setRequestProperty("Authorization"," " + strEncode);\n            \n      result = uc.getHeaderField(0);\n          uc = null;\n          u = null;\n          iAttempts++;\n        }\n\n      }\n      catch (Exception me) {\n      System.out.println("MalformedURLException: "+me);\n      }\n      return(strPassword);\n  }\n \n   public int getAttempts(){\n    return (iAttempts);\n  };\n  \n  public static void  main(String arg[]){\n     timeStart = 0;\n     timeEnd = 0;\n    \n    if (arg.length == 3) {\n    Dictionary BF = new Dictionary(arg[0],arg[1],arg[2]);\n\n    System.out.println("Processing ... ");\n    timeStart = System.currentTimeMillis();\n    System.out.println("Password = " + BF.getPassword());\n    timeEnd = System.currentTimeMillis();\n    System.out.println("Total Time Taken = " + (timeEnd - timeStart) + " (msec)");\n    System.out.println("Total Attempts  = " + BF.getAttempts());\n    }\n    else {\n       System.out.println("[Usage] java BruteForce <URL> <USERNAME> <Dictionary path>");\n\n    }\n\n  }\n}\n\n\nclass PassGenDict {\n\n  private char[] password;\n  private String line;\n  int iPassLenght;\n  private BufferedReader inputFile;\n  public PassGenDict(int lenght, String strDictPath) {\n    try{\n      inputFile = new BufferedReader(new FileReader(strDictPath));\n    }\n    catch (Exception e){\n    }\n    iPassLenght = lenght;\n  }\n   \n public String getNewPassword()\n    throws PasswordFailureException{\n    try {\n      {\n        line = inputFile.readLine();\n      }while (line.length() != iPassLenght);\n\n    }\n    catch (Exception e){\n      throw new PasswordFailureException ();\n    }\n    return (line);\n  }\n}\n\nclass PasswordFailureException extends RuntimeException {\n\n  public PasswordFailureException() {\n  }\n}',
    'import java.util.*;\nimport java.io.*;\nimport javax.swing.text.html.*;\n\n\npublic class WatchDog {\n\n  public WatchDog() {\n\n  }\n  public static void main (String args[]) {\n    DataInputStream newin;\n\n    try{\n      System.out.println("ishti");\n\n      System.out.println("Downloading first copy");\n      Runtime.getRuntime().exec("wget http://www.cs.rmit.edu./students/ -O oldfile.html");\n      String[] cmdDiff = {"//sh", "-c", "diff oldfile.html newfile.html > Diff.txt"};\n      String[] cmdMail = {"//sh", "-c", "mailx -s \\"Diffrence\\" \\"@cs.rmit.edu.\\" < Diff.txt"};\n      while(true){\n            Thread.sleep(24*60*60*1000);\n            System.out.println("Downloading new copy");\n            Runtime.getRuntime().exec("wget http://www.cs.rmit.edu./students/ -O newfile.html");\n            Thread.sleep(2000);\n            Runtime.getRuntime().exec(cmdDiff);\n            Thread.sleep(2000);\n            newin = new DataInputStream( new FileInputStream( "Diff.txt"));\n            if (newin.readLine() != null){\n               System.out.println("Sending Mail");\n               Runtime.getRuntime().exec(cmdMail);\n               Runtime.getRuntime().exec("cp newfile.html oldfile.html");\n\n            }\n        }\n\n    }\n    catch(Exception e){\n      e.printStackTrace();\n    }\n\n    }\n\n}',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

soco_train_java

• Dataset: soco_train_java at 44ca4ff
• Size: 38,664 training samples
• Columns: anchor_code, positive_code, and negative_code
• Approximate statistics based on the first 1000 samples:

  	anchor_code	positive_code	negative_code
  type	string	string	string
  details	min: 51 tokens mean: 466.15 tokens max: 512 tokens	min: 51 tokens mean: 467.06 tokens max: 512 tokens	min: 51 tokens mean: 454.38 tokens max: 512 tokens

• Samples:

  anchor_code	positive_code	negative_code
  import java.io.; import java.net.; import java.misc.BASE64Encoder; public class Dictionary { public Dictionary() {} public boolean fetchURL(String urlString,String username,String password) { StringWriter sw= new StringWriter(); PrintWriter pw = new PrintWriter(); try{ URL url=new URL(urlString); String userPwd= username+":"+password; BASE64Encoder encoder = new BASE64Encoder(); String encodedStr = encoder.encode (userPwd.getBytes()); System.out.println("Original String = " + userPwd); System.out.println("Encoded String = " + encodedStr); HttpURLConnection huc=(HttpURLConnection) url.openConnection(); huc.setRequestProperty( "Authorization"," "+encodedStr); InputStream content = (InputStream)huc.getInputStream(); BufferedReader in = new BufferedReader (new InputStreamReader (content)); String line; while ((line = in.readLine())...	import java.io.; import java.net.; import java.misc.BASE64Encoder; public class BruteForce { public BruteForce() {} public boolean fetchURL(String urlString,String username,String password) { StringWriter = new StringWriter(); PrintWriter pw = new PrintWriter(); try{ URL url=new URL(urlString); String userPwd= username+":"+password; BASE64Encoder encoder = new BASE64Encoder(); String encodedStr = encoder.encode (userPwd.getBytes()); System.out.println("Original String = " + userPwd); System.out.println("Encoded String = " + encodedStr); HttpURLConnection huc=(HttpURLConnection) url.openConnection(); huc.setRequestProperty( "Authorization"," "+encodedStr); InputStream content = (InputStream)huc.getInputStream(); BufferedReader in = new BufferedReader (new InputStreamReader (content)); String line; while ((line = in.readLine()) ...	import java.net.; import java.io.; import java.util.*; public class Dictionary{ private static URL location; private static String user; private BufferedReader input; private static BufferedReader dictionary; private int maxLetters = 3; public Dictionary() { Authenticator.setDefault(new MyAuthenticator ()); startTime = System.currentTimeMillis(); boolean passwordMatched = false; while (!passwordMatched) { try { input = new BufferedReader(new InputStreamReader(location.openStream())); String line = input.readLine(); while (line != null) { System.out.println(line); line = input.readLine(); } input.close(); passwordMatched = true; } catch (ProtocolException e) { } catch (ConnectException e) { System.out.println("Failed connect"); } catch (IOException e) ...
  import java.io.InputStream; import java.util.Properties; import javax.naming.Context; import javax.naming.InitialContext; import javax.rmi.PortableRemoteObject; import javax.sql.DataSource; public class WatchdogPropertyHelper { private static Properties testProps; public WatchdogPropertyHelper() { } public static String getProperty(String pKey){ try{ initProps(); } catch(Exception e){ System.err.println("Error init'ing the watchddog Props"); e.printStackTrace(); } return testProps.getProperty(pKey); } private static void initProps() throws Exception{ if(testProps == null){ testProps = new Properties(); InputStream fis = WatchdogPropertyHelper.class.getResourceAsStream("/watchdog.properties"); testProps.load(fis); } } }	import java.io.InputStream; import java.util.Properties; import javax.naming.Context; import javax.naming.InitialContext; import javax.rmi.PortableRemoteObject; import javax.sql.DataSource; public class BruteForcePropertyHelper { private static Properties bruteForceProps; public BruteForcePropertyHelper() { } public static String getProperty(String pKey){ try{ initProps(); } catch(Exception e){ System.err.println("Error init'ing the burteforce Props"); e.printStackTrace(); } return bruteForceProps.getProperty(pKey); } private static void initProps() throws Exception{ if(bruteForceProps == null){ bruteForceProps = new Properties(); InputStream fis = BruteForcePropertyHelper.class.getResourceAsStream("/bruteforce.properties"); bruteForceProps.load(fis); } } }	import java.io.; import java.net.; import javax.swing.Timer; import java.awt.event.*; import javax.swing.JOptionPane; public class WatchDog { private static Process pro = null; private static Runtime run = Runtime.getRuntime(); public static void main(String[] args) { String cmd = null; try { cmd = new String("wget -O original.txt http://www.cs.rmit.edu./students/"); pro = run.exec(cmd); System.out.println(cmd); } catch (IOException e) { } class Watch implements ActionListener { BufferedReader in = null; String str = null; Socket socket; public void actionPerformed (ActionEvent event) { try { System.out.println("in Watch!"); String cmd = new String(); int ERROR = 1; cmd = new String("wget -O new.txt http://www.cs.rmit.edu./students/"); System.out.println(cmd); cmd = new String("diff original.txt new.txt"); pro = run.exec(cmd); System.out.println(cmd); in = new Buf...
  import java.net.; import java.io.; public class BruteForce { private static String password=" "; public static void main(String[] args) { String Result=""; if (args.length<1) { System.out.println("Error: Correct Format Filename, username e.g<>"); System.exit(1); } BruteForce bruteForce1 = new BruteForce(); Result=bruteForce1.Password("http://sec-crack.cs.rmit.edu./SEC/2/",args[0]); System.out.println("The Password of "+args[0]+"is.."+Result); } private String Password(String urlString,String username) { int cnt=0; t0 = System.currentTimeMillis(); for ( char ch = 'A'; ch <= 'z'; ch++ ) { if (ch>'Z' && ch<'a') { ch='a'; } for ( char ch1 = 'A'; ch1 <= 'z'; ch1++ ) { if (ch1>'Z' && ch1<'a') { ch1='a'; } for ( char ch2 = 'A'; ch2 <= 'z'; ch2++ ) { if (ch2>'Z' && ch2<'a') { ...	import java.net.; import java.io.; import java.util.Date; public class Dictionary{ private static String password=" "; public static void main(String[] args) { String Result=""; if (args.length<1) { System.out.println("Correct Format Filename username e.g<>"); System.exit(1); } Dictionary dicton1 = new Dictionary(); Result=dicton1.Dict("http://sec-crack.cs.rmit.edu./SEC/2/",args[0]); System.out.println("Cracked Password for The User "+args[0]+" The Password is.."+Result); } private String Dict(String urlString,String username) { int cnt=0; FileInputStream stream=null; DataInputStream word=null; try{ stream = new FileInputStream ("/usr/share/lib/dict/words"); word =new DataInputStream(stream); t0 = System.currentTimeMillis(); while (word.available() !=0) { password=word.readLine(); if (password.length()!=3) { continue; } System.out.print("...	package java.httputils; import java.io.IOException; import java.net.MalformedURLException; import java.util.ArrayList; import java.util.Iterator; public class RunnableHttpRequest extends Thread { protected String targetURL = "http://localhost:8080/"; protected int requestCount = 1; protected ArrayList timingList = new ArrayList(); protected HttpRequestClient req; Boolean finished = new Boolean(false); HttpRequestThreadPool pool; public void run() { try { for (int i = 0; i < getRequestCount() && !getFinished().booleanValue(); i++) { try { req = new HttpRequestClient(getTargetURL()); } catch (MalformedURLException e) { e.printStackTrace(); break; } catch (IOException e) { ...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Evaluation Dataset

soco_train_java

• Dataset: soco_train_java at 44ca4ff
• Size: 4,296 evaluation samples
• Columns: anchor_code, positive_code, and negative_code
• Approximate statistics based on the first 1000 samples:

  	anchor_code	positive_code	negative_code
  type	string	string	string
  details	min: 51 tokens mean: 465.22 tokens max: 512 tokens	min: 51 tokens mean: 464.66 tokens max: 512 tokens	min: 51 tokens mean: 458.05 tokens max: 512 tokens

• Samples:

  anchor_code	positive_code	negative_code
  import java.util.; import java.io.; public class WatchDog { public static void main(String args[]) { Runtime rt1 = Runtime.getRuntime(); Process prss1= null; try { prss1 = rt1.exec("wget -R mpg,mpeg, --output-document=first.html http://www.cs.rmit.edu./students/"); }catch(java.io.IOException e){} MyWatchDogTimer w = new MyWatchDogTimer(); Timer time = new Timer(); time.schedule(w,864000000,864000000); } }	import java.util.; import java.io.; public class MyTimer { public static void main(String args[]) { Watchdog watch = new Watchdog(); Timer time = new Timer(); time.schedule(watch,864000000,864000000); } }	import java.net.; import java.io.; import java.util.Vector; import java.util.Date; import java.security.; public class Dictionary { public static BufferedReader in; public static void main(String[] args) throws Exception { String baseURL = "http://sec-crack.cs.rmit.edu./SEC/2/index.php"; int count=0; Date date = new Date(); startTime=date.getTime(); int LIMITINMINUTES=45; int TIMELIMIT=LIMITINMINUTES1000*60; boolean timedOut=false; boolean found=false; Vector dictionary=new Vector(readWords()); System.out.println("Words in dictionary: "+dictionary.size()); while (found==false && timedOut==false && dictionary.elementAt(count)!=null) { Date endDate = new Date(); endTime=endDate.getTime(); if (endTime>(TIMELIMIT+startTime)){ System.out.println("Timed out"); timedOut=true; } String password = ""; ...
  import java.io.InputStream; import java.util.Properties; import javax.naming.Context; import javax.naming.InitialContext; import javax.rmi.PortableRemoteObject; import javax.sql.DataSource; public class MailsendPropertyHelper { private static Properties testProps; public MailsendPropertyHelper() { } public static String getProperty(String pKey){ try{ initProps(); } catch(Exception e){ System.err.println("Error init'ing the watchddog Props"); e.printStackTrace(); } return testProps.getProperty(pKey); } private static void initProps() throws Exception{ if(testProps == null){ testProps = new Properties(); InputStream fis = MailsendPropertyHelper.class.getResourceAsStream("/mailsend.properties"); testProps.load(fis); } } }	import java.io.InputStream; import java.util.Properties; import javax.naming.Context; import javax.naming.InitialContext; import javax.rmi.PortableRemoteObject; import javax.sql.DataSource; public class BruteForcePropertyHelper { private static Properties bruteForceProps; public BruteForcePropertyHelper() { } public static String getProperty(String pKey){ try{ initProps(); } catch(Exception e){ System.err.println("Error init'ing the burteforce Props"); e.printStackTrace(); } return bruteForceProps.getProperty(pKey); } private static void initProps() throws Exception{ if(bruteForceProps == null){ bruteForceProps = new Properties(); InputStream fis = BruteForcePropertyHelper.class.getResourceAsStream("/bruteforce.properties"); bruteForceProps.load(fis); } } }	import java.net.; import java.io.; import java.Ostermiller.util.; import java.util.; public class MyClient2 implements Runnable { private String hostname; private int port; private String filename; private Socket s; private int n; private InputStream sin; private OutputStream sout; private int dif; private String myPassword; private int status; private int myTime; private BruteForce myMaster; public MyClient2(BruteForce bf , int num, int myPort, String password) { hostname = new String("sec-crack.cs.rmit.edu."); port = myPort; status = 0; myTime = 0; myPassword = password; filename = new String("/SEC/2/"); myMaster = 0; n = num; dif = 0; } public getDif() { return dif; } public int getStatus() { return status; } public void run() { String inputLine; String[] tokens = new String[5]; int i; myTime = 0; ...
  import java.io.; import java.net.; import java.util.*; public class Dictionary { public static void main (String args[]) { Calendar cal = Calendar.getInstance(); Date now=cal.getTime(); double startTime = now.getTime(); String password=getPassword(startTime); System.out.println("The password is " + password); } public static String getPassword(double startTime) { String password=""; int requests=0; try { FileReader fRead = new FileReader("/usr/share/lib/dict/words"); BufferedReader buf = new BufferedReader(fRead); password=buf.readLine(); while (password != null) { if (password.length()<=3) { requests++; if (testPassword(password, startTime, requests)) return password; } password = buf.readLine(); } } catch (IOException ioe) { } return password; } private static boolean testPassword(String password, double startTime, int requests) { try { U...	import java.io.; import java.net.; import java.util.*; public class BruteForce { public static void main(String args[]) { Calendar cal = Calendar.getInstance(); Date now=cal.getTime(); double startTime = now.getTime(); String password=getPassword(startTime); System.out.println("The password is " + password); } public static String getPassword(double startTime) { char first, second, third; String password=""; int requests=0; for (int i=65; i<123; i++) { requests++; first = (char) i; password = first + ""; if (testPassword(password, startTime, requests)) return password; for (int j=65; j<123; j++) { requests++; second = (char) j; password = first + "" + second; if (testPassword(password, startTime, requests)) return password; for (int k=65; k<123; k++) { requests++; third = (char) k; password = first + "" + second + "" + third; if (test...	import java.misc.BASE64Encoder; import java.misc.BASE64Decoder; import java.io.; import java.net.; import java.util.*; public class Dictionary { public Dictionary(String url, String dictionaryFile) { try{ this.url = url; this.dictionaryPath = dictionaryFile; InputStream fis = new FileInputStream(this.dictionaryPath); dict = new BufferedReader(new InputStreamReader(fis)); }catch(IOException ioe){ System.out.println("Error opening dictionary file:\n" +ioe); } } private String url = null; private String dictionaryPath = null; private BufferedReader dict = null; private int attempts = 0; private int passwordSize = 3; public void setPasswordSize(int size){ this.passwordSize = size; } public String getNextPassword()throws IOException{ String line = dict.readLine(); while(line!=null&&line.length()!=this.passwordSize ) line = dict.readLine(); return line; } publ...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• num_train_epochs: 1
• warmup_ratio: 0.1
• fp16: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	Validation Loss
0.0414	100	0.9269	0.3356
0.0827	200	0.3858	0.3603
0.1241	300	0.3759	0.3187
0.1655	400	0.3333	0.3243
0.2069	500	0.3541	0.3245
0.2482	600	0.3642	0.2986
0.2896	700	0.3402	0.3425
0.3310	800	0.3569	0.2971
0.3724	900	0.3429	0.2883
0.4137	1000	0.3484	0.3116
0.4551	1100	0.3331	0.2919
0.4965	1200	0.3277	0.2827
0.5379	1300	0.3195	0.2919
0.5792	1400	0.327	0.2784
0.6206	1500	0.3207	0.2833
0.6620	1600	0.3173	0.2691
0.7034	1700	0.3036	0.2782
0.7447	1800	0.3229	0.2727
0.7861	1900	0.3025	0.2742
0.8275	2000	0.2927	0.2735
0.8688	2100	0.2983	0.2792
0.9102	2200	0.2914	0.2677
0.9516	2300	0.2767	0.2671
0.9930	2400	0.3406	0.2614

Framework Versions

• Python: 3.11.13
• Sentence Transformers: 4.1.0
• Transformers: 4.52.4
• PyTorch: 2.6.0+cu124
• Accelerate: 1.7.0
• Datasets: 3.6.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}