mstyslavity/monad_random_experimental

merge

This is a merge of pre-trained language models created using mergekit.

Merge Details

Merge Method

This model was merged using the Passthrough merge method using PleIAs/Monad as a base.

Models Merged

The following models were included in the merge:

• PleIAs/Baguettotron

Configuration

The following YAML configuration was used to produce this model:

merge_method: passthrough
dtype: bfloat16
out_dtype: float32
base_model: PleIAs/Monad
slices:
  - sources:
    - model: PleIAs/Monad
      layer_range: [0,64]
  - sources:
    - model: PleIAs/Monad
      layer_range: [10,36]
  - sources:
    - model: PleIAs/Monad
      layer_range: [12,30]
  - sources:
    - model: PleIAs/Monad
      layer_range: [0,24]
  - sources:
    - model: PleIAs/Monad
      layer_range: [16,32]
  - sources:
    - model: PleIAs/Monad
      layer_range: [48,64]
  - sources:
    - model: PleIAs/Baguettotron
      layer_range: [0,40]
  - sources:
    - model: PleIAs/Baguettotron
      layer_range: [40,80]
tokenizer:
  source: base
parameters:
  normalize: true