hallucinate, to deceiue, or blind https://extra.shu.ac.uk/emls/iemls/work/etexts/caw1604w_removed.htm
The main focus of the experiment is a steming of the corpus jsonl + 63355 vocab extended from 32752 (with those stemma).
Not intended for inference before additional vocab extentions and then slower CPT on real texts without stemming.
python auto_emoji_plot.py <(cat <(head -1 training_log.csv) <(tail -35 training_log.csv))
LOSS (EMA: β, Raw: Β·, emojis: state):
βl12.6284
βo
βπ₯
βs
β
β 11β2124
β
β π
β β
β
β 9.7964
β Β· β
β
β π
β β
β 8.3805
β Β· β π
β β
β Β· β π
β Β· β
β 6.9645 Β· β π
β Β· Β· β β π
β Β· Β· Β· β β β’ π
β Β· Β· Β· β β β β π π β π π π π β π
β Β· Β· Β· β β β β β β β β β β β β β β β β β
β
β 5.5485
β00ββββββββββββββββββββ440βββββββββββββββββββββ780βββββββββββββββββββ1120ββββββββββββββββββββ1460ββββββββββββββββββstepβ
Legend: β = EMA trend | Β· = Raw data | π₯ = Start | β/π = Increase | β/π = Decrease | β’ = Flat
Summary: π‘ Plateau or noisy (flat trend). Trend Scale: 12.0384 β 6.1669 (Ξ 5.8715)
LEARNING_RATE (EMA: β, Raw: Β·, emojis: state):
βl1.06e-04
βe
βπ π π π
βr β β β β β β π
βn Β· β β β π π
βi9.11e-05 Β· β β β π
βn Β· Β· β β
βg Β· β π
β_ Β· Β· β β π
βr Β· β π
βa7.57e-05 Β· Β· β β
βt Β· β π
βe Β· β π
β Β· β β
β Β· β π
β 6.04e-05 Β· Β· β
β Β· β π
β Β· β
β Β· β π
β Β· β π
β 4.50e-05 Β· β β
β Β· β π
β Β· β
β Β· Β·
β Β·
β
β 2.97e-05
β00ββββββββββββββββββββ440βββββββββββββββββββββ780βββββββββββββββββββ1120ββββββββββββββββββββ1460ββββββββββββββββββstepβ
Legend: β = EMA trend | Β· = Raw data | π’ = <1e-8 | πΆ = 1e-8~1e-6 | π = 1e-6~1e-4 | π = 1e-4~1e-3 | π = >1e-3
Summary: π’ Strong learning phase. Current: 3.61e-05
GRAD_NORM (EMA: β, Raw: Β·, emojis: state):
βg66.69
βr
βπ¨
βd
β_
βn52.43π¨
βo β β
βr Β·
βm β
β
β 38.16
β
β Β· π¨
β β
β Β·
β 23.89
β β
β β
β β
β
β 9.62 β β
β β
β β β
β
β Β· Β· β β β β β β β β β β β β Β· β Β· π¨
β Β· Β· Β· Β· Β· Β· Β· β β β β β β β β β β β β β β β β β β β β β β
β
β -4.64
β00ββββββββββββββββββββ440βββββββββββββββββββββ780βββββββββββββββββββ1120ββββββββββββββββββββ1460ββββββββββββββββββstepβ
Legend: β = EMA trend | Β· = Raw data | π³ = <0.3 (vanishing) | β
= 0.3~2.0 (stable) | β = 2.0~4.0 (high) | π¨ = >4.0 (explosion)
Summary: π΄ ALERT: Gradient explosion risk! Current: 4.03
ENGLISH_PPL (EMA: β, Raw: Β·, emojis: state):
βe669.7
βn
βg π« π« π«
βl π« π« Β· π« β β β β β
βi Β· π« Β· π« Β· β β β β β β Β· Β·
βs655.1 Β· β β β β
βh Β· Β· Β· π« π« β
β_ Β· π« π« Β· β β β
βp Β· Β· π« β β β β β
βp π« Β· π« Β· β
βl640.6 β β β β
β Β· β
β π«
β β
β
β 626.1 β
β Β·
β
β π«
β β
β 611.6
β
β β
βπ«
ββ
β
β 597.0
β00ββββββββββββββββββββ440βββββββββββββββββββββ780βββββββββββββββββββ1120ββββββββββββββββββββ1460ββββββββββββββββββstepβ
Legend: β = EMA trend | Β· = Raw data | Emojis show uncertainty state (π« high, β med, β low)
Summary: π΅ Stable. Current: 659.3, range: 603.1β662.2 (Ξ 59.2)
RUSSIAN_PPL (EMA: β, Raw: Β·, emojis: state):
βr103775.3
βu
βπ«
βs
βi
βa85153.8
βn β
β_
βp π«
βp β
βl66532.3
β
β
β β
β Β·
β 47910.7 π«
β β
β
β Β· β
β π«
β 29289.2 Β· β
β β π«
β β π«
β Β· β β β π« π« π« π« π« π« π« π« π« π« π« π«
β Β· Β· Β· Β· Β· Β· Β· β β β β β β β β β β β β β β β β β β β β β β β
β
β 10667.7
β00ββββββββββββββββββββ440βββββββββββββββββββββ780βββββββββββββββββββ1120ββββββββββββββββββββ1460ββββββββββββββββββstepβ
Legend: β = EMA trend | Β· = Raw data | Emojis show uncertainty state (π« high, β med, β low)
Summary: π΅ Stable. Current: 18631.2, range: 18572.9β96016.4 (Ξ 77443.5)
PPL_DELTA_EN (EMA: β, Raw: Β·, emojis: state):
βp65.3
βp
βl π« π« π«
β_ π« π« Β· π« β β β β β
βd Β· π« Β· π« Β· β β β β β β Β· Β·
βe50.8 Β· β β β β
βl Β· Β· Β· π« π« β
βt Β· π« π« Β· β β β
βa Β· Β· π« β β β β β
β_ π« Β· π« Β· β
βe36.3 β β β β
βn Β· β
β π«
β β
β
β 21.7 β
β Β·
β
β π«
β β
β 7.2
β
β β
ββ
ββ
β
β -7.3
β00ββββββββββββββββββββ440βββββββββββββββββββββ780βββββββββββββββββββ1120ββββββββββββββββββββ1460ββββββββββββββββββstepβ
Legend: β = EMA trend | Β· = Raw data | Emojis show uncertainty state (π« high, β med, β low)
Summary: π΅ Stable. Current: 55.0, range: -1.3β57.9 (Ξ 59.2)
PPL_DELTA_RU (EMA: β, Raw: Β·, emojis: state):
βp-52580.3
βp
ββ
β_
βd
βe-71201.9
βl β
βt
βa π«
β_ β
βr-89823.4
βu
β
β β
β Β·
β -108444.9 π«
β β
β
β Β· β
β π«
β -127066.5Β· β
β β π«
β β π«
β Β· β β β π« π« π« π« π« π« π« π« π« π« π« π«
β Β· Β· Β· Β· Β· Β· Β· β β β β β β β β β β β β β β β β β β β β β β β
β
β -145688.0
β00ββββββββββββββββββββ440βββββββββββββββββββββ780βββββββββββββββββββ1120ββββββββββββββββββββ1460ββββββββββββββββββstepβ
Legend: β = EMA trend | Β· = Raw data | Emojis show uncertainty state (π« high, β med, β low)
Summary: π Decreasing. Current: -137724.5, range: -137782.8β-60339.3 (Ξ 77443.5)
Root folder has BF16 run chechpkoint-1500 visualized above.
Continued F16 in a separate checkpoint-1000 (newer) run with a newer script from checkpoint-1000 to checkpoint-1000 (yes, same names... to confuse readers)):
python continual_train_v2.py --model_path training_v5/checkpoint-1000/ --train_file /dev/shm/sed-f_98_3.huniq-c.list_4sed_mystem-n.rgΠ-Π―Π°-Ρ.jsonl --output_dir training_v5 --freeze_layers 4 --max_steps 3000 --batch_size 16 --gradient_accumulation 4 --learning_rate 1e-4 --warmup_steps 50 --block_size 512 --eval_every 50 --logging_steps 10 --save_steps 500 --auto_resume --use_fp16 --gradient_checkpointing --old_vocab_size 3275
...
Step 960 | loss=5.1411 | lr=7.83e-05 | grad_norm=4.734 | tok/s=2,595
Step 970 | loss=5.1374 | lr=7.79e-05 | grad_norm=5.862 | tok/s=2,595
Step 980 | loss=5.1325 | lr=7.74e-05 | grad_norm=3.418 | tok/s=2,595
Step 990 | loss=5.1275 | lr=7.70e-05 | grad_norm=2.287 | tok/s=2,596
Step 1000 | loss=5.1224 | lr=7.65e-05 | grad_norm=3.189 | tok/s=2,596
================================================================================
π EVALUATION | Step 1000 | Epoch 1.0000
================================================================================
Loss: 5.1224
LR: 0.00e+00
Grad norm: 0.00
Elapsed: 3.51 Ρ
English PPL: 1276.14 (+616.76)
Russian PPL: 2665.49 (-16108.01)
ΠΠ΅Π½Π΅ΡΠ°ΡΠΈΠΈ:
english:
Β«The capital of France isΒ»
β divided into two parts : namely, that it were better for thee, my dear...
russian:
Β«ΠΠΎΠ³ Π΅ΡΡΡΒ»
β ΠΠΠΠΠΎΠ³ ΠΌΡ ΠΠΎΠ³ ΠΎΠ½ Π‘οΏ½ΠΠΎΠ³ ΠΠΠΠΠΠ£ΠΠΠΠΠΠΠΠΡ...
LOSS (EMA: β, Raw: Β·, emojis: state):
βl6.2119
βo
βπ₯ π
βs β
β π
β 5.974Β· β π
β β
β π
β Β· β π
β β
β 5.7365 Β· π
β Β· β π
β β π
β Β· β π
β Β· β
β 5.4988 Β· π
β Β· β π π
β Β· β β π
β Β· β π
β Β· β π
β 5.2611 Β· β π
β Β· Β· β π π
β Β· β β π
β Β· Β· β
β Β· Β·
β
β 5.0233
β0βββββββββββββββββββββ240βββββββββββββββββββββ430ββββββββββββββββββββ620βββββββββββββββββββββ810ββββββββββββββββββstepβ
Legend: β = EMA trend | Β· = Raw data | π₯ = Start | β/π = Increase | β/π = Decrease | β’ = Flat
Summary: π’ Effective learning (downward trend). Trend Scale: 6.1129 β 5.1889 (Ξ 0.9240)
- Downloads last month
- 107
Inference Providers NEW
This model isn't deployed by any Inference Provider. π Ask for provider support
Model tree for foss22/Hallucinate1900-95M
Base model
croqaz/Sprocket-and-Say