bert-small-amharic-32k-l12-384-512

This model is a fine-tuned version of yosefw/bert-small-amharic-32k-l12-384 on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 2.6089
• Model Preparation Time: 0.0031

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 64
• eval_batch_size: 64
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• lr_scheduler_warmup_steps: 1000
• num_epochs: 5
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Model Preparation Time
6.5343	0.1249	1038	6.2580	0.0031
5.406	0.2498	2076	3.3001	0.0031
3.6032	0.3746	3114	2.8935	0.0031
3.1451	0.4995	4152	2.8139	0.0031
3.0243	0.6244	5190	2.7799	0.0031
2.9753	0.7493	6228	2.7563	0.0031
2.9393	0.8742	7266	2.7340	0.0031
2.9185	0.9990	8304	2.7174	0.0031
2.8944	1.1239	9342	2.7048	0.0031
2.8848	1.2488	10380	2.7031	0.0031
2.8722	1.3737	11418	2.6815	0.0031
2.8632	1.4986	12456	2.6920	0.0031
2.8531	1.6234	13494	2.6793	0.0031
2.8492	1.7483	14532	2.6678	0.0031
2.843	1.8732	15570	2.6701	0.0031
2.8383	1.9981	16608	2.6590	0.0031
2.8307	2.1230	17646	2.6472	0.0031
2.8206	2.2478	18684	2.6565	0.0031
2.8204	2.3727	19722	2.6557	0.0031
2.8135	2.4976	20760	2.6481	0.0031
2.8153	2.6225	21798	2.6508	0.0031
2.811	2.7474	22836	2.6377	0.0031
2.8084	2.8722	23874	2.6341	0.0031
2.8062	2.9971	24912	2.6377	0.0031
2.8026	3.1220	25950	2.6340	0.0031
2.7967	3.2469	26988	2.6456	0.0031
2.7985	3.3718	28026	2.6262	0.0031
2.7953	3.4966	29064	2.6293	0.0031
2.7892	3.6215	30102	2.6321	0.0031
2.7941	3.7464	31140	2.6327	0.0031
2.7913	3.8713	32178	2.6360	0.0031
2.7866	3.9962	33216	2.6336	0.0031
2.785	4.1210	34254	2.6318	0.0031
2.7849	4.2459	35292	2.6196	0.0031
2.7846	4.3708	36330	2.6182	0.0031
2.7803	4.4957	37368	2.6323	0.0031
2.7834	4.6205	38406	2.6172	0.0031
2.7782	4.7454	39444	2.6122	0.0031
2.7797	4.8703	40482	2.6177	0.0031
2.78	4.9952	41520	2.6248	0.0031

Framework versions

• Transformers 4.51.3
• Pytorch 2.6.0+cu124
• Datasets 3.6.0
• Tokenizers 0.21.1