| nohup: ignoring input |
| Begin main_assign: Llama-2-7b-hf self_attn alpha |
|
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards: 50%|█████ | 1/2 [00:17<00:17, 17.79s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:24<00:00, 11.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:24<00:00, 12.03s/it] |
| Once upon a time, I was a new mum, with a newborn baby. I was also a full-time teacher and doing a part-time Master's degree. I was tired and stressed. I had no time to do anything that I enjoyed. And then I was given a gift. A gift of a book that would change everything. |
| I was given the book _Babywise_ , by Gary Ezzo and Robert Bucknam. I was not sure what to expect. I was |
| LlamaForCausalLM( |
| (model): LlamaModel( |
| (embed_tokens): Embedding(32000, 4096, padding_idx=0) |
| (layers): ModuleList( |
| (0-31): 32 x LlamaDecoderLayer( |
| (self_attn): LlamaAttention( |
| (q_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (k_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (v_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (o_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| ) |
| (mlp): LlamaMLP( |
| (gate_proj): Linear(in_features=4096, out_features=11008, bias=False) |
| (up_proj): Linear(in_features=4096, out_features=11008, bias=False) |
| (down_proj): Linear(in_features=11008, out_features=4096, bias=False) |
| (act_fn): SiLU() |
| ) |
| (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05) |
| (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05) |
| ) |
| ) |
| (norm): LlamaRMSNorm((4096,), eps=1e-05) |
| (rotary_emb): LlamaRotaryEmbedding() |
| ) |
| (lm_head): Linear(in_features=4096, out_features=32000, bias=False) |
| ) |
| config: |
| LlamaConfig { |
| "architectures": [ |
| "LlamaForCausalLM" |
| ], |
| "attention_bias": false, |
| "attention_dropout": 0.0, |
| "bos_token_id": 1, |
| "eos_token_id": 2, |
| "head_dim": 128, |
| "hidden_act": "silu", |
| "hidden_size": 4096, |
| "initializer_range": 0.02, |
| "intermediate_size": 11008, |
| "max_position_embeddings": 4096, |
| "mlp_bias": false, |
| "model_type": "llama", |
| "num_attention_heads": 32, |
| "num_hidden_layers": 32, |
| "num_key_value_heads": 32, |
| "pad_token_id": 0, |
| "pretraining_tp": 1, |
| "rms_norm_eps": 1e-05, |
| "rope_scaling": null, |
| "rope_theta": 10000.0, |
| "tie_word_embeddings": false, |
| "torch_dtype": "float16", |
| "transformers_version": "4.55.2", |
| "use_cache": true, |
| "vocab_size": 32000 |
| } |
| |
| Processing layer 0--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 0 ---1.5555332899093628 |
| Processing layer 1--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 1 ---2.2105441093444824 |
| Processing layer 2--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 2 ---2.6319708824157715 |
| Processing layer 3--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 3 ---2.659501791000366 |
| Processing layer 4--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 4 ---2.5770697593688965 |
| Processing layer 5--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 5 ---2.5436081886291504 |
| Processing layer 6--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 6 ---2.4908900260925293 |
| Processing layer 7--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 7 ---2.5257110595703125 |
| Processing layer 8--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 8 ---2.3653383255004883 |
| Processing layer 9--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 9 ---2.5174360275268555 |
| Processing layer 10--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 10 ---2.265111207962036 |
| Processing layer 11--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 11 ---2.1847732067108154 |
| Processing layer 12--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 12 ---2.4449100494384766 |
| Processing layer 13--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 13 ---2.679959774017334 |
| Processing layer 14--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 14 ---2.4503092765808105 |
| Processing layer 15--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 15 ---2.7230710983276367 |
| Processing layer 16--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 16 ---3.074552536010742 |
| Processing layer 17--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 17 ---3.4709739685058594 |
| Processing layer 18--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 18 ---3.67897629737854 |
| Processing layer 19--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 19 ---3.278068780899048 |
| Processing layer 20--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 20 ---3.6138486862182617 |
| Processing layer 21--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 21 ---3.5603649616241455 |
| Processing layer 22--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 22 ---3.9758076667785645 |
| Processing layer 23--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 23 ---4.087326526641846 |
| Processing layer 24--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 24 ---3.739630699157715 |
| Processing layer 25--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 25 ---4.076397895812988 |
| Processing layer 26--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 26 ---3.5009336471557617 |
| Processing layer 27--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 27 ---4.056451320648193 |
| Processing layer 28--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 28 ---3.726351737976074 |
| Processing layer 29--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 29 ---3.844115972518921 |
| Processing layer 30--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 30 ---4.4837751388549805 |
| Processing layer 31--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 31 ---3.275714874267578 |
| metric_name alpha: [30, 23, 25, 27, 22, 29, 24, 28, 18, 20, 21, 26, 17, 19, 31, 16, 15, 13, 3, 2, 4, 5, 7, 9, 6, 14, 12, 8, 10, 1, 11, 0] |
| Begin main_assign: Llama-2-7b-hf self_attn alpha_hat |
|
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards: 50%|█████ | 1/2 [00:17<00:17, 17.91s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:24<00:00, 10.98s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:24<00:00, 12.02s/it] |
| Once upon a time, there was a boy who lived in a house with his parents, and there was a girl who lived in a house with her parents. There were other children, too, but they were less important, because they lived in other houses. |
| This boy had a sister, and the sister was very beautiful. She was so beautiful that the boy’s parents decided to give her away in marriage. The sister was quite upset about this, but her parents told her that she had no choice. |
| LlamaForCausalLM( |
| (model): LlamaModel( |
| (embed_tokens): Embedding(32000, 4096, padding_idx=0) |
| (layers): ModuleList( |
| (0-31): 32 x LlamaDecoderLayer( |
| (self_attn): LlamaAttention( |
| (q_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (k_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (v_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (o_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| ) |
| (mlp): LlamaMLP( |
| (gate_proj): Linear(in_features=4096, out_features=11008, bias=False) |
| (up_proj): Linear(in_features=4096, out_features=11008, bias=False) |
| (down_proj): Linear(in_features=11008, out_features=4096, bias=False) |
| (act_fn): SiLU() |
| ) |
| (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05) |
| (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05) |
| ) |
| ) |
| (norm): LlamaRMSNorm((4096,), eps=1e-05) |
| (rotary_emb): LlamaRotaryEmbedding() |
| ) |
| (lm_head): Linear(in_features=4096, out_features=32000, bias=False) |
| ) |
| config: |
| LlamaConfig { |
| "architectures": [ |
| "LlamaForCausalLM" |
| ], |
| "attention_bias": false, |
| "attention_dropout": 0.0, |
| "bos_token_id": 1, |
| "eos_token_id": 2, |
| "head_dim": 128, |
| "hidden_act": "silu", |
| "hidden_size": 4096, |
| "initializer_range": 0.02, |
| "intermediate_size": 11008, |
| "max_position_embeddings": 4096, |
| "mlp_bias": false, |
| "model_type": "llama", |
| "num_attention_heads": 32, |
| "num_hidden_layers": 32, |
| "num_key_value_heads": 32, |
| "pad_token_id": 0, |
| "pretraining_tp": 1, |
| "rms_norm_eps": 1e-05, |
| "rope_scaling": null, |
| "rope_theta": 10000.0, |
| "tie_word_embeddings": false, |
| "torch_dtype": "float16", |
| "transformers_version": "4.55.2", |
| "use_cache": true, |
| "vocab_size": 32000 |
| } |
| |
| Processing layer 0--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 0 ---6.557916641235352 |
| Processing layer 1--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 1 ---8.015230178833008 |
| Processing layer 2--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 2 ---10.61414909362793 |
| Processing layer 3--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 3 ---10.561344146728516 |
| Processing layer 4--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 4 ---10.057601928710938 |
| Processing layer 5--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 5 ---9.833120346069336 |
| Processing layer 6--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 6 ---9.45051097869873 |
| Processing layer 7--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 7 ---9.582311630249023 |
| Processing layer 8--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 8 ---9.064985275268555 |
| Processing layer 9--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 9 ---9.556177139282227 |
| Processing layer 10--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 10 ---8.45679759979248 |
| Processing layer 11--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 11 ---8.441352844238281 |
| Processing layer 12--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 12 ---9.276637077331543 |
| Processing layer 13--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 13 ---10.002967834472656 |
| Processing layer 14--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 14 ---8.847436904907227 |
| Processing layer 15--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 15 ---9.9490327835083 |
| Processing layer 16--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 16 ---11.152729988098145 |
| Processing layer 17--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 17 ---12.680035591125488 |
| Processing layer 18--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 18 ---13.309869766235352 |
| Processing layer 19--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 19 ---12.054608345031738 |
| Processing layer 20--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 20 ---13.724580764770508 |
| Processing layer 21--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 21 ---13.702856063842773 |
| Processing layer 22--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 22 ---15.829018592834473 |
| Processing layer 23--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 23 ---15.232747077941895 |
| Processing layer 24--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 24 ---14.636650085449219 |
| Processing layer 25--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 25 ---15.008004188537598 |
| Processing layer 26--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 26 ---14.163816452026367 |
| Processing layer 27--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 27 ---15.760412216186523 |
| Processing layer 28--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 28 ---14.682222366333008 |
| Processing layer 29--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 29 ---15.929686546325684 |
| Processing layer 30--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 30 ---17.405780792236328 |
| Processing layer 31--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 31 ---14.380212783813477 |
| metric_name alpha_hat: [30, 29, 22, 27, 23, 25, 28, 24, 31, 26, 20, 21, 18, 17, 19, 16, 2, 3, 4, 13, 15, 5, 7, 9, 6, 12, 8, 14, 10, 11, 1, 0] |
| Begin main_assign: Llama-2-7b-hf self_attn stable_rank |
|
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards: 50%|█████ | 1/2 [00:19<00:19, 19.27s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:25<00:00, 11.89s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:25<00:00, 13.00s/it] |
| Once upon a time, in a land far away, there was a castle. Everyone who lived in the castle was very wealthy. But the castle was haunted! The king and queen were very scared, but they kept the castle because they wanted the money. |
| One day, the king and queen had a baby. They called him Prince Charming. Prince Charming was very happy. He had everything he could ever want. He was so happy that he could fly! |
| One day, |
| LlamaForCausalLM( |
| (model): LlamaModel( |
| (embed_tokens): Embedding(32000, 4096, padding_idx=0) |
| (layers): ModuleList( |
| (0-31): 32 x LlamaDecoderLayer( |
| (self_attn): LlamaAttention( |
| (q_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (k_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (v_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (o_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| ) |
| (mlp): LlamaMLP( |
| (gate_proj): Linear(in_features=4096, out_features=11008, bias=False) |
| (up_proj): Linear(in_features=4096, out_features=11008, bias=False) |
| (down_proj): Linear(in_features=11008, out_features=4096, bias=False) |
| (act_fn): SiLU() |
| ) |
| (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05) |
| (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05) |
| ) |
| ) |
| (norm): LlamaRMSNorm((4096,), eps=1e-05) |
| (rotary_emb): LlamaRotaryEmbedding() |
| ) |
| (lm_head): Linear(in_features=4096, out_features=32000, bias=False) |
| ) |
| config: |
| LlamaConfig { |
| "architectures": [ |
| "LlamaForCausalLM" |
| ], |
| "attention_bias": false, |
| "attention_dropout": 0.0, |
| "bos_token_id": 1, |
| "eos_token_id": 2, |
| "head_dim": 128, |
| "hidden_act": "silu", |
| "hidden_size": 4096, |
| "initializer_range": 0.02, |
| "intermediate_size": 11008, |
| "max_position_embeddings": 4096, |
| "mlp_bias": false, |
| "model_type": "llama", |
| "num_attention_heads": 32, |
| "num_hidden_layers": 32, |
| "num_key_value_heads": 32, |
| "pad_token_id": 0, |
| "pretraining_tp": 1, |
| "rms_norm_eps": 1e-05, |
| "rope_scaling": null, |
| "rope_theta": 10000.0, |
| "tie_word_embeddings": false, |
| "torch_dtype": "float16", |
| "transformers_version": "4.55.2", |
| "use_cache": true, |
| "vocab_size": 32000 |
| } |
| |
| Processing layer 0--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| frobenius_norm tensor(55.2111, device='cuda:0') |
| spectral_norm tensor(21.8310, device='cuda:0') |
| frobenius_norm tensor(61.8584, device='cuda:0') |
| spectral_norm tensor(18.5959, device='cuda:0') |
| frobenius_norm tensor(45.2757, device='cuda:0') |
| spectral_norm tensor(4.0318, device='cuda:0') |
| frobenius_norm tensor(29.4097, device='cuda:0') |
| spectral_norm tensor(4.3865, device='cuda:0') |
| alpha value of layer 0 ---47.129005432128906 |
| Processing layer 1--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| frobenius_norm tensor(108.3667, device='cuda:0') |
| spectral_norm tensor(18.4210, device='cuda:0') |
| frobenius_norm tensor(108.1698, device='cuda:0') |
| spectral_norm tensor(20.1468, device='cuda:0') |
| frobenius_norm tensor(41.1449, device='cuda:0') |
| spectral_norm tensor(3.2622, device='cuda:0') |
| frobenius_norm tensor(33.7843, device='cuda:0') |
| spectral_norm tensor(3.8698, device='cuda:0') |
| alpha value of layer 1 ---74.68388366699219 |
| Processing layer 2--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| frobenius_norm tensor(109.0373, device='cuda:0') |
| spectral_norm tensor(16.2222, device='cuda:0') |
| frobenius_norm tensor(114.6870, device='cuda:0') |
| spectral_norm tensor(19.2055, device='cuda:0') |
| frobenius_norm tensor(58.7294, device='cuda:0') |
| spectral_norm tensor(3.5355, device='cuda:0') |
| frobenius_norm tensor(56.9624, device='cuda:0') |
| spectral_norm tensor(6.0202, device='cuda:0') |
| alpha value of layer 2 ---111.57455444335938 |
| Processing layer 3--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| frobenius_norm tensor(103.1725, device='cuda:0') |
| spectral_norm tensor(13.4197, device='cuda:0') |
| frobenius_norm tensor(107.3360, device='cuda:0') |
| spectral_norm tensor(15.3238, device='cuda:0') |
| frobenius_norm tensor(55.7174, device='cuda:0') |
| spectral_norm tensor(3.0726, device='cuda:0') |
| frobenius_norm tensor(54.5137, device='cuda:0') |
| spectral_norm tensor(6.3542, device='cuda:0') |
| alpha value of layer 3 ---127.65095520019531 |
| Processing layer 4--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| frobenius_norm tensor(107.5970, device='cuda:0') |
| spectral_norm tensor(13.7911, device='cuda:0') |
| frobenius_norm tensor(109.8770, device='cuda:0') |
| spectral_norm tensor(15.7544, device='cuda:0') |
| frobenius_norm tensor(58.8042, device='cuda:0') |
| spectral_norm tensor(3.1027, device='cuda:0') |
| frobenius_norm tensor(57.5345, device='cuda:0') |
| spectral_norm tensor(5.8239, device='cuda:0') |
| alpha value of layer 4 ---141.5764617919922 |
| Processing layer 5--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| frobenius_norm tensor(108.1569, device='cuda:0') |
| spectral_norm tensor(13.7375, device='cuda:0') |
| frobenius_norm tensor(112.6002, device='cuda:0') |
| spectral_norm tensor(16.5425, device='cuda:0') |
| frobenius_norm tensor(60.3288, device='cuda:0') |
| spectral_norm tensor(2.9971, device='cuda:0') |
| frobenius_norm tensor(59.0258, device='cuda:0') |
| spectral_norm tensor(5.6560, device='cuda:0') |
| alpha value of layer 5 ---155.60284423828125 |
| Processing layer 6--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| frobenius_norm tensor(102.0697, device='cuda:0') |
| spectral_norm tensor(12.2985, device='cuda:0') |
| frobenius_norm tensor(104.0983, device='cuda:0') |
| spectral_norm tensor(14.5729, device='cuda:0') |
| frobenius_norm tensor(55.9676, device='cuda:0') |
| spectral_norm tensor(3.0027, device='cuda:0') |
| frobenius_norm tensor(55.1871, device='cuda:0') |
| spectral_norm tensor(5.7670, device='cuda:0') |
| alpha value of layer 6 ---139.72598266601562 |
| Processing layer 7--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| frobenius_norm tensor(101.8819, device='cuda:0') |
| spectral_norm tensor(11.7685, device='cuda:0') |
| frobenius_norm tensor(102.3675, device='cuda:0') |
| spectral_norm tensor(13.6839, device='cuda:0') |
| frobenius_norm tensor(56.6824, device='cuda:0') |
| spectral_norm tensor(3.1391, device='cuda:0') |
| frobenius_norm tensor(55.6199, device='cuda:0') |
| spectral_norm tensor(5.4573, device='cuda:0') |
| alpha value of layer 7 ---140.21083068847656 |
| Processing layer 8--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| frobenius_norm tensor(102.8848, device='cuda:0') |
| spectral_norm tensor(11.9707, device='cuda:0') |
| frobenius_norm tensor(103.4811, device='cuda:0') |
| spectral_norm tensor(14.2754, device='cuda:0') |
| frobenius_norm tensor(58.2330, device='cuda:0') |
| spectral_norm tensor(3.3746, device='cuda:0') |
| frobenius_norm tensor(57.2962, device='cuda:0') |
| spectral_norm tensor(4.9391, device='cuda:0') |
| alpha value of layer 8 ---139.6942138671875 |
| Processing layer 9--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| frobenius_norm tensor(103.0146, device='cuda:0') |
| spectral_norm tensor(12.2318, device='cuda:0') |
| frobenius_norm tensor(105.5969, device='cuda:0') |
| spectral_norm tensor(14.2079, device='cuda:0') |
| frobenius_norm tensor(59.3876, device='cuda:0') |
| spectral_norm tensor(3.2388, device='cuda:0') |
| frobenius_norm tensor(58.5812, device='cuda:0') |
| spectral_norm tensor(5.1232, device='cuda:0') |
| alpha value of layer 9 ---148.2813262939453 |
| Processing layer 10--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| frobenius_norm tensor(102.8745, device='cuda:0') |
| spectral_norm tensor(12.0922, device='cuda:0') |
| frobenius_norm tensor(106.0223, device='cuda:0') |
| spectral_norm tensor(14.3113, device='cuda:0') |
| frobenius_norm tensor(58.7986, device='cuda:0') |
| spectral_norm tensor(3.2255, device='cuda:0') |
| frobenius_norm tensor(58.3338, device='cuda:0') |
| spectral_norm tensor(4.3048, device='cuda:0') |
| alpha value of layer 10 ---160.80075073242188 |
| Processing layer 11--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| frobenius_norm tensor(97.7702, device='cuda:0') |
| spectral_norm tensor(11.1815, device='cuda:0') |
| frobenius_norm tensor(97.4910, device='cuda:0') |
| spectral_norm tensor(12.9324, device='cuda:0') |
| frobenius_norm tensor(61.3144, device='cuda:0') |
| spectral_norm tensor(3.4012, device='cuda:0') |
| frobenius_norm tensor(60.7354, device='cuda:0') |
| spectral_norm tensor(5.6488, device='cuda:0') |
| alpha value of layer 11 ---143.46920776367188 |
| Processing layer 12--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| frobenius_norm tensor(99.7752, device='cuda:0') |
| spectral_norm tensor(11.8016, device='cuda:0') |
| frobenius_norm tensor(102.8686, device='cuda:0') |
| spectral_norm tensor(13.5998, device='cuda:0') |
| frobenius_norm tensor(60.5482, device='cuda:0') |
| spectral_norm tensor(3.2452, device='cuda:0') |
| frobenius_norm tensor(60.0323, device='cuda:0') |
| spectral_norm tensor(4.9509, device='cuda:0') |
| alpha value of layer 12 ---155.95849609375 |
| Processing layer 13--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| frobenius_norm tensor(98.8230, device='cuda:0') |
| spectral_norm tensor(12.0874, device='cuda:0') |
| frobenius_norm tensor(100.6162, device='cuda:0') |
| spectral_norm tensor(13.8355, device='cuda:0') |
| frobenius_norm tensor(62.7593, device='cuda:0') |
| spectral_norm tensor(3.1144, device='cuda:0') |
| frobenius_norm tensor(62.1430, device='cuda:0') |
| spectral_norm tensor(5.1164, device='cuda:0') |
| alpha value of layer 13 ---168.32928466796875 |
| Processing layer 14--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| frobenius_norm tensor(98.5708, device='cuda:0') |
| spectral_norm tensor(11.6312, device='cuda:0') |
| frobenius_norm tensor(100.4783, device='cuda:0') |
| spectral_norm tensor(13.6670, device='cuda:0') |
| frobenius_norm tensor(61.6071, device='cuda:0') |
| spectral_norm tensor(2.7584, device='cuda:0') |
| frobenius_norm tensor(60.9149, device='cuda:0') |
| spectral_norm tensor(4.5464, device='cuda:0') |
| alpha value of layer 14 ---201.04998779296875 |
| Processing layer 15--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| frobenius_norm tensor(97.3649, device='cuda:0') |
| spectral_norm tensor(12.7196, device='cuda:0') |
| frobenius_norm tensor(100.7325, device='cuda:0') |
| spectral_norm tensor(14.3247, device='cuda:0') |
| frobenius_norm tensor(64.0580, device='cuda:0') |
| spectral_norm tensor(3.0007, device='cuda:0') |
| frobenius_norm tensor(63.2220, device='cuda:0') |
| spectral_norm tensor(4.5882, device='cuda:0') |
| alpha value of layer 15 ---188.4088134765625 |
| Processing layer 16--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| frobenius_norm tensor(96.6512, device='cuda:0') |
| spectral_norm tensor(12.9655, device='cuda:0') |
| frobenius_norm tensor(99.2465, device='cuda:0') |
| spectral_norm tensor(14.6775, device='cuda:0') |
| frobenius_norm tensor(66.7600, device='cuda:0') |
| spectral_norm tensor(2.8352, device='cuda:0') |
| frobenius_norm tensor(66.0842, device='cuda:0') |
| spectral_norm tensor(5.0123, device='cuda:0') |
| alpha value of layer 16 ---207.397216796875 |
| Processing layer 17--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| frobenius_norm tensor(95.8736, device='cuda:0') |
| spectral_norm tensor(12.8995, device='cuda:0') |
| frobenius_norm tensor(98.0118, device='cuda:0') |
| spectral_norm tensor(14.3731, device='cuda:0') |
| frobenius_norm tensor(66.5281, device='cuda:0') |
| spectral_norm tensor(2.8901, device='cuda:0') |
| frobenius_norm tensor(66.1344, device='cuda:0') |
| spectral_norm tensor(5.4531, device='cuda:0') |
| alpha value of layer 17 ---194.68035888671875 |
| Processing layer 18--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| frobenius_norm tensor(93.7199, device='cuda:0') |
| spectral_norm tensor(12.9969, device='cuda:0') |
| frobenius_norm tensor(95.7889, device='cuda:0') |
| spectral_norm tensor(14.0707, device='cuda:0') |
| frobenius_norm tensor(69.6604, device='cuda:0') |
| spectral_norm tensor(2.8885, device='cuda:0') |
| frobenius_norm tensor(68.6924, device='cuda:0') |
| spectral_norm tensor(5.4377, device='cuda:0') |
| alpha value of layer 18 ---209.88125610351562 |
| Processing layer 19--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| frobenius_norm tensor(92.5769, device='cuda:0') |
| spectral_norm tensor(12.7937, device='cuda:0') |
| frobenius_norm tensor(94.3567, device='cuda:0') |
| spectral_norm tensor(14.2169, device='cuda:0') |
| frobenius_norm tensor(70.2688, device='cuda:0') |
| spectral_norm tensor(2.7430, device='cuda:0') |
| frobenius_norm tensor(69.5499, device='cuda:0') |
| spectral_norm tensor(5.3632, device='cuda:0') |
| alpha value of layer 19 ---230.21255493164062 |
| Processing layer 20--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| frobenius_norm tensor(93.1738, device='cuda:0') |
| spectral_norm tensor(13.3309, device='cuda:0') |
| frobenius_norm tensor(94.8772, device='cuda:0') |
| spectral_norm tensor(14.2162, device='cuda:0') |
| frobenius_norm tensor(71.3496, device='cuda:0') |
| spectral_norm tensor(2.7475, device='cuda:0') |
| frobenius_norm tensor(70.9048, device='cuda:0') |
| spectral_norm tensor(6.5838, device='cuda:0') |
| alpha value of layer 20 ---220.93441772460938 |
| Processing layer 21--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| frobenius_norm tensor(91.0892, device='cuda:0') |
| spectral_norm tensor(12.7824, device='cuda:0') |
| frobenius_norm tensor(92.0887, device='cuda:0') |
| spectral_norm tensor(13.3415, device='cuda:0') |
| frobenius_norm tensor(73.5470, device='cuda:0') |
| spectral_norm tensor(3.2134, device='cuda:0') |
| frobenius_norm tensor(72.4853, device='cuda:0') |
| spectral_norm tensor(5.6293, device='cuda:0') |
| alpha value of layer 21 ---197.01531982421875 |
| Processing layer 22--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| frobenius_norm tensor(93.0198, device='cuda:0') |
| spectral_norm tensor(12.9645, device='cuda:0') |
| frobenius_norm tensor(94.1975, device='cuda:0') |
| spectral_norm tensor(13.3901, device='cuda:0') |
| frobenius_norm tensor(73.8086, device='cuda:0') |
| spectral_norm tensor(2.7246, device='cuda:0') |
| frobenius_norm tensor(72.6579, device='cuda:0') |
| spectral_norm tensor(7.5949, device='cuda:0') |
| alpha value of layer 22 ---231.5908660888672 |
| Processing layer 23--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| frobenius_norm tensor(92.3679, device='cuda:0') |
| spectral_norm tensor(12.5056, device='cuda:0') |
| frobenius_norm tensor(93.0806, device='cuda:0') |
| spectral_norm tensor(12.8793, device='cuda:0') |
| frobenius_norm tensor(77.2716, device='cuda:0') |
| spectral_norm tensor(3.0203, device='cuda:0') |
| frobenius_norm tensor(76.3245, device='cuda:0') |
| spectral_norm tensor(5.5035, device='cuda:0') |
| alpha value of layer 23 ---238.41143798828125 |
| Processing layer 24--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| frobenius_norm tensor(89.8033, device='cuda:0') |
| spectral_norm tensor(12.3866, device='cuda:0') |
| frobenius_norm tensor(90.2768, device='cuda:0') |
| spectral_norm tensor(13.1727, device='cuda:0') |
| frobenius_norm tensor(76.5770, device='cuda:0') |
| spectral_norm tensor(3.2186, device='cuda:0') |
| frobenius_norm tensor(75.2567, device='cuda:0') |
| spectral_norm tensor(6.6725, device='cuda:0') |
| alpha value of layer 24 ---198.1973419189453 |
| Processing layer 25--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| frobenius_norm tensor(90.5618, device='cuda:0') |
| spectral_norm tensor(11.8575, device='cuda:0') |
| frobenius_norm tensor(90.8598, device='cuda:0') |
| spectral_norm tensor(12.2949, device='cuda:0') |
| frobenius_norm tensor(79.4490, device='cuda:0') |
| spectral_norm tensor(3.1827, device='cuda:0') |
| frobenius_norm tensor(78.3357, device='cuda:0') |
| spectral_norm tensor(4.8220, device='cuda:0') |
| alpha value of layer 25 ---249.99850463867188 |
| Processing layer 26--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| frobenius_norm tensor(89.1381, device='cuda:0') |
| spectral_norm tensor(12.8790, device='cuda:0') |
| frobenius_norm tensor(89.8395, device='cuda:0') |
| spectral_norm tensor(13.2880, device='cuda:0') |
| frobenius_norm tensor(80.7266, device='cuda:0') |
| spectral_norm tensor(3.6409, device='cuda:0') |
| frobenius_norm tensor(80.1935, device='cuda:0') |
| spectral_norm tensor(6.8510, device='cuda:0') |
| alpha value of layer 26 ---180.56016540527344 |
| Processing layer 27--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| frobenius_norm tensor(92.4526, device='cuda:0') |
| spectral_norm tensor(13.1710, device='cuda:0') |
| frobenius_norm tensor(93.3447, device='cuda:0') |
| spectral_norm tensor(14.0394, device='cuda:0') |
| frobenius_norm tensor(80.8654, device='cuda:0') |
| spectral_norm tensor(3.3240, device='cuda:0') |
| frobenius_norm tensor(80.7260, device='cuda:0') |
| spectral_norm tensor(5.6484, device='cuda:0') |
| alpha value of layer 27 ---222.39707946777344 |
| Processing layer 28--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| frobenius_norm tensor(89.8511, device='cuda:0') |
| spectral_norm tensor(12.7679, device='cuda:0') |
| frobenius_norm tensor(90.9173, device='cuda:0') |
| spectral_norm tensor(13.5210, device='cuda:0') |
| frobenius_norm tensor(83.4357, device='cuda:0') |
| spectral_norm tensor(3.6919, device='cuda:0') |
| frobenius_norm tensor(83.0720, device='cuda:0') |
| spectral_norm tensor(6.0824, device='cuda:0') |
| alpha value of layer 28 ---198.00421142578125 |
| Processing layer 29--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| frobenius_norm tensor(87.5255, device='cuda:0') |
| spectral_norm tensor(13.2835, device='cuda:0') |
| frobenius_norm tensor(88.2859, device='cuda:0') |
| spectral_norm tensor(14.1804, device='cuda:0') |
| frobenius_norm tensor(83.7624, device='cuda:0') |
| spectral_norm tensor(4.7727, device='cuda:0') |
| frobenius_norm tensor(84.0506, device='cuda:0') |
| spectral_norm tensor(6.8564, device='cuda:0') |
| alpha value of layer 29 ---135.1178436279297 |
| Processing layer 30--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| frobenius_norm tensor(88.0865, device='cuda:0') |
| spectral_norm tensor(13.0963, device='cuda:0') |
| frobenius_norm tensor(89.2752, device='cuda:0') |
| spectral_norm tensor(13.7996, device='cuda:0') |
| frobenius_norm tensor(85.7229, device='cuda:0') |
| spectral_norm tensor(3.7462, device='cuda:0') |
| frobenius_norm tensor(86.1523, device='cuda:0') |
| spectral_norm tensor(6.8602, device='cuda:0') |
| alpha value of layer 30 ---192.10064697265625 |
| Processing layer 31--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| frobenius_norm tensor(89.1405, device='cuda:0') |
| spectral_norm tensor(15.0259, device='cuda:0') |
| frobenius_norm tensor(92.3933, device='cuda:0') |
| spectral_norm tensor(16.3841, device='cuda:0') |
| frobenius_norm tensor(78.1290, device='cuda:0') |
| spectral_norm tensor(4.0098, device='cuda:0') |
| frobenius_norm tensor(78.9173, device='cuda:0') |
| spectral_norm tensor(10.5689, device='cuda:0') |
| alpha value of layer 31 ---125.60063171386719 |
| metric_name stable_rank: [25, 23, 22, 19, 27, 20, 18, 16, 14, 24, 28, 21, 17, 30, 15, 26, 13, 10, 12, 5, 9, 11, 4, 7, 6, 8, 29, 3, 31, 2, 1, 0] |
| Begin main_assign: Llama-2-7b-hf self_attn effective_rank |
|
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards: 50%|█████ | 1/2 [00:19<00:19, 19.23s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:26<00:00, 11.91s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:26<00:00, 13.01s/it] |
| Once upon a time, I worked in a library. I was a page. I would shelve books, check them out, and help patrons find things. It was my first job, and it was great. I met some really neat people. |
| At the time, I had a lot of free time, so I spent a lot of time in the library. I would sit in the fiction section and read all the books that I didn’t have time to read at home. This is how I first |
| LlamaForCausalLM( |
| (model): LlamaModel( |
| (embed_tokens): Embedding(32000, 4096, padding_idx=0) |
| (layers): ModuleList( |
| (0-31): 32 x LlamaDecoderLayer( |
| (self_attn): LlamaAttention( |
| (q_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (k_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (v_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (o_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| ) |
| (mlp): LlamaMLP( |
| (gate_proj): Linear(in_features=4096, out_features=11008, bias=False) |
| (up_proj): Linear(in_features=4096, out_features=11008, bias=False) |
| (down_proj): Linear(in_features=11008, out_features=4096, bias=False) |
| (act_fn): SiLU() |
| ) |
| (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05) |
| (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05) |
| ) |
| ) |
| (norm): LlamaRMSNorm((4096,), eps=1e-05) |
| (rotary_emb): LlamaRotaryEmbedding() |
| ) |
| (lm_head): Linear(in_features=4096, out_features=32000, bias=False) |
| ) |
| config: |
| LlamaConfig { |
| "architectures": [ |
| "LlamaForCausalLM" |
| ], |
| "attention_bias": false, |
| "attention_dropout": 0.0, |
| "bos_token_id": 1, |
| "eos_token_id": 2, |
| "head_dim": 128, |
| "hidden_act": "silu", |
| "hidden_size": 4096, |
| "initializer_range": 0.02, |
| "intermediate_size": 11008, |
| "max_position_embeddings": 4096, |
| "mlp_bias": false, |
| "model_type": "llama", |
| "num_attention_heads": 32, |
| "num_hidden_layers": 32, |
| "num_key_value_heads": 32, |
| "pad_token_id": 0, |
| "pretraining_tp": 1, |
| "rms_norm_eps": 1e-05, |
| "rope_scaling": null, |
| "rope_theta": 10000.0, |
| "tie_word_embeddings": false, |
| "torch_dtype": "float16", |
| "transformers_version": "4.55.2", |
| "use_cache": true, |
| "vocab_size": 32000 |
| } |
| |
| Processing layer 0--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 0 ---1640.3033447265625 |
| Processing layer 1--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 1 ---2134.1572265625 |
| Processing layer 2--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 2 ---2669.992919921875 |
| Processing layer 3--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 3 ---2901.25439453125 |
| Processing layer 4--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 4 ---2904.2001953125 |
| Processing layer 5--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 5 ---2910.534912109375 |
| Processing layer 6--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 6 ---2883.722900390625 |
| Processing layer 7--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 7 ---2887.236328125 |
| Processing layer 8--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 8 ---2899.039306640625 |
| Processing layer 9--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 9 ---2916.92822265625 |
| Processing layer 10--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 10 ---2859.56689453125 |
| Processing layer 11--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 11 ---2818.8173828125 |
| Processing layer 12--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 12 ---2905.6064453125 |
| Processing layer 13--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 13 ---2940.74462890625 |
| Processing layer 14--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 14 ---2900.401123046875 |
| Processing layer 15--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 15 ---2949.82080078125 |
| Processing layer 16--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 16 ---2976.977783203125 |
| Processing layer 17--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 17 ---3047.2646484375 |
| Processing layer 18--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 18 ---3096.2216796875 |
| Processing layer 19--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 19 ---3061.852783203125 |
| Processing layer 20--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 20 ---3062.37353515625 |
| Processing layer 21--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 21 ---3081.3349609375 |
| Processing layer 22--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 22 ---3106.181640625 |
| Processing layer 23--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 23 ---3144.513427734375 |
| Processing layer 24--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 24 ---3072.8798828125 |
| Processing layer 25--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 25 ---3137.80224609375 |
| Processing layer 26--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 26 ---3090.37158203125 |
| Processing layer 27--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 27 ---3181.7998046875 |
| Processing layer 28--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 28 ---3147.865478515625 |
| Processing layer 29--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 29 ---3101.146484375 |
| Processing layer 30--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 30 ---3161.5263671875 |
| Processing layer 31--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 31 ---3049.5556640625 |
| metric_name effective_rank: [27, 30, 28, 23, 25, 22, 29, 18, 26, 21, 24, 20, 19, 31, 17, 16, 15, 13, 9, 5, 12, 4, 3, 14, 8, 7, 6, 10, 11, 2, 1, 0] |
| Begin main_assign: Llama-2-7b-hf self_attn ZD |
|
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards: 50%|█████ | 1/2 [00:17<00:17, 17.69s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:23<00:00, 10.92s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:23<00:00, 11.94s/it] |
| Once upon a time, there was a girl who had a heart that was so full of love that it overflowed from her chest and flowed out of her hands. |
| This girl was so full of love that she was like a fountain of love, and wherever she went, her love would flow out of her and touch the lives of those around her. |
| One day, the girl was walking through the woods when she came upon a beautiful stream. The stream was so clear and so peaceful that |
| LlamaForCausalLM( |
| (model): LlamaModel( |
| (embed_tokens): Embedding(32000, 4096, padding_idx=0) |
| (layers): ModuleList( |
| (0-31): 32 x LlamaDecoderLayer( |
| (self_attn): LlamaAttention( |
| (q_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (k_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (v_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (o_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| ) |
| (mlp): LlamaMLP( |
| (gate_proj): Linear(in_features=4096, out_features=11008, bias=False) |
| (up_proj): Linear(in_features=4096, out_features=11008, bias=False) |
| (down_proj): Linear(in_features=11008, out_features=4096, bias=False) |
| (act_fn): SiLU() |
| ) |
| (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05) |
| (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05) |
| ) |
| ) |
| (norm): LlamaRMSNorm((4096,), eps=1e-05) |
| (rotary_emb): LlamaRotaryEmbedding() |
| ) |
| (lm_head): Linear(in_features=4096, out_features=32000, bias=False) |
| ) |
| config: |
| LlamaConfig { |
| "architectures": [ |
| "LlamaForCausalLM" |
| ], |
| "attention_bias": false, |
| "attention_dropout": 0.0, |
| "bos_token_id": 1, |
| "eos_token_id": 2, |
| "head_dim": 128, |
| "hidden_act": "silu", |
| "hidden_size": 4096, |
| "initializer_range": 0.02, |
| "intermediate_size": 11008, |
| "max_position_embeddings": 4096, |
| "mlp_bias": false, |
| "model_type": "llama", |
| "num_attention_heads": 32, |
| "num_hidden_layers": 32, |
| "num_key_value_heads": 32, |
| "pad_token_id": 0, |
| "pretraining_tp": 1, |
| "rms_norm_eps": 1e-05, |
| "rope_scaling": null, |
| "rope_theta": 10000.0, |
| "tie_word_embeddings": false, |
| "torch_dtype": "float16", |
| "transformers_version": "4.55.2", |
| "use_cache": true, |
| "vocab_size": 32000 |
| } |
| |
| Processing layer 0--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 0 ---0.09540334343910217 |
| Processing layer 1--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 1 ---0.11126542091369629 |
| Processing layer 2--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 2 ---0.14089055359363556 |
| Processing layer 3--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 3 ---0.1446058303117752 |
| Processing layer 4--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 4 ---0.14712807536125183 |
| Processing layer 5--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 5 ---0.1478433907032013 |
| Processing layer 6--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 6 ---0.14464625716209412 |
| Processing layer 7--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 7 ---0.14459004998207092 |
| Processing layer 8--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 8 ---0.14641690254211426 |
| Processing layer 9--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 9 ---0.147793248295784 |
| Processing layer 10--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 10 ---0.14709556102752686 |
| Processing layer 11--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 11 ---0.14403118193149567 |
| Processing layer 12--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 12 ---0.14700128138065338 |
| Processing layer 13--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 13 ---0.1479380875825882 |
| Processing layer 14--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 14 ---0.1479010283946991 |
| Processing layer 15--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 15 ---0.1490364670753479 |
| Processing layer 16--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 16 ---0.1480296403169632 |
| Processing layer 17--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 17 ---0.15020982921123505 |
| Processing layer 18--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 18 ---0.1507750302553177 |
| Processing layer 19--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 19 ---0.14981798827648163 |
| Processing layer 20--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 20 ---0.15018826723098755 |
| Processing layer 21--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 21 ---0.1498291790485382 |
| Processing layer 22--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 22 ---0.15043966472148895 |
| Processing layer 23--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 23 ---0.151978999376297 |
| Processing layer 24--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 24 ---0.14919137954711914 |
| Processing layer 25--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 25 ---0.15175150334835052 |
| Processing layer 26--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 26 ---0.1495654433965683 |
| Processing layer 27--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 27 ---0.15338149666786194 |
| Processing layer 28--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 28 ---0.15180033445358276 |
| Processing layer 29--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 29 ---0.1501537710428238 |
| Processing layer 30--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 30 ---0.15218497812747955 |
| Processing layer 31--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 31 ---0.14980709552764893 |
| metric_name ZD: [27, 30, 23, 28, 25, 18, 22, 17, 20, 29, 21, 19, 31, 26, 24, 15, 16, 13, 14, 5, 9, 4, 10, 12, 8, 6, 3, 7, 11, 2, 1, 0] |
| Begin main_assign: Llama-2-7b-hf self_attn head_diversity |
|
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards: 50%|█████ | 1/2 [00:18<00:18, 18.58s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:25<00:00, 11.55s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:25<00:00, 12.61s/it] |
| Once upon a time, there was a city that was built with love. I was in that city. And I fell in love. I fell in love with the people. I fell in love with the architecture. I fell in love with the food. I fell in love with the art. I fell in love with the culture. I fell in love with the music. I fell in love with the history. I fell in love with the city. And I fell in love with the person I fell in love with |
| LlamaForCausalLM( |
| (model): LlamaModel( |
| (embed_tokens): Embedding(32000, 4096, padding_idx=0) |
| (layers): ModuleList( |
| (0-31): 32 x LlamaDecoderLayer( |
| (self_attn): LlamaAttention( |
| (q_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (k_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (v_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (o_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| ) |
| (mlp): LlamaMLP( |
| (gate_proj): Linear(in_features=4096, out_features=11008, bias=False) |
| (up_proj): Linear(in_features=4096, out_features=11008, bias=False) |
| (down_proj): Linear(in_features=11008, out_features=4096, bias=False) |
| (act_fn): SiLU() |
| ) |
| (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05) |
| (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05) |
| ) |
| ) |
| (norm): LlamaRMSNorm((4096,), eps=1e-05) |
| (rotary_emb): LlamaRotaryEmbedding() |
| ) |
| (lm_head): Linear(in_features=4096, out_features=32000, bias=False) |
| ) |
| config: |
| LlamaConfig { |
| "architectures": [ |
| "LlamaForCausalLM" |
| ], |
| "attention_bias": false, |
| "attention_dropout": 0.0, |
| "bos_token_id": 1, |
| "eos_token_id": 2, |
| "head_dim": 128, |
| "hidden_act": "silu", |
| "hidden_size": 4096, |
| "initializer_range": 0.02, |
| "intermediate_size": 11008, |
| "max_position_embeddings": 4096, |
| "mlp_bias": false, |
| "model_type": "llama", |
| "num_attention_heads": 32, |
| "num_hidden_layers": 32, |
| "num_key_value_heads": 32, |
| "pad_token_id": 0, |
| "pretraining_tp": 1, |
| "rms_norm_eps": 1e-05, |
| "rope_scaling": null, |
| "rope_theta": 10000.0, |
| "tie_word_embeddings": false, |
| "torch_dtype": "float16", |
| "transformers_version": "4.55.2", |
| "use_cache": true, |
| "vocab_size": 32000 |
| } |
| |
| Processing layer 0--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 0 ---0.9916330575942993 |
| Processing layer 1--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 1 ---0.9952021241188049 |
| Processing layer 2--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 2 ---0.9966323971748352 |
| Processing layer 3--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 3 ---0.9973293542861938 |
| Processing layer 4--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 4 ---0.9971895217895508 |
| Processing layer 5--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 5 ---0.9973934888839722 |
| Processing layer 6--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 6 ---0.9974462389945984 |
| Processing layer 7--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 7 ---0.9975071549415588 |
| Processing layer 8--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 8 ---0.9974231719970703 |
| Processing layer 9--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 9 ---0.9973534345626831 |
| Processing layer 10--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 10 ---0.997123122215271 |
| Processing layer 11--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 11 ---0.9970043897628784 |
| Processing layer 12--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 12 ---0.9973783493041992 |
| Processing layer 13--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 13 ---0.9974591732025146 |
| Processing layer 14--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 14 ---0.9971306324005127 |
| Processing layer 15--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 15 ---0.9973533153533936 |
| Processing layer 16--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 16 ---0.9974291324615479 |
| Processing layer 17--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 17 ---0.9976841807365417 |
| Processing layer 18--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 18 ---0.997740626335144 |
| Processing layer 19--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 19 ---0.9975850582122803 |
| Processing layer 20--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 20 ---0.9973828792572021 |
| Processing layer 21--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 21 ---0.9975684881210327 |
| Processing layer 22--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 22 ---0.9977440237998962 |
| Processing layer 23--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 23 ---0.9980273246765137 |
| Processing layer 24--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 24 ---0.9974839091300964 |
| Processing layer 25--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 25 ---0.9979180693626404 |
| Processing layer 26--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 26 ---0.9974991083145142 |
| Processing layer 27--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 27 ---0.9979188442230225 |
| Processing layer 28--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 28 ---0.997989296913147 |
| Processing layer 29--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 29 ---0.9974175691604614 |
| Processing layer 30--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 30 ---0.9975640773773193 |
| Processing layer 31--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 31 ---0.997219443321228 |
| metric_name head_diversity: [23, 28, 27, 25, 22, 18, 17, 19, 21, 30, 7, 26, 24, 13, 6, 16, 8, 29, 5, 20, 12, 9, 15, 3, 31, 4, 14, 10, 11, 2, 1, 0] |
| Begin main_assign: Llama-2-7b-hf self_attn coherence |
|
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards: 50%|█████ | 1/2 [00:17<00:17, 17.66s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:23<00:00, 10.81s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:23<00:00, 11.84s/it] |
| Once upon a time, there lived a king and a queen. They had a beautiful daughter named Cinderella. One day, the king announced that there would be a ball. The king invited all the royalty and dignitaries to the ball, including Cinderella. |
| Cinderella was so excited. She could not wait to see her friends and dress in her best dress. She was happy that she would finally be able to go to the ball. The only problem was that her step-mother |
| LlamaForCausalLM( |
| (model): LlamaModel( |
| (embed_tokens): Embedding(32000, 4096, padding_idx=0) |
| (layers): ModuleList( |
| (0-31): 32 x LlamaDecoderLayer( |
| (self_attn): LlamaAttention( |
| (q_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (k_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (v_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (o_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| ) |
| (mlp): LlamaMLP( |
| (gate_proj): Linear(in_features=4096, out_features=11008, bias=False) |
| (up_proj): Linear(in_features=4096, out_features=11008, bias=False) |
| (down_proj): Linear(in_features=11008, out_features=4096, bias=False) |
| (act_fn): SiLU() |
| ) |
| (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05) |
| (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05) |
| ) |
| ) |
| (norm): LlamaRMSNorm((4096,), eps=1e-05) |
| (rotary_emb): LlamaRotaryEmbedding() |
| ) |
| (lm_head): Linear(in_features=4096, out_features=32000, bias=False) |
| ) |
| config: |
| LlamaConfig { |
| "architectures": [ |
| "LlamaForCausalLM" |
| ], |
| "attention_bias": false, |
| "attention_dropout": 0.0, |
| "bos_token_id": 1, |
| "eos_token_id": 2, |
| "head_dim": 128, |
| "hidden_act": "silu", |
| "hidden_size": 4096, |
| "initializer_range": 0.02, |
| "intermediate_size": 11008, |
| "max_position_embeddings": 4096, |
| "mlp_bias": false, |
| "model_type": "llama", |
| "num_attention_heads": 32, |
| "num_hidden_layers": 32, |
| "num_key_value_heads": 32, |
| "pad_token_id": 0, |
| "pretraining_tp": 1, |
| "rms_norm_eps": 1e-05, |
| "rope_scaling": null, |
| "rope_theta": 10000.0, |
| "tie_word_embeddings": false, |
| "torch_dtype": "float16", |
| "transformers_version": "4.55.2", |
| "use_cache": true, |
| "vocab_size": 32000 |
| } |
| |
| Processing layer 0--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 0 ---0.08510372042655945 |
| Processing layer 1--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 1 ---0.04102545976638794 |
| Processing layer 2--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 2 ---0.028616365045309067 |
| Processing layer 3--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 3 ---0.02104165218770504 |
| Processing layer 4--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 4 ---0.022063206881284714 |
| Processing layer 5--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 5 ---0.021188031882047653 |
| Processing layer 6--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 6 ---0.020417138934135437 |
| Processing layer 7--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 7 ---0.019520433619618416 |
| Processing layer 8--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 8 ---0.020254574716091156 |
| Processing layer 9--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 9 ---0.020007748156785965 |
| Processing layer 10--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 10 ---0.021119512617588043 |
| Processing layer 11--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 11 ---0.020985007286071777 |
| Processing layer 12--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 12 ---0.019723106175661087 |
| Processing layer 13--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 13 ---0.01894117146730423 |
| Processing layer 14--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 14 ---0.01963678002357483 |
| Processing layer 15--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 15 ---0.01925666816532612 |
| Processing layer 16--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 16 ---0.018222851678729057 |
| Processing layer 17--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 17 ---0.016996942460536957 |
| Processing layer 18--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 18 ---0.016209837049245834 |
| Processing layer 19--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 19 ---0.017241276800632477 |
| Processing layer 20--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 20 ---0.017154088243842125 |
| Processing layer 21--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 21 ---0.016598742455244064 |
| Processing layer 22--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 22 ---0.016119930893182755 |
| Processing layer 23--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 23 ---0.015261407010257244 |
| Processing layer 24--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 24 ---0.01685335859656334 |
| Processing layer 25--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 25 ---0.015361565165221691 |
| Processing layer 26--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 26 ---0.01685093343257904 |
| Processing layer 27--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 27 ---0.015206292271614075 |
| Processing layer 28--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 28 ---0.01575298234820366 |
| Processing layer 29--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 29 ---0.01735319383442402 |
| Processing layer 30--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 30 ---0.016395289450883865 |
| Processing layer 31--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False)} |
| alpha value of layer 31 ---0.02029731497168541 |
| metric_name coherence: [0, 1, 2, 4, 5, 10, 3, 11, 6, 31, 8, 9, 12, 14, 7, 15, 13, 16, 29, 19, 20, 17, 24, 26, 21, 30, 18, 22, 28, 25, 23, 27] |
| Begin main_assign: Llama-2-7b-hf mlp alpha |
|
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards: 50%|█████ | 1/2 [00:18<00:18, 18.84s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:25<00:00, 11.57s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:25<00:00, 12.66s/it] |
| Once upon a time, there was a little boy named Jack who lived in the country. Jack loved to play in the forest with his friends. He was a happy boy. |
| One day, he was playing with his friends when he saw a beautiful white horse. The horse was beautiful, and he wanted to ride it. |
| Jack ran up to the horse and asked, “Can I ride you?” |
| The horse said, “Yes, Jack, you can ride me.” |
| Jack jumped up on the horse |
| LlamaForCausalLM( |
| (model): LlamaModel( |
| (embed_tokens): Embedding(32000, 4096, padding_idx=0) |
| (layers): ModuleList( |
| (0-31): 32 x LlamaDecoderLayer( |
| (self_attn): LlamaAttention( |
| (q_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (k_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (v_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (o_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| ) |
| (mlp): LlamaMLP( |
| (gate_proj): Linear(in_features=4096, out_features=11008, bias=False) |
| (up_proj): Linear(in_features=4096, out_features=11008, bias=False) |
| (down_proj): Linear(in_features=11008, out_features=4096, bias=False) |
| (act_fn): SiLU() |
| ) |
| (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05) |
| (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05) |
| ) |
| ) |
| (norm): LlamaRMSNorm((4096,), eps=1e-05) |
| (rotary_emb): LlamaRotaryEmbedding() |
| ) |
| (lm_head): Linear(in_features=4096, out_features=32000, bias=False) |
| ) |
| config: |
| LlamaConfig { |
| "architectures": [ |
| "LlamaForCausalLM" |
| ], |
| "attention_bias": false, |
| "attention_dropout": 0.0, |
| "bos_token_id": 1, |
| "eos_token_id": 2, |
| "head_dim": 128, |
| "hidden_act": "silu", |
| "hidden_size": 4096, |
| "initializer_range": 0.02, |
| "intermediate_size": 11008, |
| "max_position_embeddings": 4096, |
| "mlp_bias": false, |
| "model_type": "llama", |
| "num_attention_heads": 32, |
| "num_hidden_layers": 32, |
| "num_key_value_heads": 32, |
| "pad_token_id": 0, |
| "pretraining_tp": 1, |
| "rms_norm_eps": 1e-05, |
| "rope_scaling": null, |
| "rope_theta": 10000.0, |
| "tie_word_embeddings": false, |
| "torch_dtype": "float16", |
| "transformers_version": "4.55.2", |
| "use_cache": true, |
| "vocab_size": 32000 |
| } |
| |
| Processing layer 0--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 0 ---2.815411329269409 |
| Processing layer 1--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 1 ---3.4513909816741943 |
| Processing layer 2--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 2 ---3.7109851837158203 |
| Processing layer 3--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 3 ---3.8032162189483643 |
| Processing layer 4--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 4 ---4.195495128631592 |
| Processing layer 5--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 5 ---3.856921911239624 |
| Processing layer 6--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 6 ---3.6256275177001953 |
| Processing layer 7--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 7 ---3.660289764404297 |
| Processing layer 8--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 8 ---3.5230093002319336 |
| Processing layer 9--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 9 ---3.5076420307159424 |
| Processing layer 10--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 10 ---3.3467018604278564 |
| Processing layer 11--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 11 ---3.291457176208496 |
| Processing layer 12--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 12 ---3.5538055896759033 |
| Processing layer 13--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 13 ---3.4736461639404297 |
| Processing layer 14--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 14 ---3.715531587600708 |
| Processing layer 15--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 15 ---3.7430496215820312 |
| Processing layer 16--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 16 ---4.149142742156982 |
| Processing layer 17--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 17 ---4.197119235992432 |
| Processing layer 18--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 18 ---4.489593982696533 |
| Processing layer 19--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 19 ---4.358992576599121 |
| Processing layer 20--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 20 ---5.0549798011779785 |
| Processing layer 21--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 21 ---4.814038276672363 |
| Processing layer 22--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 22 ---4.621534824371338 |
| Processing layer 23--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 23 ---4.299625396728516 |
| Processing layer 24--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 24 ---4.563563346862793 |
| Processing layer 25--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 25 ---4.570354461669922 |
| Processing layer 26--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 26 ---4.2273688316345215 |
| Processing layer 27--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 27 ---4.423553466796875 |
| Processing layer 28--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 28 ---4.38798189163208 |
| Processing layer 29--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 29 ---4.595789432525635 |
| Processing layer 30--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 30 ---4.671386241912842 |
| Processing layer 31--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 31 ---3.7070789337158203 |
| metric_name alpha: [20, 21, 30, 22, 29, 25, 24, 18, 27, 28, 19, 23, 26, 17, 4, 16, 5, 3, 15, 14, 2, 31, 7, 6, 12, 8, 9, 13, 1, 10, 11, 0] |
| Begin main_assign: Llama-2-7b-hf mlp alpha_hat |
|
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards: 50%|█████ | 1/2 [00:18<00:18, 18.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:24<00:00, 11.18s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:24<00:00, 12.20s/it] |
| Once upon a time, there was a kingdom called Buzuran. A kingdom known for its beautiful scenery and its friendly people. |
| One day, a stranger named Gavin arrived in the kingdom. He was looking for a place to settle down. He had heard of the kingdom and its beauty, and he was interested in seeing it for himself. |
| When Gavin arrived, he was greeted by the king and queen. They were very impressed with Gavin and his ab |
| LlamaForCausalLM( |
| (model): LlamaModel( |
| (embed_tokens): Embedding(32000, 4096, padding_idx=0) |
| (layers): ModuleList( |
| (0-31): 32 x LlamaDecoderLayer( |
| (self_attn): LlamaAttention( |
| (q_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (k_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (v_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (o_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| ) |
| (mlp): LlamaMLP( |
| (gate_proj): Linear(in_features=4096, out_features=11008, bias=False) |
| (up_proj): Linear(in_features=4096, out_features=11008, bias=False) |
| (down_proj): Linear(in_features=11008, out_features=4096, bias=False) |
| (act_fn): SiLU() |
| ) |
| (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05) |
| (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05) |
| ) |
| ) |
| (norm): LlamaRMSNorm((4096,), eps=1e-05) |
| (rotary_emb): LlamaRotaryEmbedding() |
| ) |
| (lm_head): Linear(in_features=4096, out_features=32000, bias=False) |
| ) |
| config: |
| LlamaConfig { |
| "architectures": [ |
| "LlamaForCausalLM" |
| ], |
| "attention_bias": false, |
| "attention_dropout": 0.0, |
| "bos_token_id": 1, |
| "eos_token_id": 2, |
| "head_dim": 128, |
| "hidden_act": "silu", |
| "hidden_size": 4096, |
| "initializer_range": 0.02, |
| "intermediate_size": 11008, |
| "max_position_embeddings": 4096, |
| "mlp_bias": false, |
| "model_type": "llama", |
| "num_attention_heads": 32, |
| "num_hidden_layers": 32, |
| "num_key_value_heads": 32, |
| "pad_token_id": 0, |
| "pretraining_tp": 1, |
| "rms_norm_eps": 1e-05, |
| "rope_scaling": null, |
| "rope_theta": 10000.0, |
| "tie_word_embeddings": false, |
| "torch_dtype": "float16", |
| "transformers_version": "4.55.2", |
| "use_cache": true, |
| "vocab_size": 32000 |
| } |
| |
| Processing layer 0--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 0 ---13.072108268737793 |
| Processing layer 1--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 1 ---14.548068046569824 |
| Processing layer 2--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 2 ---13.608903884887695 |
| Processing layer 3--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 3 ---13.986700057983398 |
| Processing layer 4--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 4 ---15.312846183776855 |
| Processing layer 5--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 5 ---14.763805389404297 |
| Processing layer 6--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 6 ---14.188860893249512 |
| Processing layer 7--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 7 ---14.57461166381836 |
| Processing layer 8--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 8 ---14.344627380371094 |
| Processing layer 9--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 9 ---14.180407524108887 |
| Processing layer 10--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 10 ---13.752973556518555 |
| Processing layer 11--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 11 ---13.753717422485352 |
| Processing layer 12--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 12 ---14.589736938476562 |
| Processing layer 13--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 13 ---14.359493255615234 |
| Processing layer 14--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 14 ---15.248335838317871 |
| Processing layer 15--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 15 ---15.255033493041992 |
| Processing layer 16--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 16 ---16.761940002441406 |
| Processing layer 17--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 17 ---16.71654510498047 |
| Processing layer 18--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 18 ---17.673377990722656 |
| Processing layer 19--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 19 ---17.007017135620117 |
| Processing layer 20--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 20 ---19.24078941345215 |
| Processing layer 21--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 21 ---17.77168083190918 |
| Processing layer 22--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 22 ---17.303386688232422 |
| Processing layer 23--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 23 ---16.242385864257812 |
| Processing layer 24--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 24 ---16.798599243164062 |
| Processing layer 25--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 25 ---16.718074798583984 |
| Processing layer 26--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 26 ---16.25970458984375 |
| Processing layer 27--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 27 ---17.800804138183594 |
| Processing layer 28--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 28 ---18.30744171142578 |
| Processing layer 29--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 29 ---19.38349151611328 |
| Processing layer 30--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 30 ---22.61872673034668 |
| Processing layer 31--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 31 ---19.854324340820312 |
| metric_name alpha_hat: [30, 31, 29, 20, 28, 27, 21, 18, 22, 19, 24, 16, 25, 17, 26, 23, 4, 15, 14, 5, 12, 7, 1, 13, 8, 6, 9, 3, 11, 10, 2, 0] |
| Begin main_assign: Llama-2-7b-hf mlp stable_rank |
|
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards: 50%|█████ | 1/2 [00:17<00:17, 17.70s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:24<00:00, 11.03s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:24<00:00, 12.03s/it] |
| Once upon a time there was a little boy who wanted to be a fireman when he grew up. He got to go on a fire truck and play with the hoses and watch the firemen and had a great time. |
| He also wanted to be a policeman when he grew up. He got to go on a police car and get to go to jail. He got to play with the police dogs and watch the policemen and had a great time. |
| He also wanted to be a doctor |
| LlamaForCausalLM( |
| (model): LlamaModel( |
| (embed_tokens): Embedding(32000, 4096, padding_idx=0) |
| (layers): ModuleList( |
| (0-31): 32 x LlamaDecoderLayer( |
| (self_attn): LlamaAttention( |
| (q_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (k_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (v_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (o_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| ) |
| (mlp): LlamaMLP( |
| (gate_proj): Linear(in_features=4096, out_features=11008, bias=False) |
| (up_proj): Linear(in_features=4096, out_features=11008, bias=False) |
| (down_proj): Linear(in_features=11008, out_features=4096, bias=False) |
| (act_fn): SiLU() |
| ) |
| (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05) |
| (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05) |
| ) |
| ) |
| (norm): LlamaRMSNorm((4096,), eps=1e-05) |
| (rotary_emb): LlamaRotaryEmbedding() |
| ) |
| (lm_head): Linear(in_features=4096, out_features=32000, bias=False) |
| ) |
| config: |
| LlamaConfig { |
| "architectures": [ |
| "LlamaForCausalLM" |
| ], |
| "attention_bias": false, |
| "attention_dropout": 0.0, |
| "bos_token_id": 1, |
| "eos_token_id": 2, |
| "head_dim": 128, |
| "hidden_act": "silu", |
| "hidden_size": 4096, |
| "initializer_range": 0.02, |
| "intermediate_size": 11008, |
| "max_position_embeddings": 4096, |
| "mlp_bias": false, |
| "model_type": "llama", |
| "num_attention_heads": 32, |
| "num_hidden_layers": 32, |
| "num_key_value_heads": 32, |
| "pad_token_id": 0, |
| "pretraining_tp": 1, |
| "rms_norm_eps": 1e-05, |
| "rope_scaling": null, |
| "rope_theta": 10000.0, |
| "tie_word_embeddings": false, |
| "torch_dtype": "float16", |
| "transformers_version": "4.55.2", |
| "use_cache": true, |
| "vocab_size": 32000 |
| } |
| |
| Processing layer 0--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| frobenius_norm tensor(110.1283, device='cuda:0') |
| spectral_norm tensor(11.7725, device='cuda:0') |
| frobenius_norm tensor(108.2747, device='cuda:0') |
| spectral_norm tensor(10.5846, device='cuda:0') |
| frobenius_norm tensor(112.8760, device='cuda:0') |
| spectral_norm tensor(8.8966, device='cuda:0') |
| alpha value of layer 0 ---117.70869445800781 |
| Processing layer 1--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| frobenius_norm tensor(122.3248, device='cuda:0') |
| spectral_norm tensor(14.1620, device='cuda:0') |
| frobenius_norm tensor(115.1808, device='cuda:0') |
| spectral_norm tensor(7.3779, device='cuda:0') |
| frobenius_norm tensor(116.8547, device='cuda:0') |
| spectral_norm tensor(6.6577, device='cuda:0') |
| alpha value of layer 1 ---208.7972869873047 |
| Processing layer 2--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| frobenius_norm tensor(125.7614, device='cuda:0') |
| spectral_norm tensor(10.6197, device='cuda:0') |
| frobenius_norm tensor(117.2508, device='cuda:0') |
| spectral_norm tensor(4.5438, device='cuda:0') |
| frobenius_norm tensor(117.9841, device='cuda:0') |
| spectral_norm tensor(5.6738, device='cuda:0') |
| alpha value of layer 2 ---412.8428649902344 |
| Processing layer 3--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| frobenius_norm tensor(127.6067, device='cuda:0') |
| spectral_norm tensor(9.8554, device='cuda:0') |
| frobenius_norm tensor(118.0961, device='cuda:0') |
| spectral_norm tensor(4.3441, device='cuda:0') |
| frobenius_norm tensor(118.3421, device='cuda:0') |
| spectral_norm tensor(6.3601, device='cuda:0') |
| alpha value of layer 3 ---417.640380859375 |
| Processing layer 4--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| frobenius_norm tensor(130.2638, device='cuda:0') |
| spectral_norm tensor(10.9230, device='cuda:0') |
| frobenius_norm tensor(117.2583, device='cuda:0') |
| spectral_norm tensor(4.2948, device='cuda:0') |
| frobenius_norm tensor(116.9673, device='cuda:0') |
| spectral_norm tensor(6.2874, device='cuda:0') |
| alpha value of layer 4 ---411.2488708496094 |
| Processing layer 5--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| frobenius_norm tensor(131.0171, device='cuda:0') |
| spectral_norm tensor(12.3422, device='cuda:0') |
| frobenius_norm tensor(117.2556, device='cuda:0') |
| spectral_norm tensor(4.4561, device='cuda:0') |
| frobenius_norm tensor(117.0701, device='cuda:0') |
| spectral_norm tensor(6.4393, device='cuda:0') |
| alpha value of layer 5 ---378.5376892089844 |
| Processing layer 6--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| frobenius_norm tensor(133.2534, device='cuda:0') |
| spectral_norm tensor(13.9346, device='cuda:0') |
| frobenius_norm tensor(116.8471, device='cuda:0') |
| spectral_norm tensor(4.7412, device='cuda:0') |
| frobenius_norm tensor(116.4051, device='cuda:0') |
| spectral_norm tensor(6.2935, device='cuda:0') |
| alpha value of layer 6 ---346.97930908203125 |
| Processing layer 7--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| frobenius_norm tensor(133.1796, device='cuda:0') |
| spectral_norm tensor(13.7900, device='cuda:0') |
| frobenius_norm tensor(117.2589, device='cuda:0') |
| spectral_norm tensor(4.9844, device='cuda:0') |
| frobenius_norm tensor(116.5975, device='cuda:0') |
| spectral_norm tensor(6.7392, device='cuda:0') |
| alpha value of layer 7 ---315.34832763671875 |
| Processing layer 8--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| frobenius_norm tensor(131.0625, device='cuda:0') |
| spectral_norm tensor(12.9479, device='cuda:0') |
| frobenius_norm tensor(118.7164, device='cuda:0') |
| spectral_norm tensor(5.3742, device='cuda:0') |
| frobenius_norm tensor(117.8239, device='cuda:0') |
| spectral_norm tensor(7.0607, device='cuda:0') |
| alpha value of layer 8 ---289.63446044921875 |
| Processing layer 9--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| frobenius_norm tensor(129.7933, device='cuda:0') |
| spectral_norm tensor(13.0345, device='cuda:0') |
| frobenius_norm tensor(119.5203, device='cuda:0') |
| spectral_norm tensor(5.3777, device='cuda:0') |
| frobenius_norm tensor(118.6356, device='cuda:0') |
| spectral_norm tensor(7.0941, device='cuda:0') |
| alpha value of layer 9 ---290.9284362792969 |
| Processing layer 10--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| frobenius_norm tensor(129.3888, device='cuda:0') |
| spectral_norm tensor(13.1949, device='cuda:0') |
| frobenius_norm tensor(120.7769, device='cuda:0') |
| spectral_norm tensor(5.5907, device='cuda:0') |
| frobenius_norm tensor(119.6504, device='cuda:0') |
| spectral_norm tensor(7.0719, device='cuda:0') |
| alpha value of layer 10 ---283.04052734375 |
| Processing layer 11--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| frobenius_norm tensor(128.9186, device='cuda:0') |
| spectral_norm tensor(13.5576, device='cuda:0') |
| frobenius_norm tensor(121.8036, device='cuda:0') |
| spectral_norm tensor(5.6766, device='cuda:0') |
| frobenius_norm tensor(120.4323, device='cuda:0') |
| spectral_norm tensor(7.4645, device='cuda:0') |
| alpha value of layer 11 ---270.3800048828125 |
| Processing layer 12--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| frobenius_norm tensor(128.1013, device='cuda:0') |
| spectral_norm tensor(13.0662, device='cuda:0') |
| frobenius_norm tensor(122.9129, device='cuda:0') |
| spectral_norm tensor(6.0042, device='cuda:0') |
| frobenius_norm tensor(121.4419, device='cuda:0') |
| spectral_norm tensor(6.8182, device='cuda:0') |
| alpha value of layer 12 ---277.48046875 |
| Processing layer 13--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| frobenius_norm tensor(127.6379, device='cuda:0') |
| spectral_norm tensor(13.2896, device='cuda:0') |
| frobenius_norm tensor(124.1647, device='cuda:0') |
| spectral_norm tensor(6.3249, device='cuda:0') |
| frobenius_norm tensor(122.4098, device='cuda:0') |
| spectral_norm tensor(6.4258, device='cuda:0') |
| alpha value of layer 13 ---280.17169189453125 |
| Processing layer 14--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| frobenius_norm tensor(127.3744, device='cuda:0') |
| spectral_norm tensor(13.1519, device='cuda:0') |
| frobenius_norm tensor(124.2991, device='cuda:0') |
| spectral_norm tensor(6.4910, device='cuda:0') |
| frobenius_norm tensor(122.5321, device='cuda:0') |
| spectral_norm tensor(6.1543, device='cuda:0') |
| alpha value of layer 14 ---285.63787841796875 |
| Processing layer 15--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| frobenius_norm tensor(128.0534, device='cuda:0') |
| spectral_norm tensor(13.0927, device='cuda:0') |
| frobenius_norm tensor(124.9728, device='cuda:0') |
| spectral_norm tensor(6.8014, device='cuda:0') |
| frobenius_norm tensor(122.9694, device='cuda:0') |
| spectral_norm tensor(5.7687, device='cuda:0') |
| alpha value of layer 15 ---295.8956298828125 |
| Processing layer 16--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| frobenius_norm tensor(128.9326, device='cuda:0') |
| spectral_norm tensor(13.4859, device='cuda:0') |
| frobenius_norm tensor(124.8730, device='cuda:0') |
| spectral_norm tensor(7.0698, device='cuda:0') |
| frobenius_norm tensor(122.8834, device='cuda:0') |
| spectral_norm tensor(5.4697, device='cuda:0') |
| alpha value of layer 16 ---302.703857421875 |
| Processing layer 17--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| frobenius_norm tensor(130.1418, device='cuda:0') |
| spectral_norm tensor(13.9212, device='cuda:0') |
| frobenius_norm tensor(124.4075, device='cuda:0') |
| spectral_norm tensor(6.5251, device='cuda:0') |
| frobenius_norm tensor(122.7542, device='cuda:0') |
| spectral_norm tensor(5.4324, device='cuda:0') |
| alpha value of layer 17 ---320.5066833496094 |
| Processing layer 18--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| frobenius_norm tensor(131.3037, device='cuda:0') |
| spectral_norm tensor(13.5630, device='cuda:0') |
| frobenius_norm tensor(124.0418, device='cuda:0') |
| spectral_norm tensor(6.0638, device='cuda:0') |
| frobenius_norm tensor(122.6318, device='cuda:0') |
| spectral_norm tensor(5.3899, device='cuda:0') |
| alpha value of layer 18 ---343.27899169921875 |
| Processing layer 19--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| frobenius_norm tensor(131.7776, device='cuda:0') |
| spectral_norm tensor(12.6322, device='cuda:0') |
| frobenius_norm tensor(124.0385, device='cuda:0') |
| spectral_norm tensor(5.9170, device='cuda:0') |
| frobenius_norm tensor(122.9026, device='cuda:0') |
| spectral_norm tensor(5.5908, device='cuda:0') |
| alpha value of layer 19 ---343.8408203125 |
| Processing layer 20--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| frobenius_norm tensor(132.4897, device='cuda:0') |
| spectral_norm tensor(12.7395, device='cuda:0') |
| frobenius_norm tensor(123.9540, device='cuda:0') |
| spectral_norm tensor(6.1100, device='cuda:0') |
| frobenius_norm tensor(122.8897, device='cuda:0') |
| spectral_norm tensor(5.1633, device='cuda:0') |
| alpha value of layer 20 ---362.06610107421875 |
| Processing layer 21--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| frobenius_norm tensor(133.3879, device='cuda:0') |
| spectral_norm tensor(12.1366, device='cuda:0') |
| frobenius_norm tensor(123.7530, device='cuda:0') |
| spectral_norm tensor(5.6965, device='cuda:0') |
| frobenius_norm tensor(122.8804, device='cuda:0') |
| spectral_norm tensor(4.8811, device='cuda:0') |
| alpha value of layer 21 ---408.834228515625 |
| Processing layer 22--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| frobenius_norm tensor(134.1589, device='cuda:0') |
| spectral_norm tensor(11.9229, device='cuda:0') |
| frobenius_norm tensor(123.6884, device='cuda:0') |
| spectral_norm tensor(5.3241, device='cuda:0') |
| frobenius_norm tensor(122.9257, device='cuda:0') |
| spectral_norm tensor(5.2778, device='cuda:0') |
| alpha value of layer 22 ---402.9353942871094 |
| Processing layer 23--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| frobenius_norm tensor(134.1199, device='cuda:0') |
| spectral_norm tensor(10.9135, device='cuda:0') |
| frobenius_norm tensor(124.2959, device='cuda:0') |
| spectral_norm tensor(4.9033, device='cuda:0') |
| frobenius_norm tensor(123.6594, device='cuda:0') |
| spectral_norm tensor(5.8655, device='cuda:0') |
| alpha value of layer 23 ---412.6978454589844 |
| Processing layer 24--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| frobenius_norm tensor(134.4485, device='cuda:0') |
| spectral_norm tensor(10.8745, device='cuda:0') |
| frobenius_norm tensor(124.6739, device='cuda:0') |
| spectral_norm tensor(4.6898, device='cuda:0') |
| frobenius_norm tensor(124.1024, device='cuda:0') |
| spectral_norm tensor(5.6106, device='cuda:0') |
| alpha value of layer 24 ---449.614501953125 |
| Processing layer 25--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| frobenius_norm tensor(134.7100, device='cuda:0') |
| spectral_norm tensor(10.7878, device='cuda:0') |
| frobenius_norm tensor(125.2170, device='cuda:0') |
| spectral_norm tensor(4.9540, device='cuda:0') |
| frobenius_norm tensor(124.7208, device='cuda:0') |
| spectral_norm tensor(5.3156, device='cuda:0') |
| alpha value of layer 25 ---448.44183349609375 |
| Processing layer 26--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| frobenius_norm tensor(135.1252, device='cuda:0') |
| spectral_norm tensor(11.1821, device='cuda:0') |
| frobenius_norm tensor(125.6647, device='cuda:0') |
| spectral_norm tensor(5.8583, device='cuda:0') |
| frobenius_norm tensor(125.1040, device='cuda:0') |
| spectral_norm tensor(5.0074, device='cuda:0') |
| alpha value of layer 26 ---410.114013671875 |
| Processing layer 27--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| frobenius_norm tensor(135.2713, device='cuda:0') |
| spectral_norm tensor(11.5424, device='cuda:0') |
| frobenius_norm tensor(126.2993, device='cuda:0') |
| spectral_norm tensor(6.9998, device='cuda:0') |
| frobenius_norm tensor(125.7692, device='cuda:0') |
| spectral_norm tensor(5.1612, device='cuda:0') |
| alpha value of layer 27 ---352.2416687011719 |
| Processing layer 28--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| frobenius_norm tensor(134.6884, device='cuda:0') |
| spectral_norm tensor(11.4798, device='cuda:0') |
| frobenius_norm tensor(127.5331, device='cuda:0') |
| spectral_norm tensor(9.0437, device='cuda:0') |
| frobenius_norm tensor(126.5359, device='cuda:0') |
| spectral_norm tensor(4.9416, device='cuda:0') |
| alpha value of layer 28 ---330.72796630859375 |
| Processing layer 29--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| frobenius_norm tensor(135.0423, device='cuda:0') |
| spectral_norm tensor(12.0559, device='cuda:0') |
| frobenius_norm tensor(128.7051, device='cuda:0') |
| spectral_norm tensor(11.3077, device='cuda:0') |
| frobenius_norm tensor(126.9749, device='cuda:0') |
| spectral_norm tensor(4.4258, device='cuda:0') |
| alpha value of layer 29 ---359.37579345703125 |
| Processing layer 30--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| frobenius_norm tensor(138.1487, device='cuda:0') |
| spectral_norm tensor(19.7267, device='cuda:0') |
| frobenius_norm tensor(130.8807, device='cuda:0') |
| spectral_norm tensor(19.9367, device='cuda:0') |
| frobenius_norm tensor(126.2216, device='cuda:0') |
| spectral_norm tensor(4.5236, device='cuda:0') |
| alpha value of layer 30 ---290.23907470703125 |
| Processing layer 31--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| frobenius_norm tensor(144.1483, device='cuda:0') |
| spectral_norm tensor(19.9982, device='cuda:0') |
| frobenius_norm tensor(135.9538, device='cuda:0') |
| spectral_norm tensor(19.9404, device='cuda:0') |
| frobenius_norm tensor(126.0626, device='cuda:0') |
| spectral_norm tensor(7.8316, device='cuda:0') |
| alpha value of layer 31 ---119.1807861328125 |
| metric_name stable_rank: [24, 25, 3, 2, 23, 4, 26, 21, 22, 5, 20, 29, 27, 6, 19, 18, 28, 17, 7, 16, 15, 9, 30, 8, 14, 10, 13, 12, 11, 1, 31, 0] |
| Begin main_assign: Llama-2-7b-hf mlp effective_rank |
|
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards: 50%|█████ | 1/2 [00:18<00:18, 18.39s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:24<00:00, 11.39s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:24<00:00, 12.44s/it] |
| Once upon a time, my father-in-law went to the bank to change a cheque. He was asked to write his signature on the cheque and to then place a tick in the box indicating whether the cheque was a deposit or a withdrawal. |
| My father-in-law was confused. He was not accustomed to the new style of cheque, which was introduced in the 1980s. |
| “I know it says ‘deposit’, but |
| LlamaForCausalLM( |
| (model): LlamaModel( |
| (embed_tokens): Embedding(32000, 4096, padding_idx=0) |
| (layers): ModuleList( |
| (0-31): 32 x LlamaDecoderLayer( |
| (self_attn): LlamaAttention( |
| (q_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (k_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (v_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (o_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| ) |
| (mlp): LlamaMLP( |
| (gate_proj): Linear(in_features=4096, out_features=11008, bias=False) |
| (up_proj): Linear(in_features=4096, out_features=11008, bias=False) |
| (down_proj): Linear(in_features=11008, out_features=4096, bias=False) |
| (act_fn): SiLU() |
| ) |
| (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05) |
| (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05) |
| ) |
| ) |
| (norm): LlamaRMSNorm((4096,), eps=1e-05) |
| (rotary_emb): LlamaRotaryEmbedding() |
| ) |
| (lm_head): Linear(in_features=4096, out_features=32000, bias=False) |
| ) |
| config: |
| LlamaConfig { |
| "architectures": [ |
| "LlamaForCausalLM" |
| ], |
| "attention_bias": false, |
| "attention_dropout": 0.0, |
| "bos_token_id": 1, |
| "eos_token_id": 2, |
| "head_dim": 128, |
| "hidden_act": "silu", |
| "hidden_size": 4096, |
| "initializer_range": 0.02, |
| "intermediate_size": 11008, |
| "max_position_embeddings": 4096, |
| "mlp_bias": false, |
| "model_type": "llama", |
| "num_attention_heads": 32, |
| "num_hidden_layers": 32, |
| "num_key_value_heads": 32, |
| "pad_token_id": 0, |
| "pretraining_tp": 1, |
| "rms_norm_eps": 1e-05, |
| "rope_scaling": null, |
| "rope_theta": 10000.0, |
| "tie_word_embeddings": false, |
| "torch_dtype": "float16", |
| "transformers_version": "4.55.2", |
| "use_cache": true, |
| "vocab_size": 32000 |
| } |
| |
| Processing layer 0--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 0 ---3572.474365234375 |
| Processing layer 1--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 1 ---3627.507568359375 |
| Processing layer 2--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 2 ---3733.4091796875 |
| Processing layer 3--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 3 ---3785.27880859375 |
| Processing layer 4--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 4 ---3790.103271484375 |
| Processing layer 5--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 5 ---3778.20458984375 |
| Processing layer 6--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 6 ---3768.807373046875 |
| Processing layer 7--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 7 ---3756.693359375 |
| Processing layer 8--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 8 ---3748.4443359375 |
| Processing layer 9--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 9 ---3745.5185546875 |
| Processing layer 10--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 10 ---3735.50341796875 |
| Processing layer 11--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 11 ---3737.810302734375 |
| Processing layer 12--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 12 ---3740.48388671875 |
| Processing layer 13--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 13 ---3753.927734375 |
| Processing layer 14--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 14 ---3753.2763671875 |
| Processing layer 15--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 15 ---3771.952880859375 |
| Processing layer 16--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 16 ---3771.9111328125 |
| Processing layer 17--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 17 ---3772.70849609375 |
| Processing layer 18--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 18 ---3783.5595703125 |
| Processing layer 19--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 19 ---3788.821044921875 |
| Processing layer 20--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 20 ---3790.424560546875 |
| Processing layer 21--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 21 ---3790.768310546875 |
| Processing layer 22--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 22 ---3790.56396484375 |
| Processing layer 23--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 23 ---3793.769287109375 |
| Processing layer 24--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 24 ---3795.335693359375 |
| Processing layer 25--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 25 ---3797.1728515625 |
| Processing layer 26--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 26 ---3800.90966796875 |
| Processing layer 27--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 27 ---3804.0595703125 |
| Processing layer 28--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 28 ---3810.544921875 |
| Processing layer 29--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 29 ---3812.28857421875 |
| Processing layer 30--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 30 ---3802.36669921875 |
| Processing layer 31--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 31 ---3771.994873046875 |
| metric_name effective_rank: [29, 28, 27, 30, 26, 25, 24, 23, 21, 22, 20, 4, 19, 3, 18, 5, 17, 31, 15, 16, 6, 7, 13, 14, 8, 9, 12, 11, 10, 2, 1, 0] |
| Begin main_assign: Llama-2-7b-hf mlp ZD |
|
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards: 50%|█████ | 1/2 [00:18<00:18, 18.59s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:25<00:00, 11.55s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:25<00:00, 12.60s/it] |
| Once upon a time, in a faraway land, there lived a prince and a princess. They loved each other very much. The prince was very rich, but the princess was poor. The prince loved the princess so much that he wanted to give her everything he had. |
| One day, the prince decided to give the princess a gift. He went to the market and bought her a beautiful golden ring. The princess was very happy. She loved the ring. But the prince was not happy |
| LlamaForCausalLM( |
| (model): LlamaModel( |
| (embed_tokens): Embedding(32000, 4096, padding_idx=0) |
| (layers): ModuleList( |
| (0-31): 32 x LlamaDecoderLayer( |
| (self_attn): LlamaAttention( |
| (q_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (k_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (v_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (o_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| ) |
| (mlp): LlamaMLP( |
| (gate_proj): Linear(in_features=4096, out_features=11008, bias=False) |
| (up_proj): Linear(in_features=4096, out_features=11008, bias=False) |
| (down_proj): Linear(in_features=11008, out_features=4096, bias=False) |
| (act_fn): SiLU() |
| ) |
| (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05) |
| (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05) |
| ) |
| ) |
| (norm): LlamaRMSNorm((4096,), eps=1e-05) |
| (rotary_emb): LlamaRotaryEmbedding() |
| ) |
| (lm_head): Linear(in_features=4096, out_features=32000, bias=False) |
| ) |
| config: |
| LlamaConfig { |
| "architectures": [ |
| "LlamaForCausalLM" |
| ], |
| "attention_bias": false, |
| "attention_dropout": 0.0, |
| "bos_token_id": 1, |
| "eos_token_id": 2, |
| "head_dim": 128, |
| "hidden_act": "silu", |
| "hidden_size": 4096, |
| "initializer_range": 0.02, |
| "intermediate_size": 11008, |
| "max_position_embeddings": 4096, |
| "mlp_bias": false, |
| "model_type": "llama", |
| "num_attention_heads": 32, |
| "num_hidden_layers": 32, |
| "num_key_value_heads": 32, |
| "pad_token_id": 0, |
| "pretraining_tp": 1, |
| "rms_norm_eps": 1e-05, |
| "rope_scaling": null, |
| "rope_theta": 10000.0, |
| "tie_word_embeddings": false, |
| "torch_dtype": "float16", |
| "transformers_version": "4.55.2", |
| "use_cache": true, |
| "vocab_size": 32000 |
| } |
| |
| Processing layer 0--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 0 ---0.15578114986419678 |
| Processing layer 1--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 1 ---0.15707579255104065 |
| Processing layer 2--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 2 ---0.15797409415245056 |
| Processing layer 3--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 3 ---0.1574869602918625 |
| Processing layer 4--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 4 ---0.15727774798870087 |
| Processing layer 5--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 5 ---0.15676063299179077 |
| Processing layer 6--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 6 ---0.1562662124633789 |
| Processing layer 7--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 7 ---0.15596774220466614 |
| Processing layer 8--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 8 ---0.1566615253686905 |
| Processing layer 9--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 9 ---0.1561465859413147 |
| Processing layer 10--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 10 ---0.15556025505065918 |
| Processing layer 11--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 11 ---0.15571480989456177 |
| Processing layer 12--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 12 ---0.15566584467887878 |
| Processing layer 13--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 13 ---0.15572252869606018 |
| Processing layer 14--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 14 ---0.15585064888000488 |
| Processing layer 15--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 15 ---0.15579025447368622 |
| Processing layer 16--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 16 ---0.15610937774181366 |
| Processing layer 17--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 17 ---0.1567423790693283 |
| Processing layer 18--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 18 ---0.15685215592384338 |
| Processing layer 19--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 19 ---0.15697705745697021 |
| Processing layer 20--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 20 ---0.15734341740608215 |
| Processing layer 21--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 21 ---0.1579303741455078 |
| Processing layer 22--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 22 ---0.15809854865074158 |
| Processing layer 23--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 23 ---0.15812799334526062 |
| Processing layer 24--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 24 ---0.1577608585357666 |
| Processing layer 25--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 25 ---0.15747487545013428 |
| Processing layer 26--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 26 ---0.1573786735534668 |
| Processing layer 27--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 27 ---0.15686684846878052 |
| Processing layer 28--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 28 ---0.15638381242752075 |
| Processing layer 29--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 29 ---0.1558859944343567 |
| Processing layer 30--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 30 ---0.1538189947605133 |
| Processing layer 31--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 31 ---0.15391211211681366 |
| metric_name ZD: [23, 22, 2, 21, 24, 3, 25, 26, 20, 4, 1, 19, 27, 18, 5, 17, 8, 28, 6, 9, 16, 7, 29, 14, 15, 0, 13, 11, 12, 10, 31, 30] |
| Begin main_assign: Llama-2-7b-hf mlp head_diversity |
|
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards: 50%|█████ | 1/2 [00:18<00:18, 18.53s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:24<00:00, 11.41s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:24<00:00, 12.47s/it] |
| Once upon a time, there were three kingdoms. The first kingdom was ruled by a king who was very wise. The second kingdom was ruled by a king who was very greedy. The third kingdom was ruled by a king who was very lazy. |
| The first king was very happy and lived in peace with his people. The second king was very unhappy and lived in fear of his people. The third king was very sad and lived in shame with his people. |
| One day, the first king decided to |
| LlamaForCausalLM( |
| (model): LlamaModel( |
| (embed_tokens): Embedding(32000, 4096, padding_idx=0) |
| (layers): ModuleList( |
| (0-31): 32 x LlamaDecoderLayer( |
| (self_attn): LlamaAttention( |
| (q_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (k_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (v_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (o_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| ) |
| (mlp): LlamaMLP( |
| (gate_proj): Linear(in_features=4096, out_features=11008, bias=False) |
| (up_proj): Linear(in_features=4096, out_features=11008, bias=False) |
| (down_proj): Linear(in_features=11008, out_features=4096, bias=False) |
| (act_fn): SiLU() |
| ) |
| (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05) |
| (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05) |
| ) |
| ) |
| (norm): LlamaRMSNorm((4096,), eps=1e-05) |
| (rotary_emb): LlamaRotaryEmbedding() |
| ) |
| (lm_head): Linear(in_features=4096, out_features=32000, bias=False) |
| ) |
| config: |
| LlamaConfig { |
| "architectures": [ |
| "LlamaForCausalLM" |
| ], |
| "attention_bias": false, |
| "attention_dropout": 0.0, |
| "bos_token_id": 1, |
| "eos_token_id": 2, |
| "head_dim": 128, |
| "hidden_act": "silu", |
| "hidden_size": 4096, |
| "initializer_range": 0.02, |
| "intermediate_size": 11008, |
| "max_position_embeddings": 4096, |
| "mlp_bias": false, |
| "model_type": "llama", |
| "num_attention_heads": 32, |
| "num_hidden_layers": 32, |
| "num_key_value_heads": 32, |
| "pad_token_id": 0, |
| "pretraining_tp": 1, |
| "rms_norm_eps": 1e-05, |
| "rope_scaling": null, |
| "rope_theta": 10000.0, |
| "tie_word_embeddings": false, |
| "torch_dtype": "float16", |
| "transformers_version": "4.55.2", |
| "use_cache": true, |
| "vocab_size": 32000 |
| } |
| |
| Processing layer 0--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| Traceback (most recent call last): |
| File "/mnt/bn/life-mllm/users/cxr/quantization/quantization_metric/main_assign.py", line 58, in <module> |
| all_layer_alpha = calculate_expert(model, metric=metric_name, keyword=keyword) |
| File "/mnt/bn/life-mllm/users/cxr/quantization/quantization_metric/alphalora/expert_number.py", line 354, in calculate_expert |
| all_layer_alpha.append(torch.stack(layer_final_alpha).mean().item()) |
| RuntimeError: stack expects a non-empty TensorList |
| Begin main_assign: Llama-2-7b-hf mlp coherence |
|
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards: 50%|█████ | 1/2 [00:18<00:18, 18.65s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:25<00:00, 11.55s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:25<00:00, 12.61s/it] |
| Once upon a time, I was a child. I was full of dreams and visions of the future. I believed in the power of love and in the strength of family. I was the idealist, the dreamer, the one who believed that all good things were possible. |
| Then I grew up. I became a teenager. I was still full of dreams and visions of the future, but the dreams were tinged with darkness, and the visions were full of desp |
| LlamaForCausalLM( |
| (model): LlamaModel( |
| (embed_tokens): Embedding(32000, 4096, padding_idx=0) |
| (layers): ModuleList( |
| (0-31): 32 x LlamaDecoderLayer( |
| (self_attn): LlamaAttention( |
| (q_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (k_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (v_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| (o_proj): Linear(in_features=4096, out_features=4096, bias=False) |
| ) |
| (mlp): LlamaMLP( |
| (gate_proj): Linear(in_features=4096, out_features=11008, bias=False) |
| (up_proj): Linear(in_features=4096, out_features=11008, bias=False) |
| (down_proj): Linear(in_features=11008, out_features=4096, bias=False) |
| (act_fn): SiLU() |
| ) |
| (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05) |
| (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05) |
| ) |
| ) |
| (norm): LlamaRMSNorm((4096,), eps=1e-05) |
| (rotary_emb): LlamaRotaryEmbedding() |
| ) |
| (lm_head): Linear(in_features=4096, out_features=32000, bias=False) |
| ) |
| config: |
| LlamaConfig { |
| "architectures": [ |
| "LlamaForCausalLM" |
| ], |
| "attention_bias": false, |
| "attention_dropout": 0.0, |
| "bos_token_id": 1, |
| "eos_token_id": 2, |
| "head_dim": 128, |
| "hidden_act": "silu", |
| "hidden_size": 4096, |
| "initializer_range": 0.02, |
| "intermediate_size": 11008, |
| "max_position_embeddings": 4096, |
| "mlp_bias": false, |
| "model_type": "llama", |
| "num_attention_heads": 32, |
| "num_hidden_layers": 32, |
| "num_key_value_heads": 32, |
| "pad_token_id": 0, |
| "pretraining_tp": 1, |
| "rms_norm_eps": 1e-05, |
| "rope_scaling": null, |
| "rope_theta": 10000.0, |
| "tie_word_embeddings": false, |
| "torch_dtype": "float16", |
| "transformers_version": "4.55.2", |
| "use_cache": true, |
| "vocab_size": 32000 |
| } |
| |
| Processing layer 0--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 0 ---0.018758879974484444 |
| Processing layer 1--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 1 ---0.015916038304567337 |
| Processing layer 2--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 2 ---0.014297164976596832 |
| Processing layer 3--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 3 ---0.013034423813223839 |
| Processing layer 4--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 4 ---0.012964712455868721 |
| Processing layer 5--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 5 ---0.013287676498293877 |
| Processing layer 6--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 6 ---0.013558547012507915 |
| Processing layer 7--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 7 ---0.013750488869845867 |
| Processing layer 8--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 8 ---0.013907302170991898 |
| Processing layer 9--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 9 ---0.013899993151426315 |
| Processing layer 10--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 10 ---0.014043152332305908 |
| Processing layer 11--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 11 ---0.014066706411540508 |
| Processing layer 12--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 12 ---0.01396130956709385 |
| Processing layer 13--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 13 ---0.013774177059531212 |
| Processing layer 14--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 14 ---0.013727117329835892 |
| Processing layer 15--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 15 ---0.013336378149688244 |
| Processing layer 16--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 16 ---0.013238323852419853 |
| Processing layer 17--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 17 ---0.013199622742831707 |
| Processing layer 18--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 18 ---0.012931596487760544 |
| Processing layer 19--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 19 ---0.012751361355185509 |
| Processing layer 20--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 20 ---0.012693298980593681 |
| Processing layer 21--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 21 ---0.012600544840097427 |
| Processing layer 22--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 22 ---0.012644654139876366 |
| Processing layer 23--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 23 ---0.012538459151983261 |
| Processing layer 24--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 24 ---0.012461837381124496 |
| Processing layer 25--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 25 ---0.012478632852435112 |
| Processing layer 26--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 26 ---0.012449456378817558 |
| Processing layer 27--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 27 ---0.012493574991822243 |
| Processing layer 28--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 28 ---0.012300195172429085 |
| Processing layer 29--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 29 ---0.01232621818780899 |
| Processing layer 30--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 30 ---0.012763611972332 |
| Processing layer 31--subset--{'mlp.gate_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=11008, bias=False), 'mlp.down_proj': Linear(in_features=11008, out_features=4096, bias=False)} |
| alpha value of layer 31 ---0.014904310926795006 |
| metric_name coherence: [0, 1, 31, 2, 11, 10, 12, 8, 9, 13, 7, 14, 6, 15, 5, 16, 17, 3, 4, 18, 30, 19, 20, 22, 21, 23, 27, 25, 24, 26, 29, 28] |
| Begin main_assign: Qwen2.5-7B self_attn alpha |
|
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:00<00:02, 1.28it/s]
Loading checkpoint shards: 50%|█████ | 2/4 [00:01<00:01, 1.33it/s]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:02<00:00, 1.33it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00, 1.29it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00, 1.30it/s] |
| Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation. |
| Once upon a time, there was a king who loved gold. He even had his own gold mine, but he was never satisfied with what he had. One day, he decided to go on a journey to find even more gold. He sent his men to explore the world and bring back as much gold as they could find. They searched high and low, but they could not find any more gold. The king was disappointed. He felt like he had been cheated. But then, he had an idea. He decided |
| Qwen2ForCausalLM( |
| (model): Qwen2Model( |
| (embed_tokens): Embedding(152064, 3584) |
| (layers): ModuleList( |
| (0-27): 28 x Qwen2DecoderLayer( |
| (self_attn): Qwen2Attention( |
| (q_proj): Linear(in_features=3584, out_features=3584, bias=True) |
| (k_proj): Linear(in_features=3584, out_features=512, bias=True) |
| (v_proj): Linear(in_features=3584, out_features=512, bias=True) |
| (o_proj): Linear(in_features=3584, out_features=3584, bias=False) |
| ) |
| (mlp): Qwen2MLP( |
| (gate_proj): Linear(in_features=3584, out_features=18944, bias=False) |
| (up_proj): Linear(in_features=3584, out_features=18944, bias=False) |
| (down_proj): Linear(in_features=18944, out_features=3584, bias=False) |
| (act_fn): SiLU() |
| ) |
| (input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06) |
| (post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06) |
| ) |
| ) |
| (norm): Qwen2RMSNorm((3584,), eps=1e-06) |
| (rotary_emb): Qwen2RotaryEmbedding() |
| ) |
| (lm_head): Linear(in_features=3584, out_features=152064, bias=False) |
| ) |
| config: |
| Qwen2Config { |
| "architectures": [ |
| "Qwen2ForCausalLM" |
| ], |
| "attention_dropout": 0.0, |
| "bos_token_id": 151643, |
| "eos_token_id": 151643, |
| "hidden_act": "silu", |
| "hidden_size": 3584, |
| "initializer_range": 0.02, |
| "intermediate_size": 18944, |
| "layer_types": [ |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention" |
| ], |
| "max_position_embeddings": 131072, |
| "max_window_layers": 28, |
| "model_type": "qwen2", |
| "num_attention_heads": 28, |
| "num_hidden_layers": 28, |
| "num_key_value_heads": 4, |
| "rms_norm_eps": 1e-06, |
| "rope_scaling": null, |
| "rope_theta": 1000000.0, |
| "sliding_window": null, |
| "tie_word_embeddings": false, |
| "torch_dtype": "float16", |
| "transformers_version": "4.55.2", |
| "use_cache": true, |
| "use_mrope": false, |
| "use_sliding_window": false, |
| "vocab_size": 152064 |
| } |
| |
| Processing layer 0--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 0 ---3.7398271560668945 |
| Processing layer 1--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 1 ---4.61224889755249 |
| Processing layer 2--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 2 ---3.6950488090515137 |
| Processing layer 3--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 3 ---3.5232601165771484 |
| Processing layer 4--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 4 ---4.135606288909912 |
| Processing layer 5--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 5 ---3.5275750160217285 |
| Processing layer 6--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 6 ---6.992785453796387 |
| Processing layer 7--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 7 ---3.8776655197143555 |
| Processing layer 8--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 8 ---5.665498733520508 |
| Processing layer 9--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 9 ---4.215641021728516 |
| Processing layer 10--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 10 ---3.8878092765808105 |
| Processing layer 11--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 11 ---4.219095706939697 |
| Processing layer 12--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 12 ---3.7720558643341064 |
| Processing layer 13--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 13 ---7.641180515289307 |
| Processing layer 14--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 14 ---3.707275152206421 |
| Processing layer 15--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 15 ---3.25449800491333 |
| Processing layer 16--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 16 ---4.158566474914551 |
| Processing layer 17--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 17 ---3.467252492904663 |
| Processing layer 18--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 18 ---5.359913349151611 |
| Processing layer 19--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 19 ---3.048678398132324 |
| Processing layer 20--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 20 ---2.6340246200561523 |
| Processing layer 21--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 21 ---4.8504228591918945 |
| Processing layer 22--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 22 ---3.2433063983917236 |
| Processing layer 23--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 23 ---6.264342308044434 |
| Processing layer 24--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 24 ---3.845273494720459 |
| Processing layer 25--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 25 ---4.786241054534912 |
| Processing layer 26--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 26 ---2.9173049926757812 |
| Processing layer 27--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 27 ---2.295574426651001 |
| metric_name alpha: [13, 6, 23, 8, 18, 21, 25, 1, 11, 9, 16, 4, 10, 7, 24, 12, 0, 14, 2, 5, 3, 17, 15, 22, 19, 26, 20, 27] |
| Begin main_assign: Qwen2.5-7B self_attn alpha_hat |
|
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:00<00:01, 1.56it/s]
Loading checkpoint shards: 50%|█████ | 2/4 [00:01<00:01, 1.63it/s]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:01<00:00, 1.62it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00, 1.50it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00, 1.54it/s] |
| Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation. |
| Once upon a time， there was a man who was in the habit of getting up at six o'clock every morning and jogging around his neighborhood． One morning， while he was out for his usual morning run， he saw an old man sitting in a park bench． He asked the old man what he was doing there． The old man replied，"I am waiting for my son to arrive．" The man asked，"What does he do？ Where does he work？" The old man said，"He is |
| Qwen2ForCausalLM( |
| (model): Qwen2Model( |
| (embed_tokens): Embedding(152064, 3584) |
| (layers): ModuleList( |
| (0-27): 28 x Qwen2DecoderLayer( |
| (self_attn): Qwen2Attention( |
| (q_proj): Linear(in_features=3584, out_features=3584, bias=True) |
| (k_proj): Linear(in_features=3584, out_features=512, bias=True) |
| (v_proj): Linear(in_features=3584, out_features=512, bias=True) |
| (o_proj): Linear(in_features=3584, out_features=3584, bias=False) |
| ) |
| (mlp): Qwen2MLP( |
| (gate_proj): Linear(in_features=3584, out_features=18944, bias=False) |
| (up_proj): Linear(in_features=3584, out_features=18944, bias=False) |
| (down_proj): Linear(in_features=18944, out_features=3584, bias=False) |
| (act_fn): SiLU() |
| ) |
| (input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06) |
| (post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06) |
| ) |
| ) |
| (norm): Qwen2RMSNorm((3584,), eps=1e-06) |
| (rotary_emb): Qwen2RotaryEmbedding() |
| ) |
| (lm_head): Linear(in_features=3584, out_features=152064, bias=False) |
| ) |
| config: |
| Qwen2Config { |
| "architectures": [ |
| "Qwen2ForCausalLM" |
| ], |
| "attention_dropout": 0.0, |
| "bos_token_id": 151643, |
| "eos_token_id": 151643, |
| "hidden_act": "silu", |
| "hidden_size": 3584, |
| "initializer_range": 0.02, |
| "intermediate_size": 18944, |
| "layer_types": [ |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention" |
| ], |
| "max_position_embeddings": 131072, |
| "max_window_layers": 28, |
| "model_type": "qwen2", |
| "num_attention_heads": 28, |
| "num_hidden_layers": 28, |
| "num_key_value_heads": 4, |
| "rms_norm_eps": 1e-06, |
| "rope_scaling": null, |
| "rope_theta": 1000000.0, |
| "sliding_window": null, |
| "tie_word_embeddings": false, |
| "torch_dtype": "float16", |
| "transformers_version": "4.55.2", |
| "use_cache": true, |
| "use_mrope": false, |
| "use_sliding_window": false, |
| "vocab_size": 152064 |
| } |
|
|
| Processing layer 0--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 0 ---11.435125350952148 |
| Processing layer 1--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 1 ---7.411468505859375 |
| Processing layer 2--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 2 ---7.694450378417969 |
| Processing layer 3--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 3 ---7.890998363494873 |
| Processing layer 4--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 4 ---7.172995567321777 |
| Processing layer 5--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 5 ---7.2080488204956055 |
| Processing layer 6--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 6 ---11.36629867553711 |
| Processing layer 7--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 7 ---7.176176071166992 |
| Processing layer 8--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 8 ---7.694736957550049 |
| Processing layer 9--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 9 ---7.057432174682617 |
| Processing layer 10--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 10 ---6.2363386154174805 |
| Processing layer 11--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 11 ---6.52932071685791 |
| Processing layer 12--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 12 ---7.182070732116699 |
| Processing layer 13--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 13 ---8.655073165893555 |
| Processing layer 14--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 14 ---5.977964401245117 |
| Processing layer 15--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 15 ---5.821178436279297 |
| Processing layer 16--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 16 ---7.714193820953369 |
| Processing layer 17--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 17 ---6.244594573974609 |
| Processing layer 18--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 18 ---7.704196929931641 |
| Processing layer 19--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 19 ---5.97743558883667 |
| Processing layer 20--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 20 ---5.515291213989258 |
| Processing layer 21--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 21 ---7.423676490783691 |
| Processing layer 22--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 22 ---6.235415458679199 |
| Processing layer 23--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 23 ---11.209845542907715 |
| Processing layer 24--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 24 ---7.722582817077637 |
| Processing layer 25--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 25 ---10.762764930725098 |
| Processing layer 26--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 26 ---7.431804180145264 |
| Processing layer 27--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 27 ---8.220139503479004 |
| metric_name alpha_hat: [0, 6, 23, 25, 13, 27, 3, 24, 16, 18, 8, 2, 26, 21, 1, 5, 12, 7, 4, 9, 11, 17, 10, 22, 14, 19, 15, 20] |
| Begin main_assign: Qwen2.5-7B self_attn stable_rank |
|
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:00<00:02, 1.21it/s]
Loading checkpoint shards: 50%|█████ | 2/4 [00:01<00:01, 1.29it/s]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:02<00:00, 1.31it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00, 1.37it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00, 1.34it/s] |
| Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation. |
| Once upon a time, there was a family with 10 children. Each of the 10 children had a different number of books in their bookshelves. The first child had 1 book, the second had 2 books, the third had 3 books, and so on until the tenth child, who had 10 books. One day, the parents decided to redistribute the books so that each child would have the same number of books. How many books did each child end up with? To |
| Qwen2ForCausalLM( |
| (model): Qwen2Model( |
| (embed_tokens): Embedding(152064, 3584) |
| (layers): ModuleList( |
| (0-27): 28 x Qwen2DecoderLayer( |
| (self_attn): Qwen2Attention( |
| (q_proj): Linear(in_features=3584, out_features=3584, bias=True) |
| (k_proj): Linear(in_features=3584, out_features=512, bias=True) |
| (v_proj): Linear(in_features=3584, out_features=512, bias=True) |
| (o_proj): Linear(in_features=3584, out_features=3584, bias=False) |
| ) |
| (mlp): Qwen2MLP( |
| (gate_proj): Linear(in_features=3584, out_features=18944, bias=False) |
| (up_proj): Linear(in_features=3584, out_features=18944, bias=False) |
| (down_proj): Linear(in_features=18944, out_features=3584, bias=False) |
| (act_fn): SiLU() |
| ) |
| (input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06) |
| (post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06) |
| ) |
| ) |
| (norm): Qwen2RMSNorm((3584,), eps=1e-06) |
| (rotary_emb): Qwen2RotaryEmbedding() |
| ) |
| (lm_head): Linear(in_features=3584, out_features=152064, bias=False) |
| ) |
| config: |
| Qwen2Config { |
| "architectures": [ |
| "Qwen2ForCausalLM" |
| ], |
| "attention_dropout": 0.0, |
| "bos_token_id": 151643, |
| "eos_token_id": 151643, |
| "hidden_act": "silu", |
| "hidden_size": 3584, |
| "initializer_range": 0.02, |
| "intermediate_size": 18944, |
| "layer_types": [ |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention" |
| ], |
| "max_position_embeddings": 131072, |
| "max_window_layers": 28, |
| "model_type": "qwen2", |
| "num_attention_heads": 28, |
| "num_hidden_layers": 28, |
| "num_key_value_heads": 4, |
| "rms_norm_eps": 1e-06, |
| "rope_scaling": null, |
| "rope_theta": 1000000.0, |
| "sliding_window": null, |
| "tie_word_embeddings": false, |
| "torch_dtype": "float16", |
| "transformers_version": "4.55.2", |
| "use_cache": true, |
| "use_mrope": false, |
| "use_sliding_window": false, |
| "vocab_size": 152064 |
| } |
|
|
| Processing layer 0--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| frobenius_norm tensor(74.6100, device='cuda:0') |
| spectral_norm tensor(22.8962, device='cuda:0') |
| frobenius_norm tensor(37.2340, device='cuda:0') |
| spectral_norm tensor(4.1839, device='cuda:0') |
| frobenius_norm tensor(12.2050, device='cuda:0') |
| spectral_norm tensor(1.2453, device='cuda:0') |
| frobenius_norm tensor(44.6291, device='cuda:0') |
| spectral_norm tensor(6.0001, device='cuda:0') |
| alpha value of layer 0 ---60.29820251464844 |
| Processing layer 1--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| frobenius_norm tensor(56.7970, device='cuda:0') |
| spectral_norm tensor(10.4783, device='cuda:0') |
| frobenius_norm tensor(29.9237, device='cuda:0') |
| spectral_norm tensor(4.7115, device='cuda:0') |
| frobenius_norm tensor(18.3318, device='cuda:0') |
| spectral_norm tensor(1.4635, device='cuda:0') |
| frobenius_norm tensor(51.8109, device='cuda:0') |
| spectral_norm tensor(3.7208, device='cuda:0') |
| alpha value of layer 1 ---105.13154602050781 |
| Processing layer 2--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| frobenius_norm tensor(65.4367, device='cuda:0') |
| spectral_norm tensor(5.5540, device='cuda:0') |
| frobenius_norm tensor(31.8038, device='cuda:0') |
| spectral_norm tensor(3.0331, device='cuda:0') |
| frobenius_norm tensor(14.7804, device='cuda:0') |
| spectral_norm tensor(1.2128, device='cuda:0') |
| frobenius_norm tensor(50.6930, device='cuda:0') |
| spectral_norm tensor(3.9768, device='cuda:0') |
| alpha value of layer 2 ---139.9442901611328 |
| Processing layer 3--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| frobenius_norm tensor(67.9445, device='cuda:0') |
| spectral_norm tensor(7.3888, device='cuda:0') |
| frobenius_norm tensor(32.4597, device='cuda:0') |
| spectral_norm tensor(3.6898, device='cuda:0') |
| frobenius_norm tensor(17.2702, device='cuda:0') |
| spectral_norm tensor(1.2119, device='cuda:0') |
| frobenius_norm tensor(53.1248, device='cuda:0') |
| spectral_norm tensor(3.7626, device='cuda:0') |
| alpha value of layer 3 ---141.09706115722656 |
| Processing layer 4--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| frobenius_norm tensor(64.7004, device='cuda:0') |
| spectral_norm tensor(6.6270, device='cuda:0') |
| frobenius_norm tensor(29.1071, device='cuda:0') |
| spectral_norm tensor(3.3637, device='cuda:0') |
| frobenius_norm tensor(20.8693, device='cuda:0') |
| spectral_norm tensor(1.5107, device='cuda:0') |
| frobenius_norm tensor(53.3331, device='cuda:0') |
| spectral_norm tensor(3.7430, device='cuda:0') |
| alpha value of layer 4 ---141.01356506347656 |
| Processing layer 5--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| frobenius_norm tensor(63.2364, device='cuda:0') |
| spectral_norm tensor(5.9598, device='cuda:0') |
| frobenius_norm tensor(26.6552, device='cuda:0') |
| spectral_norm tensor(2.6609, device='cuda:0') |
| frobenius_norm tensor(19.8491, device='cuda:0') |
| spectral_norm tensor(1.5480, device='cuda:0') |
| frobenius_norm tensor(53.4192, device='cuda:0') |
| spectral_norm tensor(4.0186, device='cuda:0') |
| alpha value of layer 5 ---138.5137939453125 |
| Processing layer 6--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| frobenius_norm tensor(64.2853, device='cuda:0') |
| spectral_norm tensor(5.5097, device='cuda:0') |
| frobenius_norm tensor(28.5616, device='cuda:0') |
| spectral_norm tensor(2.9881, device='cuda:0') |
| frobenius_norm tensor(20.6937, device='cuda:0') |
| spectral_norm tensor(1.4517, device='cuda:0') |
| frobenius_norm tensor(54.9730, device='cuda:0') |
| spectral_norm tensor(5.0745, device='cuda:0') |
| alpha value of layer 6 ---137.01039123535156 |
| Processing layer 7--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| frobenius_norm tensor(60.9057, device='cuda:0') |
| spectral_norm tensor(4.5698, device='cuda:0') |
| frobenius_norm tensor(23.8630, device='cuda:0') |
| spectral_norm tensor(2.5382, device='cuda:0') |
| frobenius_norm tensor(24.6983, device='cuda:0') |
| spectral_norm tensor(1.7165, device='cuda:0') |
| frobenius_norm tensor(60.3206, device='cuda:0') |
| spectral_norm tensor(3.5918, device='cuda:0') |
| alpha value of layer 7 ---188.773193359375 |
| Processing layer 8--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| frobenius_norm tensor(63.3943, device='cuda:0') |
| spectral_norm tensor(3.9526, device='cuda:0') |
| frobenius_norm tensor(27.2992, device='cuda:0') |
| spectral_norm tensor(2.5236, device='cuda:0') |
| frobenius_norm tensor(20.6254, device='cuda:0') |
| spectral_norm tensor(1.4007, device='cuda:0') |
| frobenius_norm tensor(55.2465, device='cuda:0') |
| spectral_norm tensor(3.4821, device='cuda:0') |
| alpha value of layer 8 ---210.70074462890625 |
| Processing layer 9--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| frobenius_norm tensor(58.6742, device='cuda:0') |
| spectral_norm tensor(4.6933, device='cuda:0') |
| frobenius_norm tensor(23.2825, device='cuda:0') |
| spectral_norm tensor(2.6721, device='cuda:0') |
| frobenius_norm tensor(24.7824, device='cuda:0') |
| spectral_norm tensor(1.5406, device='cuda:0') |
| frobenius_norm tensor(60.4736, device='cuda:0') |
| spectral_norm tensor(6.3921, device='cuda:0') |
| alpha value of layer 9 ---145.1187744140625 |
| Processing layer 10--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| frobenius_norm tensor(63.5265, device='cuda:0') |
| spectral_norm tensor(4.3337, device='cuda:0') |
| frobenius_norm tensor(27.1029, device='cuda:0') |
| spectral_norm tensor(2.6480, device='cuda:0') |
| frobenius_norm tensor(22.3527, device='cuda:0') |
| spectral_norm tensor(1.3822, device='cuda:0') |
| frobenius_norm tensor(57.0854, device='cuda:0') |
| spectral_norm tensor(4.1727, device='cuda:0') |
| alpha value of layer 10 ---192.07977294921875 |
| Processing layer 11--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| frobenius_norm tensor(64.2904, device='cuda:0') |
| spectral_norm tensor(4.0832, device='cuda:0') |
| frobenius_norm tensor(28.0532, device='cuda:0') |
| spectral_norm tensor(2.6312, device='cuda:0') |
| frobenius_norm tensor(19.8152, device='cuda:0') |
| spectral_norm tensor(1.4528, device='cuda:0') |
| frobenius_norm tensor(55.3427, device='cuda:0') |
| spectral_norm tensor(4.3578, device='cuda:0') |
| alpha value of layer 11 ---177.22564697265625 |
| Processing layer 12--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| frobenius_norm tensor(62.3252, device='cuda:0') |
| spectral_norm tensor(3.9938, device='cuda:0') |
| frobenius_norm tensor(26.8499, device='cuda:0') |
| spectral_norm tensor(2.5529, device='cuda:0') |
| frobenius_norm tensor(20.6057, device='cuda:0') |
| spectral_norm tensor(1.3954, device='cuda:0') |
| frobenius_norm tensor(55.6693, device='cuda:0') |
| spectral_norm tensor(4.6503, device='cuda:0') |
| alpha value of layer 12 ---178.88079833984375 |
| Processing layer 13--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| frobenius_norm tensor(61.0881, device='cuda:0') |
| spectral_norm tensor(4.2847, device='cuda:0') |
| frobenius_norm tensor(25.6297, device='cuda:0') |
| spectral_norm tensor(2.9570, device='cuda:0') |
| frobenius_norm tensor(22.4836, device='cuda:0') |
| spectral_norm tensor(1.3594, device='cuda:0') |
| frobenius_norm tensor(57.5732, device='cuda:0') |
| spectral_norm tensor(5.3631, device='cuda:0') |
| alpha value of layer 13 ---166.80108642578125 |
| Processing layer 14--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| frobenius_norm tensor(59.9385, device='cuda:0') |
| spectral_norm tensor(4.1989, device='cuda:0') |
| frobenius_norm tensor(25.3315, device='cuda:0') |
| spectral_norm tensor(2.3592, device='cuda:0') |
| frobenius_norm tensor(19.7791, device='cuda:0') |
| spectral_norm tensor(1.4382, device='cuda:0') |
| frobenius_norm tensor(54.6680, device='cuda:0') |
| spectral_norm tensor(4.8688, device='cuda:0') |
| alpha value of layer 14 ---158.56539916992188 |
| Processing layer 15--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| frobenius_norm tensor(62.2632, device='cuda:0') |
| spectral_norm tensor(4.4386, device='cuda:0') |
| frobenius_norm tensor(26.5240, device='cuda:0') |
| spectral_norm tensor(2.5572, device='cuda:0') |
| frobenius_norm tensor(20.8667, device='cuda:0') |
| spectral_norm tensor(1.5160, device='cuda:0') |
| frobenius_norm tensor(55.3834, device='cuda:0') |
| spectral_norm tensor(4.4431, device='cuda:0') |
| alpha value of layer 15 ---162.2994384765625 |
| Processing layer 16--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| frobenius_norm tensor(59.5504, device='cuda:0') |
| spectral_norm tensor(4.1705, device='cuda:0') |
| frobenius_norm tensor(23.7914, device='cuda:0') |
| spectral_norm tensor(2.3827, device='cuda:0') |
| frobenius_norm tensor(22.4299, device='cuda:0') |
| spectral_norm tensor(1.8284, device='cuda:0') |
| frobenius_norm tensor(57.2100, device='cuda:0') |
| spectral_norm tensor(5.3112, device='cuda:0') |
| alpha value of layer 16 ---142.52708435058594 |
| Processing layer 17--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| frobenius_norm tensor(61.2952, device='cuda:0') |
| spectral_norm tensor(3.9247, device='cuda:0') |
| frobenius_norm tensor(24.2607, device='cuda:0') |
| spectral_norm tensor(2.3428, device='cuda:0') |
| frobenius_norm tensor(22.0993, device='cuda:0') |
| spectral_norm tensor(1.6527, device='cuda:0') |
| frobenius_norm tensor(56.9147, device='cuda:0') |
| spectral_norm tensor(4.7962, device='cuda:0') |
| alpha value of layer 17 ---167.6918182373047 |
| Processing layer 18--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| frobenius_norm tensor(57.8086, device='cuda:0') |
| spectral_norm tensor(4.1784, device='cuda:0') |
| frobenius_norm tensor(23.1346, device='cuda:0') |
| spectral_norm tensor(2.7649, device='cuda:0') |
| frobenius_norm tensor(24.9173, device='cuda:0') |
| spectral_norm tensor(1.5424, device='cuda:0') |
| frobenius_norm tensor(60.8592, device='cuda:0') |
| spectral_norm tensor(5.2139, device='cuda:0') |
| alpha value of layer 18 ---164.66351318359375 |
| Processing layer 19--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| frobenius_norm tensor(57.4449, device='cuda:0') |
| spectral_norm tensor(4.5447, device='cuda:0') |
| frobenius_norm tensor(21.3000, device='cuda:0') |
| spectral_norm tensor(2.4431, device='cuda:0') |
| frobenius_norm tensor(25.0779, device='cuda:0') |
| spectral_norm tensor(1.6237, device='cuda:0') |
| frobenius_norm tensor(59.6389, device='cuda:0') |
| spectral_norm tensor(4.9505, device='cuda:0') |
| alpha value of layer 19 ---154.86126708984375 |
| Processing layer 20--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| frobenius_norm tensor(58.7482, device='cuda:0') |
| spectral_norm tensor(4.0858, device='cuda:0') |
| frobenius_norm tensor(22.3411, device='cuda:0') |
| spectral_norm tensor(2.3521, device='cuda:0') |
| frobenius_norm tensor(25.9143, device='cuda:0') |
| spectral_norm tensor(1.7104, device='cuda:0') |
| frobenius_norm tensor(60.9286, device='cuda:0') |
| spectral_norm tensor(4.8709, device='cuda:0') |
| alpha value of layer 20 ---170.74554443359375 |
| Processing layer 21--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| frobenius_norm tensor(57.1627, device='cuda:0') |
| spectral_norm tensor(3.6813, device='cuda:0') |
| frobenius_norm tensor(19.9702, device='cuda:0') |
| spectral_norm tensor(2.2489, device='cuda:0') |
| frobenius_norm tensor(27.7738, device='cuda:0') |
| spectral_norm tensor(1.6762, device='cuda:0') |
| frobenius_norm tensor(63.2683, device='cuda:0') |
| spectral_norm tensor(5.2737, device='cuda:0') |
| alpha value of layer 21 ---184.6076202392578 |
| Processing layer 22--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| frobenius_norm tensor(58.1757, device='cuda:0') |
| spectral_norm tensor(4.0771, device='cuda:0') |
| frobenius_norm tensor(19.4699, device='cuda:0') |
| spectral_norm tensor(1.9257, device='cuda:0') |
| frobenius_norm tensor(27.1717, device='cuda:0') |
| spectral_norm tensor(2.0305, device='cuda:0') |
| frobenius_norm tensor(63.5510, device='cuda:0') |
| spectral_norm tensor(4.3551, device='cuda:0') |
| alpha value of layer 22 ---174.45973205566406 |
| Processing layer 23--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| frobenius_norm tensor(59.5436, device='cuda:0') |
| spectral_norm tensor(4.0933, device='cuda:0') |
| frobenius_norm tensor(20.5584, device='cuda:0') |
| spectral_norm tensor(2.1754, device='cuda:0') |
| frobenius_norm tensor(27.5827, device='cuda:0') |
| spectral_norm tensor(1.8698, device='cuda:0') |
| frobenius_norm tensor(64.9988, device='cuda:0') |
| spectral_norm tensor(5.3918, device='cuda:0') |
| alpha value of layer 23 ---165.96688842773438 |
| Processing layer 24--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| frobenius_norm tensor(56.8169, device='cuda:0') |
| spectral_norm tensor(3.8277, device='cuda:0') |
| frobenius_norm tensor(19.4813, device='cuda:0') |
| spectral_norm tensor(1.8291, device='cuda:0') |
| frobenius_norm tensor(31.0836, device='cuda:0') |
| spectral_norm tensor(2.3168, device='cuda:0') |
| frobenius_norm tensor(65.3215, device='cuda:0') |
| spectral_norm tensor(6.2412, device='cuda:0') |
| alpha value of layer 24 ---155.8324432373047 |
| Processing layer 25--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| frobenius_norm tensor(55.3445, device='cuda:0') |
| spectral_norm tensor(3.8164, device='cuda:0') |
| frobenius_norm tensor(17.7185, device='cuda:0') |
| spectral_norm tensor(1.8025, device='cuda:0') |
| frobenius_norm tensor(33.9660, device='cuda:0') |
| spectral_norm tensor(3.0984, device='cuda:0') |
| frobenius_norm tensor(68.8981, device='cuda:0') |
| spectral_norm tensor(5.3780, device='cuda:0') |
| alpha value of layer 25 ---147.80517578125 |
| Processing layer 26--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| frobenius_norm tensor(51.7868, device='cuda:0') |
| spectral_norm tensor(3.9548, device='cuda:0') |
| frobenius_norm tensor(16.9575, device='cuda:0') |
| spectral_norm tensor(1.9295, device='cuda:0') |
| frobenius_norm tensor(40.6153, device='cuda:0') |
| spectral_norm tensor(3.1825, device='cuda:0') |
| frobenius_norm tensor(71.6628, device='cuda:0') |
| spectral_norm tensor(6.2830, device='cuda:0') |
| alpha value of layer 26 ---135.41717529296875 |
| Processing layer 27--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| frobenius_norm tensor(56.6802, device='cuda:0') |
| spectral_norm tensor(7.8725, device='cuda:0') |
| frobenius_norm tensor(18.0258, device='cuda:0') |
| spectral_norm tensor(2.1181, device='cuda:0') |
| frobenius_norm tensor(36.7123, device='cuda:0') |
| spectral_norm tensor(4.5755, device='cuda:0') |
| frobenius_norm tensor(66.9187, device='cuda:0') |
| spectral_norm tensor(10.3449, device='cuda:0') |
| alpha value of layer 27 ---57.62150573730469 |
| metric_name stable_rank: [8, 10, 7, 21, 12, 11, 22, 20, 17, 13, 23, 18, 15, 14, 24, 19, 25, 9, 16, 3, 4, 2, 5, 6, 26, 1, 0, 27] |
| Begin main_assign: Qwen2.5-7B self_attn effective_rank |
|
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:00<00:02, 1.40it/s]
Loading checkpoint shards: 50%|█████ | 2/4 [00:01<00:01, 1.46it/s]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:02<00:00, 1.46it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00, 1.54it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00, 1.50it/s] |
| Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation. |
| Once upon a time, in a faraway land, there was a wizard named Zephyr. Zephyr had a magical garden filled with enchanted flowers that bloomed only once a year on the night of the full moon. Each flower had a unique power: some could grant wishes, others could heal, and some could even bring the dead back to life. Zephyr knew that the garden was in danger, and he needed to protect it. He decided to create a secret code to lock the garden's entrance |
| Qwen2ForCausalLM( |
| (model): Qwen2Model( |
| (embed_tokens): Embedding(152064, 3584) |
| (layers): ModuleList( |
| (0-27): 28 x Qwen2DecoderLayer( |
| (self_attn): Qwen2Attention( |
| (q_proj): Linear(in_features=3584, out_features=3584, bias=True) |
| (k_proj): Linear(in_features=3584, out_features=512, bias=True) |
| (v_proj): Linear(in_features=3584, out_features=512, bias=True) |
| (o_proj): Linear(in_features=3584, out_features=3584, bias=False) |
| ) |
| (mlp): Qwen2MLP( |
| (gate_proj): Linear(in_features=3584, out_features=18944, bias=False) |
| (up_proj): Linear(in_features=3584, out_features=18944, bias=False) |
| (down_proj): Linear(in_features=18944, out_features=3584, bias=False) |
| (act_fn): SiLU() |
| ) |
| (input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06) |
| (post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06) |
| ) |
| ) |
| (norm): Qwen2RMSNorm((3584,), eps=1e-06) |
| (rotary_emb): Qwen2RotaryEmbedding() |
| ) |
| (lm_head): Linear(in_features=3584, out_features=152064, bias=False) |
| ) |
| config: |
| Qwen2Config { |
| "architectures": [ |
| "Qwen2ForCausalLM" |
| ], |
| "attention_dropout": 0.0, |
| "bos_token_id": 151643, |
| "eos_token_id": 151643, |
| "hidden_act": "silu", |
| "hidden_size": 3584, |
| "initializer_range": 0.02, |
| "intermediate_size": 18944, |
| "layer_types": [ |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention" |
| ], |
| "max_position_embeddings": 131072, |
| "max_window_layers": 28, |
| "model_type": "qwen2", |
| "num_attention_heads": 28, |
| "num_hidden_layers": 28, |
| "num_key_value_heads": 4, |
| "rms_norm_eps": 1e-06, |
| "rope_scaling": null, |
| "rope_theta": 1000000.0, |
| "sliding_window": null, |
| "tie_word_embeddings": false, |
| "torch_dtype": "float16", |
| "transformers_version": "4.55.2", |
| "use_cache": true, |
| "use_mrope": false, |
| "use_sliding_window": false, |
| "vocab_size": 152064 |
| } |
|
|
| Processing layer 0--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 0 ---1452.84130859375 |
| Processing layer 1--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 1 ---1305.515869140625 |
| Processing layer 2--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 2 ---1505.5045166015625 |
| Processing layer 3--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 3 ---1528.22900390625 |
| Processing layer 4--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 4 ---1506.9344482421875 |
| Processing layer 5--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 5 ---1515.52099609375 |
| Processing layer 6--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 6 ---1521.9154052734375 |
| Processing layer 7--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 7 ---1530.30126953125 |
| Processing layer 8--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 8 ---1513.680908203125 |
| Processing layer 9--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 9 ---1508.0103759765625 |
| Processing layer 10--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 10 ---1541.102294921875 |
| Processing layer 11--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 11 ---1515.346923828125 |
| Processing layer 12--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 12 ---1515.414306640625 |
| Processing layer 13--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 13 ---1518.91796875 |
| Processing layer 14--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 14 ---1460.6475830078125 |
| Processing layer 15--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 15 ---1482.7188720703125 |
| Processing layer 16--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 16 ---1503.2802734375 |
| Processing layer 17--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 17 ---1507.509033203125 |
| Processing layer 18--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 18 ---1475.742431640625 |
| Processing layer 19--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 19 ---1503.236083984375 |
| Processing layer 20--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 20 ---1508.0513916015625 |
| Processing layer 21--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 21 ---1513.8974609375 |
| Processing layer 22--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 22 ---1492.510986328125 |
| Processing layer 23--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 23 ---1560.966552734375 |
| Processing layer 24--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 24 ---1547.26904296875 |
| Processing layer 25--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 25 ---1559.457763671875 |
| Processing layer 26--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 26 ---1530.5631103515625 |
| Processing layer 27--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 27 ---1474.574462890625 |
| metric_name effective_rank: [23, 25, 24, 10, 26, 7, 3, 6, 13, 5, 12, 11, 21, 8, 20, 9, 17, 4, 2, 16, 19, 22, 15, 18, 27, 14, 0, 1] |
| Begin main_assign: Qwen2.5-7B self_attn ZD |
|
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:00<00:01, 1.55it/s]
Loading checkpoint shards: 50%|█████ | 2/4 [00:01<00:01, 1.64it/s]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:01<00:00, 1.64it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00, 1.74it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00, 1.70it/s] |
| Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation. |
| Once upon a time there was a very special fish, who lived in a very special lake, and who had a very special name. |
| And when the fish was born, his parents named him Nemo. |
| Nemo was a very happy fish, who loved to swim around his lake, and who had a lot of friends. |
| There was Dory, the forgetful fish, who would always forget where she was going, and Marlin, the protective fish, who would always look after his family. |
| Nemo was also |
| Qwen2ForCausalLM( |
| (model): Qwen2Model( |
| (embed_tokens): Embedding(152064, 3584) |
| (layers): ModuleList( |
| (0-27): 28 x Qwen2DecoderLayer( |
| (self_attn): Qwen2Attention( |
| (q_proj): Linear(in_features=3584, out_features=3584, bias=True) |
| (k_proj): Linear(in_features=3584, out_features=512, bias=True) |
| (v_proj): Linear(in_features=3584, out_features=512, bias=True) |
| (o_proj): Linear(in_features=3584, out_features=3584, bias=False) |
| ) |
| (mlp): Qwen2MLP( |
| (gate_proj): Linear(in_features=3584, out_features=18944, bias=False) |
| (up_proj): Linear(in_features=3584, out_features=18944, bias=False) |
| (down_proj): Linear(in_features=18944, out_features=3584, bias=False) |
| (act_fn): SiLU() |
| ) |
| (input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06) |
| (post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06) |
| ) |
| ) |
| (norm): Qwen2RMSNorm((3584,), eps=1e-06) |
| (rotary_emb): Qwen2RotaryEmbedding() |
| ) |
| (lm_head): Linear(in_features=3584, out_features=152064, bias=False) |
| ) |
| config: |
| Qwen2Config { |
| "architectures": [ |
| "Qwen2ForCausalLM" |
| ], |
| "attention_dropout": 0.0, |
| "bos_token_id": 151643, |
| "eos_token_id": 151643, |
| "hidden_act": "silu", |
| "hidden_size": 3584, |
| "initializer_range": 0.02, |
| "intermediate_size": 18944, |
| "layer_types": [ |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention" |
| ], |
| "max_position_embeddings": 131072, |
| "max_window_layers": 28, |
| "model_type": "qwen2", |
| "num_attention_heads": 28, |
| "num_hidden_layers": 28, |
| "num_key_value_heads": 4, |
| "rms_norm_eps": 1e-06, |
| "rope_scaling": null, |
| "rope_theta": 1000000.0, |
| "sliding_window": null, |
| "tie_word_embeddings": false, |
| "torch_dtype": "float16", |
| "transformers_version": "4.55.2", |
| "use_cache": true, |
| "use_mrope": false, |
| "use_sliding_window": false, |
| "vocab_size": 152064 |
| } |
|
|
| Processing layer 0--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 0 ---0.14184384047985077 |
| Processing layer 1--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 1 ---0.13445976376533508 |
| Processing layer 2--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 2 ---0.1448441445827484 |
| Processing layer 3--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 3 ---0.14360329508781433 |
| Processing layer 4--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 4 ---0.14149385690689087 |
| Processing layer 5--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 5 ---0.14219337701797485 |
| Processing layer 6--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 6 ---0.14448319375514984 |
| Processing layer 7--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 7 ---0.14329871535301208 |
| Processing layer 8--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 8 ---0.14073410630226135 |
| Processing layer 9--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 9 ---0.14020180702209473 |
| Processing layer 10--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 10 ---0.14346933364868164 |
| Processing layer 11--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 11 ---0.14193479716777802 |
| Processing layer 12--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 12 ---0.1403268575668335 |
| Processing layer 13--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 13 ---0.14010006189346313 |
| Processing layer 14--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 14 ---0.13355384767055511 |
| Processing layer 15--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 15 ---0.13806670904159546 |
| Processing layer 16--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 16 ---0.13962170481681824 |
| Processing layer 17--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 17 ---0.13812372088432312 |
| Processing layer 18--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 18 ---0.13966597616672516 |
| Processing layer 19--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 19 ---0.13903221487998962 |
| Processing layer 20--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 20 ---0.1419043242931366 |
| Processing layer 21--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 21 ---0.13662637770175934 |
| Processing layer 22--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 22 ---0.13463997840881348 |
| Processing layer 23--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 23 ---0.13744638860225677 |
| Processing layer 24--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 24 ---0.1426282525062561 |
| Processing layer 25--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 25 ---0.1387733370065689 |
| Processing layer 26--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 26 ---0.13506180047988892 |
| Processing layer 27--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 27 ---0.13298790156841278 |
| metric_name ZD: [2, 6, 3, 10, 7, 24, 5, 11, 20, 0, 4, 8, 12, 9, 13, 18, 16, 19, 25, 17, 15, 23, 21, 26, 22, 1, 14, 27] |
| Begin main_assign: Qwen2.5-7B self_attn head_diversity |
|
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:00<00:02, 1.18it/s]
Loading checkpoint shards: 50%|█████ | 2/4 [00:01<00:01, 1.12it/s]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:02<00:00, 1.11it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00, 1.18it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00, 1.16it/s] |
| Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation. |
| Once upon a time in the land of Mathoria, there was a magical forest where every tree had a unique number of leaves. The King of Mathoria decided to plant a new tree every day for a week (7 days), starting with 1 leaf on the first day and increasing the number of leaves by 1 each day. However, a mischievous sprite named Sprinkle loved to play tricks on the trees. On every even day, Sprinkle would randomly remove a number of leaves from the tree, between |
| Qwen2ForCausalLM( |
| (model): Qwen2Model( |
| (embed_tokens): Embedding(152064, 3584) |
| (layers): ModuleList( |
| (0-27): 28 x Qwen2DecoderLayer( |
| (self_attn): Qwen2Attention( |
| (q_proj): Linear(in_features=3584, out_features=3584, bias=True) |
| (k_proj): Linear(in_features=3584, out_features=512, bias=True) |
| (v_proj): Linear(in_features=3584, out_features=512, bias=True) |
| (o_proj): Linear(in_features=3584, out_features=3584, bias=False) |
| ) |
| (mlp): Qwen2MLP( |
| (gate_proj): Linear(in_features=3584, out_features=18944, bias=False) |
| (up_proj): Linear(in_features=3584, out_features=18944, bias=False) |
| (down_proj): Linear(in_features=18944, out_features=3584, bias=False) |
| (act_fn): SiLU() |
| ) |
| (input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06) |
| (post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06) |
| ) |
| ) |
| (norm): Qwen2RMSNorm((3584,), eps=1e-06) |
| (rotary_emb): Qwen2RotaryEmbedding() |
| ) |
| (lm_head): Linear(in_features=3584, out_features=152064, bias=False) |
| ) |
| config: |
| Qwen2Config { |
| "architectures": [ |
| "Qwen2ForCausalLM" |
| ], |
| "attention_dropout": 0.0, |
| "bos_token_id": 151643, |
| "eos_token_id": 151643, |
| "hidden_act": "silu", |
| "hidden_size": 3584, |
| "initializer_range": 0.02, |
| "intermediate_size": 18944, |
| "layer_types": [ |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention" |
| ], |
| "max_position_embeddings": 131072, |
| "max_window_layers": 28, |
| "model_type": "qwen2", |
| "num_attention_heads": 28, |
| "num_hidden_layers": 28, |
| "num_key_value_heads": 4, |
| "rms_norm_eps": 1e-06, |
| "rope_scaling": null, |
| "rope_theta": 1000000.0, |
| "sliding_window": null, |
| "tie_word_embeddings": false, |
| "torch_dtype": "float16", |
| "transformers_version": "4.55.2", |
| "use_cache": true, |
| "use_mrope": false, |
| "use_sliding_window": false, |
| "vocab_size": 152064 |
| } |
|
|
| Processing layer 0--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| Traceback (most recent call last): |
| File "/mnt/bn/life-mllm/users/cxr/quantization/quantization_metric/main_assign.py", line 58, in <module> |
| all_layer_alpha = calculate_expert(model, metric=metric_name, keyword=keyword) |
| File "/mnt/bn/life-mllm/users/cxr/quantization/quantization_metric/alphalora/expert_number.py", line 350, in calculate_expert |
| layer_final_alpha = func_call[metric](num_heads, subset) |
| File "/mnt/bn/life-mllm/users/cxr/quantization/quantization_metric/alphalora/expert_number.py", line 310, in head_diversity_asssist |
| ans.append(head_diversity(W, num_heads)) |
| File "/mnt/bn/life-mllm/users/cxr/quantization/quantization_metric/alphalora/expert_number.py", line 191, in head_diversity |
| w_heads = W.view(num_heads, head_dim, d_in) |
| RuntimeError: shape '[28, 18, 3584]' is invalid for input of size 1835008 |
| Begin main_assign: Qwen2.5-7B self_attn coherence |
|
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:00<00:02, 1.19it/s]
Loading checkpoint shards: 50%|█████ | 2/4 [00:01<00:01, 1.21it/s]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:02<00:00, 1.20it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00, 1.29it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00, 1.25it/s] |
| Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation. |
| Once upon a time in the not-so-distant past, the average person had a choice of two or three local phone companies and one long distance company. Now, you have to be an expert to navigate the maze of telephone companies and options. This article will help you make the right choice for your telephone needs. |
| If you are using a cellular phone, you should only use it in an emergency. It is important to use a cell phone only in emergencies because they use a lot of battery power. If you use a |
| Qwen2ForCausalLM( |
| (model): Qwen2Model( |
| (embed_tokens): Embedding(152064, 3584) |
| (layers): ModuleList( |
| (0-27): 28 x Qwen2DecoderLayer( |
| (self_attn): Qwen2Attention( |
| (q_proj): Linear(in_features=3584, out_features=3584, bias=True) |
| (k_proj): Linear(in_features=3584, out_features=512, bias=True) |
| (v_proj): Linear(in_features=3584, out_features=512, bias=True) |
| (o_proj): Linear(in_features=3584, out_features=3584, bias=False) |
| ) |
| (mlp): Qwen2MLP( |
| (gate_proj): Linear(in_features=3584, out_features=18944, bias=False) |
| (up_proj): Linear(in_features=3584, out_features=18944, bias=False) |
| (down_proj): Linear(in_features=18944, out_features=3584, bias=False) |
| (act_fn): SiLU() |
| ) |
| (input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06) |
| (post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06) |
| ) |
| ) |
| (norm): Qwen2RMSNorm((3584,), eps=1e-06) |
| (rotary_emb): Qwen2RotaryEmbedding() |
| ) |
| (lm_head): Linear(in_features=3584, out_features=152064, bias=False) |
| ) |
| config: |
| Qwen2Config { |
| "architectures": [ |
| "Qwen2ForCausalLM" |
| ], |
| "attention_dropout": 0.0, |
| "bos_token_id": 151643, |
| "eos_token_id": 151643, |
| "hidden_act": "silu", |
| "hidden_size": 3584, |
| "initializer_range": 0.02, |
| "intermediate_size": 18944, |
| "layer_types": [ |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention" |
| ], |
| "max_position_embeddings": 131072, |
| "max_window_layers": 28, |
| "model_type": "qwen2", |
| "num_attention_heads": 28, |
| "num_hidden_layers": 28, |
| "num_key_value_heads": 4, |
| "rms_norm_eps": 1e-06, |
| "rope_scaling": null, |
| "rope_theta": 1000000.0, |
| "sliding_window": null, |
| "tie_word_embeddings": false, |
| "torch_dtype": "float16", |
| "transformers_version": "4.55.2", |
| "use_cache": true, |
| "use_mrope": false, |
| "use_sliding_window": false, |
| "vocab_size": 152064 |
| } |
|
|
| Processing layer 0--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 0 ---0.019099362194538116 |
| Processing layer 1--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 1 ---0.03270909935235977 |
| Processing layer 2--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 2 ---0.02031659334897995 |
| Processing layer 3--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 3 ---0.020667918026447296 |
| Processing layer 4--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 4 ---0.021066918969154358 |
| Processing layer 5--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 5 ---0.019672438502311707 |
| Processing layer 6--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 6 ---0.020373258739709854 |
| Processing layer 7--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 7 ---0.01879500225186348 |
| Processing layer 8--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 8 ---0.018471794202923775 |
| Processing layer 9--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 9 ---0.01999843120574951 |
| Processing layer 10--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 10 ---0.018006717786192894 |
| Processing layer 11--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 11 ---0.019990842789411545 |
| Processing layer 12--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 12 ---0.01996159367263317 |
| Processing layer 13--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 13 ---0.020418085157871246 |
| Processing layer 14--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 14 ---0.020115870982408524 |
| Processing layer 15--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 15 ---0.02094407193362713 |
| Processing layer 16--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 16 ---0.02063441462814808 |
| Processing layer 17--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 17 ---0.018953558057546616 |
| Processing layer 18--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 18 ---0.020888380706310272 |
| Processing layer 19--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 19 ---0.01945885643362999 |
| Processing layer 20--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 20 ---0.019533313810825348 |
| Processing layer 21--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 21 ---0.018554046750068665 |
| Processing layer 22--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 22 ---0.020378313958644867 |
| Processing layer 23--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 23 ---0.019100410863757133 |
| Processing layer 24--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 24 ---0.018557211384177208 |
| Processing layer 25--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 25 ---0.019475571811199188 |
| Processing layer 26--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 26 ---0.021406373009085655 |
| Processing layer 27--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False)} |
| alpha value of layer 27 ---0.02655916102230549 |
| metric_name coherence: [1, 27, 26, 4, 15, 18, 3, 16, 13, 22, 6, 2, 14, 9, 11, 12, 5, 20, 25, 19, 23, 0, 17, 7, 24, 21, 8, 10] |
| Begin main_assign: Qwen2.5-7B mlp alpha |
|
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:00<00:02, 1.32it/s]
Loading checkpoint shards: 50%|█████ | 2/4 [00:01<00:01, 1.37it/s]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:02<00:00, 1.35it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00, 1.44it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00, 1.41it/s] |
| Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation. |
| Once upon a time there was a little girl named Maria. She was 10 years old and she loved to play with her dolls. Every night before she went to sleep she would place her dolls on her bed and have a tea party with them. One night, she was feeling very lonely. She wanted someone to talk to who could understand her. She wanted to be with her dolls, but she wanted a friend too. She prayed that God would send her a friend. She prayed that God would send her a |
| Qwen2ForCausalLM( |
| (model): Qwen2Model( |
| (embed_tokens): Embedding(152064, 3584) |
| (layers): ModuleList( |
| (0-27): 28 x Qwen2DecoderLayer( |
| (self_attn): Qwen2Attention( |
| (q_proj): Linear(in_features=3584, out_features=3584, bias=True) |
| (k_proj): Linear(in_features=3584, out_features=512, bias=True) |
| (v_proj): Linear(in_features=3584, out_features=512, bias=True) |
| (o_proj): Linear(in_features=3584, out_features=3584, bias=False) |
| ) |
| (mlp): Qwen2MLP( |
| (gate_proj): Linear(in_features=3584, out_features=18944, bias=False) |
| (up_proj): Linear(in_features=3584, out_features=18944, bias=False) |
| (down_proj): Linear(in_features=18944, out_features=3584, bias=False) |
| (act_fn): SiLU() |
| ) |
| (input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06) |
| (post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06) |
| ) |
| ) |
| (norm): Qwen2RMSNorm((3584,), eps=1e-06) |
| (rotary_emb): Qwen2RotaryEmbedding() |
| ) |
| (lm_head): Linear(in_features=3584, out_features=152064, bias=False) |
| ) |
| config: |
| Qwen2Config { |
| "architectures": [ |
| "Qwen2ForCausalLM" |
| ], |
| "attention_dropout": 0.0, |
| "bos_token_id": 151643, |
| "eos_token_id": 151643, |
| "hidden_act": "silu", |
| "hidden_size": 3584, |
| "initializer_range": 0.02, |
| "intermediate_size": 18944, |
| "layer_types": [ |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention" |
| ], |
| "max_position_embeddings": 131072, |
| "max_window_layers": 28, |
| "model_type": "qwen2", |
| "num_attention_heads": 28, |
| "num_hidden_layers": 28, |
| "num_key_value_heads": 4, |
| "rms_norm_eps": 1e-06, |
| "rope_scaling": null, |
| "rope_theta": 1000000.0, |
| "sliding_window": null, |
| "tie_word_embeddings": false, |
| "torch_dtype": "float16", |
| "transformers_version": "4.55.2", |
| "use_cache": true, |
| "use_mrope": false, |
| "use_sliding_window": false, |
| "vocab_size": 152064 |
| } |
|
|
| Processing layer 0--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 0 ---5.5507073402404785 |
| Processing layer 1--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 1 ---2.925907850265503 |
| Processing layer 2--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 2 ---3.427783727645874 |
| Processing layer 3--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 3 ---4.130731105804443 |
| Processing layer 4--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 4 ---4.2790751457214355 |
| Processing layer 5--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 5 ---4.67555570602417 |
| Processing layer 6--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 6 ---5.680507659912109 |
| Processing layer 7--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 7 ---5.469402313232422 |
| Processing layer 8--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 8 ---4.489261150360107 |
| Processing layer 9--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 9 ---5.958518981933594 |
| Processing layer 10--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 10 ---5.11647367477417 |
| Processing layer 11--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 11 ---4.431467056274414 |
| Processing layer 12--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 12 ---4.447659969329834 |
| Processing layer 13--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 13 ---4.224405288696289 |
| Processing layer 14--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 14 ---4.203671932220459 |
| Processing layer 15--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 15 ---4.193532943725586 |
| Processing layer 16--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 16 ---4.277862548828125 |
| Processing layer 17--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 17 ---4.189056396484375 |
| Processing layer 18--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 18 ---4.484411716461182 |
| Processing layer 19--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 19 ---4.689056396484375 |
| Processing layer 20--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 20 ---4.993287563323975 |
| Processing layer 21--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 21 ---6.104448318481445 |
| Processing layer 22--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 22 ---6.7987060546875 |
| Processing layer 23--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 23 ---6.16623067855835 |
| Processing layer 24--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 24 ---6.090585231781006 |
| Processing layer 25--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 25 ---5.552665710449219 |
| Processing layer 26--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 26 ---5.523178577423096 |
| Processing layer 27--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 27 ---4.90322208404541 |
| metric_name alpha: [22, 23, 21, 24, 9, 6, 25, 0, 26, 7, 10, 20, 27, 19, 5, 8, 18, 12, 11, 4, 16, 13, 14, 15, 17, 3, 2, 1] |
| Begin main_assign: Qwen2.5-7B mlp alpha_hat |
|
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:00<00:01, 1.57it/s]
Loading checkpoint shards: 50%|█████ | 2/4 [00:01<00:01, 1.65it/s]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:01<00:00, 1.64it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00, 1.71it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00, 1.68it/s] |
| Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation. |
| Once upon a time, a little boy named Timmy went to visit his grandmother. On the way, he saw a beautiful rainbow in the sky. He wanted to find the pot of gold at the end of the rainbow. But the rainbow led him to a magical door that was locked with a puzzle. |
| The puzzle was: "I am not alive, but I grow; I don't have lungs, but I need air; I don't have a mouth, but water kills me. What am I?" |
|
|
| Qwen2ForCausalLM( |
| (model): Qwen2Model( |
| (embed_tokens): Embedding(152064, 3584) |
| (layers): ModuleList( |
| (0-27): 28 x Qwen2DecoderLayer( |
| (self_attn): Qwen2Attention( |
| (q_proj): Linear(in_features=3584, out_features=3584, bias=True) |
| (k_proj): Linear(in_features=3584, out_features=512, bias=True) |
| (v_proj): Linear(in_features=3584, out_features=512, bias=True) |
| (o_proj): Linear(in_features=3584, out_features=3584, bias=False) |
| ) |
| (mlp): Qwen2MLP( |
| (gate_proj): Linear(in_features=3584, out_features=18944, bias=False) |
| (up_proj): Linear(in_features=3584, out_features=18944, bias=False) |
| (down_proj): Linear(in_features=18944, out_features=3584, bias=False) |
| (act_fn): SiLU() |
| ) |
| (input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06) |
| (post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06) |
| ) |
| ) |
| (norm): Qwen2RMSNorm((3584,), eps=1e-06) |
| (rotary_emb): Qwen2RotaryEmbedding() |
| ) |
| (lm_head): Linear(in_features=3584, out_features=152064, bias=False) |
| ) |
| config: |
| Qwen2Config { |
| "architectures": [ |
| "Qwen2ForCausalLM" |
| ], |
| "attention_dropout": 0.0, |
| "bos_token_id": 151643, |
| "eos_token_id": 151643, |
| "hidden_act": "silu", |
| "hidden_size": 3584, |
| "initializer_range": 0.02, |
| "intermediate_size": 18944, |
| "layer_types": [ |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention" |
| ], |
| "max_position_embeddings": 131072, |
| "max_window_layers": 28, |
| "model_type": "qwen2", |
| "num_attention_heads": 28, |
| "num_hidden_layers": 28, |
| "num_key_value_heads": 4, |
| "rms_norm_eps": 1e-06, |
| "rope_scaling": null, |
| "rope_theta": 1000000.0, |
| "sliding_window": null, |
| "tie_word_embeddings": false, |
| "torch_dtype": "float16", |
| "transformers_version": "4.55.2", |
| "use_cache": true, |
| "use_mrope": false, |
| "use_sliding_window": false, |
| "vocab_size": 152064 |
| } |
|
|
| Processing layer 0--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 0 ---26.469465255737305 |
| Processing layer 1--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 1 ---15.14222526550293 |
| Processing layer 2--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 2 ---15.627280235290527 |
| Processing layer 3--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 3 ---21.562671661376953 |
| Processing layer 4--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 4 ---18.97063636779785 |
| Processing layer 5--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 5 ---21.275915145874023 |
| Processing layer 6--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 6 ---20.404766082763672 |
| Processing layer 7--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 7 ---21.564342498779297 |
| Processing layer 8--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 8 ---17.832380294799805 |
| Processing layer 9--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 9 ---24.982494354248047 |
| Processing layer 10--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 10 ---21.225040435791016 |
| Processing layer 11--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 11 ---18.6888484954834 |
| Processing layer 12--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 12 ---19.226669311523438 |
| Processing layer 13--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 13 ---18.279586791992188 |
| Processing layer 14--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 14 ---17.538372039794922 |
| Processing layer 15--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 15 ---17.4776611328125 |
| Processing layer 16--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 16 ---18.033130645751953 |
| Processing layer 17--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 17 ---17.052593231201172 |
| Processing layer 18--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 18 ---18.05915641784668 |
| Processing layer 19--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 19 ---18.863903045654297 |
| Processing layer 20--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 20 ---19.91156768798828 |
| Processing layer 21--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 21 ---22.948781967163086 |
| Processing layer 22--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 22 ---26.310537338256836 |
| Processing layer 23--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 23 ---24.970985412597656 |
| Processing layer 24--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 24 ---24.232511520385742 |
| Processing layer 25--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 25 ---23.717098236083984 |
| Processing layer 26--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 26 ---23.539913177490234 |
| Processing layer 27--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 27 ---24.706113815307617 |
| metric_name alpha_hat: [0, 22, 9, 23, 27, 24, 25, 26, 21, 7, 3, 5, 10, 6, 20, 12, 4, 19, 11, 13, 18, 16, 8, 14, 15, 17, 2, 1] |
| Begin main_assign: Qwen2.5-7B mlp stable_rank |
|
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:00<00:02, 1.23it/s]
Loading checkpoint shards: 50%|█████ | 2/4 [00:01<00:01, 1.28it/s]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:02<00:00, 1.28it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00, 1.35it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00, 1.32it/s] |
| Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation. |
| Once upon a time， there was a little girl． She was very beautiful， but she was so bad that nobody liked her． She had no friends． She didn't want to play with other children． She didn't want to go to school． She lived by herself in a small house． The only thing that made her happy was a little dog． The little girl was very sad． She didn't know what to do． One day， she walked out of the house and saw a beautiful bird． She wanted to |
| Qwen2ForCausalLM( |
| (model): Qwen2Model( |
| (embed_tokens): Embedding(152064, 3584) |
| (layers): ModuleList( |
| (0-27): 28 x Qwen2DecoderLayer( |
| (self_attn): Qwen2Attention( |
| (q_proj): Linear(in_features=3584, out_features=3584, bias=True) |
| (k_proj): Linear(in_features=3584, out_features=512, bias=True) |
| (v_proj): Linear(in_features=3584, out_features=512, bias=True) |
| (o_proj): Linear(in_features=3584, out_features=3584, bias=False) |
| ) |
| (mlp): Qwen2MLP( |
| (gate_proj): Linear(in_features=3584, out_features=18944, bias=False) |
| (up_proj): Linear(in_features=3584, out_features=18944, bias=False) |
| (down_proj): Linear(in_features=18944, out_features=3584, bias=False) |
| (act_fn): SiLU() |
| ) |
| (input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06) |
| (post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06) |
| ) |
| ) |
| (norm): Qwen2RMSNorm((3584,), eps=1e-06) |
| (rotary_emb): Qwen2RotaryEmbedding() |
| ) |
| (lm_head): Linear(in_features=3584, out_features=152064, bias=False) |
| ) |
| config: |
| Qwen2Config { |
| "architectures": [ |
| "Qwen2ForCausalLM" |
| ], |
| "attention_dropout": 0.0, |
| "bos_token_id": 151643, |
| "eos_token_id": 151643, |
| "hidden_act": "silu", |
| "hidden_size": 3584, |
| "initializer_range": 0.02, |
| "intermediate_size": 18944, |
| "layer_types": [ |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention" |
| ], |
| "max_position_embeddings": 131072, |
| "max_window_layers": 28, |
| "model_type": "qwen2", |
| "num_attention_heads": 28, |
| "num_hidden_layers": 28, |
| "num_key_value_heads": 4, |
| "rms_norm_eps": 1e-06, |
| "rope_scaling": null, |
| "rope_theta": 1000000.0, |
| "sliding_window": null, |
| "tie_word_embeddings": false, |
| "torch_dtype": "float16", |
| "transformers_version": "4.55.2", |
| "use_cache": true, |
| "use_mrope": false, |
| "use_sliding_window": false, |
| "vocab_size": 152064 |
| } |
|
|
| Processing layer 0--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| frobenius_norm tensor(126.6843, device='cuda:0') |
| spectral_norm tensor(37.7735, device='cuda:0') |
| frobenius_norm tensor(108.8807, device='cuda:0') |
| spectral_norm tensor(7.8392, device='cuda:0') |
| frobenius_norm tensor(115.3682, device='cuda:0') |
| spectral_norm tensor(6.5322, device='cuda:0') |
| alpha value of layer 0 ---172.02789306640625 |
| Processing layer 1--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| frobenius_norm tensor(123.2531, device='cuda:0') |
| spectral_norm tensor(20.7839, device='cuda:0') |
| frobenius_norm tensor(101.4097, device='cuda:0') |
| spectral_norm tensor(8.8381, device='cuda:0') |
| frobenius_norm tensor(101.5777, device='cuda:0') |
| spectral_norm tensor(13.5321, device='cuda:0') |
| alpha value of layer 1 ---74.3896484375 |
| Processing layer 2--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| frobenius_norm tensor(138.8778, device='cuda:0') |
| spectral_norm tensor(23.4270, device='cuda:0') |
| frobenius_norm tensor(112.7775, device='cuda:0') |
| spectral_norm tensor(6.5452, device='cuda:0') |
| frobenius_norm tensor(115.6860, device='cuda:0') |
| spectral_norm tensor(7.9797, device='cuda:0') |
| alpha value of layer 2 ---180.73681640625 |
| Processing layer 3--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| frobenius_norm tensor(153.6270, device='cuda:0') |
| spectral_norm tensor(22.2753, device='cuda:0') |
| frobenius_norm tensor(132.3810, device='cuda:0') |
| spectral_norm tensor(7.2662, device='cuda:0') |
| frobenius_norm tensor(131.2425, device='cuda:0') |
| spectral_norm tensor(17.0355, device='cuda:0') |
| alpha value of layer 3 ---146.2799072265625 |
| Processing layer 4--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| frobenius_norm tensor(156.7403, device='cuda:0') |
| spectral_norm tensor(22.8238, device='cuda:0') |
| frobenius_norm tensor(129.6389, device='cuda:0') |
| spectral_norm tensor(5.4833, device='cuda:0') |
| frobenius_norm tensor(128.7504, device='cuda:0') |
| spectral_norm tensor(8.4121, device='cuda:0') |
| alpha value of layer 4 ---280.12554931640625 |
| Processing layer 5--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| frobenius_norm tensor(149.4161, device='cuda:0') |
| spectral_norm tensor(21.1506, device='cuda:0') |
| frobenius_norm tensor(133.9768, device='cuda:0') |
| spectral_norm tensor(6.2399, device='cuda:0') |
| frobenius_norm tensor(132.0446, device='cuda:0') |
| spectral_norm tensor(8.4305, device='cuda:0') |
| alpha value of layer 5 ---252.07894897460938 |
| Processing layer 6--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| frobenius_norm tensor(151.2180, device='cuda:0') |
| spectral_norm tensor(14.3238, device='cuda:0') |
| frobenius_norm tensor(129.6978, device='cuda:0') |
| spectral_norm tensor(4.1238, device='cuda:0') |
| frobenius_norm tensor(127.6298, device='cuda:0') |
| spectral_norm tensor(7.3585, device='cuda:0') |
| alpha value of layer 6 ---467.14599609375 |
| Processing layer 7--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| frobenius_norm tensor(141.6723, device='cuda:0') |
| spectral_norm tensor(13.0737, device='cuda:0') |
| frobenius_norm tensor(133.2031, device='cuda:0') |
| spectral_norm tensor(5.1650, device='cuda:0') |
| frobenius_norm tensor(132.6786, device='cuda:0') |
| spectral_norm tensor(7.6574, device='cuda:0') |
| alpha value of layer 7 ---360.915771484375 |
| Processing layer 8--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| frobenius_norm tensor(139.1350, device='cuda:0') |
| spectral_norm tensor(12.2201, device='cuda:0') |
| frobenius_norm tensor(135.6841, device='cuda:0') |
| spectral_norm tensor(4.7790, device='cuda:0') |
| frobenius_norm tensor(134.1169, device='cuda:0') |
| spectral_norm tensor(8.0838, device='cuda:0') |
| alpha value of layer 8 ---403.66400146484375 |
| Processing layer 9--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| frobenius_norm tensor(152.4723, device='cuda:0') |
| spectral_norm tensor(28.9447, device='cuda:0') |
| frobenius_norm tensor(124.9508, device='cuda:0') |
| spectral_norm tensor(5.0401, device='cuda:0') |
| frobenius_norm tensor(123.2868, device='cuda:0') |
| spectral_norm tensor(7.9764, device='cuda:0') |
| alpha value of layer 9 ---293.7569580078125 |
| Processing layer 10--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| frobenius_norm tensor(141.8071, device='cuda:0') |
| spectral_norm tensor(14.3111, device='cuda:0') |
| frobenius_norm tensor(133.1904, device='cuda:0') |
| spectral_norm tensor(5.2121, device='cuda:0') |
| frobenius_norm tensor(132.4129, device='cuda:0') |
| spectral_norm tensor(9.0536, device='cuda:0') |
| alpha value of layer 10 ---321.69720458984375 |
| Processing layer 11--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| frobenius_norm tensor(139.0032, device='cuda:0') |
| spectral_norm tensor(12.7478, device='cuda:0') |
| frobenius_norm tensor(135.5063, device='cuda:0') |
| spectral_norm tensor(5.4040, device='cuda:0') |
| frobenius_norm tensor(134.1321, device='cuda:0') |
| spectral_norm tensor(9.8842, device='cuda:0') |
| alpha value of layer 11 ---310.609130859375 |
| Processing layer 12--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| frobenius_norm tensor(136.5591, device='cuda:0') |
| spectral_norm tensor(12.8626, device='cuda:0') |
| frobenius_norm tensor(136.9887, device='cuda:0') |
| spectral_norm tensor(5.2525, device='cuda:0') |
| frobenius_norm tensor(135.8750, device='cuda:0') |
| spectral_norm tensor(10.9823, device='cuda:0') |
| alpha value of layer 12 ---315.332763671875 |
| Processing layer 13--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| frobenius_norm tensor(139.4029, device='cuda:0') |
| spectral_norm tensor(12.9032, device='cuda:0') |
| frobenius_norm tensor(135.3871, device='cuda:0') |
| spectral_norm tensor(5.2956, device='cuda:0') |
| frobenius_norm tensor(133.7299, device='cuda:0') |
| spectral_norm tensor(10.5605, device='cuda:0') |
| alpha value of layer 13 ---310.2343444824219 |
| Processing layer 14--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| frobenius_norm tensor(136.9790, device='cuda:0') |
| spectral_norm tensor(12.1678, device='cuda:0') |
| frobenius_norm tensor(136.4235, device='cuda:0') |
| spectral_norm tensor(5.1185, device='cuda:0') |
| frobenius_norm tensor(134.9485, device='cuda:0') |
| spectral_norm tensor(9.6906, device='cuda:0') |
| alpha value of layer 14 ---343.6784362792969 |
| Processing layer 15--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| frobenius_norm tensor(135.3356, device='cuda:0') |
| spectral_norm tensor(11.3039, device='cuda:0') |
| frobenius_norm tensor(137.9369, device='cuda:0') |
| spectral_norm tensor(5.1105, device='cuda:0') |
| frobenius_norm tensor(135.9716, device='cuda:0') |
| spectral_norm tensor(10.2615, device='cuda:0') |
| alpha value of layer 15 ---349.14691162109375 |
| Processing layer 16--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| frobenius_norm tensor(135.1444, device='cuda:0') |
| spectral_norm tensor(11.1679, device='cuda:0') |
| frobenius_norm tensor(137.6841, device='cuda:0') |
| spectral_norm tensor(5.1499, device='cuda:0') |
| frobenius_norm tensor(135.1051, device='cuda:0') |
| spectral_norm tensor(10.5823, device='cuda:0') |
| alpha value of layer 16 ---341.401611328125 |
| Processing layer 17--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| frobenius_norm tensor(133.7680, device='cuda:0') |
| spectral_norm tensor(11.1640, device='cuda:0') |
| frobenius_norm tensor(138.4800, device='cuda:0') |
| spectral_norm tensor(5.2358, device='cuda:0') |
| frobenius_norm tensor(135.3299, device='cuda:0') |
| spectral_norm tensor(8.6134, device='cuda:0') |
| alpha value of layer 17 ---363.3155212402344 |
| Processing layer 18--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| frobenius_norm tensor(133.3184, device='cuda:0') |
| spectral_norm tensor(10.9756, device='cuda:0') |
| frobenius_norm tensor(140.8582, device='cuda:0') |
| spectral_norm tensor(5.6010, device='cuda:0') |
| frobenius_norm tensor(137.9481, device='cuda:0') |
| spectral_norm tensor(7.5009, device='cuda:0') |
| alpha value of layer 18 ---372.74267578125 |
| Processing layer 19--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| frobenius_norm tensor(135.1317, device='cuda:0') |
| spectral_norm tensor(12.0810, device='cuda:0') |
| frobenius_norm tensor(140.5321, device='cuda:0') |
| spectral_norm tensor(5.5793, device='cuda:0') |
| frobenius_norm tensor(137.1833, device='cuda:0') |
| spectral_norm tensor(6.9554, device='cuda:0') |
| alpha value of layer 19 ---382.8515625 |
| Processing layer 20--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| frobenius_norm tensor(134.6000, device='cuda:0') |
| spectral_norm tensor(11.1297, device='cuda:0') |
| frobenius_norm tensor(141.2537, device='cuda:0') |
| spectral_norm tensor(5.9169, device='cuda:0') |
| frobenius_norm tensor(138.0343, device='cuda:0') |
| spectral_norm tensor(6.5506, device='cuda:0') |
| alpha value of layer 20 ---386.7352294921875 |
| Processing layer 21--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| frobenius_norm tensor(136.7462, device='cuda:0') |
| spectral_norm tensor(11.6961, device='cuda:0') |
| frobenius_norm tensor(141.2342, device='cuda:0') |
| spectral_norm tensor(5.6582, device='cuda:0') |
| frobenius_norm tensor(137.9897, device='cuda:0') |
| spectral_norm tensor(5.3100, device='cuda:0') |
| alpha value of layer 21 ---478.354248046875 |
| Processing layer 22--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| frobenius_norm tensor(136.9238, device='cuda:0') |
| spectral_norm tensor(12.4543, device='cuda:0') |
| frobenius_norm tensor(141.9774, device='cuda:0') |
| spectral_norm tensor(6.5433, device='cuda:0') |
| frobenius_norm tensor(139.0448, device='cuda:0') |
| spectral_norm tensor(5.1260, device='cuda:0') |
| alpha value of layer 22 ---442.48614501953125 |
| Processing layer 23--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| frobenius_norm tensor(138.1573, device='cuda:0') |
| spectral_norm tensor(14.3250, device='cuda:0') |
| frobenius_norm tensor(141.6219, device='cuda:0') |
| spectral_norm tensor(6.6344, device='cuda:0') |
| frobenius_norm tensor(138.9087, device='cuda:0') |
| spectral_norm tensor(5.2948, device='cuda:0') |
| alpha value of layer 23 ---412.3252868652344 |
| Processing layer 24--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| frobenius_norm tensor(135.7100, device='cuda:0') |
| spectral_norm tensor(12.7437, device='cuda:0') |
| frobenius_norm tensor(143.0019, device='cuda:0') |
| spectral_norm tensor(6.6948, device='cuda:0') |
| frobenius_norm tensor(141.1184, device='cuda:0') |
| spectral_norm tensor(5.4309, device='cuda:0') |
| alpha value of layer 24 ---414.9455261230469 |
| Processing layer 25--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| frobenius_norm tensor(134.2979, device='cuda:0') |
| spectral_norm tensor(12.3842, device='cuda:0') |
| frobenius_norm tensor(144.5099, device='cuda:0') |
| spectral_norm tensor(7.7998, device='cuda:0') |
| frobenius_norm tensor(143.8913, device='cuda:0') |
| spectral_norm tensor(6.7086, device='cuda:0') |
| alpha value of layer 25 ---306.970947265625 |
| Processing layer 26--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| frobenius_norm tensor(134.3241, device='cuda:0') |
| spectral_norm tensor(10.3940, device='cuda:0') |
| frobenius_norm tensor(145.9831, device='cuda:0') |
| spectral_norm tensor(10.3400, device='cuda:0') |
| frobenius_norm tensor(143.8145, device='cuda:0') |
| spectral_norm tensor(5.9343, device='cuda:0') |
| alpha value of layer 26 ---317.8813781738281 |
| Processing layer 27--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| frobenius_norm tensor(139.7623, device='cuda:0') |
| spectral_norm tensor(14.5234, device='cuda:0') |
| frobenius_norm tensor(145.0968, device='cuda:0') |
| spectral_norm tensor(16.8470, device='cuda:0') |
| frobenius_norm tensor(133.4643, device='cuda:0') |
| spectral_norm tensor(7.9846, device='cuda:0') |
| alpha value of layer 27 ---148.7277069091797 |
| metric_name stable_rank: [21, 6, 22, 24, 23, 8, 20, 19, 18, 17, 7, 15, 14, 16, 10, 26, 12, 11, 13, 25, 9, 4, 5, 2, 0, 27, 3, 1] |
| Begin main_assign: Qwen2.5-7B mlp effective_rank |
|
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:00<00:02, 1.01it/s]
Loading checkpoint shards: 50%|█████ | 2/4 [00:01<00:01, 1.03it/s]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:02<00:00, 1.02it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00, 1.09it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00, 1.06it/s] |
| Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation. |
| Once upon a time, there was a queen who wanted to build a castle. She had a team of builders who would work day and night to complete the castle. She wanted the castle to be the most magnificent one in the kingdom. The queen would often visit the builders to see the progress of the castle. She would give them advice on how to make it even better. The builders worked hard and the castle was built in a short time. The queen was very happy with the castle and she made it the official residence |
| Qwen2ForCausalLM( |
| (model): Qwen2Model( |
| (embed_tokens): Embedding(152064, 3584) |
| (layers): ModuleList( |
| (0-27): 28 x Qwen2DecoderLayer( |
| (self_attn): Qwen2Attention( |
| (q_proj): Linear(in_features=3584, out_features=3584, bias=True) |
| (k_proj): Linear(in_features=3584, out_features=512, bias=True) |
| (v_proj): Linear(in_features=3584, out_features=512, bias=True) |
| (o_proj): Linear(in_features=3584, out_features=3584, bias=False) |
| ) |
| (mlp): Qwen2MLP( |
| (gate_proj): Linear(in_features=3584, out_features=18944, bias=False) |
| (up_proj): Linear(in_features=3584, out_features=18944, bias=False) |
| (down_proj): Linear(in_features=18944, out_features=3584, bias=False) |
| (act_fn): SiLU() |
| ) |
| (input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06) |
| (post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06) |
| ) |
| ) |
| (norm): Qwen2RMSNorm((3584,), eps=1e-06) |
| (rotary_emb): Qwen2RotaryEmbedding() |
| ) |
| (lm_head): Linear(in_features=3584, out_features=152064, bias=False) |
| ) |
| config: |
| Qwen2Config { |
| "architectures": [ |
| "Qwen2ForCausalLM" |
| ], |
| "attention_dropout": 0.0, |
| "bos_token_id": 151643, |
| "eos_token_id": 151643, |
| "hidden_act": "silu", |
| "hidden_size": 3584, |
| "initializer_range": 0.02, |
| "intermediate_size": 18944, |
| "layer_types": [ |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention" |
| ], |
| "max_position_embeddings": 131072, |
| "max_window_layers": 28, |
| "model_type": "qwen2", |
| "num_attention_heads": 28, |
| "num_hidden_layers": 28, |
| "num_key_value_heads": 4, |
| "rms_norm_eps": 1e-06, |
| "rope_scaling": null, |
| "rope_theta": 1000000.0, |
| "sliding_window": null, |
| "tie_word_embeddings": false, |
| "torch_dtype": "float16", |
| "transformers_version": "4.55.2", |
| "use_cache": true, |
| "use_mrope": false, |
| "use_sliding_window": false, |
| "vocab_size": 152064 |
| } |
|
|
| Processing layer 0--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 0 ---2950.251953125 |
| Processing layer 1--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 1 ---3178.52880859375 |
| Processing layer 2--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 2 ---3286.484375 |
| Processing layer 3--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 3 ---3382.06787109375 |
| Processing layer 4--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 4 ---3406.15234375 |
| Processing layer 5--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 5 ---3428.05908203125 |
| Processing layer 6--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 6 ---3425.3720703125 |
| Processing layer 7--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 7 ---3438.742919921875 |
| Processing layer 8--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 8 ---3428.871826171875 |
| Processing layer 9--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 9 ---3411.824951171875 |
| Processing layer 10--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 10 ---3431.89013671875 |
| Processing layer 11--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 11 ---3419.459716796875 |
| Processing layer 12--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 12 ---3425.357177734375 |
| Processing layer 13--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 13 ---3405.046875 |
| Processing layer 14--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 14 ---3417.287841796875 |
| Processing layer 15--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 15 ---3418.02490234375 |
| Processing layer 16--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 16 ---3407.5625 |
| Processing layer 17--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 17 ---3408.4970703125 |
| Processing layer 18--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 18 ---3426.13232421875 |
| Processing layer 19--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 19 ---3422.40576171875 |
| Processing layer 20--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 20 ---3435.60009765625 |
| Processing layer 21--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 21 ---3436.99267578125 |
| Processing layer 22--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 22 ---3444.177734375 |
| Processing layer 23--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 23 ---3438.670654296875 |
| Processing layer 24--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 24 ---3432.22265625 |
| Processing layer 25--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 25 ---3429.8212890625 |
| Processing layer 26--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 26 ---3432.60546875 |
| Processing layer 27--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 27 ---3447.41943359375 |
| metric_name effective_rank: [27, 22, 7, 23, 21, 20, 26, 24, 10, 25, 8, 5, 18, 6, 12, 19, 11, 15, 14, 9, 17, 16, 4, 13, 3, 2, 1, 0] |
| Begin main_assign: Qwen2.5-7B mlp ZD |
|
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:00<00:02, 1.29it/s]
Loading checkpoint shards: 50%|█████ | 2/4 [00:01<00:01, 1.33it/s]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:02<00:00, 1.31it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00, 1.41it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00, 1.37it/s] |
| Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation. |
| Once upon a time there lived a rich man. He had a servant（仆人）. He and the servant loved wine and good food very much. Each time the rich man left his home, the servant would drink the wine and eat up all the nice food in the house. The rich man knew what his servant did, but he had never caught his servant doing that. One morning, when he left home, he said to the servant, “Here are two bottles of poison（毒药） and some nice |
| Qwen2ForCausalLM( |
| (model): Qwen2Model( |
| (embed_tokens): Embedding(152064, 3584) |
| (layers): ModuleList( |
| (0-27): 28 x Qwen2DecoderLayer( |
| (self_attn): Qwen2Attention( |
| (q_proj): Linear(in_features=3584, out_features=3584, bias=True) |
| (k_proj): Linear(in_features=3584, out_features=512, bias=True) |
| (v_proj): Linear(in_features=3584, out_features=512, bias=True) |
| (o_proj): Linear(in_features=3584, out_features=3584, bias=False) |
| ) |
| (mlp): Qwen2MLP( |
| (gate_proj): Linear(in_features=3584, out_features=18944, bias=False) |
| (up_proj): Linear(in_features=3584, out_features=18944, bias=False) |
| (down_proj): Linear(in_features=18944, out_features=3584, bias=False) |
| (act_fn): SiLU() |
| ) |
| (input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06) |
| (post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06) |
| ) |
| ) |
| (norm): Qwen2RMSNorm((3584,), eps=1e-06) |
| (rotary_emb): Qwen2RotaryEmbedding() |
| ) |
| (lm_head): Linear(in_features=3584, out_features=152064, bias=False) |
| ) |
| config: |
| Qwen2Config { |
| "architectures": [ |
| "Qwen2ForCausalLM" |
| ], |
| "attention_dropout": 0.0, |
| "bos_token_id": 151643, |
| "eos_token_id": 151643, |
| "hidden_act": "silu", |
| "hidden_size": 3584, |
| "initializer_range": 0.02, |
| "intermediate_size": 18944, |
| "layer_types": [ |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention" |
| ], |
| "max_position_embeddings": 131072, |
| "max_window_layers": 28, |
| "model_type": "qwen2", |
| "num_attention_heads": 28, |
| "num_hidden_layers": 28, |
| "num_key_value_heads": 4, |
| "rms_norm_eps": 1e-06, |
| "rope_scaling": null, |
| "rope_theta": 1000000.0, |
| "sliding_window": null, |
| "tie_word_embeddings": false, |
| "torch_dtype": "float16", |
| "transformers_version": "4.55.2", |
| "use_cache": true, |
| "use_mrope": false, |
| "use_sliding_window": false, |
| "vocab_size": 152064 |
| } |
|
|
| Processing layer 0--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 0 ---0.14041712880134583 |
| Processing layer 1--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 1 ---0.12673243880271912 |
| Processing layer 2--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 2 ---0.14196857810020447 |
| Processing layer 3--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 3 ---0.15253740549087524 |
| Processing layer 4--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 4 ---0.15330302715301514 |
| Processing layer 5--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 5 ---0.1531524807214737 |
| Processing layer 6--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 6 ---0.15197408199310303 |
| Processing layer 7--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 7 ---0.15272140502929688 |
| Processing layer 8--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 8 ---0.15372541546821594 |
| Processing layer 9--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 9 ---0.15197794139385223 |
| Processing layer 10--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 10 ---0.15327925980091095 |
| Processing layer 11--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 11 ---0.15320764482021332 |
| Processing layer 12--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 12 ---0.15106868743896484 |
| Processing layer 13--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 13 ---0.15193961560726166 |
| Processing layer 14--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 14 ---0.14989005029201508 |
| Processing layer 15--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 15 ---0.15030330419540405 |
| Processing layer 16--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 16 ---0.1516457051038742 |
| Processing layer 17--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 17 ---0.15101006627082825 |
| Processing layer 18--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 18 ---0.14948342740535736 |
| Processing layer 19--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 19 ---0.15111291408538818 |
| Processing layer 20--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 20 ---0.15065661072731018 |
| Processing layer 21--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 21 ---0.15124627947807312 |
| Processing layer 22--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 22 ---0.1522490233182907 |
| Processing layer 23--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 23 ---0.15400244295597076 |
| Processing layer 24--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 24 ---0.15468579530715942 |
| Processing layer 25--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 25 ---0.15433579683303833 |
| Processing layer 26--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 26 ---0.1542406529188156 |
| Processing layer 27--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 27 ---0.1527990698814392 |
| metric_name ZD: [24, 25, 26, 23, 8, 4, 10, 11, 5, 27, 7, 3, 22, 9, 6, 13, 16, 21, 19, 12, 17, 20, 15, 14, 18, 2, 0, 1] |
| Begin main_assign: Qwen2.5-7B mlp head_diversity |
|
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:01<00:04, 1.43s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:02<00:02, 1.24s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:03<00:01, 1.28s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:04<00:00, 1.19s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:04<00:00, 1.23s/it] |
| Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation. |
| Once upon a time, there was a little boy named Timmy. Timmy loved to play outside and explore the world around him. One day, while playing in the park, he met a talking tree named Oakley. |
| Oakley told Timmy that there was a magical forest nearby that only appeared once every hundred years. The forest was filled with talking animals and magical creatures that could grant wishes. Timmy was excited to hear this news and begged Oakley to take him there. |
| Oakley agreed and led Timmy |
| Qwen2ForCausalLM( |
| (model): Qwen2Model( |
| (embed_tokens): Embedding(152064, 3584) |
| (layers): ModuleList( |
| (0-27): 28 x Qwen2DecoderLayer( |
| (self_attn): Qwen2Attention( |
| (q_proj): Linear(in_features=3584, out_features=3584, bias=True) |
| (k_proj): Linear(in_features=3584, out_features=512, bias=True) |
| (v_proj): Linear(in_features=3584, out_features=512, bias=True) |
| (o_proj): Linear(in_features=3584, out_features=3584, bias=False) |
| ) |
| (mlp): Qwen2MLP( |
| (gate_proj): Linear(in_features=3584, out_features=18944, bias=False) |
| (up_proj): Linear(in_features=3584, out_features=18944, bias=False) |
| (down_proj): Linear(in_features=18944, out_features=3584, bias=False) |
| (act_fn): SiLU() |
| ) |
| (input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06) |
| (post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06) |
| ) |
| ) |
| (norm): Qwen2RMSNorm((3584,), eps=1e-06) |
| (rotary_emb): Qwen2RotaryEmbedding() |
| ) |
| (lm_head): Linear(in_features=3584, out_features=152064, bias=False) |
| ) |
| config: |
| Qwen2Config { |
| "architectures": [ |
| "Qwen2ForCausalLM" |
| ], |
| "attention_dropout": 0.0, |
| "bos_token_id": 151643, |
| "eos_token_id": 151643, |
| "hidden_act": "silu", |
| "hidden_size": 3584, |
| "initializer_range": 0.02, |
| "intermediate_size": 18944, |
| "layer_types": [ |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention" |
| ], |
| "max_position_embeddings": 131072, |
| "max_window_layers": 28, |
| "model_type": "qwen2", |
| "num_attention_heads": 28, |
| "num_hidden_layers": 28, |
| "num_key_value_heads": 4, |
| "rms_norm_eps": 1e-06, |
| "rope_scaling": null, |
| "rope_theta": 1000000.0, |
| "sliding_window": null, |
| "tie_word_embeddings": false, |
| "torch_dtype": "float16", |
| "transformers_version": "4.55.2", |
| "use_cache": true, |
| "use_mrope": false, |
| "use_sliding_window": false, |
| "vocab_size": 152064 |
| } |
|
|
| Processing layer 0--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| Traceback (most recent call last): |
| File "/mnt/bn/life-mllm/users/cxr/quantization/quantization_metric/main_assign.py", line 58, in <module> |
| all_layer_alpha = calculate_expert(model, metric=metric_name, keyword=keyword) |
| File "/mnt/bn/life-mllm/users/cxr/quantization/quantization_metric/alphalora/expert_number.py", line 354, in calculate_expert |
| all_layer_alpha.append(torch.stack(layer_final_alpha).mean().item()) |
| RuntimeError: stack expects a non-empty TensorList |
| Begin main_assign: Qwen2.5-7B mlp coherence |
|
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:01<00:03, 1.01s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:01<00:01, 1.12it/s]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:02<00:00, 1.15it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00, 1.23it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00, 1.18it/s] |
| Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation. |
| Once upon a time, the land of Greece was ruled by three powerful and evil sorceresses: Echidna, the mother of the monsters; her daughter, the monster Medusa; and the sea-goddess Gorgon. Their leader was the most evil of the three, the monster Medusa. When the gods tried to stop her, she became so angry that she attacked them and killed them all. She then took their weapons, which she used to attack the world. She even attacked the other |
| Qwen2ForCausalLM( |
| (model): Qwen2Model( |
| (embed_tokens): Embedding(152064, 3584) |
| (layers): ModuleList( |
| (0-27): 28 x Qwen2DecoderLayer( |
| (self_attn): Qwen2Attention( |
| (q_proj): Linear(in_features=3584, out_features=3584, bias=True) |
| (k_proj): Linear(in_features=3584, out_features=512, bias=True) |
| (v_proj): Linear(in_features=3584, out_features=512, bias=True) |
| (o_proj): Linear(in_features=3584, out_features=3584, bias=False) |
| ) |
| (mlp): Qwen2MLP( |
| (gate_proj): Linear(in_features=3584, out_features=18944, bias=False) |
| (up_proj): Linear(in_features=3584, out_features=18944, bias=False) |
| (down_proj): Linear(in_features=18944, out_features=3584, bias=False) |
| (act_fn): SiLU() |
| ) |
| (input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06) |
| (post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06) |
| ) |
| ) |
| (norm): Qwen2RMSNorm((3584,), eps=1e-06) |
| (rotary_emb): Qwen2RotaryEmbedding() |
| ) |
| (lm_head): Linear(in_features=3584, out_features=152064, bias=False) |
| ) |
| config: |
| Qwen2Config { |
| "architectures": [ |
| "Qwen2ForCausalLM" |
| ], |
| "attention_dropout": 0.0, |
| "bos_token_id": 151643, |
| "eos_token_id": 151643, |
| "hidden_act": "silu", |
| "hidden_size": 3584, |
| "initializer_range": 0.02, |
| "intermediate_size": 18944, |
| "layer_types": [ |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention", |
| "full_attention" |
| ], |
| "max_position_embeddings": 131072, |
| "max_window_layers": 28, |
| "model_type": "qwen2", |
| "num_attention_heads": 28, |
| "num_hidden_layers": 28, |
| "num_key_value_heads": 4, |
| "rms_norm_eps": 1e-06, |
| "rope_scaling": null, |
| "rope_theta": 1000000.0, |
| "sliding_window": null, |
| "tie_word_embeddings": false, |
| "torch_dtype": "float16", |
| "transformers_version": "4.55.2", |
| "use_cache": true, |
| "use_mrope": false, |
| "use_sliding_window": false, |
| "vocab_size": 152064 |
| } |
|
|
| Processing layer 0--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 0 ---0.030741512775421143 |
| Processing layer 1--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 1 ---0.0886889323592186 |
| Processing layer 2--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 2 ---0.038115616887807846 |
| Processing layer 3--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 3 ---0.01555887795984745 |
| Processing layer 4--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 4 ---0.015330223366618156 |
| Processing layer 5--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 5 ---0.014795559458434582 |
| Processing layer 6--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 6 ---0.013012934476137161 |
| Processing layer 7--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 7 ---0.012255651876330376 |
| Processing layer 8--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 8 ---0.012662074528634548 |
| Processing layer 9--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 9 ---0.017650291323661804 |
| Processing layer 10--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 10 ---0.012567928992211819 |
| Processing layer 11--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 11 ---0.012772100046277046 |
| Processing layer 12--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 12 ---0.012693522498011589 |
| Processing layer 13--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 13 ---0.012766292318701744 |
| Processing layer 14--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 14 ---0.01258667092770338 |
| Processing layer 15--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 15 ---0.012480087578296661 |
| Processing layer 16--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 16 ---0.012708479538559914 |
| Processing layer 17--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 17 ---0.01283347513526678 |
| Processing layer 18--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 18 ---0.01226731389760971 |
| Processing layer 19--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 19 ---0.012299998663365841 |
| Processing layer 20--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 20 ---0.01201008539646864 |
| Processing layer 21--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 21 ---0.011824443936347961 |
| Processing layer 22--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 22 ---0.011804303154349327 |
| Processing layer 23--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 23 ---0.012257957831025124 |
| Processing layer 24--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 24 ---0.012149857357144356 |
| Processing layer 25--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 25 ---0.012290380895137787 |
| Processing layer 26--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 26 ---0.012179265730082989 |
| Processing layer 27--subset--{'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)} |
| alpha value of layer 27 ---0.01317012868821621 |
| metric_name coherence: [1, 2, 0, 9, 3, 4, 5, 27, 6, 17, 11, 13, 16, 12, 8, 14, 10, 15, 19, 25, 18, 23, 7, 26, 24, 20, 21, 22] |
|
|
