nohup: ignoring input
Namespace(save_dir='/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg', self_attn_layer_to_quant='16 17 15 14 13', mlp_layer_to_quant='16 17 15 14 13', model_id='/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-7B', cuda_id=6)
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.34it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.43it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.47it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.48it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.46it/s]
Qwen2ForCausalLM(
  (model): Qwen2Model(
    (embed_tokens): Embedding(152064, 3584)
    (layers): ModuleList(
      (0-27): 28 x Qwen2DecoderLayer(
        (self_attn): Qwen2Attention(
          (q_proj): Linear(in_features=3584, out_features=3584, bias=True)
          (k_proj): Linear(in_features=3584, out_features=512, bias=True)
          (v_proj): Linear(in_features=3584, out_features=512, bias=True)
          (o_proj): Linear(in_features=3584, out_features=3584, bias=False)
        )
        (mlp): Qwen2MLP(
          (gate_proj): Linear(in_features=3584, out_features=18944, bias=False)
          (up_proj): Linear(in_features=3584, out_features=18944, bias=False)
          (down_proj): Linear(in_features=18944, out_features=3584, bias=False)
          (act_fn): SiLUActivation()
        )
        (input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
        (post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
      )
    )
    (norm): Qwen2RMSNorm((3584,), eps=1e-06)
    (rotary_emb): Qwen2RotaryEmbedding()
  )
  (lm_head): Linear(in_features=3584, out_features=152064, bias=False)
)
Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation.
Once upon a time，there was a man who lived in a village. He was a kind man， but he was also very stupid. One day， the man went into a village far away. He was walking along a road when he saw a woman with a large basket on her back. He thought that she must be very strong. So he asked her， "Can a weak woman like you carry such a heavy basket？" The woman answered， "I'm not a woman at all， and I'm not
quanting ... 
layers.0.self_attn 4 bit quant 
layers.0.mlp 4 bit quant 
layers.1.self_attn 4 bit quant 
layers.1.mlp 4 bit quant 
layers.2.self_attn 4 bit quant 
layers.2.mlp 4 bit quant 
layers.3.self_attn 4 bit quant 
layers.3.mlp 4 bit quant 
layers.4.self_attn 4 bit quant 
layers.4.mlp 4 bit quant 
layers.5.self_attn 4 bit quant 
layers.5.mlp 4 bit quant 
layers.6.self_attn 4 bit quant 
layers.6.mlp 4 bit quant 
layers.7.self_attn 4 bit quant 
layers.7.mlp 4 bit quant 
layers.8.self_attn 4 bit quant 
layers.8.mlp 4 bit quant 
layers.9.self_attn 4 bit quant 
layers.9.mlp 4 bit quant 
layers.10.self_attn 4 bit quant 
layers.10.mlp 4 bit quant 
layers.11.self_attn 4 bit quant 
layers.11.mlp 4 bit quant 
layers.12.self_attn 4 bit quant 
layers.12.mlp 4 bit quant 
layers.13.self_attn 2 bit quant 
layers.13.mlp 2 bit quant 
layers.14.self_attn 2 bit quant 
layers.14.mlp 2 bit quant 
layers.15.self_attn 2 bit quant 
layers.15.mlp 2 bit quant 
layers.16.self_attn 2 bit quant 
layers.16.mlp 2 bit quant 
layers.17.self_attn 2 bit quant 
layers.17.mlp 2 bit quant 
layers.18.self_attn 4 bit quant 
layers.18.mlp 4 bit quant 
layers.19.self_attn 4 bit quant 
layers.19.mlp 4 bit quant 
layers.20.self_attn 4 bit quant 
layers.20.mlp 4 bit quant 
layers.21.self_attn 4 bit quant 
layers.21.mlp 4 bit quant 
layers.22.self_attn 4 bit quant 
layers.22.mlp 4 bit quant 
layers.23.self_attn 4 bit quant 
layers.23.mlp 4 bit quant 
layers.24.self_attn 4 bit quant 
layers.24.mlp 4 bit quant 
layers.25.self_attn 4 bit quant 
layers.25.mlp 4 bit quant 
layers.26.self_attn 4 bit quant 
layers.26.mlp 4 bit quant 
layers.27.self_attn 4 bit quant 
layers.27.mlp 4 bit quant 
quanted
Qwen2ForCausalLM(
  (model): Qwen2Model(
    (embed_tokens): Embedding(152064, 3584)
    (layers): ModuleList(
      (0-27): 28 x Qwen2DecoderLayer(
        (self_attn): Qwen2Attention(
          (q_proj): W8A16Linear(3584, 3584, bias=True, weight_quant=per_channel)
          (k_proj): W8A16Linear(3584, 512, bias=True, weight_quant=per_channel)
          (v_proj): W8A16Linear(3584, 512, bias=True, weight_quant=per_channel)
          (o_proj): W8A16Linear(3584, 3584, bias=False, weight_quant=per_channel)
        )
        (mlp): Qwen2MLP(
          (gate_proj): W8A16Linear(3584, 18944, bias=False, weight_quant=per_channel)
          (up_proj): W8A16Linear(3584, 18944, bias=False, weight_quant=per_channel)
          (down_proj): W8A16Linear(18944, 3584, bias=False, weight_quant=per_channel)
          (act_fn): SiLUActivation()
        )
        (input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
        (post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
      )
    )
    (norm): Qwen2RMSNorm((3584,), eps=1e-06)
    (rotary_emb): Qwen2RotaryEmbedding()
  )
  (lm_head): Linear(in_features=3584, out_features=152064, bias=False)
)
Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation.
Once upon a time，a young family went（住家的） on vacation.Of the family were Mom and Dad，a Kid Again，Daughters （女儿） No.1，Daughters（住家女儿），Dads（住家女儿），Daughters-in-law（住家妈妈），Daughters-in-law（住家女儿），Daughters-in-law，Daughters-in-law，Daughter（住家女儿），Daughters-in-law （住家女儿），Daughters-in-law （住家女儿）

==================================================
开始评估：任务=winogrande | 少样本数=5 | 模型=Qwen2.5-7B-quantization-fg
输出路径：results2/Qwen2.5-7B-quantization-fg/baseline_BI/winogrande.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-09:16:09:02 INFO     [__main__:440] Selected Tasks: ['winogrande']
2025-12-09:16:09:02 INFO     [__main__:440] Selected Tasks: ['winogrande']
2025-12-09:16:09:02 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:16:09:02 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-09:16:09:02 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:16:09:02 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-09:16:09:03 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:16:09:03 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:16:09:03 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.06s/it]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.15s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.12s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.19s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.05s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.35it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.15it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.11s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.27it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.09it/s]
2025-12-09:16:09:22 WARNING  [evaluator:309] Overwriting default num_fewshot of winogrande from None to 5
2025-12-09:16:09:22 INFO     [api.task:434] Building contexts for winogrande on rank 1...

  0%|          | 0/633 [00:00<?, ?it/s]
100%|██████████| 633/633 [00:00<00:00, 27764.65it/s]
2025-12-09:16:09:23 WARNING  [evaluator:309] Overwriting default num_fewshot of winogrande from None to 5
2025-12-09:16:09:23 INFO     [api.task:434] Building contexts for winogrande on rank 0...

  0%|          | 0/634 [00:00<?, ?it/s]
100%|██████████| 634/634 [00:00<00:00, 25925.60it/s]
n136-128-154:2216229:2216229 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2216229:2216229 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2216229:2216229 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:2216229:2216229 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2216229:2216229 [0] NCCL INFO Comm config Blocking set to 1
n136-128-154:2216230:2216230 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:2216230:2216230 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2216230:2216230 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2216230:2216230 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2216230:2216230 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:2216229:2216439 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2216229:2216439 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2216230:2216440 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2216230:2216440 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2216229:2216439 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2216229:2216439 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2216229:2216439 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2216230:2216440 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2216230:2216440 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2216230:2216440 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2216229:2216439 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2216229:2216439 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2216229:2216439 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:2216229:2216439 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2216229:2216439 [0] NCCL INFO Using network IB
n136-128-154:2216230:2216440 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2216230:2216440 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2216230:2216440 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:2216230:2216440 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2216230:2216440 [1] NCCL INFO Using network IB
n136-128-154:2216229:2216439 [0] NCCL INFO ncclCommInitRankConfig comm 0xe2c5cc0 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0xf7a531d2d3107fc6 - Init START
n136-128-154:2216230:2216440 [1] NCCL INFO ncclCommInitRankConfig comm 0x11084de0 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0xf7a531d2d3107fc6 - Init START
n136-128-154:2216229:2216439 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2216230:2216440 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2216230:2216440 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2216230:2216440 [1] NCCL INFO Retrieving state for IB
n136-128-154:2216230:2216440 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:2216230:2216440 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2216229:2216439 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2216229:2216439 [0] NCCL INFO Retrieving state for IB
n136-128-154:2216229:2216439 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:2216229:2216439 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2216230:2216440 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2216229:2216439 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2216230:2216440 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2216229:2216439 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2216230:2216440 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2216229:2216439 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2216229:2216439 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2216230:2216440 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2216229:2216439 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2216230:2216440 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2216229:2216439 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2216229:2216439 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2216229:2216439 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2216229:2216439 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2216229:2216439 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2216229:2216439 [0] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2216229:2216439 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2216229:2216439 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2216229:2216439 [0] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2216229:2216439 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2216229:2216439 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2216229:2216439 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2216229:2216439 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2216229:2216439 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2216229:2216439 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2216229:2216439 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2216229:2216439 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2216229:2216439 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2216229:2216439 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2216229:2216439 [0] NCCL INFO ==========================================
n136-128-154:2216229:2216439 [0] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2216229:2216439 [0] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2216229:2216439 [0] NCCL INFO Setting affinity for GPU 6 to 33-62,97-126
n136-128-154:2216230:2216440 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2216230:2216440 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2216230:2216440 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2216230:2216440 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2216230:2216440 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2216230:2216440 [1] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2216230:2216440 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2216230:2216440 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2216230:2216440 [1] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2216230:2216440 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2216230:2216440 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2216230:2216440 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2216230:2216440 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2216230:2216440 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2216230:2216440 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2216230:2216440 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2216230:2216440 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2216230:2216440 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2216230:2216440 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2216230:2216440 [1] NCCL INFO ==========================================
n136-128-154:2216230:2216440 [1] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2216230:2216440 [1] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2216229:2216439 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2216230:2216440 [1] NCCL INFO Setting affinity for GPU 7 to 33-62,97-126
n136-128-154:2216229:2216439 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216229:2216439 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216229:2216439 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216229:2216439 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216229:2216439 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216229:2216439 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216229:2216439 [0] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216229:2216439 [0] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216229:2216439 [0] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216229:2216439 [0] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216229:2216439 [0] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216229:2216439 [0] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216229:2216439 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2216229:2216439 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216229:2216439 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216229:2216439 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216229:2216439 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216229:2216439 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216229:2216439 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216229:2216439 [0] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2216229:2216439 [0] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2216229:2216439 [0] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2216229:2216439 [0] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2216229:2216439 [0] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2216229:2216439 [0] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2216230:2216440 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2216230:2216440 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216230:2216440 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216230:2216440 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216230:2216440 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216230:2216440 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216230:2216440 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216230:2216440 [1] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216230:2216440 [1] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216230:2216440 [1] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216230:2216440 [1] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216230:2216440 [1] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216230:2216440 [1] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216230:2216440 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2216230:2216440 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216230:2216440 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216230:2216440 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216230:2216440 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216230:2216440 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216230:2216440 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216230:2216440 [1] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2216230:2216440 [1] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2216230:2216440 [1] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2216230:2216440 [1] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2216230:2216440 [1] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2216230:2216440 [1] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2216229:2216439 [0] NCCL INFO comm 0xe2c5cc0 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:2216229:2216439 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:2216229:2216439 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:2216229:2216439 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:2216229:2216439 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:2216229:2216439 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:2216229:2216439 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:2216229:2216439 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:2216229:2216439 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:2216229:2216439 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:2216229:2216439 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:2216229:2216439 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:2216229:2216439 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:2216229:2216439 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:2216229:2216439 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:2216229:2216439 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:2216229:2216439 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:2216229:2216439 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:2216229:2216439 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:2216229:2216439 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:2216229:2216439 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:2216229:2216439 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:2216229:2216439 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:2216229:2216439 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:2216229:2216439 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:2216229:2216439 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:2216229:2216439 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:2216229:2216439 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:2216229:2216439 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:2216229:2216439 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:2216229:2216439 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:2216229:2216439 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:2216229:2216439 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:2216229:2216439 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:2216229:2216439 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:2216229:2216439 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:2216229:2216439 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:2216229:2216439 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:2216229:2216439 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:2216229:2216439 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:2216229:2216439 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:2216229:2216439 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:2216229:2216439 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:2216229:2216439 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:2216229:2216439 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:2216229:2216439 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:2216229:2216439 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:2216229:2216439 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:2216229:2216439 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:2216229:2216439 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:2216229:2216439 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:2216229:2216439 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:2216229:2216439 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:2216229:2216439 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:2216229:2216439 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:2216229:2216439 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:2216229:2216439 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:2216229:2216439 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:2216229:2216439 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:2216229:2216439 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:2216229:2216439 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:2216229:2216439 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:2216229:2216439 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:2216229:2216439 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:2216229:2216439 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:2216229:2216439 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:2216229:2216439 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:2216229:2216439 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:2216229:2216439 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:2216229:2216439 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:2216229:2216439 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:2216229:2216439 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:2216229:2216439 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:2216229:2216439 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:2216229:2216439 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2216230:2216440 [1] NCCL INFO comm 0x11084de0 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:2216230:2216440 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:2216230:2216440 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:2216230:2216440 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:2216230:2216440 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:2216230:2216440 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:2216230:2216440 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:2216230:2216440 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:2216230:2216440 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:2216230:2216440 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:2216230:2216440 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:2216230:2216440 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:2216230:2216440 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:2216230:2216440 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:2216230:2216440 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:2216230:2216440 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:2216230:2216440 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:2216230:2216440 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:2216230:2216440 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:2216230:2216440 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:2216230:2216440 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:2216230:2216440 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:2216230:2216440 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:2216230:2216440 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:2216230:2216440 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:2216230:2216440 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:2216230:2216440 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:2216230:2216440 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:2216230:2216440 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:2216230:2216440 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:2216230:2216440 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:2216230:2216440 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:2216230:2216440 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:2216230:2216440 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:2216230:2216440 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:2216230:2216440 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:2216230:2216440 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:2216230:2216440 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:2216230:2216440 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:2216230:2216440 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:2216230:2216440 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:2216230:2216440 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:2216230:2216440 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:2216230:2216440 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:2216230:2216440 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:2216230:2216440 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:2216230:2216440 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:2216230:2216440 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:2216230:2216440 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:2216230:2216440 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:2216230:2216440 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2216229:2216439 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2216229:2216439 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:2216229:2216453 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 42
n136-128-154:2216229:2216452 [0] NCCL INFO [Proxy Service] Device 0 CPU core 41
n136-128-154:2216230:2216440 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2216230:2216454 [1] NCCL INFO [Proxy Service] Device 1 CPU core 37
n136-128-154:2216230:2216455 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 102
n136-128-154:2216229:2216439 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2216229:2216439 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2216229:2216439 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:2216230:2216440 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2216230:2216440 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2216229:2216439 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2216229:2216439 [0] NCCL INFO ncclCommInitRankConfig comm 0xe2c5cc0 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0xf7a531d2d3107fc6 - Init COMPLETE
n136-128-154:2216229:2216439 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.67 (kernels 0.20, alloc 0.25, bootstrap 0.00, allgathers 0.02, topo 0.10, graphs 0.00, connections 0.04, rest 0.07)
n136-128-154:2216230:2216440 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2216230:2216440 [1] NCCL INFO ncclCommInitRankConfig comm 0x11084de0 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0xf7a531d2d3107fc6 - Init COMPLETE
n136-128-154:2216230:2216440 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.67 (kernels 0.20, alloc 0.25, bootstrap 0.00, allgathers 0.01, topo 0.10, graphs 0.00, connections 0.03, rest 0.09)
n136-128-154:2216229:2216456 [0] NCCL INFO Channel 00/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216229:2216456 [0] NCCL INFO Channel 01/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216229:2216456 [0] NCCL INFO Channel 02/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216229:2216456 [0] NCCL INFO Channel 03/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216229:2216456 [0] NCCL INFO Channel 04/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216229:2216456 [0] NCCL INFO Channel 05/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216229:2216456 [0] NCCL INFO Channel 06/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216230:2216457 [1] NCCL INFO Channel 00/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216229:2216456 [0] NCCL INFO Channel 07/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216229:2216456 [0] NCCL INFO Channel 08/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216230:2216457 [1] NCCL INFO Channel 01/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216229:2216456 [0] NCCL INFO Channel 09/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216230:2216457 [1] NCCL INFO Channel 02/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216229:2216456 [0] NCCL INFO Channel 10/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216230:2216457 [1] NCCL INFO Channel 03/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216229:2216456 [0] NCCL INFO Channel 11/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216230:2216457 [1] NCCL INFO Channel 04/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216229:2216456 [0] NCCL INFO Channel 12/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216230:2216457 [1] NCCL INFO Channel 05/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216229:2216456 [0] NCCL INFO Channel 13/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216230:2216457 [1] NCCL INFO Channel 06/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216229:2216456 [0] NCCL INFO Channel 14/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216230:2216457 [1] NCCL INFO Channel 07/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216229:2216456 [0] NCCL INFO Channel 15/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216229:2216456 [0] NCCL INFO Channel 16/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216230:2216457 [1] NCCL INFO Channel 08/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216229:2216456 [0] NCCL INFO Channel 17/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216230:2216457 [1] NCCL INFO Channel 09/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216229:2216456 [0] NCCL INFO Channel 18/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216230:2216457 [1] NCCL INFO Channel 10/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216229:2216456 [0] NCCL INFO Channel 19/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216230:2216457 [1] NCCL INFO Channel 11/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216229:2216456 [0] NCCL INFO Channel 20/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216230:2216457 [1] NCCL INFO Channel 12/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216229:2216456 [0] NCCL INFO Channel 21/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216230:2216457 [1] NCCL INFO Channel 13/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216229:2216456 [0] NCCL INFO Channel 22/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216230:2216457 [1] NCCL INFO Channel 14/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216229:2216456 [0] NCCL INFO Channel 23/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216230:2216457 [1] NCCL INFO Channel 15/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216457 [1] NCCL INFO Channel 16/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216457 [1] NCCL INFO Channel 17/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216457 [1] NCCL INFO Channel 18/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216457 [1] NCCL INFO Channel 19/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216457 [1] NCCL INFO Channel 20/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216457 [1] NCCL INFO Channel 21/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216457 [1] NCCL INFO Channel 22/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216457 [1] NCCL INFO Channel 23/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216457 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:2216229:2216456 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-09:16:09:23 INFO     [evaluator:559] Running loglikelihood requests
2025-12-09:16:09:23 INFO     [evaluator:559] Running loglikelihood requests

Running loglikelihood requests:   0%|          | 0/1268 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 64
Determined largest batch size: 64

Running loglikelihood requests:   0%|          | 1/1268 [00:05<1:55:51,  5.49s/it]
Running loglikelihood requests:   5%|▌         | 65/1268 [00:06<01:28, 13.63it/s] 
Running loglikelihood requests:  10%|█         | 129/1268 [00:07<00:43, 25.90it/s]
Running loglikelihood requests:  15%|█▌        | 193/1268 [00:08<00:29, 36.72it/s]
Running loglikelihood requests:  20%|██        | 257/1268 [00:09<00:22, 45.71it/s]
Running loglikelihood requests:  25%|██▌       | 321/1268 [00:09<00:17, 53.02it/s]
Running loglikelihood requests:  30%|███       | 385/1268 [00:10<00:14, 58.89it/s]
Running loglikelihood requests:  35%|███▌      | 449/1268 [00:11<00:12, 63.36it/s]
Running loglikelihood requests:  40%|████      | 513/1268 [00:12<00:11, 66.78it/s]
Running loglikelihood requests:  46%|████▌     | 577/1268 [00:13<00:09, 69.43it/s]
Running loglikelihood requests:  51%|█████     | 641/1268 [00:14<00:08, 71.77it/s]
Running loglikelihood requests:  56%|█████▌    | 705/1268 [00:14<00:07, 73.36it/s]
Running loglikelihood requests:  61%|██████    | 769/1268 [00:15<00:06, 74.57it/s]
Running loglikelihood requests:  66%|██████▌   | 833/1268 [00:16<00:05, 75.66it/s]
Running loglikelihood requests:  71%|███████   | 897/1268 [00:17<00:04, 76.43it/s]
Running loglikelihood requests:  76%|███████▌  | 961/1268 [00:18<00:03, 77.06it/s]
Running loglikelihood requests:  81%|████████  | 1025/1268 [00:19<00:03, 78.10it/s]
Running loglikelihood requests:  86%|████████▌ | 1089/1268 [00:19<00:02, 78.82it/s]
Running loglikelihood requests:  91%|█████████ | 1153/1268 [00:20<00:01, 79.75it/s]
Running loglikelihood requests:  96%|█████████▌| 1217/1268 [00:21<00:00, 85.13it/s]
Running loglikelihood requests: 100%|██████████| 1268/1268 [00:21<00:00, 59.72it/s]
n136-128-154:2216230:2216480 [1] NCCL INFO Channel 00/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216480 [1] NCCL INFO Channel 01/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216480 [1] NCCL INFO Channel 02/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216480 [1] NCCL INFO Channel 03/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216480 [1] NCCL INFO Channel 04/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216480 [1] NCCL INFO Channel 05/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216480 [1] NCCL INFO Channel 06/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216480 [1] NCCL INFO Channel 07/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216480 [1] NCCL INFO Channel 08/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216480 [1] NCCL INFO Channel 09/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216480 [1] NCCL INFO Channel 10/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216480 [1] NCCL INFO Channel 11/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216480 [1] NCCL INFO Channel 12/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216480 [1] NCCL INFO Channel 13/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216480 [1] NCCL INFO Channel 14/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216480 [1] NCCL INFO Channel 15/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216480 [1] NCCL INFO Channel 16/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216480 [1] NCCL INFO Channel 17/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216480 [1] NCCL INFO Channel 18/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216480 [1] NCCL INFO Channel 19/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216480 [1] NCCL INFO Channel 20/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216480 [1] NCCL INFO Channel 21/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216480 [1] NCCL INFO Channel 22/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216480 [1] NCCL INFO Channel 23/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216480 [1] NCCL INFO Channel 24/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216480 [1] NCCL INFO Channel 25/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216480 [1] NCCL INFO Channel 26/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216480 [1] NCCL INFO Channel 27/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216480 [1] NCCL INFO Channel 28/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216480 [1] NCCL INFO Channel 29/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216480 [1] NCCL INFO Channel 30/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216230:2216480 [1] NCCL INFO Channel 31/1 : 1[7] -> 0[6] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:2216230:2216486 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2216230:2216486 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2216230:2216486 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2216230:2216486 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2216230:2216486 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2216230:2216486 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2216230:2216454 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-09:16:09:55 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: auto (64)
|  Tasks   |Version|Filter|n-shot|Metric|   |Value|   |Stderr|
|----------|------:|------|-----:|------|---|----:|---|-----:|
|winogrande|      1|none  |     5|acc   |↑  |0.513|±  | 0.014|

[rank0]:[W1209 16:09:56.530616565 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:2216229:2216535 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2216229:2216535 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2216229:2216535 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2216229:2216452 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2216229:2216535 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2216229:2216535 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2216229:2216535 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2216230:2216454 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2216230:2216486 [1] NCCL INFO comm 0x11084de0 rank 1 nranks 2 cudaDev 1 busId c9000 - Abort COMPLETE
n136-128-154:2216229:2216535 [0] NCCL INFO comm 0xe2c5cc0 rank 0 nranks 2 cudaDev 0 busId c5000 - Abort COMPLETE
任务 winogrande 评估完成！

==================================================
开始评估：任务=boolq | 少样本数=0 | 模型=Qwen2.5-7B-quantization-fg
输出路径：results2/Qwen2.5-7B-quantization-fg/baseline_BI/boolq.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-09:16:10:48 INFO     [__main__:440] Selected Tasks: ['boolq']
2025-12-09:16:10:48 INFO     [__main__:440] Selected Tasks: ['boolq']
2025-12-09:16:10:48 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:16:10:48 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-09:16:10:48 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:16:10:48 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-09:16:10:49 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:16:10:50 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:16:10:50 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.26s/it]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.26s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.16s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.16s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.04s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.06s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.34it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.33it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.12it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.12it/s]
2025-12-09:16:10:53 WARNING  [api.task:844] [Task: boolq] metric acc is defined, but aggregation is not. using default aggregation=mean
2025-12-09:16:10:53 WARNING  [api.task:856] [Task: boolq] metric acc is defined, but higher_is_better is not. using default higher_is_better=True
2025-12-09:16:10:53 WARNING  [api.task:844] [Task: boolq] metric acc is defined, but aggregation is not. using default aggregation=mean
2025-12-09:16:10:53 WARNING  [api.task:856] [Task: boolq] metric acc is defined, but higher_is_better is not. using default higher_is_better=True
2025-12-09:16:11:10 WARNING  [evaluator:309] Overwriting default num_fewshot of boolq from None to 0
2025-12-09:16:11:10 INFO     [api.task:434] Building contexts for boolq on rank 1...

  0%|          | 0/1635 [00:00<?, ?it/s]
 15%|█▌        | 252/1635 [00:00<00:00, 2508.59it/s]
 31%|███       | 506/1635 [00:00<00:00, 2521.59it/s]
 46%|████▋     | 760/1635 [00:00<00:00, 2528.49it/s]
 64%|██████▍   | 1045/1635 [00:00<00:00, 2652.44it/s]
 81%|████████  | 1328/1635 [00:00<00:00, 2714.97it/s]
 99%|█████████▊| 1611/1635 [00:00<00:00, 2753.51it/s]
100%|██████████| 1635/1635 [00:00<00:00, 2679.98it/s]
2025-12-09:16:11:13 WARNING  [evaluator:309] Overwriting default num_fewshot of boolq from None to 0
2025-12-09:16:11:13 INFO     [api.task:434] Building contexts for boolq on rank 0...

  0%|          | 0/1635 [00:00<?, ?it/s]
 15%|█▍        | 241/1635 [00:00<00:00, 2407.15it/s]
 31%|███       | 501/1635 [00:00<00:00, 2516.40it/s]
 48%|████▊     | 781/1635 [00:00<00:00, 2645.65it/s]
 65%|██████▍   | 1062/1635 [00:00<00:00, 2709.62it/s]
 82%|████████▏ | 1342/1635 [00:00<00:00, 2741.93it/s]
 99%|█████████▉| 1622/1635 [00:00<00:00, 2761.30it/s]
100%|██████████| 1635/1635 [00:00<00:00, 2701.10it/s]
n136-128-154:2216582:2216582 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2216582:2216582 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2216582:2216582 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:2216582:2216582 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2216582:2216582 [0] NCCL INFO Comm config Blocking set to 1
n136-128-154:2216583:2216583 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:2216583:2216583 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2216583:2216583 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2216583:2216583 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2216583:2216583 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:2216582:2216898 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2216582:2216898 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2216583:2216899 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2216583:2216899 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2216582:2216898 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2216582:2216898 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2216582:2216898 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2216583:2216899 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2216583:2216899 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2216583:2216899 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2216582:2216898 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2216582:2216898 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2216582:2216898 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:2216582:2216898 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2216582:2216898 [0] NCCL INFO Using network IB
n136-128-154:2216583:2216899 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2216583:2216899 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2216583:2216899 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:2216583:2216899 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2216583:2216899 [1] NCCL INFO Using network IB
n136-128-154:2216582:2216898 [0] NCCL INFO ncclCommInitRankConfig comm 0xfda11b0 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0x8d0e5fd69ac7ccb1 - Init START
n136-128-154:2216583:2216899 [1] NCCL INFO ncclCommInitRankConfig comm 0x10a0bd50 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0x8d0e5fd69ac7ccb1 - Init START
n136-128-154:2216582:2216898 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2216583:2216899 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2216582:2216898 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2216582:2216898 [0] NCCL INFO Retrieving state for IB
n136-128-154:2216582:2216898 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:2216582:2216898 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2216583:2216899 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2216583:2216899 [1] NCCL INFO Retrieving state for IB
n136-128-154:2216583:2216899 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:2216582:2216898 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2216583:2216899 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2216582:2216898 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2216583:2216899 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2216582:2216898 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2216583:2216899 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2216583:2216899 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2216582:2216898 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2216582:2216898 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2216583:2216899 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2216583:2216899 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2216582:2216898 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2216582:2216898 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2216582:2216898 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2216582:2216898 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2216582:2216898 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2216582:2216898 [0] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2216582:2216898 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2216582:2216898 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2216582:2216898 [0] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2216582:2216898 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2216582:2216898 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2216582:2216898 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2216582:2216898 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2216582:2216898 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2216582:2216898 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2216582:2216898 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2216582:2216898 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2216582:2216898 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2216582:2216898 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2216582:2216898 [0] NCCL INFO ==========================================
n136-128-154:2216582:2216898 [0] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2216582:2216898 [0] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2216582:2216898 [0] NCCL INFO Setting affinity for GPU 6 to 33-62,97-126
n136-128-154:2216583:2216899 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2216582:2216898 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2216583:2216899 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2216582:2216898 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216583:2216899 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2216582:2216898 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216583:2216899 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2216582:2216898 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216582:2216898 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216583:2216899 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2216582:2216898 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216582:2216898 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216583:2216899 [1] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2216582:2216898 [0] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216583:2216899 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2216582:2216898 [0] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216583:2216899 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2216582:2216898 [0] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216583:2216899 [1] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2216582:2216898 [0] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216583:2216899 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2216582:2216898 [0] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216583:2216899 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2216582:2216898 [0] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216583:2216899 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2216583:2216899 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2216583:2216899 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2216583:2216899 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2216583:2216899 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2216583:2216899 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2216583:2216899 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2216583:2216899 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2216583:2216899 [1] NCCL INFO ==========================================
n136-128-154:2216583:2216899 [1] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2216583:2216899 [1] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2216583:2216899 [1] NCCL INFO Setting affinity for GPU 7 to 33-62,97-126
n136-128-154:2216582:2216898 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2216582:2216898 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216582:2216898 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216582:2216898 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216582:2216898 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216582:2216898 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216582:2216898 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216582:2216898 [0] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2216582:2216898 [0] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2216582:2216898 [0] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2216582:2216898 [0] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2216582:2216898 [0] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2216582:2216898 [0] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2216583:2216899 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2216583:2216899 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216583:2216899 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216583:2216899 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216583:2216899 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216583:2216899 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216583:2216899 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216583:2216899 [1] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216583:2216899 [1] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216583:2216899 [1] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216583:2216899 [1] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216583:2216899 [1] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216583:2216899 [1] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216583:2216899 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2216583:2216899 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216583:2216899 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216583:2216899 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216583:2216899 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216583:2216899 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216583:2216899 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2216583:2216899 [1] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2216583:2216899 [1] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2216583:2216899 [1] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2216583:2216899 [1] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2216583:2216899 [1] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2216583:2216899 [1] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2216583:2216899 [1] NCCL INFO comm 0x10a0bd50 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:2216583:2216899 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:2216583:2216899 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:2216582:2216898 [0] NCCL INFO comm 0xfda11b0 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:2216583:2216899 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:2216583:2216899 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:2216583:2216899 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:2216583:2216899 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:2216583:2216899 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:2216582:2216898 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:2216583:2216899 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:2216582:2216898 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:2216583:2216899 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:2216582:2216898 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:2216583:2216899 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:2216582:2216898 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:2216583:2216899 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:2216582:2216898 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:2216583:2216899 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:2216582:2216898 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:2216583:2216899 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:2216582:2216898 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:2216583:2216899 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:2216582:2216898 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:2216583:2216899 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:2216582:2216898 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:2216583:2216899 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:2216582:2216898 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:2216583:2216899 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:2216582:2216898 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:2216583:2216899 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:2216582:2216898 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:2216583:2216899 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:2216582:2216898 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:2216583:2216899 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:2216582:2216898 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:2216583:2216899 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:2216582:2216898 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:2216583:2216899 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:2216582:2216898 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:2216583:2216899 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:2216582:2216898 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:2216583:2216899 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:2216582:2216898 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:2216582:2216898 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:2216582:2216898 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:2216582:2216898 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:2216582:2216898 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:2216582:2216898 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:2216583:2216899 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:2216582:2216898 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:2216583:2216899 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:2216583:2216899 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:2216583:2216899 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:2216583:2216899 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:2216583:2216899 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:2216583:2216899 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:2216583:2216899 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:2216582:2216898 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:2216583:2216899 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:2216582:2216898 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:2216583:2216899 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:2216582:2216898 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:2216583:2216899 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:2216582:2216898 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:2216583:2216899 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:2216582:2216898 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:2216583:2216899 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:2216582:2216898 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:2216583:2216899 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:2216582:2216898 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:2216583:2216899 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:2216582:2216898 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:2216583:2216899 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:2216582:2216898 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:2216583:2216899 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:2216582:2216898 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:2216583:2216899 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:2216582:2216898 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:2216583:2216899 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:2216582:2216898 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:2216583:2216899 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:2216582:2216898 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:2216583:2216899 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:2216582:2216898 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:2216583:2216899 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:2216582:2216898 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:2216583:2216899 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:2216582:2216898 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:2216583:2216899 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:2216582:2216898 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:2216583:2216899 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:2216582:2216898 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:2216582:2216898 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:2216583:2216899 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2216582:2216898 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:2216582:2216898 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:2216582:2216898 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:2216582:2216898 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:2216582:2216898 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:2216582:2216898 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:2216582:2216898 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:2216582:2216898 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:2216582:2216898 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:2216582:2216898 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:2216582:2216898 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:2216582:2216898 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:2216582:2216898 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:2216582:2216898 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:2216582:2216898 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:2216582:2216898 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:2216582:2216898 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:2216582:2216898 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:2216582:2216898 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:2216582:2216898 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:2216582:2216898 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:2216582:2216898 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:2216582:2216898 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:2216582:2216898 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:2216582:2216898 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:2216582:2216898 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:2216582:2216898 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:2216582:2216898 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:2216582:2216898 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:2216582:2216898 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:2216582:2216898 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2216583:2216899 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2216583:2216910 [1] NCCL INFO [Proxy Service] Device 1 CPU core 39
n136-128-154:2216583:2216911 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 40
n136-128-154:2216582:2216898 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2216582:2216898 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:2216582:2216912 [0] NCCL INFO [Proxy Service] Device 0 CPU core 98
n136-128-154:2216582:2216913 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 37
n136-128-154:2216583:2216899 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2216583:2216899 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2216582:2216898 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2216582:2216898 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2216582:2216898 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:2216583:2216899 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2216583:2216899 [1] NCCL INFO ncclCommInitRankConfig comm 0x10a0bd50 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0x8d0e5fd69ac7ccb1 - Init COMPLETE
n136-128-154:2216583:2216899 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.49 (kernels 0.14, alloc 0.17, bootstrap 0.00, allgathers 0.01, topo 0.07, graphs 0.00, connections 0.05, rest 0.05)
n136-128-154:2216582:2216898 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2216582:2216898 [0] NCCL INFO ncclCommInitRankConfig comm 0xfda11b0 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0x8d0e5fd69ac7ccb1 - Init COMPLETE
n136-128-154:2216582:2216898 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.52 (kernels 0.16, alloc 0.18, bootstrap 0.00, allgathers 0.01, topo 0.07, graphs 0.00, connections 0.05, rest 0.05)
n136-128-154:2216583:2216914 [1] NCCL INFO Channel 00/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216583:2216914 [1] NCCL INFO Channel 01/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216583:2216914 [1] NCCL INFO Channel 02/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216583:2216914 [1] NCCL INFO Channel 03/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216583:2216914 [1] NCCL INFO Channel 04/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216583:2216914 [1] NCCL INFO Channel 05/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216582:2216915 [0] NCCL INFO Channel 00/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216583:2216914 [1] NCCL INFO Channel 06/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216582:2216915 [0] NCCL INFO Channel 01/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216583:2216914 [1] NCCL INFO Channel 07/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216582:2216915 [0] NCCL INFO Channel 02/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216583:2216914 [1] NCCL INFO Channel 08/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216582:2216915 [0] NCCL INFO Channel 03/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216583:2216914 [1] NCCL INFO Channel 09/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216582:2216915 [0] NCCL INFO Channel 04/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216583:2216914 [1] NCCL INFO Channel 10/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216582:2216915 [0] NCCL INFO Channel 05/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216583:2216914 [1] NCCL INFO Channel 11/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216582:2216915 [0] NCCL INFO Channel 06/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216583:2216914 [1] NCCL INFO Channel 12/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216582:2216915 [0] NCCL INFO Channel 07/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216583:2216914 [1] NCCL INFO Channel 13/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216582:2216915 [0] NCCL INFO Channel 08/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216583:2216914 [1] NCCL INFO Channel 14/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216582:2216915 [0] NCCL INFO Channel 09/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216583:2216914 [1] NCCL INFO Channel 15/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216582:2216915 [0] NCCL INFO Channel 10/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216583:2216914 [1] NCCL INFO Channel 16/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216582:2216915 [0] NCCL INFO Channel 11/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216583:2216914 [1] NCCL INFO Channel 17/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216582:2216915 [0] NCCL INFO Channel 12/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216583:2216914 [1] NCCL INFO Channel 18/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216582:2216915 [0] NCCL INFO Channel 13/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216583:2216914 [1] NCCL INFO Channel 19/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216582:2216915 [0] NCCL INFO Channel 14/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216583:2216914 [1] NCCL INFO Channel 20/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216582:2216915 [0] NCCL INFO Channel 15/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216583:2216914 [1] NCCL INFO Channel 21/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216582:2216915 [0] NCCL INFO Channel 16/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216583:2216914 [1] NCCL INFO Channel 22/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216582:2216915 [0] NCCL INFO Channel 17/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216583:2216914 [1] NCCL INFO Channel 23/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216582:2216915 [0] NCCL INFO Channel 18/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216582:2216915 [0] NCCL INFO Channel 19/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216582:2216915 [0] NCCL INFO Channel 20/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216582:2216915 [0] NCCL INFO Channel 21/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216582:2216915 [0] NCCL INFO Channel 22/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216582:2216915 [0] NCCL INFO Channel 23/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2216583:2216914 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:2216582:2216915 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-09:16:11:14 INFO     [evaluator:559] Running loglikelihood requests
2025-12-09:16:11:14 INFO     [evaluator:559] Running loglikelihood requests
Passed argument batch_size = auto:1. Detecting largest batch size

Running loglikelihood requests:   0%|          | 0/3270 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 45
Determined largest batch size: 45

Running loglikelihood requests:   0%|          | 1/3270 [00:55<50:06:21, 55.18s/it]
Running loglikelihood requests:   3%|▎         | 91/3270 [00:56<23:15,  2.28it/s]  
Running loglikelihood requests:   6%|▌         | 181/3270 [00:57<09:42,  5.30it/s]
Running loglikelihood requests:   8%|▊         | 271/3270 [00:58<05:22,  9.29it/s]
Running loglikelihood requests:  11%|█         | 361/3270 [00:59<03:21, 14.44it/s]
Running loglikelihood requests:  14%|█▍        | 451/3270 [01:00<02:14, 20.92it/s]
Running loglikelihood requests:  17%|█▋        | 541/3270 [01:01<01:34, 28.79it/s]
Running loglikelihood requests:  19%|█▉        | 631/3270 [01:01<01:09, 38.00it/s]
Running loglikelihood requests:  22%|██▏       | 721/3270 [01:02<00:52, 48.34it/s]
Running loglikelihood requests:  25%|██▍       | 811/3270 [01:03<00:41, 59.25it/s]
Running loglikelihood requests:  28%|██▊       | 901/3270 [01:04<00:33, 70.45it/s]
Running loglikelihood requests:  30%|███       | 991/3270 [01:04<00:28, 80.95it/s]
Running loglikelihood requests:  33%|███▎      | 1081/3270 [01:05<00:24, 90.68it/s]
Running loglikelihood requests:  36%|███▌      | 1171/3270 [01:06<00:21, 99.51it/s]
Running loglikelihood requests:  39%|███▊      | 1261/3270 [01:07<00:18, 107.81it/s]
Running loglikelihood requests:  41%|████▏     | 1351/3270 [01:07<00:16, 115.00it/s]
Running loglikelihood requests:  44%|████▍     | 1441/3270 [01:08<00:15, 121.30it/s]
Running loglikelihood requests:  47%|████▋     | 1531/3270 [01:08<00:13, 126.60it/s]
Running loglikelihood requests:  50%|████▉     | 1621/3270 [01:09<00:12, 131.24it/s]
Running loglikelihood requests:  52%|█████▏    | 1711/3270 [01:10<00:11, 136.30it/s]
Running loglikelihood requests:  55%|█████▌    | 1801/3270 [01:10<00:10, 141.21it/s]
Running loglikelihood requests:  58%|█████▊    | 1891/3270 [01:11<00:09, 145.37it/s]
Running loglikelihood requests:  61%|██████    | 1981/3270 [01:11<00:08, 148.94it/s]
Running loglikelihood requests:  63%|██████▎   | 2071/3270 [01:12<00:07, 152.27it/s]
Running loglikelihood requests:  66%|██████▌   | 2161/3270 [01:13<00:07, 155.34it/s]
Running loglikelihood requests:  69%|██████▉   | 2251/3270 [01:13<00:06, 159.50it/s]
Running loglikelihood requests:  72%|███████▏  | 2341/3270 [01:14<00:05, 163.23it/s]
Running loglikelihood requests:  74%|███████▍  | 2431/3270 [01:14<00:05, 167.33it/s]
Running loglikelihood requests:  77%|███████▋  | 2521/3270 [01:15<00:04, 171.18it/s]
Running loglikelihood requests:  80%|███████▉  | 2611/3270 [01:15<00:03, 174.70it/s]
Running loglikelihood requests:  83%|████████▎ | 2701/3270 [01:16<00:03, 178.81it/s]
Running loglikelihood requests:  85%|████████▌ | 2791/3270 [01:16<00:02, 184.33it/s]
Running loglikelihood requests:  88%|████████▊ | 2881/3270 [01:16<00:02, 190.68it/s]
Running loglikelihood requests:  91%|█████████ | 2971/3270 [01:17<00:01, 197.42it/s]
Running loglikelihood requests:  94%|█████████▎| 3061/3270 [01:17<00:01, 206.30it/s]
Running loglikelihood requests:  96%|█████████▋| 3151/3270 [01:18<00:00, 216.02it/s]
Running loglikelihood requests:  99%|█████████▉| 3241/3270 [01:18<00:00, 267.69it/s]
Running loglikelihood requests: 100%|██████████| 3270/3270 [01:18<00:00, 41.78it/s] 
n136-128-154:2216583:2217108 [1] NCCL INFO Channel 00/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216583:2217108 [1] NCCL INFO Channel 01/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216583:2217108 [1] NCCL INFO Channel 02/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216583:2217108 [1] NCCL INFO Channel 03/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216583:2217108 [1] NCCL INFO Channel 04/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216583:2217108 [1] NCCL INFO Channel 05/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216583:2217108 [1] NCCL INFO Channel 06/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216583:2217108 [1] NCCL INFO Channel 07/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216583:2217108 [1] NCCL INFO Channel 08/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216583:2217108 [1] NCCL INFO Channel 09/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216583:2217108 [1] NCCL INFO Channel 10/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216583:2217108 [1] NCCL INFO Channel 11/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216583:2217108 [1] NCCL INFO Channel 12/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216583:2217108 [1] NCCL INFO Channel 13/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216583:2217108 [1] NCCL INFO Channel 14/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216583:2217108 [1] NCCL INFO Channel 15/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216583:2217108 [1] NCCL INFO Channel 16/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216583:2217108 [1] NCCL INFO Channel 17/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216583:2217108 [1] NCCL INFO Channel 18/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216583:2217108 [1] NCCL INFO Channel 19/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216583:2217108 [1] NCCL INFO Channel 20/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216583:2217108 [1] NCCL INFO Channel 21/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216583:2217108 [1] NCCL INFO Channel 22/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216583:2217108 [1] NCCL INFO Channel 23/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216583:2217108 [1] NCCL INFO Channel 24/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216583:2217108 [1] NCCL INFO Channel 25/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216583:2217108 [1] NCCL INFO Channel 26/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216583:2217108 [1] NCCL INFO Channel 27/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216583:2217108 [1] NCCL INFO Channel 28/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216583:2217108 [1] NCCL INFO Channel 29/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216583:2217108 [1] NCCL INFO Channel 30/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2216583:2217108 [1] NCCL INFO Channel 31/1 : 1[7] -> 0[6] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:2216583:2217113 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2216583:2217113 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2216583:2217113 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2216583:2217113 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2216583:2217113 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2216583:2217113 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2216583:2216910 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-09:16:12:42 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: auto (45)
|Tasks|Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|-----|------:|------|-----:|------|---|-----:|---|-----:|
|boolq|      2|none  |     0|acc   |↑  |0.5076|±  |0.0087|

[rank0]:[W1209 16:12:43.746052598 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:2216582:2217160 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2216582:2217160 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2216582:2217160 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2216582:2216912 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2216582:2217160 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2216582:2217160 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2216582:2217160 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2216583:2216910 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2216583:2217113 [1] NCCL INFO comm 0x10a0bd50 rank 1 nranks 2 cudaDev 1 busId c9000 - Abort COMPLETE
n136-128-154:2216582:2217160 [0] NCCL INFO comm 0xfda11b0 rank 0 nranks 2 cudaDev 0 busId c5000 - Abort COMPLETE
任务 boolq 评估完成！

==================================================
开始评估：任务=arc_challenge | 少样本数=25 | 模型=Qwen2.5-7B-quantization-fg
输出路径：results2/Qwen2.5-7B-quantization-fg/baseline_BI/arc_challenge.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-09:16:13:33 INFO     [__main__:440] Selected Tasks: ['arc_challenge']
2025-12-09:16:13:33 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:16:13:33 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-09:16:13:33 INFO     [__main__:440] Selected Tasks: ['arc_challenge']
2025-12-09:16:13:33 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:16:13:33 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-09:16:13:34 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:16:13:35 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:16:13:35 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.04s/it]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.03s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.08s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.07s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.02s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:00,  1.00it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.40it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.20it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.43it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.22it/s]
2025-12-09:16:13:51 WARNING  [evaluator:309] Overwriting default num_fewshot of arc_challenge from None to 25
2025-12-09:16:13:51 INFO     [api.task:434] Building contexts for arc_challenge on rank 0...

  0%|          | 0/586 [00:00<?, ?it/s]
  1%|          | 4/586 [00:00<00:16, 35.98it/s]
  1%|▏         | 8/586 [00:00<00:15, 36.32it/s]
  2%|▏         | 12/586 [00:00<00:15, 36.41it/s]
  3%|▎         | 16/586 [00:00<00:15, 36.50it/s]
  3%|▎         | 20/586 [00:00<00:15, 36.54it/s]
  4%|▍         | 24/586 [00:00<00:15, 36.57it/s]
  5%|▍         | 28/586 [00:00<00:15, 36.31it/s]
  5%|▌         | 32/586 [00:00<00:15, 36.23it/s]
  6%|▌         | 36/586 [00:00<00:15, 36.20it/s]
  7%|▋         | 40/586 [00:01<00:15, 36.19it/s]
  8%|▊         | 44/586 [00:01<00:14, 36.19it/s]
  8%|▊         | 48/586 [00:01<00:15, 35.83it/s]
  9%|▉         | 52/586 [00:01<00:14, 35.80it/s]
 10%|▉         | 56/586 [00:01<00:14, 35.45it/s]
 10%|█         | 60/586 [00:01<00:14, 35.39it/s]
 11%|█         | 64/586 [00:01<00:14, 35.40it/s]
 12%|█▏        | 68/586 [00:01<00:14, 35.62it/s]
 12%|█▏        | 72/586 [00:02<00:14, 35.47it/s]
 13%|█▎        | 76/586 [00:02<00:14, 35.80it/s]
 14%|█▎        | 80/586 [00:02<00:14, 36.12it/s]
 14%|█▍        | 84/586 [00:02<00:13, 36.33it/s]
 15%|█▌        | 88/586 [00:02<00:13, 36.45it/s]
 16%|█▌        | 92/586 [00:02<00:13, 36.55it/s]
 16%|█▋        | 96/586 [00:02<00:13, 36.42it/s]
 17%|█▋        | 100/586 [00:02<00:13, 36.43it/s]
 18%|█▊        | 104/586 [00:02<00:13, 36.23it/s]
 18%|█▊        | 108/586 [00:02<00:13, 36.14it/s]
 19%|█▉        | 112/586 [00:03<00:13, 36.09it/s]
 20%|█▉        | 116/586 [00:03<00:13, 35.45it/s]
 20%|██        | 120/586 [00:03<00:13, 35.50it/s]
 21%|██        | 124/586 [00:03<00:12, 35.78it/s]
 22%|██▏       | 128/586 [00:03<00:12, 35.96it/s]
 23%|██▎       | 132/586 [00:03<00:12, 36.11it/s]
 23%|██▎       | 136/586 [00:03<00:12, 35.75it/s]
 24%|██▍       | 140/586 [00:03<00:12, 35.91it/s]
 25%|██▍       | 144/586 [00:03<00:12, 36.10it/s]
 25%|██▌       | 148/586 [00:04<00:12, 36.26it/s]
 26%|██▌       | 152/586 [00:04<00:11, 36.28it/s]
 27%|██▋       | 156/586 [00:04<00:11, 36.36it/s]
 27%|██▋       | 160/586 [00:04<00:11, 36.39it/s]
 28%|██▊       | 164/586 [00:04<00:11, 36.40it/s]
 29%|██▊       | 168/586 [00:04<00:11, 36.29it/s]2025-12-09:16:13:56 WARNING  [evaluator:309] Overwriting default num_fewshot of arc_challenge from None to 25
2025-12-09:16:13:56 INFO     [api.task:434] Building contexts for arc_challenge on rank 1...

  0%|          | 0/586 [00:00<?, ?it/s]
 29%|██▉       | 172/586 [00:04<00:11, 36.34it/s]
  1%|          | 4/586 [00:00<00:16, 36.34it/s]
 30%|███       | 176/586 [00:04<00:11, 36.49it/s]
  1%|▏         | 8/586 [00:00<00:15, 36.92it/s]
 31%|███       | 180/586 [00:04<00:11, 35.71it/s]
  2%|▏         | 12/586 [00:00<00:15, 37.10it/s]
 31%|███▏      | 184/586 [00:05<00:11, 35.50it/s]
  3%|▎         | 16/586 [00:00<00:15, 37.14it/s]
 32%|███▏      | 188/586 [00:05<00:11, 35.51it/s]
  3%|▎         | 20/586 [00:00<00:15, 37.19it/s]
 33%|███▎      | 192/586 [00:05<00:11, 35.48it/s]
  4%|▍         | 24/586 [00:00<00:15, 37.22it/s]
 33%|███▎      | 196/586 [00:05<00:10, 35.62it/s]
  5%|▍         | 28/586 [00:00<00:14, 37.24it/s]
 34%|███▍      | 200/586 [00:05<00:10, 35.71it/s]
  5%|▌         | 32/586 [00:00<00:14, 37.17it/s]
 35%|███▍      | 204/586 [00:05<00:10, 35.70it/s]
  6%|▌         | 36/586 [00:00<00:14, 37.21it/s]
 35%|███▌      | 208/586 [00:05<00:10, 35.73it/s]
  7%|▋         | 40/586 [00:01<00:14, 36.73it/s]
 36%|███▌      | 212/586 [00:05<00:10, 35.32it/s]
  8%|▊         | 44/586 [00:01<00:14, 36.85it/s]
 37%|███▋      | 216/586 [00:06<00:10, 35.42it/s]
  8%|▊         | 48/586 [00:01<00:14, 36.46it/s]
 38%|███▊      | 220/586 [00:06<00:10, 35.54it/s]
  9%|▉         | 52/586 [00:01<00:14, 35.68it/s]
 38%|███▊      | 224/586 [00:06<00:10, 35.61it/s]
 10%|▉         | 56/586 [00:01<00:14, 35.75it/s]
 39%|███▉      | 228/586 [00:06<00:10, 35.67it/s]
 10%|█         | 60/586 [00:01<00:15, 34.62it/s]
 40%|███▉      | 232/586 [00:06<00:09, 35.74it/s]
 11%|█         | 64/586 [00:01<00:15, 33.57it/s]
 40%|████      | 236/586 [00:06<00:09, 35.74it/s]
 12%|█▏        | 68/586 [00:01<00:15, 34.50it/s]
 41%|████      | 240/586 [00:06<00:09, 35.78it/s]
 12%|█▏        | 72/586 [00:01<00:14, 35.24it/s]
 42%|████▏     | 244/586 [00:06<00:09, 35.84it/s]
 13%|█▎        | 76/586 [00:02<00:14, 35.85it/s]
 42%|████▏     | 248/586 [00:06<00:09, 35.96it/s]
 14%|█▎        | 80/586 [00:02<00:13, 36.26it/s]
 43%|████▎     | 252/586 [00:07<00:09, 36.05it/s]
 14%|█▍        | 84/586 [00:02<00:13, 36.57it/s]
 44%|████▎     | 256/586 [00:07<00:09, 36.10it/s]
 15%|█▌        | 88/586 [00:02<00:13, 36.85it/s]
 44%|████▍     | 260/586 [00:07<00:09, 36.13it/s]
 16%|█▌        | 92/586 [00:02<00:13, 37.00it/s]
 45%|████▌     | 264/586 [00:07<00:08, 36.16it/s]
 16%|█▋        | 96/586 [00:02<00:13, 37.11it/s]
 46%|████▌     | 268/586 [00:07<00:08, 36.20it/s]
 17%|█▋        | 100/586 [00:02<00:13, 37.22it/s]
 46%|████▋     | 272/586 [00:07<00:08, 36.23it/s]
 18%|█▊        | 104/586 [00:02<00:12, 37.29it/s]
 47%|████▋     | 276/586 [00:07<00:08, 36.22it/s]
 18%|█▊        | 108/586 [00:02<00:12, 37.28it/s]
 48%|████▊     | 280/586 [00:07<00:08, 36.24it/s]
 19%|█▉        | 112/586 [00:03<00:12, 36.81it/s]
 48%|████▊     | 284/586 [00:07<00:08, 36.25it/s]
 20%|█▉        | 116/586 [00:03<00:12, 36.95it/s]
 20%|██        | 120/586 [00:03<00:12, 37.06it/s]
 49%|████▉     | 288/586 [00:08<00:08, 36.23it/s]
 21%|██        | 124/586 [00:03<00:12, 37.12it/s]
 50%|████▉     | 292/586 [00:08<00:08, 36.16it/s]
 22%|██▏       | 128/586 [00:03<00:12, 37.21it/s]
 51%|█████     | 296/586 [00:08<00:08, 36.18it/s]
 23%|██▎       | 132/586 [00:03<00:12, 37.24it/s]
 51%|█████     | 300/586 [00:08<00:08, 35.70it/s]
 23%|██▎       | 136/586 [00:03<00:12, 37.26it/s]
 52%|█████▏    | 304/586 [00:08<00:07, 35.77it/s]
 24%|██▍       | 140/586 [00:03<00:11, 37.23it/s]
 53%|█████▎    | 308/586 [00:08<00:07, 36.04it/s]
 25%|██▍       | 144/586 [00:03<00:11, 36.85it/s]
 53%|█████▎    | 312/586 [00:08<00:07, 36.29it/s]
 25%|██▌       | 148/586 [00:04<00:11, 36.94it/s]
 54%|█████▍    | 316/586 [00:08<00:07, 36.50it/s]
 26%|██▌       | 152/586 [00:04<00:11, 37.02it/s]
 55%|█████▍    | 320/586 [00:08<00:07, 36.37it/s]
 27%|██▋       | 156/586 [00:04<00:11, 37.09it/s]
 55%|█████▌    | 324/586 [00:08<00:07, 36.34it/s]
 27%|██▋       | 160/586 [00:04<00:11, 37.18it/s]
 56%|█████▌    | 328/586 [00:09<00:07, 36.26it/s]
 28%|██▊       | 164/586 [00:04<00:11, 37.19it/s]
 57%|█████▋    | 332/586 [00:09<00:07, 36.20it/s]
 29%|██▊       | 168/586 [00:04<00:11, 37.19it/s]
 57%|█████▋    | 336/586 [00:09<00:06, 36.19it/s]
 29%|██▉       | 172/586 [00:04<00:11, 37.22it/s]
 58%|█████▊    | 340/586 [00:09<00:06, 36.19it/s]
 30%|███       | 176/586 [00:04<00:11, 37.18it/s]
 59%|█████▊    | 344/586 [00:09<00:06, 36.19it/s]
 31%|███       | 180/586 [00:04<00:10, 37.21it/s]
 59%|█████▉    | 348/586 [00:09<00:06, 36.14it/s]
 31%|███▏      | 184/586 [00:05<00:10, 37.22it/s]
 60%|██████    | 352/586 [00:09<00:06, 36.17it/s]
 32%|███▏      | 188/586 [00:05<00:10, 37.28it/s]
 61%|██████    | 356/586 [00:09<00:06, 36.18it/s]
 33%|███▎      | 192/586 [00:05<00:10, 37.29it/s]
 61%|██████▏   | 360/586 [00:09<00:06, 36.11it/s]
 33%|███▎      | 196/586 [00:05<00:10, 37.32it/s]
 62%|██████▏   | 364/586 [00:10<00:06, 35.46it/s]
 34%|███▍      | 200/586 [00:05<00:10, 37.32it/s]
 63%|██████▎   | 368/586 [00:10<00:06, 35.59it/s]
 35%|███▍      | 204/586 [00:05<00:10, 37.29it/s]
 63%|██████▎   | 372/586 [00:10<00:06, 35.50it/s]
 35%|███▌      | 208/586 [00:05<00:10, 37.29it/s]
 36%|███▌      | 212/586 [00:05<00:10, 37.26it/s]
 64%|██████▍   | 376/586 [00:10<00:06, 32.04it/s]
 37%|███▋      | 216/586 [00:05<00:09, 37.22it/s]
 65%|██████▍   | 380/586 [00:10<00:06, 32.06it/s]
 38%|███▊      | 220/586 [00:05<00:09, 37.12it/s]
 66%|██████▌   | 384/586 [00:10<00:06, 33.27it/s]
 38%|███▊      | 224/586 [00:06<00:09, 37.13it/s]
 66%|██████▌   | 388/586 [00:10<00:05, 34.23it/s]
 39%|███▉      | 228/586 [00:06<00:09, 37.13it/s]
 67%|██████▋   | 392/586 [00:10<00:05, 34.92it/s]
 40%|███▉      | 232/586 [00:06<00:09, 37.18it/s]
 68%|██████▊   | 396/586 [00:11<00:05, 35.42it/s]
 40%|████      | 236/586 [00:06<00:09, 36.43it/s]
 68%|██████▊   | 400/586 [00:11<00:05, 35.51it/s]
 41%|████      | 240/586 [00:06<00:09, 36.62it/s]
 69%|██████▉   | 404/586 [00:11<00:05, 35.56it/s]
 42%|████▏     | 244/586 [00:06<00:09, 36.84it/s]
 70%|██████▉   | 408/586 [00:11<00:04, 35.63it/s]
 42%|████▏     | 248/586 [00:06<00:09, 37.02it/s]
 70%|███████   | 412/586 [00:11<00:04, 35.65it/s]
 43%|████▎     | 252/586 [00:06<00:08, 37.15it/s]
 71%|███████   | 416/586 [00:11<00:04, 35.64it/s]
 44%|████▎     | 256/586 [00:06<00:08, 37.22it/s]
 72%|███████▏  | 420/586 [00:11<00:04, 35.55it/s]
 44%|████▍     | 260/586 [00:07<00:08, 37.18it/s]
 72%|███████▏  | 424/586 [00:11<00:04, 35.63it/s]
 45%|████▌     | 264/586 [00:07<00:08, 37.25it/s]
 73%|███████▎  | 428/586 [00:11<00:04, 35.69it/s]
 46%|████▌     | 268/586 [00:07<00:08, 37.33it/s]
 74%|███████▎  | 432/586 [00:12<00:04, 35.74it/s]
 46%|████▋     | 272/586 [00:07<00:08, 37.37it/s]
 74%|███████▍  | 436/586 [00:12<00:04, 35.76it/s]
 47%|████▋     | 276/586 [00:07<00:08, 37.44it/s]
 75%|███████▌  | 440/586 [00:12<00:04, 35.77it/s]
 48%|████▊     | 280/586 [00:07<00:08, 37.47it/s]
 76%|███████▌  | 444/586 [00:12<00:03, 35.79it/s]
 48%|████▊     | 284/586 [00:07<00:08, 37.51it/s]
 76%|███████▋  | 448/586 [00:12<00:03, 35.80it/s]
 49%|████▉     | 288/586 [00:07<00:07, 37.50it/s]
 77%|███████▋  | 452/586 [00:12<00:03, 35.81it/s]
 50%|████▉     | 292/586 [00:07<00:07, 37.50it/s]
 78%|███████▊  | 456/586 [00:12<00:03, 35.79it/s]
 51%|█████     | 296/586 [00:08<00:07, 37.12it/s]
 51%|█████     | 300/586 [00:08<00:07, 37.20it/s]
 78%|███████▊  | 460/586 [00:12<00:03, 35.80it/s]
 52%|█████▏    | 304/586 [00:08<00:07, 37.26it/s]
 79%|███████▉  | 464/586 [00:12<00:03, 35.78it/s]
 53%|█████▎    | 308/586 [00:08<00:07, 37.28it/s]
 80%|███████▉  | 468/586 [00:13<00:03, 35.69it/s]
 53%|█████▎    | 312/586 [00:08<00:07, 37.31it/s]
 81%|████████  | 472/586 [00:13<00:03, 35.69it/s]
 54%|█████▍    | 316/586 [00:08<00:07, 37.32it/s]
 81%|████████  | 476/586 [00:13<00:03, 35.58it/s]
 55%|█████▍    | 320/586 [00:08<00:07, 37.35it/s]
 82%|████████▏ | 480/586 [00:13<00:02, 35.59it/s]
 55%|█████▌    | 324/586 [00:08<00:07, 37.34it/s]
 83%|████████▎ | 484/586 [00:13<00:02, 35.58it/s]
 56%|█████▌    | 328/586 [00:08<00:06, 37.35it/s]
 83%|████████▎ | 488/586 [00:13<00:02, 35.61it/s]
 57%|█████▋    | 332/586 [00:08<00:06, 37.35it/s]
 84%|████████▍ | 492/586 [00:13<00:02, 35.57it/s]
 57%|█████▋    | 336/586 [00:09<00:06, 37.37it/s]
 85%|████████▍ | 496/586 [00:13<00:02, 35.59it/s]
 58%|█████▊    | 340/586 [00:09<00:06, 37.33it/s]
 85%|████████▌ | 500/586 [00:13<00:02, 35.59it/s]
 59%|█████▊    | 344/586 [00:09<00:06, 37.35it/s]
 86%|████████▌ | 504/586 [00:14<00:02, 35.61it/s]
 59%|█████▉    | 348/586 [00:09<00:06, 37.38it/s]
 87%|████████▋ | 508/586 [00:14<00:02, 35.61it/s]
 60%|██████    | 352/586 [00:09<00:06, 37.42it/s]
 87%|████████▋ | 512/586 [00:14<00:02, 35.58it/s]
 61%|██████    | 356/586 [00:09<00:06, 37.38it/s]
 88%|████████▊ | 516/586 [00:14<00:01, 35.53it/s]
 61%|██████▏   | 360/586 [00:09<00:06, 37.37it/s]
 89%|████████▊ | 520/586 [00:14<00:01, 35.58it/s]
 62%|██████▏   | 364/586 [00:09<00:05, 37.36it/s]
 89%|████████▉ | 524/586 [00:14<00:01, 35.62it/s]
 63%|██████▎   | 368/586 [00:09<00:05, 37.40it/s]
 90%|█████████ | 528/586 [00:14<00:01, 35.63it/s]
 63%|██████▎   | 372/586 [00:10<00:05, 37.39it/s]
 91%|█████████ | 532/586 [00:14<00:01, 35.62it/s]
 64%|██████▍   | 376/586 [00:10<00:05, 37.43it/s]
 91%|█████████▏| 536/586 [00:14<00:01, 35.67it/s]
 65%|██████▍   | 380/586 [00:10<00:05, 37.46it/s]
 66%|██████▌   | 384/586 [00:10<00:05, 37.47it/s]
 92%|█████████▏| 540/586 [00:15<00:01, 35.61it/s]
 66%|██████▌   | 388/586 [00:10<00:05, 37.50it/s]
 93%|█████████▎| 544/586 [00:15<00:01, 35.64it/s]
 67%|██████▋   | 392/586 [00:10<00:05, 37.50it/s]
 94%|█████████▎| 548/586 [00:15<00:01, 35.65it/s]
 68%|██████▊   | 396/586 [00:10<00:05, 37.50it/s]
 94%|█████████▍| 552/586 [00:15<00:00, 35.66it/s]
 68%|██████▊   | 400/586 [00:10<00:04, 37.47it/s]
 95%|█████████▍| 556/586 [00:15<00:00, 35.60it/s]
 69%|██████▉   | 404/586 [00:10<00:04, 37.41it/s]
 96%|█████████▌| 560/586 [00:15<00:00, 35.60it/s]
 70%|██████▉   | 408/586 [00:11<00:04, 37.32it/s]
 96%|█████████▌| 564/586 [00:15<00:00, 35.58it/s]
 70%|███████   | 412/586 [00:11<00:04, 37.34it/s]
 97%|█████████▋| 568/586 [00:15<00:00, 35.57it/s]
 71%|███████   | 416/586 [00:11<00:04, 37.36it/s]
 98%|█████████▊| 572/586 [00:15<00:00, 35.55it/s]
 72%|███████▏  | 420/586 [00:11<00:04, 37.36it/s]
 98%|█████████▊| 576/586 [00:16<00:00, 35.55it/s]
 72%|███████▏  | 424/586 [00:11<00:04, 37.38it/s]
 99%|█████████▉| 580/586 [00:16<00:00, 35.59it/s]
 73%|███████▎  | 428/586 [00:11<00:04, 37.43it/s]
100%|█████████▉| 584/586 [00:16<00:00, 35.62it/s]
 74%|███████▎  | 432/586 [00:11<00:04, 37.40it/s]
100%|██████████| 586/586 [00:16<00:00, 35.78it/s]
n136-128-154:2217285:2217285 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2217285:2217285 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2217285:2217285 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:2217285:2217285 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2217285:2217285 [0] NCCL INFO Comm config Blocking set to 1

 74%|███████▍  | 436/586 [00:11<00:04, 37.41it/s]n136-128-154:2217285:2217577 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2217285:2217577 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2217285:2217577 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2217285:2217577 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2217285:2217577 [0] NCCL INFO NCCL_IB_HCA set to mlx5

 75%|███████▌  | 440/586 [00:11<00:03, 37.37it/s]n136-128-154:2217285:2217577 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2217285:2217577 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2217285:2217577 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:2217285:2217577 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2217285:2217577 [0] NCCL INFO Using network IB
n136-128-154:2217285:2217577 [0] NCCL INFO ncclCommInitRankConfig comm 0x125cd3d0 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0x6987de3c12bf2f47 - Init START

 76%|███████▌  | 444/586 [00:11<00:03, 37.33it/s]
 76%|███████▋  | 448/586 [00:12<00:03, 37.27it/s]
 77%|███████▋  | 452/586 [00:12<00:03, 37.28it/s]
 78%|███████▊  | 456/586 [00:12<00:03, 37.30it/s]
 78%|███████▊  | 460/586 [00:12<00:03, 37.26it/s]
 79%|███████▉  | 464/586 [00:12<00:03, 37.26it/s]
 80%|███████▉  | 468/586 [00:12<00:03, 37.25it/s]
 81%|████████  | 472/586 [00:12<00:03, 37.26it/s]
 81%|████████  | 476/586 [00:12<00:02, 37.24it/s]
 82%|████████▏ | 480/586 [00:12<00:02, 37.23it/s]
 83%|████████▎ | 484/586 [00:13<00:02, 37.21it/s]
 83%|████████▎ | 488/586 [00:13<00:02, 37.23it/s]
 84%|████████▍ | 492/586 [00:13<00:02, 37.22it/s]
 85%|████████▍ | 496/586 [00:13<00:02, 37.20it/s]
 85%|████████▌ | 500/586 [00:13<00:02, 37.22it/s]
 86%|████████▌ | 504/586 [00:13<00:02, 37.22it/s]
 87%|████████▋ | 508/586 [00:13<00:02, 37.19it/s]
 87%|████████▋ | 512/586 [00:13<00:01, 37.21it/s]
 88%|████████▊ | 516/586 [00:13<00:01, 37.16it/s]
 89%|████████▊ | 520/586 [00:14<00:01, 37.16it/s]
 89%|████████▉ | 524/586 [00:14<00:01, 37.19it/s]
 90%|█████████ | 528/586 [00:14<00:01, 37.10it/s]
 91%|█████████ | 532/586 [00:14<00:01, 37.12it/s]
 91%|█████████▏| 536/586 [00:14<00:01, 36.74it/s]
 92%|█████████▏| 540/586 [00:14<00:01, 36.86it/s]
 93%|█████████▎| 544/586 [00:14<00:01, 36.94it/s]
 94%|█████████▎| 548/586 [00:14<00:01, 37.02it/s]
 94%|█████████▍| 552/586 [00:14<00:00, 37.08it/s]
 95%|█████████▍| 556/586 [00:14<00:00, 37.15it/s]
 96%|█████████▌| 560/586 [00:15<00:00, 37.16it/s]
 96%|█████████▌| 564/586 [00:15<00:00, 37.23it/s]
 97%|█████████▋| 568/586 [00:15<00:00, 37.22it/s]
 98%|█████████▊| 572/586 [00:15<00:00, 37.22it/s]
 98%|█████████▊| 576/586 [00:15<00:00, 37.22it/s]
 99%|█████████▉| 580/586 [00:15<00:00, 37.26it/s]
100%|█████████▉| 584/586 [00:15<00:00, 37.29it/s]
100%|██████████| 586/586 [00:15<00:00, 37.09it/s]
n136-128-154:2217286:2217286 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:2217286:2217286 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2217286:2217286 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2217286:2217286 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2217286:2217286 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:2217286:2217585 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2217286:2217585 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2217286:2217585 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2217286:2217585 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2217286:2217585 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2217286:2217585 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2217286:2217585 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2217286:2217585 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:2217286:2217585 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2217286:2217585 [1] NCCL INFO Using network IB
n136-128-154:2217286:2217585 [1] NCCL INFO ncclCommInitRankConfig comm 0x1131e060 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0x6987de3c12bf2f47 - Init START
n136-128-154:2217286:2217585 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2217285:2217577 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2217286:2217585 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2217286:2217585 [1] NCCL INFO Retrieving state for IB
n136-128-154:2217286:2217585 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:2217286:2217585 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2217286:2217585 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2217286:2217585 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2217285:2217577 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2217285:2217577 [0] NCCL INFO Retrieving state for IB
n136-128-154:2217285:2217577 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:2217285:2217577 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2217286:2217585 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2217285:2217577 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2217285:2217577 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2217285:2217577 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2217286:2217585 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2217286:2217585 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2217285:2217577 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2217285:2217577 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2217285:2217577 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2217285:2217577 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2217285:2217577 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2217285:2217577 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2217285:2217577 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2217285:2217577 [0] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2217285:2217577 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2217285:2217577 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2217285:2217577 [0] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2217285:2217577 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2217285:2217577 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2217285:2217577 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2217285:2217577 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2217285:2217577 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2217285:2217577 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2217285:2217577 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2217285:2217577 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2217285:2217577 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2217285:2217577 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2217285:2217577 [0] NCCL INFO ==========================================
n136-128-154:2217285:2217577 [0] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2217285:2217577 [0] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2217285:2217577 [0] NCCL INFO Setting affinity for GPU 6 to 33-62,97-126
n136-128-154:2217286:2217585 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2217286:2217585 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2217286:2217585 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2217286:2217585 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2217286:2217585 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2217286:2217585 [1] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2217286:2217585 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2217286:2217585 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2217286:2217585 [1] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2217286:2217585 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2217286:2217585 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2217286:2217585 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2217286:2217585 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2217286:2217585 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2217286:2217585 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2217286:2217585 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2217286:2217585 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2217286:2217585 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2217286:2217585 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2217286:2217585 [1] NCCL INFO ==========================================
n136-128-154:2217285:2217577 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2217286:2217585 [1] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2217285:2217577 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2217285:2217577 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2217285:2217577 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2217286:2217585 [1] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2217285:2217577 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2217285:2217577 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2217285:2217577 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2217285:2217577 [0] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2217285:2217577 [0] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2217285:2217577 [0] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2217286:2217585 [1] NCCL INFO Setting affinity for GPU 7 to 33-62,97-126
n136-128-154:2217285:2217577 [0] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2217285:2217577 [0] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2217285:2217577 [0] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2217285:2217577 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2217285:2217577 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2217285:2217577 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2217285:2217577 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2217285:2217577 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2217285:2217577 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2217285:2217577 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2217285:2217577 [0] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2217285:2217577 [0] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2217285:2217577 [0] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2217285:2217577 [0] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2217285:2217577 [0] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2217285:2217577 [0] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2217286:2217585 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2217286:2217585 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2217286:2217585 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2217286:2217585 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2217286:2217585 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2217286:2217585 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2217286:2217585 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2217286:2217585 [1] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2217286:2217585 [1] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2217286:2217585 [1] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2217286:2217585 [1] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2217286:2217585 [1] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2217286:2217585 [1] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2217286:2217585 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2217286:2217585 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2217286:2217585 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2217286:2217585 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2217286:2217585 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2217286:2217585 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2217286:2217585 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2217286:2217585 [1] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2217286:2217585 [1] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2217286:2217585 [1] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2217286:2217585 [1] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2217286:2217585 [1] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2217286:2217585 [1] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2217285:2217577 [0] NCCL INFO comm 0x125cd3d0 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:2217286:2217585 [1] NCCL INFO comm 0x1131e060 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:2217285:2217577 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:2217286:2217585 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:2217285:2217577 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:2217286:2217585 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:2217285:2217577 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:2217286:2217585 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:2217285:2217577 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:2217286:2217585 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:2217285:2217577 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:2217286:2217585 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:2217285:2217577 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:2217286:2217585 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:2217285:2217577 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:2217286:2217585 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:2217285:2217577 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:2217286:2217585 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:2217285:2217577 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:2217286:2217585 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:2217285:2217577 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:2217286:2217585 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:2217285:2217577 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:2217286:2217585 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:2217285:2217577 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:2217286:2217585 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:2217285:2217577 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:2217285:2217577 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:2217285:2217577 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:2217286:2217585 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:2217285:2217577 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:2217286:2217585 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:2217285:2217577 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:2217286:2217585 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:2217285:2217577 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:2217286:2217585 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:2217285:2217577 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:2217286:2217585 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:2217285:2217577 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:2217286:2217585 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:2217285:2217577 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:2217286:2217585 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:2217285:2217577 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:2217286:2217585 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:2217285:2217577 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:2217286:2217585 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:2217285:2217577 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:2217286:2217585 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:2217286:2217585 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:2217286:2217585 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:2217285:2217577 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:2217285:2217577 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:2217285:2217577 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:2217285:2217577 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:2217285:2217577 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:2217286:2217585 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:2217285:2217577 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:2217286:2217585 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:2217285:2217577 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:2217286:2217585 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:2217285:2217577 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:2217286:2217585 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:2217285:2217577 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:2217286:2217585 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:2217285:2217577 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:2217286:2217585 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:2217285:2217577 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:2217286:2217585 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:2217285:2217577 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:2217286:2217585 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:2217285:2217577 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:2217286:2217585 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:2217285:2217577 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:2217286:2217585 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:2217285:2217577 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:2217286:2217585 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:2217285:2217577 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:2217286:2217585 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:2217285:2217577 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:2217286:2217585 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:2217285:2217577 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:2217286:2217585 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:2217285:2217577 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:2217286:2217585 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:2217285:2217577 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:2217286:2217585 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:2217285:2217577 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:2217286:2217585 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:2217285:2217577 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:2217286:2217585 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:2217285:2217577 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:2217286:2217585 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:2217285:2217577 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:2217286:2217585 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:2217285:2217577 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:2217286:2217585 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:2217285:2217577 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:2217286:2217585 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:2217285:2217577 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:2217286:2217585 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:2217285:2217577 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:2217286:2217585 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:2217285:2217577 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:2217286:2217585 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:2217285:2217577 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:2217285:2217577 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:2217286:2217585 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2217285:2217577 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:2217285:2217577 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:2217285:2217577 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:2217285:2217577 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:2217285:2217577 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:2217285:2217577 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:2217285:2217577 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:2217285:2217577 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:2217285:2217577 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:2217285:2217577 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:2217285:2217577 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:2217285:2217577 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:2217285:2217577 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:2217285:2217577 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:2217285:2217577 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:2217285:2217577 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:2217285:2217577 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:2217285:2217577 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:2217285:2217577 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2217286:2217585 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2217286:2217593 [1] NCCL INFO [Proxy Service] Device 1 CPU core 120
n136-128-154:2217286:2217594 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 58
n136-128-154:2217285:2217577 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2217285:2217577 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:2217285:2217596 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 99
n136-128-154:2217285:2217595 [0] NCCL INFO [Proxy Service] Device 0 CPU core 34
n136-128-154:2217286:2217585 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2217286:2217585 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2217285:2217577 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2217285:2217577 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2217285:2217577 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:2217286:2217585 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2217286:2217585 [1] NCCL INFO ncclCommInitRankConfig comm 0x1131e060 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0x6987de3c12bf2f47 - Init COMPLETE
n136-128-154:2217286:2217585 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.44 (kernels 0.12, alloc 0.15, bootstrap 0.00, allgathers 0.00, topo 0.06, graphs 0.00, connections 0.01, rest 0.09)
n136-128-154:2217285:2217577 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2217285:2217577 [0] NCCL INFO ncclCommInitRankConfig comm 0x125cd3d0 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0x6987de3c12bf2f47 - Init COMPLETE
n136-128-154:2217285:2217577 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 4.56 (kernels 0.11, alloc 0.13, bootstrap 4.16, allgathers 0.00, topo 0.06, graphs 0.00, connections 0.01, rest 0.09)
n136-128-154:2217286:2217597 [1] NCCL INFO Channel 00/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217597 [1] NCCL INFO Channel 01/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217597 [1] NCCL INFO Channel 02/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217597 [1] NCCL INFO Channel 03/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217597 [1] NCCL INFO Channel 04/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217597 [1] NCCL INFO Channel 05/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217597 [1] NCCL INFO Channel 06/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217597 [1] NCCL INFO Channel 07/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217597 [1] NCCL INFO Channel 08/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217597 [1] NCCL INFO Channel 09/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217597 [1] NCCL INFO Channel 10/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217597 [1] NCCL INFO Channel 11/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217597 [1] NCCL INFO Channel 12/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217597 [1] NCCL INFO Channel 13/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217597 [1] NCCL INFO Channel 14/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217597 [1] NCCL INFO Channel 15/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217597 [1] NCCL INFO Channel 16/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217597 [1] NCCL INFO Channel 17/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217597 [1] NCCL INFO Channel 18/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217597 [1] NCCL INFO Channel 19/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217597 [1] NCCL INFO Channel 20/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217597 [1] NCCL INFO Channel 21/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217597 [1] NCCL INFO Channel 22/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217597 [1] NCCL INFO Channel 23/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217285:2217598 [0] NCCL INFO Channel 00/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2217285:2217598 [0] NCCL INFO Channel 01/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2217285:2217598 [0] NCCL INFO Channel 02/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2217285:2217598 [0] NCCL INFO Channel 03/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2217285:2217598 [0] NCCL INFO Channel 04/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2217285:2217598 [0] NCCL INFO Channel 05/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2217285:2217598 [0] NCCL INFO Channel 06/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2217285:2217598 [0] NCCL INFO Channel 07/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2217285:2217598 [0] NCCL INFO Channel 08/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2217285:2217598 [0] NCCL INFO Channel 09/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2217285:2217598 [0] NCCL INFO Channel 10/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2217285:2217598 [0] NCCL INFO Channel 11/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2217285:2217598 [0] NCCL INFO Channel 12/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2217285:2217598 [0] NCCL INFO Channel 13/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2217285:2217598 [0] NCCL INFO Channel 14/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2217285:2217598 [0] NCCL INFO Channel 15/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2217285:2217598 [0] NCCL INFO Channel 16/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2217285:2217598 [0] NCCL INFO Channel 17/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2217285:2217598 [0] NCCL INFO Channel 18/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2217285:2217598 [0] NCCL INFO Channel 19/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2217285:2217598 [0] NCCL INFO Channel 20/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2217285:2217598 [0] NCCL INFO Channel 21/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2217285:2217598 [0] NCCL INFO Channel 22/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2217285:2217598 [0] NCCL INFO Channel 23/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2217285:2217598 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:2217286:2217597 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-09:16:14:12 INFO     [evaluator:559] Running loglikelihood requests
2025-12-09:16:14:12 INFO     [evaluator:559] Running loglikelihood requests
Passed argument batch_size = auto:1. Detecting largest batch size

Running loglikelihood requests:   0%|          | 0/2344 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 45
Determined largest batch size: 45

Running loglikelihood requests:   0%|          | 1/2344 [00:56<36:27:33, 56.02s/it]
Running loglikelihood requests:   2%|▏         | 46/2344 [00:59<36:07,  1.06it/s]  
Running loglikelihood requests:   4%|▍         | 94/2344 [01:03<15:55,  2.36it/s]
Running loglikelihood requests:   6%|▌         | 142/2344 [01:07<09:41,  3.79it/s]
Running loglikelihood requests:   8%|▊         | 189/2344 [01:11<06:51,  5.24it/s]
Running loglikelihood requests:  10%|█         | 242/2344 [01:14<05:00,  6.99it/s]
Running loglikelihood requests:  12%|█▏        | 290/2344 [01:18<04:06,  8.33it/s]
Running loglikelihood requests:  14%|█▍        | 336/2344 [01:21<03:33,  9.39it/s]
Running loglikelihood requests:  16%|█▋        | 381/2344 [01:25<03:11, 10.23it/s]
Running loglikelihood requests:  18%|█▊        | 429/2344 [01:28<02:51, 11.16it/s]
Running loglikelihood requests:  20%|██        | 480/2344 [01:32<02:33, 12.12it/s]
Running loglikelihood requests:  22%|██▏       | 526/2344 [01:35<02:25, 12.50it/s]
Running loglikelihood requests:  25%|██▍       | 580/2344 [01:38<02:11, 13.46it/s]
Running loglikelihood requests:  27%|██▋       | 626/2344 [01:42<02:07, 13.47it/s]
Running loglikelihood requests:  29%|██▉       | 674/2344 [01:45<02:01, 13.69it/s]
Running loglikelihood requests:  31%|███       | 719/2344 [01:49<01:59, 13.65it/s]
Running loglikelihood requests:  33%|███▎      | 765/2344 [01:52<01:55, 13.73it/s]
Running loglikelihood requests:  35%|███▍      | 813/2344 [01:55<01:49, 13.94it/s]
Running loglikelihood requests:  37%|███▋      | 863/2344 [01:58<01:43, 14.32it/s]
Running loglikelihood requests:  39%|███▉      | 914/2344 [02:02<01:37, 14.72it/s]
Running loglikelihood requests:  41%|████      | 959/2344 [02:05<01:35, 14.48it/s]
Running loglikelihood requests:  43%|████▎     | 1005/2344 [02:08<01:33, 14.39it/s]
Running loglikelihood requests:  45%|████▍     | 1052/2344 [02:11<01:29, 14.43it/s]
Running loglikelihood requests:  47%|████▋     | 1110/2344 [02:15<01:19, 15.54it/s]
Running loglikelihood requests:  49%|████▉     | 1159/2344 [02:18<01:16, 15.49it/s]
Running loglikelihood requests:  52%|█████▏    | 1208/2344 [02:21<01:13, 15.48it/s]
Running loglikelihood requests:  54%|█████▎    | 1255/2344 [02:24<01:11, 15.30it/s]
Running loglikelihood requests:  56%|█████▌    | 1303/2344 [02:27<01:08, 15.27it/s]
Running loglikelihood requests:  58%|█████▊    | 1356/2344 [02:30<01:02, 15.76it/s]
Running loglikelihood requests:  60%|█████▉    | 1401/2344 [02:33<01:01, 15.36it/s]
Running loglikelihood requests:  62%|██████▏   | 1454/2344 [02:37<00:56, 15.89it/s]
Running loglikelihood requests:  64%|██████▍   | 1504/2344 [02:40<00:52, 15.99it/s]
Running loglikelihood requests:  66%|██████▋   | 1553/2344 [02:43<00:49, 16.03it/s]
Running loglikelihood requests:  68%|██████▊   | 1601/2344 [02:46<00:46, 15.95it/s]
Running loglikelihood requests:  71%|███████   | 1653/2344 [02:49<00:42, 16.34it/s]
Running loglikelihood requests:  72%|███████▏  | 1698/2344 [02:52<00:40, 15.95it/s]
Running loglikelihood requests:  74%|███████▍  | 1745/2344 [02:55<00:37, 15.89it/s]
Running loglikelihood requests:  76%|███████▋  | 1793/2344 [02:58<00:34, 15.95it/s]
Running loglikelihood requests:  78%|███████▊  | 1840/2344 [03:01<00:31, 15.93it/s]
Running loglikelihood requests:  81%|████████  | 1891/2344 [03:04<00:27, 16.34it/s]
Running loglikelihood requests:  83%|████████▎ | 1938/2344 [03:07<00:24, 16.28it/s]
Running loglikelihood requests:  85%|████████▍ | 1983/2344 [03:09<00:22, 16.07it/s]
Running loglikelihood requests:  87%|████████▋ | 2032/2344 [03:12<00:19, 16.32it/s]
Running loglikelihood requests:  89%|████████▉ | 2089/2344 [03:15<00:14, 17.35it/s]
Running loglikelihood requests:  91%|█████████ | 2137/2344 [03:18<00:12, 17.21it/s]
Running loglikelihood requests:  93%|█████████▎| 2185/2344 [03:21<00:09, 17.11it/s]
Running loglikelihood requests:  96%|█████████▌| 2241/2344 [03:24<00:05, 17.97it/s]
Running loglikelihood requests:  98%|█████████▊| 2295/2344 [03:26<00:02, 19.21it/s]
Running loglikelihood requests: 100%|██████████| 2344/2344 [03:26<00:00, 11.35it/s]
n136-128-154:2217286:2217993 [1] NCCL INFO Channel 00/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217993 [1] NCCL INFO Channel 01/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217993 [1] NCCL INFO Channel 02/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217993 [1] NCCL INFO Channel 03/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217993 [1] NCCL INFO Channel 04/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217993 [1] NCCL INFO Channel 05/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217993 [1] NCCL INFO Channel 06/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217993 [1] NCCL INFO Channel 07/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217993 [1] NCCL INFO Channel 08/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217993 [1] NCCL INFO Channel 09/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217993 [1] NCCL INFO Channel 10/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217993 [1] NCCL INFO Channel 11/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217993 [1] NCCL INFO Channel 12/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217993 [1] NCCL INFO Channel 13/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217993 [1] NCCL INFO Channel 14/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217993 [1] NCCL INFO Channel 15/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217993 [1] NCCL INFO Channel 16/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217993 [1] NCCL INFO Channel 17/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217993 [1] NCCL INFO Channel 18/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217993 [1] NCCL INFO Channel 19/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217993 [1] NCCL INFO Channel 20/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217993 [1] NCCL INFO Channel 21/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217993 [1] NCCL INFO Channel 22/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217993 [1] NCCL INFO Channel 23/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217993 [1] NCCL INFO Channel 24/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217993 [1] NCCL INFO Channel 25/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217993 [1] NCCL INFO Channel 26/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217993 [1] NCCL INFO Channel 27/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217993 [1] NCCL INFO Channel 28/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217993 [1] NCCL INFO Channel 29/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217993 [1] NCCL INFO Channel 30/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2217286:2217993 [1] NCCL INFO Channel 31/1 : 1[7] -> 0[6] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:2217286:2218000 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2217286:2218000 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2217286:2218000 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2217286:2218000 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2217286:2218000 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2217286:2218000 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2217286:2217593 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-09:16:17:59 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 25, batch_size: auto (45)
|    Tasks    |Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|-------------|------:|------|-----:|--------|---|-----:|---|-----:|
|arc_challenge|      1|none  |    25|acc     |↑  |0.2944|±  |0.0133|
|             |       |none  |    25|acc_norm|↑  |0.3251|±  |0.0137|

[rank0]:[W1209 16:18:00.245690467 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:2217285:2218058 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2217285:2218058 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2217285:2218058 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2217285:2217595 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2217285:2218058 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2217285:2218058 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2217285:2218058 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2217286:2217593 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2217286:2218000 [1] NCCL INFO comm 0x1131e060 rank 1 nranks 2 cudaDev 1 busId c9000 - Abort COMPLETE
n136-128-154:2217285:2218058 [0] NCCL INFO comm 0x125cd3d0 rank 0 nranks 2 cudaDev 0 busId c5000 - Abort COMPLETE
任务 arc_challenge 评估完成！

==================================================
开始评估：任务=truthfulqa_mc1 | 少样本数=0 | 模型=Qwen2.5-7B-quantization-fg
输出路径：results2/Qwen2.5-7B-quantization-fg/baseline_BI/truthfulqa_mc1.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-09:16:18:50 INFO     [__main__:440] Selected Tasks: ['truthfulqa_mc1']
2025-12-09:16:18:50 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:16:18:50 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-09:16:18:50 INFO     [__main__:440] Selected Tasks: ['truthfulqa_mc1']
2025-12-09:16:18:50 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:16:18:50 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-09:16:18:51 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:16:18:52 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:16:18:52 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.03it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.04s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.05it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.50s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.47s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:04<00:00,  1.01it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:04<00:00,  1.06s/it]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:04<00:01,  1.49s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:04<00:00,  1.00s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:04<00:00,  1.15s/it]
2025-12-09:16:19:10 INFO     [evaluator:305] num_fewshot has been set to 0 for truthfulqa_mc1 in its config. Manual configuration will be ignored.
2025-12-09:16:19:10 INFO     [api.task:434] Building contexts for truthfulqa_mc1 on rank 0...

  0%|          | 0/409 [00:00<?, ?it/s]
 25%|██▍       | 101/409 [00:00<00:00, 1005.01it/s]
 50%|█████     | 205/409 [00:00<00:00, 1020.39it/s]
 75%|███████▌  | 308/409 [00:00<00:00, 1022.35it/s]
100%|██████████| 409/409 [00:00<00:00, 1020.79it/s]
n136-128-154:2218110:2218110 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2218110:2218110 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2218110:2218110 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:2218110:2218110 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2218110:2218110 [0] NCCL INFO Comm config Blocking set to 1
n136-128-154:2218110:2218336 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2218110:2218336 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2218110:2218336 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2218110:2218336 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2218110:2218336 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2218110:2218336 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2218110:2218336 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2218110:2218336 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:2218110:2218336 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2218110:2218336 [0] NCCL INFO Using network IB
n136-128-154:2218110:2218336 [0] NCCL INFO ncclCommInitRankConfig comm 0xfdabdb0 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0xb0dc6c95e12de2b7 - Init START
2025-12-09:16:19:13 INFO     [evaluator:305] num_fewshot has been set to 0 for truthfulqa_mc1 in its config. Manual configuration will be ignored.
2025-12-09:16:19:13 INFO     [api.task:434] Building contexts for truthfulqa_mc1 on rank 1...

  0%|          | 0/408 [00:00<?, ?it/s]
 25%|██▌       | 102/408 [00:00<00:00, 1016.40it/s]
 50%|█████     | 206/408 [00:00<00:00, 1025.06it/s]
 76%|███████▌  | 309/408 [00:00<00:00, 968.15it/s] 
100%|█████████▉| 407/408 [00:00<00:00, 944.94it/s]
100%|██████████| 408/408 [00:00<00:00, 963.22it/s]
n136-128-154:2218111:2218111 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:2218111:2218111 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2218111:2218111 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2218111:2218111 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2218111:2218111 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:2218111:2218347 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2218111:2218347 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2218111:2218347 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2218111:2218347 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2218111:2218347 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2218111:2218347 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2218111:2218347 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2218111:2218347 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:2218111:2218347 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2218111:2218347 [1] NCCL INFO Using network IB
n136-128-154:2218111:2218347 [1] NCCL INFO ncclCommInitRankConfig comm 0x10923990 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0xb0dc6c95e12de2b7 - Init START
n136-128-154:2218111:2218347 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2218110:2218336 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2218110:2218336 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2218110:2218336 [0] NCCL INFO Retrieving state for IB
n136-128-154:2218110:2218336 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:2218110:2218336 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2218111:2218347 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2218111:2218347 [1] NCCL INFO Retrieving state for IB
n136-128-154:2218111:2218347 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:2218110:2218336 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2218111:2218347 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2218110:2218336 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2218111:2218347 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2218110:2218336 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2218111:2218347 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2218111:2218347 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2218110:2218336 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2218110:2218336 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2218111:2218347 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2218111:2218347 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2218110:2218336 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2218110:2218336 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2218110:2218336 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2218110:2218336 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2218110:2218336 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2218110:2218336 [0] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2218110:2218336 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2218110:2218336 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2218110:2218336 [0] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2218110:2218336 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2218110:2218336 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2218110:2218336 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2218110:2218336 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2218110:2218336 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2218110:2218336 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2218110:2218336 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2218110:2218336 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2218110:2218336 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2218110:2218336 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2218110:2218336 [0] NCCL INFO ==========================================
n136-128-154:2218110:2218336 [0] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2218110:2218336 [0] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2218110:2218336 [0] NCCL INFO Setting affinity for GPU 6 to 33-62,97-126
n136-128-154:2218111:2218347 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2218111:2218347 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2218111:2218347 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2218111:2218347 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2218111:2218347 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2218111:2218347 [1] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2218111:2218347 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2218111:2218347 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2218110:2218336 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2218111:2218347 [1] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2218111:2218347 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2218110:2218336 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218111:2218347 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2218110:2218336 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218111:2218347 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2218110:2218336 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218111:2218347 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2218110:2218336 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218111:2218347 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2218110:2218336 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218111:2218347 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2218110:2218336 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218111:2218347 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2218110:2218336 [0] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218111:2218347 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2218110:2218336 [0] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218111:2218347 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2218110:2218336 [0] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218111:2218347 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2218110:2218336 [0] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218111:2218347 [1] NCCL INFO ==========================================
n136-128-154:2218110:2218336 [0] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218110:2218336 [0] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218111:2218347 [1] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2218111:2218347 [1] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2218111:2218347 [1] NCCL INFO Setting affinity for GPU 7 to 33-62,97-126
n136-128-154:2218110:2218336 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2218110:2218336 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218110:2218336 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218110:2218336 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218110:2218336 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218110:2218336 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218110:2218336 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218110:2218336 [0] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2218110:2218336 [0] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2218110:2218336 [0] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2218110:2218336 [0] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2218110:2218336 [0] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2218110:2218336 [0] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2218111:2218347 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2218111:2218347 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218111:2218347 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218111:2218347 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218111:2218347 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218111:2218347 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218111:2218347 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218111:2218347 [1] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218111:2218347 [1] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218111:2218347 [1] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218111:2218347 [1] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218111:2218347 [1] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218111:2218347 [1] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218111:2218347 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2218111:2218347 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218111:2218347 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218111:2218347 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218111:2218347 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218111:2218347 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218111:2218347 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218111:2218347 [1] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2218111:2218347 [1] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2218111:2218347 [1] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2218111:2218347 [1] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2218111:2218347 [1] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2218111:2218347 [1] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2218110:2218336 [0] NCCL INFO comm 0xfdabdb0 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:2218111:2218347 [1] NCCL INFO comm 0x10923990 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:2218110:2218336 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:2218111:2218347 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:2218110:2218336 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:2218111:2218347 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:2218110:2218336 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:2218111:2218347 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:2218110:2218336 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:2218111:2218347 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:2218110:2218336 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:2218111:2218347 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:2218110:2218336 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:2218111:2218347 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:2218110:2218336 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:2218111:2218347 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:2218110:2218336 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:2218111:2218347 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:2218110:2218336 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:2218111:2218347 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:2218110:2218336 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:2218111:2218347 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:2218110:2218336 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:2218111:2218347 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:2218110:2218336 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:2218111:2218347 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:2218110:2218336 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:2218111:2218347 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:2218110:2218336 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:2218111:2218347 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:2218110:2218336 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:2218111:2218347 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:2218110:2218336 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:2218111:2218347 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:2218110:2218336 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:2218111:2218347 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:2218110:2218336 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:2218111:2218347 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:2218110:2218336 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:2218111:2218347 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:2218110:2218336 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:2218111:2218347 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:2218110:2218336 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:2218111:2218347 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:2218110:2218336 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:2218111:2218347 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:2218110:2218336 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:2218111:2218347 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:2218110:2218336 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:2218111:2218347 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:2218110:2218336 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:2218110:2218336 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:2218111:2218347 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:2218110:2218336 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:2218111:2218347 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:2218110:2218336 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:2218111:2218347 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:2218110:2218336 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:2218111:2218347 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:2218110:2218336 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:2218111:2218347 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:2218110:2218336 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:2218111:2218347 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:2218110:2218336 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:2218111:2218347 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:2218110:2218336 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:2218111:2218347 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:2218110:2218336 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:2218111:2218347 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:2218110:2218336 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:2218111:2218347 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:2218110:2218336 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:2218111:2218347 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:2218110:2218336 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:2218111:2218347 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:2218110:2218336 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:2218111:2218347 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:2218110:2218336 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:2218111:2218347 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:2218110:2218336 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:2218111:2218347 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:2218110:2218336 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:2218111:2218347 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:2218110:2218336 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:2218111:2218347 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:2218110:2218336 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:2218111:2218347 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:2218110:2218336 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:2218111:2218347 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:2218110:2218336 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:2218111:2218347 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:2218110:2218336 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:2218111:2218347 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:2218110:2218336 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:2218111:2218347 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:2218110:2218336 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:2218111:2218347 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:2218111:2218347 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:2218110:2218336 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:2218111:2218347 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:2218110:2218336 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:2218110:2218336 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:2218111:2218347 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2218110:2218336 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:2218110:2218336 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:2218110:2218336 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:2218110:2218336 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:2218110:2218336 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:2218110:2218336 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:2218110:2218336 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:2218110:2218336 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:2218110:2218336 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:2218110:2218336 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:2218110:2218336 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:2218110:2218336 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:2218110:2218336 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:2218110:2218336 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:2218110:2218336 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:2218110:2218336 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:2218110:2218336 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:2218110:2218336 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:2218110:2218336 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:2218110:2218336 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:2218110:2218336 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:2218110:2218336 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:2218110:2218336 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2218111:2218347 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2218111:2218355 [1] NCCL INFO [Proxy Service] Device 1 CPU core 58
n136-128-154:2218111:2218356 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 124
n136-128-154:2218110:2218336 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2218110:2218336 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:2218110:2218357 [0] NCCL INFO [Proxy Service] Device 0 CPU core 98
n136-128-154:2218110:2218358 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 35
n136-128-154:2218111:2218347 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2218111:2218347 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2218110:2218336 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2218110:2218336 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2218110:2218336 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:2218111:2218347 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2218111:2218347 [1] NCCL INFO ncclCommInitRankConfig comm 0x10923990 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0xb0dc6c95e12de2b7 - Init COMPLETE
n136-128-154:2218111:2218347 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.50 (kernels 0.12, alloc 0.17, bootstrap 0.00, allgathers 0.00, topo 0.08, graphs 0.00, connections 0.03, rest 0.10)
n136-128-154:2218110:2218336 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2218110:2218336 [0] NCCL INFO ncclCommInitRankConfig comm 0xfdabdb0 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0xb0dc6c95e12de2b7 - Init COMPLETE
n136-128-154:2218110:2218336 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 3.21 (kernels 0.11, alloc 0.11, bootstrap 2.78, allgathers 0.00, topo 0.08, graphs 0.00, connections 0.02, rest 0.10)
n136-128-154:2218111:2218359 [1] NCCL INFO Channel 00/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218359 [1] NCCL INFO Channel 01/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218359 [1] NCCL INFO Channel 02/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218359 [1] NCCL INFO Channel 03/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218359 [1] NCCL INFO Channel 04/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218359 [1] NCCL INFO Channel 05/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218359 [1] NCCL INFO Channel 06/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218110:2218360 [0] NCCL INFO Channel 00/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218111:2218359 [1] NCCL INFO Channel 07/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218110:2218360 [0] NCCL INFO Channel 01/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218111:2218359 [1] NCCL INFO Channel 08/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218110:2218360 [0] NCCL INFO Channel 02/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218111:2218359 [1] NCCL INFO Channel 09/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218110:2218360 [0] NCCL INFO Channel 03/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218111:2218359 [1] NCCL INFO Channel 10/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218110:2218360 [0] NCCL INFO Channel 04/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218111:2218359 [1] NCCL INFO Channel 11/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218110:2218360 [0] NCCL INFO Channel 05/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218111:2218359 [1] NCCL INFO Channel 12/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218110:2218360 [0] NCCL INFO Channel 06/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218111:2218359 [1] NCCL INFO Channel 13/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218110:2218360 [0] NCCL INFO Channel 07/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218111:2218359 [1] NCCL INFO Channel 14/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218110:2218360 [0] NCCL INFO Channel 08/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218111:2218359 [1] NCCL INFO Channel 15/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218110:2218360 [0] NCCL INFO Channel 09/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218111:2218359 [1] NCCL INFO Channel 16/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218110:2218360 [0] NCCL INFO Channel 10/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218111:2218359 [1] NCCL INFO Channel 17/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218110:2218360 [0] NCCL INFO Channel 11/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218111:2218359 [1] NCCL INFO Channel 18/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218110:2218360 [0] NCCL INFO Channel 12/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218111:2218359 [1] NCCL INFO Channel 19/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218110:2218360 [0] NCCL INFO Channel 13/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218111:2218359 [1] NCCL INFO Channel 20/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218110:2218360 [0] NCCL INFO Channel 14/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218111:2218359 [1] NCCL INFO Channel 21/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218110:2218360 [0] NCCL INFO Channel 15/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218111:2218359 [1] NCCL INFO Channel 22/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218110:2218360 [0] NCCL INFO Channel 16/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218111:2218359 [1] NCCL INFO Channel 23/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218110:2218360 [0] NCCL INFO Channel 17/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218110:2218360 [0] NCCL INFO Channel 18/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218110:2218360 [0] NCCL INFO Channel 19/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218110:2218360 [0] NCCL INFO Channel 20/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218110:2218360 [0] NCCL INFO Channel 21/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218110:2218360 [0] NCCL INFO Channel 22/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218110:2218360 [0] NCCL INFO Channel 23/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218110:2218360 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:2218111:2218359 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-09:16:19:14 INFO     [evaluator:559] Running loglikelihood requests
2025-12-09:16:19:14 INFO     [evaluator:559] Running loglikelihood requests
Passed argument batch_size = auto:1. Detecting largest batch size

Running loglikelihood requests:   0%|          | 0/2066 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 64
Determined largest batch size: 64

Running loglikelihood requests:   0%|          | 1/2066 [00:06<3:57:36,  6.90s/it]
Running loglikelihood requests:   3%|▎         | 65/2066 [00:08<03:05, 10.78it/s] 
Running loglikelihood requests:   6%|▌         | 129/2066 [00:09<01:34, 20.53it/s]
Running loglikelihood requests:   9%|▉         | 193/2066 [00:10<01:04, 29.07it/s]
Running loglikelihood requests:  12%|█▏        | 257/2066 [00:11<00:49, 36.23it/s]
Running loglikelihood requests:  16%|█▌        | 321/2066 [00:12<00:41, 42.01it/s]
Running loglikelihood requests:  19%|█▊        | 385/2066 [00:13<00:36, 46.49it/s]
Running loglikelihood requests:  22%|██▏       | 449/2066 [00:14<00:32, 50.10it/s]
Running loglikelihood requests:  25%|██▍       | 513/2066 [00:15<00:29, 52.80it/s]
Running loglikelihood requests:  28%|██▊       | 577/2066 [00:16<00:27, 54.82it/s]
Running loglikelihood requests:  31%|███       | 641/2066 [00:17<00:25, 56.37it/s]
Running loglikelihood requests:  34%|███▍      | 705/2066 [00:18<00:23, 57.59it/s]
Running loglikelihood requests:  37%|███▋      | 769/2066 [00:19<00:22, 58.45it/s]
Running loglikelihood requests:  40%|████      | 833/2066 [00:21<00:20, 59.02it/s]
Running loglikelihood requests:  43%|████▎     | 897/2066 [00:22<00:19, 59.55it/s]
Running loglikelihood requests:  47%|████▋     | 961/2066 [00:23<00:18, 59.95it/s]
Running loglikelihood requests:  50%|████▉     | 1025/2066 [00:24<00:17, 60.25it/s]
Running loglikelihood requests:  53%|█████▎    | 1089/2066 [00:25<00:16, 60.47it/s]
Running loglikelihood requests:  56%|█████▌    | 1153/2066 [00:26<00:15, 60.80it/s]
Running loglikelihood requests:  59%|█████▉    | 1217/2066 [00:27<00:13, 60.92it/s]
Running loglikelihood requests:  62%|██████▏   | 1281/2066 [00:28<00:12, 61.07it/s]
Running loglikelihood requests:  65%|██████▌   | 1345/2066 [00:29<00:11, 61.17it/s]
Running loglikelihood requests:  68%|██████▊   | 1409/2066 [00:30<00:10, 61.41it/s]
Running loglikelihood requests:  71%|███████▏  | 1473/2066 [00:31<00:09, 61.52it/s]
Running loglikelihood requests:  74%|███████▍  | 1537/2066 [00:32<00:08, 61.65it/s]
Running loglikelihood requests:  77%|███████▋  | 1601/2066 [00:33<00:07, 61.70it/s]
Running loglikelihood requests:  81%|████████  | 1665/2066 [00:34<00:06, 61.83it/s]
Running loglikelihood requests:  84%|████████▎ | 1729/2066 [00:35<00:05, 61.93it/s]
Running loglikelihood requests:  87%|████████▋ | 1793/2066 [00:36<00:04, 62.02it/s]
Running loglikelihood requests:  90%|████████▉ | 1857/2066 [00:37<00:03, 62.14it/s]
Running loglikelihood requests:  93%|█████████▎| 1921/2066 [00:38<00:02, 62.28it/s]
Running loglikelihood requests:  96%|█████████▌| 1985/2066 [00:39<00:01, 62.79it/s]
Running loglikelihood requests:  99%|█████████▉| 2049/2066 [00:39<00:00, 79.70it/s]
Running loglikelihood requests: 100%|██████████| 2066/2066 [00:39<00:00, 51.67it/s]
n136-128-154:2218111:2218404 [1] NCCL INFO Channel 00/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218404 [1] NCCL INFO Channel 01/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218404 [1] NCCL INFO Channel 02/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218404 [1] NCCL INFO Channel 03/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218404 [1] NCCL INFO Channel 04/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218404 [1] NCCL INFO Channel 05/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218404 [1] NCCL INFO Channel 06/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218404 [1] NCCL INFO Channel 07/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218404 [1] NCCL INFO Channel 08/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218404 [1] NCCL INFO Channel 09/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218404 [1] NCCL INFO Channel 10/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218404 [1] NCCL INFO Channel 11/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218404 [1] NCCL INFO Channel 12/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218404 [1] NCCL INFO Channel 13/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218404 [1] NCCL INFO Channel 14/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218404 [1] NCCL INFO Channel 15/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218404 [1] NCCL INFO Channel 16/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218404 [1] NCCL INFO Channel 17/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218404 [1] NCCL INFO Channel 18/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218404 [1] NCCL INFO Channel 19/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218404 [1] NCCL INFO Channel 20/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218404 [1] NCCL INFO Channel 21/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218404 [1] NCCL INFO Channel 22/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218404 [1] NCCL INFO Channel 23/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218404 [1] NCCL INFO Channel 24/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218404 [1] NCCL INFO Channel 25/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218404 [1] NCCL INFO Channel 26/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218404 [1] NCCL INFO Channel 27/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218404 [1] NCCL INFO Channel 28/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218404 [1] NCCL INFO Channel 29/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218404 [1] NCCL INFO Channel 30/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218111:2218404 [1] NCCL INFO Channel 31/1 : 1[7] -> 0[6] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:2218111:2218409 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2218111:2218409 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2218111:2218409 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2218111:2218409 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2218111:2218409 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2218111:2218409 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2218111:2218355 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-09:16:20:05 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: auto (64)
|    Tasks     |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|--------------|------:|------|-----:|------|---|-----:|---|-----:|
|truthfulqa_mc1|      2|none  |     0|acc   |↑  |0.2705|±  |0.0156|

[rank0]:[W1209 16:20:06.430630291 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:2218110:2218459 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2218110:2218459 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2218110:2218459 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2218110:2218357 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2218110:2218459 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2218111:2218355 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2218110:2218459 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2218110:2218459 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2218111:2218409 [1] NCCL INFO comm 0x10923990 rank 1 nranks 2 cudaDev 1 busId c9000 - Abort COMPLETE
n136-128-154:2218110:2218459 [0] NCCL INFO comm 0xfdabdb0 rank 0 nranks 2 cudaDev 0 busId c5000 - Abort COMPLETE
任务 truthfulqa_mc1 评估完成！

==================================================
开始评估：任务=piqa | 少样本数=0 | 模型=Qwen2.5-7B-quantization-fg
输出路径：results2/Qwen2.5-7B-quantization-fg/baseline_BI/piqa.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-09:16:20:55 INFO     [__main__:440] Selected Tasks: ['piqa']
2025-12-09:16:20:55 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:16:20:55 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-09:16:20:55 INFO     [__main__:440] Selected Tasks: ['piqa']
2025-12-09:16:20:55 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:16:20:55 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-09:16:20:56 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:16:20:56 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:16:20:57 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.11it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.04s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.09it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.06s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.14it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.59it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.38it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.02s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.39it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.20it/s]
2025-12-09:16:21:13 WARNING  [evaluator:309] Overwriting default num_fewshot of piqa from None to 0
2025-12-09:16:21:13 INFO     [api.task:434] Building contexts for piqa on rank 1...

  0%|          | 0/919 [00:00<?, ?it/s]
 16%|█▌        | 149/919 [00:00<00:00, 1489.53it/s]
 32%|███▏      | 298/919 [00:00<00:00, 1395.26it/s]
 49%|████▉     | 450/919 [00:00<00:00, 1449.99it/s]
 66%|██████▌   | 605/919 [00:00<00:00, 1488.11it/s]
 83%|████████▎ | 761/919 [00:00<00:00, 1510.34it/s]
 99%|█████████▉| 913/919 [00:00<00:00, 1507.41it/s]
100%|██████████| 919/919 [00:00<00:00, 1487.82it/s]
2025-12-09:16:21:15 WARNING  [evaluator:309] Overwriting default num_fewshot of piqa from None to 0
2025-12-09:16:21:15 INFO     [api.task:434] Building contexts for piqa on rank 0...

  0%|          | 0/919 [00:00<?, ?it/s]
 17%|█▋        | 153/919 [00:00<00:00, 1523.21it/s]
 33%|███▎      | 306/919 [00:00<00:00, 1503.28it/s]
 50%|█████     | 461/919 [00:00<00:00, 1520.03it/s]
 67%|██████▋   | 617/919 [00:00<00:00, 1531.85it/s]
 84%|████████▍ | 773/919 [00:00<00:00, 1540.57it/s]
100%|██████████| 919/919 [00:00<00:00, 1532.89it/s]
n136-128-154:2218505:2218505 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2218505:2218505 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2218505:2218505 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:2218505:2218505 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2218505:2218505 [0] NCCL INFO Comm config Blocking set to 1
n136-128-154:2218506:2218506 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:2218506:2218506 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2218506:2218506 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2218506:2218506 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2218506:2218506 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:2218505:2218684 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2218505:2218684 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2218506:2218685 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2218506:2218685 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2218505:2218684 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2218505:2218684 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2218505:2218684 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2218506:2218685 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2218506:2218685 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2218506:2218685 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2218506:2218685 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2218506:2218685 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2218506:2218685 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:2218505:2218684 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2218505:2218684 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2218505:2218684 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:2218506:2218685 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2218506:2218685 [1] NCCL INFO Using network IB
n136-128-154:2218505:2218684 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2218505:2218684 [0] NCCL INFO Using network IB
n136-128-154:2218506:2218685 [1] NCCL INFO ncclCommInitRankConfig comm 0xfc899c0 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0x29a160749ad47b2e - Init START
n136-128-154:2218505:2218684 [0] NCCL INFO ncclCommInitRankConfig comm 0xe53b9c0 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0x29a160749ad47b2e - Init START
n136-128-154:2218505:2218684 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2218506:2218685 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2218506:2218685 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2218506:2218685 [1] NCCL INFO Retrieving state for IB
n136-128-154:2218506:2218685 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:2218506:2218685 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2218505:2218684 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2218505:2218684 [0] NCCL INFO Retrieving state for IB
n136-128-154:2218505:2218684 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:2218506:2218685 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2218505:2218684 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2218506:2218685 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2218505:2218684 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2218506:2218685 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2218505:2218684 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2218505:2218684 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2218505:2218684 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2218505:2218684 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2218506:2218685 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2218506:2218685 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2218505:2218684 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2218505:2218684 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2218505:2218684 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2218505:2218684 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2218505:2218684 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2218505:2218684 [0] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2218505:2218684 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2218505:2218684 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2218505:2218684 [0] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2218505:2218684 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2218505:2218684 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2218505:2218684 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2218505:2218684 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2218505:2218684 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2218505:2218684 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2218505:2218684 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2218505:2218684 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2218505:2218684 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2218505:2218684 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2218505:2218684 [0] NCCL INFO ==========================================
n136-128-154:2218505:2218684 [0] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2218505:2218684 [0] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2218505:2218684 [0] NCCL INFO Setting affinity for GPU 6 to 33-62,97-126
n136-128-154:2218506:2218685 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2218506:2218685 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2218506:2218685 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2218506:2218685 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2218506:2218685 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2218506:2218685 [1] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2218506:2218685 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2218506:2218685 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2218506:2218685 [1] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2218506:2218685 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2218506:2218685 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2218506:2218685 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2218506:2218685 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2218506:2218685 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2218506:2218685 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2218506:2218685 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2218506:2218685 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2218506:2218685 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2218506:2218685 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2218506:2218685 [1] NCCL INFO ==========================================
n136-128-154:2218506:2218685 [1] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2218506:2218685 [1] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2218506:2218685 [1] NCCL INFO Setting affinity for GPU 7 to 33-62,97-126
n136-128-154:2218506:2218685 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2218506:2218685 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218506:2218685 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218506:2218685 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218506:2218685 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218506:2218685 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218506:2218685 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218506:2218685 [1] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218506:2218685 [1] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218506:2218685 [1] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218506:2218685 [1] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218506:2218685 [1] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218506:2218685 [1] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218506:2218685 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2218506:2218685 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218506:2218685 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218506:2218685 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218506:2218685 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218506:2218685 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218506:2218685 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218506:2218685 [1] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2218506:2218685 [1] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2218506:2218685 [1] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2218506:2218685 [1] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2218506:2218685 [1] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2218506:2218685 [1] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2218505:2218684 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2218505:2218684 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218505:2218684 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218505:2218684 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218505:2218684 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218505:2218684 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218505:2218684 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218505:2218684 [0] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218505:2218684 [0] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218505:2218684 [0] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218505:2218684 [0] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218505:2218684 [0] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218505:2218684 [0] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218505:2218684 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2218505:2218684 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218505:2218684 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218505:2218684 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218505:2218684 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218505:2218684 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218505:2218684 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218505:2218684 [0] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2218505:2218684 [0] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2218505:2218684 [0] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2218505:2218684 [0] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2218505:2218684 [0] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2218505:2218684 [0] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2218505:2218684 [0] NCCL INFO comm 0xe53b9c0 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:2218506:2218685 [1] NCCL INFO comm 0xfc899c0 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:2218505:2218684 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:2218505:2218684 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:2218506:2218685 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:2218505:2218684 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:2218506:2218685 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:2218505:2218684 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:2218506:2218685 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:2218505:2218684 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:2218506:2218685 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:2218505:2218684 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:2218506:2218685 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:2218505:2218684 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:2218506:2218685 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:2218505:2218684 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:2218506:2218685 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:2218505:2218684 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:2218506:2218685 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:2218505:2218684 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:2218506:2218685 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:2218505:2218684 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:2218506:2218685 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:2218505:2218684 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:2218506:2218685 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:2218505:2218684 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:2218506:2218685 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:2218505:2218684 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:2218506:2218685 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:2218505:2218684 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:2218506:2218685 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:2218505:2218684 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:2218506:2218685 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:2218505:2218684 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:2218506:2218685 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:2218505:2218684 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:2218506:2218685 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:2218505:2218684 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:2218506:2218685 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:2218505:2218684 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:2218506:2218685 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:2218505:2218684 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:2218506:2218685 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:2218505:2218684 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:2218506:2218685 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:2218505:2218684 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:2218506:2218685 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:2218505:2218684 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:2218506:2218685 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:2218506:2218685 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:2218505:2218684 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:2218505:2218684 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:2218505:2218684 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:2218505:2218684 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:2218506:2218685 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:2218505:2218684 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:2218506:2218685 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:2218505:2218684 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:2218506:2218685 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:2218505:2218684 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:2218506:2218685 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:2218505:2218684 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:2218506:2218685 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:2218505:2218684 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:2218506:2218685 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:2218505:2218684 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:2218506:2218685 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:2218505:2218684 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:2218506:2218685 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:2218505:2218684 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:2218506:2218685 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:2218505:2218684 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:2218506:2218685 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:2218505:2218684 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:2218506:2218685 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:2218505:2218684 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:2218506:2218685 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:2218505:2218684 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:2218506:2218685 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:2218505:2218684 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:2218506:2218685 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:2218505:2218684 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:2218506:2218685 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:2218505:2218684 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:2218506:2218685 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:2218505:2218684 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:2218506:2218685 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:2218505:2218684 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:2218506:2218685 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:2218505:2218684 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:2218506:2218685 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:2218505:2218684 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:2218506:2218685 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:2218505:2218684 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:2218506:2218685 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:2218505:2218684 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:2218506:2218685 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:2218505:2218684 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:2218506:2218685 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:2218505:2218684 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:2218506:2218685 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:2218505:2218684 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:2218506:2218685 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:2218505:2218684 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:2218505:2218684 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:2218506:2218685 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2218505:2218684 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:2218505:2218684 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:2218505:2218684 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:2218505:2218684 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:2218505:2218684 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:2218505:2218684 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:2218505:2218684 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:2218505:2218684 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:2218505:2218684 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:2218505:2218684 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:2218505:2218684 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:2218505:2218684 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:2218505:2218684 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:2218505:2218684 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:2218505:2218684 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:2218505:2218684 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:2218505:2218684 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:2218505:2218684 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:2218505:2218684 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:2218505:2218684 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2218506:2218685 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2218506:2218697 [1] NCCL INFO [Proxy Service] Device 1 CPU core 100
n136-128-154:2218506:2218698 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 38
n136-128-154:2218505:2218684 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2218505:2218684 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:2218505:2218699 [0] NCCL INFO [Proxy Service] Device 0 CPU core 103
n136-128-154:2218505:2218700 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 40
n136-128-154:2218506:2218685 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2218506:2218685 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2218505:2218684 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2218505:2218684 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2218505:2218684 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:2218506:2218685 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2218506:2218685 [1] NCCL INFO ncclCommInitRankConfig comm 0xfc899c0 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0x29a160749ad47b2e - Init COMPLETE
n136-128-154:2218506:2218685 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.51 (kernels 0.12, alloc 0.25, bootstrap 0.00, allgathers 0.01, topo 0.07, graphs 0.00, connections 0.01, rest 0.06)
n136-128-154:2218505:2218684 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2218505:2218684 [0] NCCL INFO ncclCommInitRankConfig comm 0xe53b9c0 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0x29a160749ad47b2e - Init COMPLETE
n136-128-154:2218505:2218684 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.52 (kernels 0.11, alloc 0.25, bootstrap 0.00, allgathers 0.00, topo 0.07, graphs 0.00, connections 0.01, rest 0.06)
n136-128-154:2218506:2218701 [1] NCCL INFO Channel 00/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218701 [1] NCCL INFO Channel 01/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218701 [1] NCCL INFO Channel 02/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218701 [1] NCCL INFO Channel 03/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218701 [1] NCCL INFO Channel 04/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218701 [1] NCCL INFO Channel 05/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218701 [1] NCCL INFO Channel 06/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218701 [1] NCCL INFO Channel 07/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218701 [1] NCCL INFO Channel 08/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218505:2218702 [0] NCCL INFO Channel 00/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218506:2218701 [1] NCCL INFO Channel 09/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218505:2218702 [0] NCCL INFO Channel 01/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218506:2218701 [1] NCCL INFO Channel 10/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218505:2218702 [0] NCCL INFO Channel 02/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218506:2218701 [1] NCCL INFO Channel 11/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218505:2218702 [0] NCCL INFO Channel 03/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218506:2218701 [1] NCCL INFO Channel 12/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218505:2218702 [0] NCCL INFO Channel 04/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218506:2218701 [1] NCCL INFO Channel 13/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218505:2218702 [0] NCCL INFO Channel 05/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218506:2218701 [1] NCCL INFO Channel 14/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218505:2218702 [0] NCCL INFO Channel 06/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218506:2218701 [1] NCCL INFO Channel 15/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218505:2218702 [0] NCCL INFO Channel 07/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218506:2218701 [1] NCCL INFO Channel 16/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218505:2218702 [0] NCCL INFO Channel 08/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218506:2218701 [1] NCCL INFO Channel 17/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218505:2218702 [0] NCCL INFO Channel 09/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218506:2218701 [1] NCCL INFO Channel 18/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218505:2218702 [0] NCCL INFO Channel 10/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218506:2218701 [1] NCCL INFO Channel 19/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218505:2218702 [0] NCCL INFO Channel 11/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218506:2218701 [1] NCCL INFO Channel 20/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218505:2218702 [0] NCCL INFO Channel 12/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218506:2218701 [1] NCCL INFO Channel 21/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218505:2218702 [0] NCCL INFO Channel 13/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218506:2218701 [1] NCCL INFO Channel 22/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218505:2218702 [0] NCCL INFO Channel 14/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218506:2218701 [1] NCCL INFO Channel 23/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218505:2218702 [0] NCCL INFO Channel 15/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218505:2218702 [0] NCCL INFO Channel 16/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218505:2218702 [0] NCCL INFO Channel 17/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218505:2218702 [0] NCCL INFO Channel 18/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218505:2218702 [0] NCCL INFO Channel 19/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218505:2218702 [0] NCCL INFO Channel 20/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218505:2218702 [0] NCCL INFO Channel 21/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218505:2218702 [0] NCCL INFO Channel 22/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218505:2218702 [0] NCCL INFO Channel 23/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218506:2218701 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:2218505:2218702 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-09:16:21:16 INFO     [evaluator:559] Running loglikelihood requests
2025-12-09:16:21:16 INFO     [evaluator:559] Running loglikelihood requests
Passed argument batch_size = auto:1. Detecting largest batch size

Running loglikelihood requests:   0%|          | 0/1838 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 64
Determined largest batch size: 64

Running loglikelihood requests:   0%|          | 1/1838 [00:07<3:51:36,  7.56s/it]
Running loglikelihood requests:   4%|▎         | 65/1838 [00:08<02:43, 10.85it/s] 
Running loglikelihood requests:   7%|▋         | 129/1838 [00:08<01:14, 22.92it/s]
Running loglikelihood requests:  11%|█         | 194/1838 [00:09<00:45, 36.43it/s]
Running loglikelihood requests:  14%|█▍        | 258/1838 [00:09<00:31, 50.57it/s]
Running loglikelihood requests:  18%|█▊        | 322/1838 [00:10<00:23, 65.18it/s]
Running loglikelihood requests:  21%|██        | 386/1838 [00:10<00:18, 79.41it/s]
Running loglikelihood requests:  25%|██▍       | 451/1838 [00:11<00:14, 93.21it/s]
Running loglikelihood requests:  28%|██▊       | 515/1838 [00:11<00:12, 105.73it/s]
Running loglikelihood requests:  32%|███▏      | 579/1838 [00:12<00:10, 116.61it/s]
Running loglikelihood requests:  35%|███▍      | 643/1838 [00:12<00:09, 125.96it/s]
Running loglikelihood requests:  38%|███▊      | 707/1838 [00:12<00:08, 132.69it/s]
Running loglikelihood requests:  42%|████▏     | 771/1838 [00:13<00:07, 140.07it/s]
Running loglikelihood requests:  45%|████▌     | 835/1838 [00:13<00:06, 145.68it/s]
Running loglikelihood requests:  49%|████▉     | 899/1838 [00:14<00:06, 151.10it/s]
Running loglikelihood requests:  52%|█████▏    | 964/1838 [00:14<00:05, 155.33it/s]
Running loglikelihood requests:  56%|█████▌    | 1028/1838 [00:14<00:05, 159.84it/s]
Running loglikelihood requests:  59%|█████▉    | 1092/1838 [00:15<00:04, 163.56it/s]
Running loglikelihood requests:  63%|██████▎   | 1157/1838 [00:15<00:04, 167.30it/s]
Running loglikelihood requests:  66%|██████▋   | 1221/1838 [00:16<00:03, 170.96it/s]
Running loglikelihood requests:  70%|██████▉   | 1286/1838 [00:16<00:03, 173.69it/s]
Running loglikelihood requests:  73%|███████▎  | 1350/1838 [00:16<00:02, 177.10it/s]
Running loglikelihood requests:  77%|███████▋  | 1414/1838 [00:17<00:02, 179.54it/s]
Running loglikelihood requests:  80%|████████  | 1478/1838 [00:17<00:01, 182.88it/s]
Running loglikelihood requests:  84%|████████▍ | 1542/1838 [00:17<00:01, 185.22it/s]
Running loglikelihood requests:  87%|████████▋ | 1606/1838 [00:18<00:01, 187.71it/s]
Running loglikelihood requests:  91%|█████████ | 1670/1838 [00:18<00:00, 191.72it/s]
Running loglikelihood requests:  94%|█████████▍| 1734/1838 [00:18<00:00, 196.28it/s]
Running loglikelihood requests:  98%|█████████▊| 1799/1838 [00:18<00:00, 222.64it/s]
Running loglikelihood requests: 100%|██████████| 1838/1838 [00:18<00:00, 97.30it/s] 
n136-128-154:2218506:2218723 [1] NCCL INFO Channel 00/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218723 [1] NCCL INFO Channel 01/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218723 [1] NCCL INFO Channel 02/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218723 [1] NCCL INFO Channel 03/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218723 [1] NCCL INFO Channel 04/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218723 [1] NCCL INFO Channel 05/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218723 [1] NCCL INFO Channel 06/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218723 [1] NCCL INFO Channel 07/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218723 [1] NCCL INFO Channel 08/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218723 [1] NCCL INFO Channel 09/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218723 [1] NCCL INFO Channel 10/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218723 [1] NCCL INFO Channel 11/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218723 [1] NCCL INFO Channel 12/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218723 [1] NCCL INFO Channel 13/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218723 [1] NCCL INFO Channel 14/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218723 [1] NCCL INFO Channel 15/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218723 [1] NCCL INFO Channel 16/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218723 [1] NCCL INFO Channel 17/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218723 [1] NCCL INFO Channel 18/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218723 [1] NCCL INFO Channel 19/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218723 [1] NCCL INFO Channel 20/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218723 [1] NCCL INFO Channel 21/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218723 [1] NCCL INFO Channel 22/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218723 [1] NCCL INFO Channel 23/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218723 [1] NCCL INFO Channel 24/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218723 [1] NCCL INFO Channel 25/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218723 [1] NCCL INFO Channel 26/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218723 [1] NCCL INFO Channel 27/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218723 [1] NCCL INFO Channel 28/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218723 [1] NCCL INFO Channel 29/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218723 [1] NCCL INFO Channel 30/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218506:2218723 [1] NCCL INFO Channel 31/1 : 1[7] -> 0[6] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:2218506:2218729 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2218506:2218729 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2218506:2218729 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2218506:2218729 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2218506:2218729 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2218506:2218729 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2218506:2218697 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-09:16:21:45 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: auto (64)
|Tasks|Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|-----|------:|------|-----:|--------|---|-----:|---|-----:|
|piqa |      1|none  |     0|acc     |↑  |0.7067|±  |0.0106|
|     |       |none  |     0|acc_norm|↑  |0.7073|±  |0.0106|

[rank0]:[W1209 16:21:46.735703412 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:2218505:2218778 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2218505:2218778 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2218505:2218778 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2218505:2218699 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2218505:2218778 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2218505:2218778 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2218505:2218778 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2218506:2218697 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2218506:2218729 [1] NCCL INFO comm 0xfc899c0 rank 1 nranks 2 cudaDev 1 busId c9000 - Abort COMPLETE
n136-128-154:2218505:2218778 [0] NCCL INFO comm 0xe53b9c0 rank 0 nranks 2 cudaDev 0 busId c5000 - Abort COMPLETE
任务 piqa 评估完成！

==================================================
开始评估：任务=hellaswag | 少样本数=10 | 模型=Qwen2.5-7B-quantization-fg
输出路径：results2/Qwen2.5-7B-quantization-fg/baseline_BI/hellaswag.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-09:16:22:36 INFO     [__main__:440] Selected Tasks: ['hellaswag']
2025-12-09:16:22:36 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:16:22:36 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-09:16:22:36 INFO     [__main__:440] Selected Tasks: ['hellaswag']
2025-12-09:16:22:36 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:16:22:36 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-09:16:22:37 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:16:22:37 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:16:22:37 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.09it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.06s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.09it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.08s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.15it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.62it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.39it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.02s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.39it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.19it/s]
2025-12-09:16:22:56 WARNING  [evaluator:309] Overwriting default num_fewshot of hellaswag from None to 10
2025-12-09:16:22:56 INFO     [api.task:434] Building contexts for hellaswag on rank 1...
2025-12-09:16:22:57 WARNING  [evaluator:309] Overwriting default num_fewshot of hellaswag from None to 10
2025-12-09:16:22:57 INFO     [api.task:434] Building contexts for hellaswag on rank 0...

  0%|          | 0/5021 [00:00<?, ?it/s]
  0%|          | 19/5021 [00:00<00:26, 189.03it/s]
  1%|          | 39/5021 [00:00<00:25, 194.72it/s]
  1%|          | 59/5021 [00:00<00:25, 196.97it/s]
  2%|▏         | 79/5021 [00:00<00:24, 198.04it/s]
  2%|▏         | 99/5021 [00:00<00:24, 198.51it/s]
  2%|▏         | 119/5021 [00:00<00:25, 193.89it/s]
  3%|▎         | 139/5021 [00:00<00:25, 194.65it/s]
  3%|▎         | 159/5021 [00:00<00:24, 195.93it/s]
  4%|▎         | 179/5021 [00:00<00:24, 196.47it/s]
  4%|▍         | 199/5021 [00:01<00:24, 196.74it/s]
  0%|          | 0/5021 [00:00<?, ?it/s]
  4%|▍         | 219/5021 [00:01<00:24, 196.66it/s]
  0%|          | 19/5021 [00:00<00:26, 189.53it/s]
  5%|▍         | 239/5021 [00:01<00:24, 196.91it/s]
  1%|          | 39/5021 [00:00<00:25, 191.99it/s]
  5%|▌         | 259/5021 [00:01<00:24, 195.14it/s]
  1%|          | 59/5021 [00:00<00:25, 194.50it/s]
  6%|▌         | 279/5021 [00:01<00:24, 195.47it/s]
  2%|▏         | 79/5021 [00:00<00:25, 196.09it/s]
  6%|▌         | 299/5021 [00:01<00:24, 195.74it/s]
  2%|▏         | 99/5021 [00:00<00:24, 197.07it/s]
  6%|▋         | 319/5021 [00:01<00:23, 196.18it/s]
  2%|▏         | 119/5021 [00:00<00:25, 195.23it/s]
  7%|▋         | 339/5021 [00:01<00:23, 196.53it/s]
  3%|▎         | 139/5021 [00:00<00:24, 195.63it/s]
  7%|▋         | 359/5021 [00:01<00:23, 196.54it/s]
  3%|▎         | 159/5021 [00:00<00:24, 195.74it/s]
  8%|▊         | 379/5021 [00:01<00:23, 194.50it/s]
  4%|▎         | 179/5021 [00:00<00:24, 196.22it/s]
  8%|▊         | 399/5021 [00:02<00:23, 194.04it/s]
  4%|▍         | 199/5021 [00:01<00:24, 196.17it/s]
  8%|▊         | 419/5021 [00:02<00:23, 193.77it/s]
  4%|▍         | 219/5021 [00:01<00:24, 194.32it/s]
  9%|▊         | 439/5021 [00:02<00:23, 193.44it/s]
  5%|▍         | 239/5021 [00:01<00:24, 194.69it/s]
  9%|▉         | 459/5021 [00:02<00:23, 192.94it/s]
  5%|▌         | 259/5021 [00:01<00:24, 195.28it/s]
 10%|▉         | 479/5021 [00:02<00:23, 192.46it/s]
  6%|▌         | 279/5021 [00:01<00:24, 196.15it/s]
  6%|▌         | 299/5021 [00:01<00:24, 196.69it/s]
 10%|▉         | 499/5021 [00:02<00:23, 192.50it/s]
  6%|▋         | 319/5021 [00:01<00:24, 195.23it/s]
 10%|█         | 519/5021 [00:02<00:23, 190.47it/s]
  7%|▋         | 339/5021 [00:01<00:23, 195.28it/s]
 11%|█         | 539/5021 [00:02<00:23, 190.95it/s]
  7%|▋         | 359/5021 [00:01<00:23, 195.85it/s]
 11%|█         | 559/5021 [00:02<00:23, 191.59it/s]
  8%|▊         | 379/5021 [00:01<00:23, 196.21it/s]
 12%|█▏        | 579/5021 [00:02<00:23, 192.29it/s]
  8%|▊         | 399/5021 [00:02<00:23, 196.02it/s]
 12%|█▏        | 599/5021 [00:03<00:22, 192.78it/s]
  8%|▊         | 419/5021 [00:02<00:23, 196.29it/s]
 12%|█▏        | 619/5021 [00:03<00:22, 192.09it/s]
  9%|▊         | 439/5021 [00:02<00:23, 194.37it/s]
 13%|█▎        | 639/5021 [00:03<00:22, 193.05it/s]
  9%|▉         | 459/5021 [00:02<00:23, 194.73it/s]
 13%|█▎        | 659/5021 [00:03<00:22, 193.69it/s]
 10%|▉         | 479/5021 [00:02<00:23, 194.77it/s]
 14%|█▎        | 679/5021 [00:03<00:22, 194.25it/s]
 10%|▉         | 499/5021 [00:02<00:23, 194.79it/s]
 14%|█▍        | 699/5021 [00:03<00:22, 194.54it/s]
 10%|█         | 519/5021 [00:02<00:23, 194.69it/s]
 14%|█▍        | 719/5021 [00:03<00:22, 194.57it/s]
 11%|█         | 539/5021 [00:02<00:23, 192.65it/s]
 15%|█▍        | 739/5021 [00:03<00:22, 194.43it/s]
 11%|█         | 559/5021 [00:02<00:23, 193.13it/s]
 15%|█▌        | 759/5021 [00:03<00:21, 194.49it/s]
 12%|█▏        | 579/5021 [00:02<00:22, 193.71it/s]
 16%|█▌        | 779/5021 [00:04<00:22, 192.61it/s]
 12%|█▏        | 599/5021 [00:03<00:22, 194.06it/s]
 16%|█▌        | 799/5021 [00:04<00:21, 193.48it/s]
 12%|█▏        | 619/5021 [00:03<00:22, 194.52it/s]
 16%|█▋        | 819/5021 [00:04<00:21, 194.02it/s]
 13%|█▎        | 639/5021 [00:03<00:22, 194.66it/s]
 17%|█▋        | 839/5021 [00:04<00:21, 194.53it/s]
 13%|█▎        | 659/5021 [00:03<00:22, 193.09it/s]
 17%|█▋        | 859/5021 [00:04<00:21, 195.22it/s]
 14%|█▎        | 679/5021 [00:03<00:22, 191.96it/s]
 18%|█▊        | 879/5021 [00:04<00:21, 195.64it/s]
 14%|█▍        | 699/5021 [00:03<00:22, 192.67it/s]
 18%|█▊        | 899/5021 [00:04<00:21, 194.23it/s]
 14%|█▍        | 719/5021 [00:03<00:22, 193.38it/s]
 18%|█▊        | 919/5021 [00:04<00:21, 194.40it/s]
 15%|█▍        | 739/5021 [00:03<00:22, 194.43it/s]
 19%|█▊        | 939/5021 [00:04<00:21, 194.24it/s]
 15%|█▌        | 759/5021 [00:03<00:21, 195.13it/s]
 19%|█▉        | 959/5021 [00:04<00:20, 194.43it/s]
 16%|█▌        | 779/5021 [00:04<00:21, 194.34it/s]
 19%|█▉        | 979/5021 [00:05<00:20, 194.65it/s]
 16%|█▌        | 799/5021 [00:04<00:21, 194.33it/s]
 20%|█▉        | 999/5021 [00:05<00:20, 194.90it/s]
 16%|█▋        | 819/5021 [00:04<00:21, 194.41it/s]
 20%|██        | 1019/5021 [00:05<00:20, 193.14it/s]
 17%|█▋        | 839/5021 [00:04<00:21, 194.18it/s]
 21%|██        | 1039/5021 [00:05<00:20, 192.89it/s]
 17%|█▋        | 859/5021 [00:04<00:21, 195.01it/s]
 21%|██        | 1059/5021 [00:05<00:20, 192.64it/s]
 18%|█▊        | 879/5021 [00:04<00:21, 195.09it/s]
 21%|██▏       | 1079/5021 [00:05<00:20, 192.97it/s]
 18%|█▊        | 899/5021 [00:04<00:21, 195.62it/s]
 22%|██▏       | 1099/5021 [00:05<00:20, 192.79it/s]
 18%|█▊        | 919/5021 [00:04<00:21, 193.83it/s]
 22%|██▏       | 1119/5021 [00:05<00:20, 192.86it/s]
 19%|█▊        | 939/5021 [00:04<00:20, 194.51it/s]
 23%|██▎       | 1139/5021 [00:05<00:20, 193.28it/s]
 19%|█▉        | 959/5021 [00:04<00:20, 195.07it/s]
 23%|██▎       | 1159/5021 [00:05<00:19, 193.75it/s]
 19%|█▉        | 979/5021 [00:05<00:20, 196.07it/s]
 23%|██▎       | 1179/5021 [00:06<00:19, 194.00it/s]
 20%|█▉        | 999/5021 [00:05<00:20, 196.41it/s]
 24%|██▍       | 1199/5021 [00:06<00:19, 193.68it/s]
 20%|██        | 1019/5021 [00:05<00:20, 196.67it/s]
 24%|██▍       | 1219/5021 [00:06<00:19, 193.59it/s]
 21%|██        | 1039/5021 [00:05<00:20, 197.00it/s]
 25%|██▍       | 1239/5021 [00:06<00:19, 193.52it/s]
 21%|██        | 1059/5021 [00:05<00:20, 197.33it/s]
 25%|██▌       | 1259/5021 [00:06<00:19, 193.67it/s]
 21%|██▏       | 1079/5021 [00:05<00:20, 196.02it/s]
 25%|██▌       | 1279/5021 [00:06<00:19, 191.61it/s]
 22%|██▏       | 1099/5021 [00:05<00:19, 196.37it/s]
 26%|██▌       | 1299/5021 [00:06<00:19, 192.63it/s]
 22%|██▏       | 1119/5021 [00:05<00:19, 196.38it/s]
 26%|██▋       | 1319/5021 [00:06<00:19, 193.22it/s]
 23%|██▎       | 1139/5021 [00:05<00:19, 196.76it/s]
 27%|██▋       | 1339/5021 [00:06<00:19, 193.78it/s]
 23%|██▎       | 1159/5021 [00:05<00:19, 197.28it/s]
 27%|██▋       | 1359/5021 [00:07<00:18, 193.99it/s]
 23%|██▎       | 1179/5021 [00:06<00:19, 196.97it/s]
 27%|██▋       | 1379/5021 [00:07<00:18, 194.08it/s]
 24%|██▍       | 1199/5021 [00:06<00:19, 197.01it/s]
 28%|██▊       | 1399/5021 [00:07<00:18, 193.94it/s]
 24%|██▍       | 1219/5021 [00:06<00:19, 196.90it/s]
 28%|██▊       | 1419/5021 [00:07<00:18, 194.22it/s]
 25%|██▍       | 1239/5021 [00:06<00:19, 194.81it/s]
 29%|██▊       | 1439/5021 [00:07<00:18, 193.98it/s]
 25%|██▌       | 1259/5021 [00:06<00:19, 194.88it/s]
 29%|██▉       | 1459/5021 [00:07<00:18, 194.27it/s]
 25%|██▌       | 1279/5021 [00:06<00:19, 195.04it/s]
 29%|██▉       | 1479/5021 [00:07<00:18, 194.21it/s]
 26%|██▌       | 1299/5021 [00:06<00:19, 195.03it/s]
 30%|██▉       | 1499/5021 [00:07<00:18, 194.13it/s]
 26%|██▋       | 1319/5021 [00:06<00:18, 195.47it/s]
 30%|███       | 1519/5021 [00:07<00:18, 194.39it/s]
 27%|██▋       | 1339/5021 [00:06<00:18, 196.18it/s]
 31%|███       | 1539/5021 [00:07<00:17, 194.18it/s]
 27%|██▋       | 1359/5021 [00:06<00:18, 196.74it/s]
 31%|███       | 1559/5021 [00:08<00:17, 194.26it/s]
 27%|██▋       | 1379/5021 [00:07<00:18, 196.87it/s]
 31%|███▏      | 1579/5021 [00:08<00:17, 194.58it/s]
 28%|██▊       | 1399/5021 [00:07<00:18, 194.94it/s]
 32%|███▏      | 1599/5021 [00:08<00:17, 194.39it/s]
 28%|██▊       | 1419/5021 [00:07<00:18, 195.76it/s]
 32%|███▏      | 1619/5021 [00:08<00:17, 194.90it/s]
 29%|██▊       | 1439/5021 [00:07<00:18, 196.35it/s]
 33%|███▎      | 1639/5021 [00:08<00:17, 195.33it/s]
 29%|██▉       | 1459/5021 [00:07<00:18, 196.67it/s]
 33%|███▎      | 1659/5021 [00:08<00:17, 195.94it/s]
 29%|██▉       | 1479/5021 [00:07<00:17, 197.30it/s]
 33%|███▎      | 1679/5021 [00:08<00:17, 195.69it/s]
 30%|██▉       | 1499/5021 [00:07<00:17, 197.32it/s]
 34%|███▍      | 1699/5021 [00:08<00:16, 195.69it/s]
 30%|███       | 1519/5021 [00:07<00:17, 196.75it/s]
 34%|███▍      | 1719/5021 [00:08<00:16, 195.62it/s]
 31%|███       | 1539/5021 [00:07<00:17, 197.01it/s]
 35%|███▍      | 1739/5021 [00:08<00:16, 195.54it/s]
 31%|███       | 1559/5021 [00:07<00:17, 196.98it/s]
 35%|███▌      | 1759/5021 [00:09<00:16, 195.45it/s]
 31%|███▏      | 1579/5021 [00:08<00:17, 197.36it/s]
 35%|███▌      | 1779/5021 [00:09<00:16, 194.11it/s]
 32%|███▏      | 1599/5021 [00:08<00:17, 197.54it/s]
 36%|███▌      | 1799/5021 [00:09<00:16, 194.43it/s]
 32%|███▏      | 1619/5021 [00:08<00:17, 196.35it/s]
 36%|███▌      | 1819/5021 [00:09<00:16, 193.95it/s]
 33%|███▎      | 1639/5021 [00:08<00:17, 196.69it/s]
 37%|███▋      | 1839/5021 [00:09<00:16, 194.48it/s]
 33%|███▎      | 1659/5021 [00:08<00:17, 197.07it/s]
 37%|███▋      | 1859/5021 [00:09<00:16, 194.85it/s]
 33%|███▎      | 1679/5021 [00:08<00:16, 197.13it/s]
 37%|███▋      | 1879/5021 [00:09<00:16, 195.01it/s]
 34%|███▍      | 1699/5021 [00:08<00:16, 196.65it/s]
 38%|███▊      | 1899/5021 [00:09<00:16, 194.58it/s]
 34%|███▍      | 1719/5021 [00:08<00:16, 196.91it/s]
 38%|███▊      | 1919/5021 [00:09<00:15, 194.45it/s]
 35%|███▍      | 1739/5021 [00:08<00:16, 195.79it/s]
 39%|███▊      | 1939/5021 [00:09<00:15, 194.16it/s]
 35%|███▌      | 1759/5021 [00:08<00:16, 196.16it/s]
 39%|███▉      | 1959/5021 [00:10<00:15, 194.50it/s]
 35%|███▌      | 1779/5021 [00:09<00:16, 196.62it/s]
 39%|███▉      | 1979/5021 [00:10<00:15, 195.37it/s]
 36%|███▌      | 1799/5021 [00:09<00:16, 197.12it/s]
 40%|███▉      | 1999/5021 [00:10<00:15, 196.15it/s]
 36%|███▌      | 1819/5021 [00:09<00:16, 195.61it/s]
 40%|████      | 2019/5021 [00:10<00:15, 196.62it/s]
 37%|███▋      | 1839/5021 [00:09<00:16, 196.03it/s]
 41%|████      | 2039/5021 [00:10<00:15, 196.85it/s]
 37%|███▋      | 1859/5021 [00:09<00:16, 196.46it/s]
 41%|████      | 2059/5021 [00:10<00:15, 197.28it/s]
 37%|███▋      | 1879/5021 [00:09<00:15, 196.91it/s]
 41%|████▏     | 2079/5021 [00:10<00:14, 197.22it/s]
 38%|███▊      | 1899/5021 [00:09<00:15, 196.30it/s]
 42%|████▏     | 2099/5021 [00:10<00:14, 197.09it/s]
 38%|███▊      | 1919/5021 [00:09<00:15, 196.42it/s]
 42%|████▏     | 2119/5021 [00:10<00:14, 196.77it/s]
 39%|███▊      | 1939/5021 [00:09<00:15, 196.46it/s]
 43%|████▎     | 2139/5021 [00:10<00:14, 196.24it/s]
 39%|███▉      | 1959/5021 [00:10<00:15, 196.49it/s]
 43%|████▎     | 2159/5021 [00:11<00:14, 196.42it/s]
 39%|███▉      | 1979/5021 [00:10<00:15, 196.50it/s]
 43%|████▎     | 2179/5021 [00:11<00:14, 196.26it/s]
 40%|███▉      | 1999/5021 [00:10<00:15, 196.49it/s]
 44%|████▍     | 2199/5021 [00:11<00:14, 196.39it/s]
 40%|████      | 2019/5021 [00:10<00:15, 196.73it/s]
 44%|████▍     | 2219/5021 [00:11<00:14, 196.45it/s]
 41%|████      | 2039/5021 [00:10<00:15, 194.53it/s]
 45%|████▍     | 2239/5021 [00:11<00:14, 196.34it/s]
 41%|████      | 2059/5021 [00:10<00:15, 194.70it/s]
 45%|████▍     | 2259/5021 [00:11<00:14, 196.70it/s]
 41%|████▏     | 2079/5021 [00:10<00:15, 195.58it/s]
 45%|████▌     | 2279/5021 [00:11<00:14, 194.75it/s]
 42%|████▏     | 2099/5021 [00:10<00:14, 195.93it/s]
 46%|████▌     | 2299/5021 [00:11<00:13, 194.57it/s]
 42%|████▏     | 2119/5021 [00:10<00:14, 196.07it/s]
 46%|████▌     | 2319/5021 [00:11<00:13, 194.39it/s]
 43%|████▎     | 2139/5021 [00:10<00:14, 196.81it/s]
 47%|████▋     | 2339/5021 [00:12<00:13, 194.53it/s]
 43%|████▎     | 2159/5021 [00:11<00:14, 196.81it/s]
 47%|████▋     | 2359/5021 [00:12<00:13, 195.02it/s]
 43%|████▎     | 2179/5021 [00:11<00:14, 196.86it/s]
 47%|████▋     | 2379/5021 [00:12<00:13, 195.56it/s]
 44%|████▍     | 2199/5021 [00:11<00:14, 197.32it/s]
 48%|████▊     | 2399/5021 [00:12<00:13, 195.99it/s]
 44%|████▍     | 2219/5021 [00:11<00:14, 197.36it/s]
 48%|████▊     | 2419/5021 [00:12<00:13, 196.10it/s]
 45%|████▍     | 2239/5021 [00:11<00:14, 197.73it/s]
 49%|████▊     | 2439/5021 [00:12<00:13, 196.34it/s]
 45%|████▍     | 2259/5021 [00:11<00:13, 197.82it/s]
 49%|████▉     | 2459/5021 [00:12<00:13, 196.64it/s]
 45%|████▌     | 2279/5021 [00:11<00:13, 196.78it/s]
 49%|████▉     | 2479/5021 [00:12<00:12, 196.45it/s]
 46%|████▌     | 2299/5021 [00:11<00:13, 197.16it/s]
 50%|████▉     | 2499/5021 [00:12<00:12, 196.15it/s]
 46%|████▌     | 2319/5021 [00:11<00:13, 197.09it/s]
 50%|█████     | 2519/5021 [00:12<00:12, 196.20it/s]
 47%|████▋     | 2339/5021 [00:11<00:13, 196.85it/s]
 51%|█████     | 2539/5021 [00:13<00:12, 195.96it/s]
 47%|████▋     | 2359/5021 [00:12<00:13, 197.15it/s]
 51%|█████     | 2559/5021 [00:13<00:12, 195.62it/s]
 47%|████▋     | 2379/5021 [00:12<00:13, 197.31it/s]
 51%|█████▏    | 2579/5021 [00:13<00:12, 194.19it/s]
 48%|████▊     | 2399/5021 [00:12<00:13, 197.26it/s]
 52%|█████▏    | 2599/5021 [00:13<00:12, 194.32it/s]
 48%|████▊     | 2419/5021 [00:12<00:13, 197.60it/s]
 52%|█████▏    | 2619/5021 [00:13<00:12, 194.66it/s]
 49%|████▊     | 2439/5021 [00:12<00:13, 197.73it/s]
 53%|█████▎    | 2639/5021 [00:13<00:12, 194.27it/s]
 49%|████▉     | 2459/5021 [00:12<00:12, 198.07it/s]
 53%|█████▎    | 2659/5021 [00:13<00:12, 194.63it/s]
 49%|████▉     | 2479/5021 [00:12<00:12, 198.05it/s]
 53%|█████▎    | 2679/5021 [00:13<00:12, 194.79it/s]
 50%|████▉     | 2499/5021 [00:12<00:12, 197.33it/s]
 54%|█████▍    | 2699/5021 [00:13<00:11, 195.13it/s]
 50%|█████     | 2519/5021 [00:12<00:12, 197.25it/s]
 54%|█████▍    | 2719/5021 [00:13<00:11, 195.44it/s]
 51%|█████     | 2539/5021 [00:12<00:12, 195.08it/s]
 55%|█████▍    | 2739/5021 [00:14<00:11, 195.22it/s]
 51%|█████     | 2559/5021 [00:13<00:12, 193.93it/s]
 55%|█████▍    | 2759/5021 [00:14<00:11, 195.56it/s]
 51%|█████▏    | 2579/5021 [00:13<00:12, 194.77it/s]
 55%|█████▌    | 2779/5021 [00:14<00:11, 193.90it/s]
 52%|█████▏    | 2599/5021 [00:13<00:12, 195.11it/s]
 56%|█████▌    | 2799/5021 [00:14<00:11, 194.60it/s]
 52%|█████▏    | 2619/5021 [00:13<00:12, 195.26it/s]
 56%|█████▌    | 2819/5021 [00:14<00:11, 195.15it/s]
 53%|█████▎    | 2639/5021 [00:13<00:12, 195.28it/s]
 57%|█████▋    | 2839/5021 [00:14<00:11, 194.67it/s]
 53%|█████▎    | 2659/5021 [00:13<00:12, 195.05it/s]
 57%|█████▋    | 2859/5021 [00:14<00:11, 193.46it/s]
 53%|█████▎    | 2679/5021 [00:13<00:12, 194.10it/s]
 57%|█████▋    | 2879/5021 [00:14<00:11, 193.12it/s]
 54%|█████▍    | 2699/5021 [00:13<00:11, 194.39it/s]
 58%|█████▊    | 2899/5021 [00:14<00:10, 193.19it/s]
 54%|█████▍    | 2719/5021 [00:13<00:11, 193.42it/s]
 58%|█████▊    | 2919/5021 [00:15<00:10, 191.90it/s]
 55%|█████▍    | 2739/5021 [00:13<00:11, 193.39it/s]
 59%|█████▊    | 2939/5021 [00:15<00:10, 193.31it/s]
 55%|█████▍    | 2759/5021 [00:14<00:11, 194.36it/s]
 59%|█████▉    | 2959/5021 [00:15<00:10, 193.90it/s]
 55%|█████▌    | 2779/5021 [00:14<00:11, 193.04it/s]
 59%|█████▉    | 2979/5021 [00:15<00:10, 194.51it/s]
 56%|█████▌    | 2799/5021 [00:14<00:11, 194.04it/s]
 60%|█████▉    | 2999/5021 [00:15<00:10, 191.13it/s]
 56%|█████▌    | 2819/5021 [00:14<00:11, 188.44it/s]
 60%|██████    | 3019/5021 [00:15<00:10, 189.16it/s]
 57%|█████▋    | 2839/5021 [00:14<00:11, 190.23it/s]
 61%|██████    | 3039/5021 [00:15<00:10, 191.57it/s]
 57%|█████▋    | 2859/5021 [00:14<00:11, 192.38it/s]
 61%|██████    | 3059/5021 [00:15<00:10, 192.52it/s]
 57%|█████▋    | 2879/5021 [00:14<00:11, 193.81it/s]
 61%|██████▏   | 3079/5021 [00:15<00:10, 193.67it/s]
 58%|█████▊    | 2899/5021 [00:14<00:10, 194.93it/s]
 62%|██████▏   | 3099/5021 [00:15<00:09, 194.29it/s]
 58%|█████▊    | 2919/5021 [00:14<00:10, 195.73it/s]
 62%|██████▏   | 3119/5021 [00:16<00:09, 194.95it/s]
 59%|█████▊    | 2939/5021 [00:15<00:10, 196.15it/s]
 63%|██████▎   | 3139/5021 [00:16<00:09, 191.49it/s]
 59%|█████▉    | 2959/5021 [00:15<00:10, 189.18it/s]
 63%|██████▎   | 3159/5021 [00:16<00:09, 192.84it/s]
 59%|█████▉    | 2979/5021 [00:15<00:10, 191.84it/s]
 63%|██████▎   | 3179/5021 [00:16<00:09, 193.95it/s]
 60%|█████▉    | 2999/5021 [00:15<00:10, 194.14it/s]
 64%|██████▎   | 3199/5021 [00:16<00:09, 194.61it/s]
 60%|██████    | 3019/5021 [00:15<00:10, 195.71it/s]
 64%|██████▍   | 3219/5021 [00:16<00:09, 195.32it/s]
 61%|██████    | 3039/5021 [00:15<00:10, 196.64it/s]
 65%|██████▍   | 3239/5021 [00:16<00:09, 196.02it/s]
 61%|██████    | 3059/5021 [00:15<00:09, 197.28it/s]
 65%|██████▍   | 3259/5021 [00:16<00:08, 196.88it/s]
 61%|██████▏   | 3079/5021 [00:15<00:09, 196.58it/s]
 65%|██████▌   | 3279/5021 [00:16<00:08, 197.60it/s]
 62%|██████▏   | 3099/5021 [00:15<00:09, 197.08it/s]
 66%|██████▌   | 3299/5021 [00:16<00:08, 197.24it/s]
 62%|██████▏   | 3119/5021 [00:15<00:09, 197.62it/s]
 66%|██████▌   | 3319/5021 [00:17<00:08, 197.06it/s]
 63%|██████▎   | 3139/5021 [00:16<00:09, 197.86it/s]
 67%|██████▋   | 3339/5021 [00:17<00:08, 196.39it/s]
 63%|██████▎   | 3159/5021 [00:16<00:09, 197.56it/s]
 67%|██████▋   | 3359/5021 [00:17<00:08, 195.93it/s]
 63%|██████▎   | 3179/5021 [00:16<00:09, 197.48it/s]
 67%|██████▋   | 3379/5021 [00:17<00:08, 195.79it/s]
 64%|██████▎   | 3199/5021 [00:16<00:09, 197.59it/s]
 68%|██████▊   | 3399/5021 [00:17<00:08, 195.78it/s]
 64%|██████▍   | 3219/5021 [00:16<00:09, 197.51it/s]
 68%|██████▊   | 3419/5021 [00:17<00:08, 196.44it/s]
 65%|██████▍   | 3239/5021 [00:16<00:09, 196.79it/s]
 68%|██████▊   | 3439/5021 [00:17<00:08, 196.39it/s]
 65%|██████▍   | 3259/5021 [00:16<00:08, 197.08it/s]
 69%|██████▉   | 3459/5021 [00:17<00:07, 196.42it/s]
 65%|██████▌   | 3279/5021 [00:16<00:08, 197.42it/s]
 69%|██████▉   | 3479/5021 [00:17<00:07, 196.28it/s]
 66%|██████▌   | 3299/5021 [00:16<00:08, 197.65it/s]
 70%|██████▉   | 3499/5021 [00:17<00:07, 196.29it/s]
 66%|██████▌   | 3319/5021 [00:16<00:08, 197.77it/s]
 70%|███████   | 3519/5021 [00:18<00:07, 196.38it/s]
 67%|██████▋   | 3339/5021 [00:17<00:08, 198.19it/s]
 70%|███████   | 3539/5021 [00:18<00:07, 196.56it/s]
 67%|██████▋   | 3359/5021 [00:17<00:08, 198.08it/s]
 71%|███████   | 3559/5021 [00:18<00:07, 196.85it/s]
 67%|██████▋   | 3379/5021 [00:17<00:08, 196.00it/s]
 71%|███████▏  | 3579/5021 [00:18<00:07, 196.92it/s]
 68%|██████▊   | 3399/5021 [00:17<00:08, 196.02it/s]
 72%|███████▏  | 3599/5021 [00:18<00:07, 197.10it/s]
 68%|██████▊   | 3419/5021 [00:17<00:08, 196.64it/s]
 72%|███████▏  | 3619/5021 [00:18<00:07, 197.54it/s]
 68%|██████▊   | 3439/5021 [00:17<00:08, 196.80it/s]
 72%|███████▏  | 3639/5021 [00:18<00:06, 197.71it/s]
 69%|██████▉   | 3459/5021 [00:17<00:07, 196.85it/s]
 73%|███████▎  | 3659/5021 [00:18<00:06, 197.82it/s]
 69%|██████▉   | 3479/5021 [00:17<00:07, 197.36it/s]
 73%|███████▎  | 3679/5021 [00:18<00:06, 197.04it/s]
 70%|██████▉   | 3499/5021 [00:17<00:07, 197.34it/s]
 74%|███████▎  | 3699/5021 [00:18<00:06, 196.63it/s]
 70%|███████   | 3519/5021 [00:17<00:07, 197.23it/s]
 74%|███████▍  | 3719/5021 [00:19<00:06, 196.43it/s]
 70%|███████   | 3539/5021 [00:18<00:07, 196.96it/s]
 74%|███████▍  | 3739/5021 [00:19<00:06, 196.46it/s]
 71%|███████   | 3559/5021 [00:18<00:07, 196.85it/s]
 75%|███████▍  | 3759/5021 [00:19<00:06, 196.57it/s]
 71%|███████▏  | 3579/5021 [00:18<00:07, 196.50it/s]
 75%|███████▌  | 3779/5021 [00:19<00:06, 195.27it/s]
 72%|███████▏  | 3599/5021 [00:18<00:07, 196.75it/s]
 76%|███████▌  | 3799/5021 [00:19<00:06, 195.94it/s]
 72%|███████▏  | 3619/5021 [00:18<00:07, 196.97it/s]
 76%|███████▌  | 3819/5021 [00:19<00:06, 196.20it/s]
 72%|███████▏  | 3639/5021 [00:18<00:07, 197.38it/s]
 76%|███████▋  | 3839/5021 [00:19<00:06, 196.11it/s]
 73%|███████▎  | 3659/5021 [00:18<00:06, 197.60it/s]
 77%|███████▋  | 3859/5021 [00:19<00:05, 196.25it/s]
 73%|███████▎  | 3679/5021 [00:18<00:06, 195.49it/s]
 77%|███████▋  | 3879/5021 [00:19<00:05, 196.23it/s]
 74%|███████▎  | 3699/5021 [00:18<00:06, 196.01it/s]
 78%|███████▊  | 3899/5021 [00:20<00:05, 196.21it/s]
 74%|███████▍  | 3719/5021 [00:18<00:06, 196.41it/s]
 78%|███████▊  | 3919/5021 [00:20<00:05, 196.34it/s]
 74%|███████▍  | 3739/5021 [00:19<00:06, 196.52it/s]
 78%|███████▊  | 3939/5021 [00:20<00:05, 196.46it/s]
 75%|███████▍  | 3759/5021 [00:19<00:06, 189.93it/s]
 79%|███████▉  | 3959/5021 [00:20<00:05, 196.33it/s]
 75%|███████▌  | 3779/5021 [00:19<00:06, 192.22it/s]
 79%|███████▉  | 3979/5021 [00:20<00:05, 196.74it/s]
 76%|███████▌  | 3799/5021 [00:19<00:06, 193.95it/s]
 80%|███████▉  | 3999/5021 [00:20<00:05, 197.23it/s]
 76%|███████▌  | 3819/5021 [00:19<00:06, 194.92it/s]
 80%|████████  | 4019/5021 [00:20<00:05, 197.67it/s]
 76%|███████▋  | 3839/5021 [00:19<00:06, 195.68it/s]
 80%|████████  | 4039/5021 [00:20<00:04, 198.04it/s]
 77%|███████▋  | 3859/5021 [00:19<00:05, 196.26it/s]
 81%|████████  | 4059/5021 [00:20<00:04, 198.36it/s]
 77%|███████▋  | 3879/5021 [00:19<00:05, 196.77it/s]
 81%|████████  | 4079/5021 [00:20<00:04, 198.53it/s]
 78%|███████▊  | 3899/5021 [00:19<00:05, 197.17it/s]
 82%|████████▏ | 4099/5021 [00:21<00:04, 198.40it/s]
 78%|███████▊  | 3919/5021 [00:20<00:05, 197.06it/s]
 82%|████████▏ | 4119/5021 [00:21<00:04, 197.79it/s]
 78%|███████▊  | 3939/5021 [00:20<00:05, 196.60it/s]
 82%|████████▏ | 4139/5021 [00:21<00:04, 197.52it/s]
 79%|███████▉  | 3959/5021 [00:20<00:05, 196.63it/s]
 83%|████████▎ | 4159/5021 [00:21<00:04, 197.39it/s]
 79%|███████▉  | 3979/5021 [00:20<00:05, 196.78it/s]
 83%|████████▎ | 4179/5021 [00:21<00:04, 196.96it/s]
 80%|███████▉  | 3999/5021 [00:20<00:05, 196.72it/s]
 84%|████████▎ | 4199/5021 [00:21<00:04, 196.41it/s]
 80%|████████  | 4019/5021 [00:20<00:05, 196.94it/s]
 84%|████████▍ | 4219/5021 [00:21<00:04, 196.53it/s]
 80%|████████  | 4039/5021 [00:20<00:04, 197.26it/s]
 84%|████████▍ | 4239/5021 [00:21<00:03, 196.65it/s]
 81%|████████  | 4059/5021 [00:20<00:04, 196.28it/s]
 85%|████████▍ | 4259/5021 [00:21<00:03, 196.76it/s]
 81%|████████  | 4079/5021 [00:20<00:04, 196.62it/s]
 85%|████████▌ | 4279/5021 [00:21<00:03, 196.61it/s]
 82%|████████▏ | 4099/5021 [00:20<00:04, 196.75it/s]
 86%|████████▌ | 4299/5021 [00:22<00:03, 197.01it/s]
 82%|████████▏ | 4119/5021 [00:21<00:04, 196.81it/s]
 86%|████████▌ | 4319/5021 [00:22<00:03, 197.24it/s]
 82%|████████▏ | 4139/5021 [00:21<00:04, 197.50it/s]
 86%|████████▋ | 4339/5021 [00:22<00:03, 197.31it/s]
 83%|████████▎ | 4159/5021 [00:21<00:04, 197.75it/s]
 87%|████████▋ | 4359/5021 [00:22<00:03, 197.62it/s]
 83%|████████▎ | 4179/5021 [00:21<00:04, 197.09it/s]
 87%|████████▋ | 4379/5021 [00:22<00:03, 197.59it/s]
 84%|████████▎ | 4199/5021 [00:21<00:04, 196.85it/s]
 88%|████████▊ | 4399/5021 [00:22<00:03, 197.42it/s]
 84%|████████▍ | 4219/5021 [00:21<00:04, 196.88it/s]
 88%|████████▊ | 4419/5021 [00:22<00:03, 197.57it/s]
 84%|████████▍ | 4239/5021 [00:21<00:03, 196.86it/s]
 88%|████████▊ | 4439/5021 [00:22<00:02, 197.54it/s]
 85%|████████▍ | 4259/5021 [00:21<00:03, 197.42it/s]
 89%|████████▉ | 4459/5021 [00:22<00:02, 198.01it/s]
 85%|████████▌ | 4279/5021 [00:21<00:03, 197.59it/s]
 89%|████████▉ | 4479/5021 [00:22<00:02, 197.74it/s]
 86%|████████▌ | 4299/5021 [00:21<00:03, 197.84it/s]
 90%|████████▉ | 4499/5021 [00:23<00:02, 197.74it/s]
 86%|████████▌ | 4319/5021 [00:22<00:03, 197.37it/s]
 90%|█████████ | 4519/5021 [00:23<00:02, 197.61it/s]
 86%|████████▋ | 4339/5021 [00:22<00:03, 195.65it/s]
 90%|█████████ | 4539/5021 [00:23<00:02, 196.76it/s]
 87%|████████▋ | 4359/5021 [00:22<00:03, 196.52it/s]
 91%|█████████ | 4559/5021 [00:23<00:02, 196.43it/s]
 87%|████████▋ | 4379/5021 [00:22<00:03, 196.89it/s]
 91%|█████████ | 4579/5021 [00:23<00:02, 196.39it/s]
 88%|████████▊ | 4399/5021 [00:22<00:03, 197.17it/s]
 92%|█████████▏| 4599/5021 [00:23<00:02, 196.48it/s]
 88%|████████▊ | 4419/5021 [00:22<00:03, 197.79it/s]
 92%|█████████▏| 4619/5021 [00:23<00:02, 196.50it/s]
 88%|████████▊ | 4439/5021 [00:22<00:02, 195.72it/s]
 92%|█████████▏| 4639/5021 [00:23<00:01, 196.25it/s]
 89%|████████▉ | 4459/5021 [00:22<00:02, 195.31it/s]
 93%|█████████▎| 4659/5021 [00:23<00:01, 196.22it/s]
 89%|████████▉ | 4479/5021 [00:22<00:02, 195.23it/s]
 93%|█████████▎| 4679/5021 [00:23<00:01, 196.50it/s]
 90%|████████▉ | 4499/5021 [00:22<00:02, 195.21it/s]
 94%|█████████▎| 4699/5021 [00:24<00:01, 196.15it/s]
 90%|█████████ | 4519/5021 [00:23<00:02, 195.14it/s]
 94%|█████████▍| 4719/5021 [00:24<00:01, 196.50it/s]
 90%|█████████ | 4539/5021 [00:23<00:02, 195.40it/s]
 94%|█████████▍| 4739/5021 [00:24<00:01, 196.56it/s]
 91%|█████████ | 4559/5021 [00:23<00:02, 195.60it/s]
 95%|█████████▍| 4759/5021 [00:24<00:01, 194.86it/s]
 91%|█████████ | 4579/5021 [00:23<00:02, 195.46it/s]
 95%|█████████▌| 4779/5021 [00:24<00:01, 194.56it/s]
 92%|█████████▏| 4599/5021 [00:23<00:02, 195.09it/s]
 96%|█████████▌| 4799/5021 [00:24<00:01, 193.59it/s]
 92%|█████████▏| 4619/5021 [00:23<00:02, 195.63it/s]
 96%|█████████▌| 4819/5021 [00:24<00:01, 192.09it/s]
 92%|█████████▏| 4639/5021 [00:23<00:01, 196.13it/s]
 96%|█████████▋| 4839/5021 [00:24<00:00, 191.30it/s]
 93%|█████████▎| 4659/5021 [00:23<00:01, 195.41it/s]
 97%|█████████▋| 4859/5021 [00:24<00:00, 191.36it/s]
 93%|█████████▎| 4679/5021 [00:23<00:01, 194.89it/s]
 97%|█████████▋| 4879/5021 [00:25<00:00, 191.32it/s]
 94%|█████████▎| 4699/5021 [00:23<00:01, 194.53it/s]
 98%|█████████▊| 4899/5021 [00:25<00:00, 191.15it/s]
 94%|█████████▍| 4719/5021 [00:24<00:01, 194.24it/s]
 98%|█████████▊| 4919/5021 [00:25<00:00, 191.26it/s]
 94%|█████████▍| 4739/5021 [00:24<00:01, 193.70it/s]
 98%|█████████▊| 4939/5021 [00:25<00:00, 190.93it/s]
 95%|█████████▍| 4759/5021 [00:24<00:01, 193.44it/s]
 99%|█████████▉| 4959/5021 [00:25<00:00, 191.14it/s]
 95%|█████████▌| 4779/5021 [00:24<00:01, 193.21it/s]
 99%|█████████▉| 4979/5021 [00:25<00:00, 190.93it/s]
 96%|█████████▌| 4799/5021 [00:24<00:01, 192.80it/s]
100%|█████████▉| 4999/5021 [00:25<00:00, 190.76it/s]
 96%|█████████▌| 4819/5021 [00:24<00:01, 190.32it/s]
100%|█████████▉| 5019/5021 [00:25<00:00, 190.85it/s]
100%|██████████| 5021/5021 [00:25<00:00, 195.01it/s]

 96%|█████████▋| 4839/5021 [00:24<00:00, 191.03it/s]
 97%|█████████▋| 4859/5021 [00:24<00:00, 192.46it/s]
 97%|█████████▋| 4879/5021 [00:24<00:00, 194.04it/s]
 98%|█████████▊| 4899/5021 [00:25<00:00, 195.22it/s]
 98%|█████████▊| 4919/5021 [00:25<00:00, 195.91it/s]
 98%|█████████▊| 4939/5021 [00:25<00:00, 196.51it/s]
 99%|█████████▉| 4959/5021 [00:25<00:00, 196.86it/s]
 99%|█████████▉| 4979/5021 [00:25<00:00, 197.56it/s]
100%|█████████▉| 4999/5021 [00:25<00:00, 198.11it/s]
100%|█████████▉| 5019/5021 [00:25<00:00, 198.20it/s]
100%|██████████| 5021/5021 [00:25<00:00, 195.83it/s]
n136-128-154:2218825:2218825 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2218825:2218825 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2218825:2218825 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:2218825:2218825 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2218825:2218825 [0] NCCL INFO Comm config Blocking set to 1
n136-128-154:2218826:2218826 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:2218826:2218826 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2218826:2218826 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2218826:2218826 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2218826:2218826 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:2218825:2219031 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2218825:2219031 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2218826:2219032 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2218826:2219032 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2218825:2219031 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2218825:2219031 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2218825:2219031 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2218826:2219032 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2218826:2219032 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2218826:2219032 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2218825:2219031 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2218825:2219031 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2218825:2219031 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:2218825:2219031 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2218825:2219031 [0] NCCL INFO Using network IB
n136-128-154:2218826:2219032 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2218826:2219032 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2218826:2219032 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:2218826:2219032 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2218826:2219032 [1] NCCL INFO Using network IB
n136-128-154:2218825:2219031 [0] NCCL INFO ncclCommInitRankConfig comm 0xe3cc210 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0xf5eaff4420616091 - Init START
n136-128-154:2218826:2219032 [1] NCCL INFO ncclCommInitRankConfig comm 0xe7126f0 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0xf5eaff4420616091 - Init START
n136-128-154:2218825:2219031 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2218826:2219032 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2218826:2219032 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2218826:2219032 [1] NCCL INFO Retrieving state for IB
n136-128-154:2218826:2219032 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:2218826:2219032 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2218826:2219032 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2218826:2219032 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2218826:2219032 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2218825:2219031 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2218825:2219031 [0] NCCL INFO Retrieving state for IB
n136-128-154:2218825:2219031 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:2218825:2219031 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2218825:2219031 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2218825:2219031 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2218825:2219031 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2218825:2219031 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2218825:2219031 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2218825:2219031 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2218825:2219031 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2218825:2219031 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2218825:2219031 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2218825:2219031 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2218825:2219031 [0] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2218825:2219031 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2218825:2219031 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2218825:2219031 [0] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2218825:2219031 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2218825:2219031 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2218825:2219031 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2218825:2219031 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2218825:2219031 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2218825:2219031 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2218825:2219031 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2218825:2219031 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2218825:2219031 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2218825:2219031 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2218825:2219031 [0] NCCL INFO ==========================================
n136-128-154:2218825:2219031 [0] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2218825:2219031 [0] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2218825:2219031 [0] NCCL INFO Setting affinity for GPU 6 to 33-62,97-126
n136-128-154:2218826:2219032 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2218826:2219032 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2218825:2219031 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2218825:2219031 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218825:2219031 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218825:2219031 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218825:2219031 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218825:2219031 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218825:2219031 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218825:2219031 [0] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218825:2219031 [0] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218825:2219031 [0] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218825:2219031 [0] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218825:2219031 [0] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218825:2219031 [0] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218825:2219031 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2218825:2219031 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218825:2219031 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218825:2219031 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218825:2219031 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218825:2219031 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218825:2219031 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218825:2219031 [0] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2218825:2219031 [0] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2218825:2219031 [0] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2218825:2219031 [0] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2218825:2219031 [0] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2218825:2219031 [0] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2218826:2219032 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2218826:2219032 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2218826:2219032 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2218826:2219032 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2218826:2219032 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2218826:2219032 [1] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2218826:2219032 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2218826:2219032 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2218826:2219032 [1] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2218826:2219032 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2218826:2219032 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2218826:2219032 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2218826:2219032 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2218826:2219032 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2218826:2219032 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2218826:2219032 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2218826:2219032 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2218826:2219032 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2218826:2219032 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2218826:2219032 [1] NCCL INFO ==========================================
n136-128-154:2218826:2219032 [1] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2218826:2219032 [1] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2218826:2219032 [1] NCCL INFO Setting affinity for GPU 7 to 33-62,97-126
n136-128-154:2218826:2219032 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2218826:2219032 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218826:2219032 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218826:2219032 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218826:2219032 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218826:2219032 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218826:2219032 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218826:2219032 [1] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218826:2219032 [1] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218826:2219032 [1] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218826:2219032 [1] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218826:2219032 [1] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218826:2219032 [1] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218826:2219032 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2218826:2219032 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218826:2219032 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218826:2219032 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218826:2219032 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218826:2219032 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218826:2219032 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2218826:2219032 [1] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2218826:2219032 [1] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2218826:2219032 [1] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2218826:2219032 [1] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2218826:2219032 [1] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2218826:2219032 [1] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2218825:2219031 [0] NCCL INFO comm 0xe3cc210 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:2218826:2219032 [1] NCCL INFO comm 0xe7126f0 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:2218825:2219031 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:2218826:2219032 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:2218825:2219031 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:2218826:2219032 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:2218825:2219031 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:2218826:2219032 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:2218825:2219031 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:2218826:2219032 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:2218825:2219031 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:2218826:2219032 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:2218825:2219031 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:2218826:2219032 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:2218825:2219031 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:2218826:2219032 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:2218825:2219031 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:2218826:2219032 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:2218825:2219031 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:2218826:2219032 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:2218825:2219031 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:2218826:2219032 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:2218825:2219031 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:2218826:2219032 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:2218825:2219031 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:2218826:2219032 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:2218825:2219031 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:2218826:2219032 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:2218825:2219031 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:2218826:2219032 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:2218825:2219031 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:2218826:2219032 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:2218825:2219031 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:2218826:2219032 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:2218825:2219031 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:2218826:2219032 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:2218825:2219031 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:2218826:2219032 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:2218825:2219031 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:2218826:2219032 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:2218825:2219031 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:2218826:2219032 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:2218825:2219031 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:2218826:2219032 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:2218825:2219031 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:2218826:2219032 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:2218825:2219031 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:2218826:2219032 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:2218825:2219031 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:2218826:2219032 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:2218825:2219031 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:2218825:2219031 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:2218826:2219032 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:2218825:2219031 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:2218826:2219032 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:2218825:2219031 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:2218826:2219032 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:2218825:2219031 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:2218826:2219032 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:2218825:2219031 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:2218826:2219032 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:2218825:2219031 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:2218826:2219032 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:2218825:2219031 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:2218826:2219032 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:2218825:2219031 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:2218826:2219032 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:2218825:2219031 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:2218826:2219032 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:2218825:2219031 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:2218826:2219032 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:2218825:2219031 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:2218826:2219032 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:2218825:2219031 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:2218826:2219032 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:2218825:2219031 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:2218826:2219032 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:2218825:2219031 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:2218826:2219032 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:2218825:2219031 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:2218826:2219032 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:2218825:2219031 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:2218826:2219032 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:2218825:2219031 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:2218826:2219032 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:2218825:2219031 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:2218826:2219032 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:2218825:2219031 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:2218826:2219032 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:2218825:2219031 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:2218826:2219032 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:2218825:2219031 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:2218826:2219032 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:2218825:2219031 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:2218826:2219032 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:2218825:2219031 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:2218826:2219032 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:2218825:2219031 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:2218826:2219032 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:2218825:2219031 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:2218826:2219032 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:2218825:2219031 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:2218825:2219031 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:2218826:2219032 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2218825:2219031 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:2218825:2219031 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:2218825:2219031 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:2218825:2219031 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:2218825:2219031 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:2218825:2219031 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:2218825:2219031 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:2218825:2219031 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:2218825:2219031 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:2218825:2219031 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:2218825:2219031 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:2218825:2219031 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:2218825:2219031 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:2218825:2219031 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:2218825:2219031 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:2218825:2219031 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:2218825:2219031 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:2218825:2219031 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:2218825:2219031 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:2218825:2219031 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:2218825:2219031 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:2218825:2219031 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2218826:2219032 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2218826:2219044 [1] NCCL INFO [Proxy Service] Device 1 CPU core 60
n136-128-154:2218826:2219045 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 62
n136-128-154:2218825:2219031 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2218825:2219031 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:2218825:2219046 [0] NCCL INFO [Proxy Service] Device 0 CPU core 39
n136-128-154:2218825:2219047 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 42
n136-128-154:2218826:2219032 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2218826:2219032 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2218825:2219031 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2218825:2219031 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2218825:2219031 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:2218826:2219032 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2218826:2219032 [1] NCCL INFO ncclCommInitRankConfig comm 0xe7126f0 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0xf5eaff4420616091 - Init COMPLETE
n136-128-154:2218826:2219032 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.71 (kernels 0.13, alloc 0.38, bootstrap 0.00, allgathers 0.00, topo 0.05, graphs 0.00, connections 0.02, rest 0.13)
n136-128-154:2218825:2219031 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2218825:2219031 [0] NCCL INFO ncclCommInitRankConfig comm 0xe3cc210 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0xf5eaff4420616091 - Init COMPLETE
n136-128-154:2218825:2219031 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.71 (kernels 0.11, alloc 0.40, bootstrap 0.00, allgathers 0.00, topo 0.05, graphs 0.00, connections 0.01, rest 0.13)
n136-128-154:2218826:2219048 [1] NCCL INFO Channel 00/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218826:2219048 [1] NCCL INFO Channel 01/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218826:2219048 [1] NCCL INFO Channel 02/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218826:2219048 [1] NCCL INFO Channel 03/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218825:2219049 [0] NCCL INFO Channel 00/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218826:2219048 [1] NCCL INFO Channel 04/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218825:2219049 [0] NCCL INFO Channel 01/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218826:2219048 [1] NCCL INFO Channel 05/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218825:2219049 [0] NCCL INFO Channel 02/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218826:2219048 [1] NCCL INFO Channel 06/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218825:2219049 [0] NCCL INFO Channel 03/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218826:2219048 [1] NCCL INFO Channel 07/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218825:2219049 [0] NCCL INFO Channel 04/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218826:2219048 [1] NCCL INFO Channel 08/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218825:2219049 [0] NCCL INFO Channel 05/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218826:2219048 [1] NCCL INFO Channel 09/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218825:2219049 [0] NCCL INFO Channel 06/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218826:2219048 [1] NCCL INFO Channel 10/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218825:2219049 [0] NCCL INFO Channel 07/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218826:2219048 [1] NCCL INFO Channel 11/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218825:2219049 [0] NCCL INFO Channel 08/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218826:2219048 [1] NCCL INFO Channel 12/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218825:2219049 [0] NCCL INFO Channel 09/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218826:2219048 [1] NCCL INFO Channel 13/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218825:2219049 [0] NCCL INFO Channel 10/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218826:2219048 [1] NCCL INFO Channel 14/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218825:2219049 [0] NCCL INFO Channel 11/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218826:2219048 [1] NCCL INFO Channel 15/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218825:2219049 [0] NCCL INFO Channel 12/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218826:2219048 [1] NCCL INFO Channel 16/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218825:2219049 [0] NCCL INFO Channel 13/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218826:2219048 [1] NCCL INFO Channel 17/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218825:2219049 [0] NCCL INFO Channel 14/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218826:2219048 [1] NCCL INFO Channel 18/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218825:2219049 [0] NCCL INFO Channel 15/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218826:2219048 [1] NCCL INFO Channel 19/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218825:2219049 [0] NCCL INFO Channel 16/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218826:2219048 [1] NCCL INFO Channel 20/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218825:2219049 [0] NCCL INFO Channel 17/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218826:2219048 [1] NCCL INFO Channel 21/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218825:2219049 [0] NCCL INFO Channel 18/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218826:2219048 [1] NCCL INFO Channel 22/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218825:2219049 [0] NCCL INFO Channel 19/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218826:2219048 [1] NCCL INFO Channel 23/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218825:2219049 [0] NCCL INFO Channel 20/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218825:2219049 [0] NCCL INFO Channel 21/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218825:2219049 [0] NCCL INFO Channel 22/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218825:2219049 [0] NCCL INFO Channel 23/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2218826:2219048 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:2218825:2219049 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-09:16:23:25 INFO     [evaluator:559] Running loglikelihood requests
2025-12-09:16:23:25 INFO     [evaluator:559] Running loglikelihood requests

Running loglikelihood requests:   0%|          | 0/20084 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 57
Determined largest batch size: 57

Running loglikelihood requests:   0%|          | 1/20084 [00:40<225:05:18, 40.35s/it]
Running loglikelihood requests:   0%|          | 58/20084 [00:45<3:14:47,  1.71it/s] 
Running loglikelihood requests:   1%|          | 115/20084 [00:50<1:36:27,  3.45it/s]
Running loglikelihood requests:   1%|          | 172/20084 [00:55<1:04:36,  5.14it/s]
Running loglikelihood requests:   1%|          | 229/20084 [00:59<49:27,  6.69it/s]  
Running loglikelihood requests:   1%|▏         | 286/20084 [01:04<41:02,  8.04it/s]
Running loglikelihood requests:   2%|▏         | 343/20084 [01:08<35:53,  9.17it/s]
Running loglikelihood requests:   2%|▏         | 400/20084 [01:13<32:32, 10.08it/s]
Running loglikelihood requests:   2%|▏         | 457/20084 [01:17<30:19, 10.79it/s]
Running loglikelihood requests:   3%|▎         | 514/20084 [01:22<28:46, 11.34it/s]
Running loglikelihood requests:   3%|▎         | 571/20084 [01:26<27:40, 11.75it/s]
Running loglikelihood requests:   3%|▎         | 628/20084 [01:30<26:49, 12.09it/s]
Running loglikelihood requests:   3%|▎         | 685/20084 [01:35<26:12, 12.34it/s]
Running loglikelihood requests:   4%|▎         | 742/20084 [01:39<25:43, 12.53it/s]
Running loglikelihood requests:   4%|▍         | 799/20084 [01:44<25:22, 12.67it/s]
Running loglikelihood requests:   4%|▍         | 856/20084 [01:48<25:04, 12.78it/s]
Running loglikelihood requests:   5%|▍         | 913/20084 [01:52<24:50, 12.86it/s]
Running loglikelihood requests:   5%|▍         | 970/20084 [01:57<24:38, 12.93it/s]
Running loglikelihood requests:   5%|▌         | 1027/20084 [02:01<24:27, 12.98it/s]
Running loglikelihood requests:   5%|▌         | 1084/20084 [02:05<24:16, 13.04it/s]
Running loglikelihood requests:   6%|▌         | 1141/20084 [02:10<24:07, 13.09it/s]
Running loglikelihood requests:   6%|▌         | 1198/20084 [02:14<23:58, 13.13it/s]
Running loglikelihood requests:   6%|▌         | 1255/20084 [02:18<23:49, 13.17it/s]
Running loglikelihood requests:   7%|▋         | 1312/20084 [02:23<23:42, 13.20it/s]
Running loglikelihood requests:   7%|▋         | 1369/20084 [02:27<23:35, 13.22it/s]
Running loglikelihood requests:   7%|▋         | 1426/20084 [02:31<23:28, 13.25it/s]
Running loglikelihood requests:   7%|▋         | 1483/20084 [02:35<23:21, 13.27it/s]
Running loglikelihood requests:   8%|▊         | 1540/20084 [02:40<23:13, 13.30it/s]
Running loglikelihood requests:   8%|▊         | 1597/20084 [02:44<23:07, 13.33it/s]
Running loglikelihood requests:   8%|▊         | 1654/20084 [02:48<22:59, 13.36it/s]
Running loglikelihood requests:   9%|▊         | 1711/20084 [02:53<22:53, 13.38it/s]
Running loglikelihood requests:   9%|▉         | 1768/20084 [02:57<22:46, 13.40it/s]
Running loglikelihood requests:   9%|▉         | 1825/20084 [03:01<22:39, 13.43it/s]
Running loglikelihood requests:   9%|▉         | 1882/20084 [03:05<22:33, 13.45it/s]
Running loglikelihood requests:  10%|▉         | 1939/20084 [03:09<22:27, 13.47it/s]
Running loglikelihood requests:  10%|▉         | 1996/20084 [03:14<22:20, 13.49it/s]
Running loglikelihood requests:  10%|█         | 2053/20084 [03:18<22:14, 13.51it/s]
Running loglikelihood requests:  11%|█         | 2110/20084 [03:22<22:08, 13.53it/s]
Running loglikelihood requests:  11%|█         | 2167/20084 [03:26<22:02, 13.55it/s]
Running loglikelihood requests:  11%|█         | 2224/20084 [03:30<21:55, 13.58it/s]
Running loglikelihood requests:  11%|█▏        | 2281/20084 [03:35<21:48, 13.61it/s]
Running loglikelihood requests:  12%|█▏        | 2338/20084 [03:39<21:41, 13.64it/s]
Running loglikelihood requests:  12%|█▏        | 2395/20084 [03:43<21:35, 13.66it/s]
Running loglikelihood requests:  12%|█▏        | 2452/20084 [03:47<21:28, 13.68it/s]
Running loglikelihood requests:  12%|█▏        | 2509/20084 [03:51<21:22, 13.70it/s]
Running loglikelihood requests:  13%|█▎        | 2566/20084 [03:55<21:16, 13.72it/s]
Running loglikelihood requests:  13%|█▎        | 2623/20084 [03:59<21:10, 13.74it/s]
Running loglikelihood requests:  13%|█▎        | 2680/20084 [04:04<21:04, 13.76it/s]
Running loglikelihood requests:  14%|█▎        | 2737/20084 [04:08<20:58, 13.79it/s]
Running loglikelihood requests:  14%|█▍        | 2794/20084 [04:12<20:52, 13.80it/s]
Running loglikelihood requests:  14%|█▍        | 2851/20084 [04:16<20:46, 13.82it/s]
Running loglikelihood requests:  14%|█▍        | 2908/20084 [04:20<20:41, 13.83it/s]
Running loglikelihood requests:  15%|█▍        | 2965/20084 [04:24<20:36, 13.85it/s]
Running loglikelihood requests:  15%|█▌        | 3022/20084 [04:28<20:31, 13.85it/s]
Running loglikelihood requests:  15%|█▌        | 3079/20084 [04:32<20:26, 13.87it/s]
Running loglikelihood requests:  16%|█▌        | 3136/20084 [04:36<20:20, 13.88it/s]
Running loglikelihood requests:  16%|█▌        | 3193/20084 [04:41<20:15, 13.90it/s]
Running loglikelihood requests:  16%|█▌        | 3250/20084 [04:45<20:10, 13.91it/s]
Running loglikelihood requests:  16%|█▋        | 3307/20084 [04:49<20:04, 13.93it/s]
Running loglikelihood requests:  17%|█▋        | 3364/20084 [04:53<19:59, 13.94it/s]
Running loglikelihood requests:  17%|█▋        | 3421/20084 [04:57<19:53, 13.96it/s]
Running loglikelihood requests:  17%|█▋        | 3478/20084 [05:01<19:48, 13.98it/s]
Running loglikelihood requests:  18%|█▊        | 3535/20084 [05:05<19:42, 13.99it/s]
Running loglikelihood requests:  18%|█▊        | 3592/20084 [05:09<19:37, 14.00it/s]
Running loglikelihood requests:  18%|█▊        | 3649/20084 [05:13<19:32, 14.02it/s]
Running loglikelihood requests:  18%|█▊        | 3706/20084 [05:17<19:27, 14.03it/s]
Running loglikelihood requests:  19%|█▊        | 3763/20084 [05:21<19:22, 14.05it/s]
Running loglikelihood requests:  19%|█▉        | 3820/20084 [05:25<19:16, 14.06it/s]
Running loglikelihood requests:  19%|█▉        | 3877/20084 [05:29<19:11, 14.08it/s]
Running loglikelihood requests:  20%|█▉        | 3934/20084 [05:33<19:06, 14.09it/s]
Running loglikelihood requests:  20%|█▉        | 3991/20084 [05:37<19:00, 14.10it/s]
Running loglikelihood requests:  20%|██        | 4048/20084 [05:41<18:55, 14.12it/s]
Running loglikelihood requests:  20%|██        | 4105/20084 [05:45<18:50, 14.14it/s]
Running loglikelihood requests:  21%|██        | 4162/20084 [05:49<18:45, 14.15it/s]
Running loglikelihood requests:  21%|██        | 4219/20084 [05:53<18:40, 14.16it/s]
Running loglikelihood requests:  21%|██▏       | 4276/20084 [05:57<18:35, 14.17it/s]
Running loglikelihood requests:  22%|██▏       | 4333/20084 [06:01<18:30, 14.18it/s]
Running loglikelihood requests:  22%|██▏       | 4390/20084 [06:05<18:25, 14.20it/s]
Running loglikelihood requests:  22%|██▏       | 4447/20084 [06:09<18:20, 14.21it/s]
Running loglikelihood requests:  22%|██▏       | 4504/20084 [06:13<18:15, 14.22it/s]
Running loglikelihood requests:  23%|██▎       | 4561/20084 [06:17<18:10, 14.23it/s]
Running loglikelihood requests:  23%|██▎       | 4618/20084 [06:21<18:01, 14.30it/s]
Running loglikelihood requests:  23%|██▎       | 4675/20084 [06:25<17:55, 14.33it/s]
Running loglikelihood requests:  24%|██▎       | 4732/20084 [06:29<17:48, 14.37it/s]
Running loglikelihood requests:  24%|██▍       | 4789/20084 [06:33<17:42, 14.39it/s]
Running loglikelihood requests:  24%|██▍       | 4846/20084 [06:37<17:37, 14.41it/s]
Running loglikelihood requests:  24%|██▍       | 4903/20084 [06:41<17:34, 14.39it/s]
Running loglikelihood requests:  25%|██▍       | 4960/20084 [06:45<17:31, 14.38it/s]
Running loglikelihood requests:  25%|██▍       | 5017/20084 [06:49<17:29, 14.35it/s]
Running loglikelihood requests:  25%|██▌       | 5074/20084 [06:53<17:25, 14.36it/s]
Running loglikelihood requests:  26%|██▌       | 5131/20084 [06:57<17:20, 14.37it/s]
Running loglikelihood requests:  26%|██▌       | 5188/20084 [07:01<17:15, 14.38it/s]
Running loglikelihood requests:  26%|██▌       | 5245/20084 [07:05<17:10, 14.40it/s]
Running loglikelihood requests:  26%|██▋       | 5302/20084 [07:09<17:08, 14.37it/s]
Running loglikelihood requests:  27%|██▋       | 5359/20084 [07:13<17:03, 14.38it/s]
Running loglikelihood requests:  27%|██▋       | 5416/20084 [07:17<16:58, 14.40it/s]
Running loglikelihood requests:  27%|██▋       | 5473/20084 [07:21<16:54, 14.40it/s]
Running loglikelihood requests:  28%|██▊       | 5530/20084 [07:25<16:48, 14.43it/s]
Running loglikelihood requests:  28%|██▊       | 5587/20084 [07:29<16:43, 14.45it/s]
Running loglikelihood requests:  28%|██▊       | 5644/20084 [07:33<16:37, 14.47it/s]
Running loglikelihood requests:  28%|██▊       | 5701/20084 [07:37<16:32, 14.49it/s]
Running loglikelihood requests:  29%|██▊       | 5758/20084 [07:40<16:27, 14.50it/s]
Running loglikelihood requests:  29%|██▉       | 5815/20084 [07:44<16:20, 14.55it/s]
Running loglikelihood requests:  29%|██▉       | 5872/20084 [07:48<16:13, 14.60it/s]
Running loglikelihood requests:  30%|██▉       | 5929/20084 [07:52<16:07, 14.63it/s]
Running loglikelihood requests:  30%|██▉       | 5986/20084 [07:56<16:02, 14.65it/s]
Running loglikelihood requests:  30%|███       | 6043/20084 [08:00<15:59, 14.63it/s]
Running loglikelihood requests:  30%|███       | 6100/20084 [08:04<15:54, 14.65it/s]
Running loglikelihood requests:  31%|███       | 6157/20084 [08:08<15:49, 14.67it/s]
Running loglikelihood requests:  31%|███       | 6214/20084 [08:11<15:44, 14.68it/s]
Running loglikelihood requests:  31%|███       | 6271/20084 [08:15<15:39, 14.70it/s]
Running loglikelihood requests:  32%|███▏      | 6328/20084 [08:19<15:34, 14.71it/s]
Running loglikelihood requests:  32%|███▏      | 6385/20084 [08:23<15:29, 14.74it/s]
Running loglikelihood requests:  32%|███▏      | 6442/20084 [08:27<15:25, 14.74it/s]
Running loglikelihood requests:  32%|███▏      | 6499/20084 [08:31<15:21, 14.75it/s]
Running loglikelihood requests:  33%|███▎      | 6556/20084 [08:35<15:16, 14.76it/s]
Running loglikelihood requests:  33%|███▎      | 6613/20084 [08:38<15:11, 14.78it/s]
Running loglikelihood requests:  33%|███▎      | 6670/20084 [08:42<15:07, 14.78it/s]
Running loglikelihood requests:  33%|███▎      | 6727/20084 [08:46<15:02, 14.80it/s]
Running loglikelihood requests:  34%|███▍      | 6784/20084 [08:50<14:57, 14.82it/s]
Running loglikelihood requests:  34%|███▍      | 6841/20084 [08:54<14:52, 14.83it/s]
Running loglikelihood requests:  34%|███▍      | 6898/20084 [08:58<14:48, 14.84it/s]
Running loglikelihood requests:  35%|███▍      | 6955/20084 [09:02<14:43, 14.85it/s]
Running loglikelihood requests:  35%|███▍      | 7012/20084 [09:05<14:39, 14.86it/s]
Running loglikelihood requests:  35%|███▌      | 7069/20084 [09:09<14:35, 14.87it/s]
Running loglikelihood requests:  35%|███▌      | 7126/20084 [09:13<14:29, 14.90it/s]
Running loglikelihood requests:  36%|███▌      | 7183/20084 [09:17<14:23, 14.95it/s]
Running loglikelihood requests:  36%|███▌      | 7240/20084 [09:21<14:17, 14.98it/s]
Running loglikelihood requests:  36%|███▋      | 7297/20084 [09:24<14:14, 14.97it/s]
Running loglikelihood requests:  37%|███▋      | 7354/20084 [09:28<14:10, 14.97it/s]
Running loglikelihood requests:  37%|███▋      | 7411/20084 [09:32<14:07, 14.96it/s]
Running loglikelihood requests:  37%|███▋      | 7468/20084 [09:36<14:03, 14.95it/s]
Running loglikelihood requests:  37%|███▋      | 7525/20084 [09:40<13:59, 14.96it/s]
Running loglikelihood requests:  38%|███▊      | 7582/20084 [09:43<13:55, 14.97it/s]
Running loglikelihood requests:  38%|███▊      | 7639/20084 [09:47<13:51, 14.98it/s]
Running loglikelihood requests:  38%|███▊      | 7696/20084 [09:51<13:46, 14.99it/s]
Running loglikelihood requests:  39%|███▊      | 7753/20084 [09:55<13:41, 15.00it/s]
Running loglikelihood requests:  39%|███▉      | 7810/20084 [09:59<13:37, 15.01it/s]
Running loglikelihood requests:  39%|███▉      | 7867/20084 [10:02<13:32, 15.03it/s]
Running loglikelihood requests:  39%|███▉      | 7924/20084 [10:06<13:28, 15.04it/s]
Running loglikelihood requests:  40%|███▉      | 7981/20084 [10:10<13:23, 15.06it/s]
Running loglikelihood requests:  40%|████      | 8038/20084 [10:14<13:19, 15.07it/s]
Running loglikelihood requests:  40%|████      | 8095/20084 [10:17<13:15, 15.08it/s]
Running loglikelihood requests:  41%|████      | 8152/20084 [10:21<13:10, 15.09it/s]
Running loglikelihood requests:  41%|████      | 8209/20084 [10:25<13:05, 15.11it/s]
Running loglikelihood requests:  41%|████      | 8266/20084 [10:29<13:01, 15.12it/s]
Running loglikelihood requests:  41%|████▏     | 8323/20084 [10:33<12:57, 15.12it/s]
Running loglikelihood requests:  42%|████▏     | 8380/20084 [10:36<12:53, 15.14it/s]
Running loglikelihood requests:  42%|████▏     | 8437/20084 [10:40<12:49, 15.14it/s]
Running loglikelihood requests:  42%|████▏     | 8494/20084 [10:44<12:44, 15.15it/s]
Running loglikelihood requests:  43%|████▎     | 8551/20084 [10:48<12:38, 15.21it/s]
Running loglikelihood requests:  43%|████▎     | 8608/20084 [10:51<12:32, 15.25it/s]
Running loglikelihood requests:  43%|████▎     | 8665/20084 [10:55<12:27, 15.28it/s]
Running loglikelihood requests:  43%|████▎     | 8722/20084 [10:59<12:24, 15.27it/s]
Running loglikelihood requests:  44%|████▎     | 8779/20084 [11:02<12:21, 15.26it/s]
Running loglikelihood requests:  44%|████▍     | 8836/20084 [11:06<12:17, 15.25it/s]
Running loglikelihood requests:  44%|████▍     | 8893/20084 [11:10<12:13, 15.25it/s]
Running loglikelihood requests:  45%|████▍     | 8950/20084 [11:14<12:10, 15.24it/s]
Running loglikelihood requests:  45%|████▍     | 9007/20084 [11:17<12:06, 15.25it/s]
Running loglikelihood requests:  45%|████▌     | 9064/20084 [11:21<12:02, 15.25it/s]
Running loglikelihood requests:  45%|████▌     | 9121/20084 [11:25<11:56, 15.30it/s]
Running loglikelihood requests:  46%|████▌     | 9178/20084 [11:29<11:51, 15.34it/s]
Running loglikelihood requests:  46%|████▌     | 9235/20084 [11:32<11:46, 15.36it/s]
Running loglikelihood requests:  46%|████▋     | 9292/20084 [11:36<11:42, 15.36it/s]
Running loglikelihood requests:  47%|████▋     | 9349/20084 [11:40<11:39, 15.34it/s]
Running loglikelihood requests:  47%|████▋     | 9406/20084 [11:43<11:36, 15.34it/s]
Running loglikelihood requests:  47%|████▋     | 9463/20084 [11:47<11:32, 15.33it/s]
Running loglikelihood requests:  47%|████▋     | 9520/20084 [11:51<11:28, 15.34it/s]
Running loglikelihood requests:  48%|████▊     | 9577/20084 [11:55<11:24, 15.35it/s]
Running loglikelihood requests:  48%|████▊     | 9634/20084 [11:58<11:20, 15.35it/s]
Running loglikelihood requests:  48%|████▊     | 9691/20084 [12:02<11:14, 15.41it/s]
Running loglikelihood requests:  49%|████▊     | 9748/20084 [12:06<11:08, 15.45it/s]
Running loglikelihood requests:  49%|████▉     | 9805/20084 [12:09<11:03, 15.50it/s]
Running loglikelihood requests:  49%|████▉     | 9862/20084 [12:13<10:58, 15.52it/s]
Running loglikelihood requests:  49%|████▉     | 9919/20084 [12:17<10:54, 15.54it/s]
Running loglikelihood requests:  50%|████▉     | 9976/20084 [12:20<10:51, 15.52it/s]
Running loglikelihood requests:  50%|████▉     | 10033/20084 [12:24<10:47, 15.51it/s]
Running loglikelihood requests:  50%|█████     | 10090/20084 [12:28<10:44, 15.51it/s]
Running loglikelihood requests:  51%|█████     | 10147/20084 [12:31<10:40, 15.50it/s]
Running loglikelihood requests:  51%|█████     | 10204/20084 [12:35<10:37, 15.51it/s]
Running loglikelihood requests:  51%|█████     | 10261/20084 [12:39<10:33, 15.52it/s]
Running loglikelihood requests:  51%|█████▏    | 10318/20084 [12:42<10:29, 15.52it/s]
Running loglikelihood requests:  52%|█████▏    | 10375/20084 [12:46<10:25, 15.53it/s]
Running loglikelihood requests:  52%|█████▏    | 10432/20084 [12:50<10:19, 15.58it/s]
Running loglikelihood requests:  52%|█████▏    | 10489/20084 [12:53<10:14, 15.62it/s]
Running loglikelihood requests:  53%|█████▎    | 10546/20084 [12:57<10:09, 15.65it/s]
Running loglikelihood requests:  53%|█████▎    | 10603/20084 [13:00<10:04, 15.68it/s]
Running loglikelihood requests:  53%|█████▎    | 10660/20084 [13:04<10:00, 15.69it/s]
Running loglikelihood requests:  53%|█████▎    | 10717/20084 [13:08<09:56, 15.71it/s]
Running loglikelihood requests:  54%|█████▎    | 10774/20084 [13:11<09:52, 15.71it/s]
Running loglikelihood requests:  54%|█████▍    | 10831/20084 [13:15<09:49, 15.69it/s]
Running loglikelihood requests:  54%|█████▍    | 10888/20084 [13:19<09:46, 15.68it/s]
Running loglikelihood requests:  54%|█████▍    | 10945/20084 [13:22<09:43, 15.66it/s]
Running loglikelihood requests:  55%|█████▍    | 11002/20084 [13:26<09:40, 15.66it/s]
Running loglikelihood requests:  55%|█████▌    | 11059/20084 [13:30<09:35, 15.67it/s]
Running loglikelihood requests:  55%|█████▌    | 11116/20084 [13:33<09:32, 15.68it/s]
Running loglikelihood requests:  56%|█████▌    | 11173/20084 [13:37<09:28, 15.68it/s]
Running loglikelihood requests:  56%|█████▌    | 11230/20084 [13:40<09:22, 15.74it/s]
Running loglikelihood requests:  56%|█████▌    | 11287/20084 [13:44<09:17, 15.78it/s]
Running loglikelihood requests:  56%|█████▋    | 11344/20084 [13:48<09:12, 15.81it/s]
Running loglikelihood requests:  57%|█████▋    | 11401/20084 [13:51<09:08, 15.83it/s]
Running loglikelihood requests:  57%|█████▋    | 11458/20084 [13:55<09:04, 15.85it/s]
Running loglikelihood requests:  57%|█████▋    | 11515/20084 [13:58<09:00, 15.87it/s]
Running loglikelihood requests:  58%|█████▊    | 11572/20084 [14:02<08:57, 15.85it/s]
Running loglikelihood requests:  58%|█████▊    | 11629/20084 [14:06<08:53, 15.83it/s]
Running loglikelihood requests:  58%|█████▊    | 11686/20084 [14:09<08:50, 15.82it/s]
Running loglikelihood requests:  58%|█████▊    | 11743/20084 [14:13<08:47, 15.81it/s]
Running loglikelihood requests:  59%|█████▉    | 11800/20084 [14:16<08:43, 15.82it/s]
Running loglikelihood requests:  59%|█████▉    | 11857/20084 [14:20<08:40, 15.82it/s]
Running loglikelihood requests:  59%|█████▉    | 11914/20084 [14:24<08:36, 15.83it/s]
Running loglikelihood requests:  60%|█████▉    | 11971/20084 [14:27<08:31, 15.88it/s]
Running loglikelihood requests:  60%|█████▉    | 12028/20084 [14:31<08:25, 15.92it/s]
Running loglikelihood requests:  60%|██████    | 12085/20084 [14:34<08:21, 15.95it/s]
Running loglikelihood requests:  60%|██████    | 12142/20084 [14:38<08:16, 15.98it/s]
Running loglikelihood requests:  61%|██████    | 12199/20084 [14:41<08:12, 16.01it/s]
Running loglikelihood requests:  61%|██████    | 12256/20084 [14:45<08:09, 15.99it/s]
Running loglikelihood requests:  61%|██████▏   | 12313/20084 [14:48<08:06, 15.98it/s]
Running loglikelihood requests:  62%|██████▏   | 12370/20084 [14:52<08:02, 15.98it/s]
Running loglikelihood requests:  62%|██████▏   | 12427/20084 [14:56<07:59, 15.98it/s]
Running loglikelihood requests:  62%|██████▏   | 12484/20084 [14:59<07:55, 15.99it/s]
Running loglikelihood requests:  62%|██████▏   | 12541/20084 [15:03<07:51, 15.99it/s]
Running loglikelihood requests:  63%|██████▎   | 12598/20084 [15:06<07:48, 15.99it/s]
Running loglikelihood requests:  63%|██████▎   | 12655/20084 [15:10<07:44, 15.99it/s]
Running loglikelihood requests:  63%|██████▎   | 12712/20084 [15:13<07:39, 16.05it/s]
Running loglikelihood requests:  64%|██████▎   | 12769/20084 [15:17<07:34, 16.09it/s]
Running loglikelihood requests:  64%|██████▍   | 12826/20084 [15:20<07:30, 16.12it/s]
Running loglikelihood requests:  64%|██████▍   | 12883/20084 [15:24<07:25, 16.15it/s]
Running loglikelihood requests:  64%|██████▍   | 12940/20084 [15:27<07:21, 16.17it/s]
Running loglikelihood requests:  65%|██████▍   | 12997/20084 [15:31<07:18, 16.15it/s]
Running loglikelihood requests:  65%|██████▍   | 13054/20084 [15:35<07:15, 16.15it/s]
Running loglikelihood requests:  65%|██████▌   | 13111/20084 [15:38<07:10, 16.18it/s]
Running loglikelihood requests:  66%|██████▌   | 13168/20084 [15:42<07:06, 16.22it/s]
Running loglikelihood requests:  66%|██████▌   | 13225/20084 [15:45<07:02, 16.24it/s]
Running loglikelihood requests:  66%|██████▌   | 13282/20084 [15:49<06:58, 16.26it/s]
Running loglikelihood requests:  66%|██████▋   | 13339/20084 [15:52<06:54, 16.28it/s]
Running loglikelihood requests:  67%|██████▋   | 13396/20084 [15:56<06:50, 16.29it/s]
Running loglikelihood requests:  67%|██████▋   | 13453/20084 [15:59<06:46, 16.30it/s]
Running loglikelihood requests:  67%|██████▋   | 13510/20084 [16:02<06:42, 16.32it/s]
Running loglikelihood requests:  68%|██████▊   | 13567/20084 [16:06<06:38, 16.33it/s]
Running loglikelihood requests:  68%|██████▊   | 13624/20084 [16:09<06:35, 16.32it/s]
Running loglikelihood requests:  68%|██████▊   | 13681/20084 [16:13<06:32, 16.30it/s]
Running loglikelihood requests:  68%|██████▊   | 13738/20084 [16:16<06:29, 16.30it/s]
Running loglikelihood requests:  69%|██████▊   | 13795/20084 [16:20<06:25, 16.30it/s]
Running loglikelihood requests:  69%|██████▉   | 13852/20084 [16:23<06:22, 16.31it/s]
Running loglikelihood requests:  69%|██████▉   | 13909/20084 [16:27<06:18, 16.31it/s]
Running loglikelihood requests:  70%|██████▉   | 13966/20084 [16:30<06:14, 16.33it/s]
Running loglikelihood requests:  70%|██████▉   | 14023/20084 [16:34<06:09, 16.38it/s]
Running loglikelihood requests:  70%|███████   | 14080/20084 [16:37<06:05, 16.42it/s]
Running loglikelihood requests:  70%|███████   | 14137/20084 [16:41<06:01, 16.46it/s]
Running loglikelihood requests:  71%|███████   | 14194/20084 [16:44<05:57, 16.50it/s]
Running loglikelihood requests:  71%|███████   | 14251/20084 [16:48<05:53, 16.52it/s]
Running loglikelihood requests:  71%|███████   | 14308/20084 [16:51<05:49, 16.54it/s]
Running loglikelihood requests:  72%|███████▏  | 14365/20084 [16:55<05:45, 16.56it/s]
Running loglikelihood requests:  72%|███████▏  | 14422/20084 [16:58<05:41, 16.58it/s]
Running loglikelihood requests:  72%|███████▏  | 14479/20084 [17:01<05:37, 16.59it/s]
Running loglikelihood requests:  72%|███████▏  | 14536/20084 [17:05<05:34, 16.61it/s]
Running loglikelihood requests:  73%|███████▎  | 14593/20084 [17:08<05:30, 16.62it/s]
Running loglikelihood requests:  73%|███████▎  | 14650/20084 [17:12<05:26, 16.64it/s]
Running loglikelihood requests:  73%|███████▎  | 14707/20084 [17:15<05:23, 16.62it/s]
Running loglikelihood requests:  74%|███████▎  | 14764/20084 [17:19<05:20, 16.61it/s]
Running loglikelihood requests:  74%|███████▍  | 14821/20084 [17:22<05:16, 16.60it/s]
Running loglikelihood requests:  74%|███████▍  | 14878/20084 [17:25<05:13, 16.60it/s]
Running loglikelihood requests:  74%|███████▍  | 14935/20084 [17:29<05:10, 16.60it/s]
Running loglikelihood requests:  75%|███████▍  | 14992/20084 [17:32<05:06, 16.61it/s]
Running loglikelihood requests:  75%|███████▍  | 15049/20084 [17:36<05:02, 16.63it/s]
Running loglikelihood requests:  75%|███████▌  | 15106/20084 [17:39<04:58, 16.68it/s]
Running loglikelihood requests:  75%|███████▌  | 15163/20084 [17:42<04:54, 16.73it/s]
Running loglikelihood requests:  76%|███████▌  | 15220/20084 [17:46<04:50, 16.76it/s]
Running loglikelihood requests:  76%|███████▌  | 15277/20084 [17:49<04:46, 16.79it/s]
Running loglikelihood requests:  76%|███████▋  | 15334/20084 [17:53<04:42, 16.83it/s]
Running loglikelihood requests:  77%|███████▋  | 15391/20084 [17:56<04:38, 16.84it/s]
Running loglikelihood requests:  77%|███████▋  | 15448/20084 [17:59<04:34, 16.88it/s]
Running loglikelihood requests:  77%|███████▋  | 15505/20084 [18:03<04:31, 16.89it/s]
Running loglikelihood requests:  77%|███████▋  | 15562/20084 [18:06<04:27, 16.90it/s]
Running loglikelihood requests:  78%|███████▊  | 15619/20084 [18:09<04:23, 16.92it/s]
Running loglikelihood requests:  78%|███████▊  | 15676/20084 [18:13<04:20, 16.94it/s]
Running loglikelihood requests:  78%|███████▊  | 15733/20084 [18:16<04:16, 16.94it/s]
Running loglikelihood requests:  79%|███████▊  | 15790/20084 [18:19<04:12, 16.99it/s]
Running loglikelihood requests:  79%|███████▉  | 15847/20084 [18:23<04:09, 17.01it/s]
Running loglikelihood requests:  79%|███████▉  | 15904/20084 [18:26<04:05, 17.03it/s]
Running loglikelihood requests:  79%|███████▉  | 15961/20084 [18:30<04:02, 17.00it/s]
Running loglikelihood requests:  80%|███████▉  | 16018/20084 [18:33<03:59, 16.99it/s]
Running loglikelihood requests:  80%|████████  | 16075/20084 [18:36<03:55, 16.99it/s]
Running loglikelihood requests:  80%|████████  | 16132/20084 [18:40<03:52, 17.01it/s]
Running loglikelihood requests:  81%|████████  | 16189/20084 [18:43<03:48, 17.06it/s]
Running loglikelihood requests:  81%|████████  | 16246/20084 [18:46<03:44, 17.11it/s]
Running loglikelihood requests:  81%|████████  | 16303/20084 [18:50<03:40, 17.16it/s]
Running loglikelihood requests:  81%|████████▏ | 16360/20084 [18:53<03:36, 17.20it/s]
Running loglikelihood requests:  82%|████████▏ | 16417/20084 [18:56<03:32, 17.23it/s]
Running loglikelihood requests:  82%|████████▏ | 16474/20084 [18:59<03:29, 17.25it/s]
Running loglikelihood requests:  82%|████████▏ | 16531/20084 [19:03<03:25, 17.27it/s]
Running loglikelihood requests:  83%|████████▎ | 16588/20084 [19:06<03:22, 17.31it/s]
Running loglikelihood requests:  83%|████████▎ | 16645/20084 [19:09<03:18, 17.32it/s]
Running loglikelihood requests:  83%|████████▎ | 16702/20084 [19:13<03:15, 17.34it/s]
Running loglikelihood requests:  83%|████████▎ | 16759/20084 [19:16<03:11, 17.37it/s]
Running loglikelihood requests:  84%|████████▎ | 16816/20084 [19:19<03:07, 17.39it/s]
Running loglikelihood requests:  84%|████████▍ | 16873/20084 [19:22<03:04, 17.42it/s]
Running loglikelihood requests:  84%|████████▍ | 16930/20084 [19:26<03:01, 17.40it/s]
Running loglikelihood requests:  85%|████████▍ | 16987/20084 [19:29<02:57, 17.40it/s]
Running loglikelihood requests:  85%|████████▍ | 17044/20084 [19:32<02:54, 17.41it/s]
Running loglikelihood requests:  85%|████████▌ | 17101/20084 [19:35<02:50, 17.46it/s]
Running loglikelihood requests:  85%|████████▌ | 17158/20084 [19:39<02:46, 17.53it/s]
Running loglikelihood requests:  86%|████████▌ | 17215/20084 [19:42<02:43, 17.58it/s]
Running loglikelihood requests:  86%|████████▌ | 17272/20084 [19:45<02:39, 17.62it/s]
Running loglikelihood requests:  86%|████████▋ | 17329/20084 [19:48<02:36, 17.64it/s]
Running loglikelihood requests:  87%|████████▋ | 17386/20084 [19:51<02:32, 17.66it/s]
Running loglikelihood requests:  87%|████████▋ | 17443/20084 [19:55<02:29, 17.69it/s]
Running loglikelihood requests:  87%|████████▋ | 17500/20084 [19:58<02:25, 17.73it/s]
Running loglikelihood requests:  87%|████████▋ | 17557/20084 [20:01<02:22, 17.75it/s]
Running loglikelihood requests:  88%|████████▊ | 17614/20084 [20:04<02:18, 17.78it/s]
Running loglikelihood requests:  88%|████████▊ | 17671/20084 [20:07<02:15, 17.80it/s]
Running loglikelihood requests:  88%|████████▊ | 17728/20084 [20:11<02:12, 17.83it/s]
Running loglikelihood requests:  89%|████████▊ | 17785/20084 [20:14<02:08, 17.88it/s]
Running loglikelihood requests:  89%|████████▉ | 17842/20084 [20:17<02:04, 17.94it/s]
Running loglikelihood requests:  89%|████████▉ | 17899/20084 [20:20<02:01, 17.97it/s]
Running loglikelihood requests:  89%|████████▉ | 17956/20084 [20:23<01:58, 18.02it/s]
Running loglikelihood requests:  90%|████████▉ | 18013/20084 [20:26<01:54, 18.05it/s]
Running loglikelihood requests:  90%|████████▉ | 18070/20084 [20:30<01:51, 18.05it/s]
Running loglikelihood requests:  90%|█████████ | 18127/20084 [20:33<01:48, 18.06it/s]
Running loglikelihood requests:  91%|█████████ | 18184/20084 [20:36<01:44, 18.11it/s]
Running loglikelihood requests:  91%|█████████ | 18241/20084 [20:39<01:41, 18.12it/s]
Running loglikelihood requests:  91%|█████████ | 18298/20084 [20:42<01:38, 18.14it/s]
Running loglikelihood requests:  91%|█████████▏| 18355/20084 [20:45<01:35, 18.18it/s]
Running loglikelihood requests:  92%|█████████▏| 18412/20084 [20:48<01:31, 18.25it/s]
Running loglikelihood requests:  92%|█████████▏| 18469/20084 [20:51<01:28, 18.28it/s]
Running loglikelihood requests:  92%|█████████▏| 18526/20084 [20:55<01:24, 18.33it/s]
Running loglikelihood requests:  93%|█████████▎| 18583/20084 [20:58<01:21, 18.37it/s]
Running loglikelihood requests:  93%|█████████▎| 18640/20084 [21:01<01:18, 18.40it/s]
Running loglikelihood requests:  93%|█████████▎| 18697/20084 [21:04<01:15, 18.48it/s]
Running loglikelihood requests:  93%|█████████▎| 18754/20084 [21:07<01:11, 18.52it/s]
Running loglikelihood requests:  94%|█████████▎| 18811/20084 [21:10<01:08, 18.59it/s]
Running loglikelihood requests:  94%|█████████▍| 18868/20084 [21:13<01:05, 18.66it/s]
Running loglikelihood requests:  94%|█████████▍| 18925/20084 [21:16<01:01, 18.75it/s]
Running loglikelihood requests:  95%|█████████▍| 18982/20084 [21:19<00:58, 18.79it/s]
Running loglikelihood requests:  95%|█████████▍| 19039/20084 [21:22<00:55, 18.84it/s]
Running loglikelihood requests:  95%|█████████▌| 19096/20084 [21:25<00:52, 18.90it/s]
Running loglikelihood requests:  95%|█████████▌| 19153/20084 [21:28<00:49, 18.99it/s]
Running loglikelihood requests:  96%|█████████▌| 19210/20084 [21:31<00:45, 19.07it/s]
Running loglikelihood requests:  96%|█████████▌| 19267/20084 [21:34<00:42, 19.17it/s]
Running loglikelihood requests:  96%|█████████▌| 19324/20084 [21:37<00:39, 19.25it/s]
Running loglikelihood requests:  96%|█████████▋| 19381/20084 [21:40<00:36, 19.30it/s]
Running loglikelihood requests:  97%|█████████▋| 19438/20084 [21:43<00:33, 19.39it/s]
Running loglikelihood requests:  97%|█████████▋| 19495/20084 [21:45<00:30, 19.48it/s]
Running loglikelihood requests:  97%|█████████▋| 19552/20084 [21:48<00:27, 19.58it/s]
Running loglikelihood requests:  98%|█████████▊| 19609/20084 [21:51<00:24, 19.68it/s]
Running loglikelihood requests:  98%|█████████▊| 19666/20084 [21:54<00:21, 19.78it/s]
Running loglikelihood requests:  98%|█████████▊| 19723/20084 [21:57<00:18, 19.94it/s]
Running loglikelihood requests:  98%|█████████▊| 19780/20084 [22:00<00:15, 20.11it/s]
Running loglikelihood requests:  99%|█████████▉| 19837/20084 [22:02<00:12, 20.36it/s]
Running loglikelihood requests:  99%|█████████▉| 19894/20084 [22:05<00:09, 20.59it/s]
Running loglikelihood requests:  99%|█████████▉| 19951/20084 [22:08<00:06, 20.86it/s]
Running loglikelihood requests: 100%|█████████▉| 20008/20084 [22:10<00:03, 21.22it/s]
Running loglikelihood requests: 100%|█████████▉| 20065/20084 [22:11<00:00, 26.69it/s]
Running loglikelihood requests: 100%|██████████| 20084/20084 [22:11<00:00, 15.08it/s]
n136-128-154:2218826:2221263 [1] NCCL INFO Channel 00/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218826:2221263 [1] NCCL INFO Channel 01/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218826:2221263 [1] NCCL INFO Channel 02/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218826:2221263 [1] NCCL INFO Channel 03/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218826:2221263 [1] NCCL INFO Channel 04/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218826:2221263 [1] NCCL INFO Channel 05/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218826:2221263 [1] NCCL INFO Channel 06/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218826:2221263 [1] NCCL INFO Channel 07/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218826:2221263 [1] NCCL INFO Channel 08/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218826:2221263 [1] NCCL INFO Channel 09/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218826:2221263 [1] NCCL INFO Channel 10/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218826:2221263 [1] NCCL INFO Channel 11/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218826:2221263 [1] NCCL INFO Channel 12/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218826:2221263 [1] NCCL INFO Channel 13/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218826:2221263 [1] NCCL INFO Channel 14/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218826:2221263 [1] NCCL INFO Channel 15/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218826:2221263 [1] NCCL INFO Channel 16/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218826:2221263 [1] NCCL INFO Channel 17/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218826:2221263 [1] NCCL INFO Channel 18/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218826:2221263 [1] NCCL INFO Channel 19/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218826:2221263 [1] NCCL INFO Channel 20/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218826:2221263 [1] NCCL INFO Channel 21/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218826:2221263 [1] NCCL INFO Channel 22/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218826:2221263 [1] NCCL INFO Channel 23/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218826:2221263 [1] NCCL INFO Channel 24/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218826:2221263 [1] NCCL INFO Channel 25/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218826:2221263 [1] NCCL INFO Channel 26/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218826:2221263 [1] NCCL INFO Channel 27/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218826:2221263 [1] NCCL INFO Channel 28/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218826:2221263 [1] NCCL INFO Channel 29/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218826:2221263 [1] NCCL INFO Channel 30/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2218826:2221263 [1] NCCL INFO Channel 31/1 : 1[7] -> 0[6] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:2218826:2221268 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2218826:2221268 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2218826:2221268 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2218826:2221268 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2218826:2221268 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2218826:2221268 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2218826:2219044 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-09:16:47:07 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 10, batch_size: auto (57)
|  Tasks  |Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|---------|------:|------|-----:|--------|---|-----:|---|-----:|
|hellaswag|      1|none  |    10|acc     |↑  |0.3639|±  |0.0048|
|         |       |none  |    10|acc_norm|↑  |0.4923|±  |0.0050|

[rank0]:[W1209 16:47:08.661753181 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:2218825:2221318 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2218825:2221318 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2218825:2221318 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2218825:2219046 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2218825:2221318 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2218825:2221318 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2218825:2221318 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2218826:2219044 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2218826:2221268 [1] NCCL INFO comm 0xe7126f0 rank 1 nranks 2 cudaDev 1 busId c9000 - Abort COMPLETE
n136-128-154:2218825:2221318 [0] NCCL INFO comm 0xe3cc210 rank 0 nranks 2 cudaDev 0 busId c5000 - Abort COMPLETE
任务 hellaswag 评估完成！
Execution time: 2604.13 seconds
Namespace(save_dir='/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg', self_attn_layer_to_quant='24 25 23 26 27', mlp_layer_to_quant='24 25 23 26 27', model_id='/mnt/bn/life-mllm/users/cxr/quantization/models/meta-llama/Llama-3.1-8B', cuda_id=6)
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.09s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.10s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.06s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.36it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.16it/s]
LlamaForCausalLM(
  (model): LlamaModel(
    (embed_tokens): Embedding(128256, 4096)
    (layers): ModuleList(
      (0-31): 32 x LlamaDecoderLayer(
        (self_attn): LlamaAttention(
          (q_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (k_proj): Linear(in_features=4096, out_features=1024, bias=False)
          (v_proj): Linear(in_features=4096, out_features=1024, bias=False)
          (o_proj): Linear(in_features=4096, out_features=4096, bias=False)
        )
        (mlp): LlamaMLP(
          (gate_proj): Linear(in_features=4096, out_features=14336, bias=False)
          (up_proj): Linear(in_features=4096, out_features=14336, bias=False)
          (down_proj): Linear(in_features=14336, out_features=4096, bias=False)
          (act_fn): SiLUActivation()
        )
        (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
        (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
      )
    )
    (norm): LlamaRMSNorm((4096,), eps=1e-05)
    (rotary_emb): LlamaRotaryEmbedding()
  )
  (lm_head): Linear(in_features=4096, out_features=128256, bias=False)
)
Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.
Once upon a time, there was a girl who was obsessed with the idea of having a baby. She was married to a wonderful man, but she just couldn’t shake the feeling that she needed to be a mother.
So, she talked to her doctor about it. Her doctor, being a very smart and compassionate person, listened to her concerns and explained that having a baby was a big decision, and one that should be made after careful consideration. The girl took this advice to heart, and decided to take some time
quanting ... 
layers.0.self_attn 4 bit quant 
layers.0.mlp 4 bit quant 
layers.1.self_attn 4 bit quant 
layers.1.mlp 4 bit quant 
layers.2.self_attn 4 bit quant 
layers.2.mlp 4 bit quant 
layers.3.self_attn 4 bit quant 
layers.3.mlp 4 bit quant 
layers.4.self_attn 4 bit quant 
layers.4.mlp 4 bit quant 
layers.5.self_attn 4 bit quant 
layers.5.mlp 4 bit quant 
layers.6.self_attn 4 bit quant 
layers.6.mlp 4 bit quant 
layers.7.self_attn 4 bit quant 
layers.7.mlp 4 bit quant 
layers.8.self_attn 4 bit quant 
layers.8.mlp 4 bit quant 
layers.9.self_attn 4 bit quant 
layers.9.mlp 4 bit quant 
layers.10.self_attn 4 bit quant 
layers.10.mlp 4 bit quant 
layers.11.self_attn 4 bit quant 
layers.11.mlp 4 bit quant 
layers.12.self_attn 4 bit quant 
layers.12.mlp 4 bit quant 
layers.13.self_attn 4 bit quant 
layers.13.mlp 4 bit quant 
layers.14.self_attn 4 bit quant 
layers.14.mlp 4 bit quant 
layers.15.self_attn 4 bit quant 
layers.15.mlp 4 bit quant 
layers.16.self_attn 4 bit quant 
layers.16.mlp 4 bit quant 
layers.17.self_attn 4 bit quant 
layers.17.mlp 4 bit quant 
layers.18.self_attn 4 bit quant 
layers.18.mlp 4 bit quant 
layers.19.self_attn 4 bit quant 
layers.19.mlp 4 bit quant 
layers.20.self_attn 4 bit quant 
layers.20.mlp 4 bit quant 
layers.21.self_attn 4 bit quant 
layers.21.mlp 4 bit quant 
layers.22.self_attn 4 bit quant 
layers.22.mlp 4 bit quant 
layers.23.self_attn 2 bit quant 
layers.23.mlp 2 bit quant 
layers.24.self_attn 2 bit quant 
layers.24.mlp 2 bit quant 
layers.25.self_attn 2 bit quant 
layers.25.mlp 2 bit quant 
layers.26.self_attn 2 bit quant 
layers.26.mlp 2 bit quant 
layers.27.self_attn 2 bit quant 
layers.27.mlp 2 bit quant 
layers.28.self_attn 4 bit quant 
layers.28.mlp 4 bit quant 
layers.29.self_attn 4 bit quant 
layers.29.mlp 4 bit quant 
layers.30.self_attn 4 bit quant 
layers.30.mlp 4 bit quant 
layers.31.self_attn 4 bit quant 
layers.31.mlp 4 bit quant 
quanted
LlamaForCausalLM(
  (model): LlamaModel(
    (embed_tokens): Embedding(128256, 4096)
    (layers): ModuleList(
      (0-31): 32 x LlamaDecoderLayer(
        (self_attn): LlamaAttention(
          (q_proj): W8A16Linear(4096, 4096, bias=False, weight_quant=per_channel)
          (k_proj): W8A16Linear(4096, 1024, bias=False, weight_quant=per_channel)
          (v_proj): W8A16Linear(4096, 1024, bias=False, weight_quant=per_channel)
          (o_proj): W8A16Linear(4096, 4096, bias=False, weight_quant=per_channel)
        )
        (mlp): LlamaMLP(
          (gate_proj): W8A16Linear(4096, 14336, bias=False, weight_quant=per_channel)
          (up_proj): W8A16Linear(4096, 14336, bias=False, weight_quant=per_channel)
          (down_proj): W8A16Linear(14336, 4096, bias=False, weight_quant=per_channel)
          (act_fn): SiLUActivation()
        )
        (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
        (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
      )
    )
    (norm): LlamaRMSNorm((4096,), eps=1e-05)
    (rotary_emb): LlamaRotaryEmbedding()
  )
  (lm_head): Linear(in_features=4096, out_features=128256, bias=False)
)
Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.
Once upon a time there was a young woman called Emily who had been living in the dark clouds of depression since the age of eighteen years when she was diagnosed with having the disease called as 'infective mononitisque' and that had caused her severe pain in abdomen which was treated initially with antibiotics and sulthforteclaveinbriollisbiralensbub.

==================================================
开始评估：任务=winogrande | 少样本数=5 | 模型=Llama-3.1-8B-quantization-fg
输出路径：results2/Llama-3.1-8B-quantization-fg/baseline_BI/winogrande.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-09:16:52:44 INFO     [__main__:440] Selected Tasks: ['winogrande']
2025-12-09:16:52:44 INFO     [__main__:440] Selected Tasks: ['winogrande']
2025-12-09:16:52:44 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:16:52:44 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-09:16:52:44 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:16:52:44 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-09:16:52:46 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:16:52:46 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:16:52:46 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.27s/it]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:04,  1.40s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.18s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.30s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.14s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.24it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.06it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.25s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:04<00:00,  1.13it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:04<00:00,  1.04s/it]
2025-12-09:16:53:06 WARNING  [evaluator:309] Overwriting default num_fewshot of winogrande from None to 5
2025-12-09:16:53:06 INFO     [api.task:434] Building contexts for winogrande on rank 1...

  0%|          | 0/633 [00:00<?, ?it/s]
100%|██████████| 633/633 [00:00<00:00, 26561.90it/s]
2025-12-09:16:53:08 WARNING  [evaluator:309] Overwriting default num_fewshot of winogrande from None to 5
2025-12-09:16:53:08 INFO     [api.task:434] Building contexts for winogrande on rank 0...

  0%|          | 0/634 [00:00<?, ?it/s]
100%|██████████| 634/634 [00:00<00:00, 29765.82it/s]
n136-128-154:2221863:2221863 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2221863:2221863 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2221863:2221863 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:2221863:2221863 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2221863:2221863 [0] NCCL INFO Comm config Blocking set to 1
n136-128-154:2221864:2221864 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:2221864:2221864 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2221864:2221864 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2221864:2221864 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2221864:2221864 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:2221863:2222046 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2221863:2222046 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2221864:2222047 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2221864:2222047 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2221863:2222046 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2221863:2222046 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2221863:2222046 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2221864:2222047 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2221864:2222047 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2221864:2222047 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2221864:2222047 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2221864:2222047 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2221864:2222047 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:2221864:2222047 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2221864:2222047 [1] NCCL INFO Using network IB
n136-128-154:2221863:2222046 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2221863:2222046 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2221863:2222046 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:2221863:2222046 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2221863:2222046 [0] NCCL INFO Using network IB
n136-128-154:2221864:2222047 [1] NCCL INFO ncclCommInitRankConfig comm 0x14fea0c0 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0x7c29df96cd79f60b - Init START
n136-128-154:2221863:2222046 [0] NCCL INFO ncclCommInitRankConfig comm 0x166dd430 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0x7c29df96cd79f60b - Init START
n136-128-154:2221863:2222046 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2221864:2222047 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2221864:2222047 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2221864:2222047 [1] NCCL INFO Retrieving state for IB
n136-128-154:2221864:2222047 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:2221864:2222047 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2221864:2222047 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2221864:2222047 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2221864:2222047 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2221863:2222046 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2221863:2222046 [0] NCCL INFO Retrieving state for IB
n136-128-154:2221863:2222046 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:2221863:2222046 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2221863:2222046 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2221863:2222046 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2221863:2222046 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2221863:2222046 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2221863:2222046 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2221864:2222047 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2221864:2222047 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2221863:2222046 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2221863:2222046 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2221863:2222046 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2221863:2222046 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2221863:2222046 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2221863:2222046 [0] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2221863:2222046 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2221863:2222046 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2221863:2222046 [0] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2221863:2222046 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2221864:2222047 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2221863:2222046 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2221864:2222047 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2221863:2222046 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2221864:2222047 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2221863:2222046 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2221864:2222047 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2221863:2222046 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2221864:2222047 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2221863:2222046 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2221864:2222047 [1] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2221863:2222046 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2221864:2222047 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2221863:2222046 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2221864:2222047 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2221863:2222046 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2221864:2222047 [1] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2221863:2222046 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2221864:2222047 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2221863:2222046 [0] NCCL INFO ==========================================
n136-128-154:2221864:2222047 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2221864:2222047 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2221864:2222047 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2221863:2222046 [0] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2221864:2222047 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2221864:2222047 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2221863:2222046 [0] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2221864:2222047 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2221864:2222047 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2221864:2222047 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2221864:2222047 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2221863:2222046 [0] NCCL INFO Setting affinity for GPU 6 to 33-62,97-126
n136-128-154:2221864:2222047 [1] NCCL INFO ==========================================
n136-128-154:2221864:2222047 [1] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2221864:2222047 [1] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2221864:2222047 [1] NCCL INFO Setting affinity for GPU 7 to 33-62,97-126
n136-128-154:2221863:2222046 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2221863:2222046 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2221863:2222046 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2221863:2222046 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2221863:2222046 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2221863:2222046 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2221863:2222046 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2221863:2222046 [0] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2221863:2222046 [0] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2221863:2222046 [0] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2221863:2222046 [0] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2221863:2222046 [0] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2221863:2222046 [0] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2221863:2222046 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2221863:2222046 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2221863:2222046 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2221863:2222046 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2221863:2222046 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2221863:2222046 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2221863:2222046 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2221863:2222046 [0] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2221863:2222046 [0] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2221863:2222046 [0] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2221863:2222046 [0] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2221863:2222046 [0] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2221863:2222046 [0] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2221864:2222047 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2221864:2222047 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2221864:2222047 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2221864:2222047 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2221864:2222047 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2221864:2222047 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2221864:2222047 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2221864:2222047 [1] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2221864:2222047 [1] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2221864:2222047 [1] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2221864:2222047 [1] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2221864:2222047 [1] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2221864:2222047 [1] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2221864:2222047 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2221864:2222047 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2221864:2222047 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2221864:2222047 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2221864:2222047 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2221864:2222047 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2221864:2222047 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2221864:2222047 [1] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2221864:2222047 [1] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2221864:2222047 [1] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2221864:2222047 [1] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2221864:2222047 [1] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2221864:2222047 [1] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2221863:2222046 [0] NCCL INFO comm 0x166dd430 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:2221864:2222047 [1] NCCL INFO comm 0x14fea0c0 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:2221863:2222046 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:2221864:2222047 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:2221863:2222046 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:2221864:2222047 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:2221863:2222046 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:2221864:2222047 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:2221863:2222046 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:2221864:2222047 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:2221863:2222046 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:2221864:2222047 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:2221863:2222046 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:2221864:2222047 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:2221863:2222046 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:2221864:2222047 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:2221863:2222046 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:2221864:2222047 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:2221863:2222046 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:2221864:2222047 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:2221863:2222046 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:2221864:2222047 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:2221863:2222046 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:2221864:2222047 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:2221863:2222046 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:2221864:2222047 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:2221863:2222046 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:2221864:2222047 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:2221863:2222046 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:2221864:2222047 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:2221863:2222046 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:2221864:2222047 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:2221863:2222046 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:2221864:2222047 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:2221863:2222046 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:2221864:2222047 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:2221863:2222046 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:2221864:2222047 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:2221863:2222046 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:2221864:2222047 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:2221863:2222046 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:2221864:2222047 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:2221863:2222046 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:2221864:2222047 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:2221863:2222046 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:2221864:2222047 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:2221863:2222046 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:2221864:2222047 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:2221863:2222046 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:2221864:2222047 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:2221863:2222046 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:2221864:2222047 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:2221863:2222046 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:2221864:2222047 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:2221863:2222046 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:2221864:2222047 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:2221863:2222046 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:2221864:2222047 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:2221863:2222046 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:2221864:2222047 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:2221863:2222046 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:2221864:2222047 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:2221863:2222046 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:2221864:2222047 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:2221863:2222046 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:2221864:2222047 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:2221863:2222046 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:2221864:2222047 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:2221863:2222046 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:2221864:2222047 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:2221863:2222046 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:2221864:2222047 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:2221863:2222046 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:2221864:2222047 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:2221863:2222046 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:2221864:2222047 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:2221863:2222046 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:2221864:2222047 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:2221863:2222046 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:2221864:2222047 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:2221863:2222046 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:2221864:2222047 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:2221863:2222046 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:2221864:2222047 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:2221863:2222046 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:2221864:2222047 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:2221863:2222046 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:2221864:2222047 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:2221863:2222046 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:2221864:2222047 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:2221863:2222046 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:2221864:2222047 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:2221863:2222046 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:2221864:2222047 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:2221863:2222046 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:2221864:2222047 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:2221863:2222046 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:2221864:2222047 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:2221864:2222047 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:2221863:2222046 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:2221864:2222047 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2221863:2222046 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:2221863:2222046 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:2221863:2222046 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:2221863:2222046 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:2221863:2222046 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:2221863:2222046 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:2221863:2222046 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:2221863:2222046 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:2221863:2222046 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:2221863:2222046 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:2221863:2222046 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:2221863:2222046 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:2221863:2222046 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:2221863:2222046 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:2221863:2222046 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:2221863:2222046 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:2221863:2222046 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:2221863:2222046 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:2221863:2222046 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:2221863:2222046 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:2221863:2222046 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:2221863:2222046 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:2221863:2222046 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:2221863:2222046 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:2221863:2222046 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2221864:2222047 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2221864:2222060 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 46
n136-128-154:2221864:2222059 [1] NCCL INFO [Proxy Service] Device 1 CPU core 109
n136-128-154:2221863:2222046 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2221863:2222046 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:2221863:2222061 [0] NCCL INFO [Proxy Service] Device 0 CPU core 34
n136-128-154:2221863:2222062 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 36
n136-128-154:2221864:2222047 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2221864:2222047 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2221863:2222046 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2221863:2222046 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2221863:2222046 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:2221863:2222046 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2221863:2222046 [0] NCCL INFO ncclCommInitRankConfig comm 0x166dd430 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0x7c29df96cd79f60b - Init COMPLETE
n136-128-154:2221863:2222046 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.69 (kernels 0.12, alloc 0.35, bootstrap 0.00, allgathers 0.00, topo 0.06, graphs 0.00, connections 0.02, rest 0.13)
n136-128-154:2221864:2222047 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2221864:2222047 [1] NCCL INFO ncclCommInitRankConfig comm 0x14fea0c0 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0x7c29df96cd79f60b - Init COMPLETE
n136-128-154:2221864:2222047 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.68 (kernels 0.12, alloc 0.34, bootstrap 0.01, allgathers 0.00, topo 0.06, graphs 0.00, connections 0.03, rest 0.13)
n136-128-154:2221863:2222063 [0] NCCL INFO Channel 00/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2221863:2222063 [0] NCCL INFO Channel 01/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2221863:2222063 [0] NCCL INFO Channel 02/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2221863:2222063 [0] NCCL INFO Channel 03/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2221863:2222063 [0] NCCL INFO Channel 04/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2221863:2222063 [0] NCCL INFO Channel 05/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2221864:2222064 [1] NCCL INFO Channel 00/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221863:2222063 [0] NCCL INFO Channel 06/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2221864:2222064 [1] NCCL INFO Channel 01/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221863:2222063 [0] NCCL INFO Channel 07/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2221864:2222064 [1] NCCL INFO Channel 02/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221863:2222063 [0] NCCL INFO Channel 08/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2221864:2222064 [1] NCCL INFO Channel 03/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221863:2222063 [0] NCCL INFO Channel 09/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2221864:2222064 [1] NCCL INFO Channel 04/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221863:2222063 [0] NCCL INFO Channel 10/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2221864:2222064 [1] NCCL INFO Channel 05/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221863:2222063 [0] NCCL INFO Channel 11/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2221864:2222064 [1] NCCL INFO Channel 06/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221863:2222063 [0] NCCL INFO Channel 12/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2221864:2222064 [1] NCCL INFO Channel 07/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221863:2222063 [0] NCCL INFO Channel 13/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2221864:2222064 [1] NCCL INFO Channel 08/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221863:2222063 [0] NCCL INFO Channel 14/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2221864:2222064 [1] NCCL INFO Channel 09/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221863:2222063 [0] NCCL INFO Channel 15/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2221864:2222064 [1] NCCL INFO Channel 10/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221863:2222063 [0] NCCL INFO Channel 16/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2221864:2222064 [1] NCCL INFO Channel 11/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221863:2222063 [0] NCCL INFO Channel 17/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2221864:2222064 [1] NCCL INFO Channel 12/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221863:2222063 [0] NCCL INFO Channel 18/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2221864:2222064 [1] NCCL INFO Channel 13/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221863:2222063 [0] NCCL INFO Channel 19/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2221864:2222064 [1] NCCL INFO Channel 14/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221863:2222063 [0] NCCL INFO Channel 20/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2221864:2222064 [1] NCCL INFO Channel 15/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221863:2222063 [0] NCCL INFO Channel 21/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2221864:2222064 [1] NCCL INFO Channel 16/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221863:2222063 [0] NCCL INFO Channel 22/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2221864:2222064 [1] NCCL INFO Channel 17/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221863:2222063 [0] NCCL INFO Channel 23/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2221864:2222064 [1] NCCL INFO Channel 18/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221864:2222064 [1] NCCL INFO Channel 19/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221864:2222064 [1] NCCL INFO Channel 20/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221864:2222064 [1] NCCL INFO Channel 21/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221864:2222064 [1] NCCL INFO Channel 22/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221864:2222064 [1] NCCL INFO Channel 23/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221863:2222063 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:2221864:2222064 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-09:16:53:09 INFO     [evaluator:559] Running loglikelihood requests
2025-12-09:16:53:09 INFO     [evaluator:559] Running loglikelihood requests

Running loglikelihood requests:   0%|          | 0/1268 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 64
Determined largest batch size: 64

Running loglikelihood requests:   0%|          | 1/1268 [00:05<2:03:25,  5.85s/it]
Running loglikelihood requests:   5%|▌         | 65/1268 [00:06<01:34, 12.74it/s] 
Running loglikelihood requests:  10%|█         | 129/1268 [00:07<00:47, 24.13it/s]
Running loglikelihood requests:  15%|█▌        | 193/1268 [00:08<00:31, 34.04it/s]
Running loglikelihood requests:  20%|██        | 257/1268 [00:09<00:23, 42.22it/s]
Running loglikelihood requests:  25%|██▌       | 321/1268 [00:10<00:19, 49.09it/s]
Running loglikelihood requests:  30%|███       | 385/1268 [00:11<00:16, 54.41it/s]
Running loglikelihood requests:  35%|███▌      | 449/1268 [00:12<00:14, 58.43it/s]
Running loglikelihood requests:  40%|████      | 513/1268 [00:13<00:12, 61.37it/s]
Running loglikelihood requests:  46%|████▌     | 577/1268 [00:14<00:10, 63.66it/s]
Running loglikelihood requests:  51%|█████     | 641/1268 [00:15<00:09, 65.81it/s]
Running loglikelihood requests:  56%|█████▌    | 705/1268 [00:16<00:08, 67.40it/s]
Running loglikelihood requests:  61%|██████    | 769/1268 [00:17<00:07, 68.67it/s]
Running loglikelihood requests:  66%|██████▌   | 833/1268 [00:17<00:06, 69.58it/s]
Running loglikelihood requests:  71%|███████   | 897/1268 [00:18<00:05, 70.50it/s]
Running loglikelihood requests:  76%|███████▌  | 961/1268 [00:19<00:04, 71.19it/s]
Running loglikelihood requests:  81%|████████  | 1025/1268 [00:20<00:03, 71.63it/s]
Running loglikelihood requests:  86%|████████▌ | 1089/1268 [00:21<00:02, 72.50it/s]
Running loglikelihood requests:  91%|█████████ | 1153/1268 [00:22<00:01, 72.92it/s]
Running loglikelihood requests:  96%|█████████▌| 1217/1268 [00:22<00:00, 78.15it/s]
Running loglikelihood requests: 100%|██████████| 1268/1268 [00:22<00:00, 55.15it/s]
n136-128-154:2221864:2222089 [1] NCCL INFO Channel 00/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221864:2222089 [1] NCCL INFO Channel 01/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221864:2222089 [1] NCCL INFO Channel 02/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221864:2222089 [1] NCCL INFO Channel 03/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221864:2222089 [1] NCCL INFO Channel 04/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221864:2222089 [1] NCCL INFO Channel 05/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221864:2222089 [1] NCCL INFO Channel 06/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221864:2222089 [1] NCCL INFO Channel 07/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221864:2222089 [1] NCCL INFO Channel 08/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221864:2222089 [1] NCCL INFO Channel 09/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221864:2222089 [1] NCCL INFO Channel 10/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221864:2222089 [1] NCCL INFO Channel 11/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221864:2222089 [1] NCCL INFO Channel 12/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221864:2222089 [1] NCCL INFO Channel 13/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221864:2222089 [1] NCCL INFO Channel 14/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221864:2222089 [1] NCCL INFO Channel 15/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221864:2222089 [1] NCCL INFO Channel 16/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221864:2222089 [1] NCCL INFO Channel 17/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221864:2222089 [1] NCCL INFO Channel 18/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221864:2222089 [1] NCCL INFO Channel 19/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221864:2222089 [1] NCCL INFO Channel 20/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221864:2222089 [1] NCCL INFO Channel 21/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221864:2222089 [1] NCCL INFO Channel 22/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221864:2222089 [1] NCCL INFO Channel 23/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221864:2222089 [1] NCCL INFO Channel 24/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221864:2222089 [1] NCCL INFO Channel 25/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221864:2222089 [1] NCCL INFO Channel 26/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221864:2222089 [1] NCCL INFO Channel 27/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221864:2222089 [1] NCCL INFO Channel 28/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221864:2222089 [1] NCCL INFO Channel 29/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221864:2222089 [1] NCCL INFO Channel 30/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2221864:2222089 [1] NCCL INFO Channel 31/1 : 1[7] -> 0[6] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:2221864:2222095 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2221864:2222095 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2221864:2222095 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2221864:2222095 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2221864:2222095 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2221864:2222095 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2221864:2222059 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-09:16:53:42 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: auto (64)
|  Tasks   |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|----------|------:|------|-----:|------|---|-----:|---|-----:|
|winogrande|      1|none  |     5|acc   |↑  |0.6803|±  |0.0131|

[rank0]:[W1209 16:53:42.128630331 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:2221863:2222165 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2221863:2222165 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2221863:2222165 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2221863:2222061 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2221863:2222165 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2221863:2222165 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2221863:2222165 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2221864:2222059 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2221864:2222095 [1] NCCL INFO comm 0x14fea0c0 rank 1 nranks 2 cudaDev 1 busId c9000 - Abort COMPLETE
n136-128-154:2221863:2222165 [0] NCCL INFO comm 0x166dd430 rank 0 nranks 2 cudaDev 0 busId c5000 - Abort COMPLETE
任务 winogrande 评估完成！

==================================================
开始评估：任务=boolq | 少样本数=0 | 模型=Llama-3.1-8B-quantization-fg
输出路径：results2/Llama-3.1-8B-quantization-fg/baseline_BI/boolq.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-09:16:54:31 INFO     [__main__:440] Selected Tasks: ['boolq']
2025-12-09:16:54:31 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:16:54:31 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-09:16:54:31 INFO     [__main__:440] Selected Tasks: ['boolq']
2025-12-09:16:54:31 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:16:54:31 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-09:16:54:32 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:16:54:32 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:16:54:32 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.05s/it]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.03s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.05s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.06s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.06s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.08s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.35it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.17it/s]
2025-12-09:16:54:36 WARNING  [api.task:844] [Task: boolq] metric acc is defined, but aggregation is not. using default aggregation=mean
2025-12-09:16:54:36 WARNING  [api.task:856] [Task: boolq] metric acc is defined, but higher_is_better is not. using default higher_is_better=True

Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.31it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.14it/s]
2025-12-09:16:54:36 WARNING  [api.task:844] [Task: boolq] metric acc is defined, but aggregation is not. using default aggregation=mean
2025-12-09:16:54:36 WARNING  [api.task:856] [Task: boolq] metric acc is defined, but higher_is_better is not. using default higher_is_better=True
2025-12-09:16:54:50 WARNING  [evaluator:309] Overwriting default num_fewshot of boolq from None to 0
2025-12-09:16:54:50 INFO     [api.task:434] Building contexts for boolq on rank 0...

  0%|          | 0/1635 [00:00<?, ?it/s]
 17%|█▋        | 282/1635 [00:00<00:00, 2813.96it/s]
 34%|███▍      | 564/1635 [00:00<00:00, 2762.66it/s]
 52%|█████▏    | 847/1635 [00:00<00:00, 2791.10it/s]
 69%|██████▉   | 1129/1635 [00:00<00:00, 2802.18it/s]
 86%|████████▌ | 1410/1635 [00:00<00:00, 2786.93it/s]
100%|██████████| 1635/1635 [00:00<00:00, 2792.08it/s]
n136-128-154:2222211:2222211 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2222211:2222211 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2222211:2222211 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:2222211:2222211 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2222211:2222211 [0] NCCL INFO Comm config Blocking set to 1
n136-128-154:2222211:2222378 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2222211:2222378 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2222211:2222378 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2222211:2222378 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2222211:2222378 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2222211:2222378 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2222211:2222378 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2222211:2222378 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:2222211:2222378 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2222211:2222378 [0] NCCL INFO Using network IB
n136-128-154:2222211:2222378 [0] NCCL INFO ncclCommInitRankConfig comm 0x11e067c0 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0x2b3729dc5e596a90 - Init START
2025-12-09:16:54:52 WARNING  [evaluator:309] Overwriting default num_fewshot of boolq from None to 0
2025-12-09:16:54:52 INFO     [api.task:434] Building contexts for boolq on rank 1...

  0%|          | 0/1635 [00:00<?, ?it/s]
 17%|█▋        | 271/1635 [00:00<00:00, 2704.05it/s]
 33%|███▎      | 542/1635 [00:00<00:00, 2703.49it/s]
 50%|████▉     | 813/1635 [00:00<00:00, 2686.54it/s]
 66%|██████▌   | 1082/1635 [00:00<00:00, 2672.33it/s]
 83%|████████▎ | 1356/1635 [00:00<00:00, 2693.47it/s]
100%|█████████▉| 1634/1635 [00:00<00:00, 2719.23it/s]
100%|██████████| 1635/1635 [00:00<00:00, 2702.97it/s]
n136-128-154:2222212:2222212 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:2222212:2222212 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2222212:2222212 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2222212:2222212 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2222212:2222212 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:2222212:2222392 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2222212:2222392 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2222212:2222392 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2222212:2222392 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2222212:2222392 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2222212:2222392 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2222212:2222392 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2222212:2222392 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:2222212:2222392 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2222212:2222392 [1] NCCL INFO Using network IB
n136-128-154:2222212:2222392 [1] NCCL INFO ncclCommInitRankConfig comm 0x12f06230 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0x2b3729dc5e596a90 - Init START
n136-128-154:2222212:2222392 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2222211:2222378 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2222212:2222392 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2222212:2222392 [1] NCCL INFO Retrieving state for IB
n136-128-154:2222212:2222392 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:2222212:2222392 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2222211:2222378 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2222211:2222378 [0] NCCL INFO Retrieving state for IB
n136-128-154:2222211:2222378 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:2222212:2222392 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2222211:2222378 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2222212:2222392 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2222211:2222378 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2222212:2222392 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2222211:2222378 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2222211:2222378 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2222212:2222392 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2222212:2222392 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2222211:2222378 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2222211:2222378 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2222212:2222392 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2222212:2222392 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2222212:2222392 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2222212:2222392 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2222212:2222392 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2222212:2222392 [1] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2222212:2222392 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2222212:2222392 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2222212:2222392 [1] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2222212:2222392 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2222212:2222392 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2222212:2222392 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2222212:2222392 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2222212:2222392 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2222212:2222392 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2222212:2222392 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2222212:2222392 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2222212:2222392 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2222212:2222392 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2222212:2222392 [1] NCCL INFO ==========================================
n136-128-154:2222212:2222392 [1] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2222211:2222378 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2222212:2222392 [1] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2222211:2222378 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2222211:2222378 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2222211:2222378 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2222211:2222378 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2222212:2222392 [1] NCCL INFO Setting affinity for GPU 7 to 33-62,97-126
n136-128-154:2222211:2222378 [0] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2222211:2222378 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2222211:2222378 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2222211:2222378 [0] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2222211:2222378 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2222211:2222378 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2222211:2222378 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2222211:2222378 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2222211:2222378 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2222211:2222378 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2222211:2222378 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2222211:2222378 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2222211:2222378 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2222211:2222378 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2222211:2222378 [0] NCCL INFO ==========================================
n136-128-154:2222211:2222378 [0] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2222211:2222378 [0] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2222211:2222378 [0] NCCL INFO Setting affinity for GPU 6 to 33-62,97-126
n136-128-154:2222212:2222392 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2222212:2222392 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222212:2222392 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222212:2222392 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222212:2222392 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222212:2222392 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222212:2222392 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222212:2222392 [1] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222212:2222392 [1] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222212:2222392 [1] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222212:2222392 [1] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222212:2222392 [1] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222212:2222392 [1] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222212:2222392 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2222212:2222392 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222212:2222392 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222212:2222392 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222212:2222392 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222212:2222392 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222212:2222392 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222212:2222392 [1] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2222212:2222392 [1] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2222212:2222392 [1] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2222211:2222378 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2222212:2222392 [1] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2222212:2222392 [1] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2222211:2222378 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222212:2222392 [1] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2222211:2222378 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222211:2222378 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222211:2222378 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222211:2222378 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222211:2222378 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222211:2222378 [0] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222211:2222378 [0] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222211:2222378 [0] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222211:2222378 [0] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222211:2222378 [0] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222211:2222378 [0] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222211:2222378 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2222211:2222378 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222211:2222378 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222211:2222378 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222211:2222378 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222211:2222378 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222211:2222378 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222211:2222378 [0] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2222211:2222378 [0] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2222211:2222378 [0] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2222211:2222378 [0] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2222211:2222378 [0] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2222211:2222378 [0] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2222212:2222392 [1] NCCL INFO comm 0x12f06230 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:2222211:2222378 [0] NCCL INFO comm 0x11e067c0 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:2222212:2222392 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:2222211:2222378 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:2222212:2222392 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:2222211:2222378 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:2222212:2222392 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:2222211:2222378 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:2222212:2222392 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:2222211:2222378 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:2222212:2222392 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:2222211:2222378 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:2222212:2222392 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:2222211:2222378 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:2222212:2222392 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:2222211:2222378 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:2222212:2222392 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:2222211:2222378 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:2222212:2222392 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:2222211:2222378 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:2222212:2222392 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:2222211:2222378 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:2222212:2222392 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:2222211:2222378 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:2222212:2222392 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:2222211:2222378 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:2222212:2222392 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:2222211:2222378 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:2222212:2222392 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:2222211:2222378 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:2222212:2222392 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:2222211:2222378 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:2222212:2222392 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:2222211:2222378 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:2222212:2222392 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:2222211:2222378 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:2222212:2222392 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:2222211:2222378 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:2222212:2222392 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:2222211:2222378 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:2222212:2222392 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:2222211:2222378 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:2222212:2222392 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:2222211:2222378 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:2222212:2222392 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:2222211:2222378 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:2222212:2222392 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:2222211:2222378 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:2222212:2222392 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:2222211:2222378 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:2222211:2222378 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:2222211:2222378 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:2222212:2222392 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:2222211:2222378 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:2222212:2222392 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:2222211:2222378 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:2222212:2222392 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:2222211:2222378 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:2222212:2222392 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:2222211:2222378 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:2222212:2222392 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:2222211:2222378 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:2222212:2222392 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:2222211:2222378 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:2222212:2222392 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:2222211:2222378 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:2222212:2222392 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:2222211:2222378 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:2222212:2222392 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:2222211:2222378 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:2222212:2222392 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:2222211:2222378 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:2222212:2222392 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:2222211:2222378 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:2222212:2222392 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:2222211:2222378 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:2222212:2222392 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:2222211:2222378 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:2222212:2222392 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:2222211:2222378 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:2222212:2222392 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:2222211:2222378 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:2222212:2222392 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:2222211:2222378 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:2222212:2222392 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:2222211:2222378 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:2222212:2222392 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:2222211:2222378 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:2222212:2222392 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:2222211:2222378 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:2222212:2222392 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:2222211:2222378 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:2222212:2222392 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:2222211:2222378 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:2222212:2222392 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:2222211:2222378 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:2222212:2222392 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:2222212:2222392 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:2222211:2222378 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:2222212:2222392 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:2222211:2222378 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:2222211:2222378 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:2222212:2222392 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2222211:2222378 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:2222211:2222378 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:2222211:2222378 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:2222211:2222378 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:2222211:2222378 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:2222211:2222378 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:2222211:2222378 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:2222211:2222378 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:2222211:2222378 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:2222211:2222378 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:2222211:2222378 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:2222211:2222378 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:2222211:2222378 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:2222211:2222378 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:2222211:2222378 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:2222211:2222378 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:2222211:2222378 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:2222211:2222378 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:2222211:2222378 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:2222211:2222378 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:2222211:2222378 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:2222211:2222378 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:2222211:2222378 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2222211:2222378 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2222211:2222378 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:2222211:2222400 [0] NCCL INFO [Proxy Service] Device 0 CPU core 110
n136-128-154:2222211:2222401 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 47
n136-128-154:2222212:2222392 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2222212:2222402 [1] NCCL INFO [Proxy Service] Device 1 CPU core 34
n136-128-154:2222212:2222403 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 100
n136-128-154:2222212:2222392 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2222212:2222392 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2222211:2222378 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2222211:2222378 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2222211:2222378 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:2222211:2222378 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2222211:2222378 [0] NCCL INFO ncclCommInitRankConfig comm 0x11e067c0 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0x2b3729dc5e596a90 - Init COMPLETE
n136-128-154:2222211:2222378 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 2.58 (kernels 0.12, alloc 0.10, bootstrap 1.99, allgathers 0.00, topo 0.06, graphs 0.00, connections 0.17, rest 0.14)
n136-128-154:2222212:2222392 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2222212:2222392 [1] NCCL INFO ncclCommInitRankConfig comm 0x12f06230 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0x2b3729dc5e596a90 - Init COMPLETE
n136-128-154:2222212:2222392 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.66 (kernels 0.13, alloc 0.15, bootstrap 0.00, allgathers 0.00, topo 0.06, graphs 0.00, connections 0.16, rest 0.15)
n136-128-154:2222211:2222404 [0] NCCL INFO Channel 00/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222211:2222404 [0] NCCL INFO Channel 01/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222211:2222404 [0] NCCL INFO Channel 02/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222212:2222405 [1] NCCL INFO Channel 00/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222211:2222404 [0] NCCL INFO Channel 03/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222212:2222405 [1] NCCL INFO Channel 01/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222211:2222404 [0] NCCL INFO Channel 04/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222212:2222405 [1] NCCL INFO Channel 02/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222211:2222404 [0] NCCL INFO Channel 05/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222212:2222405 [1] NCCL INFO Channel 03/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222211:2222404 [0] NCCL INFO Channel 06/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222212:2222405 [1] NCCL INFO Channel 04/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222211:2222404 [0] NCCL INFO Channel 07/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222212:2222405 [1] NCCL INFO Channel 05/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222211:2222404 [0] NCCL INFO Channel 08/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222212:2222405 [1] NCCL INFO Channel 06/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222211:2222404 [0] NCCL INFO Channel 09/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222212:2222405 [1] NCCL INFO Channel 07/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222211:2222404 [0] NCCL INFO Channel 10/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222212:2222405 [1] NCCL INFO Channel 08/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222211:2222404 [0] NCCL INFO Channel 11/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222212:2222405 [1] NCCL INFO Channel 09/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222211:2222404 [0] NCCL INFO Channel 12/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222212:2222405 [1] NCCL INFO Channel 10/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222211:2222404 [0] NCCL INFO Channel 13/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222212:2222405 [1] NCCL INFO Channel 11/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222211:2222404 [0] NCCL INFO Channel 14/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222212:2222405 [1] NCCL INFO Channel 12/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222211:2222404 [0] NCCL INFO Channel 15/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222212:2222405 [1] NCCL INFO Channel 13/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222211:2222404 [0] NCCL INFO Channel 16/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222212:2222405 [1] NCCL INFO Channel 14/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222211:2222404 [0] NCCL INFO Channel 17/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222212:2222405 [1] NCCL INFO Channel 15/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222211:2222404 [0] NCCL INFO Channel 18/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222212:2222405 [1] NCCL INFO Channel 16/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222211:2222404 [0] NCCL INFO Channel 19/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222212:2222405 [1] NCCL INFO Channel 17/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222211:2222404 [0] NCCL INFO Channel 20/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222212:2222405 [1] NCCL INFO Channel 18/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222211:2222404 [0] NCCL INFO Channel 21/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222212:2222405 [1] NCCL INFO Channel 19/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222211:2222404 [0] NCCL INFO Channel 22/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222212:2222405 [1] NCCL INFO Channel 20/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222211:2222404 [0] NCCL INFO Channel 23/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222212:2222405 [1] NCCL INFO Channel 21/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222212:2222405 [1] NCCL INFO Channel 22/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222212:2222405 [1] NCCL INFO Channel 23/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222211:2222404 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:2222212:2222405 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-09:16:54:54 INFO     [evaluator:559] Running loglikelihood requests
2025-12-09:16:54:54 INFO     [evaluator:559] Running loglikelihood requests
Passed argument batch_size = auto:1. Detecting largest batch size

Running loglikelihood requests:   0%|          | 0/3270 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 64
Determined largest batch size: 64

Running loglikelihood requests:   0%|          | 1/3270 [00:35<32:01:53, 35.27s/it]
Running loglikelihood requests:   4%|▍         | 129/3270 [00:36<10:43,  4.88it/s] 
Running loglikelihood requests:   8%|▊         | 257/3270 [00:38<04:35, 10.95it/s]
Running loglikelihood requests:  12%|█▏        | 385/3270 [00:39<02:37, 18.36it/s]
Running loglikelihood requests:  16%|█▌        | 513/3270 [00:41<01:41, 27.12it/s]
Running loglikelihood requests:  20%|█▉        | 641/3270 [00:42<01:10, 37.08it/s]
Running loglikelihood requests:  24%|██▎       | 769/3270 [00:43<00:52, 47.93it/s]
Running loglikelihood requests:  27%|██▋       | 897/3270 [00:44<00:40, 59.24it/s]
Running loglikelihood requests:  31%|███▏      | 1025/3270 [00:45<00:31, 70.60it/s]
Running loglikelihood requests:  35%|███▌      | 1153/3270 [00:46<00:25, 81.46it/s]
Running loglikelihood requests:  39%|███▉      | 1281/3270 [00:47<00:21, 91.26it/s]
Running loglikelihood requests:  43%|████▎     | 1409/3270 [00:48<00:18, 100.27it/s]
Running loglikelihood requests:  47%|████▋     | 1537/3270 [00:49<00:15, 108.40it/s]
Running loglikelihood requests:  51%|█████     | 1665/3270 [00:50<00:13, 116.59it/s]
Running loglikelihood requests:  55%|█████▍    | 1793/3270 [00:51<00:11, 123.83it/s]
Running loglikelihood requests:  59%|█████▊    | 1921/3270 [00:52<00:10, 130.81it/s]
Running loglikelihood requests:  63%|██████▎   | 2049/3270 [00:53<00:08, 136.59it/s]
Running loglikelihood requests:  67%|██████▋   | 2177/3270 [00:53<00:07, 142.51it/s]
Running loglikelihood requests:  70%|███████   | 2305/3270 [00:54<00:06, 147.87it/s]
Running loglikelihood requests:  74%|███████▍  | 2433/3270 [00:55<00:05, 153.12it/s]
Running loglikelihood requests:  78%|███████▊  | 2561/3270 [00:56<00:04, 159.01it/s]
Running loglikelihood requests:  82%|████████▏ | 2689/3270 [00:56<00:03, 164.99it/s]
Running loglikelihood requests:  86%|████████▌ | 2817/3270 [00:57<00:02, 172.30it/s]
Running loglikelihood requests:  90%|█████████ | 2945/3270 [00:58<00:01, 180.50it/s]
Running loglikelihood requests:  94%|█████████▍| 3073/3270 [00:58<00:01, 190.58it/s]
Running loglikelihood requests:  98%|█████████▊| 3201/3270 [00:59<00:00, 229.74it/s]
Running loglikelihood requests: 100%|██████████| 3270/3270 [00:59<00:00, 55.39it/s] 
n136-128-154:2222212:2222473 [1] NCCL INFO Channel 00/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222212:2222473 [1] NCCL INFO Channel 01/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222212:2222473 [1] NCCL INFO Channel 02/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222212:2222473 [1] NCCL INFO Channel 03/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222212:2222473 [1] NCCL INFO Channel 04/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222212:2222473 [1] NCCL INFO Channel 05/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222212:2222473 [1] NCCL INFO Channel 06/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222212:2222473 [1] NCCL INFO Channel 07/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222212:2222473 [1] NCCL INFO Channel 08/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222212:2222473 [1] NCCL INFO Channel 09/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222212:2222473 [1] NCCL INFO Channel 10/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222212:2222473 [1] NCCL INFO Channel 11/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222212:2222473 [1] NCCL INFO Channel 12/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222212:2222473 [1] NCCL INFO Channel 13/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222212:2222473 [1] NCCL INFO Channel 14/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222212:2222473 [1] NCCL INFO Channel 15/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222212:2222473 [1] NCCL INFO Channel 16/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222212:2222473 [1] NCCL INFO Channel 17/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222212:2222473 [1] NCCL INFO Channel 18/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222212:2222473 [1] NCCL INFO Channel 19/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222212:2222473 [1] NCCL INFO Channel 20/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222212:2222473 [1] NCCL INFO Channel 21/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222212:2222473 [1] NCCL INFO Channel 22/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222212:2222473 [1] NCCL INFO Channel 23/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222212:2222473 [1] NCCL INFO Channel 24/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222212:2222473 [1] NCCL INFO Channel 25/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222212:2222473 [1] NCCL INFO Channel 26/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222212:2222473 [1] NCCL INFO Channel 27/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222212:2222473 [1] NCCL INFO Channel 28/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222212:2222473 [1] NCCL INFO Channel 29/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222212:2222473 [1] NCCL INFO Channel 30/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222212:2222473 [1] NCCL INFO Channel 31/1 : 1[7] -> 0[6] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:2222212:2222479 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2222212:2222479 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2222212:2222479 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2222212:2222479 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2222212:2222479 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2222212:2222479 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2222212:2222402 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-09:16:56:03 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: auto (64)
|Tasks|Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|-----|------:|------|-----:|------|---|-----:|---|-----:|
|boolq|      2|none  |     0|acc   |↑  |0.6869|±  |0.0081|

[rank0]:[W1209 16:56:04.857565386 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:2222211:2222528 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2222211:2222528 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2222211:2222528 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2222211:2222400 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2222211:2222528 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2222211:2222528 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2222211:2222528 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2222212:2222402 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2222212:2222479 [1] NCCL INFO comm 0x12f06230 rank 1 nranks 2 cudaDev 1 busId c9000 - Abort COMPLETE
n136-128-154:2222211:2222528 [0] NCCL INFO comm 0x11e067c0 rank 0 nranks 2 cudaDev 0 busId c5000 - Abort COMPLETE
任务 boolq 评估完成！

==================================================
开始评估：任务=arc_challenge | 少样本数=25 | 模型=Llama-3.1-8B-quantization-fg
输出路径：results2/Llama-3.1-8B-quantization-fg/baseline_BI/arc_challenge.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-09:16:56:53 INFO     [__main__:440] Selected Tasks: ['arc_challenge']
2025-12-09:16:56:53 INFO     [__main__:440] Selected Tasks: ['arc_challenge']
2025-12-09:16:56:53 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:16:56:53 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-09:16:56:53 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:16:56:53 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-09:16:56:55 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:16:56:55 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:16:56:55 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.04s/it]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.05s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.07s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.08s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.02s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.06s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.40it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.20it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.34it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.16it/s]
2025-12-09:16:57:12 WARNING  [evaluator:309] Overwriting default num_fewshot of arc_challenge from None to 25
2025-12-09:16:57:12 INFO     [api.task:434] Building contexts for arc_challenge on rank 0...

  0%|          | 0/586 [00:00<?, ?it/s]
  1%|          | 4/586 [00:00<00:16, 35.44it/s]2025-12-09:16:57:12 WARNING  [evaluator:309] Overwriting default num_fewshot of arc_challenge from None to 25
2025-12-09:16:57:12 INFO     [api.task:434] Building contexts for arc_challenge on rank 1...

  0%|          | 0/586 [00:00<?, ?it/s]
  1%|▏         | 8/586 [00:00<00:16, 35.48it/s]
  1%|          | 4/586 [00:00<00:16, 35.99it/s]
  2%|▏         | 12/586 [00:00<00:15, 36.21it/s]
  1%|▏         | 8/586 [00:00<00:15, 36.39it/s]
  3%|▎         | 16/586 [00:00<00:15, 35.80it/s]
  2%|▏         | 12/586 [00:00<00:15, 36.48it/s]
  3%|▎         | 20/586 [00:00<00:15, 36.19it/s]
  3%|▎         | 16/586 [00:00<00:15, 36.17it/s]
  4%|▍         | 24/586 [00:00<00:16, 34.67it/s]
  3%|▎         | 20/586 [00:00<00:16, 34.34it/s]
  5%|▍         | 28/586 [00:00<00:16, 34.85it/s]
  4%|▍         | 24/586 [00:00<00:15, 35.14it/s]
  5%|▌         | 32/586 [00:00<00:15, 35.05it/s]
  5%|▍         | 28/586 [00:00<00:15, 35.64it/s]
  6%|▌         | 36/586 [00:01<00:15, 35.44it/s]
  5%|▌         | 32/586 [00:00<00:15, 35.76it/s]
  7%|▋         | 40/586 [00:01<00:15, 35.52it/s]
  6%|▌         | 36/586 [00:01<00:15, 36.04it/s]
  8%|▊         | 44/586 [00:01<00:15, 35.43it/s]
  7%|▋         | 40/586 [00:01<00:15, 36.26it/s]
  8%|▊         | 48/586 [00:01<00:15, 35.71it/s]
  8%|▊         | 44/586 [00:01<00:14, 36.37it/s]
  9%|▉         | 52/586 [00:01<00:15, 34.68it/s]
  8%|▊         | 48/586 [00:01<00:14, 36.14it/s]
 10%|▉         | 56/586 [00:01<00:15, 34.93it/s]
  9%|▉         | 52/586 [00:01<00:14, 36.27it/s]
 10%|█         | 60/586 [00:01<00:14, 35.41it/s]
 10%|▉         | 56/586 [00:01<00:14, 36.39it/s]
 11%|█         | 64/586 [00:01<00:14, 35.60it/s]
 10%|█         | 60/586 [00:01<00:14, 36.54it/s]
 12%|█▏        | 68/586 [00:01<00:14, 35.99it/s]
 11%|█         | 64/586 [00:01<00:14, 36.42it/s]
 12%|█▏        | 72/586 [00:02<00:14, 36.17it/s]
 12%|█▏        | 68/586 [00:01<00:14, 36.53it/s]
 13%|█▎        | 76/586 [00:02<00:14, 36.21it/s]
 12%|█▏        | 72/586 [00:01<00:14, 36.58it/s]
 14%|█▎        | 80/586 [00:02<00:13, 36.31it/s]
 13%|█▎        | 76/586 [00:02<00:13, 36.66it/s]
 14%|█▍        | 84/586 [00:02<00:13, 36.39it/s]
 14%|█▎        | 80/586 [00:02<00:13, 36.66it/s]
 15%|█▌        | 88/586 [00:02<00:13, 35.93it/s]
 14%|█▍        | 84/586 [00:02<00:13, 36.71it/s]
 16%|█▌        | 92/586 [00:02<00:13, 36.08it/s]
 15%|█▌        | 88/586 [00:02<00:13, 36.75it/s]
 16%|█▋        | 96/586 [00:02<00:13, 36.20it/s]
 16%|█▌        | 92/586 [00:02<00:13, 36.55it/s]
 17%|█▋        | 100/586 [00:02<00:13, 36.09it/s]
 16%|█▋        | 96/586 [00:02<00:13, 36.29it/s]
 18%|█▊        | 104/586 [00:02<00:13, 36.11it/s]
 17%|█▋        | 100/586 [00:02<00:13, 36.45it/s]
 18%|█▊        | 108/586 [00:03<00:13, 36.31it/s]
 18%|█▊        | 104/586 [00:02<00:13, 36.53it/s]
 19%|█▉        | 112/586 [00:03<00:13, 36.38it/s]
 18%|█▊        | 108/586 [00:02<00:13, 36.61it/s]
 20%|█▉        | 116/586 [00:03<00:12, 36.27it/s]
 19%|█▉        | 112/586 [00:03<00:12, 36.64it/s]
 20%|██        | 120/586 [00:03<00:12, 36.36it/s]
 20%|█▉        | 116/586 [00:03<00:12, 36.69it/s]
 21%|██        | 124/586 [00:03<00:12, 36.48it/s]
 20%|██        | 120/586 [00:03<00:12, 36.70it/s]
 22%|██▏       | 128/586 [00:03<00:12, 36.32it/s]
 21%|██        | 124/586 [00:03<00:12, 36.21it/s]
 23%|██▎       | 132/586 [00:03<00:12, 36.43it/s]
 22%|██▏       | 128/586 [00:03<00:12, 36.34it/s]
 23%|██▎       | 136/586 [00:03<00:12, 36.61it/s]
 23%|██▎       | 132/586 [00:03<00:12, 36.37it/s]
 24%|██▍       | 140/586 [00:03<00:12, 36.73it/s]
 23%|██▎       | 136/586 [00:03<00:12, 36.48it/s]
 25%|██▍       | 144/586 [00:04<00:12, 36.66it/s]
 24%|██▍       | 140/586 [00:03<00:12, 36.57it/s]
 25%|██▌       | 148/586 [00:04<00:11, 36.82it/s]
 25%|██▍       | 144/586 [00:03<00:12, 36.65it/s]
 26%|██▌       | 152/586 [00:04<00:11, 36.94it/s]
 25%|██▌       | 148/586 [00:04<00:11, 36.70it/s]
 27%|██▋       | 156/586 [00:04<00:11, 37.06it/s]
 26%|██▌       | 152/586 [00:04<00:11, 36.77it/s]
 27%|██▋       | 160/586 [00:04<00:11, 37.11it/s]
 27%|██▋       | 156/586 [00:04<00:11, 36.56it/s]
 28%|██▊       | 164/586 [00:04<00:11, 36.91it/s]
 27%|██▋       | 160/586 [00:04<00:11, 36.58it/s]
 29%|██▊       | 168/586 [00:04<00:11, 36.96it/s]
 28%|██▊       | 164/586 [00:04<00:11, 36.62it/s]
 29%|██▉       | 172/586 [00:04<00:11, 36.96it/s]
 29%|██▊       | 168/586 [00:04<00:11, 35.34it/s]
 30%|███       | 176/586 [00:04<00:11, 37.02it/s]
 29%|██▉       | 172/586 [00:04<00:11, 35.71it/s]
 31%|███       | 180/586 [00:04<00:11, 36.66it/s]
 30%|███       | 176/586 [00:04<00:11, 36.02it/s]
 31%|███▏      | 184/586 [00:05<00:10, 36.62it/s]
 31%|███       | 180/586 [00:04<00:11, 36.27it/s]
 32%|███▏      | 188/586 [00:05<00:10, 36.77it/s]
 31%|███▏      | 184/586 [00:05<00:11, 36.40it/s]
 33%|███▎      | 192/586 [00:05<00:10, 36.88it/s]
 32%|███▏      | 188/586 [00:05<00:10, 36.54it/s]
 33%|███▎      | 196/586 [00:05<00:10, 36.93it/s]
 33%|███▎      | 192/586 [00:05<00:10, 36.65it/s]
 34%|███▍      | 200/586 [00:05<00:10, 36.77it/s]
 33%|███▎      | 196/586 [00:05<00:10, 36.70it/s]
 35%|███▍      | 204/586 [00:05<00:10, 36.84it/s]
 34%|███▍      | 200/586 [00:05<00:10, 36.74it/s]
 35%|███▌      | 208/586 [00:05<00:10, 36.85it/s]
 35%|███▍      | 204/586 [00:05<00:10, 36.77it/s]
 36%|███▌      | 212/586 [00:05<00:10, 36.86it/s]
 35%|███▌      | 208/586 [00:05<00:10, 36.78it/s]
 37%|███▋      | 216/586 [00:05<00:10, 36.87it/s]
 36%|███▌      | 212/586 [00:05<00:10, 36.79it/s]
 38%|███▊      | 220/586 [00:06<00:09, 36.85it/s]
 37%|███▋      | 216/586 [00:05<00:10, 36.55it/s]
 38%|███▊      | 224/586 [00:06<00:09, 36.66it/s]
 38%|███▊      | 220/586 [00:06<00:09, 36.62it/s]
 39%|███▉      | 228/586 [00:06<00:09, 36.71it/s]
 38%|███▊      | 224/586 [00:06<00:09, 36.68it/s]
 40%|███▉      | 232/586 [00:06<00:09, 36.76it/s]
 39%|███▉      | 228/586 [00:06<00:09, 36.72it/s]
 40%|████      | 236/586 [00:06<00:09, 36.76it/s]
 40%|███▉      | 232/586 [00:06<00:09, 36.74it/s]
 41%|████      | 240/586 [00:06<00:09, 36.82it/s]
 40%|████      | 236/586 [00:06<00:09, 36.76it/s]
 42%|████▏     | 244/586 [00:06<00:09, 36.22it/s]
 41%|████      | 240/586 [00:06<00:09, 36.80it/s]
 42%|████▏     | 248/586 [00:06<00:09, 36.40it/s]
 42%|████▏     | 244/586 [00:06<00:09, 36.84it/s]
 43%|████▎     | 252/586 [00:06<00:09, 36.50it/s]
 42%|████▏     | 248/586 [00:06<00:09, 36.90it/s]
 44%|████▎     | 256/586 [00:07<00:09, 36.61it/s]
 43%|████▎     | 252/586 [00:06<00:09, 36.91it/s]
 44%|████▍     | 260/586 [00:07<00:08, 36.68it/s]
 44%|████▎     | 256/586 [00:07<00:08, 36.95it/s]
 45%|████▌     | 264/586 [00:07<00:08, 36.72it/s]
 44%|████▍     | 260/586 [00:07<00:08, 36.93it/s]
 46%|████▌     | 268/586 [00:07<00:08, 36.67it/s]
 45%|████▌     | 264/586 [00:07<00:08, 36.95it/s]
 46%|████▋     | 272/586 [00:07<00:08, 36.77it/s]
 46%|████▌     | 268/586 [00:07<00:08, 36.99it/s]
 47%|████▋     | 276/586 [00:07<00:08, 36.86it/s]
 46%|████▋     | 272/586 [00:07<00:08, 37.00it/s]
 48%|████▊     | 280/586 [00:07<00:08, 36.91it/s]
 47%|████▋     | 276/586 [00:07<00:08, 36.51it/s]
 48%|████▊     | 284/586 [00:07<00:08, 36.89it/s]
 48%|████▊     | 280/586 [00:07<00:08, 36.59it/s]
 49%|████▉     | 288/586 [00:07<00:08, 36.93it/s]
 48%|████▊     | 284/586 [00:07<00:08, 36.66it/s]
 50%|████▉     | 292/586 [00:08<00:08, 36.72it/s]
 49%|████▉     | 288/586 [00:07<00:08, 36.79it/s]
 51%|█████     | 296/586 [00:08<00:07, 36.72it/s]
 50%|████▉     | 292/586 [00:07<00:07, 36.85it/s]
 51%|█████     | 300/586 [00:08<00:07, 36.83it/s]
 51%|█████     | 296/586 [00:08<00:07, 36.92it/s]
 52%|█████▏    | 304/586 [00:08<00:07, 36.90it/s]
 51%|█████     | 300/586 [00:08<00:07, 36.97it/s]
 53%|█████▎    | 308/586 [00:08<00:07, 36.93it/s]
 52%|█████▏    | 304/586 [00:08<00:07, 36.96it/s]
 53%|█████▎    | 312/586 [00:08<00:07, 36.92it/s]
 53%|█████▎    | 308/586 [00:08<00:07, 36.93it/s]
 54%|█████▍    | 316/586 [00:08<00:07, 36.41it/s]
 53%|█████▎    | 312/586 [00:08<00:07, 36.92it/s]
 55%|█████▍    | 320/586 [00:08<00:07, 36.47it/s]
 54%|█████▍    | 316/586 [00:08<00:07, 36.90it/s]
 55%|█████▌    | 324/586 [00:08<00:07, 36.57it/s]
 55%|█████▍    | 320/586 [00:08<00:07, 36.87it/s]
 56%|█████▌    | 328/586 [00:09<00:07, 35.91it/s]
 55%|█████▌    | 324/586 [00:08<00:07, 36.89it/s]
 57%|█████▋    | 332/586 [00:09<00:07, 36.11it/s]
 56%|█████▌    | 328/586 [00:08<00:06, 36.91it/s]
 57%|█████▋    | 336/586 [00:09<00:06, 36.15it/s]
 57%|█████▋    | 332/586 [00:09<00:06, 36.92it/s]
 58%|█████▊    | 340/586 [00:09<00:06, 36.41it/s]
 57%|█████▋    | 336/586 [00:09<00:06, 36.65it/s]
 59%|█████▊    | 344/586 [00:09<00:06, 36.37it/s]
 58%|█████▊    | 340/586 [00:09<00:06, 36.70it/s]
 59%|█████▉    | 348/586 [00:09<00:06, 36.52it/s]
 59%|█████▊    | 344/586 [00:09<00:06, 36.70it/s]
 60%|██████    | 352/586 [00:09<00:06, 36.46it/s]
 59%|█████▉    | 348/586 [00:09<00:06, 36.52it/s]
 61%|██████    | 356/586 [00:09<00:06, 36.43it/s]
 60%|██████    | 352/586 [00:09<00:06, 36.39it/s]
 61%|██████▏   | 360/586 [00:09<00:06, 36.37it/s]
 61%|██████    | 356/586 [00:09<00:06, 36.31it/s]
 62%|██████▏   | 364/586 [00:10<00:06, 36.37it/s]
 61%|██████▏   | 360/586 [00:09<00:06, 36.19it/s]
 63%|██████▎   | 368/586 [00:10<00:05, 36.37it/s]
 62%|██████▏   | 364/586 [00:09<00:06, 36.14it/s]
 63%|██████▎   | 372/586 [00:10<00:05, 36.37it/s]
 63%|██████▎   | 368/586 [00:10<00:06, 36.07it/s]
 64%|██████▍   | 376/586 [00:10<00:05, 36.40it/s]
 63%|██████▎   | 372/586 [00:10<00:05, 36.02it/s]
 65%|██████▍   | 380/586 [00:10<00:05, 36.40it/s]
 64%|██████▍   | 376/586 [00:10<00:05, 36.04it/s]
 66%|██████▌   | 384/586 [00:10<00:05, 36.33it/s]
 65%|██████▍   | 380/586 [00:10<00:05, 36.04it/s]
 66%|██████▌   | 388/586 [00:10<00:05, 36.39it/s]
 66%|██████▌   | 384/586 [00:10<00:05, 36.11it/s]
 67%|██████▋   | 392/586 [00:10<00:05, 36.58it/s]
 66%|██████▌   | 388/586 [00:10<00:05, 36.28it/s]
 68%|██████▊   | 396/586 [00:10<00:05, 36.69it/s]
 67%|██████▋   | 392/586 [00:10<00:05, 36.38it/s]
 68%|██████▊   | 400/586 [00:10<00:05, 36.78it/s]
 68%|██████▊   | 396/586 [00:10<00:05, 36.48it/s]
 69%|██████▉   | 404/586 [00:11<00:04, 36.73it/s]
 68%|██████▊   | 400/586 [00:10<00:05, 36.51it/s]
 70%|██████▉   | 408/586 [00:11<00:04, 36.74it/s]
 69%|██████▉   | 404/586 [00:11<00:04, 36.50it/s]
 70%|███████   | 412/586 [00:11<00:04, 36.79it/s]
 70%|██████▉   | 408/586 [00:11<00:04, 36.51it/s]
 71%|███████   | 416/586 [00:11<00:04, 36.85it/s]
 70%|███████   | 412/586 [00:11<00:04, 36.54it/s]
 72%|███████▏  | 420/586 [00:11<00:04, 36.89it/s]
 71%|███████   | 416/586 [00:11<00:04, 36.59it/s]
 72%|███████▏  | 424/586 [00:11<00:04, 36.87it/s]
 72%|███████▏  | 420/586 [00:11<00:04, 36.61it/s]
 73%|███████▎  | 428/586 [00:11<00:04, 36.94it/s]
 72%|███████▏  | 424/586 [00:11<00:04, 36.61it/s]
 74%|███████▎  | 432/586 [00:11<00:04, 36.97it/s]
 73%|███████▎  | 428/586 [00:11<00:04, 36.62it/s]
 74%|███████▍  | 436/586 [00:11<00:04, 36.95it/s]
 74%|███████▎  | 432/586 [00:11<00:04, 36.64it/s]
 75%|███████▌  | 440/586 [00:12<00:03, 36.94it/s]
 74%|███████▍  | 436/586 [00:11<00:04, 36.66it/s]
 76%|███████▌  | 444/586 [00:12<00:03, 36.98it/s]
 75%|███████▌  | 440/586 [00:12<00:03, 36.66it/s]
 76%|███████▋  | 448/586 [00:12<00:03, 37.01it/s]
 76%|███████▌  | 444/586 [00:12<00:03, 36.68it/s]
 77%|███████▋  | 452/586 [00:12<00:03, 37.01it/s]
 76%|███████▋  | 448/586 [00:12<00:03, 36.71it/s]
 78%|███████▊  | 456/586 [00:12<00:03, 37.03it/s]
 77%|███████▋  | 452/586 [00:12<00:03, 36.71it/s]
 78%|███████▊  | 460/586 [00:12<00:03, 37.05it/s]
 78%|███████▊  | 456/586 [00:12<00:03, 36.73it/s]
 79%|███████▉  | 464/586 [00:12<00:03, 37.08it/s]
 78%|███████▊  | 460/586 [00:12<00:03, 36.72it/s]
 80%|███████▉  | 468/586 [00:12<00:03, 36.82it/s]
 79%|███████▉  | 464/586 [00:12<00:03, 36.65it/s]
 81%|████████  | 472/586 [00:12<00:03, 36.89it/s]
 80%|███████▉  | 468/586 [00:12<00:03, 36.68it/s]
 81%|████████  | 476/586 [00:13<00:02, 36.90it/s]
 81%|████████  | 472/586 [00:12<00:03, 36.73it/s]
 82%|████████▏ | 480/586 [00:13<00:02, 36.98it/s]
 81%|████████  | 476/586 [00:13<00:02, 36.71it/s]
 83%|████████▎ | 484/586 [00:13<00:02, 37.01it/s]
 82%|████████▏ | 480/586 [00:13<00:02, 36.69it/s]
 83%|████████▎ | 488/586 [00:13<00:02, 37.04it/s]
 83%|████████▎ | 484/586 [00:13<00:02, 36.69it/s]
 84%|████████▍ | 492/586 [00:13<00:02, 37.07it/s]
 83%|████████▎ | 488/586 [00:13<00:02, 36.69it/s]
 85%|████████▍ | 496/586 [00:13<00:02, 37.09it/s]
 84%|████████▍ | 492/586 [00:13<00:02, 36.67it/s]
 85%|████████▌ | 500/586 [00:13<00:02, 36.55it/s]
 85%|████████▍ | 496/586 [00:13<00:02, 36.64it/s]
 86%|████████▌ | 504/586 [00:13<00:02, 36.58it/s]
 85%|████████▌ | 500/586 [00:13<00:02, 36.65it/s]
 87%|████████▋ | 508/586 [00:13<00:02, 36.68it/s]
 86%|████████▌ | 504/586 [00:13<00:02, 36.64it/s]
 87%|████████▋ | 512/586 [00:14<00:02, 36.80it/s]
 87%|████████▋ | 508/586 [00:13<00:02, 36.52it/s]
 88%|████████▊ | 516/586 [00:14<00:01, 36.80it/s]
 87%|████████▋ | 512/586 [00:14<00:02, 36.50it/s]
 89%|████████▊ | 520/586 [00:14<00:01, 36.93it/s]
 88%|████████▊ | 516/586 [00:14<00:01, 36.50it/s]
 89%|████████▉ | 524/586 [00:14<00:01, 36.94it/s]
 89%|████████▊ | 520/586 [00:14<00:01, 36.55it/s]
 90%|█████████ | 528/586 [00:14<00:01, 36.97it/s]
 89%|████████▉ | 524/586 [00:14<00:01, 36.55it/s]
 91%|█████████ | 532/586 [00:14<00:01, 37.00it/s]
 90%|█████████ | 528/586 [00:14<00:01, 36.56it/s]
 91%|█████████▏| 536/586 [00:14<00:01, 36.81it/s]
 91%|█████████ | 532/586 [00:14<00:01, 36.55it/s]
 92%|█████████▏| 540/586 [00:14<00:01, 36.91it/s]
 91%|█████████▏| 536/586 [00:14<00:01, 36.59it/s]
 93%|█████████▎| 544/586 [00:14<00:01, 37.00it/s]
 92%|█████████▏| 540/586 [00:14<00:01, 36.64it/s]
 94%|█████████▎| 548/586 [00:14<00:01, 36.90it/s]
 93%|█████████▎| 544/586 [00:14<00:01, 36.70it/s]
 94%|█████████▍| 552/586 [00:15<00:00, 37.00it/s]
 94%|█████████▎| 548/586 [00:14<00:01, 36.69it/s]
 95%|█████████▍| 556/586 [00:15<00:00, 37.10it/s]
 94%|█████████▍| 552/586 [00:15<00:00, 36.73it/s]
 96%|█████████▌| 560/586 [00:15<00:00, 37.20it/s]
 95%|█████████▍| 556/586 [00:15<00:00, 36.75it/s]
 96%|█████████▌| 564/586 [00:15<00:00, 37.18it/s]
 96%|█████████▌| 560/586 [00:15<00:00, 36.76it/s]
 97%|█████████▋| 568/586 [00:15<00:00, 37.22it/s]
 96%|█████████▌| 564/586 [00:15<00:00, 36.76it/s]
 98%|█████████▊| 572/586 [00:15<00:00, 37.25it/s]
 97%|█████████▋| 568/586 [00:15<00:00, 36.80it/s]
 98%|█████████▊| 576/586 [00:15<00:00, 36.64it/s]
 98%|█████████▊| 572/586 [00:15<00:00, 36.61it/s]
 99%|█████████▉| 580/586 [00:15<00:00, 36.74it/s]
 98%|█████████▊| 576/586 [00:15<00:00, 36.66it/s]
100%|█████████▉| 584/586 [00:15<00:00, 36.86it/s]
100%|██████████| 586/586 [00:16<00:00, 36.57it/s]
n136-128-154:2222575:2222575 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2222575:2222575 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2222575:2222575 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:2222575:2222575 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2222575:2222575 [0] NCCL INFO Comm config Blocking set to 1

 99%|█████████▉| 580/586 [00:15<00:00, 36.71it/s]
100%|█████████▉| 584/586 [00:15<00:00, 36.70it/s]
100%|██████████| 586/586 [00:16<00:00, 36.55it/s]
n136-128-154:2222576:2222576 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:2222576:2222576 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2222576:2222576 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2222576:2222576 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2222576:2222576 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:2222575:2222766 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2222575:2222766 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2222575:2222766 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2222575:2222766 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2222575:2222766 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2222575:2222766 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2222575:2222766 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2222575:2222766 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:2222575:2222766 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2222575:2222766 [0] NCCL INFO Using network IB
n136-128-154:2222575:2222766 [0] NCCL INFO ncclCommInitRankConfig comm 0x117a52e0 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0x4121f562e23c70d - Init START
n136-128-154:2222576:2222768 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2222576:2222768 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2222576:2222768 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2222576:2222768 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2222576:2222768 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2222576:2222768 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2222576:2222768 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2222576:2222768 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:2222576:2222768 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2222576:2222768 [1] NCCL INFO Using network IB
n136-128-154:2222576:2222768 [1] NCCL INFO ncclCommInitRankConfig comm 0x11f1d2b0 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0x4121f562e23c70d - Init START
n136-128-154:2222575:2222766 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2222576:2222768 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2222575:2222766 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2222575:2222766 [0] NCCL INFO Retrieving state for IB
n136-128-154:2222575:2222766 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:2222575:2222766 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2222576:2222768 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2222576:2222768 [1] NCCL INFO Retrieving state for IB
n136-128-154:2222576:2222768 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:2222575:2222766 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2222576:2222768 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2222576:2222768 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2222575:2222766 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2222576:2222768 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2222575:2222766 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2222576:2222768 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2222576:2222768 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2222576:2222768 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2222575:2222766 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2222575:2222766 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2222576:2222768 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2222576:2222768 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2222576:2222768 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2222576:2222768 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2222576:2222768 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2222576:2222768 [1] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2222576:2222768 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2222576:2222768 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2222576:2222768 [1] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2222576:2222768 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2222576:2222768 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2222576:2222768 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2222576:2222768 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2222576:2222768 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2222576:2222768 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2222576:2222768 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2222576:2222768 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2222576:2222768 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2222576:2222768 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2222576:2222768 [1] NCCL INFO ==========================================
n136-128-154:2222576:2222768 [1] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2222576:2222768 [1] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2222576:2222768 [1] NCCL INFO Setting affinity for GPU 7 to 33-62,97-126
n136-128-154:2222575:2222766 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2222575:2222766 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2222575:2222766 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2222575:2222766 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2222575:2222766 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2222575:2222766 [0] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2222575:2222766 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2222575:2222766 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2222575:2222766 [0] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2222575:2222766 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2222575:2222766 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2222575:2222766 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2222575:2222766 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2222575:2222766 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2222575:2222766 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2222575:2222766 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2222575:2222766 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2222575:2222766 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2222575:2222766 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2222575:2222766 [0] NCCL INFO ==========================================
n136-128-154:2222575:2222766 [0] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2222575:2222766 [0] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2222575:2222766 [0] NCCL INFO Setting affinity for GPU 6 to 33-62,97-126
n136-128-154:2222576:2222768 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2222576:2222768 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222576:2222768 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222576:2222768 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222576:2222768 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222576:2222768 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222576:2222768 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222576:2222768 [1] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222576:2222768 [1] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222576:2222768 [1] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222576:2222768 [1] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222576:2222768 [1] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222576:2222768 [1] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222576:2222768 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2222576:2222768 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222576:2222768 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222576:2222768 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222576:2222768 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222576:2222768 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222576:2222768 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222576:2222768 [1] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2222576:2222768 [1] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2222576:2222768 [1] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2222576:2222768 [1] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2222576:2222768 [1] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2222576:2222768 [1] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2222575:2222766 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2222575:2222766 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222575:2222766 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222575:2222766 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222575:2222766 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222575:2222766 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222575:2222766 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222575:2222766 [0] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222575:2222766 [0] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222575:2222766 [0] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222575:2222766 [0] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222575:2222766 [0] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222575:2222766 [0] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222575:2222766 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2222575:2222766 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222575:2222766 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222575:2222766 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222575:2222766 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222575:2222766 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222575:2222766 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2222575:2222766 [0] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2222575:2222766 [0] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2222575:2222766 [0] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2222575:2222766 [0] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2222575:2222766 [0] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2222575:2222766 [0] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2222575:2222766 [0] NCCL INFO comm 0x117a52e0 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:2222576:2222768 [1] NCCL INFO comm 0x11f1d2b0 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:2222575:2222766 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:2222575:2222766 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:2222575:2222766 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:2222576:2222768 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:2222575:2222766 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:2222576:2222768 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:2222575:2222766 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:2222576:2222768 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:2222575:2222766 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:2222576:2222768 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:2222575:2222766 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:2222576:2222768 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:2222575:2222766 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:2222576:2222768 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:2222575:2222766 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:2222576:2222768 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:2222575:2222766 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:2222576:2222768 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:2222575:2222766 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:2222576:2222768 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:2222575:2222766 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:2222576:2222768 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:2222575:2222766 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:2222576:2222768 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:2222575:2222766 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:2222576:2222768 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:2222575:2222766 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:2222576:2222768 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:2222575:2222766 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:2222576:2222768 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:2222575:2222766 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:2222576:2222768 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:2222575:2222766 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:2222576:2222768 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:2222575:2222766 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:2222576:2222768 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:2222575:2222766 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:2222576:2222768 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:2222575:2222766 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:2222576:2222768 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:2222575:2222766 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:2222576:2222768 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:2222575:2222766 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:2222576:2222768 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:2222575:2222766 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:2222576:2222768 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:2222576:2222768 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:2222576:2222768 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:2222575:2222766 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:2222575:2222766 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:2222575:2222766 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:2222575:2222766 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:2222575:2222766 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:2222576:2222768 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:2222575:2222766 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:2222576:2222768 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:2222575:2222766 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:2222576:2222768 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:2222575:2222766 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:2222576:2222768 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:2222575:2222766 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:2222576:2222768 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:2222575:2222766 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:2222576:2222768 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:2222575:2222766 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:2222576:2222768 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:2222575:2222766 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:2222576:2222768 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:2222575:2222766 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:2222576:2222768 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:2222575:2222766 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:2222576:2222768 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:2222575:2222766 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:2222576:2222768 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:2222575:2222766 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:2222576:2222768 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:2222575:2222766 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:2222576:2222768 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:2222575:2222766 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:2222576:2222768 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:2222575:2222766 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:2222576:2222768 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:2222575:2222766 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:2222576:2222768 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:2222575:2222766 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:2222576:2222768 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:2222575:2222766 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:2222576:2222768 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:2222575:2222766 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:2222576:2222768 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:2222575:2222766 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:2222576:2222768 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:2222575:2222766 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:2222576:2222768 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:2222575:2222766 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:2222576:2222768 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:2222575:2222766 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:2222576:2222768 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:2222575:2222766 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:2222576:2222768 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:2222575:2222766 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:2222576:2222768 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:2222575:2222766 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:2222575:2222766 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:2222576:2222768 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2222575:2222766 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:2222575:2222766 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:2222575:2222766 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:2222575:2222766 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:2222575:2222766 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:2222575:2222766 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:2222575:2222766 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:2222575:2222766 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:2222575:2222766 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:2222575:2222766 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:2222575:2222766 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:2222575:2222766 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:2222575:2222766 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:2222575:2222766 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:2222575:2222766 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:2222575:2222766 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:2222575:2222766 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:2222575:2222766 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:2222575:2222766 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2222576:2222768 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2222576:2222780 [1] NCCL INFO [Proxy Service] Device 1 CPU core 36
n136-128-154:2222576:2222781 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 37
n136-128-154:2222575:2222766 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2222575:2222766 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:2222575:2222782 [0] NCCL INFO [Proxy Service] Device 0 CPU core 104
n136-128-154:2222575:2222783 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 106
n136-128-154:2222576:2222768 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2222576:2222768 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2222575:2222766 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2222575:2222766 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2222575:2222766 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:2222575:2222766 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2222575:2222766 [0] NCCL INFO ncclCommInitRankConfig comm 0x117a52e0 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0x4121f562e23c70d - Init COMPLETE
n136-128-154:2222575:2222766 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.79 (kernels 0.13, alloc 0.19, bootstrap 0.16, allgathers 0.00, topo 0.15, graphs 0.00, connections 0.01, rest 0.14)
n136-128-154:2222576:2222768 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2222576:2222768 [1] NCCL INFO ncclCommInitRankConfig comm 0x11f1d2b0 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0x4121f562e23c70d - Init COMPLETE
n136-128-154:2222576:2222768 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.61 (kernels 0.13, alloc 0.17, bootstrap 0.00, allgathers 0.02, topo 0.15, graphs 0.00, connections 0.01, rest 0.13)
n136-128-154:2222575:2222784 [0] NCCL INFO Channel 00/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222575:2222784 [0] NCCL INFO Channel 01/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222575:2222784 [0] NCCL INFO Channel 02/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222575:2222784 [0] NCCL INFO Channel 03/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222576:2222785 [1] NCCL INFO Channel 00/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222575:2222784 [0] NCCL INFO Channel 04/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222576:2222785 [1] NCCL INFO Channel 01/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222575:2222784 [0] NCCL INFO Channel 05/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222576:2222785 [1] NCCL INFO Channel 02/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222575:2222784 [0] NCCL INFO Channel 06/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222576:2222785 [1] NCCL INFO Channel 03/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222575:2222784 [0] NCCL INFO Channel 07/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222576:2222785 [1] NCCL INFO Channel 04/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222575:2222784 [0] NCCL INFO Channel 08/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222576:2222785 [1] NCCL INFO Channel 05/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222575:2222784 [0] NCCL INFO Channel 09/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222576:2222785 [1] NCCL INFO Channel 06/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222575:2222784 [0] NCCL INFO Channel 10/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222576:2222785 [1] NCCL INFO Channel 07/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222575:2222784 [0] NCCL INFO Channel 11/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222576:2222785 [1] NCCL INFO Channel 08/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222575:2222784 [0] NCCL INFO Channel 12/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222576:2222785 [1] NCCL INFO Channel 09/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222575:2222784 [0] NCCL INFO Channel 13/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222576:2222785 [1] NCCL INFO Channel 10/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222575:2222784 [0] NCCL INFO Channel 14/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222576:2222785 [1] NCCL INFO Channel 11/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222575:2222784 [0] NCCL INFO Channel 15/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222576:2222785 [1] NCCL INFO Channel 12/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222575:2222784 [0] NCCL INFO Channel 16/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222576:2222785 [1] NCCL INFO Channel 13/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222575:2222784 [0] NCCL INFO Channel 17/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222576:2222785 [1] NCCL INFO Channel 14/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222575:2222784 [0] NCCL INFO Channel 18/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222576:2222785 [1] NCCL INFO Channel 15/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222575:2222784 [0] NCCL INFO Channel 19/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222576:2222785 [1] NCCL INFO Channel 16/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222575:2222784 [0] NCCL INFO Channel 20/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222576:2222785 [1] NCCL INFO Channel 17/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222575:2222784 [0] NCCL INFO Channel 21/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222576:2222785 [1] NCCL INFO Channel 18/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222575:2222784 [0] NCCL INFO Channel 22/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222576:2222785 [1] NCCL INFO Channel 19/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222575:2222784 [0] NCCL INFO Channel 23/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2222576:2222785 [1] NCCL INFO Channel 20/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222576:2222785 [1] NCCL INFO Channel 21/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222576:2222785 [1] NCCL INFO Channel 22/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222576:2222785 [1] NCCL INFO Channel 23/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222575:2222784 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:2222576:2222785 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-09:16:57:29 INFO     [evaluator:559] Running loglikelihood requests
2025-12-09:16:57:29 INFO     [evaluator:559] Running loglikelihood requests
Passed argument batch_size = auto:1. Detecting largest batch size

Running loglikelihood requests:   0%|          | 0/2344 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 64
Determined largest batch size: 64

Running loglikelihood requests:   0%|          | 1/2344 [00:38<25:15:25, 38.81s/it]
Running loglikelihood requests:   3%|▎         | 65/2344 [00:45<19:47,  1.92it/s]  
Running loglikelihood requests:   6%|▌         | 135/2344 [00:51<09:22,  3.93it/s]
Running loglikelihood requests:   9%|▊         | 204/2344 [00:56<06:13,  5.72it/s]
Running loglikelihood requests:  12%|█▏        | 276/2344 [01:02<04:39,  7.41it/s]
Running loglikelihood requests:  15%|█▍        | 341/2344 [01:07<03:55,  8.52it/s]
Running loglikelihood requests:  17%|█▋        | 408/2344 [01:13<03:22,  9.54it/s]
Running loglikelihood requests:  20%|██        | 475/2344 [01:18<03:00, 10.36it/s]
Running loglikelihood requests:  24%|██▎       | 551/2344 [01:23<02:36, 11.48it/s]
Running loglikelihood requests:  26%|██▋       | 616/2344 [01:28<02:27, 11.74it/s]
Running loglikelihood requests:  29%|██▉       | 684/2344 [01:34<02:16, 12.13it/s]
Running loglikelihood requests:  32%|███▏      | 748/2344 [01:39<02:10, 12.23it/s]
Running loglikelihood requests:  35%|███▍      | 815/2344 [01:44<02:02, 12.49it/s]
Running loglikelihood requests:  38%|███▊      | 883/2344 [01:49<01:54, 12.77it/s]
Running loglikelihood requests:  41%|████      | 952/2344 [01:54<01:46, 13.03it/s]
Running loglikelihood requests:  44%|████▎     | 1022/2344 [01:59<01:39, 13.31it/s]
Running loglikelihood requests:  47%|████▋     | 1094/2344 [02:04<01:31, 13.66it/s]
Running loglikelihood requests:  50%|█████     | 1172/2344 [02:09<01:22, 14.28it/s]
Running loglikelihood requests:  53%|█████▎    | 1238/2344 [02:14<01:18, 14.03it/s]
Running loglikelihood requests:  56%|█████▌    | 1306/2344 [02:19<01:14, 13.98it/s]
Running loglikelihood requests:  59%|█████▊    | 1377/2344 [02:24<01:08, 14.18it/s]
Running loglikelihood requests:  62%|██████▏   | 1451/2344 [02:28<01:01, 14.54it/s]
Running loglikelihood requests:  65%|██████▍   | 1518/2344 [02:33<00:57, 14.43it/s]
Running loglikelihood requests:  68%|██████▊   | 1587/2344 [02:38<00:52, 14.51it/s]
Running loglikelihood requests:  71%|███████   | 1654/2344 [02:42<00:47, 14.46it/s]
Running loglikelihood requests:  74%|███████▎  | 1724/2344 [02:47<00:42, 14.66it/s]
Running loglikelihood requests:  76%|███████▋  | 1793/2344 [02:52<00:37, 14.77it/s]
Running loglikelihood requests:  79%|███████▉  | 1863/2344 [02:56<00:32, 14.95it/s]
Running loglikelihood requests:  82%|████████▏ | 1933/2344 [03:01<00:27, 15.11it/s]
Running loglikelihood requests:  85%|████████▌ | 2000/2344 [03:05<00:22, 15.06it/s]
Running loglikelihood requests:  88%|████████▊ | 2074/2344 [03:10<00:17, 15.53it/s]
Running loglikelihood requests:  92%|█████████▏| 2146/2344 [03:14<00:12, 15.78it/s]
Running loglikelihood requests:  94%|█████████▍| 2214/2344 [03:18<00:08, 15.77it/s]
Running loglikelihood requests:  98%|█████████▊| 2297/2344 [03:21<00:02, 19.20it/s]
Running loglikelihood requests: 100%|██████████| 2344/2344 [03:21<00:00, 11.64it/s]
n136-128-154:2222576:2223034 [1] NCCL INFO Channel 00/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222576:2223034 [1] NCCL INFO Channel 01/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222576:2223034 [1] NCCL INFO Channel 02/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222576:2223034 [1] NCCL INFO Channel 03/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222576:2223034 [1] NCCL INFO Channel 04/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222576:2223034 [1] NCCL INFO Channel 05/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222576:2223034 [1] NCCL INFO Channel 06/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222576:2223034 [1] NCCL INFO Channel 07/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222576:2223034 [1] NCCL INFO Channel 08/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222576:2223034 [1] NCCL INFO Channel 09/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222576:2223034 [1] NCCL INFO Channel 10/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222576:2223034 [1] NCCL INFO Channel 11/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222576:2223034 [1] NCCL INFO Channel 12/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222576:2223034 [1] NCCL INFO Channel 13/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222576:2223034 [1] NCCL INFO Channel 14/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222576:2223034 [1] NCCL INFO Channel 15/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222576:2223034 [1] NCCL INFO Channel 16/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222576:2223034 [1] NCCL INFO Channel 17/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222576:2223034 [1] NCCL INFO Channel 18/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222576:2223034 [1] NCCL INFO Channel 19/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222576:2223034 [1] NCCL INFO Channel 20/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222576:2223034 [1] NCCL INFO Channel 21/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222576:2223034 [1] NCCL INFO Channel 22/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222576:2223034 [1] NCCL INFO Channel 23/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222576:2223034 [1] NCCL INFO Channel 24/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222576:2223034 [1] NCCL INFO Channel 25/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222576:2223034 [1] NCCL INFO Channel 26/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222576:2223034 [1] NCCL INFO Channel 27/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222576:2223034 [1] NCCL INFO Channel 28/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222576:2223034 [1] NCCL INFO Channel 29/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222576:2223034 [1] NCCL INFO Channel 30/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2222576:2223034 [1] NCCL INFO Channel 31/1 : 1[7] -> 0[6] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:2222576:2223038 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2222576:2223038 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2222576:2223038 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2222576:2223038 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2222576:2223038 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2222576:2223038 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2222576:2222780 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-09:17:01:10 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 25, batch_size: auto (64)
|    Tasks    |Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|-------------|------:|------|-----:|--------|---|-----:|---|-----:|
|arc_challenge|      1|none  |    25|acc     |↑  |0.3712|±  |0.0141|
|             |       |none  |    25|acc_norm|↑  |0.4061|±  |0.0144|

[rank0]:[W1209 17:01:11.668862667 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:2222575:2223088 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2222575:2223088 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2222575:2223088 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2222575:2222782 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2222575:2223088 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2222575:2223088 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2222575:2223088 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2222576:2222780 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2222576:2223038 [1] NCCL INFO comm 0x11f1d2b0 rank 1 nranks 2 cudaDev 1 busId c9000 - Abort COMPLETE
n136-128-154:2222575:2223088 [0] NCCL INFO comm 0x117a52e0 rank 0 nranks 2 cudaDev 0 busId c5000 - Abort COMPLETE
任务 arc_challenge 评估完成！

==================================================
开始评估：任务=truthfulqa_mc1 | 少样本数=0 | 模型=Llama-3.1-8B-quantization-fg
输出路径：results2/Llama-3.1-8B-quantization-fg/baseline_BI/truthfulqa_mc1.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-09:17:02:00 INFO     [__main__:440] Selected Tasks: ['truthfulqa_mc1']
2025-12-09:17:02:00 INFO     [__main__:440] Selected Tasks: ['truthfulqa_mc1']
2025-12-09:17:02:00 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:17:02:00 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-09:17:02:00 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:17:02:00 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-09:17:02:01 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:17:02:01 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:17:02:01 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.04s/it]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.03s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.03s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.06s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.03s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.07s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.36it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.18it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.31it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.15it/s]
2025-12-09:17:02:31 INFO     [evaluator:305] num_fewshot has been set to 0 for truthfulqa_mc1 in its config. Manual configuration will be ignored.
2025-12-09:17:02:31 INFO     [api.task:434] Building contexts for truthfulqa_mc1 on rank 1...

  0%|          | 0/408 [00:00<?, ?it/s]
 24%|██▍       | 98/408 [00:00<00:00, 973.73it/s]
 49%|████▊     | 198/408 [00:00<00:00, 987.73it/s]
 74%|███████▍  | 301/408 [00:00<00:00, 1003.42it/s]
 99%|█████████▉| 404/408 [00:00<00:00, 1011.19it/s]
100%|██████████| 408/408 [00:00<00:00, 1003.80it/s]
2025-12-09:17:02:33 INFO     [evaluator:305] num_fewshot has been set to 0 for truthfulqa_mc1 in its config. Manual configuration will be ignored.
2025-12-09:17:02:33 INFO     [api.task:434] Building contexts for truthfulqa_mc1 on rank 0...

  0%|          | 0/409 [00:00<?, ?it/s]
 24%|██▍       | 99/409 [00:00<00:00, 980.32it/s]
 50%|████▉     | 204/409 [00:00<00:00, 1016.26it/s]
 75%|███████▌  | 308/409 [00:00<00:00, 1025.38it/s]
100%|██████████| 409/409 [00:00<00:00, 1017.25it/s]
n136-128-154:2223134:2223134 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2223134:2223134 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2223134:2223134 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:2223134:2223134 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2223134:2223134 [0] NCCL INFO Comm config Blocking set to 1
n136-128-154:2223135:2223135 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:2223135:2223135 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2223135:2223135 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2223135:2223135 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2223135:2223135 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:2223135:2223317 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2223135:2223317 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2223134:2223316 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2223134:2223316 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2223135:2223317 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2223135:2223317 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2223135:2223317 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2223134:2223316 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2223134:2223316 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2223134:2223316 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2223134:2223316 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2223134:2223316 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2223134:2223316 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:2223135:2223317 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2223135:2223317 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2223135:2223317 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:2223134:2223316 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2223134:2223316 [0] NCCL INFO Using network IB
n136-128-154:2223135:2223317 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2223135:2223317 [1] NCCL INFO Using network IB
n136-128-154:2223135:2223317 [1] NCCL INFO ncclCommInitRankConfig comm 0x14fc5940 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0xfde0410e5cf6efa8 - Init START
n136-128-154:2223134:2223316 [0] NCCL INFO ncclCommInitRankConfig comm 0x14cd8680 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0xfde0410e5cf6efa8 - Init START
n136-128-154:2223135:2223317 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2223134:2223316 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2223135:2223317 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2223135:2223317 [1] NCCL INFO Retrieving state for IB
n136-128-154:2223135:2223317 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:2223135:2223317 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2223134:2223316 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2223134:2223316 [0] NCCL INFO Retrieving state for IB
n136-128-154:2223134:2223316 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:2223135:2223317 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2223134:2223316 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2223135:2223317 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2223134:2223316 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2223135:2223317 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2223134:2223316 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2223134:2223316 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2223134:2223316 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2223134:2223316 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2223135:2223317 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2223135:2223317 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2223134:2223316 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2223134:2223316 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2223134:2223316 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2223134:2223316 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2223134:2223316 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2223134:2223316 [0] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2223134:2223316 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2223134:2223316 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2223134:2223316 [0] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2223134:2223316 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2223134:2223316 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2223134:2223316 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2223134:2223316 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2223134:2223316 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2223134:2223316 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2223134:2223316 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2223134:2223316 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2223134:2223316 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2223134:2223316 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2223134:2223316 [0] NCCL INFO ==========================================
n136-128-154:2223134:2223316 [0] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2223134:2223316 [0] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2223134:2223316 [0] NCCL INFO Setting affinity for GPU 6 to 33-62,97-126
n136-128-154:2223134:2223316 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2223134:2223316 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223134:2223316 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223134:2223316 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223134:2223316 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223134:2223316 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223134:2223316 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223134:2223316 [0] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223134:2223316 [0] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223134:2223316 [0] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223134:2223316 [0] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223134:2223316 [0] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223135:2223317 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2223134:2223316 [0] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223135:2223317 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2223135:2223317 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2223135:2223317 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2223135:2223317 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2223135:2223317 [1] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2223135:2223317 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2223135:2223317 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2223135:2223317 [1] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2223135:2223317 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2223135:2223317 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2223135:2223317 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2223135:2223317 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2223135:2223317 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2223135:2223317 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2223135:2223317 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2223135:2223317 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2223135:2223317 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2223135:2223317 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2223135:2223317 [1] NCCL INFO ==========================================
n136-128-154:2223134:2223316 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2223134:2223316 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223135:2223317 [1] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2223134:2223316 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223134:2223316 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223135:2223317 [1] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2223134:2223316 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223134:2223316 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223134:2223316 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223134:2223316 [0] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2223135:2223317 [1] NCCL INFO Setting affinity for GPU 7 to 33-62,97-126
n136-128-154:2223134:2223316 [0] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2223134:2223316 [0] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2223134:2223316 [0] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2223134:2223316 [0] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2223134:2223316 [0] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2223135:2223317 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2223135:2223317 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223135:2223317 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223135:2223317 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223135:2223317 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223135:2223317 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223135:2223317 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223135:2223317 [1] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223135:2223317 [1] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223135:2223317 [1] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223135:2223317 [1] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223135:2223317 [1] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223135:2223317 [1] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223135:2223317 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2223135:2223317 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223135:2223317 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223135:2223317 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223135:2223317 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223135:2223317 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223135:2223317 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223135:2223317 [1] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2223135:2223317 [1] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2223135:2223317 [1] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2223135:2223317 [1] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2223135:2223317 [1] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2223135:2223317 [1] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2223134:2223316 [0] NCCL INFO comm 0x14cd8680 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:2223135:2223317 [1] NCCL INFO comm 0x14fc5940 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:2223134:2223316 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:2223134:2223316 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:2223135:2223317 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:2223134:2223316 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:2223135:2223317 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:2223134:2223316 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:2223135:2223317 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:2223134:2223316 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:2223135:2223317 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:2223134:2223316 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:2223135:2223317 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:2223134:2223316 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:2223135:2223317 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:2223134:2223316 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:2223135:2223317 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:2223134:2223316 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:2223135:2223317 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:2223134:2223316 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:2223135:2223317 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:2223134:2223316 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:2223135:2223317 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:2223134:2223316 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:2223135:2223317 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:2223134:2223316 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:2223135:2223317 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:2223134:2223316 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:2223135:2223317 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:2223134:2223316 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:2223135:2223317 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:2223134:2223316 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:2223135:2223317 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:2223134:2223316 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:2223135:2223317 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:2223134:2223316 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:2223135:2223317 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:2223134:2223316 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:2223135:2223317 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:2223134:2223316 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:2223135:2223317 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:2223134:2223316 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:2223135:2223317 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:2223134:2223316 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:2223135:2223317 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:2223134:2223316 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:2223135:2223317 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:2223134:2223316 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:2223135:2223317 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:2223135:2223317 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:2223134:2223316 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:2223134:2223316 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:2223134:2223316 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:2223134:2223316 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:2223135:2223317 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:2223134:2223316 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:2223135:2223317 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:2223134:2223316 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:2223135:2223317 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:2223134:2223316 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:2223135:2223317 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:2223134:2223316 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:2223135:2223317 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:2223134:2223316 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:2223135:2223317 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:2223134:2223316 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:2223135:2223317 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:2223134:2223316 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:2223135:2223317 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:2223134:2223316 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:2223135:2223317 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:2223134:2223316 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:2223135:2223317 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:2223134:2223316 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:2223135:2223317 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:2223134:2223316 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:2223135:2223317 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:2223134:2223316 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:2223135:2223317 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:2223134:2223316 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:2223135:2223317 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:2223134:2223316 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:2223135:2223317 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:2223134:2223316 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:2223135:2223317 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:2223134:2223316 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:2223135:2223317 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:2223135:2223317 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:2223134:2223316 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:2223135:2223317 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:2223134:2223316 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:2223135:2223317 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:2223134:2223316 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:2223135:2223317 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:2223134:2223316 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:2223135:2223317 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:2223134:2223316 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:2223135:2223317 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:2223134:2223316 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:2223135:2223317 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:2223134:2223316 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:2223135:2223317 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:2223134:2223316 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:2223135:2223317 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2223134:2223316 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:2223134:2223316 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:2223134:2223316 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:2223134:2223316 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:2223134:2223316 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:2223134:2223316 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:2223134:2223316 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:2223134:2223316 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:2223134:2223316 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:2223134:2223316 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:2223134:2223316 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:2223134:2223316 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:2223134:2223316 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:2223134:2223316 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:2223134:2223316 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:2223134:2223316 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:2223134:2223316 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:2223134:2223316 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:2223134:2223316 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:2223134:2223316 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:2223134:2223316 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:2223134:2223316 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2223134:2223316 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2223134:2223316 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:2223134:2223328 [0] NCCL INFO [Proxy Service] Device 0 CPU core 106
n136-128-154:2223134:2223329 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 43
n136-128-154:2223135:2223317 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2223135:2223330 [1] NCCL INFO [Proxy Service] Device 1 CPU core 35
n136-128-154:2223135:2223331 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 36
n136-128-154:2223134:2223316 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2223134:2223316 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2223134:2223316 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:2223135:2223317 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2223135:2223317 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2223134:2223316 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2223134:2223316 [0] NCCL INFO ncclCommInitRankConfig comm 0x14cd8680 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0xfde0410e5cf6efa8 - Init COMPLETE
n136-128-154:2223134:2223316 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.71 (kernels 0.13, alloc 0.36, bootstrap 0.00, allgathers 0.00, topo 0.08, graphs 0.00, connections 0.01, rest 0.12)
n136-128-154:2223135:2223317 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2223135:2223317 [1] NCCL INFO ncclCommInitRankConfig comm 0x14fc5940 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0xfde0410e5cf6efa8 - Init COMPLETE
n136-128-154:2223135:2223317 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.71 (kernels 0.12, alloc 0.37, bootstrap 0.00, allgathers 0.00, topo 0.08, graphs 0.00, connections 0.01, rest 0.12)
n136-128-154:2223134:2223332 [0] NCCL INFO Channel 00/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223134:2223332 [0] NCCL INFO Channel 01/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223134:2223332 [0] NCCL INFO Channel 02/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223134:2223332 [0] NCCL INFO Channel 03/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223134:2223332 [0] NCCL INFO Channel 04/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223134:2223332 [0] NCCL INFO Channel 05/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223134:2223332 [0] NCCL INFO Channel 06/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223134:2223332 [0] NCCL INFO Channel 07/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223134:2223332 [0] NCCL INFO Channel 08/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223134:2223332 [0] NCCL INFO Channel 09/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223134:2223332 [0] NCCL INFO Channel 10/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223134:2223332 [0] NCCL INFO Channel 11/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223135:2223333 [1] NCCL INFO Channel 00/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223134:2223332 [0] NCCL INFO Channel 12/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223135:2223333 [1] NCCL INFO Channel 01/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223134:2223332 [0] NCCL INFO Channel 13/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223135:2223333 [1] NCCL INFO Channel 02/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223134:2223332 [0] NCCL INFO Channel 14/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223135:2223333 [1] NCCL INFO Channel 03/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223134:2223332 [0] NCCL INFO Channel 15/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223135:2223333 [1] NCCL INFO Channel 04/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223134:2223332 [0] NCCL INFO Channel 16/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223135:2223333 [1] NCCL INFO Channel 05/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223134:2223332 [0] NCCL INFO Channel 17/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223135:2223333 [1] NCCL INFO Channel 06/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223134:2223332 [0] NCCL INFO Channel 18/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223135:2223333 [1] NCCL INFO Channel 07/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223134:2223332 [0] NCCL INFO Channel 19/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223135:2223333 [1] NCCL INFO Channel 08/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223134:2223332 [0] NCCL INFO Channel 20/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223135:2223333 [1] NCCL INFO Channel 09/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223134:2223332 [0] NCCL INFO Channel 21/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223135:2223333 [1] NCCL INFO Channel 10/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223134:2223332 [0] NCCL INFO Channel 22/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223135:2223333 [1] NCCL INFO Channel 11/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223134:2223332 [0] NCCL INFO Channel 23/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223135:2223333 [1] NCCL INFO Channel 12/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223333 [1] NCCL INFO Channel 13/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223333 [1] NCCL INFO Channel 14/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223333 [1] NCCL INFO Channel 15/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223333 [1] NCCL INFO Channel 16/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223333 [1] NCCL INFO Channel 17/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223333 [1] NCCL INFO Channel 18/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223333 [1] NCCL INFO Channel 19/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223333 [1] NCCL INFO Channel 20/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223333 [1] NCCL INFO Channel 21/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223333 [1] NCCL INFO Channel 22/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223333 [1] NCCL INFO Channel 23/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223333 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:2223134:2223332 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-09:17:02:34 INFO     [evaluator:559] Running loglikelihood requests
2025-12-09:17:02:34 INFO     [evaluator:559] Running loglikelihood requests

Running loglikelihood requests:   0%|          | 0/2066 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 64
Determined largest batch size: 64

Running loglikelihood requests:   0%|          | 1/2066 [00:07<4:08:03,  7.21s/it]
Running loglikelihood requests:   3%|▎         | 65/2066 [00:08<03:13, 10.33it/s] 
Running loglikelihood requests:   6%|▌         | 129/2066 [00:09<01:38, 19.66it/s]
Running loglikelihood requests:   9%|▉         | 193/2066 [00:10<01:07, 27.83it/s]
Running loglikelihood requests:  12%|█▏        | 257/2066 [00:11<00:52, 34.64it/s]
Running loglikelihood requests:  16%|█▌        | 321/2066 [00:13<00:43, 40.13it/s]
Running loglikelihood requests:  19%|█▊        | 385/2066 [00:14<00:37, 44.44it/s]
Running loglikelihood requests:  22%|██▏       | 449/2066 [00:15<00:33, 47.68it/s]
Running loglikelihood requests:  25%|██▍       | 513/2066 [00:16<00:30, 50.37it/s]
Running loglikelihood requests:  28%|██▊       | 577/2066 [00:17<00:28, 52.27it/s]
Running loglikelihood requests:  31%|███       | 641/2066 [00:18<00:26, 53.73it/s]
Running loglikelihood requests:  34%|███▍      | 705/2066 [00:19<00:24, 54.79it/s]
Running loglikelihood requests:  37%|███▋      | 769/2066 [00:20<00:23, 55.63it/s]
Running loglikelihood requests:  40%|████      | 833/2066 [00:22<00:21, 56.21it/s]
Running loglikelihood requests:  43%|████▎     | 897/2066 [00:23<00:20, 56.75it/s]
Running loglikelihood requests:  47%|████▋     | 961/2066 [00:24<00:19, 57.04it/s]
Running loglikelihood requests:  50%|████▉     | 1025/2066 [00:25<00:18, 57.30it/s]
Running loglikelihood requests:  53%|█████▎    | 1089/2066 [00:26<00:16, 57.59it/s]
Running loglikelihood requests:  56%|█████▌    | 1153/2066 [00:27<00:15, 57.74it/s]
Running loglikelihood requests:  59%|█████▉    | 1217/2066 [00:28<00:14, 58.01it/s]
Running loglikelihood requests:  62%|██████▏   | 1281/2066 [00:29<00:13, 58.19it/s]
Running loglikelihood requests:  65%|██████▌   | 1345/2066 [00:30<00:12, 58.30it/s]
Running loglikelihood requests:  68%|██████▊   | 1409/2066 [00:31<00:11, 58.34it/s]
Running loglikelihood requests:  71%|███████▏  | 1473/2066 [00:33<00:10, 58.41it/s]
Running loglikelihood requests:  74%|███████▍  | 1537/2066 [00:34<00:09, 58.60it/s]
Running loglikelihood requests:  77%|███████▋  | 1601/2066 [00:35<00:07, 58.72it/s]
Running loglikelihood requests:  81%|████████  | 1665/2066 [00:36<00:06, 58.83it/s]
Running loglikelihood requests:  84%|████████▎ | 1729/2066 [00:37<00:05, 59.29it/s]
Running loglikelihood requests:  87%|████████▋ | 1793/2066 [00:38<00:04, 59.31it/s]
Running loglikelihood requests:  90%|████████▉ | 1857/2066 [00:39<00:03, 59.60it/s]
Running loglikelihood requests:  93%|█████████▎| 1921/2066 [00:40<00:02, 59.98it/s]
Running loglikelihood requests:  96%|█████████▌| 1985/2066 [00:41<00:01, 60.27it/s]
Running loglikelihood requests:  99%|█████████▉| 2049/2066 [00:41<00:00, 76.53it/s]
Running loglikelihood requests: 100%|██████████| 2066/2066 [00:41<00:00, 49.33it/s]
n136-128-154:2223135:2223378 [1] NCCL INFO Channel 00/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223378 [1] NCCL INFO Channel 01/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223378 [1] NCCL INFO Channel 02/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223378 [1] NCCL INFO Channel 03/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223378 [1] NCCL INFO Channel 04/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223378 [1] NCCL INFO Channel 05/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223378 [1] NCCL INFO Channel 06/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223378 [1] NCCL INFO Channel 07/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223378 [1] NCCL INFO Channel 08/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223378 [1] NCCL INFO Channel 09/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223378 [1] NCCL INFO Channel 10/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223378 [1] NCCL INFO Channel 11/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223378 [1] NCCL INFO Channel 12/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223378 [1] NCCL INFO Channel 13/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223378 [1] NCCL INFO Channel 14/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223378 [1] NCCL INFO Channel 15/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223378 [1] NCCL INFO Channel 16/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223378 [1] NCCL INFO Channel 17/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223378 [1] NCCL INFO Channel 18/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223378 [1] NCCL INFO Channel 19/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223378 [1] NCCL INFO Channel 20/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223378 [1] NCCL INFO Channel 21/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223378 [1] NCCL INFO Channel 22/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223378 [1] NCCL INFO Channel 23/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223378 [1] NCCL INFO Channel 24/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223378 [1] NCCL INFO Channel 25/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223378 [1] NCCL INFO Channel 26/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223378 [1] NCCL INFO Channel 27/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223378 [1] NCCL INFO Channel 28/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223378 [1] NCCL INFO Channel 29/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223378 [1] NCCL INFO Channel 30/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223135:2223378 [1] NCCL INFO Channel 31/1 : 1[7] -> 0[6] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:2223135:2223384 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2223135:2223384 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2223135:2223384 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2223135:2223384 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2223135:2223384 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2223135:2223384 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2223135:2223330 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-09:17:03:27 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: auto (64)
|    Tasks     |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|--------------|------:|------|-----:|------|---|-----:|---|-----:|
|truthfulqa_mc1|      2|none  |     0|acc   |↑  |0.2644|±  |0.0154|

[rank0]:[W1209 17:03:28.487046361 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:2223134:2223432 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2223134:2223432 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2223134:2223328 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2223134:2223432 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2223134:2223432 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2223134:2223432 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2223134:2223432 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2223135:2223330 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2223135:2223384 [1] NCCL INFO comm 0x14fc5940 rank 1 nranks 2 cudaDev 1 busId c9000 - Abort COMPLETE
n136-128-154:2223134:2223432 [0] NCCL INFO comm 0x14cd8680 rank 0 nranks 2 cudaDev 0 busId c5000 - Abort COMPLETE
任务 truthfulqa_mc1 评估完成！

==================================================
开始评估：任务=piqa | 少样本数=0 | 模型=Llama-3.1-8B-quantization-fg
输出路径：results2/Llama-3.1-8B-quantization-fg/baseline_BI/piqa.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-09:17:04:17 INFO     [__main__:440] Selected Tasks: ['piqa']
2025-12-09:17:04:17 INFO     [__main__:440] Selected Tasks: ['piqa']
2025-12-09:17:04:17 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:17:04:17 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-09:17:04:17 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:17:04:17 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-09:17:04:18 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:17:04:19 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:17:04:19 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
`torch_dtype` is deprecated! Use `dtype` instead!
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.08it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.05it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.08it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.05it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.08it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:01,  1.02s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.53it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.32it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.37it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.22it/s]
2025-12-09:17:04:33 WARNING  [evaluator:309] Overwriting default num_fewshot of piqa from None to 0
2025-12-09:17:04:33 INFO     [api.task:434] Building contexts for piqa on rank 1...

  0%|          | 0/919 [00:00<?, ?it/s]
 16%|█▌        | 146/919 [00:00<00:00, 1457.00it/s]
 33%|███▎      | 300/919 [00:00<00:00, 1499.54it/s]
 50%|████▉     | 458/919 [00:00<00:00, 1533.26it/s]
 67%|██████▋   | 612/919 [00:00<00:00, 1519.15it/s]
 84%|████████▍ | 770/919 [00:00<00:00, 1539.50it/s]
100%|██████████| 919/919 [00:00<00:00, 1534.18it/s]
2025-12-09:17:04:34 WARNING  [evaluator:309] Overwriting default num_fewshot of piqa from None to 0
2025-12-09:17:04:34 INFO     [api.task:434] Building contexts for piqa on rank 0...

  0%|          | 0/919 [00:00<?, ?it/s]
 16%|█▌        | 145/919 [00:00<00:00, 1435.23it/s]
 32%|███▏      | 298/919 [00:00<00:00, 1488.63it/s]
 49%|████▉     | 451/919 [00:00<00:00, 1503.78it/s]
 66%|██████▌   | 602/919 [00:00<00:00, 1499.55it/s]
 82%|████████▏ | 756/919 [00:00<00:00, 1512.29it/s]
 99%|█████████▉| 909/919 [00:00<00:00, 1517.55it/s]
100%|██████████| 919/919 [00:00<00:00, 1506.26it/s]
n136-128-154:2223501:2223501 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2223501:2223501 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2223501:2223501 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:2223501:2223501 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2223501:2223501 [0] NCCL INFO Comm config Blocking set to 1
n136-128-154:2223502:2223502 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:2223502:2223502 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2223502:2223502 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2223502:2223502 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2223502:2223502 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:2223502:2223673 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2223502:2223673 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2223501:2223672 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2223501:2223672 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2223502:2223673 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2223502:2223673 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2223502:2223673 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2223501:2223672 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2223501:2223672 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2223501:2223672 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2223502:2223673 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2223502:2223673 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2223502:2223673 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:2223502:2223673 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2223502:2223673 [1] NCCL INFO Using network IB
n136-128-154:2223501:2223672 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2223501:2223672 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2223501:2223672 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:2223501:2223672 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2223501:2223672 [0] NCCL INFO Using network IB
n136-128-154:2223502:2223673 [1] NCCL INFO ncclCommInitRankConfig comm 0xd992cd0 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0x38c55482cacbc41c - Init START
n136-128-154:2223501:2223672 [0] NCCL INFO ncclCommInitRankConfig comm 0x1530f880 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0x38c55482cacbc41c - Init START
n136-128-154:2223502:2223673 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2223501:2223672 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2223502:2223673 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2223502:2223673 [1] NCCL INFO Retrieving state for IB
n136-128-154:2223502:2223673 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:2223502:2223673 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2223502:2223673 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2223502:2223673 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2223502:2223673 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2223501:2223672 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2223501:2223672 [0] NCCL INFO Retrieving state for IB
n136-128-154:2223501:2223672 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:2223501:2223672 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2223501:2223672 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2223501:2223672 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2223501:2223672 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2223501:2223672 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2223501:2223672 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2223502:2223673 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2223502:2223673 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2223501:2223672 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2223501:2223672 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2223501:2223672 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2223501:2223672 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2223501:2223672 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2223501:2223672 [0] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2223501:2223672 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2223501:2223672 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2223501:2223672 [0] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2223501:2223672 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2223501:2223672 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2223501:2223672 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2223501:2223672 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2223501:2223672 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2223501:2223672 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2223501:2223672 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2223501:2223672 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2223501:2223672 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2223501:2223672 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2223501:2223672 [0] NCCL INFO ==========================================
n136-128-154:2223501:2223672 [0] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2223501:2223672 [0] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2223501:2223672 [0] NCCL INFO Setting affinity for GPU 6 to 33-62,97-126
n136-128-154:2223502:2223673 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2223502:2223673 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2223502:2223673 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2223502:2223673 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2223502:2223673 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2223502:2223673 [1] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2223502:2223673 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2223502:2223673 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2223502:2223673 [1] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2223502:2223673 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2223502:2223673 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2223502:2223673 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2223502:2223673 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2223502:2223673 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2223502:2223673 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2223502:2223673 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2223502:2223673 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2223502:2223673 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2223502:2223673 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2223502:2223673 [1] NCCL INFO ==========================================
n136-128-154:2223502:2223673 [1] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2223502:2223673 [1] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2223502:2223673 [1] NCCL INFO Setting affinity for GPU 7 to 33-62,97-126
n136-128-154:2223501:2223672 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2223501:2223672 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223501:2223672 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223501:2223672 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223501:2223672 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223501:2223672 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223501:2223672 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223501:2223672 [0] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223501:2223672 [0] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223502:2223673 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2223501:2223672 [0] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223502:2223673 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223501:2223672 [0] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223502:2223673 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223501:2223672 [0] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223502:2223673 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223501:2223672 [0] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223502:2223673 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223502:2223673 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223502:2223673 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223502:2223673 [1] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223502:2223673 [1] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223502:2223673 [1] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223502:2223673 [1] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223502:2223673 [1] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223502:2223673 [1] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223501:2223672 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2223502:2223673 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2223501:2223672 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223502:2223673 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223501:2223672 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223502:2223673 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223501:2223672 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223502:2223673 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223501:2223672 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223502:2223673 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223501:2223672 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223502:2223673 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223501:2223672 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223502:2223673 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223501:2223672 [0] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2223502:2223673 [1] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2223501:2223672 [0] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2223502:2223673 [1] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2223501:2223672 [0] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2223502:2223673 [1] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2223501:2223672 [0] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2223502:2223673 [1] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2223501:2223672 [0] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2223502:2223673 [1] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2223501:2223672 [0] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2223502:2223673 [1] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2223502:2223673 [1] NCCL INFO comm 0xd992cd0 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:2223502:2223673 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:2223501:2223672 [0] NCCL INFO comm 0x1530f880 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:2223502:2223673 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:2223502:2223673 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:2223502:2223673 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:2223502:2223673 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:2223501:2223672 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:2223502:2223673 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:2223501:2223672 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:2223501:2223672 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:2223501:2223672 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:2223501:2223672 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:2223501:2223672 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:2223501:2223672 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:2223501:2223672 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:2223501:2223672 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:2223501:2223672 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:2223501:2223672 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:2223501:2223672 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:2223501:2223672 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:2223501:2223672 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:2223501:2223672 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:2223501:2223672 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:2223501:2223672 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:2223501:2223672 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:2223501:2223672 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:2223501:2223672 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:2223501:2223672 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:2223501:2223672 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:2223502:2223673 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2223501:2223672 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:2223501:2223672 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:2223501:2223672 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:2223501:2223672 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:2223501:2223672 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:2223501:2223672 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:2223501:2223672 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:2223501:2223672 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:2223501:2223672 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:2223501:2223672 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:2223501:2223672 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:2223501:2223672 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:2223501:2223672 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:2223501:2223672 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:2223501:2223672 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:2223501:2223672 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:2223501:2223672 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:2223501:2223672 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:2223501:2223672 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:2223501:2223672 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:2223501:2223672 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:2223501:2223672 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:2223501:2223672 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:2223501:2223672 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:2223501:2223672 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:2223501:2223672 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:2223501:2223672 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:2223501:2223672 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:2223501:2223672 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:2223501:2223672 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:2223501:2223672 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:2223501:2223672 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:2223501:2223672 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:2223501:2223672 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:2223501:2223672 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:2223501:2223672 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:2223501:2223672 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:2223501:2223672 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:2223501:2223672 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:2223501:2223672 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:2223501:2223672 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:2223501:2223672 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:2223501:2223672 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:2223501:2223672 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:2223501:2223672 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:2223501:2223672 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:2223501:2223672 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:2223501:2223672 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:2223501:2223672 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:2223501:2223672 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:2223501:2223672 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:2223501:2223672 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2223502:2223673 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2223502:2223685 [1] NCCL INFO [Proxy Service] Device 1 CPU core 122
n136-128-154:2223502:2223686 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 123
n136-128-154:2223501:2223672 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2223501:2223672 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:2223501:2223687 [0] NCCL INFO [Proxy Service] Device 0 CPU core 100
n136-128-154:2223501:2223688 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 102
n136-128-154:2223502:2223673 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2223502:2223673 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2223501:2223672 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2223501:2223672 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2223501:2223672 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:2223502:2223673 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2223502:2223673 [1] NCCL INFO ncclCommInitRankConfig comm 0xd992cd0 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0x38c55482cacbc41c - Init COMPLETE
n136-128-154:2223502:2223673 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.47 (kernels 0.12, alloc 0.21, bootstrap 0.00, allgathers 0.01, topo 0.07, graphs 0.00, connections 0.02, rest 0.03)
n136-128-154:2223501:2223672 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2223501:2223672 [0] NCCL INFO ncclCommInitRankConfig comm 0x1530f880 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0x38c55482cacbc41c - Init COMPLETE
n136-128-154:2223501:2223672 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.48 (kernels 0.13, alloc 0.22, bootstrap 0.00, allgathers 0.01, topo 0.07, graphs 0.00, connections 0.03, rest 0.03)
n136-128-154:2223502:2223689 [1] NCCL INFO Channel 00/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223502:2223689 [1] NCCL INFO Channel 01/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223502:2223689 [1] NCCL INFO Channel 02/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223502:2223689 [1] NCCL INFO Channel 03/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223502:2223689 [1] NCCL INFO Channel 04/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223501:2223690 [0] NCCL INFO Channel 00/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223502:2223689 [1] NCCL INFO Channel 05/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223501:2223690 [0] NCCL INFO Channel 01/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223502:2223689 [1] NCCL INFO Channel 06/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223501:2223690 [0] NCCL INFO Channel 02/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223502:2223689 [1] NCCL INFO Channel 07/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223501:2223690 [0] NCCL INFO Channel 03/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223502:2223689 [1] NCCL INFO Channel 08/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223501:2223690 [0] NCCL INFO Channel 04/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223502:2223689 [1] NCCL INFO Channel 09/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223501:2223690 [0] NCCL INFO Channel 05/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223502:2223689 [1] NCCL INFO Channel 10/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223501:2223690 [0] NCCL INFO Channel 06/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223502:2223689 [1] NCCL INFO Channel 11/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223501:2223690 [0] NCCL INFO Channel 07/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223502:2223689 [1] NCCL INFO Channel 12/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223501:2223690 [0] NCCL INFO Channel 08/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223502:2223689 [1] NCCL INFO Channel 13/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223501:2223690 [0] NCCL INFO Channel 09/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223502:2223689 [1] NCCL INFO Channel 14/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223501:2223690 [0] NCCL INFO Channel 10/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223502:2223689 [1] NCCL INFO Channel 15/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223501:2223690 [0] NCCL INFO Channel 11/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223502:2223689 [1] NCCL INFO Channel 16/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223501:2223690 [0] NCCL INFO Channel 12/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223502:2223689 [1] NCCL INFO Channel 17/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223501:2223690 [0] NCCL INFO Channel 13/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223502:2223689 [1] NCCL INFO Channel 18/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223501:2223690 [0] NCCL INFO Channel 14/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223502:2223689 [1] NCCL INFO Channel 19/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223501:2223690 [0] NCCL INFO Channel 15/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223502:2223689 [1] NCCL INFO Channel 20/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223501:2223690 [0] NCCL INFO Channel 16/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223502:2223689 [1] NCCL INFO Channel 21/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223501:2223690 [0] NCCL INFO Channel 17/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223502:2223689 [1] NCCL INFO Channel 22/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223501:2223690 [0] NCCL INFO Channel 18/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223502:2223689 [1] NCCL INFO Channel 23/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223501:2223690 [0] NCCL INFO Channel 19/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223501:2223690 [0] NCCL INFO Channel 20/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223501:2223690 [0] NCCL INFO Channel 21/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223501:2223690 [0] NCCL INFO Channel 22/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223501:2223690 [0] NCCL INFO Channel 23/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223502:2223689 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:2223501:2223690 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-09:17:04:36 INFO     [evaluator:559] Running loglikelihood requests
2025-12-09:17:04:36 INFO     [evaluator:559] Running loglikelihood requests

Running loglikelihood requests:   0%|          | 0/1838 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 64
Determined largest batch size: 64

Running loglikelihood requests:   0%|          | 1/1838 [00:08<4:12:49,  8.26s/it]
Running loglikelihood requests:   4%|▎         | 65/1838 [00:08<02:58,  9.96it/s] 
Running loglikelihood requests:   7%|▋         | 129/1838 [00:09<01:21, 21.05it/s]
Running loglikelihood requests:  11%|█         | 194/1838 [00:10<00:48, 33.60it/s]
Running loglikelihood requests:  14%|█▍        | 258/1838 [00:10<00:33, 46.88it/s]
Running loglikelihood requests:  18%|█▊        | 322/1838 [00:11<00:24, 60.66it/s]
Running loglikelihood requests:  21%|██        | 386/1838 [00:11<00:19, 74.07it/s]
Running loglikelihood requests:  25%|██▍       | 451/1838 [00:12<00:15, 86.94it/s]
Running loglikelihood requests:  28%|██▊       | 515/1838 [00:12<00:13, 98.39it/s]
Running loglikelihood requests:  32%|███▏      | 579/1838 [00:13<00:11, 108.44it/s]
Running loglikelihood requests:  35%|███▍      | 643/1838 [00:13<00:10, 117.34it/s]
Running loglikelihood requests:  38%|███▊      | 707/1838 [00:14<00:09, 124.70it/s]
Running loglikelihood requests:  42%|████▏     | 771/1838 [00:14<00:08, 130.98it/s]
Running loglikelihood requests:  45%|████▌     | 835/1838 [00:14<00:07, 136.39it/s]
Running loglikelihood requests:  49%|████▉     | 899/1838 [00:15<00:06, 141.21it/s]
Running loglikelihood requests:  52%|█████▏    | 964/1838 [00:15<00:06, 145.45it/s]
Running loglikelihood requests:  56%|█████▌    | 1028/1838 [00:16<00:05, 149.51it/s]
Running loglikelihood requests:  59%|█████▉    | 1092/1838 [00:16<00:04, 152.53it/s]
Running loglikelihood requests:  63%|██████▎   | 1156/1838 [00:16<00:04, 154.30it/s]
Running loglikelihood requests:  66%|██████▋   | 1221/1838 [00:17<00:03, 157.78it/s]
Running loglikelihood requests:  70%|██████▉   | 1286/1838 [00:17<00:03, 161.25it/s]
Running loglikelihood requests:  73%|███████▎  | 1350/1838 [00:18<00:02, 164.25it/s]
Running loglikelihood requests:  77%|███████▋  | 1414/1838 [00:18<00:02, 165.96it/s]
Running loglikelihood requests:  80%|████████  | 1478/1838 [00:18<00:02, 169.10it/s]
Running loglikelihood requests:  84%|████████▍ | 1542/1838 [00:19<00:01, 171.48it/s]
Running loglikelihood requests:  87%|████████▋ | 1606/1838 [00:19<00:01, 174.56it/s]
Running loglikelihood requests:  91%|█████████ | 1670/1838 [00:19<00:00, 177.62it/s]
Running loglikelihood requests:  94%|█████████▍| 1734/1838 [00:20<00:00, 181.66it/s]
Running loglikelihood requests:  98%|█████████▊| 1799/1838 [00:20<00:00, 203.22it/s]
Running loglikelihood requests: 100%|██████████| 1838/1838 [00:20<00:00, 90.02it/s] 
n136-128-154:2223502:2223711 [1] NCCL INFO Channel 00/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223502:2223711 [1] NCCL INFO Channel 01/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223502:2223711 [1] NCCL INFO Channel 02/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223502:2223711 [1] NCCL INFO Channel 03/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223502:2223711 [1] NCCL INFO Channel 04/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223502:2223711 [1] NCCL INFO Channel 05/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223502:2223711 [1] NCCL INFO Channel 06/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223502:2223711 [1] NCCL INFO Channel 07/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223502:2223711 [1] NCCL INFO Channel 08/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223502:2223711 [1] NCCL INFO Channel 09/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223502:2223711 [1] NCCL INFO Channel 10/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223502:2223711 [1] NCCL INFO Channel 11/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223502:2223711 [1] NCCL INFO Channel 12/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223502:2223711 [1] NCCL INFO Channel 13/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223502:2223711 [1] NCCL INFO Channel 14/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223502:2223711 [1] NCCL INFO Channel 15/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223502:2223711 [1] NCCL INFO Channel 16/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223502:2223711 [1] NCCL INFO Channel 17/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223502:2223711 [1] NCCL INFO Channel 18/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223502:2223711 [1] NCCL INFO Channel 19/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223502:2223711 [1] NCCL INFO Channel 20/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223502:2223711 [1] NCCL INFO Channel 21/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223502:2223711 [1] NCCL INFO Channel 22/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223502:2223711 [1] NCCL INFO Channel 23/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223502:2223711 [1] NCCL INFO Channel 24/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223502:2223711 [1] NCCL INFO Channel 25/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223502:2223711 [1] NCCL INFO Channel 26/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223502:2223711 [1] NCCL INFO Channel 27/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223502:2223711 [1] NCCL INFO Channel 28/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223502:2223711 [1] NCCL INFO Channel 29/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223502:2223711 [1] NCCL INFO Channel 30/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223502:2223711 [1] NCCL INFO Channel 31/1 : 1[7] -> 0[6] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:2223502:2223717 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2223502:2223717 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2223502:2223717 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2223502:2223717 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2223502:2223717 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2223502:2223717 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2223502:2223685 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-09:17:05:06 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: auto (64)
|Tasks|Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|-----|------:|------|-----:|--------|---|-----:|---|-----:|
|piqa |      1|none  |     0|acc     |↑  |0.6795|±  |0.0109|
|     |       |none  |     0|acc_norm|↑  |0.6866|±  |0.0108|

[rank0]:[W1209 17:05:07.286703568 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:2223501:2223767 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2223501:2223767 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2223501:2223767 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2223501:2223687 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2223501:2223767 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2223501:2223767 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2223501:2223767 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2223502:2223685 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2223502:2223717 [1] NCCL INFO comm 0xd992cd0 rank 1 nranks 2 cudaDev 1 busId c9000 - Abort COMPLETE
n136-128-154:2223501:2223767 [0] NCCL INFO comm 0x1530f880 rank 0 nranks 2 cudaDev 0 busId c5000 - Abort COMPLETE
任务 piqa 评估完成！

==================================================
开始评估：任务=hellaswag | 少样本数=10 | 模型=Llama-3.1-8B-quantization-fg
输出路径：results2/Llama-3.1-8B-quantization-fg/baseline_BI/hellaswag.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-09:17:05:57 INFO     [__main__:440] Selected Tasks: ['hellaswag']
2025-12-09:17:05:57 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:17:05:57 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-09:17:05:57 INFO     [__main__:440] Selected Tasks: ['hellaswag']
2025-12-09:17:05:57 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:17:05:57 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-09:17:05:58 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:17:05:58 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:17:05:59 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.13it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.00it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.08it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.00it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.09it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.53it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.33it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.02it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.45it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.25it/s]
2025-12-09:17:06:18 WARNING  [evaluator:309] Overwriting default num_fewshot of hellaswag from None to 10
2025-12-09:17:06:18 INFO     [api.task:434] Building contexts for hellaswag on rank 1...
2025-12-09:17:06:19 WARNING  [evaluator:309] Overwriting default num_fewshot of hellaswag from None to 10
2025-12-09:17:06:19 INFO     [api.task:434] Building contexts for hellaswag on rank 0...

  0%|          | 0/5021 [00:00<?, ?it/s]
  0%|          | 18/5021 [00:00<00:29, 172.00it/s]
  1%|          | 38/5021 [00:00<00:26, 185.73it/s]
  1%|          | 58/5021 [00:00<00:25, 191.52it/s]
  2%|▏         | 78/5021 [00:00<00:25, 192.64it/s]
  2%|▏         | 98/5021 [00:00<00:25, 193.38it/s]
  0%|          | 0/5021 [00:00<?, ?it/s]
  2%|▏         | 118/5021 [00:00<00:25, 192.05it/s]
  0%|          | 19/5021 [00:00<00:27, 184.74it/s]
  3%|▎         | 138/5021 [00:00<00:25, 193.19it/s]
  1%|          | 39/5021 [00:00<00:26, 189.95it/s]
  3%|▎         | 158/5021 [00:00<00:25, 194.03it/s]
  1%|          | 59/5021 [00:00<00:25, 193.02it/s]
  4%|▎         | 178/5021 [00:00<00:24, 195.01it/s]
  2%|▏         | 79/5021 [00:00<00:25, 195.26it/s]
  4%|▍         | 198/5021 [00:01<00:24, 195.65it/s]
  2%|▏         | 99/5021 [00:00<00:25, 194.46it/s]
  4%|▍         | 218/5021 [00:01<00:24, 192.79it/s]
  2%|▏         | 119/5021 [00:00<00:25, 195.74it/s]
  5%|▍         | 238/5021 [00:01<00:24, 193.64it/s]
  3%|▎         | 139/5021 [00:00<00:24, 196.04it/s]
  5%|▌         | 258/5021 [00:01<00:24, 194.40it/s]
  3%|▎         | 159/5021 [00:00<00:24, 196.26it/s]
  6%|▌         | 278/5021 [00:01<00:24, 194.89it/s]
  4%|▎         | 179/5021 [00:00<00:24, 196.37it/s]
  6%|▌         | 298/5021 [00:01<00:24, 195.06it/s]
  4%|▍         | 199/5021 [00:01<00:24, 193.24it/s]
  6%|▋         | 318/5021 [00:01<00:24, 192.69it/s]
  4%|▍         | 219/5021 [00:01<00:24, 193.24it/s]
  7%|▋         | 338/5021 [00:01<00:24, 193.69it/s]
  5%|▍         | 239/5021 [00:01<00:24, 193.44it/s]
  7%|▋         | 358/5021 [00:01<00:23, 194.61it/s]
  5%|▌         | 259/5021 [00:01<00:24, 193.76it/s]
  8%|▊         | 378/5021 [00:01<00:23, 195.19it/s]
  8%|▊         | 398/5021 [00:02<00:23, 195.89it/s]
  6%|▌         | 279/5021 [00:01<00:25, 189.33it/s]
  8%|▊         | 418/5021 [00:02<00:23, 196.47it/s]
  6%|▌         | 298/5021 [00:01<00:25, 187.17it/s]
  9%|▊         | 438/5021 [00:02<00:23, 196.22it/s]
  6%|▋         | 318/5021 [00:01<00:24, 188.81it/s]
  9%|▉         | 458/5021 [00:02<00:23, 196.40it/s]
  7%|▋         | 337/5021 [00:01<00:24, 189.13it/s]
 10%|▉         | 478/5021 [00:02<00:23, 196.04it/s]
  7%|▋         | 356/5021 [00:01<00:24, 188.54it/s]
 10%|▉         | 498/5021 [00:02<00:23, 196.12it/s]
  7%|▋         | 376/5021 [00:01<00:24, 190.57it/s]
 10%|█         | 518/5021 [00:02<00:22, 195.94it/s]
  8%|▊         | 396/5021 [00:02<00:24, 191.26it/s]
 11%|█         | 538/5021 [00:02<00:22, 195.76it/s]
  8%|▊         | 416/5021 [00:02<00:24, 190.56it/s]
 11%|█         | 558/5021 [00:02<00:22, 195.25it/s]
  9%|▊         | 436/5021 [00:02<00:23, 191.25it/s]
 12%|█▏        | 578/5021 [00:02<00:22, 194.97it/s]
  9%|▉         | 456/5021 [00:02<00:23, 191.62it/s]
 12%|█▏        | 598/5021 [00:03<00:22, 194.25it/s]
  9%|▉         | 476/5021 [00:02<00:23, 191.82it/s]
 12%|█▏        | 618/5021 [00:03<00:22, 193.79it/s]
 10%|▉         | 496/5021 [00:02<00:23, 192.14it/s]
 13%|█▎        | 638/5021 [00:03<00:22, 193.76it/s]
 10%|█         | 516/5021 [00:02<00:23, 190.65it/s]
 13%|█▎        | 658/5021 [00:03<00:22, 193.64it/s]
 11%|█         | 536/5021 [00:02<00:23, 189.66it/s]
 14%|█▎        | 678/5021 [00:03<00:22, 193.74it/s]
 11%|█         | 555/5021 [00:02<00:23, 189.54it/s]
 14%|█▍        | 698/5021 [00:03<00:22, 193.78it/s]
 11%|█▏        | 574/5021 [00:02<00:23, 189.66it/s]
 14%|█▍        | 718/5021 [00:03<00:22, 193.73it/s]
 12%|█▏        | 593/5021 [00:03<00:23, 189.64it/s]
 15%|█▍        | 738/5021 [00:03<00:22, 194.68it/s]
 12%|█▏        | 613/5021 [00:03<00:23, 190.89it/s]
 15%|█▌        | 758/5021 [00:03<00:21, 195.49it/s]
 13%|█▎        | 633/5021 [00:03<00:23, 189.84it/s]
 15%|█▌        | 778/5021 [00:04<00:21, 196.42it/s]
 13%|█▎        | 653/5021 [00:03<00:22, 191.10it/s]
 16%|█▌        | 798/5021 [00:04<00:21, 197.31it/s]
 13%|█▎        | 673/5021 [00:03<00:22, 191.67it/s]
 16%|█▋        | 818/5021 [00:04<00:21, 197.60it/s]
 14%|█▍        | 693/5021 [00:03<00:22, 191.70it/s]
 17%|█▋        | 838/5021 [00:04<00:21, 197.16it/s]
 14%|█▍        | 713/5021 [00:03<00:22, 192.17it/s]
 17%|█▋        | 858/5021 [00:04<00:21, 197.90it/s]
 15%|█▍        | 733/5021 [00:03<00:22, 191.62it/s]
 17%|█▋        | 878/5021 [00:04<00:20, 198.41it/s]
 15%|█▍        | 753/5021 [00:03<00:22, 192.16it/s]
 18%|█▊        | 898/5021 [00:04<00:20, 198.45it/s]
 15%|█▌        | 773/5021 [00:04<00:22, 192.60it/s]
 18%|█▊        | 918/5021 [00:04<00:20, 198.58it/s]
 16%|█▌        | 793/5021 [00:04<00:22, 191.86it/s]
 19%|█▊        | 938/5021 [00:04<00:20, 198.56it/s]
 16%|█▌        | 813/5021 [00:04<00:21, 191.98it/s]
 19%|█▉        | 958/5021 [00:04<00:20, 198.82it/s]
 17%|█▋        | 833/5021 [00:04<00:21, 192.82it/s]
 19%|█▉        | 979/5021 [00:05<00:20, 199.29it/s]
 17%|█▋        | 853/5021 [00:04<00:21, 193.24it/s]
 20%|█▉        | 999/5021 [00:05<00:20, 199.28it/s]
 17%|█▋        | 873/5021 [00:04<00:21, 193.53it/s]
 20%|██        | 1019/5021 [00:05<00:20, 198.99it/s]
 18%|█▊        | 893/5021 [00:04<00:21, 193.91it/s]
 21%|██        | 1040/5021 [00:05<00:19, 199.28it/s]
 18%|█▊        | 913/5021 [00:04<00:21, 192.54it/s]
 21%|██        | 1060/5021 [00:05<00:19, 199.39it/s]
 19%|█▊        | 933/5021 [00:04<00:21, 192.35it/s]
 22%|██▏       | 1080/5021 [00:05<00:19, 199.34it/s]
 19%|█▉        | 953/5021 [00:04<00:21, 192.80it/s]
 22%|██▏       | 1101/5021 [00:05<00:19, 199.98it/s]
 19%|█▉        | 973/5021 [00:05<00:20, 193.57it/s]
 22%|██▏       | 1122/5021 [00:05<00:19, 200.68it/s]
 20%|█▉        | 993/5021 [00:05<00:20, 193.65it/s]
 23%|██▎       | 1143/5021 [00:05<00:19, 201.31it/s]
 20%|██        | 1013/5021 [00:05<00:20, 193.64it/s]
 23%|██▎       | 1164/5021 [00:05<00:19, 201.91it/s]
 21%|██        | 1033/5021 [00:05<00:20, 193.65it/s]
 24%|██▎       | 1185/5021 [00:06<00:18, 202.53it/s]
 21%|██        | 1053/5021 [00:05<00:20, 191.61it/s]
 24%|██▍       | 1206/5021 [00:06<00:18, 202.53it/s]
 21%|██▏       | 1073/5021 [00:05<00:20, 192.44it/s]
 24%|██▍       | 1227/5021 [00:06<00:18, 202.82it/s]
 22%|██▏       | 1093/5021 [00:05<00:20, 192.80it/s]
 25%|██▍       | 1248/5021 [00:06<00:18, 202.70it/s]
 22%|██▏       | 1113/5021 [00:05<00:20, 193.38it/s]
 25%|██▌       | 1269/5021 [00:06<00:18, 202.54it/s]
 23%|██▎       | 1133/5021 [00:05<00:20, 193.38it/s]
 26%|██▌       | 1290/5021 [00:06<00:18, 201.65it/s]
 23%|██▎       | 1153/5021 [00:06<00:19, 193.53it/s]
 26%|██▌       | 1311/5021 [00:06<00:18, 201.24it/s]
 23%|██▎       | 1173/5021 [00:06<00:19, 193.70it/s]
 27%|██▋       | 1332/5021 [00:06<00:18, 201.09it/s]
 24%|██▍       | 1193/5021 [00:06<00:19, 194.06it/s]
 27%|██▋       | 1353/5021 [00:06<00:18, 200.64it/s]
 24%|██▍       | 1213/5021 [00:06<00:19, 194.34it/s]
 27%|██▋       | 1374/5021 [00:06<00:18, 200.65it/s]
 25%|██▍       | 1233/5021 [00:06<00:19, 193.61it/s]
 28%|██▊       | 1395/5021 [00:07<00:18, 200.73it/s]
 25%|██▍       | 1253/5021 [00:06<00:19, 194.09it/s]
 28%|██▊       | 1416/5021 [00:07<00:17, 200.92it/s]
 25%|██▌       | 1273/5021 [00:06<00:19, 194.75it/s]
 29%|██▊       | 1437/5021 [00:07<00:17, 200.76it/s]
 26%|██▌       | 1293/5021 [00:06<00:19, 195.17it/s]
 29%|██▉       | 1458/5021 [00:07<00:17, 201.24it/s]
 26%|██▌       | 1313/5021 [00:06<00:18, 195.71it/s]
 29%|██▉       | 1479/5021 [00:07<00:17, 201.40it/s]
 27%|██▋       | 1333/5021 [00:06<00:18, 195.82it/s]
 30%|██▉       | 1500/5021 [00:07<00:17, 201.38it/s]
 27%|██▋       | 1353/5021 [00:07<00:18, 196.14it/s]
 30%|███       | 1521/5021 [00:07<00:17, 201.61it/s]
 27%|██▋       | 1373/5021 [00:07<00:18, 196.44it/s]
 31%|███       | 1542/5021 [00:07<00:17, 201.87it/s]
 28%|██▊       | 1393/5021 [00:07<00:18, 197.03it/s]
 31%|███       | 1563/5021 [00:07<00:17, 202.17it/s]
 28%|██▊       | 1413/5021 [00:07<00:18, 194.97it/s]
 32%|███▏      | 1584/5021 [00:08<00:17, 201.99it/s]
 29%|██▊       | 1433/5021 [00:07<00:18, 194.80it/s]
 32%|███▏      | 1605/5021 [00:08<00:16, 201.82it/s]
 29%|██▉       | 1453/5021 [00:07<00:18, 195.17it/s]
 32%|███▏      | 1626/5021 [00:08<00:16, 200.43it/s]
 29%|██▉       | 1473/5021 [00:07<00:18, 195.31it/s]
 33%|███▎      | 1647/5021 [00:08<00:16, 201.03it/s]
 30%|██▉       | 1493/5021 [00:07<00:18, 195.65it/s]
 33%|███▎      | 1668/5021 [00:08<00:16, 200.64it/s]
 30%|███       | 1513/5021 [00:07<00:17, 195.40it/s]
 34%|███▎      | 1689/5021 [00:08<00:16, 200.50it/s]
 31%|███       | 1533/5021 [00:07<00:17, 194.80it/s]
 34%|███▍      | 1710/5021 [00:08<00:16, 200.56it/s]
 31%|███       | 1553/5021 [00:08<00:17, 194.63it/s]
 34%|███▍      | 1731/5021 [00:08<00:16, 200.22it/s]
 31%|███▏      | 1573/5021 [00:08<00:17, 194.78it/s]
 35%|███▍      | 1752/5021 [00:08<00:16, 197.74it/s]
 32%|███▏      | 1593/5021 [00:08<00:17, 192.79it/s]
 35%|███▌      | 1772/5021 [00:08<00:16, 197.53it/s]
 32%|███▏      | 1613/5021 [00:08<00:17, 193.42it/s]
 36%|███▌      | 1792/5021 [00:09<00:16, 197.29it/s]
 33%|███▎      | 1633/5021 [00:08<00:17, 193.85it/s]
 36%|███▌      | 1812/5021 [00:09<00:16, 197.35it/s]
 33%|███▎      | 1653/5021 [00:08<00:17, 194.32it/s]
 36%|███▋      | 1832/5021 [00:09<00:16, 196.56it/s]
 33%|███▎      | 1673/5021 [00:08<00:17, 194.49it/s]
 37%|███▋      | 1852/5021 [00:09<00:16, 196.20it/s]
 34%|███▎      | 1693/5021 [00:08<00:17, 194.44it/s]
 37%|███▋      | 1872/5021 [00:09<00:16, 193.98it/s]
 34%|███▍      | 1713/5021 [00:08<00:17, 194.47it/s]
 38%|███▊      | 1892/5021 [00:09<00:16, 195.16it/s]
 35%|███▍      | 1733/5021 [00:08<00:16, 194.78it/s]
 38%|███▊      | 1912/5021 [00:09<00:15, 195.75it/s]
 35%|███▍      | 1753/5021 [00:09<00:16, 195.29it/s]
 38%|███▊      | 1932/5021 [00:09<00:15, 196.22it/s]
 35%|███▌      | 1773/5021 [00:09<00:16, 195.59it/s]
 39%|███▉      | 1952/5021 [00:09<00:15, 196.87it/s]
 36%|███▌      | 1793/5021 [00:09<00:16, 196.10it/s]
 39%|███▉      | 1972/5021 [00:09<00:15, 197.51it/s]
 36%|███▌      | 1813/5021 [00:09<00:16, 194.91it/s]
 40%|███▉      | 1992/5021 [00:10<00:15, 198.03it/s]
 37%|███▋      | 1833/5021 [00:09<00:16, 195.42it/s]
 40%|████      | 2012/5021 [00:10<00:15, 198.31it/s]
 37%|███▋      | 1853/5021 [00:09<00:16, 195.75it/s]
 40%|████      | 2032/5021 [00:10<00:15, 196.93it/s]
 37%|███▋      | 1873/5021 [00:09<00:16, 196.33it/s]
 41%|████      | 2052/5021 [00:10<00:15, 197.55it/s]
 38%|███▊      | 1893/5021 [00:09<00:15, 196.74it/s]
 41%|████▏     | 2072/5021 [00:10<00:14, 198.20it/s]
 38%|███▊      | 1913/5021 [00:09<00:15, 196.69it/s]
 42%|████▏     | 2093/5021 [00:10<00:14, 198.93it/s]
 38%|███▊      | 1933/5021 [00:09<00:15, 196.33it/s]
 42%|████▏     | 2113/5021 [00:10<00:14, 199.10it/s]
 39%|███▉      | 1953/5021 [00:10<00:15, 196.21it/s]
 42%|████▏     | 2133/5021 [00:10<00:14, 199.27it/s]
 39%|███▉      | 1973/5021 [00:10<00:15, 196.17it/s]
 43%|████▎     | 2154/5021 [00:10<00:14, 199.62it/s]
 40%|███▉      | 1993/5021 [00:10<00:15, 196.12it/s]
 43%|████▎     | 2175/5021 [00:11<00:14, 200.44it/s]
 40%|████      | 2013/5021 [00:10<00:15, 194.40it/s]
 44%|████▎     | 2196/5021 [00:11<00:14, 198.66it/s]
 40%|████      | 2033/5021 [00:10<00:15, 193.58it/s]
 44%|████▍     | 2217/5021 [00:11<00:14, 199.27it/s]
 41%|████      | 2053/5021 [00:10<00:15, 193.34it/s]
 45%|████▍     | 2237/5021 [00:11<00:13, 199.29it/s]
 41%|████▏     | 2073/5021 [00:10<00:15, 193.19it/s]
 45%|████▍     | 2257/5021 [00:11<00:13, 199.21it/s]
 42%|████▏     | 2093/5021 [00:10<00:15, 193.21it/s]
 45%|████▌     | 2277/5021 [00:11<00:13, 199.40it/s]
 42%|████▏     | 2113/5021 [00:10<00:15, 193.08it/s]
 46%|████▌     | 2298/5021 [00:11<00:13, 199.74it/s]
 42%|████▏     | 2133/5021 [00:11<00:14, 193.23it/s]
 46%|████▌     | 2319/5021 [00:11<00:13, 200.11it/s]
 43%|████▎     | 2153/5021 [00:11<00:14, 193.84it/s]
 47%|████▋     | 2340/5021 [00:11<00:13, 197.41it/s]
 43%|████▎     | 2173/5021 [00:11<00:14, 194.59it/s]
 47%|████▋     | 2361/5021 [00:11<00:13, 198.36it/s]
 44%|████▎     | 2193/5021 [00:11<00:14, 194.90it/s]
 47%|████▋     | 2381/5021 [00:12<00:13, 198.52it/s]
 44%|████▍     | 2213/5021 [00:11<00:14, 195.03it/s]
 48%|████▊     | 2402/5021 [00:12<00:13, 199.10it/s]
 44%|████▍     | 2233/5021 [00:11<00:14, 192.64it/s]
 48%|████▊     | 2423/5021 [00:12<00:13, 199.45it/s]
 45%|████▍     | 2253/5021 [00:11<00:14, 193.26it/s]
 49%|████▊     | 2444/5021 [00:12<00:12, 199.88it/s]
 45%|████▌     | 2273/5021 [00:11<00:14, 193.82it/s]
 49%|████▉     | 2465/5021 [00:12<00:12, 200.49it/s]
 46%|████▌     | 2293/5021 [00:11<00:14, 194.35it/s]
 50%|████▉     | 2486/5021 [00:12<00:12, 200.74it/s]
 46%|████▌     | 2313/5021 [00:11<00:13, 194.32it/s]
 50%|████▉     | 2507/5021 [00:12<00:12, 200.46it/s]
 46%|████▋     | 2333/5021 [00:12<00:13, 194.40it/s]
 50%|█████     | 2528/5021 [00:12<00:12, 198.73it/s]
 47%|████▋     | 2353/5021 [00:12<00:13, 194.28it/s]
 51%|█████     | 2549/5021 [00:12<00:12, 199.38it/s]
 47%|████▋     | 2373/5021 [00:12<00:13, 194.14it/s]
 51%|█████     | 2570/5021 [00:12<00:12, 199.91it/s]
 48%|████▊     | 2393/5021 [00:12<00:13, 193.94it/s]
 48%|████▊     | 2413/5021 [00:12<00:13, 193.77it/s]
 52%|█████▏    | 2591/5021 [00:13<00:12, 200.63it/s]
 48%|████▊     | 2433/5021 [00:12<00:13, 194.27it/s]
 52%|█████▏    | 2612/5021 [00:13<00:11, 200.97it/s]
 49%|████▉     | 2453/5021 [00:12<00:13, 194.53it/s]
 52%|█████▏    | 2633/5021 [00:13<00:11, 201.29it/s]
 49%|████▉     | 2473/5021 [00:12<00:13, 194.59it/s]
 53%|█████▎    | 2654/5021 [00:13<00:11, 201.46it/s]
 50%|████▉     | 2493/5021 [00:12<00:12, 194.56it/s]
 53%|█████▎    | 2675/5021 [00:13<00:11, 201.50it/s]
 50%|█████     | 2513/5021 [00:12<00:12, 193.68it/s]
 54%|█████▎    | 2696/5021 [00:13<00:11, 201.40it/s]
 50%|█████     | 2533/5021 [00:13<00:12, 193.93it/s]
 54%|█████▍    | 2717/5021 [00:13<00:11, 198.73it/s]
 51%|█████     | 2553/5021 [00:13<00:12, 194.22it/s]
 55%|█████▍    | 2737/5021 [00:13<00:11, 198.47it/s]
 51%|█████     | 2573/5021 [00:13<00:12, 194.82it/s]
 55%|█████▍    | 2757/5021 [00:13<00:11, 198.62it/s]
 52%|█████▏    | 2593/5021 [00:13<00:12, 195.68it/s]
 55%|█████▌    | 2777/5021 [00:14<00:11, 198.92it/s]
 52%|█████▏    | 2613/5021 [00:13<00:12, 195.92it/s]
 56%|█████▌    | 2797/5021 [00:14<00:11, 198.99it/s]
 52%|█████▏    | 2633/5021 [00:13<00:12, 196.16it/s]
 56%|█████▌    | 2817/5021 [00:14<00:11, 199.08it/s]
 53%|█████▎    | 2653/5021 [00:13<00:12, 196.39it/s]
 57%|█████▋    | 2837/5021 [00:14<00:10, 199.32it/s]
 53%|█████▎    | 2673/5021 [00:13<00:11, 196.70it/s]
 57%|█████▋    | 2857/5021 [00:14<00:10, 198.01it/s]
 54%|█████▎    | 2693/5021 [00:13<00:11, 197.08it/s]
 57%|█████▋    | 2877/5021 [00:14<00:10, 198.54it/s]
 54%|█████▍    | 2713/5021 [00:14<00:11, 197.19it/s]
 58%|█████▊    | 2897/5021 [00:14<00:10, 195.90it/s]
 54%|█████▍    | 2733/5021 [00:14<00:11, 196.62it/s]
 58%|█████▊    | 2917/5021 [00:14<00:10, 196.52it/s]
 55%|█████▍    | 2753/5021 [00:14<00:11, 196.05it/s]
 58%|█████▊    | 2937/5021 [00:14<00:10, 197.43it/s]
 55%|█████▌    | 2773/5021 [00:14<00:11, 193.78it/s]
 59%|█████▉    | 2958/5021 [00:14<00:10, 198.31it/s]
 59%|█████▉    | 2978/5021 [00:15<00:10, 198.59it/s]
 56%|█████▌    | 2793/5021 [00:14<00:11, 193.46it/s]
 60%|█████▉    | 2999/5021 [00:15<00:10, 199.28it/s]
 56%|█████▌    | 2813/5021 [00:14<00:11, 192.74it/s]
 56%|█████▋    | 2833/5021 [00:14<00:11, 192.74it/s]
 60%|██████    | 3020/5021 [00:15<00:10, 199.66it/s]
 57%|█████▋    | 2853/5021 [00:14<00:11, 192.97it/s]
 61%|██████    | 3041/5021 [00:15<00:09, 199.75it/s]
 57%|█████▋    | 2873/5021 [00:14<00:11, 193.06it/s]
 61%|██████    | 3062/5021 [00:15<00:09, 200.21it/s]
 58%|█████▊    | 2893/5021 [00:14<00:11, 193.21it/s]
 61%|██████▏   | 3083/5021 [00:15<00:09, 200.61it/s]
 58%|█████▊    | 2913/5021 [00:15<00:10, 193.42it/s]
 62%|██████▏   | 3104/5021 [00:15<00:09, 200.86it/s]
 58%|█████▊    | 2933/5021 [00:15<00:10, 193.61it/s]
 62%|██████▏   | 3125/5021 [00:15<00:09, 199.58it/s]
 59%|█████▉    | 2953/5021 [00:15<00:10, 194.10it/s]
 63%|██████▎   | 3146/5021 [00:15<00:09, 199.97it/s]
 59%|█████▉    | 2973/5021 [00:15<00:10, 194.33it/s]
 63%|██████▎   | 3167/5021 [00:15<00:09, 200.62it/s]
 60%|█████▉    | 2993/5021 [00:15<00:10, 194.70it/s]
 63%|██████▎   | 3188/5021 [00:16<00:09, 200.85it/s]
 60%|██████    | 3013/5021 [00:15<00:10, 195.06it/s]
 64%|██████▍   | 3209/5021 [00:16<00:08, 201.35it/s]
 60%|██████    | 3033/5021 [00:15<00:10, 195.65it/s]
 64%|██████▍   | 3230/5021 [00:16<00:08, 201.13it/s]
 61%|██████    | 3053/5021 [00:15<00:10, 193.67it/s]
 65%|██████▍   | 3251/5021 [00:16<00:08, 201.38it/s]
 61%|██████    | 3073/5021 [00:15<00:10, 194.61it/s]
 65%|██████▌   | 3272/5021 [00:16<00:08, 201.26it/s]
 62%|██████▏   | 3093/5021 [00:15<00:09, 194.87it/s]
 66%|██████▌   | 3293/5021 [00:16<00:08, 201.38it/s]
 62%|██████▏   | 3113/5021 [00:16<00:09, 195.18it/s]
 66%|██████▌   | 3314/5021 [00:16<00:08, 201.20it/s]
 62%|██████▏   | 3133/5021 [00:16<00:09, 195.61it/s]
 66%|██████▋   | 3335/5021 [00:16<00:08, 199.35it/s]
 63%|██████▎   | 3153/5021 [00:16<00:09, 195.21it/s]
 67%|██████▋   | 3355/5021 [00:16<00:08, 199.24it/s]
 63%|██████▎   | 3173/5021 [00:16<00:09, 194.92it/s]
 67%|██████▋   | 3375/5021 [00:17<00:08, 199.16it/s]
 64%|██████▎   | 3193/5021 [00:16<00:09, 194.67it/s]
 68%|██████▊   | 3395/5021 [00:17<00:08, 198.66it/s]
 64%|██████▍   | 3213/5021 [00:16<00:09, 194.72it/s]
 68%|██████▊   | 3415/5021 [00:17<00:08, 198.50it/s]
 64%|██████▍   | 3233/5021 [00:16<00:09, 194.79it/s]
 68%|██████▊   | 3435/5021 [00:17<00:07, 198.30it/s]
 65%|██████▍   | 3253/5021 [00:16<00:09, 194.66it/s]
 69%|██████▉   | 3455/5021 [00:17<00:07, 198.67it/s]
 65%|██████▌   | 3273/5021 [00:16<00:08, 194.23it/s]
 69%|██████▉   | 3475/5021 [00:17<00:07, 199.00it/s]
 66%|██████▌   | 3293/5021 [00:16<00:08, 194.40it/s]
 70%|██████▉   | 3495/5021 [00:17<00:07, 199.25it/s]
 66%|██████▌   | 3313/5021 [00:17<00:08, 194.47it/s]
 70%|███████   | 3516/5021 [00:17<00:07, 199.89it/s]
 66%|██████▋   | 3333/5021 [00:17<00:08, 194.61it/s]
 70%|███████   | 3537/5021 [00:17<00:07, 200.12it/s]
 67%|██████▋   | 3353/5021 [00:17<00:08, 195.16it/s]
 71%|███████   | 3558/5021 [00:17<00:07, 197.13it/s]
 67%|██████▋   | 3373/5021 [00:17<00:08, 193.98it/s]
 71%|███████▏  | 3579/5021 [00:18<00:07, 198.06it/s]
 68%|██████▊   | 3393/5021 [00:17<00:08, 194.64it/s]
 72%|███████▏  | 3600/5021 [00:18<00:07, 199.08it/s]
 68%|██████▊   | 3413/5021 [00:17<00:08, 195.26it/s]
 72%|███████▏  | 3621/5021 [00:18<00:07, 199.76it/s]
 68%|██████▊   | 3433/5021 [00:17<00:08, 195.59it/s]
 73%|███████▎  | 3642/5021 [00:18<00:06, 200.36it/s]
 69%|██████▉   | 3453/5021 [00:17<00:08, 195.89it/s]
 73%|███████▎  | 3663/5021 [00:18<00:06, 200.51it/s]
 69%|██████▉   | 3473/5021 [00:17<00:07, 196.20it/s]
 73%|███████▎  | 3684/5021 [00:18<00:06, 200.71it/s]
 70%|██████▉   | 3493/5021 [00:18<00:07, 196.65it/s]
 74%|███████▍  | 3705/5021 [00:18<00:06, 201.03it/s]
 70%|██████▉   | 3513/5021 [00:18<00:07, 196.89it/s]
 74%|███████▍  | 3726/5021 [00:18<00:06, 200.87it/s]
 70%|███████   | 3533/5021 [00:18<00:07, 196.82it/s]
 75%|███████▍  | 3747/5021 [00:18<00:06, 200.89it/s]
 71%|███████   | 3553/5021 [00:18<00:07, 196.40it/s]
 75%|███████▌  | 3768/5021 [00:18<00:06, 200.78it/s]
 71%|███████   | 3573/5021 [00:18<00:07, 196.28it/s]
 75%|███████▌  | 3789/5021 [00:19<00:06, 200.39it/s]
 72%|███████▏  | 3593/5021 [00:18<00:07, 196.06it/s]
 76%|███████▌  | 3810/5021 [00:19<00:06, 200.60it/s]
 72%|███████▏  | 3613/5021 [00:18<00:07, 195.93it/s]
 76%|███████▋  | 3831/5021 [00:19<00:05, 201.16it/s]
 72%|███████▏  | 3633/5021 [00:18<00:07, 195.62it/s]
 77%|███████▋  | 3852/5021 [00:19<00:05, 199.74it/s]
 73%|███████▎  | 3653/5021 [00:18<00:06, 195.83it/s]
 77%|███████▋  | 3873/5021 [00:19<00:05, 200.43it/s]
 73%|███████▎  | 3673/5021 [00:18<00:06, 195.86it/s]
 78%|███████▊  | 3894/5021 [00:19<00:05, 199.43it/s]
 74%|███████▎  | 3693/5021 [00:19<00:06, 193.48it/s]
 78%|███████▊  | 3914/5021 [00:19<00:05, 199.48it/s]
 74%|███████▍  | 3713/5021 [00:19<00:06, 192.90it/s]
 78%|███████▊  | 3934/5021 [00:19<00:05, 199.54it/s]
 74%|███████▍  | 3733/5021 [00:19<00:06, 192.38it/s]
 79%|███████▉  | 3955/5021 [00:19<00:05, 199.96it/s]
 75%|███████▍  | 3753/5021 [00:19<00:06, 192.67it/s]
 79%|███████▉  | 3975/5021 [00:20<00:05, 199.96it/s]
 75%|███████▌  | 3773/5021 [00:19<00:06, 192.58it/s]
 80%|███████▉  | 3996/5021 [00:20<00:05, 199.63it/s]
 76%|███████▌  | 3793/5021 [00:19<00:06, 192.63it/s]
 80%|████████  | 4017/5021 [00:20<00:05, 200.43it/s]
 76%|███████▌  | 3813/5021 [00:19<00:06, 193.07it/s]
 80%|████████  | 4038/5021 [00:20<00:04, 200.62it/s]
 76%|███████▋  | 3833/5021 [00:19<00:06, 193.68it/s]
 81%|████████  | 4059/5021 [00:20<00:04, 199.03it/s]
 77%|███████▋  | 3853/5021 [00:19<00:06, 194.01it/s]
 81%|████████▏ | 4080/5021 [00:20<00:04, 199.77it/s]
 77%|███████▋  | 3873/5021 [00:19<00:05, 194.32it/s]
 82%|████████▏ | 4101/5021 [00:20<00:04, 200.34it/s]
 78%|███████▊  | 3893/5021 [00:20<00:05, 194.25it/s]
 82%|████████▏ | 4122/5021 [00:20<00:04, 198.18it/s]
 78%|███████▊  | 3913/5021 [00:20<00:05, 194.64it/s]
 82%|████████▏ | 4142/5021 [00:20<00:04, 197.72it/s]
 78%|███████▊  | 3933/5021 [00:20<00:05, 194.52it/s]
 83%|████████▎ | 4162/5021 [00:20<00:04, 198.01it/s]
 79%|███████▊  | 3953/5021 [00:20<00:05, 193.96it/s]
 83%|████████▎ | 4182/5021 [00:21<00:04, 197.72it/s]
 79%|███████▉  | 3973/5021 [00:20<00:05, 194.19it/s]
 84%|████████▎ | 4202/5021 [00:21<00:04, 198.15it/s]
 80%|███████▉  | 3993/5021 [00:20<00:05, 194.46it/s]
 84%|████████▍ | 4223/5021 [00:21<00:04, 198.98it/s]
 80%|███████▉  | 4013/5021 [00:20<00:05, 192.55it/s]
 85%|████████▍ | 4244/5021 [00:21<00:03, 199.36it/s]
 80%|████████  | 4033/5021 [00:20<00:05, 193.36it/s]
 85%|████████▍ | 4264/5021 [00:21<00:03, 199.43it/s]
 81%|████████  | 4053/5021 [00:20<00:04, 193.96it/s]
 85%|████████▌ | 4284/5021 [00:21<00:03, 199.54it/s]
 81%|████████  | 4073/5021 [00:20<00:04, 194.34it/s]
 86%|████████▌ | 4305/5021 [00:21<00:03, 200.02it/s]
 82%|████████▏ | 4093/5021 [00:21<00:04, 194.56it/s]
 86%|████████▌ | 4326/5021 [00:21<00:03, 200.16it/s]
 82%|████████▏ | 4113/5021 [00:21<00:04, 194.63it/s]
 87%|████████▋ | 4347/5021 [00:21<00:03, 200.27it/s]
 82%|████████▏ | 4133/5021 [00:21<00:04, 194.93it/s]
 87%|████████▋ | 4368/5021 [00:21<00:03, 200.23it/s]
 83%|████████▎ | 4153/5021 [00:21<00:04, 195.05it/s]
 87%|████████▋ | 4389/5021 [00:22<00:03, 197.85it/s]
 83%|████████▎ | 4173/5021 [00:21<00:04, 195.31it/s]
 88%|████████▊ | 4410/5021 [00:22<00:03, 198.62it/s]
 84%|████████▎ | 4193/5021 [00:21<00:04, 195.31it/s]
 88%|████████▊ | 4431/5021 [00:22<00:02, 199.42it/s]
 84%|████████▍ | 4213/5021 [00:21<00:04, 195.75it/s]
 89%|████████▊ | 4452/5021 [00:22<00:02, 199.69it/s]
 84%|████████▍ | 4233/5021 [00:21<00:04, 196.09it/s]
 89%|████████▉ | 4472/5021 [00:22<00:02, 195.55it/s]
 85%|████████▍ | 4253/5021 [00:21<00:03, 196.00it/s]
 89%|████████▉ | 4492/5021 [00:22<00:02, 196.06it/s]
 85%|████████▌ | 4273/5021 [00:22<00:03, 196.46it/s]
 90%|████████▉ | 4512/5021 [00:22<00:02, 196.10it/s]
 86%|████████▌ | 4293/5021 [00:22<00:03, 196.31it/s]
 90%|█████████ | 4532/5021 [00:22<00:02, 197.14it/s]
 86%|████████▌ | 4313/5021 [00:22<00:03, 196.45it/s]
 91%|█████████ | 4552/5021 [00:22<00:02, 197.40it/s]
 86%|████████▋ | 4333/5021 [00:22<00:03, 196.35it/s]
 91%|█████████ | 4572/5021 [00:23<00:02, 195.40it/s]
 87%|████████▋ | 4353/5021 [00:22<00:03, 196.54it/s]
 91%|█████████▏| 4593/5021 [00:23<00:02, 196.99it/s]
 87%|████████▋ | 4373/5021 [00:22<00:03, 196.28it/s]
 92%|█████████▏| 4614/5021 [00:23<00:02, 197.98it/s]
 87%|████████▋ | 4393/5021 [00:22<00:03, 195.92it/s]
 92%|█████████▏| 4634/5021 [00:23<00:01, 198.22it/s]
 88%|████████▊ | 4413/5021 [00:22<00:03, 194.62it/s]
 93%|█████████▎| 4654/5021 [00:23<00:01, 198.65it/s]
 88%|████████▊ | 4433/5021 [00:22<00:03, 194.76it/s]
 93%|█████████▎| 4674/5021 [00:23<00:01, 198.78it/s]
 89%|████████▊ | 4453/5021 [00:22<00:02, 194.69it/s]
 93%|█████████▎| 4694/5021 [00:23<00:01, 198.82it/s]
 89%|████████▉ | 4473/5021 [00:23<00:02, 192.52it/s]
 94%|█████████▍| 4714/5021 [00:23<00:01, 193.09it/s]
 89%|████████▉ | 4493/5021 [00:23<00:02, 186.78it/s]
 94%|█████████▍| 4734/5021 [00:23<00:01, 193.67it/s]
 90%|████████▉ | 4513/5021 [00:23<00:02, 188.62it/s]
 95%|█████████▍| 4755/5021 [00:23<00:01, 195.86it/s]
 90%|█████████ | 4533/5021 [00:23<00:02, 190.61it/s]
 95%|█████████▌| 4776/5021 [00:24<00:01, 197.39it/s]
 91%|█████████ | 4553/5021 [00:23<00:02, 192.07it/s]
 96%|█████████▌| 4797/5021 [00:24<00:01, 198.68it/s]
 91%|█████████ | 4573/5021 [00:23<00:02, 193.39it/s]
 96%|█████████▌| 4818/5021 [00:24<00:01, 199.66it/s]
 91%|█████████▏| 4593/5021 [00:23<00:02, 194.30it/s]
 96%|█████████▋| 4839/5021 [00:24<00:00, 200.19it/s]
 92%|█████████▏| 4613/5021 [00:23<00:02, 194.75it/s]
 97%|█████████▋| 4860/5021 [00:24<00:00, 200.34it/s]
 92%|█████████▏| 4633/5021 [00:23<00:01, 195.41it/s]
 97%|█████████▋| 4881/5021 [00:24<00:00, 200.14it/s]
 93%|█████████▎| 4653/5021 [00:23<00:01, 195.97it/s]
 98%|█████████▊| 4902/5021 [00:24<00:00, 200.53it/s]
 93%|█████████▎| 4673/5021 [00:24<00:01, 195.98it/s]
 98%|█████████▊| 4923/5021 [00:24<00:00, 200.35it/s]
 93%|█████████▎| 4693/5021 [00:24<00:01, 196.20it/s]
 98%|█████████▊| 4944/5021 [00:24<00:00, 200.18it/s]
 94%|█████████▍| 4713/5021 [00:24<00:01, 196.45it/s]
 94%|█████████▍| 4733/5021 [00:24<00:01, 196.52it/s]
 99%|█████████▉| 4965/5021 [00:25<00:00, 200.49it/s]
 95%|█████████▍| 4753/5021 [00:24<00:01, 196.16it/s]
 99%|█████████▉| 4986/5021 [00:25<00:00, 200.89it/s]
 95%|█████████▌| 4773/5021 [00:24<00:01, 195.63it/s]
100%|█████████▉| 5007/5021 [00:25<00:00, 201.19it/s]
100%|██████████| 5021/5021 [00:25<00:00, 198.62it/s]

 95%|█████████▌| 4793/5021 [00:24<00:01, 192.87it/s]
 96%|█████████▌| 4813/5021 [00:24<00:01, 192.28it/s]
 96%|█████████▋| 4833/5021 [00:24<00:00, 192.19it/s]
 97%|█████████▋| 4853/5021 [00:25<00:00, 192.17it/s]
 97%|█████████▋| 4873/5021 [00:25<00:00, 192.17it/s]
 97%|█████████▋| 4893/5021 [00:25<00:00, 192.60it/s]
 98%|█████████▊| 4913/5021 [00:25<00:00, 192.52it/s]
 98%|█████████▊| 4933/5021 [00:25<00:00, 192.86it/s]
 99%|█████████▊| 4953/5021 [00:25<00:00, 192.36it/s]
 99%|█████████▉| 4973/5021 [00:25<00:00, 192.65it/s]
 99%|█████████▉| 4993/5021 [00:25<00:00, 192.85it/s]
100%|█████████▉| 5013/5021 [00:25<00:00, 193.31it/s]
100%|██████████| 5021/5021 [00:25<00:00, 194.00it/s]
n136-128-154:2223813:2223813 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2223813:2223813 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2223813:2223813 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:2223813:2223813 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2223813:2223813 [0] NCCL INFO Comm config Blocking set to 1
n136-128-154:2223814:2223814 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:2223814:2223814 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2223814:2223814 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2223814:2223814 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2223814:2223814 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:2223813:2224024 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2223813:2224024 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2223814:2224025 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2223814:2224025 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2223813:2224024 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2223813:2224024 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2223813:2224024 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2223814:2224025 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2223814:2224025 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2223814:2224025 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2223813:2224024 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2223813:2224024 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2223813:2224024 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:2223813:2224024 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2223813:2224024 [0] NCCL INFO Using network IB
n136-128-154:2223814:2224025 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2223814:2224025 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2223814:2224025 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:2223814:2224025 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2223814:2224025 [1] NCCL INFO Using network IB
n136-128-154:2223813:2224024 [0] NCCL INFO ncclCommInitRankConfig comm 0x155a4e40 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0x16c35cf1206b8bda - Init START
n136-128-154:2223814:2224025 [1] NCCL INFO ncclCommInitRankConfig comm 0x12aba270 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0x16c35cf1206b8bda - Init START
n136-128-154:2223813:2224024 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2223814:2224025 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2223814:2224025 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2223814:2224025 [1] NCCL INFO Retrieving state for IB
n136-128-154:2223814:2224025 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:2223814:2224025 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2223814:2224025 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2223814:2224025 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2223813:2224024 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2223813:2224024 [0] NCCL INFO Retrieving state for IB
n136-128-154:2223813:2224024 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:2223814:2224025 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2223813:2224024 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2223813:2224024 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2223813:2224024 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2223813:2224024 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2223813:2224024 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2223813:2224024 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2223814:2224025 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2223814:2224025 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2223814:2224025 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2223814:2224025 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2223814:2224025 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2223814:2224025 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2223814:2224025 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2223814:2224025 [1] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2223814:2224025 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2223814:2224025 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2223814:2224025 [1] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2223814:2224025 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2223814:2224025 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2223814:2224025 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2223814:2224025 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2223814:2224025 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2223814:2224025 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2223814:2224025 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2223814:2224025 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2223814:2224025 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2223814:2224025 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2223814:2224025 [1] NCCL INFO ==========================================
n136-128-154:2223814:2224025 [1] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2223814:2224025 [1] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2223814:2224025 [1] NCCL INFO Setting affinity for GPU 7 to 33-62,97-126
n136-128-154:2223814:2224025 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2223813:2224024 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2223814:2224025 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223813:2224024 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2223814:2224025 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223814:2224025 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223813:2224024 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2223814:2224025 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223813:2224024 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2223814:2224025 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223813:2224024 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2223814:2224025 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223813:2224024 [0] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2223814:2224025 [1] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223813:2224024 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2223814:2224025 [1] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223813:2224024 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2223814:2224025 [1] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223813:2224024 [0] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2223814:2224025 [1] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223813:2224024 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2223814:2224025 [1] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223813:2224024 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2223814:2224025 [1] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223813:2224024 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2223813:2224024 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2223813:2224024 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2223813:2224024 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2223813:2224024 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2223813:2224024 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2223813:2224024 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2223813:2224024 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2223813:2224024 [0] NCCL INFO ==========================================
n136-128-154:2223813:2224024 [0] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2223813:2224024 [0] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2223813:2224024 [0] NCCL INFO Setting affinity for GPU 6 to 33-62,97-126
n136-128-154:2223814:2224025 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2223814:2224025 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223814:2224025 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223814:2224025 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223814:2224025 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223814:2224025 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223814:2224025 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223814:2224025 [1] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2223814:2224025 [1] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2223814:2224025 [1] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2223814:2224025 [1] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2223814:2224025 [1] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2223814:2224025 [1] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2223813:2224024 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2223813:2224024 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223813:2224024 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223813:2224024 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223813:2224024 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223813:2224024 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223813:2224024 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223813:2224024 [0] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223813:2224024 [0] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223813:2224024 [0] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223813:2224024 [0] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223813:2224024 [0] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223813:2224024 [0] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223813:2224024 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2223813:2224024 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223813:2224024 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223813:2224024 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223813:2224024 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223813:2224024 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223813:2224024 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2223813:2224024 [0] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2223813:2224024 [0] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2223813:2224024 [0] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2223813:2224024 [0] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2223813:2224024 [0] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2223813:2224024 [0] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2223813:2224024 [0] NCCL INFO comm 0x155a4e40 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:2223814:2224025 [1] NCCL INFO comm 0x12aba270 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:2223813:2224024 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:2223813:2224024 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:2223814:2224025 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:2223813:2224024 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:2223814:2224025 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:2223813:2224024 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:2223814:2224025 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:2223813:2224024 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:2223814:2224025 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:2223813:2224024 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:2223814:2224025 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:2223813:2224024 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:2223814:2224025 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:2223813:2224024 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:2223814:2224025 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:2223813:2224024 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:2223814:2224025 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:2223813:2224024 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:2223814:2224025 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:2223813:2224024 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:2223814:2224025 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:2223813:2224024 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:2223814:2224025 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:2223813:2224024 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:2223814:2224025 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:2223813:2224024 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:2223814:2224025 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:2223813:2224024 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:2223814:2224025 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:2223813:2224024 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:2223814:2224025 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:2223813:2224024 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:2223814:2224025 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:2223813:2224024 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:2223814:2224025 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:2223813:2224024 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:2223814:2224025 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:2223813:2224024 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:2223814:2224025 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:2223813:2224024 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:2223814:2224025 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:2223813:2224024 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:2223814:2224025 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:2223813:2224024 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:2223814:2224025 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:2223813:2224024 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:2223814:2224025 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:2223814:2224025 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:2223813:2224024 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:2223813:2224024 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:2223813:2224024 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:2223813:2224024 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:2223814:2224025 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:2223813:2224024 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:2223814:2224025 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:2223813:2224024 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:2223814:2224025 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:2223813:2224024 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:2223814:2224025 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:2223813:2224024 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:2223814:2224025 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:2223813:2224024 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:2223814:2224025 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:2223813:2224024 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:2223814:2224025 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:2223813:2224024 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:2223814:2224025 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:2223813:2224024 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:2223814:2224025 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:2223813:2224024 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:2223814:2224025 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:2223813:2224024 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:2223814:2224025 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:2223813:2224024 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:2223814:2224025 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:2223813:2224024 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:2223814:2224025 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:2223813:2224024 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:2223814:2224025 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:2223813:2224024 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:2223814:2224025 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:2223813:2224024 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:2223814:2224025 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:2223813:2224024 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:2223814:2224025 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:2223813:2224024 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:2223814:2224025 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:2223813:2224024 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:2223814:2224025 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:2223813:2224024 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:2223814:2224025 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:2223813:2224024 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:2223814:2224025 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:2223813:2224024 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:2223814:2224025 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:2223813:2224024 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:2223814:2224025 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:2223813:2224024 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:2223814:2224025 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:2223813:2224024 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:2223814:2224025 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:2223813:2224024 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:2223813:2224024 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:2223814:2224025 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2223813:2224024 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:2223813:2224024 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:2223813:2224024 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:2223813:2224024 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:2223813:2224024 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:2223813:2224024 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:2223813:2224024 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:2223813:2224024 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:2223813:2224024 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:2223813:2224024 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:2223813:2224024 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:2223813:2224024 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:2223813:2224024 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:2223813:2224024 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:2223813:2224024 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:2223813:2224024 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:2223813:2224024 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:2223813:2224024 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:2223813:2224024 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:2223813:2224024 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2223814:2224025 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2223814:2224037 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 116
n136-128-154:2223814:2224036 [1] NCCL INFO [Proxy Service] Device 1 CPU core 50
n136-128-154:2223813:2224024 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2223813:2224024 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:2223813:2224038 [0] NCCL INFO [Proxy Service] Device 0 CPU core 108
n136-128-154:2223813:2224039 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 119
n136-128-154:2223814:2224025 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2223814:2224025 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2223813:2224024 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2223813:2224024 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2223813:2224024 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:2223813:2224024 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2223813:2224024 [0] NCCL INFO ncclCommInitRankConfig comm 0x155a4e40 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0x16c35cf1206b8bda - Init COMPLETE
n136-128-154:2223813:2224024 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.71 (kernels 0.12, alloc 0.36, bootstrap 0.01, allgathers 0.00, topo 0.05, graphs 0.00, connections 0.03, rest 0.14)
n136-128-154:2223814:2224025 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2223814:2224025 [1] NCCL INFO ncclCommInitRankConfig comm 0x12aba270 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0x16c35cf1206b8bda - Init COMPLETE
n136-128-154:2223814:2224025 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.70 (kernels 0.12, alloc 0.35, bootstrap 0.00, allgathers 0.00, topo 0.05, graphs 0.00, connections 0.03, rest 0.14)
n136-128-154:2223813:2224040 [0] NCCL INFO Channel 00/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223813:2224040 [0] NCCL INFO Channel 01/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223813:2224040 [0] NCCL INFO Channel 02/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223813:2224040 [0] NCCL INFO Channel 03/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223813:2224040 [0] NCCL INFO Channel 04/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223813:2224040 [0] NCCL INFO Channel 05/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223813:2224040 [0] NCCL INFO Channel 06/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223813:2224040 [0] NCCL INFO Channel 07/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223814:2224041 [1] NCCL INFO Channel 00/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223813:2224040 [0] NCCL INFO Channel 08/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223814:2224041 [1] NCCL INFO Channel 01/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223813:2224040 [0] NCCL INFO Channel 09/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223814:2224041 [1] NCCL INFO Channel 02/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223813:2224040 [0] NCCL INFO Channel 10/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223814:2224041 [1] NCCL INFO Channel 03/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223813:2224040 [0] NCCL INFO Channel 11/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223814:2224041 [1] NCCL INFO Channel 04/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223813:2224040 [0] NCCL INFO Channel 12/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223814:2224041 [1] NCCL INFO Channel 05/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223813:2224040 [0] NCCL INFO Channel 13/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223814:2224041 [1] NCCL INFO Channel 06/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223813:2224040 [0] NCCL INFO Channel 14/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223814:2224041 [1] NCCL INFO Channel 07/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223813:2224040 [0] NCCL INFO Channel 15/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223814:2224041 [1] NCCL INFO Channel 08/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223813:2224040 [0] NCCL INFO Channel 16/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223814:2224041 [1] NCCL INFO Channel 09/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223813:2224040 [0] NCCL INFO Channel 17/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223814:2224041 [1] NCCL INFO Channel 10/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223813:2224040 [0] NCCL INFO Channel 18/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223814:2224041 [1] NCCL INFO Channel 11/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223813:2224040 [0] NCCL INFO Channel 19/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223814:2224041 [1] NCCL INFO Channel 12/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223813:2224040 [0] NCCL INFO Channel 20/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223814:2224041 [1] NCCL INFO Channel 13/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223813:2224040 [0] NCCL INFO Channel 21/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223814:2224041 [1] NCCL INFO Channel 14/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223813:2224040 [0] NCCL INFO Channel 22/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223814:2224041 [1] NCCL INFO Channel 15/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223813:2224040 [0] NCCL INFO Channel 23/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2223814:2224041 [1] NCCL INFO Channel 16/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2224041 [1] NCCL INFO Channel 17/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2224041 [1] NCCL INFO Channel 18/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2224041 [1] NCCL INFO Channel 19/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2224041 [1] NCCL INFO Channel 20/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2224041 [1] NCCL INFO Channel 21/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2224041 [1] NCCL INFO Channel 22/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2224041 [1] NCCL INFO Channel 23/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223813:2224040 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:2223814:2224041 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-09:17:06:47 INFO     [evaluator:559] Running loglikelihood requests
2025-12-09:17:06:47 INFO     [evaluator:559] Running loglikelihood requests
Passed argument batch_size = auto:1. Detecting largest batch size

Running loglikelihood requests:   0%|          | 0/20084 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 64
Determined largest batch size: 64

Running loglikelihood requests:   0%|          | 1/20084 [00:35<200:04:34, 35.86s/it]
Running loglikelihood requests:   0%|          | 65/20084 [00:42<2:42:16,  2.06it/s] 
Running loglikelihood requests:   1%|          | 129/20084 [00:48<1:25:01,  3.91it/s]
Running loglikelihood requests:   1%|          | 193/20084 [00:53<59:20,  5.59it/s]  
Running loglikelihood requests:   1%|▏         | 257/20084 [00:59<47:08,  7.01it/s]
Running loglikelihood requests:   2%|▏         | 321/20084 [01:04<40:18,  8.17it/s]
Running loglikelihood requests:   2%|▏         | 385/20084 [01:10<36:05,  9.10it/s]
Running loglikelihood requests:   2%|▏         | 449/20084 [01:15<33:20,  9.81it/s]
Running loglikelihood requests:   3%|▎         | 513/20084 [01:21<31:30, 10.35it/s]
Running loglikelihood requests:   3%|▎         | 577/20084 [01:26<30:09, 10.78it/s]
Running loglikelihood requests:   3%|▎         | 641/20084 [01:32<29:11, 11.10it/s]
Running loglikelihood requests:   4%|▎         | 705/20084 [01:37<28:27, 11.35it/s]
Running loglikelihood requests:   4%|▍         | 769/20084 [01:42<27:54, 11.54it/s]
Running loglikelihood requests:   4%|▍         | 833/20084 [01:48<27:27, 11.68it/s]
Running loglikelihood requests:   4%|▍         | 897/20084 [01:53<27:06, 11.80it/s]
Running loglikelihood requests:   5%|▍         | 961/20084 [01:58<26:48, 11.89it/s]
Running loglikelihood requests:   5%|▌         | 1025/20084 [02:03<26:33, 11.96it/s]
Running loglikelihood requests:   5%|▌         | 1089/20084 [02:09<26:20, 12.02it/s]
Running loglikelihood requests:   6%|▌         | 1153/20084 [02:14<26:07, 12.08it/s]
Running loglikelihood requests:   6%|▌         | 1217/20084 [02:19<25:56, 12.12it/s]
Running loglikelihood requests:   6%|▋         | 1281/20084 [02:24<25:47, 12.15it/s]
Running loglikelihood requests:   7%|▋         | 1345/20084 [02:30<25:37, 12.19it/s]
Running loglikelihood requests:   7%|▋         | 1409/20084 [02:35<25:28, 12.22it/s]
Running loglikelihood requests:   7%|▋         | 1473/20084 [02:40<25:19, 12.25it/s]
Running loglikelihood requests:   8%|▊         | 1537/20084 [02:45<25:10, 12.28it/s]
Running loglikelihood requests:   8%|▊         | 1601/20084 [02:50<25:02, 12.30it/s]
Running loglikelihood requests:   8%|▊         | 1665/20084 [02:55<24:53, 12.33it/s]
Running loglikelihood requests:   9%|▊         | 1729/20084 [03:01<24:45, 12.36it/s]
Running loglikelihood requests:   9%|▉         | 1793/20084 [03:06<24:36, 12.39it/s]
Running loglikelihood requests:   9%|▉         | 1857/20084 [03:11<24:28, 12.41it/s]
Running loglikelihood requests:  10%|▉         | 1921/20084 [03:16<24:19, 12.44it/s]
Running loglikelihood requests:  10%|▉         | 1985/20084 [03:21<24:11, 12.47it/s]
Running loglikelihood requests:  10%|█         | 2049/20084 [03:26<24:03, 12.50it/s]
Running loglikelihood requests:  11%|█         | 2113/20084 [03:31<23:55, 12.52it/s]
Running loglikelihood requests:  11%|█         | 2177/20084 [03:36<23:48, 12.54it/s]
Running loglikelihood requests:  11%|█         | 2241/20084 [03:41<23:41, 12.55it/s]
Running loglikelihood requests:  11%|█▏        | 2305/20084 [03:47<23:34, 12.57it/s]
Running loglikelihood requests:  12%|█▏        | 2369/20084 [03:52<23:27, 12.59it/s]
Running loglikelihood requests:  12%|█▏        | 2433/20084 [03:57<23:20, 12.61it/s]
Running loglikelihood requests:  12%|█▏        | 2497/20084 [04:02<23:15, 12.60it/s]
Running loglikelihood requests:  13%|█▎        | 2561/20084 [04:07<23:08, 12.62it/s]
Running loglikelihood requests:  13%|█▎        | 2625/20084 [04:12<23:02, 12.63it/s]
Running loglikelihood requests:  13%|█▎        | 2689/20084 [04:17<22:55, 12.65it/s]
Running loglikelihood requests:  14%|█▎        | 2753/20084 [04:22<22:48, 12.67it/s]
Running loglikelihood requests:  14%|█▍        | 2817/20084 [04:27<22:41, 12.68it/s]
Running loglikelihood requests:  14%|█▍        | 2881/20084 [04:32<22:34, 12.70it/s]
Running loglikelihood requests:  15%|█▍        | 2945/20084 [04:37<22:27, 12.72it/s]
Running loglikelihood requests:  15%|█▍        | 3009/20084 [04:42<22:20, 12.74it/s]
Running loglikelihood requests:  15%|█▌        | 3073/20084 [04:47<22:13, 12.75it/s]
Running loglikelihood requests:  16%|█▌        | 3137/20084 [04:52<22:07, 12.77it/s]
Running loglikelihood requests:  16%|█▌        | 3201/20084 [04:57<22:00, 12.78it/s]
Running loglikelihood requests:  16%|█▋        | 3265/20084 [05:02<21:54, 12.80it/s]
Running loglikelihood requests:  17%|█▋        | 3329/20084 [05:07<21:47, 12.82it/s]
Running loglikelihood requests:  17%|█▋        | 3393/20084 [05:12<21:41, 12.83it/s]
Running loglikelihood requests:  17%|█▋        | 3457/20084 [05:17<21:34, 12.84it/s]
Running loglikelihood requests:  18%|█▊        | 3521/20084 [05:22<21:28, 12.86it/s]
Running loglikelihood requests:  18%|█▊        | 3585/20084 [05:27<21:21, 12.87it/s]
Running loglikelihood requests:  18%|█▊        | 3649/20084 [05:32<21:14, 12.89it/s]
Running loglikelihood requests:  18%|█▊        | 3713/20084 [05:37<21:08, 12.91it/s]
Running loglikelihood requests:  19%|█▉        | 3777/20084 [05:42<21:01, 12.93it/s]
Running loglikelihood requests:  19%|█▉        | 3841/20084 [05:47<20:54, 12.94it/s]
Running loglikelihood requests:  19%|█▉        | 3905/20084 [05:52<20:48, 12.96it/s]
Running loglikelihood requests:  20%|█▉        | 3969/20084 [05:56<20:41, 12.98it/s]
Running loglikelihood requests:  20%|██        | 4033/20084 [06:01<20:35, 13.00it/s]
Running loglikelihood requests:  20%|██        | 4097/20084 [06:06<20:29, 13.01it/s]
Running loglikelihood requests:  21%|██        | 4161/20084 [06:11<20:23, 13.02it/s]
Running loglikelihood requests:  21%|██        | 4225/20084 [06:16<20:16, 13.03it/s]
Running loglikelihood requests:  21%|██▏       | 4289/20084 [06:21<20:11, 13.04it/s]
Running loglikelihood requests:  22%|██▏       | 4353/20084 [06:26<20:05, 13.05it/s]
Running loglikelihood requests:  22%|██▏       | 4417/20084 [06:31<19:58, 13.07it/s]
Running loglikelihood requests:  22%|██▏       | 4481/20084 [06:36<19:52, 13.08it/s]
Running loglikelihood requests:  23%|██▎       | 4545/20084 [06:41<19:46, 13.10it/s]
Running loglikelihood requests:  23%|██▎       | 4609/20084 [06:45<19:39, 13.12it/s]
Running loglikelihood requests:  23%|██▎       | 4673/20084 [06:50<19:33, 13.13it/s]
Running loglikelihood requests:  24%|██▎       | 4737/20084 [06:55<19:27, 13.14it/s]
Running loglikelihood requests:  24%|██▍       | 4801/20084 [07:00<19:20, 13.17it/s]
Running loglikelihood requests:  24%|██▍       | 4865/20084 [07:05<19:14, 13.19it/s]
Running loglikelihood requests:  25%|██▍       | 4929/20084 [07:10<19:08, 13.20it/s]
Running loglikelihood requests:  25%|██▍       | 4993/20084 [07:14<19:01, 13.22it/s]
Running loglikelihood requests:  25%|██▌       | 5057/20084 [07:19<18:55, 13.23it/s]
Running loglikelihood requests:  25%|██▌       | 5121/20084 [07:24<18:50, 13.24it/s]
Running loglikelihood requests:  26%|██▌       | 5185/20084 [07:29<18:44, 13.25it/s]
Running loglikelihood requests:  26%|██▌       | 5249/20084 [07:34<18:37, 13.27it/s]
Running loglikelihood requests:  26%|██▋       | 5313/20084 [07:39<18:31, 13.29it/s]
Running loglikelihood requests:  27%|██▋       | 5377/20084 [07:43<18:25, 13.30it/s]
Running loglikelihood requests:  27%|██▋       | 5441/20084 [07:48<18:19, 13.31it/s]
Running loglikelihood requests:  27%|██▋       | 5505/20084 [07:53<18:13, 13.33it/s]
Running loglikelihood requests:  28%|██▊       | 5569/20084 [07:58<18:07, 13.34it/s]
Running loglikelihood requests:  28%|██▊       | 5633/20084 [08:02<18:01, 13.36it/s]
Running loglikelihood requests:  28%|██▊       | 5697/20084 [08:07<17:53, 13.40it/s]
Running loglikelihood requests:  29%|██▊       | 5761/20084 [08:12<17:46, 13.43it/s]
Running loglikelihood requests:  29%|██▉       | 5825/20084 [08:17<17:40, 13.45it/s]
Running loglikelihood requests:  29%|██▉       | 5889/20084 [08:21<17:34, 13.46it/s]
Running loglikelihood requests:  30%|██▉       | 5953/20084 [08:26<17:28, 13.47it/s]
Running loglikelihood requests:  30%|██▉       | 6017/20084 [08:31<17:23, 13.49it/s]
Running loglikelihood requests:  30%|███       | 6081/20084 [08:36<17:17, 13.50it/s]
Running loglikelihood requests:  31%|███       | 6145/20084 [08:40<17:11, 13.51it/s]
Running loglikelihood requests:  31%|███       | 6209/20084 [08:45<17:06, 13.52it/s]
Running loglikelihood requests:  31%|███       | 6273/20084 [08:50<17:01, 13.53it/s]
Running loglikelihood requests:  32%|███▏      | 6337/20084 [08:55<16:56, 13.53it/s]
Running loglikelihood requests:  32%|███▏      | 6401/20084 [08:59<16:50, 13.54it/s]
Running loglikelihood requests:  32%|███▏      | 6465/20084 [09:04<16:44, 13.56it/s]
Running loglikelihood requests:  33%|███▎      | 6529/20084 [09:09<16:38, 13.57it/s]
Running loglikelihood requests:  33%|███▎      | 6593/20084 [09:13<16:32, 13.59it/s]
Running loglikelihood requests:  33%|███▎      | 6657/20084 [09:18<16:27, 13.60it/s]
Running loglikelihood requests:  33%|███▎      | 6721/20084 [09:23<16:21, 13.62it/s]
Running loglikelihood requests:  34%|███▍      | 6785/20084 [09:27<16:15, 13.63it/s]
Running loglikelihood requests:  34%|███▍      | 6849/20084 [09:32<16:09, 13.64it/s]
Running loglikelihood requests:  34%|███▍      | 6913/20084 [09:37<16:04, 13.65it/s]
Running loglikelihood requests:  35%|███▍      | 6977/20084 [09:41<15:59, 13.66it/s]
Running loglikelihood requests:  35%|███▌      | 7041/20084 [09:46<15:53, 13.68it/s]
Running loglikelihood requests:  35%|███▌      | 7105/20084 [09:51<15:47, 13.69it/s]
Running loglikelihood requests:  36%|███▌      | 7169/20084 [09:55<15:42, 13.70it/s]
Running loglikelihood requests:  36%|███▌      | 7233/20084 [10:00<15:36, 13.72it/s]
Running loglikelihood requests:  36%|███▋      | 7297/20084 [10:05<15:31, 13.73it/s]
Running loglikelihood requests:  37%|███▋      | 7361/20084 [10:09<15:25, 13.74it/s]
Running loglikelihood requests:  37%|███▋      | 7425/20084 [10:14<15:20, 13.75it/s]
Running loglikelihood requests:  37%|███▋      | 7489/20084 [10:19<15:15, 13.76it/s]
Running loglikelihood requests:  38%|███▊      | 7553/20084 [10:23<15:10, 13.77it/s]
Running loglikelihood requests:  38%|███▊      | 7617/20084 [10:28<15:04, 13.78it/s]
Running loglikelihood requests:  38%|███▊      | 7681/20084 [10:33<14:59, 13.79it/s]
Running loglikelihood requests:  39%|███▊      | 7745/20084 [10:37<14:53, 13.81it/s]
Running loglikelihood requests:  39%|███▉      | 7809/20084 [10:42<14:47, 13.82it/s]
Running loglikelihood requests:  39%|███▉      | 7873/20084 [10:46<14:42, 13.84it/s]
Running loglikelihood requests:  40%|███▉      | 7937/20084 [10:51<14:37, 13.85it/s]
Running loglikelihood requests:  40%|███▉      | 8001/20084 [10:56<14:31, 13.86it/s]
Running loglikelihood requests:  40%|████      | 8065/20084 [11:00<14:26, 13.87it/s]
Running loglikelihood requests:  40%|████      | 8129/20084 [11:05<14:21, 13.88it/s]
Running loglikelihood requests:  41%|████      | 8193/20084 [11:10<14:15, 13.89it/s]
Running loglikelihood requests:  41%|████      | 8257/20084 [11:14<14:10, 13.91it/s]
Running loglikelihood requests:  41%|████▏     | 8321/20084 [11:19<14:05, 13.92it/s]
Running loglikelihood requests:  42%|████▏     | 8385/20084 [11:23<14:00, 13.92it/s]
Running loglikelihood requests:  42%|████▏     | 8449/20084 [11:28<13:54, 13.94it/s]
Running loglikelihood requests:  42%|████▏     | 8513/20084 [11:32<13:49, 13.95it/s]
Running loglikelihood requests:  43%|████▎     | 8577/20084 [11:37<13:43, 13.97it/s]
Running loglikelihood requests:  43%|████▎     | 8641/20084 [11:42<13:38, 13.98it/s]
Running loglikelihood requests:  43%|████▎     | 8705/20084 [11:46<13:33, 14.00it/s]
Running loglikelihood requests:  44%|████▎     | 8769/20084 [11:51<13:28, 14.00it/s]
Running loglikelihood requests:  44%|████▍     | 8833/20084 [11:55<13:23, 14.01it/s]
Running loglikelihood requests:  44%|████▍     | 8897/20084 [12:00<13:17, 14.03it/s]
Running loglikelihood requests:  45%|████▍     | 8961/20084 [12:04<13:12, 14.04it/s]
Running loglikelihood requests:  45%|████▍     | 9025/20084 [12:09<13:06, 14.06it/s]
Running loglikelihood requests:  45%|████▌     | 9089/20084 [12:13<13:01, 14.07it/s]
Running loglikelihood requests:  46%|████▌     | 9153/20084 [12:18<12:55, 14.09it/s]
Running loglikelihood requests:  46%|████▌     | 9217/20084 [12:23<12:50, 14.10it/s]
Running loglikelihood requests:  46%|████▌     | 9281/20084 [12:27<12:44, 14.12it/s]
Running loglikelihood requests:  47%|████▋     | 9345/20084 [12:32<12:39, 14.14it/s]
Running loglikelihood requests:  47%|████▋     | 9409/20084 [12:36<12:33, 14.16it/s]
Running loglikelihood requests:  47%|████▋     | 9473/20084 [12:41<12:28, 14.17it/s]
Running loglikelihood requests:  47%|████▋     | 9537/20084 [12:45<12:23, 14.18it/s]
Running loglikelihood requests:  48%|████▊     | 9601/20084 [12:50<12:18, 14.20it/s]
Running loglikelihood requests:  48%|████▊     | 9665/20084 [12:54<12:12, 14.21it/s]
Running loglikelihood requests:  48%|████▊     | 9729/20084 [12:59<12:08, 14.22it/s]
Running loglikelihood requests:  49%|████▉     | 9793/20084 [13:03<12:02, 14.23it/s]
Running loglikelihood requests:  49%|████▉     | 9857/20084 [13:07<11:57, 14.25it/s]
Running loglikelihood requests:  49%|████▉     | 9921/20084 [13:12<11:52, 14.27it/s]
Running loglikelihood requests:  50%|████▉     | 9985/20084 [13:16<11:47, 14.28it/s]
Running loglikelihood requests:  50%|█████     | 10049/20084 [13:21<11:42, 14.29it/s]
Running loglikelihood requests:  50%|█████     | 10113/20084 [13:25<11:37, 14.30it/s]
Running loglikelihood requests:  51%|█████     | 10177/20084 [13:30<11:32, 14.31it/s]
Running loglikelihood requests:  51%|█████     | 10241/20084 [13:34<11:27, 14.32it/s]
Running loglikelihood requests:  51%|█████▏    | 10305/20084 [13:39<11:22, 14.34it/s]
Running loglikelihood requests:  52%|█████▏    | 10369/20084 [13:43<11:17, 14.35it/s]
Running loglikelihood requests:  52%|█████▏    | 10433/20084 [13:48<11:12, 14.35it/s]
Running loglikelihood requests:  52%|█████▏    | 10497/20084 [13:52<11:07, 14.37it/s]
Running loglikelihood requests:  53%|█████▎    | 10561/20084 [13:57<11:02, 14.37it/s]
Running loglikelihood requests:  53%|█████▎    | 10625/20084 [14:01<10:58, 14.37it/s]
Running loglikelihood requests:  53%|█████▎    | 10689/20084 [14:05<10:53, 14.39it/s]
Running loglikelihood requests:  54%|█████▎    | 10753/20084 [14:10<10:47, 14.40it/s]
Running loglikelihood requests:  54%|█████▍    | 10817/20084 [14:14<10:42, 14.43it/s]
Running loglikelihood requests:  54%|█████▍    | 10881/20084 [14:19<10:37, 14.44it/s]
Running loglikelihood requests:  54%|█████▍    | 10945/20084 [14:23<10:31, 14.46it/s]
Running loglikelihood requests:  55%|█████▍    | 11009/20084 [14:28<10:27, 14.47it/s]
Running loglikelihood requests:  55%|█████▌    | 11073/20084 [14:32<10:21, 14.49it/s]
Running loglikelihood requests:  55%|█████▌    | 11137/20084 [14:36<10:16, 14.51it/s]
Running loglikelihood requests:  56%|█████▌    | 11201/20084 [14:41<10:11, 14.52it/s]
Running loglikelihood requests:  56%|█████▌    | 11265/20084 [14:45<10:06, 14.53it/s]
Running loglikelihood requests:  56%|█████▋    | 11329/20084 [14:50<10:02, 14.53it/s]
Running loglikelihood requests:  57%|█████▋    | 11393/20084 [14:54<09:57, 14.54it/s]
Running loglikelihood requests:  57%|█████▋    | 11457/20084 [14:58<09:53, 14.55it/s]
Running loglikelihood requests:  57%|█████▋    | 11521/20084 [15:03<09:48, 14.56it/s]
Running loglikelihood requests:  58%|█████▊    | 11585/20084 [15:07<09:42, 14.58it/s]
Running loglikelihood requests:  58%|█████▊    | 11649/20084 [15:11<09:37, 14.60it/s]
Running loglikelihood requests:  58%|█████▊    | 11713/20084 [15:16<09:32, 14.61it/s]
Running loglikelihood requests:  59%|█████▊    | 11777/20084 [15:20<09:28, 14.61it/s]
Running loglikelihood requests:  59%|█████▉    | 11841/20084 [15:25<09:23, 14.64it/s]
Running loglikelihood requests:  59%|█████▉    | 11905/20084 [15:29<09:18, 14.63it/s]
Running loglikelihood requests:  60%|█████▉    | 11969/20084 [15:33<09:13, 14.65it/s]
Running loglikelihood requests:  60%|█████▉    | 12033/20084 [15:38<09:09, 14.66it/s]
Running loglikelihood requests:  60%|██████    | 12097/20084 [15:42<09:04, 14.68it/s]
Running loglikelihood requests:  61%|██████    | 12161/20084 [15:46<08:59, 14.69it/s]
Running loglikelihood requests:  61%|██████    | 12225/20084 [15:51<08:54, 14.69it/s]
Running loglikelihood requests:  61%|██████    | 12289/20084 [15:55<08:49, 14.71it/s]
Running loglikelihood requests:  62%|██████▏   | 12353/20084 [15:59<08:45, 14.70it/s]
Running loglikelihood requests:  62%|██████▏   | 12417/20084 [16:04<08:41, 14.70it/s]
Running loglikelihood requests:  62%|██████▏   | 12481/20084 [16:08<08:36, 14.71it/s]
Running loglikelihood requests:  62%|██████▏   | 12545/20084 [16:12<08:32, 14.72it/s]
Running loglikelihood requests:  63%|██████▎   | 12609/20084 [16:17<08:26, 14.75it/s]
Running loglikelihood requests:  63%|██████▎   | 12673/20084 [16:21<08:22, 14.76it/s]
Running loglikelihood requests:  63%|██████▎   | 12737/20084 [16:25<08:17, 14.78it/s]
Running loglikelihood requests:  64%|██████▎   | 12801/20084 [16:30<08:12, 14.79it/s]
Running loglikelihood requests:  64%|██████▍   | 12865/20084 [16:34<08:07, 14.81it/s]
Running loglikelihood requests:  64%|██████▍   | 12929/20084 [16:38<08:02, 14.83it/s]
Running loglikelihood requests:  65%|██████▍   | 12993/20084 [16:43<07:57, 14.85it/s]
Running loglikelihood requests:  65%|██████▌   | 13057/20084 [16:47<07:52, 14.87it/s]
Running loglikelihood requests:  65%|██████▌   | 13121/20084 [16:51<07:47, 14.89it/s]
Running loglikelihood requests:  66%|██████▌   | 13185/20084 [16:56<07:43, 14.90it/s]
Running loglikelihood requests:  66%|██████▌   | 13249/20084 [17:00<07:38, 14.90it/s]
Running loglikelihood requests:  66%|██████▋   | 13313/20084 [17:04<07:33, 14.93it/s]
Running loglikelihood requests:  67%|██████▋   | 13377/20084 [17:08<07:28, 14.94it/s]
Running loglikelihood requests:  67%|██████▋   | 13441/20084 [17:13<07:24, 14.95it/s]
Running loglikelihood requests:  67%|██████▋   | 13505/20084 [17:17<07:19, 14.96it/s]
Running loglikelihood requests:  68%|██████▊   | 13569/20084 [17:21<07:14, 14.98it/s]
Running loglikelihood requests:  68%|██████▊   | 13633/20084 [17:25<07:09, 15.00it/s]
Running loglikelihood requests:  68%|██████▊   | 13697/20084 [17:30<07:05, 15.02it/s]
Running loglikelihood requests:  69%|██████▊   | 13761/20084 [17:34<07:00, 15.05it/s]
Running loglikelihood requests:  69%|██████▉   | 13825/20084 [17:38<06:55, 15.07it/s]
Running loglikelihood requests:  69%|██████▉   | 13889/20084 [17:42<06:50, 15.08it/s]
Running loglikelihood requests:  69%|██████▉   | 13953/20084 [17:47<06:45, 15.10it/s]
Running loglikelihood requests:  70%|██████▉   | 14017/20084 [17:51<06:41, 15.12it/s]
Running loglikelihood requests:  70%|███████   | 14081/20084 [17:55<06:36, 15.15it/s]
Running loglikelihood requests:  70%|███████   | 14145/20084 [17:59<06:31, 15.16it/s]
Running loglikelihood requests:  71%|███████   | 14209/20084 [18:03<06:27, 15.17it/s]
Running loglikelihood requests:  71%|███████   | 14273/20084 [18:08<06:22, 15.20it/s]
Running loglikelihood requests:  71%|███████▏  | 14337/20084 [18:12<06:17, 15.22it/s]
Running loglikelihood requests:  72%|███████▏  | 14401/20084 [18:16<06:13, 15.23it/s]
Running loglikelihood requests:  72%|███████▏  | 14465/20084 [18:20<06:08, 15.25it/s]
Running loglikelihood requests:  72%|███████▏  | 14529/20084 [18:24<06:04, 15.26it/s]
Running loglikelihood requests:  73%|███████▎  | 14593/20084 [18:29<05:59, 15.28it/s]
Running loglikelihood requests:  73%|███████▎  | 14657/20084 [18:33<05:54, 15.30it/s]
Running loglikelihood requests:  73%|███████▎  | 14721/20084 [18:37<05:50, 15.31it/s]
Running loglikelihood requests:  74%|███████▎  | 14785/20084 [18:41<05:45, 15.33it/s]
Running loglikelihood requests:  74%|███████▍  | 14849/20084 [18:45<05:41, 15.35it/s]
Running loglikelihood requests:  74%|███████▍  | 14913/20084 [18:49<05:36, 15.37it/s]
Running loglikelihood requests:  75%|███████▍  | 14977/20084 [18:54<05:31, 15.38it/s]
Running loglikelihood requests:  75%|███████▍  | 15041/20084 [18:58<05:27, 15.40it/s]
Running loglikelihood requests:  75%|███████▌  | 15105/20084 [19:02<05:22, 15.42it/s]
Running loglikelihood requests:  76%|███████▌  | 15169/20084 [19:06<05:18, 15.44it/s]
Running loglikelihood requests:  76%|███████▌  | 15233/20084 [19:10<05:14, 15.44it/s]
Running loglikelihood requests:  76%|███████▌  | 15297/20084 [19:14<05:09, 15.47it/s]
Running loglikelihood requests:  76%|███████▋  | 15361/20084 [19:18<05:04, 15.49it/s]
Running loglikelihood requests:  77%|███████▋  | 15425/20084 [19:22<05:00, 15.50it/s]
Running loglikelihood requests:  77%|███████▋  | 15489/20084 [19:27<04:56, 15.51it/s]
Running loglikelihood requests:  77%|███████▋  | 15553/20084 [19:31<04:52, 15.51it/s]
Running loglikelihood requests:  78%|███████▊  | 15617/20084 [19:35<04:46, 15.57it/s]
Running loglikelihood requests:  78%|███████▊  | 15681/20084 [19:39<04:42, 15.59it/s]
Running loglikelihood requests:  78%|███████▊  | 15745/20084 [19:43<04:37, 15.62it/s]
Running loglikelihood requests:  79%|███████▊  | 15809/20084 [19:47<04:33, 15.65it/s]
Running loglikelihood requests:  79%|███████▉  | 15873/20084 [19:51<04:28, 15.67it/s]
Running loglikelihood requests:  79%|███████▉  | 15937/20084 [19:55<04:24, 15.70it/s]
Running loglikelihood requests:  80%|███████▉  | 16001/20084 [19:59<04:19, 15.72it/s]
Running loglikelihood requests:  80%|███████▉  | 16065/20084 [20:03<04:15, 15.73it/s]
Running loglikelihood requests:  80%|████████  | 16129/20084 [20:07<04:11, 15.75it/s]
Running loglikelihood requests:  81%|████████  | 16193/20084 [20:11<04:06, 15.76it/s]
Running loglikelihood requests:  81%|████████  | 16257/20084 [20:15<04:02, 15.78it/s]
Running loglikelihood requests:  81%|████████▏ | 16321/20084 [20:19<03:58, 15.80it/s]
Running loglikelihood requests:  82%|████████▏ | 16385/20084 [20:23<03:53, 15.84it/s]
Running loglikelihood requests:  82%|████████▏ | 16449/20084 [20:27<03:48, 15.88it/s]
Running loglikelihood requests:  82%|████████▏ | 16513/20084 [20:32<03:44, 15.90it/s]
Running loglikelihood requests:  83%|████████▎ | 16577/20084 [20:35<03:40, 15.93it/s]
Running loglikelihood requests:  83%|████████▎ | 16641/20084 [20:39<03:35, 15.95it/s]
Running loglikelihood requests:  83%|████████▎ | 16705/20084 [20:43<03:31, 15.98it/s]
Running loglikelihood requests:  83%|████████▎ | 16769/20084 [20:47<03:27, 16.00it/s]
Running loglikelihood requests:  84%|████████▍ | 16833/20084 [20:51<03:22, 16.03it/s]
Running loglikelihood requests:  84%|████████▍ | 16897/20084 [20:55<03:18, 16.05it/s]
Running loglikelihood requests:  84%|████████▍ | 16961/20084 [20:59<03:14, 16.05it/s]
Running loglikelihood requests:  85%|████████▍ | 17025/20084 [21:03<03:10, 16.09it/s]
Running loglikelihood requests:  85%|████████▌ | 17089/20084 [21:07<03:05, 16.12it/s]
Running loglikelihood requests:  85%|████████▌ | 17153/20084 [21:11<03:01, 16.15it/s]
Running loglikelihood requests:  86%|████████▌ | 17217/20084 [21:15<02:57, 16.19it/s]
Running loglikelihood requests:  86%|████████▌ | 17281/20084 [21:19<02:52, 16.21it/s]
Running loglikelihood requests:  86%|████████▋ | 17345/20084 [21:23<02:48, 16.24it/s]
Running loglikelihood requests:  87%|████████▋ | 17409/20084 [21:27<02:44, 16.28it/s]
Running loglikelihood requests:  87%|████████▋ | 17473/20084 [21:31<02:40, 16.30it/s]
Running loglikelihood requests:  87%|████████▋ | 17537/20084 [21:35<02:36, 16.32it/s]
Running loglikelihood requests:  88%|████████▊ | 17601/20084 [21:39<02:31, 16.35it/s]
Running loglikelihood requests:  88%|████████▊ | 17665/20084 [21:43<02:27, 16.38it/s]
Running loglikelihood requests:  88%|████████▊ | 17729/20084 [21:46<02:23, 16.42it/s]
Running loglikelihood requests:  89%|████████▊ | 17793/20084 [21:50<02:19, 16.45it/s]
Running loglikelihood requests:  89%|████████▉ | 17857/20084 [21:54<02:15, 16.48it/s]
Running loglikelihood requests:  89%|████████▉ | 17921/20084 [21:58<02:11, 16.50it/s]
Running loglikelihood requests:  90%|████████▉ | 17985/20084 [22:02<02:06, 16.54it/s]
Running loglikelihood requests:  90%|████████▉ | 18049/20084 [22:06<02:02, 16.58it/s]
Running loglikelihood requests:  90%|█████████ | 18113/20084 [22:10<01:58, 16.62it/s]
Running loglikelihood requests:  91%|█████████ | 18177/20084 [22:13<01:54, 16.64it/s]
Running loglikelihood requests:  91%|█████████ | 18241/20084 [22:17<01:50, 16.69it/s]
Running loglikelihood requests:  91%|█████████ | 18305/20084 [22:21<01:46, 16.73it/s]
Running loglikelihood requests:  91%|█████████▏| 18369/20084 [22:25<01:42, 16.77it/s]
Running loglikelihood requests:  92%|█████████▏| 18433/20084 [22:29<01:38, 16.80it/s]
Running loglikelihood requests:  92%|█████████▏| 18497/20084 [22:32<01:34, 16.86it/s]
Running loglikelihood requests:  92%|█████████▏| 18561/20084 [22:36<01:30, 16.90it/s]
Running loglikelihood requests:  93%|█████████▎| 18625/20084 [22:40<01:26, 16.94it/s]
Running loglikelihood requests:  93%|█████████▎| 18689/20084 [22:44<01:22, 16.99it/s]
Running loglikelihood requests:  93%|█████████▎| 18753/20084 [22:47<01:18, 17.04it/s]
Running loglikelihood requests:  94%|█████████▎| 18817/20084 [22:51<01:14, 17.10it/s]
Running loglikelihood requests:  94%|█████████▍| 18881/20084 [22:55<01:10, 17.16it/s]
Running loglikelihood requests:  94%|█████████▍| 18945/20084 [22:58<01:06, 17.20it/s]
Running loglikelihood requests:  95%|█████████▍| 19009/20084 [23:02<01:02, 17.20it/s]
Running loglikelihood requests:  95%|█████████▍| 19073/20084 [23:06<00:58, 17.29it/s]
Running loglikelihood requests:  95%|█████████▌| 19137/20084 [23:10<00:54, 17.35it/s]
Running loglikelihood requests:  96%|█████████▌| 19201/20084 [23:13<00:50, 17.44it/s]
Running loglikelihood requests:  96%|█████████▌| 19265/20084 [23:17<00:46, 17.52it/s]
Running loglikelihood requests:  96%|█████████▌| 19329/20084 [23:20<00:42, 17.59it/s]
Running loglikelihood requests:  97%|█████████▋| 19393/20084 [23:24<00:39, 17.69it/s]
Running loglikelihood requests:  97%|█████████▋| 19457/20084 [23:27<00:35, 17.79it/s]
Running loglikelihood requests:  97%|█████████▋| 19521/20084 [23:31<00:31, 17.89it/s]
Running loglikelihood requests:  98%|█████████▊| 19585/20084 [23:35<00:27, 18.00it/s]
Running loglikelihood requests:  98%|█████████▊| 19649/20084 [23:38<00:24, 18.11it/s]
Running loglikelihood requests:  98%|█████████▊| 19713/20084 [23:41<00:20, 18.27it/s]
Running loglikelihood requests:  98%|█████████▊| 19777/20084 [23:45<00:16, 18.45it/s]
Running loglikelihood requests:  99%|█████████▉| 19841/20084 [23:48<00:13, 18.68it/s]
Running loglikelihood requests:  99%|█████████▉| 19905/20084 [23:51<00:09, 18.90it/s]
Running loglikelihood requests:  99%|█████████▉| 19969/20084 [23:55<00:05, 19.21it/s]
Running loglikelihood requests: 100%|█████████▉| 20033/20084 [23:57<00:02, 20.76it/s]
Running loglikelihood requests: 100%|██████████| 20084/20084 [23:57<00:00, 13.97it/s]
n136-128-154:2223814:2226553 [1] NCCL INFO Channel 00/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2226553 [1] NCCL INFO Channel 01/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2226553 [1] NCCL INFO Channel 02/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2226553 [1] NCCL INFO Channel 03/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2226553 [1] NCCL INFO Channel 04/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2226553 [1] NCCL INFO Channel 05/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2226553 [1] NCCL INFO Channel 06/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2226553 [1] NCCL INFO Channel 07/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2226553 [1] NCCL INFO Channel 08/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2226553 [1] NCCL INFO Channel 09/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2226553 [1] NCCL INFO Channel 10/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2226553 [1] NCCL INFO Channel 11/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2226553 [1] NCCL INFO Channel 12/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2226553 [1] NCCL INFO Channel 13/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2226553 [1] NCCL INFO Channel 14/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2226553 [1] NCCL INFO Channel 15/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2226553 [1] NCCL INFO Channel 16/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2226553 [1] NCCL INFO Channel 17/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2226553 [1] NCCL INFO Channel 18/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2226553 [1] NCCL INFO Channel 19/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2226553 [1] NCCL INFO Channel 20/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2226553 [1] NCCL INFO Channel 21/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2226553 [1] NCCL INFO Channel 22/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2226553 [1] NCCL INFO Channel 23/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2226553 [1] NCCL INFO Channel 24/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2226553 [1] NCCL INFO Channel 25/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2226553 [1] NCCL INFO Channel 26/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2226553 [1] NCCL INFO Channel 27/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2226553 [1] NCCL INFO Channel 28/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2226553 [1] NCCL INFO Channel 29/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2226553 [1] NCCL INFO Channel 30/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2223814:2226553 [1] NCCL INFO Channel 31/1 : 1[7] -> 0[6] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:2223814:2226574 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2223814:2226574 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2223814:2226574 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2223814:2226574 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2223814:2226574 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2223814:2226574 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2223814:2224036 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-09:17:32:06 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 10, batch_size: auto (64)
|  Tasks  |Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|---------|------:|------|-----:|--------|---|-----:|---|-----:|
|hellaswag|      1|none  |    10|acc     |↑  |0.4457|±  |0.0050|
|         |       |none  |    10|acc_norm|↑  |0.6242|±  |0.0048|

[rank0]:[W1209 17:32:07.519485233 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:2223813:2226625 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2223813:2226625 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2223813:2226625 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2223813:2224038 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2223813:2226625 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2223813:2226625 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2223813:2226625 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2223814:2224036 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2223814:2226574 [1] NCCL INFO comm 0x12aba270 rank 1 nranks 2 cudaDev 1 busId c9000 - Abort COMPLETE
n136-128-154:2223813:2226625 [0] NCCL INFO comm 0x155a4e40 rank 0 nranks 2 cudaDev 0 busId c5000 - Abort COMPLETE
任务 hellaswag 评估完成！
Execution time: 2699.23 seconds