nohup: ignoring input
Namespace(save_dir='/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg', self_attn_layer_to_quant='4 1 2 8 23 22 25 5 24 7 26 6 20 12 19 17 21 11 10 9 18', mlp_layer_to_quant='27 16 19', model_id='/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-7B', cuda_id=6)
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.01it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.10it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.15it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.14it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.13it/s]
Qwen2ForCausalLM(
  (model): Qwen2Model(
    (embed_tokens): Embedding(152064, 3584)
    (layers): ModuleList(
      (0-27): 28 x Qwen2DecoderLayer(
        (self_attn): Qwen2Attention(
          (q_proj): Linear(in_features=3584, out_features=3584, bias=True)
          (k_proj): Linear(in_features=3584, out_features=512, bias=True)
          (v_proj): Linear(in_features=3584, out_features=512, bias=True)
          (o_proj): Linear(in_features=3584, out_features=3584, bias=False)
        )
        (mlp): Qwen2MLP(
          (gate_proj): Linear(in_features=3584, out_features=18944, bias=False)
          (up_proj): Linear(in_features=3584, out_features=18944, bias=False)
          (down_proj): Linear(in_features=18944, out_features=3584, bias=False)
          (act_fn): SiLUActivation()
        )
        (input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
        (post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
      )
    )
    (norm): Qwen2RMSNorm((3584,), eps=1e-06)
    (rotary_emb): Qwen2RotaryEmbedding()
  )
  (lm_head): Linear(in_features=3584, out_features=152064, bias=False)
)
Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation.
Once upon a time, a farmer owned a field. The farmer planted wheat in the field and harvested it every year. However, one year, a storm destroyed the wheat crop. The farmer decided to plant rice instead. The rice crop was successful, and the farmer was able to sell the rice and make a profit. The farmer then used the profit to buy more land, and eventually became a wealthy landowner. The moral of the story is that sometimes, one must be willing to adapt to changing circumstances in order to
quanting ... 
layers.0.self_attn 4 bit quant 
layers.0.mlp 4 bit quant 
layers.1.self_attn 2 bit quant 
layers.1.mlp 4 bit quant 
layers.2.self_attn 2 bit quant 
layers.2.mlp 4 bit quant 
layers.3.self_attn 4 bit quant 
layers.3.mlp 4 bit quant 
layers.4.self_attn 2 bit quant 
layers.4.mlp 4 bit quant 
layers.5.self_attn 2 bit quant 
layers.5.mlp 4 bit quant 
layers.6.self_attn 2 bit quant 
layers.6.mlp 4 bit quant 
layers.7.self_attn 2 bit quant 
layers.7.mlp 4 bit quant 
layers.8.self_attn 2 bit quant 
layers.8.mlp 4 bit quant 
layers.9.self_attn 2 bit quant 
layers.9.mlp 4 bit quant 
layers.10.self_attn 2 bit quant 
layers.10.mlp 4 bit quant 
layers.11.self_attn 2 bit quant 
layers.11.mlp 4 bit quant 
layers.12.self_attn 2 bit quant 
layers.12.mlp 4 bit quant 
layers.13.self_attn 4 bit quant 
layers.13.mlp 4 bit quant 
layers.14.self_attn 4 bit quant 
layers.14.mlp 4 bit quant 
layers.15.self_attn 4 bit quant 
layers.15.mlp 4 bit quant 
layers.16.self_attn 4 bit quant 
layers.16.mlp 2 bit quant 
layers.17.self_attn 2 bit quant 
layers.17.mlp 4 bit quant 
layers.18.self_attn 2 bit quant 
layers.18.mlp 4 bit quant 
layers.19.self_attn 2 bit quant 
layers.19.mlp 2 bit quant 
layers.20.self_attn 2 bit quant 
layers.20.mlp 4 bit quant 
layers.21.self_attn 2 bit quant 
layers.21.mlp 4 bit quant 
layers.22.self_attn 2 bit quant 
layers.22.mlp 4 bit quant 
layers.23.self_attn 2 bit quant 
layers.23.mlp 4 bit quant 
layers.24.self_attn 2 bit quant 
layers.24.mlp 4 bit quant 
layers.25.self_attn 2 bit quant 
layers.25.mlp 4 bit quant 
layers.26.self_attn 2 bit quant 
layers.26.mlp 4 bit quant 
layers.27.self_attn 4 bit quant 
layers.27.mlp 2 bit quant 
quanted
Qwen2ForCausalLM(
  (model): Qwen2Model(
    (embed_tokens): Embedding(152064, 3584)
    (layers): ModuleList(
      (0-27): 28 x Qwen2DecoderLayer(
        (self_attn): Qwen2Attention(
          (q_proj): W8A16Linear(3584, 3584, bias=True, weight_quant=per_channel)
          (k_proj): W8A16Linear(3584, 512, bias=True, weight_quant=per_channel)
          (v_proj): W8A16Linear(3584, 512, bias=True, weight_quant=per_channel)
          (o_proj): W8A16Linear(3584, 3584, bias=False, weight_quant=per_channel)
        )
        (mlp): Qwen2MLP(
          (gate_proj): W8A16Linear(3584, 18944, bias=False, weight_quant=per_channel)
          (up_proj): W8A16Linear(3584, 18944, bias=False, weight_quant=per_channel)
          (down_proj): W8A16Linear(18944, 3584, bias=False, weight_quant=per_channel)
          (act_fn): SiLUActivation()
        )
        (input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
        (post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
      )
    )
    (norm): Qwen2RMSNorm((3584,), eps=1e-06)
    (rotary_emb): Qwen2RotaryEmbedding()
  )
  (lm_head): Linear(in_features=3584, out_features=152064, bias=False)
)
Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation.
Once upon a time, a至尊观看  25-1 strugg of (, 11 in  strugg for 2  5 1 ( (142  a new 1/operators ( to c c ( (  ( strugg as, trous s for  to in the'icon" an over ( 1 act (27 ( 500 convin as bä and in  suche for (3, (, (0  to 11 0 0  to at

==================================================
开始评估：任务=winogrande | 少样本数=5 | 模型=Qwen2.5-7B-quantization-fg
输出路径：results2/Qwen2.5-7B-quantization-fg/f6/winogrande.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-09:19:11:02 INFO     [__main__:440] Selected Tasks: ['winogrande']
2025-12-09:19:11:02 INFO     [__main__:440] Selected Tasks: ['winogrande']
2025-12-09:19:11:02 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:19:11:02 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-09:19:11:02 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:19:11:02 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-09:19:11:03 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:19:11:03 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:19:11:03 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.07it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.06s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.21it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.05it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.31it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.89it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.57it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.16it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.67it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.39it/s]
2025-12-09:19:11:55 WARNING  [evaluator:309] Overwriting default num_fewshot of winogrande from None to 5
2025-12-09:19:11:55 INFO     [api.task:434] Building contexts for winogrande on rank 1...

  0%|          | 0/633 [00:00<?, ?it/s]
100%|██████████| 633/633 [00:00<00:00, 25923.88it/s]
2025-12-09:19:12:14 WARNING  [evaluator:309] Overwriting default num_fewshot of winogrande from None to 5
2025-12-09:19:12:14 INFO     [api.task:434] Building contexts for winogrande on rank 0...

  0%|          | 0/634 [00:00<?, ?it/s]
100%|██████████| 634/634 [00:00<00:00, 26915.14it/s]
n136-128-154:2239781:2239781 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2239781:2239781 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2239781:2239781 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:2239781:2239781 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2239781:2239781 [0] NCCL INFO Comm config Blocking set to 1
n136-128-154:2239782:2239782 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:2239782:2239782 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2239782:2239782 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2239782:2239782 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2239782:2239782 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:2239781:2240030 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2239781:2240030 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2239782:2240031 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2239782:2240031 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2239781:2240030 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2239781:2240030 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2239781:2240030 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2239782:2240031 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2239782:2240031 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2239782:2240031 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2239781:2240030 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2239781:2240030 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2239781:2240030 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:2239781:2240030 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2239781:2240030 [0] NCCL INFO Using network IB
n136-128-154:2239782:2240031 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2239782:2240031 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2239782:2240031 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:2239782:2240031 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2239782:2240031 [1] NCCL INFO Using network IB
n136-128-154:2239782:2240031 [1] NCCL INFO ncclCommInitRankConfig comm 0x10b4d040 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0xc1a41f07abad06fa - Init START
n136-128-154:2239781:2240030 [0] NCCL INFO ncclCommInitRankConfig comm 0x102eb980 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0xc1a41f07abad06fa - Init START
n136-128-154:2239781:2240030 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2239782:2240031 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2239781:2240030 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2239781:2240030 [0] NCCL INFO Retrieving state for IB
n136-128-154:2239781:2240030 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:2239781:2240030 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2239781:2240030 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2239781:2240030 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2239782:2240031 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2239782:2240031 [1] NCCL INFO Retrieving state for IB
n136-128-154:2239782:2240031 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:2239782:2240031 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2239781:2240030 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2239782:2240031 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2239782:2240031 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2239782:2240031 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2239782:2240031 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2239782:2240031 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2239781:2240030 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2239781:2240030 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2239782:2240031 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2239782:2240031 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2239782:2240031 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2239782:2240031 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2239782:2240031 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2239782:2240031 [1] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2239781:2240030 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2239782:2240031 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2239781:2240030 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2239782:2240031 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2239781:2240030 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2239782:2240031 [1] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2239781:2240030 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2239782:2240031 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2239781:2240030 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2239782:2240031 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2239781:2240030 [0] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2239782:2240031 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2239781:2240030 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2239782:2240031 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2239781:2240030 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2239782:2240031 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2239781:2240030 [0] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2239782:2240031 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2239781:2240030 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2239782:2240031 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2239781:2240030 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2239782:2240031 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2239781:2240030 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2239782:2240031 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2239781:2240030 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2239782:2240031 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2239781:2240030 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2239782:2240031 [1] NCCL INFO ==========================================
n136-128-154:2239781:2240030 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2239781:2240030 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2239781:2240030 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2239782:2240031 [1] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2239781:2240030 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2239781:2240030 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2239781:2240030 [0] NCCL INFO ==========================================
n136-128-154:2239782:2240031 [1] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2239781:2240030 [0] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2239782:2240031 [1] NCCL INFO Setting affinity for GPU 7 to 33-62,97-126
n136-128-154:2239781:2240030 [0] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2239781:2240030 [0] NCCL INFO Setting affinity for GPU 6 to 33-62,97-126
n136-128-154:2239781:2240030 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2239782:2240031 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2239781:2240030 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2239781:2240030 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2239782:2240031 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2239781:2240030 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2239782:2240031 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2239781:2240030 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2239782:2240031 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2239781:2240030 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2239782:2240031 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2239781:2240030 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2239782:2240031 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2239781:2240030 [0] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2239782:2240031 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2239781:2240030 [0] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2239782:2240031 [1] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2239781:2240030 [0] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2239782:2240031 [1] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2239781:2240030 [0] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2239782:2240031 [1] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2239781:2240030 [0] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2239782:2240031 [1] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2239781:2240030 [0] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2239782:2240031 [1] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2239782:2240031 [1] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2239781:2240030 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2239781:2240030 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2239781:2240030 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2239782:2240031 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2239781:2240030 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2239782:2240031 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2239781:2240030 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2239782:2240031 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2239781:2240030 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2239782:2240031 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2239781:2240030 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2239782:2240031 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2239781:2240030 [0] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2239782:2240031 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2239781:2240030 [0] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2239782:2240031 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2239781:2240030 [0] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2239782:2240031 [1] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2239781:2240030 [0] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2239782:2240031 [1] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2239781:2240030 [0] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2239782:2240031 [1] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2239781:2240030 [0] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2239782:2240031 [1] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2239782:2240031 [1] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2239782:2240031 [1] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2239781:2240030 [0] NCCL INFO comm 0x102eb980 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:2239782:2240031 [1] NCCL INFO comm 0x10b4d040 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:2239781:2240030 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:2239781:2240030 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:2239782:2240031 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:2239781:2240030 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:2239782:2240031 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:2239781:2240030 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:2239782:2240031 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:2239781:2240030 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:2239782:2240031 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:2239781:2240030 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:2239782:2240031 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:2239781:2240030 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:2239782:2240031 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:2239781:2240030 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:2239782:2240031 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:2239781:2240030 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:2239782:2240031 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:2239781:2240030 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:2239782:2240031 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:2239781:2240030 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:2239782:2240031 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:2239781:2240030 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:2239782:2240031 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:2239781:2240030 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:2239782:2240031 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:2239781:2240030 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:2239782:2240031 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:2239781:2240030 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:2239782:2240031 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:2239781:2240030 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:2239782:2240031 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:2239781:2240030 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:2239782:2240031 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:2239781:2240030 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:2239782:2240031 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:2239781:2240030 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:2239782:2240031 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:2239781:2240030 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:2239782:2240031 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:2239781:2240030 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:2239782:2240031 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:2239781:2240030 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:2239782:2240031 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:2239781:2240030 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:2239782:2240031 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:2239781:2240030 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:2239782:2240031 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:2239782:2240031 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:2239781:2240030 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:2239781:2240030 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:2239781:2240030 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:2239781:2240030 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:2239782:2240031 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:2239781:2240030 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:2239782:2240031 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:2239781:2240030 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:2239782:2240031 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:2239781:2240030 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:2239782:2240031 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:2239781:2240030 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:2239782:2240031 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:2239781:2240030 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:2239782:2240031 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:2239781:2240030 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:2239782:2240031 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:2239781:2240030 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:2239782:2240031 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:2239781:2240030 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:2239782:2240031 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:2239781:2240030 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:2239782:2240031 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:2239781:2240030 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:2239782:2240031 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:2239781:2240030 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:2239782:2240031 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:2239781:2240030 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:2239782:2240031 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:2239781:2240030 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:2239782:2240031 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:2239781:2240030 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:2239782:2240031 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:2239781:2240030 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:2239782:2240031 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:2239781:2240030 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:2239782:2240031 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:2239781:2240030 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:2239782:2240031 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:2239781:2240030 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:2239782:2240031 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:2239781:2240030 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:2239782:2240031 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:2239781:2240030 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:2239782:2240031 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:2239782:2240031 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:2239781:2240030 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:2239782:2240031 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:2239781:2240030 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:2239782:2240031 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:2239781:2240030 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:2239782:2240031 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:2239781:2240030 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:2239781:2240030 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:2239782:2240031 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2239781:2240030 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:2239781:2240030 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:2239781:2240030 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:2239781:2240030 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:2239781:2240030 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:2239781:2240030 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:2239781:2240030 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:2239781:2240030 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:2239781:2240030 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:2239781:2240030 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:2239781:2240030 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:2239781:2240030 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:2239781:2240030 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:2239781:2240030 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:2239781:2240030 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:2239781:2240030 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:2239781:2240030 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:2239781:2240030 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:2239781:2240030 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:2239781:2240030 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:2239781:2240030 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2239781:2240030 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2239781:2240030 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:2239781:2240042 [0] NCCL INFO [Proxy Service] Device 0 CPU core 107
n136-128-154:2239781:2240043 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 108
n136-128-154:2239782:2240031 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2239782:2240044 [1] NCCL INFO [Proxy Service] Device 1 CPU core 40
n136-128-154:2239782:2240045 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 41
n136-128-154:2239781:2240030 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2239782:2240031 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2239781:2240030 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2239782:2240031 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2239781:2240030 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:2239781:2240030 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2239781:2240030 [0] NCCL INFO ncclCommInitRankConfig comm 0x102eb980 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0xc1a41f07abad06fa - Init COMPLETE
n136-128-154:2239781:2240030 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.79 (kernels 0.12, alloc 0.43, bootstrap 0.00, allgathers 0.01, topo 0.07, graphs 0.00, connections 0.05, rest 0.12)
n136-128-154:2239782:2240031 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2239782:2240031 [1] NCCL INFO ncclCommInitRankConfig comm 0x10b4d040 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0xc1a41f07abad06fa - Init COMPLETE
n136-128-154:2239782:2240031 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.78 (kernels 0.11, alloc 0.42, bootstrap 0.00, allgathers 0.01, topo 0.07, graphs 0.00, connections 0.05, rest 0.12)
n136-128-154:2239781:2240046 [0] NCCL INFO Channel 00/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2239781:2240046 [0] NCCL INFO Channel 01/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2239781:2240046 [0] NCCL INFO Channel 02/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2239781:2240046 [0] NCCL INFO Channel 03/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2239782:2240047 [1] NCCL INFO Channel 00/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239781:2240046 [0] NCCL INFO Channel 04/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2239782:2240047 [1] NCCL INFO Channel 01/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239781:2240046 [0] NCCL INFO Channel 05/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2239782:2240047 [1] NCCL INFO Channel 02/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239781:2240046 [0] NCCL INFO Channel 06/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2239782:2240047 [1] NCCL INFO Channel 03/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239781:2240046 [0] NCCL INFO Channel 07/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2239781:2240046 [0] NCCL INFO Channel 08/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2239782:2240047 [1] NCCL INFO Channel 04/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239781:2240046 [0] NCCL INFO Channel 09/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2239782:2240047 [1] NCCL INFO Channel 05/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239781:2240046 [0] NCCL INFO Channel 10/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2239782:2240047 [1] NCCL INFO Channel 06/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239781:2240046 [0] NCCL INFO Channel 11/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2239782:2240047 [1] NCCL INFO Channel 07/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239781:2240046 [0] NCCL INFO Channel 12/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2239782:2240047 [1] NCCL INFO Channel 08/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239781:2240046 [0] NCCL INFO Channel 13/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2239782:2240047 [1] NCCL INFO Channel 09/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239781:2240046 [0] NCCL INFO Channel 14/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2239782:2240047 [1] NCCL INFO Channel 10/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239781:2240046 [0] NCCL INFO Channel 15/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2239782:2240047 [1] NCCL INFO Channel 11/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239781:2240046 [0] NCCL INFO Channel 16/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2239782:2240047 [1] NCCL INFO Channel 12/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239781:2240046 [0] NCCL INFO Channel 17/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2239782:2240047 [1] NCCL INFO Channel 13/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239781:2240046 [0] NCCL INFO Channel 18/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2239781:2240046 [0] NCCL INFO Channel 19/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2239782:2240047 [1] NCCL INFO Channel 14/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239781:2240046 [0] NCCL INFO Channel 20/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2239782:2240047 [1] NCCL INFO Channel 15/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239781:2240046 [0] NCCL INFO Channel 21/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2239782:2240047 [1] NCCL INFO Channel 16/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239781:2240046 [0] NCCL INFO Channel 22/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2239782:2240047 [1] NCCL INFO Channel 17/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239781:2240046 [0] NCCL INFO Channel 23/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2239782:2240047 [1] NCCL INFO Channel 18/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240047 [1] NCCL INFO Channel 19/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240047 [1] NCCL INFO Channel 20/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240047 [1] NCCL INFO Channel 21/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240047 [1] NCCL INFO Channel 22/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240047 [1] NCCL INFO Channel 23/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240047 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:2239781:2240046 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-09:19:12:15 INFO     [evaluator:559] Running loglikelihood requests
2025-12-09:19:12:15 INFO     [evaluator:559] Running loglikelihood requests

Running loglikelihood requests:   0%|          | 0/1268 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 64
Determined largest batch size: 64

Running loglikelihood requests:   0%|          | 1/1268 [00:05<1:54:05,  5.40s/it]
Running loglikelihood requests:   5%|▌         | 65/1268 [00:06<01:27, 13.81it/s] 
Running loglikelihood requests:  10%|█         | 129/1268 [00:07<00:43, 26.19it/s]
Running loglikelihood requests:  15%|█▌        | 193/1268 [00:08<00:28, 37.19it/s]
Running loglikelihood requests:  20%|██        | 257/1268 [00:08<00:21, 46.30it/s]
Running loglikelihood requests:  25%|██▌       | 321/1268 [00:09<00:17, 53.69it/s]
Running loglikelihood requests:  30%|███       | 385/1268 [00:10<00:14, 59.57it/s]
Running loglikelihood requests:  35%|███▌      | 449/1268 [00:11<00:12, 64.01it/s]
Running loglikelihood requests:  40%|████      | 513/1268 [00:12<00:11, 67.52it/s]
Running loglikelihood requests:  46%|████▌     | 577/1268 [00:13<00:09, 70.20it/s]
Running loglikelihood requests:  51%|█████     | 641/1268 [00:13<00:08, 72.45it/s]
Running loglikelihood requests:  56%|█████▌    | 705/1268 [00:14<00:07, 74.10it/s]
Running loglikelihood requests:  61%|██████    | 769/1268 [00:15<00:06, 75.46it/s]
Running loglikelihood requests:  66%|██████▌   | 833/1268 [00:16<00:05, 76.35it/s]
Running loglikelihood requests:  71%|███████   | 897/1268 [00:17<00:04, 77.05it/s]
Running loglikelihood requests:  76%|███████▌  | 961/1268 [00:18<00:03, 77.60it/s]
Running loglikelihood requests:  81%|████████  | 1025/1268 [00:18<00:03, 78.58it/s]
Running loglikelihood requests:  86%|████████▌ | 1089/1268 [00:19<00:02, 79.49it/s]
Running loglikelihood requests:  91%|█████████ | 1153/1268 [00:20<00:01, 80.37it/s]
Running loglikelihood requests:  96%|█████████▌| 1217/1268 [00:21<00:00, 85.72it/s]
Running loglikelihood requests: 100%|██████████| 1268/1268 [00:21<00:00, 60.34it/s]
n136-128-154:2239782:2240072 [1] NCCL INFO Channel 00/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240072 [1] NCCL INFO Channel 01/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240072 [1] NCCL INFO Channel 02/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240072 [1] NCCL INFO Channel 03/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240072 [1] NCCL INFO Channel 04/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240072 [1] NCCL INFO Channel 05/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240072 [1] NCCL INFO Channel 06/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240072 [1] NCCL INFO Channel 07/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240072 [1] NCCL INFO Channel 08/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240072 [1] NCCL INFO Channel 09/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240072 [1] NCCL INFO Channel 10/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240072 [1] NCCL INFO Channel 11/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240072 [1] NCCL INFO Channel 12/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240072 [1] NCCL INFO Channel 13/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240072 [1] NCCL INFO Channel 14/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240072 [1] NCCL INFO Channel 15/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240072 [1] NCCL INFO Channel 16/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240072 [1] NCCL INFO Channel 17/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240072 [1] NCCL INFO Channel 18/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240072 [1] NCCL INFO Channel 19/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240072 [1] NCCL INFO Channel 20/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240072 [1] NCCL INFO Channel 21/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240072 [1] NCCL INFO Channel 22/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240072 [1] NCCL INFO Channel 23/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240072 [1] NCCL INFO Channel 24/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240072 [1] NCCL INFO Channel 25/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240072 [1] NCCL INFO Channel 26/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240072 [1] NCCL INFO Channel 27/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240072 [1] NCCL INFO Channel 28/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240072 [1] NCCL INFO Channel 29/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240072 [1] NCCL INFO Channel 30/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2239782:2240072 [1] NCCL INFO Channel 31/1 : 1[7] -> 0[6] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:2239782:2240076 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2239782:2240076 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2239782:2240076 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2239782:2240076 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2239782:2240076 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2239782:2240076 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2239782:2240044 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-09:19:12:46 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: auto (64)
|  Tasks   |Version|Filter|n-shot|Metric|   |Value|   |Stderr|
|----------|------:|------|-----:|------|---|----:|---|-----:|
|winogrande|      1|none  |     5|acc   |↑  |0.498|±  |0.0141|

[rank0]:[W1209 19:12:47.537418135 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:2239781:2240127 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2239781:2240127 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2239781:2240127 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2239781:2240042 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2239781:2240127 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2239781:2240127 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2239781:2240127 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2239782:2240044 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2239782:2240076 [1] NCCL INFO comm 0x10b4d040 rank 1 nranks 2 cudaDev 1 busId c9000 - Abort COMPLETE
n136-128-154:2239781:2240127 [0] NCCL INFO comm 0x102eb980 rank 0 nranks 2 cudaDev 0 busId c5000 - Abort COMPLETE
任务 winogrande 评估完成！

==================================================
开始评估：任务=boolq | 少样本数=0 | 模型=Qwen2.5-7B-quantization-fg
输出路径：results2/Qwen2.5-7B-quantization-fg/f6/boolq.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-09:19:13:38 INFO     [__main__:440] Selected Tasks: ['boolq']
2025-12-09:19:13:38 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:19:13:38 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-09:19:13:38 INFO     [__main__:440] Selected Tasks: ['boolq']
2025-12-09:19:13:38 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:19:13:38 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-09:19:13:39 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:19:13:39 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:19:13:39 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.37it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.40it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.32it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.37it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.38it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.43it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.98it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.70it/s]
2025-12-09:19:13:42 WARNING  [api.task:844] [Task: boolq] metric acc is defined, but aggregation is not. using default aggregation=mean
2025-12-09:19:13:42 WARNING  [api.task:856] [Task: boolq] metric acc is defined, but higher_is_better is not. using default higher_is_better=True

Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  2.05it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.76it/s]
2025-12-09:19:13:42 WARNING  [api.task:844] [Task: boolq] metric acc is defined, but aggregation is not. using default aggregation=mean
2025-12-09:19:13:42 WARNING  [api.task:856] [Task: boolq] metric acc is defined, but higher_is_better is not. using default higher_is_better=True
2025-12-09:19:14:06 WARNING  [evaluator:309] Overwriting default num_fewshot of boolq from None to 0
2025-12-09:19:14:06 INFO     [api.task:434] Building contexts for boolq on rank 0...

  0%|          | 0/1635 [00:00<?, ?it/s]
 17%|█▋        | 280/1635 [00:00<00:00, 2795.17it/s]
 35%|███▍      | 565/1635 [00:00<00:00, 2822.57it/s]
 52%|█████▏    | 848/1635 [00:00<00:00, 2769.50it/s]
 69%|██████▉   | 1126/1635 [00:00<00:00, 2766.26it/s]
 86%|████████▌ | 1409/1635 [00:00<00:00, 2786.08it/s]
100%|██████████| 1635/1635 [00:00<00:00, 2791.98it/s]
n136-128-154:2240172:2240172 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2240172:2240172 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2240172:2240172 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:2240172:2240172 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2240172:2240172 [0] NCCL INFO Comm config Blocking set to 1
n136-128-154:2240172:2240375 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2240172:2240375 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2240172:2240375 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2240172:2240375 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2240172:2240375 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2240172:2240375 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2240172:2240375 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2240172:2240375 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:2240172:2240375 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2240172:2240375 [0] NCCL INFO Using network IB
n136-128-154:2240172:2240375 [0] NCCL INFO ncclCommInitRankConfig comm 0x11218200 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0xe55dfc9887a40091 - Init START
2025-12-09:19:14:08 WARNING  [evaluator:309] Overwriting default num_fewshot of boolq from None to 0
2025-12-09:19:14:08 INFO     [api.task:434] Building contexts for boolq on rank 1...

  0%|          | 0/1635 [00:00<?, ?it/s]
 17%|█▋        | 280/1635 [00:00<00:00, 2796.50it/s]
 35%|███▍      | 565/1635 [00:00<00:00, 2827.26it/s]
 52%|█████▏    | 850/1635 [00:00<00:00, 2833.93it/s]
 69%|██████▉   | 1134/1635 [00:00<00:00, 2807.29it/s]
 87%|████████▋ | 1419/1635 [00:00<00:00, 2822.45it/s]
100%|██████████| 1635/1635 [00:00<00:00, 2826.18it/s]
n136-128-154:2240173:2240173 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:2240173:2240173 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2240173:2240173 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2240173:2240173 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2240173:2240173 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:2240173:2240390 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2240173:2240390 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2240173:2240390 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2240173:2240390 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2240173:2240390 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2240173:2240390 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2240173:2240390 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2240173:2240390 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:2240173:2240390 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2240173:2240390 [1] NCCL INFO Using network IB
n136-128-154:2240173:2240390 [1] NCCL INFO ncclCommInitRankConfig comm 0xfab04c0 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0xe55dfc9887a40091 - Init START
n136-128-154:2240173:2240390 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2240172:2240375 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2240173:2240390 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2240173:2240390 [1] NCCL INFO Retrieving state for IB
n136-128-154:2240173:2240390 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:2240173:2240390 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2240172:2240375 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2240172:2240375 [0] NCCL INFO Retrieving state for IB
n136-128-154:2240172:2240375 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:2240173:2240390 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2240172:2240375 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2240173:2240390 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2240172:2240375 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2240173:2240390 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2240172:2240375 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2240172:2240375 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2240173:2240390 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2240173:2240390 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2240172:2240375 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2240172:2240375 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2240172:2240375 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2240172:2240375 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2240172:2240375 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2240172:2240375 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2240172:2240375 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2240172:2240375 [0] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2240172:2240375 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2240172:2240375 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2240172:2240375 [0] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2240172:2240375 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2240172:2240375 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2240172:2240375 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2240172:2240375 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2240172:2240375 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2240172:2240375 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2240172:2240375 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2240172:2240375 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2240172:2240375 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2240172:2240375 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2240172:2240375 [0] NCCL INFO ==========================================
n136-128-154:2240172:2240375 [0] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2240172:2240375 [0] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2240172:2240375 [0] NCCL INFO Setting affinity for GPU 6 to 33-62,97-126
n136-128-154:2240173:2240390 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2240173:2240390 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2240173:2240390 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2240173:2240390 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2240173:2240390 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2240173:2240390 [1] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2240173:2240390 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2240173:2240390 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2240173:2240390 [1] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2240173:2240390 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2240173:2240390 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2240173:2240390 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2240173:2240390 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2240173:2240390 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2240173:2240390 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2240173:2240390 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2240173:2240390 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2240173:2240390 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2240173:2240390 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2240173:2240390 [1] NCCL INFO ==========================================
n136-128-154:2240173:2240390 [1] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2240173:2240390 [1] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2240173:2240390 [1] NCCL INFO Setting affinity for GPU 7 to 33-62,97-126
n136-128-154:2240172:2240375 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2240172:2240375 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240172:2240375 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240172:2240375 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240172:2240375 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240172:2240375 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240172:2240375 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240172:2240375 [0] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240172:2240375 [0] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240172:2240375 [0] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240172:2240375 [0] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240172:2240375 [0] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240172:2240375 [0] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240172:2240375 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2240172:2240375 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240172:2240375 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240172:2240375 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240172:2240375 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240172:2240375 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240172:2240375 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240172:2240375 [0] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2240172:2240375 [0] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2240172:2240375 [0] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2240172:2240375 [0] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2240172:2240375 [0] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2240172:2240375 [0] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2240173:2240390 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2240173:2240390 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240173:2240390 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240173:2240390 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240173:2240390 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240173:2240390 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240173:2240390 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240173:2240390 [1] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240173:2240390 [1] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240173:2240390 [1] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240173:2240390 [1] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240173:2240390 [1] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240173:2240390 [1] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240173:2240390 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2240173:2240390 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240173:2240390 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240173:2240390 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240173:2240390 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240173:2240390 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240173:2240390 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240173:2240390 [1] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2240173:2240390 [1] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2240173:2240390 [1] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2240173:2240390 [1] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2240173:2240390 [1] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2240173:2240390 [1] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2240172:2240375 [0] NCCL INFO comm 0x11218200 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:2240173:2240390 [1] NCCL INFO comm 0xfab04c0 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:2240172:2240375 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:2240172:2240375 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:2240173:2240390 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:2240172:2240375 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:2240173:2240390 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:2240172:2240375 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:2240173:2240390 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:2240172:2240375 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:2240173:2240390 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:2240172:2240375 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:2240173:2240390 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:2240172:2240375 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:2240173:2240390 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:2240172:2240375 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:2240173:2240390 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:2240172:2240375 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:2240173:2240390 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:2240172:2240375 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:2240173:2240390 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:2240172:2240375 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:2240173:2240390 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:2240172:2240375 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:2240173:2240390 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:2240172:2240375 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:2240173:2240390 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:2240172:2240375 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:2240173:2240390 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:2240172:2240375 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:2240173:2240390 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:2240172:2240375 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:2240173:2240390 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:2240172:2240375 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:2240173:2240390 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:2240172:2240375 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:2240173:2240390 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:2240172:2240375 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:2240173:2240390 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:2240172:2240375 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:2240173:2240390 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:2240172:2240375 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:2240173:2240390 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:2240172:2240375 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:2240173:2240390 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:2240172:2240375 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:2240173:2240390 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:2240172:2240375 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:2240173:2240390 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:2240173:2240390 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:2240172:2240375 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:2240172:2240375 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:2240172:2240375 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:2240172:2240375 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:2240173:2240390 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:2240172:2240375 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:2240173:2240390 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:2240172:2240375 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:2240173:2240390 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:2240172:2240375 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:2240173:2240390 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:2240172:2240375 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:2240173:2240390 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:2240172:2240375 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:2240173:2240390 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:2240172:2240375 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:2240173:2240390 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:2240172:2240375 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:2240173:2240390 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:2240172:2240375 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:2240173:2240390 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:2240172:2240375 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:2240173:2240390 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:2240172:2240375 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:2240173:2240390 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:2240172:2240375 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:2240173:2240390 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:2240172:2240375 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:2240173:2240390 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:2240172:2240375 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:2240173:2240390 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:2240172:2240375 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:2240173:2240390 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:2240172:2240375 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:2240173:2240390 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:2240172:2240375 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:2240173:2240390 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:2240172:2240375 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:2240173:2240390 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:2240172:2240375 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:2240173:2240390 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:2240172:2240375 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:2240173:2240390 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:2240172:2240375 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:2240173:2240390 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:2240172:2240375 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:2240173:2240390 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:2240172:2240375 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:2240173:2240390 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:2240172:2240375 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:2240173:2240390 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:2240172:2240375 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:2240173:2240390 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:2240172:2240375 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:2240173:2240390 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2240172:2240375 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:2240172:2240375 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:2240172:2240375 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:2240172:2240375 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:2240172:2240375 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:2240172:2240375 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:2240172:2240375 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:2240172:2240375 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:2240172:2240375 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:2240172:2240375 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:2240172:2240375 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:2240172:2240375 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:2240172:2240375 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:2240172:2240375 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:2240172:2240375 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:2240172:2240375 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:2240172:2240375 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:2240172:2240375 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:2240172:2240375 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:2240172:2240375 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:2240172:2240375 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2240173:2240390 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2240173:2240397 [1] NCCL INFO [Proxy Service] Device 1 CPU core 119
n136-128-154:2240173:2240398 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 57
n136-128-154:2240172:2240375 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2240172:2240375 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:2240172:2240399 [0] NCCL INFO [Proxy Service] Device 0 CPU core 107
n136-128-154:2240172:2240400 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 109
n136-128-154:2240173:2240390 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2240173:2240390 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2240172:2240375 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2240172:2240375 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2240172:2240375 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:2240173:2240390 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2240173:2240390 [1] NCCL INFO ncclCommInitRankConfig comm 0xfab04c0 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0xe55dfc9887a40091 - Init COMPLETE
n136-128-154:2240173:2240390 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.50 (kernels 0.11, alloc 0.20, bootstrap 0.00, allgathers 0.00, topo 0.06, graphs 0.00, connections 0.01, rest 0.12)
n136-128-154:2240172:2240375 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2240172:2240375 [0] NCCL INFO ncclCommInitRankConfig comm 0x11218200 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0xe55dfc9887a40091 - Init COMPLETE
n136-128-154:2240172:2240375 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 2.12 (kernels 0.19, alloc 0.26, bootstrap 1.47, allgathers 0.00, topo 0.06, graphs 0.00, connections 0.01, rest 0.12)
n136-128-154:2240173:2240401 [1] NCCL INFO Channel 00/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240173:2240401 [1] NCCL INFO Channel 01/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240172:2240402 [0] NCCL INFO Channel 00/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240173:2240401 [1] NCCL INFO Channel 02/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240172:2240402 [0] NCCL INFO Channel 01/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240173:2240401 [1] NCCL INFO Channel 03/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240172:2240402 [0] NCCL INFO Channel 02/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240173:2240401 [1] NCCL INFO Channel 04/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240172:2240402 [0] NCCL INFO Channel 03/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240173:2240401 [1] NCCL INFO Channel 05/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240172:2240402 [0] NCCL INFO Channel 04/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240173:2240401 [1] NCCL INFO Channel 06/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240172:2240402 [0] NCCL INFO Channel 05/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240173:2240401 [1] NCCL INFO Channel 07/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240172:2240402 [0] NCCL INFO Channel 06/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240173:2240401 [1] NCCL INFO Channel 08/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240172:2240402 [0] NCCL INFO Channel 07/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240173:2240401 [1] NCCL INFO Channel 09/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240172:2240402 [0] NCCL INFO Channel 08/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240173:2240401 [1] NCCL INFO Channel 10/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240172:2240402 [0] NCCL INFO Channel 09/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240173:2240401 [1] NCCL INFO Channel 11/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240172:2240402 [0] NCCL INFO Channel 10/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240173:2240401 [1] NCCL INFO Channel 12/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240172:2240402 [0] NCCL INFO Channel 11/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240173:2240401 [1] NCCL INFO Channel 13/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240172:2240402 [0] NCCL INFO Channel 12/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240173:2240401 [1] NCCL INFO Channel 14/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240172:2240402 [0] NCCL INFO Channel 13/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240173:2240401 [1] NCCL INFO Channel 15/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240172:2240402 [0] NCCL INFO Channel 14/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240173:2240401 [1] NCCL INFO Channel 16/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240172:2240402 [0] NCCL INFO Channel 15/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240173:2240401 [1] NCCL INFO Channel 17/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240172:2240402 [0] NCCL INFO Channel 16/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240173:2240401 [1] NCCL INFO Channel 18/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240172:2240402 [0] NCCL INFO Channel 17/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240173:2240401 [1] NCCL INFO Channel 19/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240172:2240402 [0] NCCL INFO Channel 18/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240173:2240401 [1] NCCL INFO Channel 20/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240172:2240402 [0] NCCL INFO Channel 19/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240173:2240401 [1] NCCL INFO Channel 21/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240172:2240402 [0] NCCL INFO Channel 20/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240173:2240401 [1] NCCL INFO Channel 22/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240172:2240402 [0] NCCL INFO Channel 21/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240173:2240401 [1] NCCL INFO Channel 23/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240172:2240402 [0] NCCL INFO Channel 22/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240172:2240402 [0] NCCL INFO Channel 23/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240173:2240401 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:2240172:2240402 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-09:19:14:09 INFO     [evaluator:559] Running loglikelihood requests
2025-12-09:19:14:09 INFO     [evaluator:559] Running loglikelihood requests

Running loglikelihood requests:   0%|          | 0/3270 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 45
Determined largest batch size: 45

Running loglikelihood requests:   0%|          | 1/3270 [00:55<50:01:44, 55.09s/it]
Running loglikelihood requests:   3%|▎         | 91/3270 [00:56<23:13,  2.28it/s]  
Running loglikelihood requests:   6%|▌         | 181/3270 [00:57<09:41,  5.31it/s]
Running loglikelihood requests:   8%|▊         | 271/3270 [00:58<05:22,  9.31it/s]
Running loglikelihood requests:  11%|█         | 361/3270 [00:59<03:21, 14.46it/s]
Running loglikelihood requests:  14%|█▍        | 451/3270 [01:00<02:14, 20.95it/s]
Running loglikelihood requests:  17%|█▋        | 541/3270 [01:00<01:34, 28.82it/s]
Running loglikelihood requests:  19%|█▉        | 631/3270 [01:01<01:09, 38.02it/s]
Running loglikelihood requests:  22%|██▏       | 721/3270 [01:02<00:52, 48.31it/s]
Running loglikelihood requests:  25%|██▍       | 811/3270 [01:03<00:41, 59.22it/s]
Running loglikelihood requests:  28%|██▊       | 901/3270 [01:04<00:33, 70.43it/s]
Running loglikelihood requests:  30%|███       | 991/3270 [01:04<00:28, 81.01it/s]
Running loglikelihood requests:  33%|███▎      | 1081/3270 [01:05<00:24, 90.89it/s]
Running loglikelihood requests:  36%|███▌      | 1171/3270 [01:06<00:21, 99.71it/s]
Running loglikelihood requests:  39%|███▊      | 1261/3270 [01:06<00:18, 108.07it/s]
Running loglikelihood requests:  41%|████▏     | 1351/3270 [01:07<00:16, 115.36it/s]
Running loglikelihood requests:  44%|████▍     | 1441/3270 [01:08<00:15, 121.72it/s]
Running loglikelihood requests:  47%|████▋     | 1531/3270 [01:08<00:13, 127.11it/s]
Running loglikelihood requests:  50%|████▉     | 1621/3270 [01:09<00:12, 131.85it/s]
Running loglikelihood requests:  52%|█████▏    | 1711/3270 [01:10<00:11, 136.99it/s]
Running loglikelihood requests:  55%|█████▌    | 1801/3270 [01:10<00:10, 141.65it/s]
Running loglikelihood requests:  58%|█████▊    | 1891/3270 [01:11<00:09, 145.52it/s]
Running loglikelihood requests:  61%|██████    | 1981/3270 [01:11<00:08, 149.08it/s]
Running loglikelihood requests:  63%|██████▎   | 2071/3270 [01:12<00:07, 152.30it/s]
Running loglikelihood requests:  66%|██████▌   | 2161/3270 [01:12<00:07, 155.23it/s]
Running loglikelihood requests:  69%|██████▉   | 2251/3270 [01:13<00:06, 159.70it/s]
Running loglikelihood requests:  72%|███████▏  | 2341/3270 [01:13<00:05, 163.50it/s]
Running loglikelihood requests:  74%|███████▍  | 2431/3270 [01:14<00:05, 167.67it/s]
Running loglikelihood requests:  77%|███████▋  | 2521/3270 [01:14<00:04, 171.54it/s]
Running loglikelihood requests:  80%|███████▉  | 2611/3270 [01:15<00:03, 175.09it/s]
Running loglikelihood requests:  83%|████████▎ | 2701/3270 [01:15<00:03, 179.08it/s]
Running loglikelihood requests:  85%|████████▌ | 2791/3270 [01:16<00:02, 185.06it/s]
Running loglikelihood requests:  88%|████████▊ | 2881/3270 [01:16<00:02, 191.36it/s]
Running loglikelihood requests:  91%|█████████ | 2971/3270 [01:17<00:01, 198.42it/s]
Running loglikelihood requests:  94%|█████████▎| 3061/3270 [01:17<00:01, 206.97it/s]
Running loglikelihood requests:  96%|█████████▋| 3151/3270 [01:17<00:00, 216.23it/s]
Running loglikelihood requests:  99%|█████████▉| 3241/3270 [01:18<00:00, 272.26it/s]
Running loglikelihood requests: 100%|██████████| 3270/3270 [01:18<00:00, 41.86it/s] 
n136-128-154:2240173:2240489 [1] NCCL INFO Channel 00/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240173:2240489 [1] NCCL INFO Channel 01/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240173:2240489 [1] NCCL INFO Channel 02/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240173:2240489 [1] NCCL INFO Channel 03/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240173:2240489 [1] NCCL INFO Channel 04/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240173:2240489 [1] NCCL INFO Channel 05/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240173:2240489 [1] NCCL INFO Channel 06/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240173:2240489 [1] NCCL INFO Channel 07/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240173:2240489 [1] NCCL INFO Channel 08/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240173:2240489 [1] NCCL INFO Channel 09/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240173:2240489 [1] NCCL INFO Channel 10/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240173:2240489 [1] NCCL INFO Channel 11/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240173:2240489 [1] NCCL INFO Channel 12/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240173:2240489 [1] NCCL INFO Channel 13/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240173:2240489 [1] NCCL INFO Channel 14/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240173:2240489 [1] NCCL INFO Channel 15/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240173:2240489 [1] NCCL INFO Channel 16/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240173:2240489 [1] NCCL INFO Channel 17/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240173:2240489 [1] NCCL INFO Channel 18/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240173:2240489 [1] NCCL INFO Channel 19/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240173:2240489 [1] NCCL INFO Channel 20/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240173:2240489 [1] NCCL INFO Channel 21/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240173:2240489 [1] NCCL INFO Channel 22/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240173:2240489 [1] NCCL INFO Channel 23/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240173:2240489 [1] NCCL INFO Channel 24/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240173:2240489 [1] NCCL INFO Channel 25/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240173:2240489 [1] NCCL INFO Channel 26/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240173:2240489 [1] NCCL INFO Channel 27/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240173:2240489 [1] NCCL INFO Channel 28/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240173:2240489 [1] NCCL INFO Channel 29/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240173:2240489 [1] NCCL INFO Channel 30/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240173:2240489 [1] NCCL INFO Channel 31/1 : 1[7] -> 0[6] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:2240173:2240493 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2240173:2240493 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2240173:2240493 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2240173:2240493 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2240173:2240493 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2240173:2240493 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2240173:2240397 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-09:19:15:39 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: auto (45)
|Tasks|Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|-----|------:|------|-----:|------|---|-----:|---|-----:|
|boolq|      2|none  |     0|acc   |↑  |0.3801|±  |0.0085|

[rank0]:[W1209 19:15:39.023581183 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:2240172:2240544 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2240172:2240544 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2240172:2240544 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2240172:2240399 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2240172:2240544 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2240172:2240544 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2240172:2240544 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2240173:2240397 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2240173:2240493 [1] NCCL INFO comm 0xfab04c0 rank 1 nranks 2 cudaDev 1 busId c9000 - Abort COMPLETE
n136-128-154:2240172:2240544 [0] NCCL INFO comm 0x11218200 rank 0 nranks 2 cudaDev 0 busId c5000 - Abort COMPLETE
任务 boolq 评估完成！

==================================================
开始评估：任务=arc_challenge | 少样本数=25 | 模型=Qwen2.5-7B-quantization-fg
输出路径：results2/Qwen2.5-7B-quantization-fg/f6/arc_challenge.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-09:19:16:29 INFO     [__main__:440] Selected Tasks: ['arc_challenge']
2025-12-09:19:16:29 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:19:16:29 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-09:19:16:29 INFO     [__main__:440] Selected Tasks: ['arc_challenge']
2025-12-09:19:16:29 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:19:16:29 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-09:19:16:30 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:19:16:30 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:19:16:30 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.13it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.13it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.12it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.12it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.16it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.15it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.70it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.44it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.70it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.44it/s]
2025-12-09:19:16:48 WARNING  [evaluator:309] Overwriting default num_fewshot of arc_challenge from None to 25
2025-12-09:19:16:48 INFO     [api.task:434] Building contexts for arc_challenge on rank 0...

  0%|          | 0/586 [00:00<?, ?it/s]
  1%|          | 4/586 [00:00<00:16, 35.72it/s]
  1%|▏         | 8/586 [00:00<00:15, 36.45it/s]
  2%|▏         | 12/586 [00:00<00:15, 36.52it/s]
  3%|▎         | 16/586 [00:00<00:15, 36.57it/s]
  3%|▎         | 20/586 [00:00<00:15, 36.65it/s]
  4%|▍         | 24/586 [00:00<00:15, 36.73it/s]
  5%|▍         | 28/586 [00:00<00:15, 36.83it/s]
  5%|▌         | 32/586 [00:00<00:15, 36.87it/s]
  6%|▌         | 36/586 [00:00<00:15, 36.43it/s]
  7%|▋         | 40/586 [00:01<00:14, 36.58it/s]
  8%|▊         | 44/586 [00:01<00:14, 36.43it/s]2025-12-09:19:16:50 WARNING  [evaluator:309] Overwriting default num_fewshot of arc_challenge from None to 25
2025-12-09:19:16:50 INFO     [api.task:434] Building contexts for arc_challenge on rank 1...

  0%|          | 0/586 [00:00<?, ?it/s]
  8%|▊         | 48/586 [00:01<00:14, 36.61it/s]
  1%|          | 4/586 [00:00<00:16, 35.64it/s]
  9%|▉         | 52/586 [00:01<00:14, 36.75it/s]
  1%|▏         | 8/586 [00:00<00:16, 35.83it/s]
 10%|▉         | 56/586 [00:01<00:14, 36.77it/s]
  2%|▏         | 12/586 [00:00<00:15, 36.06it/s]
 10%|█         | 60/586 [00:01<00:14, 36.31it/s]
  3%|▎         | 16/586 [00:00<00:15, 36.39it/s]
 11%|█         | 64/586 [00:01<00:14, 36.47it/s]
  3%|▎         | 20/586 [00:00<00:15, 36.51it/s]
 12%|█▏        | 68/586 [00:01<00:14, 36.63it/s]
  4%|▍         | 24/586 [00:00<00:15, 36.60it/s]
 12%|█▏        | 72/586 [00:01<00:13, 36.79it/s]
  5%|▍         | 28/586 [00:00<00:15, 36.31it/s]
 13%|█▎        | 76/586 [00:02<00:13, 36.88it/s]
  5%|▌         | 32/586 [00:00<00:15, 36.50it/s]
 14%|█▎        | 80/586 [00:02<00:13, 36.81it/s]
  6%|▌         | 36/586 [00:00<00:15, 36.37it/s]
 14%|█▍        | 84/586 [00:02<00:13, 36.95it/s]
  7%|▋         | 40/586 [00:01<00:14, 36.55it/s]
 15%|█▌        | 88/586 [00:02<00:13, 37.02it/s]
  8%|▊         | 44/586 [00:01<00:14, 36.69it/s]
 16%|█▌        | 92/586 [00:02<00:13, 37.06it/s]
  8%|▊         | 48/586 [00:01<00:14, 36.80it/s]
 16%|█▋        | 96/586 [00:02<00:13, 37.11it/s]
  9%|▉         | 52/586 [00:01<00:14, 36.41it/s]
 17%|█▋        | 100/586 [00:02<00:13, 37.12it/s]
 10%|▉         | 56/586 [00:01<00:14, 36.49it/s]
 18%|█▊        | 104/586 [00:02<00:12, 37.11it/s]
 10%|█         | 60/586 [00:01<00:14, 36.40it/s]
 18%|█▊        | 108/586 [00:02<00:13, 36.49it/s]
 11%|█         | 64/586 [00:01<00:14, 36.33it/s]
 19%|█▉        | 112/586 [00:03<00:12, 36.49it/s]
 12%|█▏        | 68/586 [00:01<00:14, 36.37it/s]
 20%|█▉        | 116/586 [00:03<00:12, 36.51it/s]
 12%|█▏        | 72/586 [00:01<00:14, 36.47it/s]
 20%|██        | 120/586 [00:03<00:12, 36.62it/s]
 13%|█▎        | 76/586 [00:02<00:14, 36.14it/s]
 21%|██        | 124/586 [00:03<00:12, 36.53it/s]
 14%|█▎        | 80/586 [00:02<00:13, 36.16it/s]
 22%|██▏       | 128/586 [00:03<00:12, 36.43it/s]
 14%|█▍        | 84/586 [00:02<00:13, 36.17it/s]
 23%|██▎       | 132/586 [00:03<00:12, 36.35it/s]
 15%|█▌        | 88/586 [00:02<00:13, 36.18it/s]
 23%|██▎       | 136/586 [00:03<00:12, 36.27it/s]
 16%|█▌        | 92/586 [00:02<00:13, 36.22it/s]
 24%|██▍       | 140/586 [00:03<00:12, 35.85it/s]
 16%|█▋        | 96/586 [00:02<00:13, 35.88it/s]
 25%|██▍       | 144/586 [00:03<00:12, 35.92it/s]
 17%|█▋        | 100/586 [00:02<00:13, 35.89it/s]
 25%|██▌       | 148/586 [00:04<00:12, 35.98it/s]
 18%|█▊        | 104/586 [00:02<00:13, 35.99it/s]
 26%|██▌       | 152/586 [00:04<00:12, 36.07it/s]
 18%|█▊        | 108/586 [00:02<00:13, 36.05it/s]
 27%|██▋       | 156/586 [00:04<00:11, 36.13it/s]
 19%|█▉        | 112/586 [00:03<00:13, 36.14it/s]
 27%|██▋       | 160/586 [00:04<00:11, 36.17it/s]
 20%|█▉        | 116/586 [00:03<00:12, 36.35it/s]
 28%|██▊       | 164/586 [00:04<00:11, 36.41it/s]
 20%|██        | 120/586 [00:03<00:12, 36.55it/s]
 29%|██▊       | 168/586 [00:04<00:11, 36.62it/s]
 21%|██        | 124/586 [00:03<00:12, 36.68it/s]
 29%|██▉       | 172/586 [00:04<00:11, 36.73it/s]
 22%|██▏       | 128/586 [00:03<00:12, 36.77it/s]
 30%|███       | 176/586 [00:04<00:11, 36.89it/s]
 31%|███       | 180/586 [00:04<00:10, 37.02it/s]
 23%|██▎       | 132/586 [00:03<00:12, 36.41it/s]
 31%|███▏      | 184/586 [00:05<00:10, 37.11it/s]
 23%|██▎       | 136/586 [00:03<00:12, 36.57it/s]
 32%|███▏      | 188/586 [00:05<00:10, 37.15it/s]
 24%|██▍       | 140/586 [00:03<00:12, 36.73it/s]
 33%|███▎      | 192/586 [00:05<00:10, 37.13it/s]
 25%|██▍       | 144/586 [00:03<00:12, 36.71it/s]
 33%|███▎      | 196/586 [00:05<00:10, 37.06it/s]
 25%|██▌       | 148/586 [00:04<00:12, 36.44it/s]
 34%|███▍      | 200/586 [00:05<00:10, 37.05it/s]
 26%|██▌       | 152/586 [00:04<00:11, 36.32it/s]
 35%|███▍      | 204/586 [00:05<00:10, 37.09it/s]
 27%|██▋       | 156/586 [00:04<00:11, 36.22it/s]
 35%|███▌      | 208/586 [00:05<00:10, 37.00it/s]
 27%|██▋       | 160/586 [00:04<00:11, 36.06it/s]
 36%|███▌      | 212/586 [00:05<00:10, 37.05it/s]
 28%|██▊       | 164/586 [00:04<00:11, 36.08it/s]
 37%|███▋      | 216/586 [00:05<00:09, 37.08it/s]
 29%|██▊       | 168/586 [00:04<00:11, 36.30it/s]
 38%|███▊      | 220/586 [00:05<00:09, 37.03it/s]
 29%|██▉       | 172/586 [00:04<00:11, 36.45it/s]
 38%|███▊      | 224/586 [00:06<00:09, 37.07it/s]
 30%|███       | 176/586 [00:04<00:11, 36.59it/s]
 39%|███▉      | 228/586 [00:06<00:09, 37.10it/s]
 31%|███       | 180/586 [00:04<00:11, 36.72it/s]
 40%|███▉      | 232/586 [00:06<00:09, 36.58it/s]
 31%|███▏      | 184/586 [00:05<00:10, 36.79it/s]
 40%|████      | 236/586 [00:06<00:09, 36.68it/s]
 32%|███▏      | 188/586 [00:05<00:10, 36.85it/s]
 41%|████      | 240/586 [00:06<00:09, 36.82it/s]
 33%|███▎      | 192/586 [00:05<00:10, 36.88it/s]
 42%|████▏     | 244/586 [00:06<00:09, 36.86it/s]
 33%|███▎      | 196/586 [00:05<00:10, 36.85it/s]
 42%|████▏     | 248/586 [00:06<00:09, 36.92it/s]
 34%|███▍      | 200/586 [00:05<00:10, 36.85it/s]
 43%|████▎     | 252/586 [00:06<00:09, 36.96it/s]
 35%|███▍      | 204/586 [00:05<00:10, 36.82it/s]
 44%|████▎     | 256/586 [00:06<00:08, 36.99it/s]
 35%|███▌      | 208/586 [00:05<00:10, 36.83it/s]
 44%|████▍     | 260/586 [00:07<00:08, 37.04it/s]
 36%|███▌      | 212/586 [00:05<00:10, 36.81it/s]
 45%|████▌     | 264/586 [00:07<00:08, 37.03it/s]
 37%|███▋      | 216/586 [00:05<00:10, 36.85it/s]
 46%|████▌     | 268/586 [00:07<00:08, 37.03it/s]
 38%|███▊      | 220/586 [00:06<00:10, 36.28it/s]
 46%|████▋     | 272/586 [00:07<00:08, 37.00it/s]
 38%|███▊      | 224/586 [00:06<00:09, 36.39it/s]
 47%|████▋     | 276/586 [00:07<00:08, 37.06it/s]
 39%|███▉      | 228/586 [00:06<00:09, 36.47it/s]
 48%|████▊     | 280/586 [00:07<00:08, 37.08it/s]
 40%|███▉      | 232/586 [00:06<00:09, 36.59it/s]
 48%|████▊     | 284/586 [00:07<00:08, 37.09it/s]
 40%|████      | 236/586 [00:06<00:09, 36.38it/s]
 49%|████▉     | 288/586 [00:07<00:08, 37.11it/s]
 41%|████      | 240/586 [00:06<00:09, 36.35it/s]
 50%|████▉     | 292/586 [00:07<00:07, 37.10it/s]
 42%|████▏     | 244/586 [00:06<00:09, 36.36it/s]
 51%|█████     | 296/586 [00:08<00:07, 36.64it/s]
 42%|████▏     | 248/586 [00:06<00:09, 36.39it/s]
 51%|█████     | 300/586 [00:08<00:07, 36.73it/s]
 43%|████▎     | 252/586 [00:06<00:09, 36.43it/s]
 52%|█████▏    | 304/586 [00:08<00:07, 36.80it/s]
 44%|████▎     | 256/586 [00:07<00:09, 36.27it/s]
 53%|█████▎    | 308/586 [00:08<00:07, 36.94it/s]
 44%|████▍     | 260/586 [00:07<00:08, 36.35it/s]
 53%|█████▎    | 312/586 [00:08<00:07, 37.01it/s]
 45%|████▌     | 264/586 [00:07<00:08, 36.43it/s]
 54%|█████▍    | 316/586 [00:08<00:07, 36.99it/s]
 46%|████▌     | 268/586 [00:07<00:08, 36.48it/s]
 55%|█████▍    | 320/586 [00:08<00:07, 36.98it/s]
 46%|████▋     | 272/586 [00:07<00:08, 36.46it/s]
 55%|█████▌    | 324/586 [00:08<00:07, 37.04it/s]
 47%|████▋     | 276/586 [00:07<00:08, 36.48it/s]
 56%|█████▌    | 328/586 [00:08<00:06, 37.04it/s]
 48%|████▊     | 280/586 [00:07<00:08, 36.21it/s]
 57%|█████▋    | 332/586 [00:09<00:06, 37.01it/s]
 48%|████▊     | 284/586 [00:07<00:08, 36.17it/s]
 57%|█████▋    | 336/586 [00:09<00:06, 37.03it/s]
 49%|████▉     | 288/586 [00:07<00:08, 36.21it/s]
 58%|█████▊    | 340/586 [00:09<00:06, 37.09it/s]
 50%|████▉     | 292/586 [00:08<00:08, 36.25it/s]
 59%|█████▊    | 344/586 [00:09<00:06, 37.07it/s]
 51%|█████     | 296/586 [00:08<00:07, 36.37it/s]
 59%|█████▉    | 348/586 [00:09<00:06, 37.08it/s]
 51%|█████     | 300/586 [00:08<00:07, 36.45it/s]
 60%|██████    | 352/586 [00:09<00:06, 37.11it/s]
 52%|█████▏    | 304/586 [00:08<00:07, 36.49it/s]
 61%|██████    | 356/586 [00:09<00:06, 37.09it/s]
 53%|█████▎    | 308/586 [00:08<00:07, 36.52it/s]
 61%|██████▏   | 360/586 [00:09<00:06, 37.13it/s]
 53%|█████▎    | 312/586 [00:08<00:07, 36.55it/s]
 62%|██████▏   | 364/586 [00:09<00:05, 37.14it/s]
 54%|█████▍    | 316/586 [00:08<00:07, 36.58it/s]
 63%|██████▎   | 368/586 [00:09<00:05, 37.20it/s]
 55%|█████▍    | 320/586 [00:08<00:07, 36.64it/s]
 63%|██████▎   | 372/586 [00:10<00:05, 37.16it/s]
 55%|█████▌    | 324/586 [00:08<00:07, 36.70it/s]
 64%|██████▍   | 376/586 [00:10<00:05, 37.14it/s]
 56%|█████▌    | 328/586 [00:09<00:07, 36.64it/s]
 65%|██████▍   | 380/586 [00:10<00:05, 37.08it/s]
 57%|█████▋    | 332/586 [00:09<00:06, 36.63it/s]
 66%|██████▌   | 384/586 [00:10<00:05, 37.06it/s]
 57%|█████▋    | 336/586 [00:09<00:06, 36.60it/s]
 66%|██████▌   | 388/586 [00:10<00:05, 37.05it/s]
 58%|█████▊    | 340/586 [00:09<00:06, 36.51it/s]
 67%|██████▋   | 392/586 [00:10<00:05, 37.06it/s]
 59%|█████▊    | 344/586 [00:09<00:06, 36.44it/s]
 68%|██████▊   | 396/586 [00:10<00:05, 36.94it/s]
 59%|█████▉    | 348/586 [00:09<00:06, 36.40it/s]
 68%|██████▊   | 400/586 [00:10<00:05, 36.93it/s]
 60%|██████    | 352/586 [00:09<00:06, 36.46it/s]
 69%|██████▉   | 404/586 [00:10<00:04, 36.93it/s]
 61%|██████    | 356/586 [00:09<00:06, 36.32it/s]
 70%|██████▉   | 408/586 [00:11<00:04, 36.92it/s]
 61%|██████▏   | 360/586 [00:09<00:06, 36.35it/s]
 70%|███████   | 412/586 [00:11<00:04, 36.89it/s]
 62%|██████▏   | 364/586 [00:09<00:06, 36.40it/s]
 71%|███████   | 416/586 [00:11<00:04, 36.92it/s]
 63%|██████▎   | 368/586 [00:10<00:05, 36.47it/s]
 72%|███████▏  | 420/586 [00:11<00:04, 36.85it/s]
 63%|██████▎   | 372/586 [00:10<00:05, 36.49it/s]
 72%|███████▏  | 424/586 [00:11<00:04, 36.83it/s]
 64%|██████▍   | 376/586 [00:10<00:05, 36.17it/s]
 73%|███████▎  | 428/586 [00:11<00:04, 36.72it/s]
 65%|██████▍   | 380/586 [00:10<00:05, 36.19it/s]
 74%|███████▎  | 432/586 [00:11<00:04, 36.65it/s]
 66%|██████▌   | 384/586 [00:10<00:05, 36.18it/s]
 74%|███████▍  | 436/586 [00:11<00:04, 36.62it/s]
 66%|██████▌   | 388/586 [00:10<00:05, 36.24it/s]
 75%|███████▌  | 440/586 [00:11<00:03, 36.64it/s]
 67%|██████▋   | 392/586 [00:10<00:05, 36.30it/s]
 76%|███████▌  | 444/586 [00:12<00:03, 36.64it/s]
 68%|██████▊   | 396/586 [00:10<00:05, 36.34it/s]
 76%|███████▋  | 448/586 [00:12<00:03, 36.58it/s]
 68%|██████▊   | 400/586 [00:10<00:05, 35.97it/s]
 77%|███████▋  | 452/586 [00:12<00:03, 36.61it/s]
 69%|██████▉   | 404/586 [00:11<00:05, 36.17it/s]
 78%|███████▊  | 456/586 [00:12<00:03, 36.64it/s]
 70%|██████▉   | 408/586 [00:11<00:04, 36.34it/s]
 78%|███████▊  | 460/586 [00:12<00:03, 36.68it/s]
 70%|███████   | 412/586 [00:11<00:04, 36.39it/s]
 79%|███████▉  | 464/586 [00:12<00:03, 36.70it/s]
 71%|███████   | 416/586 [00:11<00:04, 36.40it/s]
 80%|███████▉  | 468/586 [00:12<00:03, 36.61it/s]
 72%|███████▏  | 420/586 [00:11<00:04, 36.46it/s]
 81%|████████  | 472/586 [00:12<00:03, 36.63it/s]
 72%|███████▏  | 424/586 [00:11<00:04, 36.02it/s]
 81%|████████  | 476/586 [00:12<00:03, 36.56it/s]
 73%|███████▎  | 428/586 [00:11<00:04, 36.13it/s]
 82%|████████▏ | 480/586 [00:13<00:02, 36.57it/s]
 74%|███████▎  | 432/586 [00:11<00:04, 36.28it/s]
 83%|████████▎ | 484/586 [00:13<00:02, 36.61it/s]
 74%|███████▍  | 436/586 [00:11<00:04, 36.35it/s]
 83%|████████▎ | 488/586 [00:13<00:02, 36.59it/s]
 75%|███████▌  | 440/586 [00:12<00:04, 36.39it/s]
 84%|████████▍ | 492/586 [00:13<00:02, 36.63it/s]
 85%|████████▍ | 496/586 [00:13<00:02, 36.71it/s]
 76%|███████▌  | 444/586 [00:12<00:03, 36.46it/s]
 85%|████████▌ | 500/586 [00:13<00:02, 36.71it/s]
 76%|███████▋  | 448/586 [00:12<00:03, 36.52it/s]
 86%|████████▌ | 504/586 [00:13<00:02, 36.76it/s]
 77%|███████▋  | 452/586 [00:12<00:03, 36.65it/s]
 87%|████████▋ | 508/586 [00:13<00:02, 36.80it/s]
 78%|███████▊  | 456/586 [00:12<00:03, 36.71it/s]
 87%|████████▋ | 512/586 [00:13<00:02, 36.82it/s]
 78%|███████▊  | 460/586 [00:12<00:03, 36.74it/s]
 88%|████████▊ | 516/586 [00:14<00:01, 36.82it/s]
 79%|███████▉  | 464/586 [00:12<00:03, 36.75it/s]
 89%|████████▊ | 520/586 [00:14<00:01, 36.83it/s]
 80%|███████▉  | 468/586 [00:12<00:03, 36.79it/s]
 89%|████████▉ | 524/586 [00:14<00:01, 36.82it/s]
 81%|████████  | 472/586 [00:12<00:03, 36.80it/s]
 81%|████████  | 476/586 [00:13<00:02, 36.83it/s]
 90%|█████████ | 528/586 [00:14<00:01, 36.80it/s]
 82%|████████▏ | 480/586 [00:13<00:02, 36.86it/s]
 91%|█████████ | 532/586 [00:14<00:01, 36.27it/s]
 83%|████████▎ | 484/586 [00:13<00:02, 36.86it/s]
 91%|█████████▏| 536/586 [00:14<00:01, 36.37it/s]
 83%|████████▎ | 488/586 [00:13<00:02, 36.85it/s]
 92%|█████████▏| 540/586 [00:14<00:01, 36.45it/s]
 84%|████████▍ | 492/586 [00:13<00:02, 36.77it/s]
 93%|█████████▎| 544/586 [00:14<00:01, 36.50it/s]
 85%|████████▍ | 496/586 [00:13<00:02, 36.68it/s]
 94%|█████████▎| 548/586 [00:14<00:01, 36.58it/s]
 85%|████████▌ | 500/586 [00:13<00:02, 36.44it/s]
 94%|█████████▍| 552/586 [00:15<00:00, 36.63it/s]
 86%|████████▌ | 504/586 [00:13<00:02, 36.24it/s]
 95%|█████████▍| 556/586 [00:15<00:00, 36.66it/s]
 96%|█████████▌| 560/586 [00:15<00:00, 36.74it/s]
 87%|████████▋ | 508/586 [00:13<00:02, 36.33it/s]
 96%|█████████▌| 564/586 [00:15<00:00, 36.74it/s]
 87%|████████▋ | 512/586 [00:14<00:02, 36.09it/s]
 97%|█████████▋| 568/586 [00:15<00:00, 36.78it/s]
 88%|████████▊ | 516/586 [00:14<00:01, 35.69it/s]
 98%|█████████▊| 572/586 [00:15<00:00, 36.77it/s]
 89%|████████▊ | 520/586 [00:14<00:01, 35.95it/s]
 98%|█████████▊| 576/586 [00:15<00:00, 36.81it/s]
 89%|████████▉ | 524/586 [00:14<00:01, 36.20it/s]
 99%|█████████▉| 580/586 [00:15<00:00, 36.83it/s]
 90%|█████████ | 528/586 [00:14<00:01, 36.34it/s]
100%|█████████▉| 584/586 [00:15<00:00, 36.77it/s]
 91%|█████████ | 532/586 [00:14<00:01, 36.38it/s]
100%|██████████| 586/586 [00:15<00:00, 36.78it/s]
n136-128-154:2240589:2240589 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2240589:2240589 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2240589:2240589 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:2240589:2240589 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2240589:2240589 [0] NCCL INFO Comm config Blocking set to 1

 91%|█████████▏| 536/586 [00:14<00:01, 36.46it/s]
 92%|█████████▏| 540/586 [00:14<00:01, 36.54it/s]n136-128-154:2240589:2240786 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2240589:2240786 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2240589:2240786 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2240589:2240786 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2240589:2240786 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2240589:2240786 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2240589:2240786 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2240589:2240786 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:2240589:2240786 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2240589:2240786 [0] NCCL INFO Using network IB

 93%|█████████▎| 544/586 [00:14<00:01, 36.65it/s]n136-128-154:2240589:2240786 [0] NCCL INFO ncclCommInitRankConfig comm 0x119f6810 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0xc11a58f1b0081fd7 - Init START

 94%|█████████▎| 548/586 [00:15<00:01, 36.76it/s]
 94%|█████████▍| 552/586 [00:15<00:00, 36.80it/s]
 95%|█████████▍| 556/586 [00:15<00:00, 36.79it/s]
 96%|█████████▌| 560/586 [00:15<00:00, 36.72it/s]
 96%|█████████▌| 564/586 [00:15<00:00, 36.67it/s]
 97%|█████████▋| 568/586 [00:15<00:00, 36.58it/s]
 98%|█████████▊| 572/586 [00:15<00:00, 36.58it/s]
 98%|█████████▊| 576/586 [00:15<00:00, 36.56it/s]
 99%|█████████▉| 580/586 [00:15<00:00, 36.58it/s]
100%|█████████▉| 584/586 [00:16<00:00, 36.61it/s]
100%|██████████| 586/586 [00:16<00:00, 36.44it/s]
n136-128-154:2240590:2240590 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:2240590:2240590 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2240590:2240590 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2240590:2240590 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2240590:2240590 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:2240590:2240791 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2240590:2240791 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2240590:2240791 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2240590:2240791 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2240590:2240791 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2240590:2240791 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2240590:2240791 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2240590:2240791 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:2240590:2240791 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2240590:2240791 [1] NCCL INFO Using network IB
n136-128-154:2240590:2240791 [1] NCCL INFO ncclCommInitRankConfig comm 0x109605b0 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0xc11a58f1b0081fd7 - Init START
n136-128-154:2240590:2240791 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2240589:2240786 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2240589:2240786 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2240589:2240786 [0] NCCL INFO Retrieving state for IB
n136-128-154:2240589:2240786 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:2240589:2240786 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2240589:2240786 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2240589:2240786 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2240589:2240786 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2240590:2240791 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2240590:2240791 [1] NCCL INFO Retrieving state for IB
n136-128-154:2240590:2240791 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:2240590:2240791 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2240590:2240791 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2240590:2240791 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2240590:2240791 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2240590:2240791 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2240590:2240791 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2240589:2240786 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2240589:2240786 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2240589:2240786 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2240589:2240786 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2240589:2240786 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2240589:2240786 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2240589:2240786 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2240589:2240786 [0] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2240589:2240786 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2240589:2240786 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2240589:2240786 [0] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2240589:2240786 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2240589:2240786 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2240589:2240786 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2240589:2240786 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2240589:2240786 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2240589:2240786 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2240589:2240786 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2240589:2240786 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2240589:2240786 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2240589:2240786 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2240589:2240786 [0] NCCL INFO ==========================================
n136-128-154:2240589:2240786 [0] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2240589:2240786 [0] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2240589:2240786 [0] NCCL INFO Setting affinity for GPU 6 to 33-62,97-126
n136-128-154:2240590:2240791 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2240590:2240791 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2240590:2240791 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2240590:2240791 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2240590:2240791 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2240590:2240791 [1] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2240590:2240791 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2240590:2240791 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2240590:2240791 [1] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2240590:2240791 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2240590:2240791 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2240590:2240791 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2240590:2240791 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2240590:2240791 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2240590:2240791 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2240590:2240791 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2240590:2240791 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2240590:2240791 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2240590:2240791 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2240590:2240791 [1] NCCL INFO ==========================================
n136-128-154:2240590:2240791 [1] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2240590:2240791 [1] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2240590:2240791 [1] NCCL INFO Setting affinity for GPU 7 to 33-62,97-126
n136-128-154:2240589:2240786 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2240589:2240786 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240589:2240786 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240589:2240786 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240589:2240786 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240589:2240786 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240589:2240786 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240589:2240786 [0] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240589:2240786 [0] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240589:2240786 [0] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240589:2240786 [0] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240589:2240786 [0] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240589:2240786 [0] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240589:2240786 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2240589:2240786 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240589:2240786 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240589:2240786 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240589:2240786 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240589:2240786 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240589:2240786 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240589:2240786 [0] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2240589:2240786 [0] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2240589:2240786 [0] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2240589:2240786 [0] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2240589:2240786 [0] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2240589:2240786 [0] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2240590:2240791 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2240590:2240791 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240590:2240791 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240590:2240791 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240590:2240791 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240590:2240791 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240590:2240791 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240590:2240791 [1] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240590:2240791 [1] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240590:2240791 [1] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240590:2240791 [1] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240590:2240791 [1] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240590:2240791 [1] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240590:2240791 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2240590:2240791 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240590:2240791 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240590:2240791 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240590:2240791 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240590:2240791 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240590:2240791 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2240590:2240791 [1] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2240590:2240791 [1] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2240590:2240791 [1] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2240590:2240791 [1] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2240590:2240791 [1] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2240590:2240791 [1] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2240590:2240791 [1] NCCL INFO comm 0x109605b0 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:2240590:2240791 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:2240590:2240791 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:2240590:2240791 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:2240590:2240791 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:2240589:2240786 [0] NCCL INFO comm 0x119f6810 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:2240590:2240791 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:2240590:2240791 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:2240590:2240791 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:2240589:2240786 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:2240590:2240791 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:2240589:2240786 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:2240590:2240791 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:2240589:2240786 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:2240590:2240791 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:2240589:2240786 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:2240590:2240791 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:2240589:2240786 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:2240590:2240791 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:2240589:2240786 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:2240590:2240791 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:2240589:2240786 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:2240590:2240791 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:2240589:2240786 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:2240590:2240791 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:2240589:2240786 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:2240590:2240791 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:2240589:2240786 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:2240590:2240791 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:2240589:2240786 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:2240590:2240791 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:2240589:2240786 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:2240590:2240791 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:2240589:2240786 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:2240590:2240791 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:2240589:2240786 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:2240590:2240791 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:2240589:2240786 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:2240590:2240791 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:2240589:2240786 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:2240590:2240791 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:2240589:2240786 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:2240590:2240791 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:2240589:2240786 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:2240589:2240786 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:2240589:2240786 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:2240589:2240786 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:2240589:2240786 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:2240589:2240786 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:2240590:2240791 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:2240589:2240786 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:2240590:2240791 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:2240590:2240791 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:2240590:2240791 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:2240590:2240791 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:2240590:2240791 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:2240590:2240791 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:2240590:2240791 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:2240590:2240791 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:2240590:2240791 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:2240590:2240791 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:2240590:2240791 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:2240590:2240791 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:2240590:2240791 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:2240590:2240791 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:2240590:2240791 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:2240590:2240791 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:2240590:2240791 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:2240590:2240791 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:2240590:2240791 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:2240590:2240791 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:2240590:2240791 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:2240590:2240791 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:2240590:2240791 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:2240590:2240791 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:2240590:2240791 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2240589:2240786 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:2240589:2240786 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:2240589:2240786 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:2240589:2240786 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:2240589:2240786 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:2240589:2240786 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:2240589:2240786 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:2240589:2240786 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:2240589:2240786 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:2240589:2240786 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:2240589:2240786 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:2240589:2240786 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:2240589:2240786 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:2240589:2240786 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:2240589:2240786 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:2240589:2240786 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:2240589:2240786 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:2240589:2240786 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:2240589:2240786 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:2240589:2240786 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:2240589:2240786 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:2240589:2240786 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:2240589:2240786 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:2240589:2240786 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:2240589:2240786 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:2240589:2240786 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:2240589:2240786 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:2240589:2240786 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:2240589:2240786 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:2240589:2240786 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:2240589:2240786 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:2240589:2240786 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:2240589:2240786 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:2240589:2240786 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:2240589:2240786 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:2240589:2240786 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:2240589:2240786 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:2240589:2240786 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:2240589:2240786 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:2240589:2240786 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:2240589:2240786 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:2240589:2240786 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:2240589:2240786 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:2240589:2240786 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:2240589:2240786 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:2240589:2240786 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:2240589:2240786 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:2240589:2240786 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:2240589:2240786 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:2240589:2240786 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2240590:2240791 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2240590:2240800 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 55
n136-128-154:2240590:2240799 [1] NCCL INFO [Proxy Service] Device 1 CPU core 117
n136-128-154:2240589:2240786 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2240589:2240786 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:2240589:2240802 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 56
n136-128-154:2240589:2240801 [0] NCCL INFO [Proxy Service] Device 0 CPU core 50
n136-128-154:2240590:2240791 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2240590:2240791 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2240589:2240786 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2240589:2240786 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2240589:2240786 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:2240589:2240786 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2240589:2240786 [0] NCCL INFO ncclCommInitRankConfig comm 0x119f6810 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0xc11a58f1b0081fd7 - Init COMPLETE
n136-128-154:2240589:2240786 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 1.95 (kernels 0.11, alloc 0.17, bootstrap 1.50, allgathers 0.01, topo 0.06, graphs 0.00, connections 0.01, rest 0.10)
n136-128-154:2240590:2240791 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2240590:2240791 [1] NCCL INFO ncclCommInitRankConfig comm 0x109605b0 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0xc11a58f1b0081fd7 - Init COMPLETE
n136-128-154:2240590:2240791 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.52 (kernels 0.11, alloc 0.23, bootstrap 0.00, allgathers 0.01, topo 0.06, graphs 0.00, connections 0.01, rest 0.10)
n136-128-154:2240589:2240803 [0] NCCL INFO Channel 00/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240590:2240804 [1] NCCL INFO Channel 00/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240589:2240803 [0] NCCL INFO Channel 01/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240590:2240804 [1] NCCL INFO Channel 01/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240589:2240803 [0] NCCL INFO Channel 02/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240590:2240804 [1] NCCL INFO Channel 02/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240589:2240803 [0] NCCL INFO Channel 03/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240590:2240804 [1] NCCL INFO Channel 03/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240589:2240803 [0] NCCL INFO Channel 04/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240590:2240804 [1] NCCL INFO Channel 04/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240589:2240803 [0] NCCL INFO Channel 05/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240590:2240804 [1] NCCL INFO Channel 05/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240589:2240803 [0] NCCL INFO Channel 06/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240590:2240804 [1] NCCL INFO Channel 06/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240589:2240803 [0] NCCL INFO Channel 07/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240590:2240804 [1] NCCL INFO Channel 07/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240589:2240803 [0] NCCL INFO Channel 08/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240590:2240804 [1] NCCL INFO Channel 08/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240589:2240803 [0] NCCL INFO Channel 09/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240590:2240804 [1] NCCL INFO Channel 09/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240589:2240803 [0] NCCL INFO Channel 10/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240590:2240804 [1] NCCL INFO Channel 10/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240589:2240803 [0] NCCL INFO Channel 11/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240590:2240804 [1] NCCL INFO Channel 11/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240589:2240803 [0] NCCL INFO Channel 12/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240590:2240804 [1] NCCL INFO Channel 12/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240589:2240803 [0] NCCL INFO Channel 13/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240590:2240804 [1] NCCL INFO Channel 13/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240589:2240803 [0] NCCL INFO Channel 14/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240590:2240804 [1] NCCL INFO Channel 14/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240589:2240803 [0] NCCL INFO Channel 15/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240590:2240804 [1] NCCL INFO Channel 15/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240589:2240803 [0] NCCL INFO Channel 16/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240590:2240804 [1] NCCL INFO Channel 16/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240589:2240803 [0] NCCL INFO Channel 17/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240590:2240804 [1] NCCL INFO Channel 17/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240589:2240803 [0] NCCL INFO Channel 18/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240590:2240804 [1] NCCL INFO Channel 18/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240589:2240803 [0] NCCL INFO Channel 19/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240590:2240804 [1] NCCL INFO Channel 19/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240589:2240803 [0] NCCL INFO Channel 20/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240590:2240804 [1] NCCL INFO Channel 20/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240589:2240803 [0] NCCL INFO Channel 21/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240590:2240804 [1] NCCL INFO Channel 21/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240589:2240803 [0] NCCL INFO Channel 22/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240590:2240804 [1] NCCL INFO Channel 22/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240589:2240803 [0] NCCL INFO Channel 23/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2240590:2240804 [1] NCCL INFO Channel 23/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240590:2240804 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:2240589:2240803 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-09:19:17:07 INFO     [evaluator:559] Running loglikelihood requests
2025-12-09:19:17:07 INFO     [evaluator:559] Running loglikelihood requests

Running loglikelihood requests:   0%|          | 0/2344 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 45
Determined largest batch size: 45

Running loglikelihood requests:   0%|          | 1/2344 [00:56<36:28:00, 56.03s/it]
Running loglikelihood requests:   2%|▏         | 46/2344 [00:59<36:07,  1.06it/s]  
Running loglikelihood requests:   4%|▍         | 94/2344 [01:03<15:55,  2.36it/s]
Running loglikelihood requests:   6%|▌         | 142/2344 [01:07<09:41,  3.79it/s]
Running loglikelihood requests:   8%|▊         | 189/2344 [01:11<06:51,  5.24it/s]
Running loglikelihood requests:  10%|█         | 242/2344 [01:14<05:01,  6.98it/s]
Running loglikelihood requests:  12%|█▏        | 290/2344 [01:18<04:06,  8.32it/s]
Running loglikelihood requests:  14%|█▍        | 336/2344 [01:21<03:34,  9.38it/s]
Running loglikelihood requests:  16%|█▋        | 381/2344 [01:25<03:12, 10.22it/s]
Running loglikelihood requests:  18%|█▊        | 429/2344 [01:28<02:51, 11.15it/s]
Running loglikelihood requests:  20%|██        | 480/2344 [01:32<02:33, 12.11it/s]
Running loglikelihood requests:  22%|██▏       | 526/2344 [01:35<02:25, 12.49it/s]
Running loglikelihood requests:  25%|██▍       | 580/2344 [01:38<02:11, 13.45it/s]
Running loglikelihood requests:  27%|██▋       | 626/2344 [01:42<02:07, 13.47it/s]
Running loglikelihood requests:  29%|██▉       | 674/2344 [01:45<02:02, 13.69it/s]
Running loglikelihood requests:  31%|███       | 719/2344 [01:49<01:59, 13.65it/s]
Running loglikelihood requests:  33%|███▎      | 765/2344 [01:52<01:55, 13.71it/s]
Running loglikelihood requests:  35%|███▍      | 813/2344 [01:55<01:49, 13.92it/s]
Running loglikelihood requests:  37%|███▋      | 863/2344 [01:58<01:43, 14.29it/s]
Running loglikelihood requests:  39%|███▉      | 914/2344 [02:02<01:37, 14.69it/s]
Running loglikelihood requests:  41%|████      | 959/2344 [02:05<01:35, 14.45it/s]
Running loglikelihood requests:  43%|████▎     | 1005/2344 [02:08<01:33, 14.35it/s]
Running loglikelihood requests:  45%|████▍     | 1052/2344 [02:11<01:29, 14.39it/s]
Running loglikelihood requests:  47%|████▋     | 1110/2344 [02:15<01:19, 15.49it/s]
Running loglikelihood requests:  49%|████▉     | 1159/2344 [02:18<01:16, 15.45it/s]
Running loglikelihood requests:  52%|█████▏    | 1208/2344 [02:21<01:13, 15.44it/s]
Running loglikelihood requests:  54%|█████▎    | 1255/2344 [02:24<01:11, 15.25it/s]
Running loglikelihood requests:  56%|█████▌    | 1303/2344 [02:27<01:08, 15.23it/s]
Running loglikelihood requests:  58%|█████▊    | 1356/2344 [02:31<01:02, 15.71it/s]
Running loglikelihood requests:  60%|█████▉    | 1401/2344 [02:34<01:01, 15.32it/s]
Running loglikelihood requests:  62%|██████▏   | 1454/2344 [02:37<00:56, 15.84it/s]
Running loglikelihood requests:  64%|██████▍   | 1504/2344 [02:40<00:52, 15.94it/s]
Running loglikelihood requests:  66%|██████▋   | 1553/2344 [02:43<00:49, 15.99it/s]
Running loglikelihood requests:  68%|██████▊   | 1601/2344 [02:46<00:46, 15.92it/s]
Running loglikelihood requests:  71%|███████   | 1653/2344 [02:49<00:42, 16.30it/s]
Running loglikelihood requests:  72%|███████▏  | 1698/2344 [02:52<00:40, 15.91it/s]
Running loglikelihood requests:  74%|███████▍  | 1745/2344 [02:55<00:37, 15.86it/s]
Running loglikelihood requests:  76%|███████▋  | 1793/2344 [02:58<00:34, 15.94it/s]
Running loglikelihood requests:  78%|███████▊  | 1840/2344 [03:01<00:31, 15.91it/s]
Running loglikelihood requests:  81%|████████  | 1891/2344 [03:04<00:27, 16.31it/s]
Running loglikelihood requests:  83%|████████▎ | 1938/2344 [03:07<00:24, 16.26it/s]
Running loglikelihood requests:  85%|████████▍ | 1983/2344 [03:10<00:22, 16.04it/s]
Running loglikelihood requests:  87%|████████▋ | 2032/2344 [03:13<00:19, 16.28it/s]
Running loglikelihood requests:  89%|████████▉ | 2089/2344 [03:15<00:14, 17.32it/s]
Running loglikelihood requests:  91%|█████████ | 2137/2344 [03:18<00:12, 17.18it/s]
Running loglikelihood requests:  93%|█████████▎| 2185/2344 [03:21<00:09, 17.13it/s]
Running loglikelihood requests:  96%|█████████▌| 2241/2344 [03:24<00:05, 17.97it/s]
Running loglikelihood requests:  98%|█████████▊| 2295/2344 [03:26<00:02, 19.21it/s]
Running loglikelihood requests: 100%|██████████| 2344/2344 [03:26<00:00, 11.33it/s]
n136-128-154:2240590:2241157 [1] NCCL INFO Channel 00/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240590:2241157 [1] NCCL INFO Channel 01/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240590:2241157 [1] NCCL INFO Channel 02/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240590:2241157 [1] NCCL INFO Channel 03/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240590:2241157 [1] NCCL INFO Channel 04/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240590:2241157 [1] NCCL INFO Channel 05/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240590:2241157 [1] NCCL INFO Channel 06/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240590:2241157 [1] NCCL INFO Channel 07/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240590:2241157 [1] NCCL INFO Channel 08/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240590:2241157 [1] NCCL INFO Channel 09/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240590:2241157 [1] NCCL INFO Channel 10/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240590:2241157 [1] NCCL INFO Channel 11/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240590:2241157 [1] NCCL INFO Channel 12/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240590:2241157 [1] NCCL INFO Channel 13/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240590:2241157 [1] NCCL INFO Channel 14/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240590:2241157 [1] NCCL INFO Channel 15/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240590:2241157 [1] NCCL INFO Channel 16/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240590:2241157 [1] NCCL INFO Channel 17/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240590:2241157 [1] NCCL INFO Channel 18/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240590:2241157 [1] NCCL INFO Channel 19/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240590:2241157 [1] NCCL INFO Channel 20/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240590:2241157 [1] NCCL INFO Channel 21/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240590:2241157 [1] NCCL INFO Channel 22/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240590:2241157 [1] NCCL INFO Channel 23/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240590:2241157 [1] NCCL INFO Channel 24/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240590:2241157 [1] NCCL INFO Channel 25/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240590:2241157 [1] NCCL INFO Channel 26/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240590:2241157 [1] NCCL INFO Channel 27/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240590:2241157 [1] NCCL INFO Channel 28/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240590:2241157 [1] NCCL INFO Channel 29/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240590:2241157 [1] NCCL INFO Channel 30/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2240590:2241157 [1] NCCL INFO Channel 31/1 : 1[7] -> 0[6] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:2240590:2241162 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2240590:2241162 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2240590:2241162 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2240590:2241162 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2240590:2241162 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2240590:2241162 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2240590:2240799 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-09:19:20:53 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 25, batch_size: auto (45)
|    Tasks    |Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|-------------|------:|------|-----:|--------|---|-----:|---|-----:|
|arc_challenge|      1|none  |    25|acc     |↑  |0.2176|±  |0.0121|
|             |       |none  |    25|acc_norm|↑  |0.2662|±  |0.0129|

[rank0]:[W1209 19:20:54.772226287 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:2240589:2241211 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2240589:2241211 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2240589:2241211 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2240589:2240801 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2240590:2240799 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2240589:2241211 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2240589:2241211 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2240589:2241211 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2240590:2241162 [1] NCCL INFO comm 0x109605b0 rank 1 nranks 2 cudaDev 1 busId c9000 - Abort COMPLETE
n136-128-154:2240589:2241211 [0] NCCL INFO comm 0x119f6810 rank 0 nranks 2 cudaDev 0 busId c5000 - Abort COMPLETE
任务 arc_challenge 评估完成！

==================================================
开始评估：任务=truthfulqa_mc1 | 少样本数=0 | 模型=Qwen2.5-7B-quantization-fg
输出路径：results2/Qwen2.5-7B-quantization-fg/f6/truthfulqa_mc1.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-09:19:21:44 INFO     [__main__:440] Selected Tasks: ['truthfulqa_mc1']
2025-12-09:19:21:44 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:19:21:44 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-09:19:21:44 INFO     [__main__:440] Selected Tasks: ['truthfulqa_mc1']
2025-12-09:19:21:44 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:19:21:44 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-09:19:21:45 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:19:21:45 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:19:21:45 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.15it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.01it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.09it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.02s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.14it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.63it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.40it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.03it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.49it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.27it/s]
2025-12-09:19:22:05 INFO     [evaluator:305] num_fewshot has been set to 0 for truthfulqa_mc1 in its config. Manual configuration will be ignored.
2025-12-09:19:22:05 INFO     [api.task:434] Building contexts for truthfulqa_mc1 on rank 1...

  0%|          | 0/408 [00:00<?, ?it/s]
 25%|██▍       | 100/408 [00:00<00:00, 993.77it/s]
 50%|█████     | 204/408 [00:00<00:00, 1017.16it/s]
 75%|███████▌  | 306/408 [00:00<00:00, 1016.56it/s]
100%|██████████| 408/408 [00:00<00:00, 1013.68it/s]
100%|██████████| 408/408 [00:00<00:00, 1012.63it/s]
2025-12-09:19:22:11 INFO     [evaluator:305] num_fewshot has been set to 0 for truthfulqa_mc1 in its config. Manual configuration will be ignored.
2025-12-09:19:22:11 INFO     [api.task:434] Building contexts for truthfulqa_mc1 on rank 0...

  0%|          | 0/409 [00:00<?, ?it/s]
 25%|██▌       | 103/409 [00:00<00:00, 1022.12it/s]
 51%|█████     | 208/409 [00:00<00:00, 1035.34it/s]
 77%|███████▋  | 313/409 [00:00<00:00, 1037.67it/s]
100%|██████████| 409/409 [00:00<00:00, 1035.09it/s]
n136-128-154:2241272:2241272 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2241272:2241272 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2241272:2241272 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:2241272:2241272 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2241272:2241272 [0] NCCL INFO Comm config Blocking set to 1
n136-128-154:2241273:2241273 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:2241273:2241273 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2241273:2241273 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2241273:2241273 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2241273:2241273 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:2241272:2241458 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2241272:2241458 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2241273:2241459 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2241273:2241459 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2241272:2241458 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2241272:2241458 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2241272:2241458 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2241273:2241459 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2241273:2241459 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2241273:2241459 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2241273:2241459 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2241273:2241459 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2241273:2241459 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:2241272:2241458 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2241272:2241458 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2241272:2241458 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:2241273:2241459 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2241273:2241459 [1] NCCL INFO Using network IB
n136-128-154:2241272:2241458 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2241272:2241458 [0] NCCL INFO Using network IB
n136-128-154:2241273:2241459 [1] NCCL INFO ncclCommInitRankConfig comm 0x10e35c90 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0xcadda8d18b391fd7 - Init START
n136-128-154:2241272:2241458 [0] NCCL INFO ncclCommInitRankConfig comm 0xf6198f0 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0xcadda8d18b391fd7 - Init START
n136-128-154:2241273:2241459 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2241272:2241458 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2241273:2241459 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2241273:2241459 [1] NCCL INFO Retrieving state for IB
n136-128-154:2241273:2241459 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:2241273:2241459 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2241272:2241458 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2241272:2241458 [0] NCCL INFO Retrieving state for IB
n136-128-154:2241272:2241458 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:2241273:2241459 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2241272:2241458 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2241272:2241458 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2241273:2241459 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2241273:2241459 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2241272:2241458 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2241272:2241458 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2241272:2241458 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2241272:2241458 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2241272:2241458 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2241272:2241458 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2241272:2241458 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2241272:2241458 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2241272:2241458 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2241272:2241458 [0] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2241272:2241458 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2241272:2241458 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2241272:2241458 [0] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2241272:2241458 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2241272:2241458 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2241272:2241458 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2241272:2241458 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2241272:2241458 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2241272:2241458 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2241272:2241458 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2241272:2241458 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2241272:2241458 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2241272:2241458 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2241272:2241458 [0] NCCL INFO ==========================================
n136-128-154:2241272:2241458 [0] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2241272:2241458 [0] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2241272:2241458 [0] NCCL INFO Setting affinity for GPU 6 to 33-62,97-126
n136-128-154:2241272:2241458 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2241272:2241458 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241272:2241458 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241272:2241458 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241272:2241458 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241272:2241458 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241272:2241458 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241272:2241458 [0] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241272:2241458 [0] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241272:2241458 [0] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241272:2241458 [0] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241272:2241458 [0] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241272:2241458 [0] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241272:2241458 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2241272:2241458 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241273:2241459 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2241273:2241459 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2241273:2241459 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2241273:2241459 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2241273:2241459 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2241273:2241459 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2241273:2241459 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2241273:2241459 [1] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2241273:2241459 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2241273:2241459 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2241273:2241459 [1] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2241273:2241459 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2241273:2241459 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2241273:2241459 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2241273:2241459 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2241273:2241459 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2241273:2241459 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2241273:2241459 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2241273:2241459 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2241273:2241459 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2241273:2241459 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2241273:2241459 [1] NCCL INFO ==========================================
n136-128-154:2241273:2241459 [1] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2241273:2241459 [1] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2241273:2241459 [1] NCCL INFO Setting affinity for GPU 7 to 33-62,97-126
n136-128-154:2241273:2241459 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2241273:2241459 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241273:2241459 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241273:2241459 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241273:2241459 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241273:2241459 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241273:2241459 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241273:2241459 [1] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241273:2241459 [1] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241273:2241459 [1] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241273:2241459 [1] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241273:2241459 [1] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241273:2241459 [1] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241273:2241459 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2241273:2241459 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241273:2241459 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241273:2241459 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241273:2241459 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241273:2241459 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241273:2241459 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241273:2241459 [1] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2241273:2241459 [1] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2241273:2241459 [1] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2241273:2241459 [1] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2241273:2241459 [1] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2241273:2241459 [1] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2241272:2241458 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241272:2241458 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241272:2241458 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241272:2241458 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241272:2241458 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241272:2241458 [0] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2241272:2241458 [0] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2241272:2241458 [0] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2241272:2241458 [0] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2241272:2241458 [0] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2241272:2241458 [0] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2241273:2241459 [1] NCCL INFO comm 0x10e35c90 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:2241272:2241458 [0] NCCL INFO comm 0xf6198f0 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:2241273:2241459 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:2241273:2241459 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:2241273:2241459 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:2241273:2241459 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:2241273:2241459 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:2241273:2241459 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:2241273:2241459 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:2241272:2241458 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:2241273:2241459 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:2241272:2241458 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:2241273:2241459 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:2241272:2241458 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:2241273:2241459 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:2241272:2241458 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:2241273:2241459 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:2241272:2241458 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:2241273:2241459 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:2241272:2241458 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:2241273:2241459 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:2241272:2241458 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:2241273:2241459 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:2241272:2241458 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:2241273:2241459 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:2241272:2241458 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:2241273:2241459 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:2241272:2241458 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:2241273:2241459 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:2241272:2241458 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:2241273:2241459 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:2241272:2241458 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:2241273:2241459 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:2241272:2241458 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:2241273:2241459 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:2241272:2241458 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:2241273:2241459 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:2241272:2241458 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:2241273:2241459 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:2241272:2241458 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:2241273:2241459 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:2241272:2241458 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:2241273:2241459 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:2241272:2241458 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:2241272:2241458 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:2241272:2241458 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:2241272:2241458 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:2241272:2241458 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:2241272:2241458 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:2241272:2241458 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:2241273:2241459 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:2241273:2241459 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:2241273:2241459 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:2241273:2241459 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:2241272:2241458 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:2241273:2241459 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:2241272:2241458 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:2241273:2241459 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:2241272:2241458 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:2241273:2241459 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:2241272:2241458 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:2241273:2241459 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:2241272:2241458 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:2241273:2241459 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:2241272:2241458 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:2241273:2241459 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:2241272:2241458 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:2241273:2241459 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:2241272:2241458 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:2241273:2241459 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:2241272:2241458 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:2241273:2241459 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:2241272:2241458 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:2241273:2241459 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:2241272:2241458 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:2241273:2241459 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:2241272:2241458 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:2241273:2241459 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:2241272:2241458 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:2241273:2241459 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:2241272:2241458 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:2241273:2241459 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:2241272:2241458 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:2241273:2241459 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:2241272:2241458 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:2241273:2241459 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:2241272:2241458 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:2241273:2241459 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:2241272:2241458 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:2241273:2241459 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:2241272:2241458 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:2241273:2241459 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:2241272:2241458 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:2241273:2241459 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:2241272:2241458 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:2241273:2241459 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:2241272:2241458 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:2241272:2241458 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:2241273:2241459 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2241272:2241458 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:2241272:2241458 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:2241272:2241458 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:2241272:2241458 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:2241272:2241458 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:2241272:2241458 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:2241272:2241458 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:2241272:2241458 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:2241272:2241458 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:2241272:2241458 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:2241272:2241458 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:2241272:2241458 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:2241272:2241458 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:2241272:2241458 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:2241272:2241458 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:2241272:2241458 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:2241272:2241458 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:2241272:2241458 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:2241272:2241458 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:2241272:2241458 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:2241272:2241458 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:2241272:2241458 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:2241272:2241458 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:2241272:2241458 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:2241272:2241458 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:2241272:2241458 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:2241272:2241458 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2241273:2241459 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2241273:2241471 [1] NCCL INFO [Proxy Service] Device 1 CPU core 61
n136-128-154:2241273:2241472 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 37
n136-128-154:2241272:2241458 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2241272:2241458 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:2241272:2241473 [0] NCCL INFO [Proxy Service] Device 0 CPU core 99
n136-128-154:2241272:2241474 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 103
n136-128-154:2241273:2241459 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2241273:2241459 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2241272:2241458 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2241272:2241458 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2241272:2241458 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:2241272:2241458 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2241272:2241458 [0] NCCL INFO ncclCommInitRankConfig comm 0xf6198f0 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0xcadda8d18b391fd7 - Init COMPLETE
n136-128-154:2241272:2241458 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.77 (kernels 0.12, alloc 0.42, bootstrap 0.00, allgathers 0.00, topo 0.06, graphs 0.00, connections 0.03, rest 0.12)
n136-128-154:2241273:2241459 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2241273:2241459 [1] NCCL INFO ncclCommInitRankConfig comm 0x10e35c90 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0xcadda8d18b391fd7 - Init COMPLETE
n136-128-154:2241273:2241459 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.76 (kernels 0.12, alloc 0.42, bootstrap 0.00, allgathers 0.00, topo 0.06, graphs 0.00, connections 0.02, rest 0.14)
n136-128-154:2241272:2241475 [0] NCCL INFO Channel 00/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241272:2241475 [0] NCCL INFO Channel 01/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241272:2241475 [0] NCCL INFO Channel 02/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241272:2241475 [0] NCCL INFO Channel 03/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241272:2241475 [0] NCCL INFO Channel 04/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241273:2241476 [1] NCCL INFO Channel 00/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241272:2241475 [0] NCCL INFO Channel 05/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241273:2241476 [1] NCCL INFO Channel 01/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241272:2241475 [0] NCCL INFO Channel 06/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241273:2241476 [1] NCCL INFO Channel 02/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241272:2241475 [0] NCCL INFO Channel 07/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241273:2241476 [1] NCCL INFO Channel 03/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241272:2241475 [0] NCCL INFO Channel 08/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241273:2241476 [1] NCCL INFO Channel 04/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241272:2241475 [0] NCCL INFO Channel 09/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241273:2241476 [1] NCCL INFO Channel 05/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241272:2241475 [0] NCCL INFO Channel 10/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241273:2241476 [1] NCCL INFO Channel 06/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241272:2241475 [0] NCCL INFO Channel 11/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241273:2241476 [1] NCCL INFO Channel 07/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241272:2241475 [0] NCCL INFO Channel 12/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241273:2241476 [1] NCCL INFO Channel 08/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241272:2241475 [0] NCCL INFO Channel 13/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241273:2241476 [1] NCCL INFO Channel 09/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241272:2241475 [0] NCCL INFO Channel 14/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241273:2241476 [1] NCCL INFO Channel 10/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241272:2241475 [0] NCCL INFO Channel 15/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241273:2241476 [1] NCCL INFO Channel 11/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241272:2241475 [0] NCCL INFO Channel 16/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241273:2241476 [1] NCCL INFO Channel 12/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241272:2241475 [0] NCCL INFO Channel 17/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241273:2241476 [1] NCCL INFO Channel 13/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241272:2241475 [0] NCCL INFO Channel 18/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241273:2241476 [1] NCCL INFO Channel 14/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241272:2241475 [0] NCCL INFO Channel 19/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241273:2241476 [1] NCCL INFO Channel 15/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241272:2241475 [0] NCCL INFO Channel 20/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241273:2241476 [1] NCCL INFO Channel 16/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241272:2241475 [0] NCCL INFO Channel 21/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241273:2241476 [1] NCCL INFO Channel 17/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241272:2241475 [0] NCCL INFO Channel 22/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241273:2241476 [1] NCCL INFO Channel 18/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241272:2241475 [0] NCCL INFO Channel 23/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241273:2241476 [1] NCCL INFO Channel 19/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241273:2241476 [1] NCCL INFO Channel 20/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241273:2241476 [1] NCCL INFO Channel 21/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241273:2241476 [1] NCCL INFO Channel 22/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241273:2241476 [1] NCCL INFO Channel 23/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241272:2241475 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:2241273:2241476 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-09:19:22:12 INFO     [evaluator:559] Running loglikelihood requests
2025-12-09:19:22:12 INFO     [evaluator:559] Running loglikelihood requests

Running loglikelihood requests:   0%|          | 0/2066 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 64
Determined largest batch size: 64

Running loglikelihood requests:   0%|          | 1/2066 [00:07<4:02:15,  7.04s/it]
Running loglikelihood requests:   3%|▎         | 65/2066 [00:08<03:08, 10.61it/s] 
Running loglikelihood requests:   6%|▌         | 129/2066 [00:09<01:35, 20.27it/s]
Running loglikelihood requests:   9%|▉         | 193/2066 [00:10<01:05, 28.81it/s]
Running loglikelihood requests:  12%|█▏        | 257/2066 [00:11<00:50, 36.01it/s]
Running loglikelihood requests:  16%|█▌        | 321/2066 [00:12<00:41, 41.83it/s]
Running loglikelihood requests:  19%|█▊        | 385/2066 [00:13<00:36, 46.41it/s]
Running loglikelihood requests:  22%|██▏       | 449/2066 [00:14<00:32, 50.09it/s]
Running loglikelihood requests:  25%|██▍       | 513/2066 [00:15<00:29, 52.82it/s]
Running loglikelihood requests:  28%|██▊       | 577/2066 [00:16<00:27, 54.89it/s]
Running loglikelihood requests:  31%|███       | 641/2066 [00:17<00:25, 56.46it/s]
Running loglikelihood requests:  34%|███▍      | 705/2066 [00:19<00:23, 57.67it/s]
Running loglikelihood requests:  37%|███▋      | 769/2066 [00:20<00:22, 58.52it/s]
Running loglikelihood requests:  40%|████      | 833/2066 [00:21<00:20, 59.11it/s]
Running loglikelihood requests:  43%|████▎     | 897/2066 [00:22<00:19, 59.72it/s]
Running loglikelihood requests:  47%|████▋     | 961/2066 [00:23<00:18, 60.09it/s]
Running loglikelihood requests:  50%|████▉     | 1025/2066 [00:24<00:17, 60.35it/s]
Running loglikelihood requests:  53%|█████▎    | 1089/2066 [00:25<00:16, 60.53it/s]
Running loglikelihood requests:  56%|█████▌    | 1153/2066 [00:26<00:15, 60.80it/s]
Running loglikelihood requests:  59%|█████▉    | 1217/2066 [00:27<00:13, 60.94it/s]
Running loglikelihood requests:  62%|██████▏   | 1281/2066 [00:28<00:12, 60.96it/s]
Running loglikelihood requests:  65%|██████▌   | 1345/2066 [00:29<00:11, 61.11it/s]
Running loglikelihood requests:  68%|██████▊   | 1409/2066 [00:30<00:10, 61.35it/s]
Running loglikelihood requests:  71%|███████▏  | 1473/2066 [00:31<00:09, 61.52it/s]
Running loglikelihood requests:  74%|███████▍  | 1537/2066 [00:32<00:08, 61.68it/s]
Running loglikelihood requests:  77%|███████▋  | 1601/2066 [00:33<00:07, 61.76it/s]
Running loglikelihood requests:  81%|████████  | 1665/2066 [00:34<00:06, 61.93it/s]
Running loglikelihood requests:  84%|████████▎ | 1729/2066 [00:35<00:05, 62.09it/s]
Running loglikelihood requests:  87%|████████▋ | 1793/2066 [00:36<00:04, 62.15it/s]
Running loglikelihood requests:  90%|████████▉ | 1857/2066 [00:37<00:03, 62.29it/s]
Running loglikelihood requests:  93%|█████████▎| 1921/2066 [00:38<00:02, 62.16it/s]
Running loglikelihood requests:  96%|█████████▌| 1985/2066 [00:39<00:01, 62.64it/s]
Running loglikelihood requests:  99%|█████████▉| 2049/2066 [00:40<00:00, 79.54it/s]
Running loglikelihood requests: 100%|██████████| 2066/2066 [00:40<00:00, 51.54it/s]
n136-128-154:2241273:2241528 [1] NCCL INFO Channel 00/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241273:2241528 [1] NCCL INFO Channel 01/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241273:2241528 [1] NCCL INFO Channel 02/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241273:2241528 [1] NCCL INFO Channel 03/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241273:2241528 [1] NCCL INFO Channel 04/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241273:2241528 [1] NCCL INFO Channel 05/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241273:2241528 [1] NCCL INFO Channel 06/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241273:2241528 [1] NCCL INFO Channel 07/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241273:2241528 [1] NCCL INFO Channel 08/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241273:2241528 [1] NCCL INFO Channel 09/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241273:2241528 [1] NCCL INFO Channel 10/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241273:2241528 [1] NCCL INFO Channel 11/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241273:2241528 [1] NCCL INFO Channel 12/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241273:2241528 [1] NCCL INFO Channel 13/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241273:2241528 [1] NCCL INFO Channel 14/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241273:2241528 [1] NCCL INFO Channel 15/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241273:2241528 [1] NCCL INFO Channel 16/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241273:2241528 [1] NCCL INFO Channel 17/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241273:2241528 [1] NCCL INFO Channel 18/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241273:2241528 [1] NCCL INFO Channel 19/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241273:2241528 [1] NCCL INFO Channel 20/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241273:2241528 [1] NCCL INFO Channel 21/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241273:2241528 [1] NCCL INFO Channel 22/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241273:2241528 [1] NCCL INFO Channel 23/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241273:2241528 [1] NCCL INFO Channel 24/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241273:2241528 [1] NCCL INFO Channel 25/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241273:2241528 [1] NCCL INFO Channel 26/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241273:2241528 [1] NCCL INFO Channel 27/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241273:2241528 [1] NCCL INFO Channel 28/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241273:2241528 [1] NCCL INFO Channel 29/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241273:2241528 [1] NCCL INFO Channel 30/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241273:2241528 [1] NCCL INFO Channel 31/1 : 1[7] -> 0[6] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:2241273:2241532 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2241273:2241532 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2241273:2241532 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2241273:2241532 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2241273:2241532 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2241273:2241532 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2241273:2241471 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-09:19:23:03 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: auto (64)
|    Tasks     |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|--------------|------:|------|-----:|------|---|-----:|---|-----:|
|truthfulqa_mc1|      2|none  |     0|acc   |↑  |0.2387|±  |0.0149|

[rank0]:[W1209 19:23:04.417594440 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:2241272:2241583 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2241272:2241583 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2241272:2241583 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2241272:2241473 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2241272:2241583 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2241272:2241583 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2241272:2241583 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2241273:2241471 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2241273:2241532 [1] NCCL INFO comm 0x10e35c90 rank 1 nranks 2 cudaDev 1 busId c9000 - Abort COMPLETE
n136-128-154:2241272:2241583 [0] NCCL INFO comm 0xf6198f0 rank 0 nranks 2 cudaDev 0 busId c5000 - Abort COMPLETE
任务 truthfulqa_mc1 评估完成！

==================================================
开始评估：任务=piqa | 少样本数=0 | 模型=Qwen2.5-7B-quantization-fg
输出路径：results2/Qwen2.5-7B-quantization-fg/f6/piqa.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-09:19:23:54 INFO     [__main__:440] Selected Tasks: ['piqa']
2025-12-09:19:23:54 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:19:23:54 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-09:19:23:54 INFO     [__main__:440] Selected Tasks: ['piqa']
2025-12-09:19:23:54 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:19:23:54 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-09:19:23:55 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:19:23:55 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:19:23:55 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.15it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.08it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:02,  1.00s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.02it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.02it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.04it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.47it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.27it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.48it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.29it/s]
2025-12-09:19:24:13 WARNING  [evaluator:309] Overwriting default num_fewshot of piqa from None to 0
2025-12-09:19:24:13 INFO     [api.task:434] Building contexts for piqa on rank 0...

  0%|          | 0/919 [00:00<?, ?it/s]
 16%|█▋        | 150/919 [00:00<00:00, 1496.96it/s]
 33%|███▎      | 301/919 [00:00<00:00, 1501.53it/s]
 50%|████▉     | 455/919 [00:00<00:00, 1518.99it/s]
 66%|██████▋   | 609/919 [00:00<00:00, 1526.12it/s]
 83%|████████▎ | 762/919 [00:00<00:00, 1527.37it/s]
100%|█████████▉| 915/919 [00:00<00:00, 1501.66it/s]
100%|██████████| 919/919 [00:00<00:00, 1508.87it/s]
n136-128-154:2241630:2241630 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2241630:2241630 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2241630:2241630 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:2241630:2241630 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2241630:2241630 [0] NCCL INFO Comm config Blocking set to 1
n136-128-154:2241630:2241849 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2241630:2241849 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2241630:2241849 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2241630:2241849 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2241630:2241849 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2241630:2241849 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2241630:2241849 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2241630:2241849 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:2241630:2241849 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2241630:2241849 [0] NCCL INFO Using network IB
n136-128-154:2241630:2241849 [0] NCCL INFO ncclCommInitRankConfig comm 0xe995780 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0x53eacb4832208b8c - Init START
2025-12-09:19:24:16 WARNING  [evaluator:309] Overwriting default num_fewshot of piqa from None to 0
2025-12-09:19:24:16 INFO     [api.task:434] Building contexts for piqa on rank 1...

  0%|          | 0/919 [00:00<?, ?it/s]
 17%|█▋        | 152/919 [00:00<00:00, 1510.30it/s]
 33%|███▎      | 305/919 [00:00<00:00, 1521.49it/s]
 50%|████▉     | 459/919 [00:00<00:00, 1509.28it/s]
 66%|██████▋   | 611/919 [00:00<00:00, 1512.47it/s]
 83%|████████▎ | 764/919 [00:00<00:00, 1517.83it/s]
100%|█████████▉| 916/919 [00:00<00:00, 1509.61it/s]
100%|██████████| 919/919 [00:00<00:00, 1511.42it/s]
n136-128-154:2241631:2241631 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:2241631:2241631 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2241631:2241631 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2241631:2241631 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2241631:2241631 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:2241631:2241886 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2241631:2241886 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2241631:2241886 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2241631:2241886 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2241631:2241886 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2241631:2241886 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2241631:2241886 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2241631:2241886 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:2241631:2241886 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2241631:2241886 [1] NCCL INFO Using network IB
n136-128-154:2241631:2241886 [1] NCCL INFO ncclCommInitRankConfig comm 0xea03000 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0x53eacb4832208b8c - Init START
n136-128-154:2241631:2241886 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2241630:2241849 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2241630:2241849 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2241630:2241849 [0] NCCL INFO Retrieving state for IB
n136-128-154:2241630:2241849 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:2241630:2241849 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2241631:2241886 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2241631:2241886 [1] NCCL INFO Retrieving state for IB
n136-128-154:2241631:2241886 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:2241630:2241849 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2241631:2241886 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2241630:2241849 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2241631:2241886 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2241630:2241849 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2241631:2241886 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2241631:2241886 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2241630:2241849 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2241630:2241849 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2241631:2241886 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2241631:2241886 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2241630:2241849 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2241630:2241849 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2241630:2241849 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2241630:2241849 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2241630:2241849 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2241630:2241849 [0] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2241630:2241849 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2241630:2241849 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2241630:2241849 [0] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2241630:2241849 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2241630:2241849 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2241630:2241849 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2241630:2241849 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2241630:2241849 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2241630:2241849 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2241630:2241849 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2241630:2241849 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2241630:2241849 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2241630:2241849 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2241630:2241849 [0] NCCL INFO ==========================================
n136-128-154:2241630:2241849 [0] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2241630:2241849 [0] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2241630:2241849 [0] NCCL INFO Setting affinity for GPU 6 to 33-62,97-126
n136-128-154:2241631:2241886 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2241631:2241886 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2241631:2241886 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2241631:2241886 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2241631:2241886 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2241631:2241886 [1] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2241631:2241886 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2241631:2241886 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2241631:2241886 [1] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2241631:2241886 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2241631:2241886 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2241631:2241886 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2241631:2241886 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2241631:2241886 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2241631:2241886 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2241631:2241886 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2241631:2241886 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2241631:2241886 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2241631:2241886 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2241631:2241886 [1] NCCL INFO ==========================================
n136-128-154:2241631:2241886 [1] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2241631:2241886 [1] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2241631:2241886 [1] NCCL INFO Setting affinity for GPU 7 to 33-62,97-126
n136-128-154:2241630:2241849 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2241630:2241849 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241630:2241849 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241630:2241849 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241630:2241849 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241630:2241849 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241630:2241849 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241630:2241849 [0] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241630:2241849 [0] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241630:2241849 [0] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241630:2241849 [0] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241630:2241849 [0] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241630:2241849 [0] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241630:2241849 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2241630:2241849 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241630:2241849 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241630:2241849 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241630:2241849 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241630:2241849 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241630:2241849 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241630:2241849 [0] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2241630:2241849 [0] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2241630:2241849 [0] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2241630:2241849 [0] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2241630:2241849 [0] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2241630:2241849 [0] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2241631:2241886 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2241631:2241886 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241631:2241886 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241631:2241886 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241631:2241886 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241631:2241886 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241631:2241886 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241631:2241886 [1] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241631:2241886 [1] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241631:2241886 [1] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241631:2241886 [1] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241631:2241886 [1] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241631:2241886 [1] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241631:2241886 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2241631:2241886 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241631:2241886 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241631:2241886 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241631:2241886 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241631:2241886 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241631:2241886 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2241631:2241886 [1] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2241631:2241886 [1] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2241631:2241886 [1] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2241631:2241886 [1] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2241631:2241886 [1] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2241631:2241886 [1] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2241630:2241849 [0] NCCL INFO comm 0xe995780 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:2241631:2241886 [1] NCCL INFO comm 0xea03000 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:2241631:2241886 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:2241630:2241849 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:2241631:2241886 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:2241630:2241849 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:2241631:2241886 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:2241630:2241849 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:2241631:2241886 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:2241630:2241849 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:2241631:2241886 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:2241630:2241849 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:2241631:2241886 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:2241630:2241849 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:2241631:2241886 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:2241630:2241849 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:2241631:2241886 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:2241630:2241849 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:2241631:2241886 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:2241630:2241849 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:2241631:2241886 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:2241630:2241849 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:2241631:2241886 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:2241630:2241849 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:2241631:2241886 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:2241630:2241849 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:2241631:2241886 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:2241630:2241849 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:2241631:2241886 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:2241630:2241849 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:2241631:2241886 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:2241630:2241849 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:2241631:2241886 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:2241630:2241849 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:2241631:2241886 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:2241630:2241849 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:2241631:2241886 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:2241630:2241849 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:2241631:2241886 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:2241630:2241849 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:2241631:2241886 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:2241630:2241849 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:2241631:2241886 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:2241630:2241849 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:2241631:2241886 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:2241630:2241849 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:2241631:2241886 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:2241630:2241849 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:2241631:2241886 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:2241630:2241849 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:2241630:2241849 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:2241630:2241849 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:2241631:2241886 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:2241630:2241849 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:2241631:2241886 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:2241630:2241849 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:2241631:2241886 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:2241630:2241849 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:2241631:2241886 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:2241630:2241849 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:2241631:2241886 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:2241630:2241849 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:2241631:2241886 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:2241630:2241849 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:2241631:2241886 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:2241630:2241849 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:2241631:2241886 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:2241630:2241849 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:2241631:2241886 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:2241630:2241849 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:2241631:2241886 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:2241630:2241849 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:2241631:2241886 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:2241630:2241849 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:2241631:2241886 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:2241630:2241849 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:2241631:2241886 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:2241630:2241849 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:2241631:2241886 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:2241630:2241849 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:2241631:2241886 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:2241630:2241849 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:2241631:2241886 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:2241630:2241849 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:2241631:2241886 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:2241630:2241849 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:2241631:2241886 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:2241630:2241849 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:2241631:2241886 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:2241630:2241849 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:2241631:2241886 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:2241630:2241849 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:2241631:2241886 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:2241630:2241849 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:2241631:2241886 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:2241630:2241849 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:2241631:2241886 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:2241630:2241849 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:2241631:2241886 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:2241630:2241849 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:2241631:2241886 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:2241630:2241849 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:2241631:2241886 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2241630:2241849 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:2241630:2241849 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:2241630:2241849 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:2241630:2241849 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:2241630:2241849 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:2241630:2241849 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:2241630:2241849 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:2241630:2241849 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:2241630:2241849 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:2241630:2241849 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:2241630:2241849 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:2241630:2241849 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:2241630:2241849 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:2241630:2241849 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:2241630:2241849 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:2241630:2241849 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:2241630:2241849 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:2241630:2241849 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:2241630:2241849 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:2241630:2241849 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:2241630:2241849 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:2241630:2241849 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:2241630:2241849 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2241631:2241886 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2241631:2241895 [1] NCCL INFO [Proxy Service] Device 1 CPU core 54
n136-128-154:2241631:2241896 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 55
n136-128-154:2241630:2241849 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2241630:2241849 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:2241630:2241897 [0] NCCL INFO [Proxy Service] Device 0 CPU core 35
n136-128-154:2241630:2241898 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 37
n136-128-154:2241631:2241886 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2241631:2241886 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2241630:2241849 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2241630:2241849 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2241630:2241849 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:2241631:2241886 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2241631:2241886 [1] NCCL INFO ncclCommInitRankConfig comm 0xea03000 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0x53eacb4832208b8c - Init COMPLETE
n136-128-154:2241631:2241886 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.70 (kernels 0.37, alloc 0.19, bootstrap 0.00, allgathers 0.00, topo 0.10, graphs 0.00, connections 0.01, rest 0.03)
n136-128-154:2241630:2241849 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2241630:2241849 [0] NCCL INFO ncclCommInitRankConfig comm 0xe995780 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0x53eacb4832208b8c - Init COMPLETE
n136-128-154:2241630:2241849 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 3.25 (kernels 0.20, alloc 0.11, bootstrap 2.80, allgathers 0.00, topo 0.10, graphs 0.00, connections 0.01, rest 0.03)
n136-128-154:2241631:2241899 [1] NCCL INFO Channel 00/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2241899 [1] NCCL INFO Channel 01/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2241899 [1] NCCL INFO Channel 02/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241630:2241900 [0] NCCL INFO Channel 00/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241631:2241899 [1] NCCL INFO Channel 03/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2241899 [1] NCCL INFO Channel 04/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241630:2241900 [0] NCCL INFO Channel 01/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241631:2241899 [1] NCCL INFO Channel 05/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241630:2241900 [0] NCCL INFO Channel 02/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241631:2241899 [1] NCCL INFO Channel 06/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241630:2241900 [0] NCCL INFO Channel 03/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241631:2241899 [1] NCCL INFO Channel 07/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241630:2241900 [0] NCCL INFO Channel 04/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241631:2241899 [1] NCCL INFO Channel 08/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241630:2241900 [0] NCCL INFO Channel 05/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241630:2241900 [0] NCCL INFO Channel 06/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241630:2241900 [0] NCCL INFO Channel 07/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241631:2241899 [1] NCCL INFO Channel 09/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2241899 [1] NCCL INFO Channel 10/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2241899 [1] NCCL INFO Channel 11/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241630:2241900 [0] NCCL INFO Channel 08/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241630:2241900 [0] NCCL INFO Channel 09/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241630:2241900 [0] NCCL INFO Channel 10/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241631:2241899 [1] NCCL INFO Channel 12/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241630:2241900 [0] NCCL INFO Channel 11/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241631:2241899 [1] NCCL INFO Channel 13/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2241899 [1] NCCL INFO Channel 14/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2241899 [1] NCCL INFO Channel 15/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2241899 [1] NCCL INFO Channel 16/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241630:2241900 [0] NCCL INFO Channel 12/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241630:2241900 [0] NCCL INFO Channel 13/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241631:2241899 [1] NCCL INFO Channel 17/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241630:2241900 [0] NCCL INFO Channel 14/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241631:2241899 [1] NCCL INFO Channel 18/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2241899 [1] NCCL INFO Channel 19/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241630:2241900 [0] NCCL INFO Channel 15/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241630:2241900 [0] NCCL INFO Channel 16/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241630:2241900 [0] NCCL INFO Channel 17/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241630:2241900 [0] NCCL INFO Channel 18/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241631:2241899 [1] NCCL INFO Channel 20/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241630:2241900 [0] NCCL INFO Channel 19/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241631:2241899 [1] NCCL INFO Channel 21/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241630:2241900 [0] NCCL INFO Channel 20/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241631:2241899 [1] NCCL INFO Channel 22/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2241899 [1] NCCL INFO Channel 23/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241630:2241900 [0] NCCL INFO Channel 21/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241630:2241900 [0] NCCL INFO Channel 22/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241630:2241900 [0] NCCL INFO Channel 23/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2241631:2241899 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:2241630:2241900 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-09:19:24:18 INFO     [evaluator:559] Running loglikelihood requests
2025-12-09:19:24:18 INFO     [evaluator:559] Running loglikelihood requests

Running loglikelihood requests:   0%|          | 0/1838 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 64
Determined largest batch size: 64

Running loglikelihood requests:   0%|          | 1/1838 [00:07<3:47:06,  7.42s/it]
Running loglikelihood requests:   4%|▎         | 65/1838 [00:08<02:40, 11.06it/s] 
Running loglikelihood requests:   7%|▋         | 129/1838 [00:08<01:13, 23.34it/s]
Running loglikelihood requests:  11%|█         | 194/1838 [00:09<00:44, 37.02it/s]
Running loglikelihood requests:  14%|█▍        | 258/1838 [00:09<00:30, 51.25it/s]
Running loglikelihood requests:  18%|█▊        | 322/1838 [00:10<00:22, 66.07it/s]
Running loglikelihood requests:  21%|██        | 386/1838 [00:10<00:18, 80.46it/s]
Running loglikelihood requests:  25%|██▍       | 451/1838 [00:11<00:14, 93.95it/s]
Running loglikelihood requests:  28%|██▊       | 515/1838 [00:11<00:12, 106.27it/s]
Running loglikelihood requests:  32%|███▏      | 579/1838 [00:11<00:10, 117.52it/s]
Running loglikelihood requests:  35%|███▍      | 643/1838 [00:12<00:09, 127.70it/s]
Running loglikelihood requests:  38%|███▊      | 707/1838 [00:12<00:08, 134.94it/s]
Running loglikelihood requests:  42%|████▏     | 771/1838 [00:13<00:07, 142.27it/s]
Running loglikelihood requests:  45%|████▌     | 835/1838 [00:13<00:06, 148.54it/s]
Running loglikelihood requests:  49%|████▉     | 899/1838 [00:13<00:06, 154.20it/s]
Running loglikelihood requests:  52%|█████▏    | 964/1838 [00:14<00:05, 159.79it/s]
Running loglikelihood requests:  56%|█████▌    | 1028/1838 [00:14<00:04, 164.50it/s]
Running loglikelihood requests:  59%|█████▉    | 1092/1838 [00:15<00:04, 168.20it/s]
Running loglikelihood requests:  63%|██████▎   | 1157/1838 [00:15<00:04, 169.91it/s]
Running loglikelihood requests:  66%|██████▋   | 1221/1838 [00:15<00:03, 172.51it/s]
Running loglikelihood requests:  70%|██████▉   | 1286/1838 [00:16<00:03, 173.65it/s]
Running loglikelihood requests:  73%|███████▎  | 1350/1838 [00:16<00:02, 175.76it/s]
Running loglikelihood requests:  77%|███████▋  | 1414/1838 [00:16<00:02, 176.40it/s]
Running loglikelihood requests:  80%|████████  | 1478/1838 [00:17<00:02, 177.94it/s]
Running loglikelihood requests:  84%|████████▍ | 1542/1838 [00:17<00:01, 179.89it/s]
Running loglikelihood requests:  87%|████████▋ | 1606/1838 [00:17<00:01, 181.55it/s]
Running loglikelihood requests:  91%|█████████ | 1670/1838 [00:18<00:00, 185.00it/s]
Running loglikelihood requests:  94%|█████████▍| 1734/1838 [00:18<00:00, 188.07it/s]
Running loglikelihood requests:  98%|█████████▊| 1799/1838 [00:18<00:00, 213.02it/s]
Running loglikelihood requests: 100%|██████████| 1838/1838 [00:18<00:00, 98.02it/s] 
n136-128-154:2241631:2242079 [1] NCCL INFO Channel 00/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2242079 [1] NCCL INFO Channel 01/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2242079 [1] NCCL INFO Channel 02/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2242079 [1] NCCL INFO Channel 03/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2242079 [1] NCCL INFO Channel 04/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2242079 [1] NCCL INFO Channel 05/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2242079 [1] NCCL INFO Channel 06/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2242079 [1] NCCL INFO Channel 07/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2242079 [1] NCCL INFO Channel 08/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2242079 [1] NCCL INFO Channel 09/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2242079 [1] NCCL INFO Channel 10/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2242079 [1] NCCL INFO Channel 11/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2242079 [1] NCCL INFO Channel 12/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2242079 [1] NCCL INFO Channel 13/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2242079 [1] NCCL INFO Channel 14/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2242079 [1] NCCL INFO Channel 15/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2242079 [1] NCCL INFO Channel 16/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2242079 [1] NCCL INFO Channel 17/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2242079 [1] NCCL INFO Channel 18/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2242079 [1] NCCL INFO Channel 19/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2242079 [1] NCCL INFO Channel 20/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2242079 [1] NCCL INFO Channel 21/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2242079 [1] NCCL INFO Channel 22/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2242079 [1] NCCL INFO Channel 23/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2242079 [1] NCCL INFO Channel 24/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2242079 [1] NCCL INFO Channel 25/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2242079 [1] NCCL INFO Channel 26/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2242079 [1] NCCL INFO Channel 27/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2242079 [1] NCCL INFO Channel 28/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2242079 [1] NCCL INFO Channel 29/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2242079 [1] NCCL INFO Channel 30/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2241631:2242079 [1] NCCL INFO Channel 31/1 : 1[7] -> 0[6] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:2241631:2242083 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2241631:2242083 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2241631:2242083 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2241631:2242083 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2241631:2242083 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2241631:2242083 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2241631:2241895 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-09:19:24:46 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: auto (64)
|Tasks|Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|-----|------:|------|-----:|--------|---|-----:|---|-----:|
|piqa |      1|none  |     0|acc     |↑  |0.5332|±  |0.0116|
|     |       |none  |     0|acc_norm|↑  |0.5152|±  |0.0117|

[rank0]:[W1209 19:24:47.098294767 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:2241630:2242193 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2241630:2242193 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2241630:2242193 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2241630:2241897 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2241631:2241895 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2241630:2242193 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2241630:2242193 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2241630:2242193 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2241631:2242083 [1] NCCL INFO comm 0xea03000 rank 1 nranks 2 cudaDev 1 busId c9000 - Abort COMPLETE
n136-128-154:2241630:2242193 [0] NCCL INFO comm 0xe995780 rank 0 nranks 2 cudaDev 0 busId c5000 - Abort COMPLETE
任务 piqa 评估完成！

==================================================
开始评估：任务=hellaswag | 少样本数=10 | 模型=Qwen2.5-7B-quantization-fg
输出路径：results2/Qwen2.5-7B-quantization-fg/f6/hellaswag.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-09:19:25:39 INFO     [__main__:440] Selected Tasks: ['hellaswag']
2025-12-09:19:25:39 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:19:25:39 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-09:19:25:39 INFO     [__main__:440] Selected Tasks: ['hellaswag']
2025-12-09:19:25:39 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:19:25:39 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-09:19:25:40 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:19:25:40 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:19:25:40 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.17it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.15it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.14it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.13it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.20it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.19it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.70it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.46it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.69it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.45it/s]
2025-12-09:19:26:05 WARNING  [evaluator:309] Overwriting default num_fewshot of hellaswag from None to 10
2025-12-09:19:26:05 INFO     [api.task:434] Building contexts for hellaswag on rank 1...

  0%|          | 0/5021 [00:00<?, ?it/s]
  0%|          | 19/5021 [00:00<00:26, 187.57it/s]2025-12-09:19:26:07 WARNING  [evaluator:309] Overwriting default num_fewshot of hellaswag from None to 10
2025-12-09:19:26:07 INFO     [api.task:434] Building contexts for hellaswag on rank 0...

  1%|          | 39/5021 [00:00<00:25, 192.71it/s]
  1%|          | 59/5021 [00:00<00:25, 192.49it/s]
  2%|▏         | 79/5021 [00:00<00:25, 193.72it/s]
  2%|▏         | 99/5021 [00:00<00:25, 192.77it/s]
  2%|▏         | 119/5021 [00:00<00:26, 185.30it/s]
  3%|▎         | 138/5021 [00:00<00:26, 185.01it/s]
  3%|▎         | 157/5021 [00:00<00:26, 186.31it/s]
  4%|▎         | 177/5021 [00:00<00:25, 189.56it/s]
  4%|▍         | 197/5021 [00:01<00:25, 191.58it/s]
  4%|▍         | 217/5021 [00:01<00:24, 193.22it/s]
  0%|          | 0/5021 [00:00<?, ?it/s]
  5%|▍         | 237/5021 [00:01<00:24, 192.94it/s]
  0%|          | 19/5021 [00:00<00:26, 185.84it/s]
  5%|▌         | 257/5021 [00:01<00:24, 193.54it/s]
  1%|          | 39/5021 [00:00<00:25, 192.40it/s]
  6%|▌         | 277/5021 [00:01<00:24, 194.62it/s]
  1%|          | 59/5021 [00:00<00:25, 193.86it/s]
  6%|▌         | 297/5021 [00:01<00:24, 195.67it/s]
  2%|▏         | 79/5021 [00:00<00:25, 195.16it/s]
  6%|▋         | 317/5021 [00:01<00:23, 196.82it/s]
  2%|▏         | 99/5021 [00:00<00:25, 194.43it/s]
  7%|▋         | 337/5021 [00:01<00:24, 194.28it/s]
  2%|▏         | 119/5021 [00:00<00:25, 194.79it/s]
  7%|▋         | 357/5021 [00:01<00:23, 194.92it/s]
  3%|▎         | 139/5021 [00:00<00:24, 195.57it/s]
  8%|▊         | 377/5021 [00:01<00:23, 195.38it/s]
  3%|▎         | 159/5021 [00:00<00:25, 194.43it/s]
  8%|▊         | 397/5021 [00:02<00:23, 196.50it/s]
  4%|▎         | 179/5021 [00:00<00:24, 195.39it/s]
  8%|▊         | 417/5021 [00:02<00:23, 196.85it/s]
  4%|▍         | 199/5021 [00:01<00:24, 196.17it/s]
  9%|▊         | 437/5021 [00:02<00:23, 196.54it/s]
  4%|▍         | 219/5021 [00:01<00:24, 196.65it/s]
  9%|▉         | 457/5021 [00:02<00:23, 196.40it/s]
  5%|▍         | 239/5021 [00:01<00:24, 197.02it/s]
 10%|▉         | 477/5021 [00:02<00:23, 195.89it/s]
  5%|▌         | 259/5021 [00:01<00:24, 197.24it/s]
 10%|▉         | 497/5021 [00:02<00:23, 195.76it/s]
  6%|▌         | 279/5021 [00:01<00:23, 197.68it/s]
 10%|█         | 517/5021 [00:02<00:23, 194.82it/s]
  6%|▌         | 299/5021 [00:01<00:23, 197.60it/s]
 11%|█         | 537/5021 [00:02<00:23, 194.55it/s]
  6%|▋         | 319/5021 [00:01<00:23, 197.06it/s]
 11%|█         | 557/5021 [00:02<00:22, 194.87it/s]
  7%|▋         | 339/5021 [00:01<00:23, 196.99it/s]
 11%|█▏        | 577/5021 [00:02<00:22, 195.28it/s]
  7%|▋         | 359/5021 [00:01<00:23, 196.53it/s]
 12%|█▏        | 597/5021 [00:03<00:22, 192.47it/s]
  8%|▊         | 379/5021 [00:01<00:23, 195.64it/s]
 12%|█▏        | 617/5021 [00:03<00:22, 191.83it/s]
  8%|▊         | 399/5021 [00:02<00:23, 195.47it/s]
 13%|█▎        | 637/5021 [00:03<00:22, 192.00it/s]
  8%|▊         | 419/5021 [00:02<00:23, 195.22it/s]
 13%|█▎        | 657/5021 [00:03<00:22, 192.29it/s]
  9%|▊         | 439/5021 [00:02<00:23, 195.31it/s]
 13%|█▎        | 677/5021 [00:03<00:22, 192.59it/s]
  9%|▉         | 459/5021 [00:02<00:23, 194.60it/s]
 14%|█▍        | 697/5021 [00:03<00:22, 192.38it/s]
 10%|▉         | 479/5021 [00:02<00:23, 192.98it/s]
 14%|█▍        | 717/5021 [00:03<00:22, 193.12it/s]
 10%|▉         | 499/5021 [00:02<00:23, 192.37it/s]
 15%|█▍        | 737/5021 [00:03<00:22, 193.81it/s]
 10%|█         | 519/5021 [00:02<00:23, 192.07it/s]
 15%|█▌        | 757/5021 [00:03<00:21, 194.06it/s]
 11%|█         | 539/5021 [00:02<00:23, 192.36it/s]
 15%|█▌        | 777/5021 [00:04<00:21, 192.96it/s]
 11%|█         | 559/5021 [00:02<00:23, 192.95it/s]
 16%|█▌        | 797/5021 [00:04<00:21, 192.67it/s]
 12%|█▏        | 579/5021 [00:02<00:22, 193.37it/s]
 16%|█▋        | 817/5021 [00:04<00:21, 193.66it/s]
 12%|█▏        | 599/5021 [00:03<00:22, 193.60it/s]
 17%|█▋        | 837/5021 [00:04<00:21, 194.40it/s]
 12%|█▏        | 619/5021 [00:03<00:22, 193.68it/s]
 17%|█▋        | 857/5021 [00:04<00:21, 194.87it/s]
 13%|█▎        | 639/5021 [00:03<00:22, 193.68it/s]
 17%|█▋        | 877/5021 [00:04<00:21, 195.89it/s]
 13%|█▎        | 659/5021 [00:03<00:22, 193.57it/s]
 18%|█▊        | 897/5021 [00:04<00:21, 195.73it/s]
 14%|█▎        | 679/5021 [00:03<00:22, 193.98it/s]
 18%|█▊        | 917/5021 [00:04<00:20, 196.30it/s]
 14%|█▍        | 699/5021 [00:03<00:22, 194.26it/s]
 19%|█▊        | 937/5021 [00:04<00:20, 195.56it/s]
 14%|█▍        | 719/5021 [00:03<00:22, 194.55it/s]
 19%|█▉        | 957/5021 [00:04<00:20, 196.05it/s]
 15%|█▍        | 739/5021 [00:03<00:21, 195.03it/s]
 19%|█▉        | 977/5021 [00:05<00:20, 196.10it/s]
 15%|█▌        | 759/5021 [00:03<00:21, 195.04it/s]
 20%|█▉        | 997/5021 [00:05<00:21, 190.88it/s]
 16%|█▌        | 779/5021 [00:03<00:21, 194.85it/s]
 20%|██        | 1017/5021 [00:05<00:21, 188.30it/s]
 16%|█▌        | 799/5021 [00:04<00:21, 195.11it/s]
 21%|██        | 1037/5021 [00:05<00:21, 189.17it/s]
 16%|█▋        | 819/5021 [00:04<00:21, 192.82it/s]
 21%|██        | 1057/5021 [00:05<00:20, 191.11it/s]
 17%|█▋        | 839/5021 [00:04<00:21, 192.28it/s]
 21%|██▏       | 1077/5021 [00:05<00:20, 189.63it/s]
 17%|█▋        | 859/5021 [00:04<00:21, 192.42it/s]
 22%|██▏       | 1097/5021 [00:05<00:20, 190.97it/s]
 18%|█▊        | 879/5021 [00:04<00:22, 188.11it/s]
 22%|██▏       | 1117/5021 [00:05<00:20, 191.92it/s]
 18%|█▊        | 898/5021 [00:04<00:22, 180.89it/s]
 23%|██▎       | 1137/5021 [00:05<00:20, 192.84it/s]
 18%|█▊        | 917/5021 [00:04<00:22, 183.41it/s]
 23%|██▎       | 1157/5021 [00:05<00:19, 193.56it/s]
 19%|█▊        | 937/5021 [00:04<00:21, 185.97it/s]
 23%|██▎       | 1177/5021 [00:06<00:19, 194.40it/s]
 19%|█▉        | 957/5021 [00:04<00:21, 187.99it/s]
 24%|██▍       | 1197/5021 [00:06<00:19, 194.92it/s]
 19%|█▉        | 977/5021 [00:05<00:21, 189.51it/s]
 24%|██▍       | 1217/5021 [00:06<00:19, 195.17it/s]
 20%|█▉        | 997/5021 [00:05<00:21, 190.76it/s]
 25%|██▍       | 1237/5021 [00:06<00:19, 195.72it/s]
 20%|██        | 1017/5021 [00:05<00:20, 191.67it/s]
 25%|██▌       | 1257/5021 [00:06<00:19, 196.11it/s]
 21%|██        | 1037/5021 [00:05<00:20, 192.46it/s]
 25%|██▌       | 1277/5021 [00:06<00:19, 196.25it/s]
 26%|██▌       | 1297/5021 [00:06<00:19, 194.42it/s]
 21%|██        | 1057/5021 [00:05<00:20, 188.88it/s]
 26%|██▌       | 1317/5021 [00:06<00:19, 194.13it/s]
 21%|██▏       | 1077/5021 [00:05<00:20, 190.52it/s]
 27%|██▋       | 1337/5021 [00:06<00:18, 195.69it/s]
 22%|██▏       | 1097/5021 [00:05<00:20, 192.03it/s]
 27%|██▋       | 1357/5021 [00:07<00:18, 196.81it/s]
 22%|██▏       | 1117/5021 [00:05<00:20, 191.20it/s]
 27%|██▋       | 1377/5021 [00:07<00:18, 197.65it/s]
 23%|██▎       | 1137/5021 [00:05<00:20, 192.71it/s]
 28%|██▊       | 1397/5021 [00:07<00:18, 198.19it/s]
 23%|██▎       | 1157/5021 [00:05<00:19, 194.06it/s]
 28%|██▊       | 1417/5021 [00:07<00:18, 198.53it/s]
 23%|██▎       | 1177/5021 [00:06<00:19, 194.88it/s]
 29%|██▊       | 1437/5021 [00:07<00:18, 198.27it/s]
 24%|██▍       | 1197/5021 [00:06<00:19, 195.70it/s]
 29%|██▉       | 1457/5021 [00:07<00:17, 198.26it/s]
 24%|██▍       | 1217/5021 [00:06<00:19, 196.09it/s]
 29%|██▉       | 1477/5021 [00:07<00:17, 198.61it/s]
 25%|██▍       | 1237/5021 [00:06<00:19, 196.38it/s]
 30%|██▉       | 1497/5021 [00:07<00:17, 198.48it/s]
 25%|██▌       | 1257/5021 [00:06<00:19, 196.89it/s]
 30%|███       | 1517/5021 [00:07<00:17, 198.81it/s]
 25%|██▌       | 1277/5021 [00:06<00:18, 197.47it/s]
 31%|███       | 1537/5021 [00:07<00:17, 198.86it/s]
 26%|██▌       | 1297/5021 [00:06<00:18, 197.71it/s]
 31%|███       | 1557/5021 [00:08<00:17, 197.97it/s]
 26%|██▌       | 1317/5021 [00:06<00:18, 197.27it/s]
 31%|███▏      | 1577/5021 [00:08<00:17, 197.91it/s]
 27%|██▋       | 1337/5021 [00:06<00:18, 196.74it/s]
 32%|███▏      | 1597/5021 [00:08<00:17, 198.02it/s]
 27%|██▋       | 1357/5021 [00:07<00:18, 196.80it/s]
 32%|███▏      | 1617/5021 [00:08<00:17, 198.12it/s]
 27%|██▋       | 1377/5021 [00:07<00:18, 196.77it/s]
 33%|███▎      | 1637/5021 [00:08<00:17, 197.60it/s]
 28%|██▊       | 1397/5021 [00:07<00:18, 196.85it/s]
 33%|███▎      | 1657/5021 [00:08<00:17, 197.85it/s]
 28%|██▊       | 1417/5021 [00:07<00:18, 197.18it/s]
 33%|███▎      | 1677/5021 [00:08<00:17, 195.16it/s]
 29%|██▊       | 1437/5021 [00:07<00:18, 197.06it/s]
 34%|███▍      | 1697/5021 [00:08<00:17, 195.51it/s]
 29%|██▉       | 1457/5021 [00:07<00:18, 197.16it/s]
 34%|███▍      | 1717/5021 [00:08<00:16, 196.19it/s]
 29%|██▉       | 1477/5021 [00:07<00:17, 197.16it/s]
 35%|███▍      | 1737/5021 [00:08<00:16, 196.92it/s]
 30%|██▉       | 1497/5021 [00:07<00:17, 197.19it/s]
 35%|███▍      | 1757/5021 [00:09<00:16, 197.51it/s]
 30%|███       | 1517/5021 [00:07<00:17, 197.27it/s]
 35%|███▌      | 1778/5021 [00:09<00:16, 198.38it/s]
 31%|███       | 1537/5021 [00:07<00:17, 197.18it/s]
 36%|███▌      | 1798/5021 [00:09<00:16, 198.45it/s]
 31%|███       | 1557/5021 [00:08<00:17, 197.08it/s]
 36%|███▌      | 1818/5021 [00:09<00:16, 198.56it/s]
 31%|███▏      | 1577/5021 [00:08<00:17, 197.01it/s]
 37%|███▋      | 1838/5021 [00:09<00:16, 197.50it/s]
 32%|███▏      | 1597/5021 [00:08<00:17, 196.98it/s]
 37%|███▋      | 1858/5021 [00:09<00:15, 197.85it/s]
 32%|███▏      | 1617/5021 [00:08<00:17, 197.10it/s]
 37%|███▋      | 1878/5021 [00:09<00:15, 198.02it/s]
 33%|███▎      | 1637/5021 [00:08<00:17, 197.15it/s]
 38%|███▊      | 1898/5021 [00:09<00:15, 198.02it/s]
 33%|███▎      | 1657/5021 [00:08<00:17, 197.19it/s]
 38%|███▊      | 1918/5021 [00:09<00:15, 198.13it/s]
 33%|███▎      | 1677/5021 [00:08<00:16, 197.22it/s]
 39%|███▊      | 1938/5021 [00:09<00:15, 198.21it/s]
 34%|███▍      | 1697/5021 [00:08<00:16, 197.62it/s]
 39%|███▉      | 1958/5021 [00:10<00:15, 198.35it/s]
 34%|███▍      | 1717/5021 [00:08<00:16, 197.87it/s]
 39%|███▉      | 1978/5021 [00:10<00:15, 198.32it/s]
 35%|███▍      | 1737/5021 [00:08<00:16, 198.14it/s]
 40%|███▉      | 1998/5021 [00:10<00:15, 198.31it/s]
 35%|███▍      | 1757/5021 [00:09<00:16, 198.29it/s]
 40%|████      | 2018/5021 [00:10<00:15, 198.55it/s]
 35%|███▌      | 1777/5021 [00:09<00:16, 196.58it/s]
 41%|████      | 2038/5021 [00:10<00:15, 198.51it/s]
 36%|███▌      | 1797/5021 [00:09<00:16, 196.93it/s]
 41%|████      | 2058/5021 [00:10<00:14, 198.54it/s]
 36%|███▌      | 1817/5021 [00:09<00:16, 196.74it/s]
 41%|████▏     | 2078/5021 [00:10<00:14, 198.52it/s]
 37%|███▋      | 1837/5021 [00:09<00:16, 196.91it/s]
 42%|████▏     | 2098/5021 [00:10<00:14, 198.23it/s]
 37%|███▋      | 1857/5021 [00:09<00:16, 196.55it/s]
 42%|████▏     | 2118/5021 [00:10<00:14, 198.59it/s]
 37%|███▋      | 1877/5021 [00:09<00:16, 196.41it/s]
 43%|████▎     | 2138/5021 [00:10<00:14, 197.83it/s]
 38%|███▊      | 1897/5021 [00:09<00:15, 196.75it/s]
 43%|████▎     | 2158/5021 [00:11<00:14, 198.42it/s]
 38%|███▊      | 1917/5021 [00:09<00:15, 197.04it/s]
 43%|████▎     | 2178/5021 [00:11<00:14, 198.85it/s]
 39%|███▊      | 1937/5021 [00:09<00:15, 197.47it/s]
 44%|████▍     | 2198/5021 [00:11<00:14, 198.98it/s]
 39%|███▉      | 1957/5021 [00:10<00:15, 197.21it/s]
 44%|████▍     | 2219/5021 [00:11<00:14, 199.40it/s]
 39%|███▉      | 1977/5021 [00:10<00:15, 197.36it/s]
 45%|████▍     | 2239/5021 [00:11<00:13, 199.52it/s]
 40%|███▉      | 1997/5021 [00:10<00:15, 197.60it/s]
 45%|████▍     | 2259/5021 [00:11<00:13, 199.33it/s]
 40%|████      | 2017/5021 [00:10<00:15, 198.07it/s]
 45%|████▌     | 2279/5021 [00:11<00:13, 196.66it/s]
 41%|████      | 2037/5021 [00:10<00:15, 198.00it/s]
 46%|████▌     | 2299/5021 [00:11<00:13, 195.96it/s]
 41%|████      | 2057/5021 [00:10<00:14, 197.92it/s]
 46%|████▌     | 2319/5021 [00:11<00:13, 195.94it/s]
 41%|████▏     | 2077/5021 [00:10<00:14, 198.36it/s]
 47%|████▋     | 2339/5021 [00:11<00:13, 196.45it/s]
 42%|████▏     | 2098/5021 [00:10<00:14, 198.97it/s]
 47%|████▋     | 2359/5021 [00:12<00:13, 196.81it/s]
 42%|████▏     | 2118/5021 [00:10<00:14, 199.12it/s]
 47%|████▋     | 2379/5021 [00:12<00:13, 197.46it/s]
 43%|████▎     | 2139/5021 [00:10<00:14, 199.34it/s]
 48%|████▊     | 2399/5021 [00:12<00:13, 197.96it/s]
 43%|████▎     | 2159/5021 [00:11<00:14, 199.40it/s]
 48%|████▊     | 2419/5021 [00:12<00:13, 198.09it/s]
 43%|████▎     | 2179/5021 [00:11<00:14, 199.33it/s]
 49%|████▊     | 2439/5021 [00:12<00:13, 198.25it/s]
 44%|████▍     | 2199/5021 [00:11<00:14, 199.17it/s]
 49%|████▉     | 2459/5021 [00:12<00:12, 198.38it/s]
 44%|████▍     | 2219/5021 [00:11<00:14, 198.85it/s]
 49%|████▉     | 2479/5021 [00:12<00:12, 198.62it/s]
 45%|████▍     | 2239/5021 [00:11<00:14, 198.63it/s]
 50%|████▉     | 2499/5021 [00:12<00:12, 198.79it/s]
 45%|████▍     | 2259/5021 [00:11<00:13, 198.56it/s]
 50%|█████     | 2519/5021 [00:12<00:12, 199.09it/s]
 45%|████▌     | 2279/5021 [00:11<00:13, 198.97it/s]
 51%|█████     | 2539/5021 [00:12<00:12, 199.17it/s]
 46%|████▌     | 2299/5021 [00:11<00:13, 198.76it/s]
 51%|█████     | 2560/5021 [00:13<00:12, 199.81it/s]
 46%|████▌     | 2319/5021 [00:11<00:13, 198.29it/s]
 51%|█████▏    | 2581/5021 [00:13<00:12, 200.13it/s]
 47%|████▋     | 2339/5021 [00:11<00:13, 198.09it/s]
 52%|█████▏    | 2602/5021 [00:13<00:12, 200.53it/s]
 47%|████▋     | 2359/5021 [00:12<00:13, 198.09it/s]
 52%|█████▏    | 2623/5021 [00:13<00:12, 199.61it/s]
 47%|████▋     | 2379/5021 [00:12<00:13, 198.09it/s]
 53%|█████▎    | 2644/5021 [00:13<00:11, 200.01it/s]
 48%|████▊     | 2399/5021 [00:12<00:13, 198.30it/s]
 53%|█████▎    | 2665/5021 [00:13<00:11, 200.08it/s]
 48%|████▊     | 2419/5021 [00:12<00:13, 196.21it/s]
 49%|████▊     | 2439/5021 [00:12<00:13, 196.04it/s]
 53%|█████▎    | 2686/5021 [00:13<00:11, 199.66it/s]
 54%|█████▍    | 2706/5021 [00:13<00:11, 199.67it/s]
 49%|████▉     | 2459/5021 [00:12<00:13, 195.96it/s]
 54%|█████▍    | 2726/5021 [00:13<00:11, 199.05it/s]
 49%|████▉     | 2479/5021 [00:12<00:12, 196.05it/s]
 55%|█████▍    | 2746/5021 [00:14<00:11, 198.65it/s]
 50%|████▉     | 2499/5021 [00:12<00:12, 196.29it/s]
 55%|█████▌    | 2766/5021 [00:14<00:11, 198.76it/s]
 50%|█████     | 2519/5021 [00:12<00:12, 195.72it/s]
 55%|█████▌    | 2786/5021 [00:14<00:11, 198.27it/s]
 51%|█████     | 2539/5021 [00:12<00:12, 193.98it/s]
 56%|█████▌    | 2806/5021 [00:14<00:11, 198.53it/s]
 51%|█████     | 2559/5021 [00:13<00:12, 193.84it/s]
 56%|█████▋    | 2826/5021 [00:14<00:11, 198.94it/s]
 51%|█████▏    | 2579/5021 [00:13<00:12, 194.81it/s]
 57%|█████▋    | 2846/5021 [00:14<00:10, 199.07it/s]
 52%|█████▏    | 2599/5021 [00:13<00:12, 195.74it/s]
 57%|█████▋    | 2866/5021 [00:14<00:10, 198.72it/s]
 52%|█████▏    | 2619/5021 [00:13<00:12, 196.12it/s]
 57%|█████▋    | 2886/5021 [00:14<00:10, 198.99it/s]
 53%|█████▎    | 2639/5021 [00:13<00:12, 196.40it/s]
 58%|█████▊    | 2907/5021 [00:14<00:10, 199.49it/s]
 53%|█████▎    | 2659/5021 [00:13<00:12, 196.52it/s]
 58%|█████▊    | 2928/5021 [00:14<00:10, 199.69it/s]
 53%|█████▎    | 2679/5021 [00:13<00:11, 196.91it/s]
 59%|█████▊    | 2949/5021 [00:15<00:10, 199.95it/s]
 54%|█████▍    | 2699/5021 [00:13<00:12, 192.63it/s]
 59%|█████▉    | 2969/5021 [00:15<00:10, 199.88it/s]
 54%|█████▍    | 2719/5021 [00:13<00:11, 193.13it/s]
 60%|█████▉    | 2990/5021 [00:15<00:10, 200.16it/s]
 55%|█████▍    | 2739/5021 [00:14<00:11, 193.81it/s]
 60%|█████▉    | 3011/5021 [00:15<00:10, 200.23it/s]
 55%|█████▍    | 2759/5021 [00:14<00:11, 193.33it/s]
 60%|██████    | 3032/5021 [00:15<00:09, 199.04it/s]
 55%|█████▌    | 2779/5021 [00:14<00:11, 193.13it/s]
 61%|██████    | 3052/5021 [00:15<00:09, 197.89it/s]
 56%|█████▌    | 2799/5021 [00:14<00:11, 192.79it/s]
 61%|██████    | 3072/5021 [00:15<00:09, 197.04it/s]
 56%|█████▌    | 2819/5021 [00:14<00:11, 192.53it/s]
 62%|██████▏   | 3092/5021 [00:15<00:09, 196.32it/s]
 57%|█████▋    | 2839/5021 [00:14<00:11, 192.82it/s]
 62%|██████▏   | 3112/5021 [00:15<00:09, 195.95it/s]
 57%|█████▋    | 2859/5021 [00:14<00:11, 192.79it/s]
 62%|██████▏   | 3132/5021 [00:15<00:09, 195.75it/s]
 57%|█████▋    | 2879/5021 [00:14<00:11, 193.19it/s]
 63%|██████▎   | 3152/5021 [00:16<00:09, 195.66it/s]
 58%|█████▊    | 2899/5021 [00:14<00:10, 193.42it/s]
 63%|██████▎   | 3172/5021 [00:16<00:09, 195.85it/s]
 58%|█████▊    | 2919/5021 [00:14<00:10, 193.24it/s]
 64%|██████▎   | 3192/5021 [00:16<00:09, 195.62it/s]
 59%|█████▊    | 2939/5021 [00:15<00:10, 193.50it/s]
 64%|██████▍   | 3212/5021 [00:16<00:09, 195.36it/s]
 59%|█████▉    | 2959/5021 [00:15<00:10, 193.84it/s]
 64%|██████▍   | 3232/5021 [00:16<00:09, 196.30it/s]
 59%|█████▉    | 2979/5021 [00:15<00:10, 195.39it/s]
 65%|██████▍   | 3253/5021 [00:16<00:08, 197.56it/s]
 60%|█████▉    | 2999/5021 [00:15<00:10, 196.16it/s]
 65%|██████▌   | 3274/5021 [00:16<00:08, 198.36it/s]
 60%|██████    | 3019/5021 [00:15<00:10, 196.48it/s]
 66%|██████▌   | 3294/5021 [00:16<00:08, 198.83it/s]
 61%|██████    | 3039/5021 [00:15<00:10, 194.40it/s]
 66%|██████▌   | 3314/5021 [00:16<00:08, 199.06it/s]
 61%|██████    | 3059/5021 [00:15<00:10, 195.50it/s]
 66%|██████▋   | 3335/5021 [00:16<00:08, 199.39it/s]
 61%|██████▏   | 3079/5021 [00:15<00:09, 196.11it/s]
 67%|██████▋   | 3356/5021 [00:17<00:08, 199.63it/s]
 62%|██████▏   | 3099/5021 [00:15<00:09, 196.66it/s]
 67%|██████▋   | 3377/5021 [00:17<00:08, 199.90it/s]
 62%|██████▏   | 3119/5021 [00:15<00:09, 197.06it/s]
 68%|██████▊   | 3398/5021 [00:17<00:08, 200.28it/s]
 63%|██████▎   | 3139/5021 [00:16<00:09, 197.21it/s]
 68%|██████▊   | 3419/5021 [00:17<00:07, 200.37it/s]
 63%|██████▎   | 3159/5021 [00:16<00:09, 196.96it/s]
 63%|██████▎   | 3179/5021 [00:16<00:09, 196.61it/s]
 69%|██████▊   | 3440/5021 [00:17<00:07, 200.31it/s]
 64%|██████▎   | 3199/5021 [00:16<00:09, 196.54it/s]
 69%|██████▉   | 3461/5021 [00:17<00:07, 200.19it/s]
 64%|██████▍   | 3219/5021 [00:16<00:09, 196.72it/s]
 69%|██████▉   | 3482/5021 [00:17<00:07, 199.88it/s]
 65%|██████▍   | 3239/5021 [00:16<00:09, 197.38it/s]
 70%|██████▉   | 3502/5021 [00:17<00:07, 199.68it/s]
 65%|██████▍   | 3259/5021 [00:16<00:08, 197.65it/s]
 70%|███████   | 3522/5021 [00:17<00:07, 199.60it/s]
 65%|██████▌   | 3279/5021 [00:16<00:08, 197.89it/s]
 71%|███████   | 3542/5021 [00:18<00:07, 199.61it/s]
 66%|██████▌   | 3299/5021 [00:16<00:08, 198.02it/s]
 71%|███████   | 3562/5021 [00:18<00:07, 199.53it/s]
 66%|██████▌   | 3319/5021 [00:16<00:08, 198.06it/s]
 71%|███████▏  | 3582/5021 [00:18<00:07, 199.35it/s]
 67%|██████▋   | 3339/5021 [00:17<00:08, 198.50it/s]
 72%|███████▏  | 3603/5021 [00:18<00:07, 199.62it/s]
 67%|██████▋   | 3359/5021 [00:17<00:08, 198.80it/s]
 72%|███████▏  | 3624/5021 [00:18<00:06, 199.80it/s]
 67%|██████▋   | 3379/5021 [00:17<00:08, 199.07it/s]
 73%|███████▎  | 3644/5021 [00:18<00:06, 198.23it/s]
 68%|██████▊   | 3399/5021 [00:17<00:08, 198.82it/s]
 73%|███████▎  | 3664/5021 [00:18<00:06, 197.44it/s]
 68%|██████▊   | 3419/5021 [00:17<00:08, 198.21it/s]
 73%|███████▎  | 3684/5021 [00:18<00:06, 196.82it/s]
 68%|██████▊   | 3439/5021 [00:17<00:08, 197.61it/s]
 74%|███████▍  | 3704/5021 [00:18<00:06, 196.43it/s]
 69%|██████▉   | 3459/5021 [00:17<00:07, 197.44it/s]
 74%|███████▍  | 3724/5021 [00:18<00:06, 196.53it/s]
 69%|██████▉   | 3479/5021 [00:17<00:07, 197.66it/s]
 75%|███████▍  | 3744/5021 [00:19<00:06, 197.15it/s]
 70%|██████▉   | 3499/5021 [00:17<00:07, 197.67it/s]
 75%|███████▍  | 3764/5021 [00:19<00:06, 197.74it/s]
 70%|███████   | 3519/5021 [00:17<00:07, 197.67it/s]
 75%|███████▌  | 3784/5021 [00:19<00:06, 198.16it/s]
 70%|███████   | 3539/5021 [00:18<00:07, 197.68it/s]
 76%|███████▌  | 3804/5021 [00:19<00:06, 198.44it/s]
 71%|███████   | 3559/5021 [00:18<00:07, 197.43it/s]
 76%|███████▌  | 3824/5021 [00:19<00:06, 196.82it/s]
 71%|███████▏  | 3579/5021 [00:18<00:07, 197.56it/s]
 77%|███████▋  | 3844/5021 [00:19<00:05, 197.55it/s]
 72%|███████▏  | 3599/5021 [00:18<00:07, 197.63it/s]
 77%|███████▋  | 3865/5021 [00:19<00:05, 198.40it/s]
 72%|███████▏  | 3619/5021 [00:18<00:07, 197.83it/s]
 77%|███████▋  | 3885/5021 [00:19<00:05, 198.72it/s]
 72%|███████▏  | 3640/5021 [00:18<00:06, 198.72it/s]
 78%|███████▊  | 3905/5021 [00:19<00:05, 198.39it/s]
 73%|███████▎  | 3660/5021 [00:18<00:06, 198.84it/s]
 78%|███████▊  | 3925/5021 [00:19<00:05, 198.67it/s]
 73%|███████▎  | 3680/5021 [00:18<00:06, 198.30it/s]
 79%|███████▊  | 3945/5021 [00:20<00:05, 198.75it/s]
 74%|███████▎  | 3700/5021 [00:18<00:06, 198.16it/s]
 79%|███████▉  | 3965/5021 [00:20<00:05, 198.49it/s]
 74%|███████▍  | 3720/5021 [00:19<00:06, 198.37it/s]
 79%|███████▉  | 3985/5021 [00:20<00:05, 198.71it/s]
 74%|███████▍  | 3740/5021 [00:19<00:06, 198.32it/s]
 80%|███████▉  | 4006/5021 [00:20<00:05, 199.27it/s]
 75%|███████▍  | 3760/5021 [00:19<00:06, 198.61it/s]
 80%|████████  | 4026/5021 [00:20<00:04, 199.05it/s]
 75%|███████▌  | 3780/5021 [00:19<00:06, 198.89it/s]
 81%|████████  | 4046/5021 [00:20<00:04, 197.83it/s]
 76%|███████▌  | 3800/5021 [00:19<00:06, 198.74it/s]
 81%|████████  | 4066/5021 [00:20<00:04, 198.45it/s]
 76%|███████▌  | 3820/5021 [00:19<00:06, 198.38it/s]
 81%|████████▏ | 4086/5021 [00:20<00:04, 198.64it/s]
 76%|███████▋  | 3840/5021 [00:19<00:05, 197.77it/s]
 82%|████████▏ | 4107/5021 [00:20<00:04, 199.17it/s]
 77%|███████▋  | 3860/5021 [00:19<00:05, 197.99it/s]
 82%|████████▏ | 4127/5021 [00:20<00:04, 199.39it/s]
 77%|███████▋  | 3880/5021 [00:19<00:05, 198.06it/s]
 83%|████████▎ | 4148/5021 [00:21<00:04, 199.61it/s]
 78%|███████▊  | 3900/5021 [00:19<00:05, 197.98it/s]
 83%|████████▎ | 4168/5021 [00:21<00:04, 199.60it/s]
 78%|███████▊  | 3920/5021 [00:20<00:05, 197.76it/s]
 83%|████████▎ | 4188/5021 [00:21<00:04, 199.56it/s]
 78%|███████▊  | 3940/5021 [00:20<00:05, 197.85it/s]
 84%|████████▍ | 4209/5021 [00:21<00:04, 199.82it/s]
 79%|███████▉  | 3960/5021 [00:20<00:05, 197.73it/s]
 84%|████████▍ | 4229/5021 [00:21<00:03, 199.78it/s]
 79%|███████▉  | 3980/5021 [00:20<00:05, 197.92it/s]
 85%|████████▍ | 4250/5021 [00:21<00:03, 200.19it/s]
 80%|███████▉  | 4000/5021 [00:20<00:05, 198.03it/s]
 85%|████████▌ | 4271/5021 [00:21<00:03, 199.67it/s]
 80%|████████  | 4020/5021 [00:20<00:05, 197.46it/s]
 85%|████████▌ | 4291/5021 [00:21<00:03, 198.99it/s]
 80%|████████  | 4040/5021 [00:20<00:04, 197.49it/s]
 86%|████████▌ | 4311/5021 [00:21<00:03, 198.83it/s]
 81%|████████  | 4060/5021 [00:20<00:04, 197.97it/s]
 86%|████████▋ | 4331/5021 [00:21<00:03, 198.71it/s]
 81%|████████▏ | 4080/5021 [00:20<00:04, 197.86it/s]
 87%|████████▋ | 4351/5021 [00:22<00:03, 198.58it/s]
 82%|████████▏ | 4100/5021 [00:20<00:04, 198.04it/s]
 87%|████████▋ | 4371/5021 [00:22<00:03, 198.71it/s]
 82%|████████▏ | 4120/5021 [00:21<00:04, 198.51it/s]
 87%|████████▋ | 4392/5021 [00:22<00:03, 199.24it/s]
 82%|████████▏ | 4140/5021 [00:21<00:04, 198.83it/s]
 88%|████████▊ | 4413/5021 [00:22<00:03, 199.63it/s]
 83%|████████▎ | 4160/5021 [00:21<00:04, 198.89it/s]
 88%|████████▊ | 4434/5021 [00:22<00:02, 199.83it/s]
 83%|████████▎ | 4180/5021 [00:21<00:04, 199.00it/s]
 89%|████████▊ | 4455/5021 [00:22<00:02, 200.08it/s]
 84%|████████▎ | 4200/5021 [00:21<00:04, 198.78it/s]
 89%|████████▉ | 4476/5021 [00:22<00:02, 199.78it/s]
 84%|████████▍ | 4220/5021 [00:21<00:04, 198.64it/s]
 90%|████████▉ | 4497/5021 [00:22<00:02, 200.13it/s]
 84%|████████▍ | 4240/5021 [00:21<00:03, 198.08it/s]
 90%|████████▉ | 4518/5021 [00:22<00:02, 200.28it/s]
 85%|████████▍ | 4260/5021 [00:21<00:03, 197.79it/s]
 90%|█████████ | 4539/5021 [00:23<00:02, 200.45it/s]
 85%|████████▌ | 4280/5021 [00:21<00:03, 197.86it/s]
 91%|█████████ | 4560/5021 [00:23<00:02, 200.44it/s]
 86%|████████▌ | 4300/5021 [00:21<00:03, 197.78it/s]
 91%|█████████ | 4581/5021 [00:23<00:02, 200.63it/s]
 86%|████████▌ | 4320/5021 [00:22<00:03, 196.99it/s]
 92%|█████████▏| 4602/5021 [00:23<00:02, 201.15it/s]
 86%|████████▋ | 4340/5021 [00:22<00:03, 196.88it/s]
 92%|█████████▏| 4623/5021 [00:23<00:01, 201.55it/s]
 87%|████████▋ | 4360/5021 [00:22<00:03, 196.89it/s]
 92%|█████████▏| 4644/5021 [00:23<00:01, 201.51it/s]
 87%|████████▋ | 4380/5021 [00:22<00:03, 197.21it/s]
 93%|█████████▎| 4665/5021 [00:23<00:01, 201.20it/s]
 88%|████████▊ | 4400/5021 [00:22<00:03, 197.49it/s]
 93%|█████████▎| 4686/5021 [00:23<00:01, 201.26it/s]
 88%|████████▊ | 4420/5021 [00:22<00:03, 197.48it/s]
 94%|█████████▎| 4707/5021 [00:23<00:01, 200.87it/s]
 88%|████████▊ | 4440/5021 [00:22<00:02, 197.68it/s]
 94%|█████████▍| 4728/5021 [00:23<00:01, 200.69it/s]
 89%|████████▉ | 4460/5021 [00:22<00:02, 198.08it/s]
 89%|████████▉ | 4480/5021 [00:22<00:02, 198.23it/s]
 95%|█████████▍| 4749/5021 [00:24<00:01, 200.83it/s]
 90%|████████▉ | 4500/5021 [00:22<00:02, 198.73it/s]
 95%|█████████▌| 4770/5021 [00:24<00:01, 200.64it/s]
 90%|█████████ | 4521/5021 [00:23<00:02, 199.13it/s]
 95%|█████████▌| 4791/5021 [00:24<00:01, 200.21it/s]
 90%|█████████ | 4541/5021 [00:23<00:02, 199.32it/s]
 96%|█████████▌| 4812/5021 [00:24<00:01, 200.10it/s]
 91%|█████████ | 4561/5021 [00:23<00:02, 199.51it/s]
 96%|█████████▋| 4833/5021 [00:24<00:00, 199.81it/s]
 91%|█████████▏| 4582/5021 [00:23<00:02, 199.67it/s]
 97%|█████████▋| 4853/5021 [00:24<00:00, 199.76it/s]
 92%|█████████▏| 4602/5021 [00:23<00:02, 199.43it/s]
 97%|█████████▋| 4873/5021 [00:24<00:00, 199.82it/s]
 92%|█████████▏| 4622/5021 [00:23<00:02, 199.40it/s]
 97%|█████████▋| 4894/5021 [00:24<00:00, 199.89it/s]
 92%|█████████▏| 4642/5021 [00:23<00:01, 199.38it/s]
 98%|█████████▊| 4914/5021 [00:24<00:00, 199.58it/s]
 93%|█████████▎| 4662/5021 [00:23<00:01, 198.91it/s]
 98%|█████████▊| 4934/5021 [00:25<00:00, 199.55it/s]
 93%|█████████▎| 4682/5021 [00:23<00:01, 198.34it/s]
 99%|█████████▊| 4954/5021 [00:25<00:00, 198.53it/s]
 94%|█████████▎| 4702/5021 [00:23<00:01, 197.81it/s]
 99%|█████████▉| 4974/5021 [00:25<00:00, 198.21it/s]
 94%|█████████▍| 4722/5021 [00:24<00:01, 197.82it/s]
 99%|█████████▉| 4994/5021 [00:25<00:00, 198.40it/s]
 94%|█████████▍| 4742/5021 [00:24<00:01, 197.81it/s]
100%|█████████▉| 5015/5021 [00:25<00:00, 199.47it/s]
100%|██████████| 5021/5021 [00:25<00:00, 197.37it/s]

 95%|█████████▍| 4762/5021 [00:24<00:01, 197.64it/s]
 95%|█████████▌| 4782/5021 [00:24<00:01, 197.54it/s]
 96%|█████████▌| 4802/5021 [00:24<00:01, 197.38it/s]
 96%|█████████▌| 4822/5021 [00:24<00:01, 197.51it/s]
 96%|█████████▋| 4842/5021 [00:24<00:00, 197.50it/s]
 97%|█████████▋| 4862/5021 [00:24<00:00, 198.00it/s]
 97%|█████████▋| 4882/5021 [00:24<00:00, 198.10it/s]
 98%|█████████▊| 4902/5021 [00:24<00:00, 198.32it/s]
 98%|█████████▊| 4922/5021 [00:25<00:00, 198.62it/s]
 98%|█████████▊| 4943/5021 [00:25<00:00, 199.00it/s]
 99%|█████████▉| 4964/5021 [00:25<00:00, 199.73it/s]
 99%|█████████▉| 4984/5021 [00:25<00:00, 199.56it/s]
100%|█████████▉| 5004/5021 [00:25<00:00, 199.56it/s]
100%|██████████| 5021/5021 [00:25<00:00, 196.39it/s]
n136-128-154:2242239:2242239 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2242239:2242239 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2242239:2242239 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:2242239:2242239 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2242239:2242239 [0] NCCL INFO Comm config Blocking set to 1
n136-128-154:2242240:2242240 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:2242240:2242240 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2242240:2242240 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2242240:2242240 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2242240:2242240 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:2242239:2242479 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2242239:2242479 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2242240:2242480 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2242240:2242480 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2242239:2242479 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2242239:2242479 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2242239:2242479 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2242240:2242480 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2242240:2242480 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2242240:2242480 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2242239:2242479 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2242239:2242479 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2242239:2242479 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:2242239:2242479 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2242239:2242479 [0] NCCL INFO Using network IB
n136-128-154:2242240:2242480 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2242240:2242480 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2242240:2242480 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:2242240:2242480 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2242240:2242480 [1] NCCL INFO Using network IB
n136-128-154:2242240:2242480 [1] NCCL INFO ncclCommInitRankConfig comm 0x47be4560 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0x624137be80e2f945 - Init START
n136-128-154:2242239:2242479 [0] NCCL INFO ncclCommInitRankConfig comm 0xe2505d0 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0x624137be80e2f945 - Init START
n136-128-154:2242239:2242479 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2242240:2242480 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2242240:2242480 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2242240:2242480 [1] NCCL INFO Retrieving state for IB
n136-128-154:2242240:2242480 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:2242240:2242480 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2242239:2242479 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2242239:2242479 [0] NCCL INFO Retrieving state for IB
n136-128-154:2242239:2242479 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:2242240:2242480 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2242239:2242479 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2242240:2242480 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2242239:2242479 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2242240:2242480 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2242239:2242479 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2242239:2242479 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2242240:2242480 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2242240:2242480 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2242239:2242479 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2242239:2242479 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2242240:2242480 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2242240:2242480 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2242240:2242480 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2242240:2242480 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2242240:2242480 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2242240:2242480 [1] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2242240:2242480 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2242240:2242480 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2242240:2242480 [1] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2242240:2242480 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2242240:2242480 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2242240:2242480 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2242240:2242480 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2242240:2242480 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2242240:2242480 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2242240:2242480 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2242240:2242480 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2242240:2242480 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2242240:2242480 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2242240:2242480 [1] NCCL INFO ==========================================
n136-128-154:2242240:2242480 [1] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2242240:2242480 [1] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2242239:2242479 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2242239:2242479 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2242239:2242479 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2242240:2242480 [1] NCCL INFO Setting affinity for GPU 7 to 33-62,97-126
n136-128-154:2242239:2242479 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2242239:2242479 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2242239:2242479 [0] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2242239:2242479 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2242239:2242479 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2242239:2242479 [0] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2242239:2242479 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2242239:2242479 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2242239:2242479 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2242239:2242479 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2242239:2242479 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2242239:2242479 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2242239:2242479 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2242239:2242479 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2242239:2242479 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2242239:2242479 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2242239:2242479 [0] NCCL INFO ==========================================
n136-128-154:2242239:2242479 [0] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2242239:2242479 [0] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2242239:2242479 [0] NCCL INFO Setting affinity for GPU 6 to 33-62,97-126
n136-128-154:2242239:2242479 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2242239:2242479 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2242239:2242479 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2242239:2242479 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2242239:2242479 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2242239:2242479 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2242239:2242479 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2242239:2242479 [0] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2242239:2242479 [0] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2242239:2242479 [0] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2242240:2242480 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2242239:2242479 [0] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2242239:2242479 [0] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2242240:2242480 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2242239:2242479 [0] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2242240:2242480 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2242240:2242480 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2242240:2242480 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2242240:2242480 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2242240:2242480 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2242240:2242480 [1] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2242240:2242480 [1] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2242240:2242480 [1] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2242240:2242480 [1] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2242240:2242480 [1] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2242240:2242480 [1] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2242240:2242480 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2242240:2242480 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2242240:2242480 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2242240:2242480 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2242240:2242480 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2242240:2242480 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2242239:2242479 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2242240:2242480 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2242239:2242479 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2242240:2242480 [1] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2242239:2242479 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2242240:2242480 [1] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2242239:2242479 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2242240:2242480 [1] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2242239:2242479 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2242240:2242480 [1] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2242239:2242479 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2242240:2242480 [1] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2242239:2242479 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2242240:2242480 [1] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2242239:2242479 [0] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2242239:2242479 [0] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2242239:2242479 [0] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2242239:2242479 [0] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2242239:2242479 [0] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2242239:2242479 [0] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2242240:2242480 [1] NCCL INFO comm 0x47be4560 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:2242239:2242479 [0] NCCL INFO comm 0xe2505d0 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:2242240:2242480 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:2242240:2242480 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:2242239:2242479 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:2242240:2242480 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:2242239:2242479 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:2242240:2242480 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:2242239:2242479 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:2242240:2242480 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:2242239:2242479 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:2242240:2242480 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:2242239:2242479 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:2242240:2242480 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:2242239:2242479 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:2242240:2242480 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:2242239:2242479 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:2242240:2242480 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:2242239:2242479 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:2242240:2242480 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:2242239:2242479 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:2242240:2242480 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:2242239:2242479 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:2242240:2242480 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:2242239:2242479 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:2242240:2242480 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:2242239:2242479 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:2242240:2242480 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:2242239:2242479 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:2242240:2242480 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:2242239:2242479 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:2242240:2242480 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:2242239:2242479 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:2242240:2242480 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:2242239:2242479 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:2242240:2242480 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:2242239:2242479 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:2242240:2242480 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:2242239:2242479 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:2242240:2242480 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:2242239:2242479 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:2242240:2242480 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:2242239:2242479 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:2242240:2242480 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:2242239:2242479 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:2242240:2242480 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:2242239:2242479 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:2242240:2242480 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:2242239:2242479 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:2242239:2242479 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:2242239:2242479 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:2242239:2242479 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:2242240:2242480 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:2242239:2242479 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:2242240:2242480 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:2242239:2242479 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:2242240:2242480 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:2242239:2242479 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:2242240:2242480 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:2242239:2242479 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:2242240:2242480 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:2242239:2242479 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:2242240:2242480 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:2242239:2242479 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:2242240:2242480 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:2242239:2242479 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:2242240:2242480 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:2242239:2242479 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:2242240:2242480 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:2242239:2242479 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:2242240:2242480 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:2242239:2242479 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:2242240:2242480 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:2242239:2242479 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:2242240:2242480 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:2242239:2242479 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:2242240:2242480 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:2242239:2242479 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:2242240:2242480 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:2242239:2242479 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:2242240:2242480 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:2242239:2242479 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:2242240:2242480 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:2242239:2242479 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:2242240:2242480 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:2242239:2242479 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:2242240:2242480 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:2242239:2242479 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:2242240:2242480 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:2242239:2242479 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:2242240:2242480 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:2242239:2242479 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:2242240:2242480 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:2242239:2242479 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:2242240:2242480 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:2242239:2242479 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:2242240:2242480 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:2242239:2242479 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:2242240:2242480 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:2242239:2242479 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:2242240:2242480 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:2242239:2242479 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:2242239:2242479 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:2242240:2242480 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2242239:2242479 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:2242239:2242479 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:2242239:2242479 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:2242239:2242479 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:2242239:2242479 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:2242239:2242479 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:2242239:2242479 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:2242239:2242479 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:2242239:2242479 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:2242239:2242479 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:2242239:2242479 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:2242239:2242479 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:2242239:2242479 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:2242239:2242479 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:2242239:2242479 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:2242239:2242479 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:2242239:2242479 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:2242239:2242479 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:2242239:2242479 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:2242239:2242479 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:2242239:2242479 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:2242239:2242479 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2242239:2242479 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2242239:2242479 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:2242239:2242492 [0] NCCL INFO [Proxy Service] Device 0 CPU core 99
n136-128-154:2242239:2242493 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 100
n136-128-154:2242240:2242480 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2242240:2242494 [1] NCCL INFO [Proxy Service] Device 1 CPU core 104
n136-128-154:2242240:2242495 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 39
n136-128-154:2242240:2242480 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2242240:2242480 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2242239:2242479 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2242239:2242479 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2242239:2242479 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:2242240:2242480 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2242240:2242480 [1] NCCL INFO ncclCommInitRankConfig comm 0x47be4560 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0x624137be80e2f945 - Init COMPLETE
n136-128-154:2242240:2242480 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.86 (kernels 0.14, alloc 0.47, bootstrap 0.00, allgathers 0.00, topo 0.07, graphs 0.00, connections 0.02, rest 0.16)
n136-128-154:2242239:2242479 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2242239:2242479 [0] NCCL INFO ncclCommInitRankConfig comm 0xe2505d0 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0x624137be80e2f945 - Init COMPLETE
n136-128-154:2242239:2242479 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.87 (kernels 0.13, alloc 0.48, bootstrap 0.00, allgathers 0.00, topo 0.07, graphs 0.00, connections 0.02, rest 0.16)
n136-128-154:2242240:2242496 [1] NCCL INFO Channel 00/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242240:2242496 [1] NCCL INFO Channel 01/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242240:2242496 [1] NCCL INFO Channel 02/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242240:2242496 [1] NCCL INFO Channel 03/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242240:2242496 [1] NCCL INFO Channel 04/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242239:2242497 [0] NCCL INFO Channel 00/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2242240:2242496 [1] NCCL INFO Channel 05/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242239:2242497 [0] NCCL INFO Channel 01/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2242240:2242496 [1] NCCL INFO Channel 06/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242239:2242497 [0] NCCL INFO Channel 02/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2242240:2242496 [1] NCCL INFO Channel 07/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242239:2242497 [0] NCCL INFO Channel 03/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2242240:2242496 [1] NCCL INFO Channel 08/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242239:2242497 [0] NCCL INFO Channel 04/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2242240:2242496 [1] NCCL INFO Channel 09/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242239:2242497 [0] NCCL INFO Channel 05/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2242240:2242496 [1] NCCL INFO Channel 10/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242239:2242497 [0] NCCL INFO Channel 06/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2242240:2242496 [1] NCCL INFO Channel 11/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242239:2242497 [0] NCCL INFO Channel 07/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2242240:2242496 [1] NCCL INFO Channel 12/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242239:2242497 [0] NCCL INFO Channel 08/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2242240:2242496 [1] NCCL INFO Channel 13/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242239:2242497 [0] NCCL INFO Channel 09/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2242240:2242496 [1] NCCL INFO Channel 14/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242239:2242497 [0] NCCL INFO Channel 10/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2242240:2242496 [1] NCCL INFO Channel 15/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242239:2242497 [0] NCCL INFO Channel 11/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2242240:2242496 [1] NCCL INFO Channel 16/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242239:2242497 [0] NCCL INFO Channel 12/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2242240:2242496 [1] NCCL INFO Channel 17/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242239:2242497 [0] NCCL INFO Channel 13/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2242240:2242496 [1] NCCL INFO Channel 18/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242239:2242497 [0] NCCL INFO Channel 14/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2242240:2242496 [1] NCCL INFO Channel 19/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242239:2242497 [0] NCCL INFO Channel 15/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2242240:2242496 [1] NCCL INFO Channel 20/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242239:2242497 [0] NCCL INFO Channel 16/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2242240:2242496 [1] NCCL INFO Channel 21/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242239:2242497 [0] NCCL INFO Channel 17/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2242240:2242496 [1] NCCL INFO Channel 22/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242239:2242497 [0] NCCL INFO Channel 18/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2242240:2242496 [1] NCCL INFO Channel 23/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242239:2242497 [0] NCCL INFO Channel 19/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2242239:2242497 [0] NCCL INFO Channel 20/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2242239:2242497 [0] NCCL INFO Channel 21/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2242239:2242497 [0] NCCL INFO Channel 22/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2242239:2242497 [0] NCCL INFO Channel 23/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2242240:2242496 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:2242239:2242497 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-09:19:26:34 INFO     [evaluator:559] Running loglikelihood requests
2025-12-09:19:26:34 INFO     [evaluator:559] Running loglikelihood requests
Passed argument batch_size = auto:1. Detecting largest batch size

Running loglikelihood requests:   0%|          | 0/20084 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 57
Determined largest batch size: 57

Running loglikelihood requests:   0%|          | 1/20084 [00:39<222:29:51, 39.88s/it]
Running loglikelihood requests:   0%|          | 58/20084 [00:45<3:11:10,  1.75it/s] 
Running loglikelihood requests:   1%|          | 115/20084 [00:49<1:35:21,  3.49it/s]
Running loglikelihood requests:   1%|          | 172/20084 [00:54<1:04:01,  5.18it/s]
Running loglikelihood requests:   1%|          | 229/20084 [00:58<49:06,  6.74it/s]  
Running loglikelihood requests:   1%|▏         | 286/20084 [01:03<40:49,  8.08it/s]
Running loglikelihood requests:   2%|▏         | 343/20084 [01:08<35:46,  9.20it/s]
Running loglikelihood requests:   2%|▏         | 400/20084 [01:12<32:28, 10.10it/s]
Running loglikelihood requests:   2%|▏         | 457/20084 [01:17<30:16, 10.81it/s]
Running loglikelihood requests:   3%|▎         | 514/20084 [01:21<28:44, 11.35it/s]
Running loglikelihood requests:   3%|▎         | 571/20084 [01:25<27:39, 11.76it/s]
Running loglikelihood requests:   3%|▎         | 628/20084 [01:30<26:48, 12.10it/s]
Running loglikelihood requests:   3%|▎         | 685/20084 [01:34<26:11, 12.34it/s]
Running loglikelihood requests:   4%|▎         | 742/20084 [01:39<25:42, 12.54it/s]
Running loglikelihood requests:   4%|▍         | 799/20084 [01:43<25:20, 12.68it/s]
Running loglikelihood requests:   4%|▍         | 856/20084 [01:47<25:03, 12.79it/s]
Running loglikelihood requests:   5%|▍         | 913/20084 [01:52<24:48, 12.88it/s]
Running loglikelihood requests:   5%|▍         | 970/20084 [01:56<24:36, 12.94it/s]
Running loglikelihood requests:   5%|▌         | 1027/20084 [02:00<24:25, 13.00it/s]
Running loglikelihood requests:   5%|▌         | 1084/20084 [02:05<24:15, 13.05it/s]
Running loglikelihood requests:   6%|▌         | 1141/20084 [02:09<24:06, 13.10it/s]
Running loglikelihood requests:   6%|▌         | 1198/20084 [02:13<23:57, 13.13it/s]
Running loglikelihood requests:   6%|▌         | 1255/20084 [02:18<23:49, 13.17it/s]
Running loglikelihood requests:   7%|▋         | 1312/20084 [02:22<23:42, 13.20it/s]
Running loglikelihood requests:   7%|▋         | 1369/20084 [02:26<23:35, 13.22it/s]
Running loglikelihood requests:   7%|▋         | 1426/20084 [02:31<23:28, 13.25it/s]
Running loglikelihood requests:   7%|▋         | 1483/20084 [02:35<23:20, 13.28it/s]
Running loglikelihood requests:   8%|▊         | 1540/20084 [02:39<23:14, 13.30it/s]
Running loglikelihood requests:   8%|▊         | 1597/20084 [02:43<23:07, 13.32it/s]
Running loglikelihood requests:   8%|▊         | 1654/20084 [02:48<23:01, 13.34it/s]
Running loglikelihood requests:   9%|▊         | 1711/20084 [02:52<22:54, 13.36it/s]
Running loglikelihood requests:   9%|▉         | 1768/20084 [02:56<22:48, 13.39it/s]
Running loglikelihood requests:   9%|▉         | 1825/20084 [03:00<22:41, 13.41it/s]
Running loglikelihood requests:   9%|▉         | 1882/20084 [03:05<22:34, 13.44it/s]
Running loglikelihood requests:  10%|▉         | 1939/20084 [03:09<22:28, 13.46it/s]
Running loglikelihood requests:  10%|▉         | 1996/20084 [03:13<22:21, 13.49it/s]
Running loglikelihood requests:  10%|█         | 2053/20084 [03:17<22:15, 13.50it/s]
Running loglikelihood requests:  11%|█         | 2110/20084 [03:21<22:09, 13.52it/s]
Running loglikelihood requests:  11%|█         | 2167/20084 [03:26<22:02, 13.54it/s]
Running loglikelihood requests:  11%|█         | 2224/20084 [03:30<21:57, 13.56it/s]
Running loglikelihood requests:  11%|█▏        | 2281/20084 [03:34<21:50, 13.58it/s]
Running loglikelihood requests:  12%|█▏        | 2338/20084 [03:38<21:44, 13.60it/s]
Running loglikelihood requests:  12%|█▏        | 2395/20084 [03:42<21:37, 13.63it/s]
Running loglikelihood requests:  12%|█▏        | 2452/20084 [03:46<21:31, 13.65it/s]
Running loglikelihood requests:  12%|█▏        | 2509/20084 [03:51<21:26, 13.66it/s]
Running loglikelihood requests:  13%|█▎        | 2566/20084 [03:55<21:20, 13.68it/s]
Running loglikelihood requests:  13%|█▎        | 2623/20084 [03:59<21:14, 13.70it/s]
Running loglikelihood requests:  13%|█▎        | 2680/20084 [04:03<21:09, 13.71it/s]
Running loglikelihood requests:  14%|█▎        | 2737/20084 [04:07<21:02, 13.74it/s]
Running loglikelihood requests:  14%|█▍        | 2794/20084 [04:11<20:57, 13.75it/s]
Running loglikelihood requests:  14%|█▍        | 2851/20084 [04:15<20:52, 13.76it/s]
Running loglikelihood requests:  14%|█▍        | 2908/20084 [04:20<20:47, 13.77it/s]
Running loglikelihood requests:  15%|█▍        | 2965/20084 [04:24<20:41, 13.79it/s]
Running loglikelihood requests:  15%|█▌        | 3022/20084 [04:28<20:35, 13.81it/s]
Running loglikelihood requests:  15%|█▌        | 3079/20084 [04:32<20:30, 13.82it/s]
Running loglikelihood requests:  16%|█▌        | 3136/20084 [04:36<20:24, 13.84it/s]
Running loglikelihood requests:  16%|█▌        | 3193/20084 [04:40<20:19, 13.85it/s]
Running loglikelihood requests:  16%|█▌        | 3250/20084 [04:44<20:14, 13.86it/s]
Running loglikelihood requests:  16%|█▋        | 3307/20084 [04:48<20:08, 13.88it/s]
Running loglikelihood requests:  17%|█▋        | 3364/20084 [04:52<20:03, 13.90it/s]
Running loglikelihood requests:  17%|█▋        | 3421/20084 [04:57<19:57, 13.91it/s]
Running loglikelihood requests:  17%|█▋        | 3478/20084 [05:01<19:52, 13.93it/s]
Running loglikelihood requests:  18%|█▊        | 3535/20084 [05:05<19:47, 13.93it/s]
Running loglikelihood requests:  18%|█▊        | 3592/20084 [05:09<19:45, 13.92it/s]
Running loglikelihood requests:  18%|█▊        | 3649/20084 [05:13<19:38, 13.95it/s]
Running loglikelihood requests:  18%|█▊        | 3706/20084 [05:17<19:32, 13.97it/s]
Running loglikelihood requests:  19%|█▊        | 3763/20084 [05:21<19:26, 13.99it/s]
Running loglikelihood requests:  19%|█▉        | 3820/20084 [05:25<19:20, 14.01it/s]
Running loglikelihood requests:  19%|█▉        | 3877/20084 [05:29<19:15, 14.03it/s]
Running loglikelihood requests:  20%|█▉        | 3934/20084 [05:33<19:09, 14.05it/s]
Running loglikelihood requests:  20%|█▉        | 3991/20084 [05:37<19:04, 14.07it/s]
Running loglikelihood requests:  20%|██        | 4048/20084 [05:41<18:58, 14.09it/s]
Running loglikelihood requests:  20%|██        | 4105/20084 [05:45<18:52, 14.11it/s]
Running loglikelihood requests:  21%|██        | 4162/20084 [05:49<18:46, 14.13it/s]
Running loglikelihood requests:  21%|██        | 4219/20084 [05:53<18:41, 14.14it/s]
Running loglikelihood requests:  21%|██▏       | 4276/20084 [05:57<18:36, 14.15it/s]
Running loglikelihood requests:  22%|██▏       | 4333/20084 [06:01<18:31, 14.17it/s]
Running loglikelihood requests:  22%|██▏       | 4390/20084 [06:05<18:26, 14.18it/s]
Running loglikelihood requests:  22%|██▏       | 4447/20084 [06:09<18:21, 14.20it/s]
Running loglikelihood requests:  22%|██▏       | 4504/20084 [06:13<18:15, 14.22it/s]
Running loglikelihood requests:  23%|██▎       | 4561/20084 [06:17<18:11, 14.23it/s]
Running loglikelihood requests:  23%|██▎       | 4618/20084 [06:21<18:02, 14.28it/s]
Running loglikelihood requests:  23%|██▎       | 4675/20084 [06:25<17:55, 14.32it/s]
Running loglikelihood requests:  24%|██▎       | 4732/20084 [06:29<17:49, 14.36it/s]
Running loglikelihood requests:  24%|██▍       | 4789/20084 [06:33<17:42, 14.39it/s]
Running loglikelihood requests:  24%|██▍       | 4846/20084 [06:37<17:37, 14.41it/s]
Running loglikelihood requests:  24%|██▍       | 4903/20084 [06:41<17:34, 14.40it/s]
Running loglikelihood requests:  25%|██▍       | 4960/20084 [06:45<17:32, 14.37it/s]
Running loglikelihood requests:  25%|██▍       | 5017/20084 [06:49<17:28, 14.37it/s]
Running loglikelihood requests:  25%|██▌       | 5074/20084 [06:53<17:23, 14.38it/s]
Running loglikelihood requests:  26%|██▌       | 5131/20084 [06:57<17:19, 14.39it/s]
Running loglikelihood requests:  26%|██▌       | 5188/20084 [07:01<17:14, 14.40it/s]
Running loglikelihood requests:  26%|██▌       | 5245/20084 [07:05<17:10, 14.40it/s]
Running loglikelihood requests:  26%|██▋       | 5302/20084 [07:09<17:06, 14.39it/s]
Running loglikelihood requests:  27%|██▋       | 5359/20084 [07:13<17:02, 14.40it/s]
Running loglikelihood requests:  27%|██▋       | 5416/20084 [07:17<16:57, 14.41it/s]
Running loglikelihood requests:  27%|██▋       | 5473/20084 [07:21<16:52, 14.43it/s]
Running loglikelihood requests:  28%|██▊       | 5530/20084 [07:25<16:47, 14.44it/s]
Running loglikelihood requests:  28%|██▊       | 5587/20084 [07:29<16:43, 14.45it/s]
Running loglikelihood requests:  28%|██▊       | 5644/20084 [07:32<16:37, 14.47it/s]
Running loglikelihood requests:  28%|██▊       | 5701/20084 [07:36<16:32, 14.49it/s]
Running loglikelihood requests:  29%|██▊       | 5758/20084 [07:40<16:27, 14.51it/s]
Running loglikelihood requests:  29%|██▉       | 5815/20084 [07:44<16:19, 14.57it/s]
Running loglikelihood requests:  29%|██▉       | 5872/20084 [07:48<16:12, 14.61it/s]
Running loglikelihood requests:  30%|██▉       | 5929/20084 [07:52<16:07, 14.64it/s]
Running loglikelihood requests:  30%|██▉       | 5986/20084 [07:56<16:01, 14.66it/s]
Running loglikelihood requests:  30%|███       | 6043/20084 [08:00<15:58, 14.65it/s]
Running loglikelihood requests:  30%|███       | 6100/20084 [08:04<15:53, 14.66it/s]
Running loglikelihood requests:  31%|███       | 6157/20084 [08:07<15:50, 14.66it/s]
Running loglikelihood requests:  31%|███       | 6214/20084 [08:11<15:46, 14.66it/s]
Running loglikelihood requests:  31%|███       | 6271/20084 [08:15<15:41, 14.67it/s]
Running loglikelihood requests:  32%|███▏      | 6328/20084 [08:19<15:37, 14.68it/s]
Running loglikelihood requests:  32%|███▏      | 6385/20084 [08:23<15:32, 14.68it/s]
Running loglikelihood requests:  32%|███▏      | 6442/20084 [08:27<15:28, 14.70it/s]
Running loglikelihood requests:  32%|███▏      | 6499/20084 [08:31<15:23, 14.71it/s]
Running loglikelihood requests:  33%|███▎      | 6556/20084 [08:35<15:18, 14.73it/s]
Running loglikelihood requests:  33%|███▎      | 6613/20084 [08:38<15:13, 14.74it/s]
Running loglikelihood requests:  33%|███▎      | 6670/20084 [08:42<15:09, 14.75it/s]
Running loglikelihood requests:  33%|███▎      | 6727/20084 [08:46<15:04, 14.76it/s]
Running loglikelihood requests:  34%|███▍      | 6784/20084 [08:50<15:00, 14.76it/s]
Running loglikelihood requests:  34%|███▍      | 6841/20084 [08:54<14:56, 14.78it/s]
Running loglikelihood requests:  34%|███▍      | 6898/20084 [08:58<14:51, 14.78it/s]
Running loglikelihood requests:  35%|███▍      | 6955/20084 [09:02<14:47, 14.79it/s]
Running loglikelihood requests:  35%|███▍      | 7012/20084 [09:05<14:43, 14.80it/s]
Running loglikelihood requests:  35%|███▌      | 7069/20084 [09:09<14:38, 14.81it/s]
Running loglikelihood requests:  35%|███▌      | 7126/20084 [09:13<14:31, 14.86it/s]
Running loglikelihood requests:  36%|███▌      | 7183/20084 [09:17<14:25, 14.91it/s]
Running loglikelihood requests:  36%|███▌      | 7240/20084 [09:21<14:20, 14.92it/s]
Running loglikelihood requests:  36%|███▋      | 7297/20084 [09:25<14:19, 14.87it/s]
Running loglikelihood requests:  37%|███▋      | 7354/20084 [09:28<14:19, 14.81it/s]
Running loglikelihood requests:  37%|███▋      | 7411/20084 [09:32<14:13, 14.84it/s]
Running loglikelihood requests:  37%|███▋      | 7468/20084 [09:36<14:08, 14.87it/s]
Running loglikelihood requests:  37%|███▋      | 7525/20084 [09:40<14:03, 14.89it/s]
Running loglikelihood requests:  38%|███▊      | 7582/20084 [09:44<13:58, 14.91it/s]
Running loglikelihood requests:  38%|███▊      | 7639/20084 [09:47<13:53, 14.93it/s]
Running loglikelihood requests:  38%|███▊      | 7696/20084 [09:51<13:49, 14.94it/s]
Running loglikelihood requests:  39%|███▊      | 7753/20084 [09:55<13:44, 14.96it/s]
Running loglikelihood requests:  39%|███▉      | 7810/20084 [09:59<13:39, 14.97it/s]
Running loglikelihood requests:  39%|███▉      | 7867/20084 [10:03<13:35, 14.98it/s]
Running loglikelihood requests:  39%|███▉      | 7924/20084 [10:06<13:30, 15.00it/s]
Running loglikelihood requests:  40%|███▉      | 7981/20084 [10:10<13:26, 15.01it/s]
Running loglikelihood requests:  40%|████      | 8038/20084 [10:14<13:21, 15.03it/s]
Running loglikelihood requests:  40%|████      | 8095/20084 [10:18<13:17, 15.04it/s]
Running loglikelihood requests:  41%|████      | 8152/20084 [10:22<13:12, 15.05it/s]
Running loglikelihood requests:  41%|████      | 8209/20084 [10:25<13:08, 15.07it/s]
Running loglikelihood requests:  41%|████      | 8266/20084 [10:29<13:03, 15.08it/s]
Running loglikelihood requests:  41%|████▏     | 8323/20084 [10:33<12:59, 15.09it/s]
Running loglikelihood requests:  42%|████▏     | 8380/20084 [10:37<12:55, 15.10it/s]
Running loglikelihood requests:  42%|████▏     | 8437/20084 [10:40<12:50, 15.12it/s]
Running loglikelihood requests:  42%|████▏     | 8494/20084 [10:44<12:46, 15.12it/s]
Running loglikelihood requests:  43%|████▎     | 8551/20084 [10:48<12:39, 15.18it/s]
Running loglikelihood requests:  43%|████▎     | 8608/20084 [10:52<12:33, 15.23it/s]
Running loglikelihood requests:  43%|████▎     | 8665/20084 [10:55<12:28, 15.26it/s]
Running loglikelihood requests:  43%|████▎     | 8722/20084 [10:59<12:25, 15.25it/s]
Running loglikelihood requests:  44%|████▎     | 8779/20084 [11:03<12:21, 15.24it/s]
Running loglikelihood requests:  44%|████▍     | 8836/20084 [11:07<12:18, 15.23it/s]
Running loglikelihood requests:  44%|████▍     | 8893/20084 [11:10<12:14, 15.23it/s]
Running loglikelihood requests:  45%|████▍     | 8950/20084 [11:14<12:11, 15.22it/s]
Running loglikelihood requests:  45%|████▍     | 9007/20084 [11:18<12:07, 15.23it/s]
Running loglikelihood requests:  45%|████▌     | 9064/20084 [11:22<12:03, 15.23it/s]
Running loglikelihood requests:  45%|████▌     | 9121/20084 [11:25<11:57, 15.29it/s]
Running loglikelihood requests:  46%|████▌     | 9178/20084 [11:29<11:51, 15.33it/s]
Running loglikelihood requests:  46%|████▌     | 9235/20084 [11:33<11:47, 15.34it/s]
Running loglikelihood requests:  46%|████▋     | 9292/20084 [11:36<11:43, 15.34it/s]
Running loglikelihood requests:  47%|████▋     | 9349/20084 [11:40<11:39, 15.34it/s]
Running loglikelihood requests:  47%|████▋     | 9406/20084 [11:44<11:35, 15.35it/s]
Running loglikelihood requests:  47%|████▋     | 9463/20084 [11:48<11:32, 15.35it/s]
Running loglikelihood requests:  47%|████▋     | 9520/20084 [11:51<11:27, 15.36it/s]
Running loglikelihood requests:  48%|████▊     | 9577/20084 [11:55<11:23, 15.37it/s]
Running loglikelihood requests:  48%|████▊     | 9634/20084 [11:59<11:19, 15.38it/s]
Running loglikelihood requests:  48%|████▊     | 9691/20084 [12:02<11:13, 15.43it/s]
Running loglikelihood requests:  49%|████▊     | 9748/20084 [12:06<11:08, 15.47it/s]
Running loglikelihood requests:  49%|████▉     | 9805/20084 [12:10<11:02, 15.51it/s]
Running loglikelihood requests:  49%|████▉     | 9862/20084 [12:13<10:58, 15.53it/s]
Running loglikelihood requests:  49%|████▉     | 9919/20084 [12:17<10:53, 15.55it/s]
Running loglikelihood requests:  50%|████▉     | 9976/20084 [12:21<10:50, 15.54it/s]
Running loglikelihood requests:  50%|████▉     | 10033/20084 [12:24<10:47, 15.52it/s]
Running loglikelihood requests:  50%|█████     | 10090/20084 [12:28<10:44, 15.51it/s]
Running loglikelihood requests:  51%|█████     | 10147/20084 [12:32<10:41, 15.50it/s]
Running loglikelihood requests:  51%|█████     | 10204/20084 [12:35<10:37, 15.50it/s]
Running loglikelihood requests:  51%|█████     | 10261/20084 [12:39<10:33, 15.51it/s]
Running loglikelihood requests:  51%|█████▏    | 10318/20084 [12:43<10:29, 15.51it/s]
Running loglikelihood requests:  52%|█████▏    | 10375/20084 [12:46<10:25, 15.52it/s]
Running loglikelihood requests:  52%|█████▏    | 10432/20084 [12:50<10:19, 15.57it/s]
Running loglikelihood requests:  52%|█████▏    | 10489/20084 [12:54<10:14, 15.60it/s]
Running loglikelihood requests:  53%|█████▎    | 10546/20084 [12:57<10:10, 15.62it/s]
Running loglikelihood requests:  53%|█████▎    | 10603/20084 [13:01<10:05, 15.65it/s]
Running loglikelihood requests:  53%|█████▎    | 10660/20084 [13:05<10:01, 15.67it/s]
Running loglikelihood requests:  53%|█████▎    | 10717/20084 [13:08<09:57, 15.68it/s]
Running loglikelihood requests:  54%|█████▎    | 10774/20084 [13:12<09:53, 15.69it/s]
Running loglikelihood requests:  54%|█████▍    | 10831/20084 [13:15<09:50, 15.68it/s]
Running loglikelihood requests:  54%|█████▍    | 10888/20084 [13:19<09:46, 15.67it/s]
Running loglikelihood requests:  54%|█████▍    | 10945/20084 [13:23<09:43, 15.66it/s]
Running loglikelihood requests:  55%|█████▍    | 11002/20084 [13:26<09:39, 15.67it/s]
Running loglikelihood requests:  55%|█████▌    | 11059/20084 [13:30<09:35, 15.67it/s]
Running loglikelihood requests:  55%|█████▌    | 11116/20084 [13:34<09:31, 15.68it/s]
Running loglikelihood requests:  56%|█████▌    | 11173/20084 [13:37<09:28, 15.69it/s]
Running loglikelihood requests:  56%|█████▌    | 11230/20084 [13:41<09:22, 15.75it/s]
Running loglikelihood requests:  56%|█████▌    | 11287/20084 [13:44<09:17, 15.79it/s]
Running loglikelihood requests:  56%|█████▋    | 11344/20084 [13:48<09:12, 15.82it/s]
Running loglikelihood requests:  57%|█████▋    | 11401/20084 [13:52<09:07, 15.85it/s]
Running loglikelihood requests:  57%|█████▋    | 11458/20084 [13:55<09:03, 15.86it/s]
Running loglikelihood requests:  57%|█████▋    | 11515/20084 [13:59<08:59, 15.87it/s]
Running loglikelihood requests:  58%|█████▊    | 11572/20084 [14:02<08:56, 15.86it/s]
Running loglikelihood requests:  58%|█████▊    | 11629/20084 [14:06<08:53, 15.84it/s]
Running loglikelihood requests:  58%|█████▊    | 11686/20084 [14:10<08:50, 15.82it/s]
Running loglikelihood requests:  58%|█████▊    | 11743/20084 [14:13<08:47, 15.83it/s]
Running loglikelihood requests:  59%|█████▉    | 11800/20084 [14:17<08:43, 15.83it/s]
Running loglikelihood requests:  59%|█████▉    | 11857/20084 [14:20<08:39, 15.84it/s]
Running loglikelihood requests:  59%|█████▉    | 11914/20084 [14:24<08:35, 15.84it/s]
Running loglikelihood requests:  60%|█████▉    | 11971/20084 [14:28<08:30, 15.88it/s]
Running loglikelihood requests:  60%|█████▉    | 12028/20084 [14:31<08:26, 15.92it/s]
Running loglikelihood requests:  60%|██████    | 12085/20084 [14:35<08:21, 15.93it/s]
Running loglikelihood requests:  60%|██████    | 12142/20084 [14:38<08:17, 15.96it/s]
Running loglikelihood requests:  61%|██████    | 12199/20084 [14:42<08:13, 15.99it/s]
Running loglikelihood requests:  61%|██████    | 12256/20084 [14:45<08:09, 15.98it/s]
Running loglikelihood requests:  61%|██████▏   | 12313/20084 [14:49<08:06, 15.96it/s]
Running loglikelihood requests:  62%|██████▏   | 12370/20084 [14:52<08:03, 15.96it/s]
Running loglikelihood requests:  62%|██████▏   | 12427/20084 [14:56<07:59, 15.97it/s]
Running loglikelihood requests:  62%|██████▏   | 12484/20084 [15:00<07:55, 15.98it/s]
Running loglikelihood requests:  62%|██████▏   | 12541/20084 [15:03<07:51, 15.99it/s]
Running loglikelihood requests:  63%|██████▎   | 12598/20084 [15:07<07:47, 16.00it/s]
Running loglikelihood requests:  63%|██████▎   | 12655/20084 [15:10<07:44, 16.00it/s]
Running loglikelihood requests:  63%|██████▎   | 12712/20084 [15:14<07:39, 16.05it/s]
Running loglikelihood requests:  64%|██████▎   | 12769/20084 [15:17<07:34, 16.09it/s]
Running loglikelihood requests:  64%|██████▍   | 12826/20084 [15:21<07:29, 16.14it/s]
Running loglikelihood requests:  64%|██████▍   | 12883/20084 [15:24<07:25, 16.16it/s]
Running loglikelihood requests:  64%|██████▍   | 12940/20084 [15:28<07:21, 16.17it/s]
Running loglikelihood requests:  65%|██████▍   | 12997/20084 [15:31<07:18, 16.16it/s]
Running loglikelihood requests:  65%|██████▍   | 13054/20084 [15:35<07:15, 16.15it/s]
Running loglikelihood requests:  65%|██████▌   | 13111/20084 [15:38<07:10, 16.20it/s]
Running loglikelihood requests:  66%|██████▌   | 13168/20084 [15:42<07:06, 16.23it/s]
Running loglikelihood requests:  66%|██████▌   | 13225/20084 [15:45<07:02, 16.25it/s]
Running loglikelihood requests:  66%|██████▌   | 13282/20084 [15:49<06:58, 16.26it/s]
Running loglikelihood requests:  66%|██████▋   | 13339/20084 [15:52<06:53, 16.29it/s]
Running loglikelihood requests:  67%|██████▋   | 13396/20084 [15:56<06:50, 16.31it/s]
Running loglikelihood requests:  67%|██████▋   | 13453/20084 [15:59<06:46, 16.32it/s]
Running loglikelihood requests:  67%|██████▋   | 13510/20084 [16:03<06:42, 16.34it/s]
Running loglikelihood requests:  68%|██████▊   | 13567/20084 [16:06<06:38, 16.35it/s]
Running loglikelihood requests:  68%|██████▊   | 13624/20084 [16:10<06:35, 16.33it/s]
Running loglikelihood requests:  68%|██████▊   | 13681/20084 [16:13<06:32, 16.32it/s]
Running loglikelihood requests:  68%|██████▊   | 13738/20084 [16:17<06:28, 16.31it/s]
Running loglikelihood requests:  69%|██████▊   | 13795/20084 [16:20<06:25, 16.33it/s]
Running loglikelihood requests:  69%|██████▉   | 13852/20084 [16:24<06:21, 16.33it/s]
Running loglikelihood requests:  69%|██████▉   | 13909/20084 [16:27<06:17, 16.34it/s]
Running loglikelihood requests:  70%|██████▉   | 13966/20084 [16:31<06:13, 16.40it/s]
Running loglikelihood requests:  70%|██████▉   | 14023/20084 [16:34<06:08, 16.44it/s]
Running loglikelihood requests:  70%|███████   | 14080/20084 [16:38<06:04, 16.46it/s]
Running loglikelihood requests:  70%|███████   | 14137/20084 [16:41<06:00, 16.49it/s]
Running loglikelihood requests:  71%|███████   | 14194/20084 [16:45<05:56, 16.52it/s]
Running loglikelihood requests:  71%|███████   | 14251/20084 [16:48<05:52, 16.55it/s]
Running loglikelihood requests:  71%|███████   | 14308/20084 [16:51<05:48, 16.58it/s]
Running loglikelihood requests:  72%|███████▏  | 14365/20084 [16:55<05:44, 16.59it/s]
Running loglikelihood requests:  72%|███████▏  | 14422/20084 [16:58<05:40, 16.61it/s]
Running loglikelihood requests:  72%|███████▏  | 14479/20084 [17:02<05:37, 16.61it/s]
Running loglikelihood requests:  72%|███████▏  | 14536/20084 [17:05<05:33, 16.63it/s]
Running loglikelihood requests:  73%|███████▎  | 14593/20084 [17:08<05:29, 16.65it/s]
Running loglikelihood requests:  73%|███████▎  | 14650/20084 [17:12<05:26, 16.66it/s]
Running loglikelihood requests:  73%|███████▎  | 14707/20084 [17:15<05:23, 16.63it/s]
Running loglikelihood requests:  74%|███████▎  | 14764/20084 [17:19<05:20, 16.62it/s]
Running loglikelihood requests:  74%|███████▍  | 14821/20084 [17:22<05:16, 16.62it/s]
Running loglikelihood requests:  74%|███████▍  | 14878/20084 [17:26<05:13, 16.62it/s]
Running loglikelihood requests:  74%|███████▍  | 14935/20084 [17:29<05:09, 16.63it/s]
Running loglikelihood requests:  75%|███████▍  | 14992/20084 [17:32<05:05, 16.65it/s]
Running loglikelihood requests:  75%|███████▍  | 15049/20084 [17:36<05:02, 16.66it/s]
Running loglikelihood requests:  75%|███████▌  | 15106/20084 [17:39<04:57, 16.71it/s]
Running loglikelihood requests:  75%|███████▌  | 15163/20084 [17:43<04:53, 16.76it/s]
Running loglikelihood requests:  76%|███████▌  | 15220/20084 [17:46<04:49, 16.79it/s]
Running loglikelihood requests:  76%|███████▌  | 15277/20084 [17:49<04:45, 16.82it/s]
Running loglikelihood requests:  76%|███████▋  | 15334/20084 [17:53<04:41, 16.85it/s]
Running loglikelihood requests:  77%|███████▋  | 15391/20084 [17:56<04:38, 16.86it/s]
Running loglikelihood requests:  77%|███████▋  | 15448/20084 [18:00<04:34, 16.88it/s]
Running loglikelihood requests:  77%|███████▋  | 15505/20084 [18:03<04:30, 16.90it/s]
Running loglikelihood requests:  77%|███████▋  | 15562/20084 [18:06<04:27, 16.92it/s]
Running loglikelihood requests:  78%|███████▊  | 15619/20084 [18:10<04:23, 16.94it/s]
Running loglikelihood requests:  78%|███████▊  | 15676/20084 [18:13<04:19, 16.97it/s]
Running loglikelihood requests:  78%|███████▊  | 15733/20084 [18:16<04:16, 16.98it/s]
Running loglikelihood requests:  79%|███████▊  | 15790/20084 [18:20<04:12, 17.03it/s]
Running loglikelihood requests:  79%|███████▉  | 15847/20084 [18:23<04:08, 17.05it/s]
Running loglikelihood requests:  79%|███████▉  | 15904/20084 [18:26<04:04, 17.07it/s]
Running loglikelihood requests:  79%|███████▉  | 15961/20084 [18:30<04:01, 17.05it/s]
Running loglikelihood requests:  80%|███████▉  | 16018/20084 [18:33<03:58, 17.04it/s]
Running loglikelihood requests:  80%|████████  | 16075/20084 [18:36<03:55, 17.05it/s]
Running loglikelihood requests:  80%|████████  | 16132/20084 [18:40<03:51, 17.05it/s]
Running loglikelihood requests:  81%|████████  | 16189/20084 [18:43<03:47, 17.10it/s]
Running loglikelihood requests:  81%|████████  | 16246/20084 [18:46<03:43, 17.15it/s]
Running loglikelihood requests:  81%|████████  | 16303/20084 [18:50<03:39, 17.20it/s]
Running loglikelihood requests:  81%|████████▏ | 16360/20084 [18:53<03:35, 17.25it/s]
Running loglikelihood requests:  82%|████████▏ | 16417/20084 [18:56<03:32, 17.27it/s]
Running loglikelihood requests:  82%|████████▏ | 16474/20084 [18:59<03:28, 17.30it/s]
Running loglikelihood requests:  82%|████████▏ | 16531/20084 [19:03<03:25, 17.33it/s]
Running loglikelihood requests:  83%|████████▎ | 16588/20084 [19:06<03:21, 17.34it/s]
Running loglikelihood requests:  83%|████████▎ | 16645/20084 [19:09<03:18, 17.35it/s]
Running loglikelihood requests:  83%|████████▎ | 16702/20084 [19:13<03:14, 17.38it/s]
Running loglikelihood requests:  83%|████████▎ | 16759/20084 [19:16<03:11, 17.41it/s]
Running loglikelihood requests:  84%|████████▎ | 16816/20084 [19:19<03:07, 17.43it/s]
Running loglikelihood requests:  84%|████████▍ | 16873/20084 [19:22<03:04, 17.43it/s]
Running loglikelihood requests:  84%|████████▍ | 16930/20084 [19:26<03:01, 17.42it/s]
Running loglikelihood requests:  85%|████████▍ | 16987/20084 [19:29<02:57, 17.41it/s]
Running loglikelihood requests:  85%|████████▍ | 17044/20084 [19:32<02:54, 17.42it/s]
Running loglikelihood requests:  85%|████████▌ | 17101/20084 [19:35<02:50, 17.47it/s]
Running loglikelihood requests:  85%|████████▌ | 17158/20084 [19:39<02:46, 17.53it/s]
Running loglikelihood requests:  86%|████████▌ | 17215/20084 [19:42<02:43, 17.57it/s]
Running loglikelihood requests:  86%|████████▌ | 17272/20084 [19:45<02:39, 17.61it/s]
Running loglikelihood requests:  86%|████████▋ | 17329/20084 [19:48<02:36, 17.63it/s]
Running loglikelihood requests:  87%|████████▋ | 17386/20084 [19:52<02:32, 17.67it/s]
Running loglikelihood requests:  87%|████████▋ | 17443/20084 [19:55<02:29, 17.71it/s]
Running loglikelihood requests:  87%|████████▋ | 17500/20084 [19:58<02:25, 17.74it/s]
Running loglikelihood requests:  87%|████████▋ | 17557/20084 [20:01<02:22, 17.76it/s]
Running loglikelihood requests:  88%|████████▊ | 17614/20084 [20:04<02:18, 17.79it/s]
Running loglikelihood requests:  88%|████████▊ | 17671/20084 [20:07<02:15, 17.81it/s]
Running loglikelihood requests:  88%|████████▊ | 17728/20084 [20:11<02:12, 17.84it/s]
Running loglikelihood requests:  89%|████████▊ | 17785/20084 [20:14<02:08, 17.89it/s]
Running loglikelihood requests:  89%|████████▉ | 17842/20084 [20:17<02:05, 17.93it/s]
Running loglikelihood requests:  89%|████████▉ | 17899/20084 [20:20<02:01, 17.97it/s]
Running loglikelihood requests:  89%|████████▉ | 17956/20084 [20:23<01:58, 18.01it/s]
Running loglikelihood requests:  90%|████████▉ | 18013/20084 [20:26<01:54, 18.03it/s]
Running loglikelihood requests:  90%|████████▉ | 18070/20084 [20:30<01:51, 18.03it/s]
Running loglikelihood requests:  90%|█████████ | 18127/20084 [20:33<01:48, 18.05it/s]
Running loglikelihood requests:  91%|█████████ | 18184/20084 [20:36<01:45, 18.09it/s]
Running loglikelihood requests:  91%|█████████ | 18241/20084 [20:39<01:41, 18.12it/s]
Running loglikelihood requests:  91%|█████████ | 18298/20084 [20:42<01:38, 18.13it/s]
Running loglikelihood requests:  91%|█████████▏| 18355/20084 [20:45<01:35, 18.18it/s]
Running loglikelihood requests:  92%|█████████▏| 18412/20084 [20:48<01:31, 18.25it/s]
Running loglikelihood requests:  92%|█████████▏| 18469/20084 [20:51<01:28, 18.28it/s]
Running loglikelihood requests:  92%|█████████▏| 18526/20084 [20:55<01:24, 18.35it/s]
Running loglikelihood requests:  93%|█████████▎| 18583/20084 [20:58<01:21, 18.38it/s]
Running loglikelihood requests:  93%|█████████▎| 18640/20084 [21:01<01:18, 18.40it/s]
Running loglikelihood requests:  93%|█████████▎| 18697/20084 [21:04<01:15, 18.47it/s]
Running loglikelihood requests:  93%|█████████▎| 18754/20084 [21:07<01:11, 18.51it/s]
Running loglikelihood requests:  94%|█████████▎| 18811/20084 [21:10<01:08, 18.58it/s]
Running loglikelihood requests:  94%|█████████▍| 18868/20084 [21:13<01:05, 18.66it/s]
Running loglikelihood requests:  94%|█████████▍| 18925/20084 [21:16<01:01, 18.74it/s]
Running loglikelihood requests:  95%|█████████▍| 18982/20084 [21:19<00:58, 18.79it/s]
Running loglikelihood requests:  95%|█████████▍| 19039/20084 [21:22<00:55, 18.84it/s]
Running loglikelihood requests:  95%|█████████▌| 19096/20084 [21:25<00:52, 18.90it/s]
Running loglikelihood requests:  95%|█████████▌| 19153/20084 [21:28<00:49, 18.99it/s]
Running loglikelihood requests:  96%|█████████▌| 19210/20084 [21:31<00:45, 19.06it/s]
Running loglikelihood requests:  96%|█████████▌| 19267/20084 [21:34<00:42, 19.17it/s]
Running loglikelihood requests:  96%|█████████▌| 19324/20084 [21:37<00:39, 19.23it/s]
Running loglikelihood requests:  96%|█████████▋| 19381/20084 [21:40<00:36, 19.29it/s]
Running loglikelihood requests:  97%|█████████▋| 19438/20084 [21:43<00:33, 19.35it/s]
Running loglikelihood requests:  97%|█████████▋| 19495/20084 [21:46<00:30, 19.46it/s]
Running loglikelihood requests:  97%|█████████▋| 19552/20084 [21:48<00:27, 19.57it/s]
Running loglikelihood requests:  98%|█████████▊| 19609/20084 [21:51<00:24, 19.67it/s]
Running loglikelihood requests:  98%|█████████▊| 19666/20084 [21:54<00:21, 19.77it/s]
Running loglikelihood requests:  98%|█████████▊| 19723/20084 [21:57<00:18, 19.94it/s]
Running loglikelihood requests:  98%|█████████▊| 19780/20084 [22:00<00:15, 20.12it/s]
Running loglikelihood requests:  99%|█████████▉| 19837/20084 [22:02<00:12, 20.36it/s]
Running loglikelihood requests:  99%|█████████▉| 19894/20084 [22:05<00:09, 20.58it/s]
Running loglikelihood requests:  99%|█████████▉| 19951/20084 [22:08<00:06, 20.83it/s]
Running loglikelihood requests: 100%|█████████▉| 20008/20084 [22:10<00:03, 21.22it/s]
Running loglikelihood requests: 100%|█████████▉| 20065/20084 [22:11<00:00, 26.70it/s]
Running loglikelihood requests: 100%|██████████| 20084/20084 [22:11<00:00, 15.08it/s]
n136-128-154:2242240:2245500 [1] NCCL INFO Channel 00/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242240:2245500 [1] NCCL INFO Channel 01/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242240:2245500 [1] NCCL INFO Channel 02/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242240:2245500 [1] NCCL INFO Channel 03/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242240:2245500 [1] NCCL INFO Channel 04/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242240:2245500 [1] NCCL INFO Channel 05/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242240:2245500 [1] NCCL INFO Channel 06/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242240:2245500 [1] NCCL INFO Channel 07/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242240:2245500 [1] NCCL INFO Channel 08/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242240:2245500 [1] NCCL INFO Channel 09/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242240:2245500 [1] NCCL INFO Channel 10/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242240:2245500 [1] NCCL INFO Channel 11/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242240:2245500 [1] NCCL INFO Channel 12/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242240:2245500 [1] NCCL INFO Channel 13/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242240:2245500 [1] NCCL INFO Channel 14/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242240:2245500 [1] NCCL INFO Channel 15/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242240:2245500 [1] NCCL INFO Channel 16/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242240:2245500 [1] NCCL INFO Channel 17/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242240:2245500 [1] NCCL INFO Channel 18/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242240:2245500 [1] NCCL INFO Channel 19/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242240:2245500 [1] NCCL INFO Channel 20/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242240:2245500 [1] NCCL INFO Channel 21/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242240:2245500 [1] NCCL INFO Channel 22/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242240:2245500 [1] NCCL INFO Channel 23/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242240:2245500 [1] NCCL INFO Channel 24/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242240:2245500 [1] NCCL INFO Channel 25/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242240:2245500 [1] NCCL INFO Channel 26/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242240:2245500 [1] NCCL INFO Channel 27/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242240:2245500 [1] NCCL INFO Channel 28/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242240:2245500 [1] NCCL INFO Channel 29/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242240:2245500 [1] NCCL INFO Channel 30/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2242240:2245500 [1] NCCL INFO Channel 31/1 : 1[7] -> 0[6] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:2242240:2245537 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2242240:2245537 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2242240:2245537 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2242240:2245537 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2242240:2245537 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2242240:2245537 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2242240:2242494 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-09:19:50:17 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 10, batch_size: auto (57)
|  Tasks  |Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|---------|------:|------|-----:|--------|---|-----:|---|-----:|
|hellaswag|      1|none  |    10|acc     |↑  |0.2590|±  |0.0044|
|         |       |none  |    10|acc_norm|↑  |0.2554|±  |0.0044|

[rank0]:[W1209 19:50:18.642585790 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:2242239:2245602 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2242239:2245602 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2242239:2245602 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2242239:2242492 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2242240:2242494 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2242239:2245602 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2242239:2245602 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2242239:2245602 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2242240:2245537 [1] NCCL INFO comm 0x47be4560 rank 1 nranks 2 cudaDev 1 busId c9000 - Abort COMPLETE
n136-128-154:2242239:2245602 [0] NCCL INFO comm 0xe2505d0 rank 0 nranks 2 cudaDev 0 busId c5000 - Abort COMPLETE
任务 hellaswag 评估完成！
Execution time: 2673.92 seconds
Namespace(save_dir='/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg', self_attn_layer_to_quant='29 23 24 30 18 28 26 20 16 27 25 17 19 21', mlp_layer_to_quant='26 20 22', model_id='/mnt/bn/life-mllm/users/cxr/quantization/models/meta-llama/Llama-3.1-8B', cuda_id=6)
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.00s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.02s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.02it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.48it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.26it/s]
LlamaForCausalLM(
  (model): LlamaModel(
    (embed_tokens): Embedding(128256, 4096)
    (layers): ModuleList(
      (0-31): 32 x LlamaDecoderLayer(
        (self_attn): LlamaAttention(
          (q_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (k_proj): Linear(in_features=4096, out_features=1024, bias=False)
          (v_proj): Linear(in_features=4096, out_features=1024, bias=False)
          (o_proj): Linear(in_features=4096, out_features=4096, bias=False)
        )
        (mlp): LlamaMLP(
          (gate_proj): Linear(in_features=4096, out_features=14336, bias=False)
          (up_proj): Linear(in_features=4096, out_features=14336, bias=False)
          (down_proj): Linear(in_features=14336, out_features=4096, bias=False)
          (act_fn): SiLUActivation()
        )
        (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
        (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
      )
    )
    (norm): LlamaRMSNorm((4096,), eps=1e-05)
    (rotary_emb): LlamaRotaryEmbedding()
  )
  (lm_head): Linear(in_features=4096, out_features=128256, bias=False)
)
Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.
Once upon a time, there was a young man named Peter. He was the son of an English schoolmaster and his wife, and he was growing up in London. Peter had always been a good student, and he was very interested in science and mathematics. He was particularly interested in physics, and he loved to read about the latest discoveries in the field.
One day, when Peter was 14 years old, he went to a local library to check out some books on physics. While he was there, he saw
quanting ... 
layers.0.self_attn 4 bit quant 
layers.0.mlp 4 bit quant 
layers.1.self_attn 4 bit quant 
layers.1.mlp 4 bit quant 
layers.2.self_attn 4 bit quant 
layers.2.mlp 4 bit quant 
layers.3.self_attn 4 bit quant 
layers.3.mlp 4 bit quant 
layers.4.self_attn 4 bit quant 
layers.4.mlp 4 bit quant 
layers.5.self_attn 4 bit quant 
layers.5.mlp 4 bit quant 
layers.6.self_attn 4 bit quant 
layers.6.mlp 4 bit quant 
layers.7.self_attn 4 bit quant 
layers.7.mlp 4 bit quant 
layers.8.self_attn 4 bit quant 
layers.8.mlp 4 bit quant 
layers.9.self_attn 4 bit quant 
layers.9.mlp 4 bit quant 
layers.10.self_attn 4 bit quant 
layers.10.mlp 4 bit quant 
layers.11.self_attn 4 bit quant 
layers.11.mlp 4 bit quant 
layers.12.self_attn 4 bit quant 
layers.12.mlp 4 bit quant 
layers.13.self_attn 4 bit quant 
layers.13.mlp 4 bit quant 
layers.14.self_attn 4 bit quant 
layers.14.mlp 4 bit quant 
layers.15.self_attn 4 bit quant 
layers.15.mlp 4 bit quant 
layers.16.self_attn 2 bit quant 
layers.16.mlp 4 bit quant 
layers.17.self_attn 2 bit quant 
layers.17.mlp 4 bit quant 
layers.18.self_attn 2 bit quant 
layers.18.mlp 4 bit quant 
layers.19.self_attn 2 bit quant 
layers.19.mlp 4 bit quant 
layers.20.self_attn 2 bit quant 
layers.20.mlp 2 bit quant 
layers.21.self_attn 2 bit quant 
layers.21.mlp 4 bit quant 
layers.22.self_attn 4 bit quant 
layers.22.mlp 2 bit quant 
layers.23.self_attn 2 bit quant 
layers.23.mlp 4 bit quant 
layers.24.self_attn 2 bit quant 
layers.24.mlp 4 bit quant 
layers.25.self_attn 2 bit quant 
layers.25.mlp 4 bit quant 
layers.26.self_attn 2 bit quant 
layers.26.mlp 2 bit quant 
layers.27.self_attn 2 bit quant 
layers.27.mlp 4 bit quant 
layers.28.self_attn 2 bit quant 
layers.28.mlp 4 bit quant 
layers.29.self_attn 2 bit quant 
layers.29.mlp 4 bit quant 
layers.30.self_attn 2 bit quant 
layers.30.mlp 4 bit quant 
layers.31.self_attn 4 bit quant 
layers.31.mlp 4 bit quant 
quanted
LlamaForCausalLM(
  (model): LlamaModel(
    (embed_tokens): Embedding(128256, 4096)
    (layers): ModuleList(
      (0-31): 32 x LlamaDecoderLayer(
        (self_attn): LlamaAttention(
          (q_proj): W8A16Linear(4096, 4096, bias=False, weight_quant=per_channel)
          (k_proj): W8A16Linear(4096, 1024, bias=False, weight_quant=per_channel)
          (v_proj): W8A16Linear(4096, 1024, bias=False, weight_quant=per_channel)
          (o_proj): W8A16Linear(4096, 4096, bias=False, weight_quant=per_channel)
        )
        (mlp): LlamaMLP(
          (gate_proj): W8A16Linear(4096, 14336, bias=False, weight_quant=per_channel)
          (up_proj): W8A16Linear(4096, 14336, bias=False, weight_quant=per_channel)
          (down_proj): W8A16Linear(14336, 4096, bias=False, weight_quant=per_channel)
          (act_fn): SiLUActivation()
        )
        (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
        (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
      )
    )
    (norm): LlamaRMSNorm((4096,), eps=1e-05)
    (rotary_emb): LlamaRotaryEmbedding()
  )
  (lm_head): Linear(in_features=4096, out_features=128256, bias=False)
)
Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.
Once upon a time there was a guy named Gary Lundling, but the high school c
What's the day being held in a town in a flat land
In the 14.5m high m's. 100m game
What's the day in a, b, c...
Once upon a half race in half a, a
.5x14.5m at what a half race in half
.5x14.5m at what you do with half of, in half


==================================================
开始评估：任务=winogrande | 少样本数=5 | 模型=Llama-3.1-8B-quantization-fg
输出路径：results2/Llama-3.1-8B-quantization-fg/fg6/winogrande.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-09:19:55:52 INFO     [__main__:440] Selected Tasks: ['winogrande']
2025-12-09:19:55:52 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:19:55:52 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-09:19:55:52 INFO     [__main__:440] Selected Tasks: ['winogrande']
2025-12-09:19:55:52 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:19:55:52 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-09:19:55:53 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:19:55:53 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:19:55:53 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.22it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.09s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.17it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.05s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.21it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.70it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.47it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.01s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.40it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.20it/s]
2025-12-09:19:56:15 WARNING  [evaluator:309] Overwriting default num_fewshot of winogrande from None to 5
2025-12-09:19:56:15 INFO     [api.task:434] Building contexts for winogrande on rank 1...

  0%|          | 0/633 [00:00<?, ?it/s]
100%|██████████| 633/633 [00:00<00:00, 28074.68it/s]
2025-12-09:19:56:27 WARNING  [evaluator:309] Overwriting default num_fewshot of winogrande from None to 5
2025-12-09:19:56:27 INFO     [api.task:434] Building contexts for winogrande on rank 0...

  0%|          | 0/634 [00:00<?, ?it/s]
100%|██████████| 634/634 [00:00<00:00, 23629.67it/s]
n136-128-154:2246512:2246512 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2246512:2246512 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2246512:2246512 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:2246512:2246512 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2246512:2246512 [0] NCCL INFO Comm config Blocking set to 1
n136-128-154:2246513:2246513 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:2246513:2246513 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2246513:2246513 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2246513:2246513 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2246513:2246513 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:2246513:2246883 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2246513:2246883 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2246512:2246882 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2246512:2246882 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2246513:2246883 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2246513:2246883 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2246513:2246883 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2246512:2246882 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2246512:2246882 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2246512:2246882 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2246512:2246882 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2246512:2246882 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2246512:2246882 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:2246513:2246883 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2246513:2246883 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2246513:2246883 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:2246512:2246882 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2246512:2246882 [0] NCCL INFO Using network IB
n136-128-154:2246513:2246883 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2246513:2246883 [1] NCCL INFO Using network IB
n136-128-154:2246512:2246882 [0] NCCL INFO ncclCommInitRankConfig comm 0x14c90df0 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0xa8d5bc1f63708c6 - Init START
n136-128-154:2246513:2246883 [1] NCCL INFO ncclCommInitRankConfig comm 0x16758670 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0xa8d5bc1f63708c6 - Init START
n136-128-154:2246512:2246882 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2246513:2246883 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2246512:2246882 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2246512:2246882 [0] NCCL INFO Retrieving state for IB
n136-128-154:2246512:2246882 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:2246512:2246882 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2246513:2246883 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2246513:2246883 [1] NCCL INFO Retrieving state for IB
n136-128-154:2246513:2246883 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:2246512:2246882 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2246513:2246883 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2246512:2246882 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2246513:2246883 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2246512:2246882 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2246513:2246883 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2246513:2246883 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2246513:2246883 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2246513:2246883 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2246512:2246882 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2246512:2246882 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2246513:2246883 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2246513:2246883 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2246513:2246883 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2246513:2246883 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2246513:2246883 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2246513:2246883 [1] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2246513:2246883 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2246513:2246883 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2246513:2246883 [1] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2246513:2246883 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2246513:2246883 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2246513:2246883 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2246513:2246883 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2246513:2246883 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2246513:2246883 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2246513:2246883 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2246513:2246883 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2246513:2246883 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2246512:2246882 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2246513:2246883 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2246512:2246882 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2246513:2246883 [1] NCCL INFO ==========================================
n136-128-154:2246512:2246882 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2246512:2246882 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2246512:2246882 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2246513:2246883 [1] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2246512:2246882 [0] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2246512:2246882 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2246513:2246883 [1] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2246512:2246882 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2246512:2246882 [0] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2246512:2246882 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2246512:2246882 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2246513:2246883 [1] NCCL INFO Setting affinity for GPU 7 to 33-62,97-126
n136-128-154:2246512:2246882 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2246512:2246882 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2246512:2246882 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2246512:2246882 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2246512:2246882 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2246512:2246882 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2246512:2246882 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2246512:2246882 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2246512:2246882 [0] NCCL INFO ==========================================
n136-128-154:2246512:2246882 [0] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2246512:2246882 [0] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2246512:2246882 [0] NCCL INFO Setting affinity for GPU 6 to 33-62,97-126
n136-128-154:2246513:2246883 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2246513:2246883 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2246513:2246883 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2246513:2246883 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2246513:2246883 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2246513:2246883 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2246513:2246883 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2246513:2246883 [1] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2246513:2246883 [1] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2246513:2246883 [1] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2246513:2246883 [1] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2246513:2246883 [1] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2246513:2246883 [1] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2246513:2246883 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2246513:2246883 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2246513:2246883 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2246513:2246883 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2246513:2246883 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2246513:2246883 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2246513:2246883 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2246513:2246883 [1] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2246513:2246883 [1] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2246513:2246883 [1] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2246513:2246883 [1] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2246513:2246883 [1] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2246513:2246883 [1] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2246512:2246882 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2246512:2246882 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2246512:2246882 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2246512:2246882 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2246512:2246882 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2246512:2246882 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2246512:2246882 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2246512:2246882 [0] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2246512:2246882 [0] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2246512:2246882 [0] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2246512:2246882 [0] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2246512:2246882 [0] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2246512:2246882 [0] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2246512:2246882 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2246512:2246882 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2246512:2246882 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2246512:2246882 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2246512:2246882 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2246512:2246882 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2246512:2246882 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2246512:2246882 [0] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2246512:2246882 [0] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2246512:2246882 [0] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2246512:2246882 [0] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2246512:2246882 [0] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2246512:2246882 [0] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2246513:2246883 [1] NCCL INFO comm 0x16758670 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:2246512:2246882 [0] NCCL INFO comm 0x14c90df0 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:2246513:2246883 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:2246513:2246883 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:2246512:2246882 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:2246513:2246883 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:2246512:2246882 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:2246513:2246883 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:2246512:2246882 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:2246513:2246883 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:2246512:2246882 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:2246513:2246883 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:2246512:2246882 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:2246513:2246883 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:2246512:2246882 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:2246513:2246883 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:2246512:2246882 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:2246513:2246883 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:2246512:2246882 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:2246513:2246883 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:2246512:2246882 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:2246513:2246883 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:2246512:2246882 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:2246513:2246883 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:2246512:2246882 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:2246513:2246883 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:2246512:2246882 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:2246513:2246883 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:2246512:2246882 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:2246513:2246883 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:2246512:2246882 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:2246513:2246883 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:2246512:2246882 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:2246513:2246883 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:2246512:2246882 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:2246513:2246883 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:2246512:2246882 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:2246513:2246883 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:2246512:2246882 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:2246513:2246883 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:2246512:2246882 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:2246513:2246883 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:2246512:2246882 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:2246513:2246883 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:2246512:2246882 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:2246513:2246883 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:2246512:2246882 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:2246513:2246883 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:2246512:2246882 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:2246512:2246882 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:2246512:2246882 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:2246512:2246882 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:2246513:2246883 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:2246512:2246882 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:2246513:2246883 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:2246512:2246882 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:2246513:2246883 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:2246512:2246882 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:2246513:2246883 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:2246512:2246882 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:2246513:2246883 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:2246512:2246882 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:2246513:2246883 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:2246512:2246882 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:2246513:2246883 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:2246512:2246882 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:2246513:2246883 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:2246512:2246882 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:2246513:2246883 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:2246512:2246882 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:2246513:2246883 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:2246512:2246882 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:2246513:2246883 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:2246512:2246882 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:2246513:2246883 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:2246512:2246882 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:2246513:2246883 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:2246512:2246882 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:2246513:2246883 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:2246512:2246882 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:2246513:2246883 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:2246512:2246882 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:2246513:2246883 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:2246512:2246882 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:2246513:2246883 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:2246512:2246882 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:2246513:2246883 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:2246512:2246882 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:2246513:2246883 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:2246512:2246882 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:2246513:2246883 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:2246512:2246882 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:2246513:2246883 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:2246512:2246882 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:2246513:2246883 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:2246512:2246882 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:2246513:2246883 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:2246512:2246882 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:2246513:2246883 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:2246512:2246882 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:2246513:2246883 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:2246512:2246882 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:2246512:2246882 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:2246513:2246883 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2246512:2246882 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:2246512:2246882 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:2246512:2246882 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:2246512:2246882 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:2246512:2246882 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:2246512:2246882 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:2246512:2246882 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:2246512:2246882 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:2246512:2246882 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:2246512:2246882 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:2246512:2246882 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:2246512:2246882 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:2246512:2246882 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:2246512:2246882 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:2246512:2246882 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:2246512:2246882 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:2246512:2246882 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:2246512:2246882 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:2246512:2246882 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:2246512:2246882 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:2246512:2246882 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:2246512:2246882 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2246513:2246883 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2246513:2246894 [1] NCCL INFO [Proxy Service] Device 1 CPU core 109
n136-128-154:2246513:2246895 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 110
n136-128-154:2246512:2246882 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2246512:2246882 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:2246512:2246896 [0] NCCL INFO [Proxy Service] Device 0 CPU core 99
n136-128-154:2246512:2246897 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 36
n136-128-154:2246513:2246883 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2246513:2246883 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2246512:2246882 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2246512:2246882 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2246512:2246882 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:2246513:2246883 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2246513:2246883 [1] NCCL INFO ncclCommInitRankConfig comm 0x16758670 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0xa8d5bc1f63708c6 - Init COMPLETE
n136-128-154:2246513:2246883 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.41 (kernels 0.12, alloc 0.18, bootstrap 0.00, allgathers 0.00, topo 0.07, graphs 0.00, connections 0.01, rest 0.03)
n136-128-154:2246512:2246882 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2246512:2246882 [0] NCCL INFO ncclCommInitRankConfig comm 0x14c90df0 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0xa8d5bc1f63708c6 - Init COMPLETE
n136-128-154:2246512:2246882 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.41 (kernels 0.13, alloc 0.18, bootstrap 0.00, allgathers 0.00, topo 0.07, graphs 0.00, connections 0.01, rest 0.03)
n136-128-154:2246513:2246898 [1] NCCL INFO Channel 00/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2246898 [1] NCCL INFO Channel 01/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2246898 [1] NCCL INFO Channel 02/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2246898 [1] NCCL INFO Channel 03/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2246898 [1] NCCL INFO Channel 04/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2246898 [1] NCCL INFO Channel 05/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2246898 [1] NCCL INFO Channel 06/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2246898 [1] NCCL INFO Channel 07/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2246898 [1] NCCL INFO Channel 08/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2246898 [1] NCCL INFO Channel 09/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2246898 [1] NCCL INFO Channel 10/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2246898 [1] NCCL INFO Channel 11/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2246898 [1] NCCL INFO Channel 12/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2246898 [1] NCCL INFO Channel 13/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2246898 [1] NCCL INFO Channel 14/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2246898 [1] NCCL INFO Channel 15/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2246898 [1] NCCL INFO Channel 16/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2246898 [1] NCCL INFO Channel 17/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2246898 [1] NCCL INFO Channel 18/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2246898 [1] NCCL INFO Channel 19/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2246898 [1] NCCL INFO Channel 20/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246512:2246899 [0] NCCL INFO Channel 00/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2246513:2246898 [1] NCCL INFO Channel 21/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246512:2246899 [0] NCCL INFO Channel 01/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2246513:2246898 [1] NCCL INFO Channel 22/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246512:2246899 [0] NCCL INFO Channel 02/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2246513:2246898 [1] NCCL INFO Channel 23/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246512:2246899 [0] NCCL INFO Channel 03/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2246512:2246899 [0] NCCL INFO Channel 04/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2246512:2246899 [0] NCCL INFO Channel 05/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2246512:2246899 [0] NCCL INFO Channel 06/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2246512:2246899 [0] NCCL INFO Channel 07/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2246512:2246899 [0] NCCL INFO Channel 08/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2246512:2246899 [0] NCCL INFO Channel 09/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2246512:2246899 [0] NCCL INFO Channel 10/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2246512:2246899 [0] NCCL INFO Channel 11/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2246512:2246899 [0] NCCL INFO Channel 12/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2246512:2246899 [0] NCCL INFO Channel 13/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2246512:2246899 [0] NCCL INFO Channel 14/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2246512:2246899 [0] NCCL INFO Channel 15/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2246512:2246899 [0] NCCL INFO Channel 16/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2246512:2246899 [0] NCCL INFO Channel 17/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2246512:2246899 [0] NCCL INFO Channel 18/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2246512:2246899 [0] NCCL INFO Channel 19/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2246512:2246899 [0] NCCL INFO Channel 20/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2246512:2246899 [0] NCCL INFO Channel 21/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2246512:2246899 [0] NCCL INFO Channel 22/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2246512:2246899 [0] NCCL INFO Channel 23/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2246513:2246898 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:2246512:2246899 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-09:19:56:28 INFO     [evaluator:559] Running loglikelihood requests
2025-12-09:19:56:28 INFO     [evaluator:559] Running loglikelihood requests
Passed argument batch_size = auto:1. Detecting largest batch size

Running loglikelihood requests:   0%|          | 0/1268 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 64
Determined largest batch size: 64

Running loglikelihood requests:   0%|          | 1/1268 [00:06<2:09:00,  6.11s/it]
Running loglikelihood requests:   5%|▌         | 65/1268 [00:07<01:37, 12.28it/s] 
Running loglikelihood requests:  10%|█         | 129/1268 [00:08<00:48, 23.41it/s]
Running loglikelihood requests:  15%|█▌        | 193/1268 [00:09<00:32, 33.17it/s]
Running loglikelihood requests:  20%|██        | 257/1268 [00:10<00:24, 41.28it/s]
Running loglikelihood requests:  25%|██▌       | 321/1268 [00:10<00:19, 48.10it/s]
Running loglikelihood requests:  30%|███       | 385/1268 [00:11<00:16, 53.48it/s]
Running loglikelihood requests:  35%|███▌      | 449/1268 [00:12<00:14, 57.60it/s]
Running loglikelihood requests:  40%|████      | 513/1268 [00:13<00:12, 60.82it/s]
Running loglikelihood requests:  46%|████▌     | 577/1268 [00:14<00:10, 63.12it/s]
Running loglikelihood requests:  51%|█████     | 641/1268 [00:15<00:09, 65.17it/s]
Running loglikelihood requests:  56%|█████▌    | 705/1268 [00:16<00:08, 66.70it/s]
Running loglikelihood requests:  61%|██████    | 769/1268 [00:17<00:07, 67.87it/s]
Running loglikelihood requests:  66%|██████▌   | 833/1268 [00:18<00:06, 68.75it/s]
Running loglikelihood requests:  71%|███████   | 897/1268 [00:19<00:05, 69.47it/s]
Running loglikelihood requests:  76%|███████▌  | 961/1268 [00:20<00:04, 70.17it/s]
Running loglikelihood requests:  81%|████████  | 1025/1268 [00:20<00:03, 70.63it/s]
Running loglikelihood requests:  86%|████████▌ | 1089/1268 [00:21<00:02, 71.42it/s]
Running loglikelihood requests:  91%|█████████ | 1153/1268 [00:22<00:01, 72.08it/s]
Running loglikelihood requests:  96%|█████████▌| 1217/1268 [00:23<00:00, 77.12it/s]
Running loglikelihood requests: 100%|██████████| 1268/1268 [00:23<00:00, 54.09it/s]
n136-128-154:2246513:2247195 [1] NCCL INFO Channel 00/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2247195 [1] NCCL INFO Channel 01/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2247195 [1] NCCL INFO Channel 02/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2247195 [1] NCCL INFO Channel 03/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2247195 [1] NCCL INFO Channel 04/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2247195 [1] NCCL INFO Channel 05/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2247195 [1] NCCL INFO Channel 06/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2247195 [1] NCCL INFO Channel 07/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2247195 [1] NCCL INFO Channel 08/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2247195 [1] NCCL INFO Channel 09/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2247195 [1] NCCL INFO Channel 10/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2247195 [1] NCCL INFO Channel 11/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2247195 [1] NCCL INFO Channel 12/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2247195 [1] NCCL INFO Channel 13/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2247195 [1] NCCL INFO Channel 14/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2247195 [1] NCCL INFO Channel 15/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2247195 [1] NCCL INFO Channel 16/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2247195 [1] NCCL INFO Channel 17/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2247195 [1] NCCL INFO Channel 18/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2247195 [1] NCCL INFO Channel 19/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2247195 [1] NCCL INFO Channel 20/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2247195 [1] NCCL INFO Channel 21/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2247195 [1] NCCL INFO Channel 22/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2247195 [1] NCCL INFO Channel 23/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2247195 [1] NCCL INFO Channel 24/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2247195 [1] NCCL INFO Channel 25/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2247195 [1] NCCL INFO Channel 26/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2247195 [1] NCCL INFO Channel 27/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2247195 [1] NCCL INFO Channel 28/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2247195 [1] NCCL INFO Channel 29/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2247195 [1] NCCL INFO Channel 30/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2246513:2247195 [1] NCCL INFO Channel 31/1 : 1[7] -> 0[6] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:2246513:2247200 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2246513:2247200 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2246513:2247200 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2246513:2247200 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2246513:2247200 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2246513:2247200 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2246513:2246894 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-09:19:57:00 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: auto (64)
|  Tasks   |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|----------|------:|------|-----:|------|---|-----:|---|-----:|
|winogrande|      1|none  |     5|acc   |↑  |0.6693|±  |0.0132|

[rank0]:[W1209 19:57:00.913929289 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:2246512:2247311 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2246512:2247311 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2246512:2247311 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2246512:2246896 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2246512:2247311 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2246512:2247311 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2246512:2247311 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2246513:2246894 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2246513:2247200 [1] NCCL INFO comm 0x16758670 rank 1 nranks 2 cudaDev 1 busId c9000 - Abort COMPLETE
n136-128-154:2246512:2247311 [0] NCCL INFO comm 0x14c90df0 rank 0 nranks 2 cudaDev 0 busId c5000 - Abort COMPLETE
任务 winogrande 评估完成！

==================================================
开始评估：任务=boolq | 少样本数=0 | 模型=Llama-3.1-8B-quantization-fg
输出路径：results2/Llama-3.1-8B-quantization-fg/fg6/boolq.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-09:19:57:52 INFO     [__main__:440] Selected Tasks: ['boolq']
2025-12-09:19:57:52 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:19:57:52 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-09:19:57:52 INFO     [__main__:440] Selected Tasks: ['boolq']
2025-12-09:19:57:52 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:19:57:52 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-09:19:57:53 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:19:57:53 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:19:57:53 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.06s/it]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.06s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.09s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.08s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.06s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.33it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.15it/s]
2025-12-09:19:57:56 WARNING  [api.task:844] [Task: boolq] metric acc is defined, but aggregation is not. using default aggregation=mean
2025-12-09:19:57:56 WARNING  [api.task:856] [Task: boolq] metric acc is defined, but higher_is_better is not. using default higher_is_better=True

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.05s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.33it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.15it/s]
2025-12-09:19:57:57 WARNING  [api.task:844] [Task: boolq] metric acc is defined, but aggregation is not. using default aggregation=mean
2025-12-09:19:57:57 WARNING  [api.task:856] [Task: boolq] metric acc is defined, but higher_is_better is not. using default higher_is_better=True
2025-12-09:19:58:17 WARNING  [evaluator:309] Overwriting default num_fewshot of boolq from None to 0
2025-12-09:19:58:17 INFO     [api.task:434] Building contexts for boolq on rank 1...

  0%|          | 0/1635 [00:00<?, ?it/s]
 17%|█▋        | 274/1635 [00:00<00:00, 2735.80it/s]
 34%|███▍      | 557/1635 [00:00<00:00, 2788.18it/s]
 51%|█████▏    | 839/1635 [00:00<00:00, 2798.87it/s]
 68%|██████▊   | 1119/1635 [00:00<00:00, 2785.23it/s]
 86%|████████▌ | 1399/1635 [00:00<00:00, 2789.89it/s]
100%|██████████| 1635/1635 [00:00<00:00, 2780.70it/s]
2025-12-09:19:58:19 WARNING  [evaluator:309] Overwriting default num_fewshot of boolq from None to 0
2025-12-09:19:58:19 INFO     [api.task:434] Building contexts for boolq on rank 0...

  0%|          | 0/1635 [00:00<?, ?it/s]
 17%|█▋        | 282/1635 [00:00<00:00, 2815.11it/s]
 34%|███▍      | 564/1635 [00:00<00:00, 2780.63it/s]
 52%|█████▏    | 846/1635 [00:00<00:00, 2794.87it/s]
 69%|██████▉   | 1128/1635 [00:00<00:00, 2803.33it/s]
 86%|████████▌ | 1409/1635 [00:00<00:00, 2783.46it/s]
100%|██████████| 1635/1635 [00:00<00:00, 2793.89it/s]
n136-128-154:2247419:2247419 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2247419:2247419 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2247419:2247419 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:2247419:2247419 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2247419:2247419 [0] NCCL INFO Comm config Blocking set to 1
n136-128-154:2247420:2247420 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:2247420:2247420 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2247420:2247420 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2247420:2247420 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2247420:2247420 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:2247419:2247676 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2247419:2247676 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2247420:2247677 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2247420:2247677 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2247419:2247676 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2247419:2247676 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2247419:2247676 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2247420:2247677 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2247420:2247677 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2247420:2247677 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2247419:2247676 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2247419:2247676 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2247419:2247676 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:2247419:2247676 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2247419:2247676 [0] NCCL INFO Using network IB
n136-128-154:2247420:2247677 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2247420:2247677 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2247420:2247677 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:2247420:2247677 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2247420:2247677 [1] NCCL INFO Using network IB
n136-128-154:2247419:2247676 [0] NCCL INFO ncclCommInitRankConfig comm 0x13269030 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0xc5295f96de61b1e1 - Init START
n136-128-154:2247420:2247677 [1] NCCL INFO ncclCommInitRankConfig comm 0xfc4f9e0 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0xc5295f96de61b1e1 - Init START
n136-128-154:2247419:2247676 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2247420:2247677 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2247420:2247677 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2247420:2247677 [1] NCCL INFO Retrieving state for IB
n136-128-154:2247420:2247677 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:2247420:2247677 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2247419:2247676 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2247419:2247676 [0] NCCL INFO Retrieving state for IB
n136-128-154:2247419:2247676 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:2247420:2247677 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2247419:2247676 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2247420:2247677 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2247419:2247676 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2247420:2247677 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2247419:2247676 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2247419:2247676 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2247420:2247677 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2247420:2247677 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2247419:2247676 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2247419:2247676 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2247420:2247677 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2247420:2247677 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2247420:2247677 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2247419:2247676 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2247420:2247677 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2247419:2247676 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2247420:2247677 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2247419:2247676 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2247420:2247677 [1] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2247419:2247676 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2247420:2247677 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2247419:2247676 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2247420:2247677 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2247419:2247676 [0] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2247420:2247677 [1] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2247419:2247676 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2247420:2247677 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2247419:2247676 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2247420:2247677 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2247419:2247676 [0] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2247420:2247677 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2247419:2247676 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2247420:2247677 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2247419:2247676 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2247420:2247677 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2247419:2247676 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2247420:2247677 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2247419:2247676 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2247420:2247677 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2247419:2247676 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2247420:2247677 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2247419:2247676 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2247420:2247677 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2247419:2247676 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2247420:2247677 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2247419:2247676 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2247420:2247677 [1] NCCL INFO ==========================================
n136-128-154:2247419:2247676 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2247419:2247676 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2247419:2247676 [0] NCCL INFO ==========================================
n136-128-154:2247420:2247677 [1] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2247419:2247676 [0] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2247420:2247677 [1] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2247419:2247676 [0] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2247420:2247677 [1] NCCL INFO Setting affinity for GPU 7 to 33-62,97-126
n136-128-154:2247419:2247676 [0] NCCL INFO Setting affinity for GPU 6 to 33-62,97-126
n136-128-154:2247420:2247677 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2247420:2247677 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247420:2247677 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247420:2247677 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247420:2247677 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247420:2247677 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247420:2247677 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247420:2247677 [1] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247420:2247677 [1] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247420:2247677 [1] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247420:2247677 [1] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247420:2247677 [1] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247420:2247677 [1] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247420:2247677 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2247420:2247677 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247420:2247677 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247420:2247677 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247420:2247677 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247420:2247677 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247420:2247677 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247420:2247677 [1] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2247420:2247677 [1] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2247420:2247677 [1] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2247420:2247677 [1] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2247420:2247677 [1] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2247420:2247677 [1] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2247419:2247676 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2247419:2247676 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247419:2247676 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247419:2247676 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247419:2247676 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247419:2247676 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247419:2247676 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247419:2247676 [0] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247419:2247676 [0] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247419:2247676 [0] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247419:2247676 [0] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247419:2247676 [0] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247419:2247676 [0] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247419:2247676 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2247419:2247676 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247419:2247676 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247419:2247676 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247419:2247676 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247419:2247676 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247419:2247676 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247419:2247676 [0] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2247419:2247676 [0] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2247419:2247676 [0] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2247419:2247676 [0] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2247419:2247676 [0] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2247419:2247676 [0] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2247420:2247677 [1] NCCL INFO comm 0xfc4f9e0 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:2247419:2247676 [0] NCCL INFO comm 0x13269030 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:2247419:2247676 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:2247420:2247677 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:2247419:2247676 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:2247420:2247677 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:2247419:2247676 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:2247420:2247677 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:2247419:2247676 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:2247420:2247677 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:2247419:2247676 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:2247420:2247677 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:2247419:2247676 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:2247420:2247677 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:2247419:2247676 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:2247420:2247677 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:2247419:2247676 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:2247420:2247677 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:2247419:2247676 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:2247420:2247677 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:2247419:2247676 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:2247420:2247677 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:2247419:2247676 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:2247420:2247677 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:2247419:2247676 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:2247420:2247677 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:2247419:2247676 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:2247420:2247677 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:2247419:2247676 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:2247420:2247677 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:2247419:2247676 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:2247420:2247677 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:2247419:2247676 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:2247420:2247677 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:2247419:2247676 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:2247420:2247677 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:2247419:2247676 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:2247420:2247677 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:2247419:2247676 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:2247420:2247677 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:2247419:2247676 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:2247420:2247677 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:2247419:2247676 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:2247420:2247677 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:2247419:2247676 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:2247420:2247677 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:2247419:2247676 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:2247420:2247677 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:2247419:2247676 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:2247420:2247677 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:2247419:2247676 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:2247419:2247676 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:2247420:2247677 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:2247419:2247676 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:2247420:2247677 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:2247419:2247676 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:2247420:2247677 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:2247419:2247676 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:2247420:2247677 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:2247419:2247676 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:2247420:2247677 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:2247419:2247676 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:2247420:2247677 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:2247419:2247676 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:2247420:2247677 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:2247419:2247676 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:2247420:2247677 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:2247419:2247676 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:2247420:2247677 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:2247419:2247676 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:2247420:2247677 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:2247419:2247676 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:2247420:2247677 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:2247419:2247676 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:2247420:2247677 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:2247419:2247676 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:2247420:2247677 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:2247419:2247676 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:2247420:2247677 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:2247419:2247676 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:2247420:2247677 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:2247419:2247676 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:2247420:2247677 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:2247419:2247676 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:2247420:2247677 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:2247419:2247676 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:2247420:2247677 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:2247419:2247676 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:2247420:2247677 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:2247419:2247676 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:2247420:2247677 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:2247419:2247676 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:2247420:2247677 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:2247419:2247676 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:2247420:2247677 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:2247419:2247676 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:2247420:2247677 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:2247419:2247676 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:2247420:2247677 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:2247419:2247676 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:2247420:2247677 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:2247419:2247676 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:2247419:2247676 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:2247420:2247677 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2247419:2247676 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:2247419:2247676 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:2247419:2247676 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:2247419:2247676 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:2247419:2247676 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:2247419:2247676 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:2247419:2247676 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:2247419:2247676 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:2247419:2247676 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:2247419:2247676 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:2247419:2247676 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:2247419:2247676 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:2247419:2247676 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:2247419:2247676 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:2247419:2247676 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:2247419:2247676 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:2247419:2247676 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:2247419:2247676 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:2247419:2247676 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:2247419:2247676 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:2247419:2247676 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:2247419:2247676 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2247420:2247677 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2247420:2247689 [1] NCCL INFO [Proxy Service] Device 1 CPU core 102
n136-128-154:2247420:2247690 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 39
n136-128-154:2247419:2247676 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2247419:2247676 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:2247419:2247691 [0] NCCL INFO [Proxy Service] Device 0 CPU core 122
n136-128-154:2247419:2247692 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 123
n136-128-154:2247420:2247677 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2247420:2247677 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2247419:2247676 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2247419:2247676 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2247419:2247676 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:2247420:2247677 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2247420:2247677 [1] NCCL INFO ncclCommInitRankConfig comm 0xfc4f9e0 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0xc5295f96de61b1e1 - Init COMPLETE
n136-128-154:2247420:2247677 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.40 (kernels 0.13, alloc 0.17, bootstrap 0.00, allgathers 0.00, topo 0.05, graphs 0.00, connections 0.01, rest 0.04)
n136-128-154:2247419:2247676 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2247419:2247676 [0] NCCL INFO ncclCommInitRankConfig comm 0x13269030 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0xc5295f96de61b1e1 - Init COMPLETE
n136-128-154:2247419:2247676 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.40 (kernels 0.13, alloc 0.17, bootstrap 0.00, allgathers 0.00, topo 0.05, graphs 0.00, connections 0.01, rest 0.04)
n136-128-154:2247419:2247693 [0] NCCL INFO Channel 00/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247419:2247693 [0] NCCL INFO Channel 01/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247419:2247693 [0] NCCL INFO Channel 02/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247419:2247693 [0] NCCL INFO Channel 03/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247420:2247694 [1] NCCL INFO Channel 00/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247419:2247693 [0] NCCL INFO Channel 04/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247420:2247694 [1] NCCL INFO Channel 01/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247419:2247693 [0] NCCL INFO Channel 05/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247420:2247694 [1] NCCL INFO Channel 02/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247419:2247693 [0] NCCL INFO Channel 06/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247420:2247694 [1] NCCL INFO Channel 03/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247419:2247693 [0] NCCL INFO Channel 07/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247420:2247694 [1] NCCL INFO Channel 04/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247419:2247693 [0] NCCL INFO Channel 08/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247420:2247694 [1] NCCL INFO Channel 05/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247419:2247693 [0] NCCL INFO Channel 09/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247420:2247694 [1] NCCL INFO Channel 06/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247419:2247693 [0] NCCL INFO Channel 10/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247420:2247694 [1] NCCL INFO Channel 07/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247419:2247693 [0] NCCL INFO Channel 11/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247420:2247694 [1] NCCL INFO Channel 08/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247419:2247693 [0] NCCL INFO Channel 12/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247420:2247694 [1] NCCL INFO Channel 09/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247419:2247693 [0] NCCL INFO Channel 13/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247420:2247694 [1] NCCL INFO Channel 10/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247419:2247693 [0] NCCL INFO Channel 14/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247420:2247694 [1] NCCL INFO Channel 11/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247419:2247693 [0] NCCL INFO Channel 15/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247420:2247694 [1] NCCL INFO Channel 12/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247419:2247693 [0] NCCL INFO Channel 16/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247420:2247694 [1] NCCL INFO Channel 13/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247419:2247693 [0] NCCL INFO Channel 17/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247420:2247694 [1] NCCL INFO Channel 14/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247419:2247693 [0] NCCL INFO Channel 18/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247420:2247694 [1] NCCL INFO Channel 15/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247419:2247693 [0] NCCL INFO Channel 19/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247420:2247694 [1] NCCL INFO Channel 16/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247419:2247693 [0] NCCL INFO Channel 20/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247420:2247694 [1] NCCL INFO Channel 17/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247419:2247693 [0] NCCL INFO Channel 21/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247420:2247694 [1] NCCL INFO Channel 18/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247419:2247693 [0] NCCL INFO Channel 22/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247420:2247694 [1] NCCL INFO Channel 19/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247419:2247693 [0] NCCL INFO Channel 23/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247420:2247694 [1] NCCL INFO Channel 20/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247420:2247694 [1] NCCL INFO Channel 21/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247420:2247694 [1] NCCL INFO Channel 22/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247420:2247694 [1] NCCL INFO Channel 23/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247419:2247693 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:2247420:2247694 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-09:19:58:21 INFO     [evaluator:559] Running loglikelihood requests
2025-12-09:19:58:21 INFO     [evaluator:559] Running loglikelihood requests
Passed argument batch_size = auto:1. Detecting largest batch size

Running loglikelihood requests:   0%|          | 0/3270 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 64
Determined largest batch size: 64

Running loglikelihood requests:   0%|          | 1/3270 [00:35<32:01:44, 35.27s/it]
Running loglikelihood requests:   4%|▍         | 129/3270 [00:36<10:43,  4.88it/s] 
Running loglikelihood requests:   8%|▊         | 257/3270 [00:38<04:35, 10.94it/s]
Running loglikelihood requests:  12%|█▏        | 385/3270 [00:39<02:37, 18.32it/s]
Running loglikelihood requests:  16%|█▌        | 513/3270 [00:41<01:41, 27.04it/s]
Running loglikelihood requests:  20%|█▉        | 641/3270 [00:42<01:11, 36.93it/s]
Running loglikelihood requests:  24%|██▎       | 769/3270 [00:43<00:52, 47.68it/s]
Running loglikelihood requests:  27%|██▋       | 897/3270 [00:44<00:40, 58.93it/s]
Running loglikelihood requests:  31%|███▏      | 1025/3270 [00:45<00:31, 70.21it/s]
Running loglikelihood requests:  35%|███▌      | 1153/3270 [00:46<00:26, 80.94it/s]
Running loglikelihood requests:  39%|███▉      | 1281/3270 [00:47<00:21, 91.01it/s]
Running loglikelihood requests:  43%|████▎     | 1409/3270 [00:48<00:18, 100.09it/s]
Running loglikelihood requests:  47%|████▋     | 1537/3270 [00:49<00:16, 108.09it/s]
Running loglikelihood requests:  51%|█████     | 1665/3270 [00:50<00:13, 116.09it/s]
Running loglikelihood requests:  55%|█████▍    | 1793/3270 [00:51<00:12, 122.41it/s]
Running loglikelihood requests:  59%|█████▊    | 1921/3270 [00:52<00:10, 129.26it/s]
Running loglikelihood requests:  63%|██████▎   | 2049/3270 [00:53<00:09, 134.74it/s]
Running loglikelihood requests:  67%|██████▋   | 2177/3270 [00:54<00:07, 140.85it/s]
Running loglikelihood requests:  70%|███████   | 2305/3270 [00:54<00:06, 146.31it/s]
Running loglikelihood requests:  74%|███████▍  | 2433/3270 [00:55<00:05, 151.36it/s]
Running loglikelihood requests:  78%|███████▊  | 2561/3270 [00:56<00:04, 157.41it/s]
Running loglikelihood requests:  82%|████████▏ | 2689/3270 [00:57<00:03, 163.47it/s]
Running loglikelihood requests:  86%|████████▌ | 2817/3270 [00:57<00:02, 170.92it/s]
Running loglikelihood requests:  90%|█████████ | 2945/3270 [00:58<00:01, 178.92it/s]
Running loglikelihood requests:  94%|█████████▍| 3073/3270 [00:58<00:01, 188.91it/s]
Running loglikelihood requests:  98%|█████████▊| 3201/3270 [00:59<00:00, 227.44it/s]
Running loglikelihood requests: 100%|██████████| 3270/3270 [00:59<00:00, 55.19it/s] 
n136-128-154:2247420:2247798 [1] NCCL INFO Channel 00/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247420:2247798 [1] NCCL INFO Channel 01/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247420:2247798 [1] NCCL INFO Channel 02/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247420:2247798 [1] NCCL INFO Channel 03/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247420:2247798 [1] NCCL INFO Channel 04/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247420:2247798 [1] NCCL INFO Channel 05/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247420:2247798 [1] NCCL INFO Channel 06/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247420:2247798 [1] NCCL INFO Channel 07/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247420:2247798 [1] NCCL INFO Channel 08/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247420:2247798 [1] NCCL INFO Channel 09/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247420:2247798 [1] NCCL INFO Channel 10/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247420:2247798 [1] NCCL INFO Channel 11/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247420:2247798 [1] NCCL INFO Channel 12/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247420:2247798 [1] NCCL INFO Channel 13/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247420:2247798 [1] NCCL INFO Channel 14/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247420:2247798 [1] NCCL INFO Channel 15/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247420:2247798 [1] NCCL INFO Channel 16/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247420:2247798 [1] NCCL INFO Channel 17/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247420:2247798 [1] NCCL INFO Channel 18/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247420:2247798 [1] NCCL INFO Channel 19/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247420:2247798 [1] NCCL INFO Channel 20/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247420:2247798 [1] NCCL INFO Channel 21/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247420:2247798 [1] NCCL INFO Channel 22/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247420:2247798 [1] NCCL INFO Channel 23/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247420:2247798 [1] NCCL INFO Channel 24/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247420:2247798 [1] NCCL INFO Channel 25/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247420:2247798 [1] NCCL INFO Channel 26/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247420:2247798 [1] NCCL INFO Channel 27/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247420:2247798 [1] NCCL INFO Channel 28/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247420:2247798 [1] NCCL INFO Channel 29/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247420:2247798 [1] NCCL INFO Channel 30/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247420:2247798 [1] NCCL INFO Channel 31/1 : 1[7] -> 0[6] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:2247420:2247804 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2247420:2247804 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2247420:2247804 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2247420:2247804 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2247420:2247804 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2247420:2247804 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2247420:2247689 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-09:19:59:31 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: auto (64)
|Tasks|Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|-----|------:|------|-----:|------|---|-----:|---|-----:|
|boolq|      2|none  |     0|acc   |↑  |0.7202|±  |0.0079|

[rank0]:[W1209 19:59:31.887913696 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:2247419:2247891 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2247419:2247891 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2247419:2247891 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2247419:2247691 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2247419:2247891 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2247419:2247891 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2247419:2247891 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2247420:2247689 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2247420:2247804 [1] NCCL INFO comm 0xfc4f9e0 rank 1 nranks 2 cudaDev 1 busId c9000 - Abort COMPLETE
n136-128-154:2247419:2247891 [0] NCCL INFO comm 0x13269030 rank 0 nranks 2 cudaDev 0 busId c5000 - Abort COMPLETE
任务 boolq 评估完成！

==================================================
开始评估：任务=arc_challenge | 少样本数=25 | 模型=Llama-3.1-8B-quantization-fg
输出路径：results2/Llama-3.1-8B-quantization-fg/fg6/arc_challenge.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-09:20:00:34 INFO     [__main__:440] Selected Tasks: ['arc_challenge']
2025-12-09:20:00:34 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:20:00:34 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-09:20:00:34 INFO     [__main__:440] Selected Tasks: ['arc_challenge']
2025-12-09:20:00:34 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:20:00:34 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-09:20:00:37 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:20:00:37 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:20:00:37 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.07s/it]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.06it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.08s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.06it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.03s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.09it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.38it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.18it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.56it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.34it/s]
2025-12-09:20:00:55 WARNING  [evaluator:309] Overwriting default num_fewshot of arc_challenge from None to 25
2025-12-09:20:00:55 INFO     [api.task:434] Building contexts for arc_challenge on rank 1...

  0%|          | 0/586 [00:00<?, ?it/s]
  1%|          | 4/586 [00:00<00:16, 36.23it/s]
  1%|▏         | 8/586 [00:00<00:15, 36.65it/s]
  2%|▏         | 12/586 [00:00<00:15, 36.85it/s]
  3%|▎         | 16/586 [00:00<00:15, 37.00it/s]
  3%|▎         | 20/586 [00:00<00:15, 36.98it/s]
  4%|▍         | 24/586 [00:00<00:15, 36.94it/s]
  5%|▍         | 28/586 [00:00<00:15, 37.05it/s]
  5%|▌         | 32/586 [00:00<00:14, 37.16it/s]
  6%|▌         | 36/586 [00:00<00:14, 37.32it/s]
  7%|▋         | 40/586 [00:01<00:14, 37.11it/s]
  8%|▊         | 44/586 [00:01<00:14, 37.23it/s]
  8%|▊         | 48/586 [00:01<00:14, 37.31it/s]
  9%|▉         | 52/586 [00:01<00:14, 37.36it/s]
 10%|▉         | 56/586 [00:01<00:14, 36.73it/s]
 10%|█         | 60/586 [00:01<00:14, 36.82it/s]
 11%|█         | 64/586 [00:01<00:14, 36.88it/s]
 12%|█▏        | 68/586 [00:01<00:14, 36.73it/s]
 12%|█▏        | 72/586 [00:01<00:13, 36.85it/s]
 13%|█▎        | 76/586 [00:02<00:13, 36.96it/s]
 14%|█▎        | 80/586 [00:02<00:13, 37.07it/s]
 14%|█▍        | 84/586 [00:02<00:13, 36.98it/s]
 15%|█▌        | 88/586 [00:02<00:13, 37.10it/s]
 16%|█▌        | 92/586 [00:02<00:13, 37.19it/s]
 16%|█▋        | 96/586 [00:02<00:13, 37.29it/s]
 17%|█▋        | 100/586 [00:02<00:13, 37.37it/s]
 18%|█▊        | 104/586 [00:02<00:12, 37.26it/s]
 18%|█▊        | 108/586 [00:02<00:12, 37.27it/s]
 19%|█▉        | 112/586 [00:03<00:12, 37.24it/s]
 20%|█▉        | 116/586 [00:03<00:12, 37.05it/s]
 20%|██        | 120/586 [00:03<00:12, 37.13it/s]
 21%|██        | 124/586 [00:03<00:12, 37.16it/s]
 22%|██▏       | 128/586 [00:03<00:12, 37.09it/s]
 23%|██▎       | 132/586 [00:03<00:12, 37.08it/s]
 23%|██▎       | 136/586 [00:03<00:12, 37.12it/s]
 24%|██▍       | 140/586 [00:03<00:11, 37.18it/s]
 25%|██▍       | 144/586 [00:03<00:11, 37.25it/s]
 25%|██▌       | 148/586 [00:03<00:11, 36.62it/s]
 26%|██▌       | 152/586 [00:04<00:11, 36.77it/s]
 27%|██▋       | 156/586 [00:04<00:11, 36.77it/s]
 27%|██▋       | 160/586 [00:04<00:11, 36.84it/s]
 28%|██▊       | 164/586 [00:04<00:11, 36.93it/s]
 29%|██▊       | 168/586 [00:04<00:11, 36.86it/s]2025-12-09:20:01:00 WARNING  [evaluator:309] Overwriting default num_fewshot of arc_challenge from None to 25
2025-12-09:20:01:00 INFO     [api.task:434] Building contexts for arc_challenge on rank 0...

  0%|          | 0/586 [00:00<?, ?it/s]
 29%|██▉       | 172/586 [00:04<00:11, 36.96it/s]
  1%|          | 4/586 [00:00<00:16, 35.83it/s]
 30%|███       | 176/586 [00:04<00:11, 37.06it/s]
  1%|▏         | 8/586 [00:00<00:16, 35.75it/s]
 31%|███       | 180/586 [00:04<00:11, 36.82it/s]
  2%|▏         | 12/586 [00:00<00:15, 35.92it/s]
 31%|███▏      | 184/586 [00:04<00:10, 36.92it/s]
  3%|▎         | 16/586 [00:00<00:15, 36.13it/s]
 32%|███▏      | 188/586 [00:05<00:10, 36.97it/s]
  3%|▎         | 20/586 [00:00<00:15, 36.16it/s]
 33%|███▎      | 192/586 [00:05<00:10, 37.07it/s]
  4%|▍         | 24/586 [00:00<00:15, 36.35it/s]
 33%|███▎      | 196/586 [00:05<00:10, 36.83it/s]
  5%|▍         | 28/586 [00:00<00:15, 36.01it/s]
 34%|███▍      | 200/586 [00:05<00:10, 36.99it/s]
  5%|▌         | 32/586 [00:00<00:15, 36.28it/s]
 35%|███▍      | 204/586 [00:05<00:10, 37.09it/s]
  6%|▌         | 36/586 [00:00<00:15, 36.12it/s]
 35%|███▌      | 208/586 [00:05<00:10, 37.13it/s]
  7%|▋         | 40/586 [00:01<00:15, 36.38it/s]
 36%|███▌      | 212/586 [00:05<00:10, 36.51it/s]
  8%|▊         | 44/586 [00:01<00:14, 36.55it/s]
 37%|███▋      | 216/586 [00:05<00:10, 36.65it/s]
  8%|▊         | 48/586 [00:01<00:14, 36.45it/s]
 38%|███▊      | 220/586 [00:05<00:09, 36.81it/s]
  9%|▉         | 52/586 [00:01<00:14, 36.50it/s]
 38%|███▊      | 224/586 [00:06<00:09, 36.98it/s]
 10%|▉         | 56/586 [00:01<00:14, 36.44it/s]
 39%|███▉      | 228/586 [00:06<00:09, 37.05it/s]
 10%|█         | 60/586 [00:01<00:14, 35.96it/s]
 40%|███▉      | 232/586 [00:06<00:09, 36.91it/s]
 11%|█         | 64/586 [00:01<00:14, 36.18it/s]
 40%|████      | 236/586 [00:06<00:09, 36.93it/s]
 12%|█▏        | 68/586 [00:01<00:14, 36.42it/s]
 41%|████      | 240/586 [00:06<00:09, 37.04it/s]
 12%|█▏        | 72/586 [00:01<00:14, 36.64it/s]
 42%|████▏     | 244/586 [00:06<00:09, 37.13it/s]
 13%|█▎        | 76/586 [00:02<00:13, 36.71it/s]
 42%|████▏     | 248/586 [00:06<00:09, 37.20it/s]
 14%|█▎        | 80/586 [00:02<00:13, 36.81it/s]
 43%|████▎     | 252/586 [00:06<00:08, 37.27it/s]
 14%|█▍        | 84/586 [00:02<00:13, 36.88it/s]
 44%|████▎     | 256/586 [00:06<00:08, 37.35it/s]
 15%|█▌        | 88/586 [00:02<00:13, 36.68it/s]
 44%|████▍     | 260/586 [00:07<00:08, 36.66it/s]
 16%|█▌        | 92/586 [00:02<00:13, 36.71it/s]
 45%|████▌     | 264/586 [00:07<00:08, 36.25it/s]
 16%|█▋        | 96/586 [00:02<00:13, 36.73it/s]
 46%|████▌     | 268/586 [00:07<00:08, 36.05it/s]
 17%|█▋        | 100/586 [00:02<00:13, 36.21it/s]
 46%|████▋     | 272/586 [00:07<00:08, 36.19it/s]
 18%|█▊        | 104/586 [00:02<00:13, 36.34it/s]
 47%|████▋     | 276/586 [00:07<00:08, 36.38it/s]
 18%|█▊        | 108/586 [00:02<00:13, 36.45it/s]
 48%|████▊     | 280/586 [00:07<00:08, 34.79it/s]
 19%|█▉        | 112/586 [00:03<00:12, 36.54it/s]
 48%|████▊     | 284/586 [00:07<00:08, 35.29it/s]
 20%|█▉        | 116/586 [00:03<00:12, 36.50it/s]
 49%|████▉     | 288/586 [00:07<00:08, 35.70it/s]
 20%|██        | 120/586 [00:03<00:12, 36.60it/s]
 50%|████▉     | 292/586 [00:07<00:08, 35.84it/s]
 21%|██        | 124/586 [00:03<00:12, 36.71it/s]
 51%|█████     | 296/586 [00:08<00:08, 36.12it/s]
 22%|██▏       | 128/586 [00:03<00:12, 36.78it/s]
 51%|█████     | 300/586 [00:08<00:07, 36.45it/s]
 23%|██▎       | 132/586 [00:03<00:12, 36.63it/s]
 52%|█████▏    | 304/586 [00:08<00:07, 36.65it/s]
 23%|██▎       | 136/586 [00:03<00:12, 36.74it/s]
 53%|█████▎    | 308/586 [00:08<00:07, 36.76it/s]
 24%|██▍       | 140/586 [00:03<00:12, 36.78it/s]
 53%|█████▎    | 312/586 [00:08<00:07, 36.87it/s]
 25%|██▍       | 144/586 [00:03<00:11, 36.86it/s]
 54%|█████▍    | 316/586 [00:08<00:07, 36.88it/s]
 25%|██▌       | 148/586 [00:04<00:11, 36.90it/s]
 55%|█████▍    | 320/586 [00:08<00:07, 36.70it/s]
 26%|██▌       | 152/586 [00:04<00:11, 36.38it/s]
 55%|█████▌    | 324/586 [00:08<00:07, 36.80it/s]
 27%|██▋       | 156/586 [00:04<00:11, 36.56it/s]
 56%|█████▌    | 328/586 [00:08<00:06, 36.93it/s]
 27%|██▋       | 160/586 [00:04<00:11, 36.74it/s]
 57%|█████▋    | 332/586 [00:09<00:06, 37.03it/s]
 28%|██▊       | 164/586 [00:04<00:11, 36.85it/s]
 57%|█████▋    | 336/586 [00:09<00:06, 36.45it/s]
 29%|██▊       | 168/586 [00:04<00:11, 36.90it/s]
 58%|█████▊    | 340/586 [00:09<00:06, 36.63it/s]
 29%|██▉       | 172/586 [00:04<00:11, 36.88it/s]
 59%|█████▊    | 344/586 [00:09<00:06, 36.80it/s]
 30%|███       | 176/586 [00:04<00:11, 36.94it/s]
 59%|█████▉    | 348/586 [00:09<00:06, 36.96it/s]
 31%|███       | 180/586 [00:04<00:10, 37.03it/s]
 60%|██████    | 352/586 [00:09<00:06, 37.10it/s]
 31%|███▏      | 184/586 [00:05<00:10, 37.06it/s]
 61%|██████    | 356/586 [00:09<00:06, 36.84it/s]
 32%|███▏      | 188/586 [00:05<00:10, 37.06it/s]
 61%|██████▏   | 360/586 [00:09<00:06, 36.88it/s]
 33%|███▎      | 192/586 [00:05<00:10, 36.88it/s]
 62%|██████▏   | 364/586 [00:09<00:06, 36.96it/s]
 33%|███▎      | 196/586 [00:05<00:10, 36.96it/s]
 63%|██████▎   | 368/586 [00:09<00:05, 37.02it/s]
 34%|███▍      | 200/586 [00:05<00:10, 36.98it/s]
 63%|██████▎   | 372/586 [00:10<00:05, 37.03it/s]
 35%|███▍      | 204/586 [00:05<00:10, 36.98it/s]
 64%|██████▍   | 376/586 [00:10<00:05, 37.04it/s]
 35%|███▌      | 208/586 [00:05<00:10, 37.05it/s]
 65%|██████▍   | 380/586 [00:10<00:05, 37.10it/s]
 36%|███▌      | 212/586 [00:05<00:10, 36.50it/s]
 66%|██████▌   | 384/586 [00:10<00:05, 37.04it/s]
 37%|███▋      | 216/586 [00:05<00:10, 36.60it/s]
 66%|██████▌   | 388/586 [00:10<00:05, 37.02it/s]
 38%|███▊      | 220/586 [00:06<00:09, 36.73it/s]
 67%|██████▋   | 392/586 [00:10<00:05, 37.01it/s]
 38%|███▊      | 224/586 [00:06<00:09, 36.81it/s]
 68%|██████▊   | 396/586 [00:10<00:05, 37.03it/s]
 39%|███▉      | 228/586 [00:06<00:09, 36.88it/s]
 68%|██████▊   | 400/586 [00:10<00:05, 37.07it/s]
 40%|███▉      | 232/586 [00:06<00:09, 36.94it/s]
 69%|██████▉   | 404/586 [00:10<00:04, 37.12it/s]
 40%|████      | 236/586 [00:06<00:09, 36.83it/s]
 70%|██████▉   | 408/586 [00:11<00:04, 37.14it/s]
 41%|████      | 240/586 [00:06<00:09, 36.67it/s]
 70%|███████   | 412/586 [00:11<00:04, 37.17it/s]
 42%|████▏     | 244/586 [00:06<00:09, 36.76it/s]
 71%|███████   | 416/586 [00:11<00:04, 37.15it/s]
 42%|████▏     | 248/586 [00:06<00:09, 36.72it/s]
 72%|███████▏  | 420/586 [00:11<00:04, 37.16it/s]
 43%|████▎     | 252/586 [00:06<00:09, 36.68it/s]
 72%|███████▏  | 424/586 [00:11<00:04, 37.17it/s]
 44%|████▎     | 256/586 [00:06<00:09, 36.58it/s]
 73%|███████▎  | 428/586 [00:11<00:04, 37.16it/s]
 44%|████▍     | 260/586 [00:07<00:08, 36.66it/s]
 74%|███████▎  | 432/586 [00:11<00:04, 37.13it/s]
 45%|████▌     | 264/586 [00:07<00:08, 36.43it/s]
 74%|███████▍  | 436/586 [00:11<00:04, 37.12it/s]
 46%|████▌     | 268/586 [00:07<00:08, 36.45it/s]
 75%|███████▌  | 440/586 [00:11<00:03, 37.14it/s]
 46%|████▋     | 272/586 [00:07<00:08, 36.60it/s]
 76%|███████▌  | 444/586 [00:12<00:03, 37.10it/s]
 47%|████▋     | 276/586 [00:07<00:08, 36.74it/s]
 76%|███████▋  | 448/586 [00:12<00:03, 37.06it/s]
 48%|████▊     | 280/586 [00:07<00:08, 36.82it/s]
 77%|███████▋  | 452/586 [00:12<00:03, 37.05it/s]
 48%|████▊     | 284/586 [00:07<00:08, 36.84it/s]
 78%|███████▊  | 456/586 [00:12<00:03, 37.12it/s]
 49%|████▉     | 288/586 [00:07<00:08, 36.84it/s]
 78%|███████▊  | 460/586 [00:12<00:03, 37.19it/s]
 50%|████▉     | 292/586 [00:07<00:08, 36.29it/s]
 79%|███████▉  | 464/586 [00:12<00:03, 36.93it/s]
 51%|█████     | 296/586 [00:08<00:07, 36.38it/s]
 80%|███████▉  | 468/586 [00:12<00:03, 36.92it/s]
 51%|█████     | 300/586 [00:08<00:07, 36.49it/s]
 81%|████████  | 472/586 [00:12<00:03, 36.70it/s]
 52%|█████▏    | 304/586 [00:08<00:07, 36.59it/s]
 81%|████████  | 476/586 [00:12<00:02, 36.82it/s]
 53%|█████▎    | 308/586 [00:08<00:07, 36.63it/s]
 82%|████████▏ | 480/586 [00:13<00:02, 36.90it/s]
 53%|█████▎    | 312/586 [00:08<00:07, 36.68it/s]
 83%|████████▎ | 484/586 [00:13<00:02, 36.91it/s]
 54%|█████▍    | 316/586 [00:08<00:07, 36.71it/s]
 83%|████████▎ | 488/586 [00:13<00:02, 36.98it/s]
 55%|█████▍    | 320/586 [00:08<00:07, 36.56it/s]
 84%|████████▍ | 492/586 [00:13<00:02, 36.97it/s]
 55%|█████▌    | 324/586 [00:08<00:07, 36.62it/s]
 85%|████████▍ | 496/586 [00:13<00:02, 36.71it/s]
 56%|█████▌    | 328/586 [00:08<00:07, 36.69it/s]
 85%|████████▌ | 500/586 [00:13<00:02, 36.83it/s]
 57%|█████▋    | 332/586 [00:09<00:06, 36.64it/s]
 86%|████████▌ | 504/586 [00:13<00:02, 36.16it/s]
 57%|█████▋    | 336/586 [00:09<00:06, 36.67it/s]
 87%|████████▋ | 508/586 [00:13<00:02, 36.24it/s]
 58%|█████▊    | 340/586 [00:09<00:06, 36.70it/s]
 87%|████████▋ | 512/586 [00:13<00:02, 36.46it/s]
 59%|█████▊    | 344/586 [00:09<00:06, 36.73it/s]
 88%|████████▊ | 516/586 [00:13<00:01, 36.63it/s]
 59%|█████▉    | 348/586 [00:09<00:06, 36.77it/s]
 89%|████████▊ | 520/586 [00:14<00:01, 36.76it/s]
 60%|██████    | 352/586 [00:09<00:06, 36.57it/s]
 89%|████████▉ | 524/586 [00:14<00:01, 36.85it/s]
 61%|██████    | 356/586 [00:09<00:06, 36.67it/s]
 90%|█████████ | 528/586 [00:14<00:01, 36.95it/s]
 61%|██████▏   | 360/586 [00:09<00:06, 36.66it/s]
 91%|█████████ | 532/586 [00:14<00:01, 36.99it/s]
 62%|██████▏   | 364/586 [00:09<00:06, 36.67it/s]
 91%|█████████▏| 536/586 [00:14<00:01, 37.13it/s]
 63%|██████▎   | 368/586 [00:10<00:05, 36.61it/s]
 92%|█████████▏| 540/586 [00:14<00:01, 36.96it/s]
 93%|█████████▎| 544/586 [00:14<00:01, 37.11it/s]
 63%|██████▎   | 372/586 [00:10<00:05, 36.59it/s]
 94%|█████████▎| 548/586 [00:14<00:01, 37.01it/s]
 64%|██████▍   | 376/586 [00:10<00:05, 36.53it/s]
 94%|█████████▍| 552/586 [00:14<00:00, 37.06it/s]
 65%|██████▍   | 380/586 [00:10<00:05, 36.54it/s]
 95%|█████████▍| 556/586 [00:15<00:00, 37.13it/s]
 66%|██████▌   | 384/586 [00:10<00:05, 36.15it/s]
 96%|█████████▌| 560/586 [00:15<00:00, 36.73it/s]
 66%|██████▌   | 388/586 [00:10<00:05, 36.31it/s]
 67%|██████▋   | 392/586 [00:10<00:05, 36.43it/s]
 96%|█████████▌| 564/586 [00:15<00:00, 35.17it/s]
 68%|██████▊   | 396/586 [00:10<00:05, 36.11it/s]
 97%|█████████▋| 568/586 [00:15<00:00, 35.50it/s]
 68%|██████▊   | 400/586 [00:10<00:05, 36.13it/s]
 98%|█████████▊| 572/586 [00:15<00:00, 34.39it/s]
 69%|██████▉   | 404/586 [00:11<00:05, 36.29it/s]
 98%|█████████▊| 576/586 [00:15<00:00, 34.95it/s]
 70%|██████▉   | 408/586 [00:11<00:04, 36.35it/s]
 99%|█████████▉| 580/586 [00:15<00:00, 35.53it/s]
 70%|███████   | 412/586 [00:11<00:04, 36.36it/s]
100%|█████████▉| 584/586 [00:15<00:00, 35.97it/s]
100%|██████████| 586/586 [00:15<00:00, 36.80it/s]

 71%|███████   | 416/586 [00:11<00:04, 36.24it/s]
 72%|███████▏  | 420/586 [00:11<00:04, 36.22it/s]
 72%|███████▏  | 424/586 [00:11<00:04, 34.87it/s]
 73%|███████▎  | 428/586 [00:11<00:04, 35.30it/s]
 74%|███████▎  | 432/586 [00:11<00:04, 35.64it/s]
 74%|███████▍  | 436/586 [00:11<00:04, 35.92it/s]
 75%|███████▌  | 440/586 [00:12<00:04, 36.16it/s]
 76%|███████▌  | 444/586 [00:12<00:03, 36.31it/s]
 76%|███████▋  | 448/586 [00:12<00:03, 36.39it/s]
 77%|███████▋  | 452/586 [00:12<00:03, 36.53it/s]
 78%|███████▊  | 456/586 [00:12<00:03, 36.36it/s]
 78%|███████▊  | 460/586 [00:12<00:03, 36.50it/s]
 79%|███████▉  | 464/586 [00:12<00:03, 36.62it/s]
 80%|███████▉  | 468/586 [00:12<00:03, 36.64it/s]
 81%|████████  | 472/586 [00:12<00:03, 36.70it/s]
 81%|████████  | 476/586 [00:13<00:03, 35.87it/s]
 82%|████████▏ | 480/586 [00:13<00:02, 36.14it/s]
 83%|████████▎ | 484/586 [00:13<00:02, 36.31it/s]
 83%|████████▎ | 488/586 [00:13<00:02, 36.47it/s]
 84%|████████▍ | 492/586 [00:13<00:02, 36.00it/s]
 85%|████████▍ | 496/586 [00:13<00:02, 36.15it/s]
 85%|████████▌ | 500/586 [00:13<00:02, 36.32it/s]
 86%|████████▌ | 504/586 [00:13<00:02, 36.42it/s]
 87%|████████▋ | 508/586 [00:13<00:02, 36.52it/s]
 87%|████████▋ | 512/586 [00:14<00:02, 36.60it/s]
 88%|████████▊ | 516/586 [00:14<00:01, 36.66it/s]
 89%|████████▊ | 520/586 [00:14<00:01, 36.72it/s]
 89%|████████▉ | 524/586 [00:14<00:01, 36.77it/s]
 90%|█████████ | 528/586 [00:14<00:01, 36.80it/s]
 91%|█████████ | 532/586 [00:14<00:01, 36.70it/s]
 91%|█████████▏| 536/586 [00:14<00:01, 36.76it/s]
 92%|█████████▏| 540/586 [00:14<00:01, 36.77it/s]
 93%|█████████▎| 544/586 [00:14<00:01, 36.78it/s]
 94%|█████████▎| 548/586 [00:14<00:01, 36.79it/s]
 94%|█████████▍| 552/586 [00:15<00:00, 36.81it/s]
 95%|█████████▍| 556/586 [00:15<00:00, 36.83it/s]
 96%|█████████▌| 560/586 [00:15<00:00, 36.82it/s]
 96%|█████████▌| 564/586 [00:15<00:00, 36.80it/s]
 97%|█████████▋| 568/586 [00:15<00:00, 36.78it/s]
 98%|█████████▊| 572/586 [00:15<00:00, 36.81it/s]
 98%|█████████▊| 576/586 [00:15<00:00, 36.80it/s]
 99%|█████████▉| 580/586 [00:15<00:00, 36.62it/s]
100%|█████████▉| 584/586 [00:15<00:00, 36.74it/s]
100%|██████████| 586/586 [00:16<00:00, 36.55it/s]
n136-128-154:2247970:2247970 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2247970:2247970 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2247970:2247970 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:2247970:2247970 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2247970:2247970 [0] NCCL INFO Comm config Blocking set to 1
n136-128-154:2247972:2247972 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:2247972:2247972 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2247972:2247972 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2247972:2247972 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2247972:2247972 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:2247970:2248483 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2247970:2248483 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2247972:2248484 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2247972:2248484 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2247970:2248483 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2247970:2248483 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2247970:2248483 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2247972:2248484 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2247972:2248484 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2247972:2248484 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2247972:2248484 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2247972:2248484 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2247972:2248484 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:2247972:2248484 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2247972:2248484 [1] NCCL INFO Using network IB
n136-128-154:2247970:2248483 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2247970:2248483 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2247970:2248483 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:2247970:2248483 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2247970:2248483 [0] NCCL INFO Using network IB
n136-128-154:2247970:2248483 [0] NCCL INFO ncclCommInitRankConfig comm 0x127dd0e0 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0x306930abc4c7a324 - Init START
n136-128-154:2247972:2248484 [1] NCCL INFO ncclCommInitRankConfig comm 0x112e94a0 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0x306930abc4c7a324 - Init START
n136-128-154:2247970:2248483 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2247972:2248484 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2247970:2248483 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2247970:2248483 [0] NCCL INFO Retrieving state for IB
n136-128-154:2247970:2248483 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:2247970:2248483 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2247972:2248484 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2247972:2248484 [1] NCCL INFO Retrieving state for IB
n136-128-154:2247972:2248484 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:2247970:2248483 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2247972:2248484 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2247970:2248483 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2247972:2248484 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2247970:2248483 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2247972:2248484 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2247972:2248484 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2247970:2248483 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2247970:2248483 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2247972:2248484 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2247972:2248484 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2247972:2248484 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2247972:2248484 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2247972:2248484 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2247972:2248484 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2247972:2248484 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2247972:2248484 [1] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2247972:2248484 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2247972:2248484 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2247972:2248484 [1] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2247972:2248484 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2247972:2248484 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2247972:2248484 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2247972:2248484 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2247972:2248484 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2247972:2248484 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2247972:2248484 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2247972:2248484 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2247972:2248484 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2247972:2248484 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2247972:2248484 [1] NCCL INFO ==========================================
n136-128-154:2247972:2248484 [1] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2247972:2248484 [1] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2247972:2248484 [1] NCCL INFO Setting affinity for GPU 7 to 33-62,97-126
n136-128-154:2247970:2248483 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2247970:2248483 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2247970:2248483 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2247970:2248483 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2247970:2248483 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2247970:2248483 [0] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2247970:2248483 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2247970:2248483 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2247970:2248483 [0] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2247970:2248483 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2247970:2248483 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2247970:2248483 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2247970:2248483 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2247970:2248483 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2247970:2248483 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2247970:2248483 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2247970:2248483 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2247970:2248483 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2247970:2248483 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2247970:2248483 [0] NCCL INFO ==========================================
n136-128-154:2247970:2248483 [0] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2247970:2248483 [0] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2247970:2248483 [0] NCCL INFO Setting affinity for GPU 6 to 33-62,97-126
n136-128-154:2247972:2248484 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2247972:2248484 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247972:2248484 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247972:2248484 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247972:2248484 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247972:2248484 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247972:2248484 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247972:2248484 [1] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247972:2248484 [1] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247972:2248484 [1] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247972:2248484 [1] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247972:2248484 [1] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247972:2248484 [1] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247972:2248484 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2247970:2248483 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2247972:2248484 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247970:2248483 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247972:2248484 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247970:2248483 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247972:2248484 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247970:2248483 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247972:2248484 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247970:2248483 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247972:2248484 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247970:2248483 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247972:2248484 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247970:2248483 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247972:2248484 [1] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2247970:2248483 [0] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247972:2248484 [1] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2247970:2248483 [0] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247972:2248484 [1] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2247970:2248483 [0] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247972:2248484 [1] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2247970:2248483 [0] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247972:2248484 [1] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2247970:2248483 [0] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247972:2248484 [1] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2247970:2248483 [0] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247970:2248483 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2247970:2248483 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247970:2248483 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247970:2248483 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247970:2248483 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247970:2248483 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247970:2248483 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2247970:2248483 [0] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2247970:2248483 [0] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2247970:2248483 [0] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2247970:2248483 [0] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2247970:2248483 [0] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2247970:2248483 [0] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2247970:2248483 [0] NCCL INFO comm 0x127dd0e0 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:2247972:2248484 [1] NCCL INFO comm 0x112e94a0 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:2247970:2248483 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:2247972:2248484 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:2247970:2248483 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:2247972:2248484 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:2247970:2248483 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:2247972:2248484 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:2247970:2248483 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:2247972:2248484 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:2247970:2248483 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:2247972:2248484 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:2247970:2248483 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:2247972:2248484 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:2247970:2248483 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:2247972:2248484 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:2247970:2248483 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:2247972:2248484 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:2247970:2248483 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:2247972:2248484 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:2247970:2248483 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:2247972:2248484 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:2247970:2248483 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:2247972:2248484 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:2247970:2248483 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:2247972:2248484 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:2247970:2248483 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:2247972:2248484 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:2247970:2248483 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:2247972:2248484 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:2247970:2248483 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:2247972:2248484 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:2247970:2248483 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:2247972:2248484 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:2247970:2248483 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:2247972:2248484 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:2247970:2248483 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:2247972:2248484 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:2247970:2248483 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:2247972:2248484 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:2247970:2248483 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:2247972:2248484 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:2247970:2248483 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:2247972:2248484 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:2247970:2248483 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:2247972:2248484 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:2247970:2248483 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:2247972:2248484 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:2247970:2248483 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:2247972:2248484 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:2247970:2248483 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:2247970:2248483 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:2247972:2248484 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:2247970:2248483 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:2247972:2248484 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:2247970:2248483 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:2247972:2248484 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:2247970:2248483 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:2247972:2248484 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:2247970:2248483 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:2247972:2248484 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:2247970:2248483 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:2247972:2248484 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:2247970:2248483 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:2247972:2248484 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:2247970:2248483 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:2247972:2248484 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:2247970:2248483 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:2247972:2248484 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:2247970:2248483 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:2247972:2248484 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:2247970:2248483 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:2247970:2248483 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:2247972:2248484 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:2247970:2248483 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:2247972:2248484 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:2247970:2248483 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:2247972:2248484 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:2247970:2248483 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:2247972:2248484 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:2247970:2248483 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:2247972:2248484 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:2247970:2248483 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:2247972:2248484 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:2247970:2248483 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:2247972:2248484 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:2247970:2248483 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:2247972:2248484 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:2247970:2248483 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:2247972:2248484 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:2247970:2248483 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:2247972:2248484 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:2247970:2248483 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:2247972:2248484 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:2247970:2248483 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:2247972:2248484 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:2247970:2248483 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:2247972:2248484 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:2247970:2248483 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:2247972:2248484 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:2247970:2248483 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:2247972:2248484 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:2247970:2248483 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:2247970:2248483 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:2247972:2248484 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2247970:2248483 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:2247970:2248483 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:2247970:2248483 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:2247970:2248483 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:2247970:2248483 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:2247970:2248483 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:2247970:2248483 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:2247970:2248483 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:2247970:2248483 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:2247970:2248483 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:2247970:2248483 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:2247970:2248483 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:2247970:2248483 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:2247970:2248483 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:2247970:2248483 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:2247970:2248483 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:2247970:2248483 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:2247970:2248483 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:2247970:2248483 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:2247970:2248483 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:2247970:2248483 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2247970:2248483 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2247970:2248483 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:2247970:2248498 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 115
n136-128-154:2247970:2248497 [0] NCCL INFO [Proxy Service] Device 0 CPU core 113
n136-128-154:2247972:2248484 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2247972:2248499 [1] NCCL INFO [Proxy Service] Device 1 CPU core 52
n136-128-154:2247972:2248500 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 53
n136-128-154:2247970:2248483 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2247970:2248483 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2247972:2248484 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2247972:2248484 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2247970:2248483 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:2247970:2248483 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2247970:2248483 [0] NCCL INFO ncclCommInitRankConfig comm 0x127dd0e0 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0x306930abc4c7a324 - Init COMPLETE
n136-128-154:2247970:2248483 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.63 (kernels 0.12, alloc 0.30, bootstrap 0.00, allgathers 0.00, topo 0.05, graphs 0.00, connections 0.07, rest 0.08)
n136-128-154:2247972:2248484 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2247972:2248484 [1] NCCL INFO ncclCommInitRankConfig comm 0x112e94a0 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0x306930abc4c7a324 - Init COMPLETE
n136-128-154:2247972:2248484 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.62 (kernels 0.12, alloc 0.29, bootstrap 0.00, allgathers 0.00, topo 0.05, graphs 0.00, connections 0.07, rest 0.08)
n136-128-154:2247970:2248501 [0] NCCL INFO Channel 00/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247970:2248501 [0] NCCL INFO Channel 01/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247970:2248501 [0] NCCL INFO Channel 02/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247970:2248501 [0] NCCL INFO Channel 03/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247970:2248501 [0] NCCL INFO Channel 04/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247970:2248501 [0] NCCL INFO Channel 05/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247970:2248501 [0] NCCL INFO Channel 06/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247970:2248501 [0] NCCL INFO Channel 07/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247970:2248501 [0] NCCL INFO Channel 08/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247970:2248501 [0] NCCL INFO Channel 09/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247970:2248501 [0] NCCL INFO Channel 10/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247972:2248502 [1] NCCL INFO Channel 00/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247970:2248501 [0] NCCL INFO Channel 11/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247972:2248502 [1] NCCL INFO Channel 01/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247970:2248501 [0] NCCL INFO Channel 12/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247972:2248502 [1] NCCL INFO Channel 02/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247970:2248501 [0] NCCL INFO Channel 13/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247972:2248502 [1] NCCL INFO Channel 03/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247970:2248501 [0] NCCL INFO Channel 14/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247972:2248502 [1] NCCL INFO Channel 04/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247970:2248501 [0] NCCL INFO Channel 15/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247972:2248502 [1] NCCL INFO Channel 05/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247970:2248501 [0] NCCL INFO Channel 16/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247972:2248502 [1] NCCL INFO Channel 06/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247970:2248501 [0] NCCL INFO Channel 17/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247972:2248502 [1] NCCL INFO Channel 07/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247970:2248501 [0] NCCL INFO Channel 18/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247972:2248502 [1] NCCL INFO Channel 08/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247970:2248501 [0] NCCL INFO Channel 19/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247972:2248502 [1] NCCL INFO Channel 09/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247970:2248501 [0] NCCL INFO Channel 20/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247972:2248502 [1] NCCL INFO Channel 10/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247970:2248501 [0] NCCL INFO Channel 21/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247972:2248502 [1] NCCL INFO Channel 11/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247970:2248501 [0] NCCL INFO Channel 22/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247972:2248502 [1] NCCL INFO Channel 12/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247970:2248501 [0] NCCL INFO Channel 23/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2247972:2248502 [1] NCCL INFO Channel 13/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2248502 [1] NCCL INFO Channel 14/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2248502 [1] NCCL INFO Channel 15/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2248502 [1] NCCL INFO Channel 16/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2248502 [1] NCCL INFO Channel 17/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2248502 [1] NCCL INFO Channel 18/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2248502 [1] NCCL INFO Channel 19/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2248502 [1] NCCL INFO Channel 20/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2248502 [1] NCCL INFO Channel 21/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2248502 [1] NCCL INFO Channel 22/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2248502 [1] NCCL INFO Channel 23/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2248502 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:2247970:2248501 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-09:20:01:17 INFO     [evaluator:559] Running loglikelihood requests
2025-12-09:20:01:17 INFO     [evaluator:559] Running loglikelihood requests

Running loglikelihood requests:   0%|          | 0/2344 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 64
Determined largest batch size: 64

Running loglikelihood requests:   0%|          | 1/2344 [00:39<25:37:08, 39.36s/it]
Running loglikelihood requests:   3%|▎         | 65/2344 [00:46<20:20,  1.87it/s]  
Running loglikelihood requests:   6%|▌         | 135/2344 [00:52<09:35,  3.84it/s]
Running loglikelihood requests:   9%|▊         | 204/2344 [00:57<06:20,  5.62it/s]
Running loglikelihood requests:  12%|█▏        | 276/2344 [01:03<04:42,  7.31it/s]
Running loglikelihood requests:  15%|█▍        | 341/2344 [01:08<03:57,  8.44it/s]
Running loglikelihood requests:  17%|█▋        | 408/2344 [01:14<03:24,  9.47it/s]
Running loglikelihood requests:  20%|██        | 475/2344 [01:19<03:01, 10.30it/s]
Running loglikelihood requests:  24%|██▎       | 551/2344 [01:24<02:36, 11.44it/s]
Running loglikelihood requests:  26%|██▋       | 616/2344 [01:29<02:27, 11.72it/s]
Running loglikelihood requests:  29%|██▉       | 684/2344 [01:35<02:16, 12.12it/s]
Running loglikelihood requests:  32%|███▏      | 748/2344 [01:40<02:10, 12.22it/s]
Running loglikelihood requests:  35%|███▍      | 815/2344 [01:45<02:02, 12.48it/s]
Running loglikelihood requests:  38%|███▊      | 883/2344 [01:50<01:54, 12.76it/s]
Running loglikelihood requests:  41%|████      | 952/2344 [01:55<01:46, 13.03it/s]
Running loglikelihood requests:  44%|████▎     | 1022/2344 [02:00<01:39, 13.31it/s]
Running loglikelihood requests:  47%|████▋     | 1094/2344 [02:05<01:31, 13.66it/s]
Running loglikelihood requests:  50%|█████     | 1172/2344 [02:10<01:22, 14.29it/s]
Running loglikelihood requests:  53%|█████▎    | 1238/2344 [02:15<01:18, 14.04it/s]
Running loglikelihood requests:  56%|█████▌    | 1306/2344 [02:20<01:14, 14.00it/s]
Running loglikelihood requests:  59%|█████▊    | 1377/2344 [02:25<01:08, 14.18it/s]
Running loglikelihood requests:  62%|██████▏   | 1451/2344 [02:29<01:01, 14.54it/s]
Running loglikelihood requests:  65%|██████▍   | 1518/2344 [02:34<00:57, 14.41it/s]
Running loglikelihood requests:  68%|██████▊   | 1587/2344 [02:39<00:52, 14.48it/s]
Running loglikelihood requests:  71%|███████   | 1654/2344 [02:43<00:47, 14.45it/s]
Running loglikelihood requests:  74%|███████▎  | 1724/2344 [02:48<00:42, 14.63it/s]
Running loglikelihood requests:  76%|███████▋  | 1793/2344 [02:53<00:37, 14.74it/s]
Running loglikelihood requests:  79%|███████▉  | 1863/2344 [02:57<00:32, 14.91it/s]
Running loglikelihood requests:  82%|████████▏ | 1933/2344 [03:02<00:27, 15.07it/s]
Running loglikelihood requests:  85%|████████▌ | 2000/2344 [03:06<00:22, 15.02it/s]
Running loglikelihood requests:  88%|████████▊ | 2074/2344 [03:11<00:17, 15.48it/s]
Running loglikelihood requests:  92%|█████████▏| 2146/2344 [03:15<00:12, 15.76it/s]
Running loglikelihood requests:  94%|█████████▍| 2214/2344 [03:20<00:08, 15.76it/s]
Running loglikelihood requests:  98%|█████████▊| 2297/2344 [03:22<00:02, 19.19it/s]
Running loglikelihood requests: 100%|██████████| 2344/2344 [03:22<00:00, 11.58it/s]
n136-128-154:2247972:2249091 [1] NCCL INFO Channel 00/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2249091 [1] NCCL INFO Channel 01/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2249091 [1] NCCL INFO Channel 02/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2249091 [1] NCCL INFO Channel 03/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2249091 [1] NCCL INFO Channel 04/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2249091 [1] NCCL INFO Channel 05/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2249091 [1] NCCL INFO Channel 06/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2249091 [1] NCCL INFO Channel 07/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2249091 [1] NCCL INFO Channel 08/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2249091 [1] NCCL INFO Channel 09/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2249091 [1] NCCL INFO Channel 10/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2249091 [1] NCCL INFO Channel 11/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2249091 [1] NCCL INFO Channel 12/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2249091 [1] NCCL INFO Channel 13/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2249091 [1] NCCL INFO Channel 14/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2249091 [1] NCCL INFO Channel 15/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2249091 [1] NCCL INFO Channel 16/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2249091 [1] NCCL INFO Channel 17/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2249091 [1] NCCL INFO Channel 18/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2249091 [1] NCCL INFO Channel 19/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2249091 [1] NCCL INFO Channel 20/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2249091 [1] NCCL INFO Channel 21/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2249091 [1] NCCL INFO Channel 22/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2249091 [1] NCCL INFO Channel 23/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2249091 [1] NCCL INFO Channel 24/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2249091 [1] NCCL INFO Channel 25/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2249091 [1] NCCL INFO Channel 26/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2249091 [1] NCCL INFO Channel 27/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2249091 [1] NCCL INFO Channel 28/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2249091 [1] NCCL INFO Channel 29/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2249091 [1] NCCL INFO Channel 30/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2247972:2249091 [1] NCCL INFO Channel 31/1 : 1[7] -> 0[6] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:2247972:2249096 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2247972:2249096 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2247972:2249096 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2247972:2249096 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2247972:2249096 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2247972:2249096 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2247972:2248499 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-09:20:04:57 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 25, batch_size: auto (64)
|    Tasks    |Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|-------------|------:|------|-----:|--------|---|-----:|---|-----:|
|arc_challenge|      1|none  |    25|acc     |↑  |0.3089|±  |0.0135|
|             |       |none  |    25|acc_norm|↑  |0.3439|±  |0.0139|

[rank0]:[W1209 20:04:58.604574678 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:2247970:2249190 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2247970:2249190 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2247970:2249190 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2247970:2248497 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2247970:2249190 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2247970:2249190 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2247970:2249190 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2247972:2248499 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2247972:2249096 [1] NCCL INFO comm 0x112e94a0 rank 1 nranks 2 cudaDev 1 busId c9000 - Abort COMPLETE
n136-128-154:2247970:2249190 [0] NCCL INFO comm 0x127dd0e0 rank 0 nranks 2 cudaDev 0 busId c5000 - Abort COMPLETE
任务 arc_challenge 评估完成！

==================================================
开始评估：任务=truthfulqa_mc1 | 少样本数=0 | 模型=Llama-3.1-8B-quantization-fg
输出路径：results2/Llama-3.1-8B-quantization-fg/fg6/truthfulqa_mc1.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-09:20:05:48 INFO     [__main__:440] Selected Tasks: ['truthfulqa_mc1']
2025-12-09:20:05:48 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:20:05:48 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-09:20:05:48 INFO     [__main__:440] Selected Tasks: ['truthfulqa_mc1']
2025-12-09:20:05:48 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:20:05:48 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-09:20:05:49 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:20:05:49 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:20:05:49 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.16it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.16it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.13it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.13it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.14it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.59it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.39it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.10it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.55it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.36it/s]
2025-12-09:20:06:11 INFO     [evaluator:305] num_fewshot has been set to 0 for truthfulqa_mc1 in its config. Manual configuration will be ignored.
2025-12-09:20:06:11 INFO     [api.task:434] Building contexts for truthfulqa_mc1 on rank 0...

  0%|          | 0/409 [00:00<?, ?it/s]
 23%|██▎       | 95/409 [00:00<00:00, 942.96it/s]
 47%|████▋     | 194/409 [00:00<00:00, 969.60it/s]
 71%|███████▏  | 292/409 [00:00<00:00, 971.29it/s]
 96%|█████████▌| 393/409 [00:00<00:00, 984.82it/s]
100%|██████████| 409/409 [00:00<00:00, 977.69it/s]
n136-128-154:2249344:2249344 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2249344:2249344 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2249344:2249344 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:2249344:2249344 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2249344:2249344 [0] NCCL INFO Comm config Blocking set to 1
n136-128-154:2249344:2249774 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2249344:2249774 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2249344:2249774 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2249344:2249774 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2249344:2249774 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2249344:2249774 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2249344:2249774 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2249344:2249774 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:2249344:2249774 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2249344:2249774 [0] NCCL INFO Using network IB
n136-128-154:2249344:2249774 [0] NCCL INFO ncclCommInitRankConfig comm 0x12ab5e20 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0xdc5d9a9594c2fed0 - Init START
2025-12-09:20:06:17 INFO     [evaluator:305] num_fewshot has been set to 0 for truthfulqa_mc1 in its config. Manual configuration will be ignored.
2025-12-09:20:06:17 INFO     [api.task:434] Building contexts for truthfulqa_mc1 on rank 1...

  0%|          | 0/408 [00:00<?, ?it/s]
 24%|██▍       | 99/408 [00:00<00:00, 988.67it/s]
 49%|████▊     | 198/408 [00:00<00:00, 974.06it/s]
 73%|███████▎  | 299/408 [00:00<00:00, 987.02it/s]
 98%|█████████▊| 400/408 [00:00<00:00, 993.86it/s]
100%|██████████| 408/408 [00:00<00:00, 989.65it/s]
n136-128-154:2249345:2249345 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:2249345:2249345 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2249345:2249345 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2249345:2249345 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2249345:2249345 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:2249345:2249848 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2249345:2249848 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2249345:2249848 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2249345:2249848 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2249345:2249848 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2249345:2249848 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2249345:2249848 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2249345:2249848 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:2249345:2249848 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2249345:2249848 [1] NCCL INFO Using network IB
n136-128-154:2249345:2249848 [1] NCCL INFO ncclCommInitRankConfig comm 0x151d1bb0 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0xdc5d9a9594c2fed0 - Init START
n136-128-154:2249345:2249848 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2249344:2249774 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2249344:2249774 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2249344:2249774 [0] NCCL INFO Retrieving state for IB
n136-128-154:2249344:2249774 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:2249344:2249774 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2249345:2249848 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2249345:2249848 [1] NCCL INFO Retrieving state for IB
n136-128-154:2249345:2249848 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:2249344:2249774 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2249345:2249848 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2249344:2249774 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2249345:2249848 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2249344:2249774 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2249345:2249848 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2249345:2249848 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2249345:2249848 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2249345:2249848 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2249344:2249774 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2249344:2249774 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2249345:2249848 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2249345:2249848 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2249345:2249848 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2249345:2249848 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2249345:2249848 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2249345:2249848 [1] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2249345:2249848 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2249345:2249848 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2249345:2249848 [1] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2249345:2249848 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2249345:2249848 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2249345:2249848 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2249345:2249848 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2249345:2249848 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2249345:2249848 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2249345:2249848 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2249345:2249848 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2249345:2249848 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2249345:2249848 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2249345:2249848 [1] NCCL INFO ==========================================
n136-128-154:2249345:2249848 [1] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2249345:2249848 [1] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2249345:2249848 [1] NCCL INFO Setting affinity for GPU 7 to 33-62,97-126
n136-128-154:2249345:2249848 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2249345:2249848 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2249345:2249848 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2249345:2249848 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2249345:2249848 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2249345:2249848 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2249345:2249848 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2249345:2249848 [1] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2249345:2249848 [1] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2249345:2249848 [1] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2249345:2249848 [1] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2249345:2249848 [1] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2249345:2249848 [1] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2249345:2249848 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2249345:2249848 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2249345:2249848 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2249345:2249848 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2249345:2249848 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2249345:2249848 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2249345:2249848 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2249345:2249848 [1] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2249345:2249848 [1] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2249345:2249848 [1] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2249345:2249848 [1] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2249345:2249848 [1] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2249345:2249848 [1] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2249344:2249774 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2249344:2249774 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2249344:2249774 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2249344:2249774 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2249344:2249774 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2249344:2249774 [0] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2249344:2249774 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2249344:2249774 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2249344:2249774 [0] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2249344:2249774 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2249344:2249774 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2249344:2249774 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2249344:2249774 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2249344:2249774 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2249344:2249774 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2249344:2249774 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2249344:2249774 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2249344:2249774 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2249344:2249774 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2249344:2249774 [0] NCCL INFO ==========================================
n136-128-154:2249344:2249774 [0] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2249344:2249774 [0] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2249344:2249774 [0] NCCL INFO Setting affinity for GPU 6 to 33-62,97-126
n136-128-154:2249344:2249774 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2249344:2249774 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2249344:2249774 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2249344:2249774 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2249344:2249774 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2249344:2249774 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2249344:2249774 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2249344:2249774 [0] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2249344:2249774 [0] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2249344:2249774 [0] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2249344:2249774 [0] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2249344:2249774 [0] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2249344:2249774 [0] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2249344:2249774 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2249344:2249774 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2249344:2249774 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2249344:2249774 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2249344:2249774 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2249344:2249774 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2249344:2249774 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2249344:2249774 [0] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2249344:2249774 [0] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2249344:2249774 [0] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2249344:2249774 [0] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2249344:2249774 [0] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2249344:2249774 [0] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2249345:2249848 [1] NCCL INFO comm 0x151d1bb0 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:2249344:2249774 [0] NCCL INFO comm 0x12ab5e20 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:2249345:2249848 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:2249344:2249774 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:2249345:2249848 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:2249344:2249774 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:2249345:2249848 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:2249344:2249774 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:2249345:2249848 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:2249344:2249774 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:2249345:2249848 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:2249344:2249774 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:2249345:2249848 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:2249344:2249774 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:2249345:2249848 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:2249344:2249774 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:2249345:2249848 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:2249344:2249774 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:2249345:2249848 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:2249344:2249774 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:2249345:2249848 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:2249344:2249774 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:2249345:2249848 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:2249344:2249774 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:2249345:2249848 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:2249344:2249774 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:2249345:2249848 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:2249344:2249774 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:2249345:2249848 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:2249344:2249774 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:2249345:2249848 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:2249344:2249774 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:2249345:2249848 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:2249344:2249774 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:2249345:2249848 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:2249344:2249774 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:2249345:2249848 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:2249344:2249774 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:2249345:2249848 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:2249344:2249774 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:2249345:2249848 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:2249344:2249774 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:2249345:2249848 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:2249344:2249774 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:2249345:2249848 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:2249344:2249774 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:2249345:2249848 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:2249344:2249774 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:2249345:2249848 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:2249344:2249774 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:2249344:2249774 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:2249345:2249848 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:2249344:2249774 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:2249345:2249848 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:2249344:2249774 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:2249345:2249848 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:2249344:2249774 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:2249345:2249848 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:2249344:2249774 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:2249345:2249848 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:2249344:2249774 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:2249345:2249848 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:2249344:2249774 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:2249345:2249848 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:2249344:2249774 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:2249345:2249848 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:2249344:2249774 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:2249345:2249848 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:2249344:2249774 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:2249345:2249848 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:2249344:2249774 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:2249345:2249848 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:2249344:2249774 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:2249345:2249848 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:2249344:2249774 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:2249345:2249848 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:2249344:2249774 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:2249345:2249848 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:2249344:2249774 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:2249345:2249848 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:2249344:2249774 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:2249345:2249848 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:2249344:2249774 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:2249345:2249848 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:2249344:2249774 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:2249345:2249848 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:2249344:2249774 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:2249345:2249848 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:2249344:2249774 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:2249345:2249848 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:2249344:2249774 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:2249345:2249848 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:2249344:2249774 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:2249345:2249848 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:2249344:2249774 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:2249345:2249848 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:2249344:2249774 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:2249345:2249848 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:2249345:2249848 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:2249344:2249774 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:2249344:2249774 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:2249345:2249848 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2249344:2249774 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:2249344:2249774 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:2249344:2249774 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:2249344:2249774 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:2249344:2249774 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:2249344:2249774 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:2249344:2249774 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:2249344:2249774 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:2249344:2249774 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:2249344:2249774 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:2249344:2249774 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:2249344:2249774 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:2249344:2249774 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:2249344:2249774 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:2249344:2249774 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:2249344:2249774 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:2249344:2249774 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:2249344:2249774 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:2249344:2249774 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:2249344:2249774 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:2249344:2249774 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:2249344:2249774 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:2249344:2249774 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:2249344:2249774 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2249345:2249848 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2249345:2249857 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 103
n136-128-154:2249345:2249856 [1] NCCL INFO [Proxy Service] Device 1 CPU core 101
n136-128-154:2249344:2249774 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2249344:2249774 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:2249344:2249858 [0] NCCL INFO [Proxy Service] Device 0 CPU core 104
n136-128-154:2249344:2249859 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 41
n136-128-154:2249345:2249848 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2249345:2249848 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2249344:2249774 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2249344:2249774 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2249344:2249774 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:2249345:2249848 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2249345:2249848 [1] NCCL INFO ncclCommInitRankConfig comm 0x151d1bb0 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0xdc5d9a9594c2fed0 - Init COMPLETE
n136-128-154:2249345:2249848 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.47 (kernels 0.13, alloc 0.18, bootstrap 0.00, allgathers 0.01, topo 0.09, graphs 0.00, connections 0.02, rest 0.04)
n136-128-154:2249344:2249774 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2249344:2249774 [0] NCCL INFO ncclCommInitRankConfig comm 0x12ab5e20 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0xdc5d9a9594c2fed0 - Init COMPLETE
n136-128-154:2249344:2249774 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 6.41 (kernels 0.12, alloc 0.17, bootstrap 5.96, allgathers 0.01, topo 0.09, graphs 0.00, connections 0.02, rest 0.05)
n136-128-154:2249345:2249860 [1] NCCL INFO Channel 00/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249860 [1] NCCL INFO Channel 01/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249860 [1] NCCL INFO Channel 02/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249860 [1] NCCL INFO Channel 03/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249860 [1] NCCL INFO Channel 04/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249860 [1] NCCL INFO Channel 05/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249860 [1] NCCL INFO Channel 06/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249860 [1] NCCL INFO Channel 07/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249860 [1] NCCL INFO Channel 08/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249344:2249861 [0] NCCL INFO Channel 00/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2249345:2249860 [1] NCCL INFO Channel 09/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249344:2249861 [0] NCCL INFO Channel 01/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2249345:2249860 [1] NCCL INFO Channel 10/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249344:2249861 [0] NCCL INFO Channel 02/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2249345:2249860 [1] NCCL INFO Channel 11/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249344:2249861 [0] NCCL INFO Channel 03/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2249345:2249860 [1] NCCL INFO Channel 12/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249344:2249861 [0] NCCL INFO Channel 04/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2249345:2249860 [1] NCCL INFO Channel 13/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249344:2249861 [0] NCCL INFO Channel 05/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2249345:2249860 [1] NCCL INFO Channel 14/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249344:2249861 [0] NCCL INFO Channel 06/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2249345:2249860 [1] NCCL INFO Channel 15/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249344:2249861 [0] NCCL INFO Channel 07/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2249345:2249860 [1] NCCL INFO Channel 16/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249344:2249861 [0] NCCL INFO Channel 08/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2249345:2249860 [1] NCCL INFO Channel 17/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249344:2249861 [0] NCCL INFO Channel 09/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2249345:2249860 [1] NCCL INFO Channel 18/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249344:2249861 [0] NCCL INFO Channel 10/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2249345:2249860 [1] NCCL INFO Channel 19/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249344:2249861 [0] NCCL INFO Channel 11/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2249345:2249860 [1] NCCL INFO Channel 20/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249344:2249861 [0] NCCL INFO Channel 12/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2249345:2249860 [1] NCCL INFO Channel 21/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249344:2249861 [0] NCCL INFO Channel 13/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2249345:2249860 [1] NCCL INFO Channel 22/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249344:2249861 [0] NCCL INFO Channel 14/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2249345:2249860 [1] NCCL INFO Channel 23/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249344:2249861 [0] NCCL INFO Channel 15/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2249344:2249861 [0] NCCL INFO Channel 16/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2249344:2249861 [0] NCCL INFO Channel 17/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2249344:2249861 [0] NCCL INFO Channel 18/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2249344:2249861 [0] NCCL INFO Channel 19/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2249344:2249861 [0] NCCL INFO Channel 20/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2249344:2249861 [0] NCCL INFO Channel 21/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2249344:2249861 [0] NCCL INFO Channel 22/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2249344:2249861 [0] NCCL INFO Channel 23/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2249345:2249860 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:2249344:2249861 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-09:20:06:18 INFO     [evaluator:559] Running loglikelihood requests
2025-12-09:20:06:18 INFO     [evaluator:559] Running loglikelihood requests

Running loglikelihood requests:   0%|          | 0/2066 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 64
Determined largest batch size: 64

Running loglikelihood requests:   0%|          | 1/2066 [00:07<4:10:50,  7.29s/it]
Running loglikelihood requests:   3%|▎         | 65/2066 [00:08<03:15, 10.21it/s] 
Running loglikelihood requests:   6%|▌         | 129/2066 [00:09<01:39, 19.46it/s]
Running loglikelihood requests:   9%|▉         | 193/2066 [00:10<01:07, 27.55it/s]
Running loglikelihood requests:  12%|█▏        | 257/2066 [00:12<00:52, 34.32it/s]
Running loglikelihood requests:  16%|█▌        | 321/2066 [00:13<00:44, 39.51it/s]
Running loglikelihood requests:  19%|█▊        | 385/2066 [00:14<00:38, 43.82it/s]
Running loglikelihood requests:  22%|██▏       | 449/2066 [00:15<00:34, 47.09it/s]
Running loglikelihood requests:  25%|██▍       | 513/2066 [00:16<00:31, 49.43it/s]
Running loglikelihood requests:  28%|██▊       | 577/2066 [00:17<00:28, 51.54it/s]
Running loglikelihood requests:  31%|███       | 641/2066 [00:18<00:27, 52.49it/s]
Running loglikelihood requests:  34%|███▍      | 705/2066 [00:20<00:25, 53.45it/s]
Running loglikelihood requests:  37%|███▋      | 769/2066 [00:21<00:23, 54.59it/s]
Running loglikelihood requests:  40%|████      | 833/2066 [00:22<00:22, 55.47it/s]
Running loglikelihood requests:  43%|████▎     | 897/2066 [00:23<00:20, 56.11it/s]
Running loglikelihood requests:  47%|████▋     | 961/2066 [00:24<00:19, 56.63it/s]
Running loglikelihood requests:  50%|████▉     | 1025/2066 [00:25<00:18, 57.04it/s]
Running loglikelihood requests:  53%|█████▎    | 1089/2066 [00:26<00:17, 57.35it/s]
Running loglikelihood requests:  56%|█████▌    | 1153/2066 [00:27<00:15, 57.59it/s]
Running loglikelihood requests:  59%|█████▉    | 1217/2066 [00:28<00:14, 57.86it/s]
Running loglikelihood requests:  62%|██████▏   | 1281/2066 [00:30<00:13, 57.96it/s]
Running loglikelihood requests:  65%|██████▌   | 1345/2066 [00:31<00:12, 58.07it/s]
Running loglikelihood requests:  68%|██████▊   | 1409/2066 [00:32<00:11, 58.21it/s]
Running loglikelihood requests:  71%|███████▏  | 1473/2066 [00:33<00:10, 58.28it/s]
Running loglikelihood requests:  74%|███████▍  | 1537/2066 [00:34<00:09, 58.45it/s]
Running loglikelihood requests:  77%|███████▋  | 1601/2066 [00:35<00:07, 58.55it/s]
Running loglikelihood requests:  81%|████████  | 1665/2066 [00:36<00:06, 58.70it/s]
Running loglikelihood requests:  84%|████████▎ | 1729/2066 [00:37<00:05, 59.09it/s]
Running loglikelihood requests:  87%|████████▋ | 1793/2066 [00:38<00:04, 59.41it/s]
Running loglikelihood requests:  90%|████████▉ | 1857/2066 [00:39<00:03, 59.59it/s]
Running loglikelihood requests:  93%|█████████▎| 1921/2066 [00:40<00:02, 59.85it/s]
Running loglikelihood requests:  96%|█████████▌| 1985/2066 [00:41<00:01, 60.18it/s]
Running loglikelihood requests:  99%|█████████▉| 2049/2066 [00:42<00:00, 76.41it/s]
Running loglikelihood requests: 100%|██████████| 2066/2066 [00:42<00:00, 48.93it/s]
n136-128-154:2249345:2249988 [1] NCCL INFO Channel 00/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249988 [1] NCCL INFO Channel 01/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249988 [1] NCCL INFO Channel 02/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249988 [1] NCCL INFO Channel 03/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249988 [1] NCCL INFO Channel 04/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249988 [1] NCCL INFO Channel 05/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249988 [1] NCCL INFO Channel 06/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249988 [1] NCCL INFO Channel 07/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249988 [1] NCCL INFO Channel 08/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249988 [1] NCCL INFO Channel 09/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249988 [1] NCCL INFO Channel 10/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249988 [1] NCCL INFO Channel 11/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249988 [1] NCCL INFO Channel 12/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249988 [1] NCCL INFO Channel 13/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249988 [1] NCCL INFO Channel 14/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249988 [1] NCCL INFO Channel 15/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249988 [1] NCCL INFO Channel 16/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249988 [1] NCCL INFO Channel 17/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249988 [1] NCCL INFO Channel 18/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249988 [1] NCCL INFO Channel 19/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249988 [1] NCCL INFO Channel 20/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249988 [1] NCCL INFO Channel 21/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249988 [1] NCCL INFO Channel 22/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249988 [1] NCCL INFO Channel 23/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249988 [1] NCCL INFO Channel 24/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249988 [1] NCCL INFO Channel 25/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249988 [1] NCCL INFO Channel 26/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249988 [1] NCCL INFO Channel 27/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249988 [1] NCCL INFO Channel 28/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249988 [1] NCCL INFO Channel 29/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249988 [1] NCCL INFO Channel 30/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2249345:2249988 [1] NCCL INFO Channel 31/1 : 1[7] -> 0[6] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:2249345:2249994 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2249345:2249994 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2249345:2249994 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2249345:2249994 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2249345:2249994 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2249345:2249994 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2249345:2249856 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-09:20:07:09 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: auto (64)
|    Tasks     |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|--------------|------:|------|-----:|------|---|-----:|---|-----:|
|truthfulqa_mc1|      2|none  |     0|acc   |↑  |0.2448|±  |0.0151|

[rank0]:[W1209 20:07:10.511964655 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:2249344:2250056 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2249344:2250056 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2249344:2250056 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2249344:2249858 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2249344:2250056 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2249344:2250056 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2249344:2250056 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2249345:2249856 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2249345:2249994 [1] NCCL INFO comm 0x151d1bb0 rank 1 nranks 2 cudaDev 1 busId c9000 - Abort COMPLETE
n136-128-154:2249344:2250056 [0] NCCL INFO comm 0x12ab5e20 rank 0 nranks 2 cudaDev 0 busId c5000 - Abort COMPLETE
任务 truthfulqa_mc1 评估完成！

==================================================
开始评估：任务=piqa | 少样本数=0 | 模型=Llama-3.1-8B-quantization-fg
输出路径：results2/Llama-3.1-8B-quantization-fg/fg6/piqa.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-09:20:08:00 INFO     [__main__:440] Selected Tasks: ['piqa']
2025-12-09:20:08:00 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:20:08:00 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-09:20:08:00 INFO     [__main__:440] Selected Tasks: ['piqa']
2025-12-09:20:08:00 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:20:08:00 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-09:20:08:01 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:20:08:01 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:20:08:02 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.15it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.14it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.12it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.11it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.13it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.60it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.39it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.13it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.60it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.38it/s]
2025-12-09:20:08:21 WARNING  [evaluator:309] Overwriting default num_fewshot of piqa from None to 0
2025-12-09:20:08:21 INFO     [api.task:434] Building contexts for piqa on rank 1...

  0%|          | 0/919 [00:00<?, ?it/s]
 16%|█▌        | 147/919 [00:00<00:00, 1463.38it/s]
 32%|███▏      | 298/919 [00:00<00:00, 1486.30it/s]
 49%|████▉     | 450/919 [00:00<00:00, 1500.94it/s]
 66%|██████▌   | 607/919 [00:00<00:00, 1525.50it/s]
 83%|████████▎ | 763/919 [00:00<00:00, 1534.90it/s]
100%|█████████▉| 917/919 [00:00<00:00, 1492.10it/s]
100%|██████████| 919/919 [00:00<00:00, 1499.69it/s]
2025-12-09:20:08:27 WARNING  [evaluator:309] Overwriting default num_fewshot of piqa from None to 0
2025-12-09:20:08:27 INFO     [api.task:434] Building contexts for piqa on rank 0...

  0%|          | 0/919 [00:00<?, ?it/s]
 14%|█▍        | 133/919 [00:00<00:00, 1324.76it/s]
 31%|███       | 281/919 [00:00<00:00, 1411.41it/s]
 47%|████▋     | 432/919 [00:00<00:00, 1454.63it/s]
 63%|██████▎   | 582/919 [00:00<00:00, 1470.08it/s]
 80%|███████▉  | 731/919 [00:00<00:00, 1476.75it/s]
 96%|█████████▌| 879/919 [00:00<00:00, 1472.78it/s]
100%|██████████| 919/919 [00:00<00:00, 1459.53it/s]
n136-128-154:2250247:2250247 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2250247:2250247 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2250247:2250247 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:2250247:2250247 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2250247:2250247 [0] NCCL INFO Comm config Blocking set to 1
n136-128-154:2250248:2250248 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:2250248:2250248 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2250248:2250248 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2250248:2250248 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2250248:2250248 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:2250247:2250742 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2250247:2250742 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2250248:2250743 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2250248:2250743 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2250247:2250742 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2250247:2250742 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2250247:2250742 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2250248:2250743 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2250248:2250743 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2250248:2250743 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2250248:2250743 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2250248:2250743 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2250248:2250743 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:2250248:2250743 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2250248:2250743 [1] NCCL INFO Using network IB
n136-128-154:2250247:2250742 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2250247:2250742 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2250247:2250742 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:2250247:2250742 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2250247:2250742 [0] NCCL INFO Using network IB
n136-128-154:2250248:2250743 [1] NCCL INFO ncclCommInitRankConfig comm 0x14b39720 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0x6c6b8b0590fd8c7a - Init START
n136-128-154:2250247:2250742 [0] NCCL INFO ncclCommInitRankConfig comm 0x167514c0 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0x6c6b8b0590fd8c7a - Init START
n136-128-154:2250247:2250742 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2250248:2250743 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2250248:2250743 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2250248:2250743 [1] NCCL INFO Retrieving state for IB
n136-128-154:2250248:2250743 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:2250248:2250743 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2250247:2250742 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2250247:2250742 [0] NCCL INFO Retrieving state for IB
n136-128-154:2250247:2250742 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:2250248:2250743 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2250247:2250742 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2250248:2250743 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2250247:2250742 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2250248:2250743 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2250247:2250742 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2250247:2250742 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2250248:2250743 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2250248:2250743 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2250247:2250742 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2250247:2250742 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2250248:2250743 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2250248:2250743 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2250248:2250743 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2250248:2250743 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2250248:2250743 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2250248:2250743 [1] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2250248:2250743 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2250248:2250743 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2250248:2250743 [1] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2250248:2250743 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2250248:2250743 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2250248:2250743 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2250248:2250743 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2250248:2250743 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2250248:2250743 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2250248:2250743 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2250248:2250743 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2250248:2250743 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2250248:2250743 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2250248:2250743 [1] NCCL INFO ==========================================
n136-128-154:2250248:2250743 [1] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2250248:2250743 [1] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2250248:2250743 [1] NCCL INFO Setting affinity for GPU 7 to 33-62,97-126
n136-128-154:2250247:2250742 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2250247:2250742 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2250247:2250742 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2250247:2250742 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2250247:2250742 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2250247:2250742 [0] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2250247:2250742 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2250247:2250742 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2250247:2250742 [0] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2250247:2250742 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2250247:2250742 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2250247:2250742 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2250247:2250742 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2250247:2250742 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2250247:2250742 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2250247:2250742 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2250247:2250742 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2250247:2250742 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2250247:2250742 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2250247:2250742 [0] NCCL INFO ==========================================
n136-128-154:2250247:2250742 [0] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2250247:2250742 [0] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2250247:2250742 [0] NCCL INFO Setting affinity for GPU 6 to 33-62,97-126
n136-128-154:2250247:2250742 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2250247:2250742 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250247:2250742 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250247:2250742 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250247:2250742 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250247:2250742 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250247:2250742 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250247:2250742 [0] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250247:2250742 [0] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250247:2250742 [0] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250247:2250742 [0] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250247:2250742 [0] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250247:2250742 [0] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250247:2250742 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2250247:2250742 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250247:2250742 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250247:2250742 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250247:2250742 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250247:2250742 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250247:2250742 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250247:2250742 [0] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2250247:2250742 [0] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2250247:2250742 [0] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2250247:2250742 [0] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2250247:2250742 [0] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2250247:2250742 [0] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2250248:2250743 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2250248:2250743 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250248:2250743 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250248:2250743 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250248:2250743 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250248:2250743 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250248:2250743 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250248:2250743 [1] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250248:2250743 [1] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250248:2250743 [1] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250248:2250743 [1] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250248:2250743 [1] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250248:2250743 [1] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250248:2250743 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2250248:2250743 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250248:2250743 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250248:2250743 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250248:2250743 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250248:2250743 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250248:2250743 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250248:2250743 [1] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2250248:2250743 [1] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2250248:2250743 [1] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2250248:2250743 [1] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2250248:2250743 [1] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2250248:2250743 [1] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2250248:2250743 [1] NCCL INFO comm 0x14b39720 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:2250248:2250743 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:2250247:2250742 [0] NCCL INFO comm 0x167514c0 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:2250248:2250743 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:2250248:2250743 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:2250248:2250743 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:2250248:2250743 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:2250248:2250743 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:2250248:2250743 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:2250247:2250742 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:2250248:2250743 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:2250247:2250742 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:2250248:2250743 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:2250247:2250742 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:2250248:2250743 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:2250247:2250742 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:2250248:2250743 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:2250247:2250742 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:2250248:2250743 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:2250247:2250742 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:2250248:2250743 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:2250247:2250742 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:2250248:2250743 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:2250247:2250742 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:2250248:2250743 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:2250247:2250742 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:2250248:2250743 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:2250247:2250742 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:2250248:2250743 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:2250247:2250742 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:2250248:2250743 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:2250247:2250742 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:2250248:2250743 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:2250247:2250742 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:2250248:2250743 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:2250247:2250742 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:2250248:2250743 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:2250247:2250742 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:2250248:2250743 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:2250247:2250742 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:2250248:2250743 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:2250247:2250742 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:2250248:2250743 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:2250247:2250742 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:2250247:2250742 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:2250247:2250742 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:2250247:2250742 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:2250247:2250742 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:2250247:2250742 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:2250247:2250742 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:2250248:2250743 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:2250248:2250743 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:2250248:2250743 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:2250248:2250743 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:2250248:2250743 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:2250247:2250742 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:2250248:2250743 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:2250247:2250742 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:2250248:2250743 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:2250247:2250742 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:2250248:2250743 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:2250247:2250742 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:2250248:2250743 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:2250247:2250742 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:2250248:2250743 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:2250247:2250742 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:2250248:2250743 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:2250247:2250742 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:2250248:2250743 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:2250247:2250742 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:2250248:2250743 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:2250247:2250742 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:2250248:2250743 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:2250247:2250742 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:2250248:2250743 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:2250247:2250742 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:2250248:2250743 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:2250247:2250742 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:2250248:2250743 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:2250247:2250742 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:2250248:2250743 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:2250247:2250742 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:2250248:2250743 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:2250247:2250742 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:2250248:2250743 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:2250247:2250742 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:2250248:2250743 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:2250247:2250742 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:2250248:2250743 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:2250247:2250742 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:2250248:2250743 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:2250247:2250742 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:2250248:2250743 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:2250247:2250742 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:2250248:2250743 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:2250247:2250742 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:2250247:2250742 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:2250248:2250743 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2250247:2250742 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:2250247:2250742 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:2250247:2250742 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:2250247:2250742 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:2250247:2250742 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:2250247:2250742 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:2250247:2250742 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:2250247:2250742 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:2250247:2250742 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:2250247:2250742 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:2250247:2250742 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:2250247:2250742 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:2250247:2250742 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:2250247:2250742 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:2250247:2250742 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:2250247:2250742 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:2250247:2250742 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:2250247:2250742 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:2250247:2250742 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:2250247:2250742 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:2250247:2250742 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:2250247:2250742 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:2250247:2250742 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:2250247:2250742 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:2250247:2250742 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:2250247:2250742 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:2250247:2250742 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:2250247:2250742 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2250247:2250742 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2250247:2250742 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:2250247:2250755 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 104
n136-128-154:2250247:2250754 [0] NCCL INFO [Proxy Service] Device 0 CPU core 39
n136-128-154:2250248:2250743 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2250248:2250756 [1] NCCL INFO [Proxy Service] Device 1 CPU core 34
n136-128-154:2250248:2250757 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 41
n136-128-154:2250247:2250742 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2250247:2250742 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2250248:2250743 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2250248:2250743 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2250247:2250742 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:2250247:2250742 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2250247:2250742 [0] NCCL INFO ncclCommInitRankConfig comm 0x167514c0 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0x6c6b8b0590fd8c7a - Init COMPLETE
n136-128-154:2250247:2250742 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.52 (kernels 0.12, alloc 0.19, bootstrap 0.00, allgathers 0.01, topo 0.08, graphs 0.00, connections 0.06, rest 0.06)
n136-128-154:2250248:2250743 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2250248:2250743 [1] NCCL INFO ncclCommInitRankConfig comm 0x14b39720 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0x6c6b8b0590fd8c7a - Init COMPLETE
n136-128-154:2250248:2250743 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.51 (kernels 0.12, alloc 0.17, bootstrap 0.00, allgathers 0.01, topo 0.08, graphs 0.00, connections 0.06, rest 0.06)
n136-128-154:2250247:2250758 [0] NCCL INFO Channel 00/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250247:2250758 [0] NCCL INFO Channel 01/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250247:2250758 [0] NCCL INFO Channel 02/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250247:2250758 [0] NCCL INFO Channel 03/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250247:2250758 [0] NCCL INFO Channel 04/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250247:2250758 [0] NCCL INFO Channel 05/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250247:2250758 [0] NCCL INFO Channel 06/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250247:2250758 [0] NCCL INFO Channel 07/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250247:2250758 [0] NCCL INFO Channel 08/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250247:2250758 [0] NCCL INFO Channel 09/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250247:2250758 [0] NCCL INFO Channel 10/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250247:2250758 [0] NCCL INFO Channel 11/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250247:2250758 [0] NCCL INFO Channel 12/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250247:2250758 [0] NCCL INFO Channel 13/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250248:2250759 [1] NCCL INFO Channel 00/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250247:2250758 [0] NCCL INFO Channel 14/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250248:2250759 [1] NCCL INFO Channel 01/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250247:2250758 [0] NCCL INFO Channel 15/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250248:2250759 [1] NCCL INFO Channel 02/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250247:2250758 [0] NCCL INFO Channel 16/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250248:2250759 [1] NCCL INFO Channel 03/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250247:2250758 [0] NCCL INFO Channel 17/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250248:2250759 [1] NCCL INFO Channel 04/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250247:2250758 [0] NCCL INFO Channel 18/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250248:2250759 [1] NCCL INFO Channel 05/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250247:2250758 [0] NCCL INFO Channel 19/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250248:2250759 [1] NCCL INFO Channel 06/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250247:2250758 [0] NCCL INFO Channel 20/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250248:2250759 [1] NCCL INFO Channel 07/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250247:2250758 [0] NCCL INFO Channel 21/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250248:2250759 [1] NCCL INFO Channel 08/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250247:2250758 [0] NCCL INFO Channel 22/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250248:2250759 [1] NCCL INFO Channel 09/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250247:2250758 [0] NCCL INFO Channel 23/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250248:2250759 [1] NCCL INFO Channel 10/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250759 [1] NCCL INFO Channel 11/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250759 [1] NCCL INFO Channel 12/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250759 [1] NCCL INFO Channel 13/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250759 [1] NCCL INFO Channel 14/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250759 [1] NCCL INFO Channel 15/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250759 [1] NCCL INFO Channel 16/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250759 [1] NCCL INFO Channel 17/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250759 [1] NCCL INFO Channel 18/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250759 [1] NCCL INFO Channel 19/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250759 [1] NCCL INFO Channel 20/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250759 [1] NCCL INFO Channel 21/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250759 [1] NCCL INFO Channel 22/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250759 [1] NCCL INFO Channel 23/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250247:2250758 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:2250248:2250759 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-09:20:08:28 INFO     [evaluator:559] Running loglikelihood requests
2025-12-09:20:08:28 INFO     [evaluator:559] Running loglikelihood requests
Passed argument batch_size = auto:1. Detecting largest batch size

Running loglikelihood requests:   0%|          | 0/1838 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 64
Determined largest batch size: 64

Running loglikelihood requests:   0%|          | 1/1838 [00:08<4:09:49,  8.16s/it]
Running loglikelihood requests:   4%|▎         | 65/1838 [00:08<02:56, 10.02it/s] 
Running loglikelihood requests:   7%|▋         | 129/1838 [00:09<01:20, 21.20it/s]
Running loglikelihood requests:  11%|█         | 194/1838 [00:10<00:48, 33.81it/s]
Running loglikelihood requests:  14%|█▍        | 258/1838 [00:10<00:33, 47.07it/s]
Running loglikelihood requests:  18%|█▊        | 322/1838 [00:11<00:25, 60.61it/s]
Running loglikelihood requests:  21%|██        | 386/1838 [00:11<00:19, 74.12it/s]
Running loglikelihood requests:  25%|██▍       | 451/1838 [00:12<00:15, 87.04it/s]
Running loglikelihood requests:  28%|██▊       | 515/1838 [00:12<00:13, 98.42it/s]
Running loglikelihood requests:  32%|███▏      | 579/1838 [00:13<00:11, 108.94it/s]
Running loglikelihood requests:  35%|███▍      | 643/1838 [00:13<00:10, 117.79it/s]
Running loglikelihood requests:  38%|███▊      | 707/1838 [00:13<00:09, 124.72it/s]
Running loglikelihood requests:  42%|████▏     | 771/1838 [00:14<00:08, 130.07it/s]
Running loglikelihood requests:  45%|████▌     | 835/1838 [00:14<00:07, 136.03it/s]
Running loglikelihood requests:  49%|████▉     | 899/1838 [00:15<00:06, 141.15it/s]
Running loglikelihood requests:  52%|█████▏    | 964/1838 [00:15<00:05, 145.89it/s]
Running loglikelihood requests:  56%|█████▌    | 1028/1838 [00:16<00:05, 150.01it/s]
Running loglikelihood requests:  59%|█████▉    | 1092/1838 [00:16<00:04, 154.11it/s]
Running loglikelihood requests:  63%|██████▎   | 1156/1838 [00:16<00:04, 157.51it/s]
Running loglikelihood requests:  66%|██████▋   | 1221/1838 [00:17<00:03, 161.23it/s]
Running loglikelihood requests:  70%|██████▉   | 1286/1838 [00:17<00:03, 164.98it/s]
Running loglikelihood requests:  73%|███████▎  | 1350/1838 [00:17<00:02, 167.67it/s]
Running loglikelihood requests:  77%|███████▋  | 1414/1838 [00:18<00:02, 169.41it/s]
Running loglikelihood requests:  80%|████████  | 1478/1838 [00:18<00:02, 172.13it/s]
Running loglikelihood requests:  84%|████████▍ | 1542/1838 [00:19<00:01, 174.89it/s]
Running loglikelihood requests:  87%|████████▋ | 1606/1838 [00:19<00:01, 177.84it/s]
Running loglikelihood requests:  91%|█████████ | 1670/1838 [00:19<00:00, 180.97it/s]
Running loglikelihood requests:  94%|█████████▍| 1734/1838 [00:20<00:00, 185.24it/s]
Running loglikelihood requests:  98%|█████████▊| 1799/1838 [00:20<00:00, 208.00it/s]
Running loglikelihood requests: 100%|██████████| 1838/1838 [00:20<00:00, 90.78it/s] 
n136-128-154:2250248:2250834 [1] NCCL INFO Channel 00/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250834 [1] NCCL INFO Channel 01/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250834 [1] NCCL INFO Channel 02/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250834 [1] NCCL INFO Channel 03/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250834 [1] NCCL INFO Channel 04/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250834 [1] NCCL INFO Channel 05/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250834 [1] NCCL INFO Channel 06/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250834 [1] NCCL INFO Channel 07/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250834 [1] NCCL INFO Channel 08/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250834 [1] NCCL INFO Channel 09/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250834 [1] NCCL INFO Channel 10/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250834 [1] NCCL INFO Channel 11/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250834 [1] NCCL INFO Channel 12/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250834 [1] NCCL INFO Channel 13/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250834 [1] NCCL INFO Channel 14/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250834 [1] NCCL INFO Channel 15/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250834 [1] NCCL INFO Channel 16/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250834 [1] NCCL INFO Channel 17/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250834 [1] NCCL INFO Channel 18/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250834 [1] NCCL INFO Channel 19/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250834 [1] NCCL INFO Channel 20/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250834 [1] NCCL INFO Channel 21/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250834 [1] NCCL INFO Channel 22/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250834 [1] NCCL INFO Channel 23/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250834 [1] NCCL INFO Channel 24/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250834 [1] NCCL INFO Channel 25/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250834 [1] NCCL INFO Channel 26/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250834 [1] NCCL INFO Channel 27/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250834 [1] NCCL INFO Channel 28/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250834 [1] NCCL INFO Channel 29/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250834 [1] NCCL INFO Channel 30/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250248:2250834 [1] NCCL INFO Channel 31/1 : 1[7] -> 0[6] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:2250248:2250839 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2250248:2250839 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2250248:2250839 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2250248:2250839 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2250248:2250839 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2250248:2250839 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2250248:2250756 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-09:20:08:58 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: auto (64)
|Tasks|Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|-----|------:|------|-----:|--------|---|-----:|---|-----:|
|piqa |      1|none  |     0|acc     |↑  |0.6431|±  |0.0112|
|     |       |none  |     0|acc_norm|↑  |0.6458|±  |0.0112|

[rank0]:[W1209 20:08:59.922555932 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:2250247:2250886 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2250247:2250886 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2250247:2250754 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2250247:2250886 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2250247:2250886 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2250247:2250886 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2250247:2250886 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2250248:2250756 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2250248:2250839 [1] NCCL INFO comm 0x14b39720 rank 1 nranks 2 cudaDev 1 busId c9000 - Abort COMPLETE
n136-128-154:2250247:2250886 [0] NCCL INFO comm 0x167514c0 rank 0 nranks 2 cudaDev 0 busId c5000 - Abort COMPLETE
任务 piqa 评估完成！

==================================================
开始评估：任务=hellaswag | 少样本数=10 | 模型=Llama-3.1-8B-quantization-fg
输出路径：results2/Llama-3.1-8B-quantization-fg/fg6/hellaswag.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-09:20:09:49 INFO     [__main__:440] Selected Tasks: ['hellaswag']
2025-12-09:20:09:49 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:20:09:49 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-09:20:09:49 INFO     [__main__:440] Selected Tasks: ['hellaswag']
2025-12-09:20:09:49 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:20:09:49 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-09:20:09:50 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:20:09:51 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:20:09:51 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.09s/it]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.04s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.13s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.19s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.22s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.19s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.19it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.05it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.17it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.02it/s]
2025-12-09:20:10:16 WARNING  [evaluator:309] Overwriting default num_fewshot of hellaswag from None to 10
2025-12-09:20:10:16 INFO     [api.task:434] Building contexts for hellaswag on rank 1...
2025-12-09:20:10:17 WARNING  [evaluator:309] Overwriting default num_fewshot of hellaswag from None to 10
2025-12-09:20:10:17 INFO     [api.task:434] Building contexts for hellaswag on rank 0...

  0%|          | 0/5021 [00:00<?, ?it/s]
  0%|          | 19/5021 [00:00<00:26, 186.84it/s]
  1%|          | 39/5021 [00:00<00:25, 191.68it/s]
  1%|          | 59/5021 [00:00<00:25, 191.85it/s]
  2%|▏         | 79/5021 [00:00<00:26, 188.45it/s]
  2%|▏         | 99/5021 [00:00<00:25, 189.92it/s]
  2%|▏         | 119/5021 [00:00<00:25, 191.13it/s]
  3%|▎         | 139/5021 [00:00<00:25, 192.43it/s]
  3%|▎         | 159/5021 [00:00<00:25, 193.61it/s]
  4%|▎         | 179/5021 [00:00<00:24, 194.16it/s]
  4%|▍         | 199/5021 [00:01<00:24, 194.48it/s]
  0%|          | 0/5021 [00:00<?, ?it/s]
  4%|▍         | 219/5021 [00:01<00:24, 194.48it/s]
  0%|          | 19/5021 [00:00<00:27, 181.10it/s]
  5%|▍         | 239/5021 [00:01<00:24, 192.78it/s]
  1%|          | 38/5021 [00:00<00:26, 185.61it/s]
  5%|▌         | 259/5021 [00:01<00:24, 192.60it/s]
  1%|          | 58/5021 [00:00<00:26, 189.18it/s]
  6%|▌         | 279/5021 [00:01<00:24, 193.12it/s]
  2%|▏         | 78/5021 [00:00<00:25, 192.50it/s]
  6%|▌         | 299/5021 [00:01<00:24, 193.58it/s]
  2%|▏         | 98/5021 [00:00<00:25, 194.96it/s]
  6%|▋         | 319/5021 [00:01<00:24, 194.21it/s]
  2%|▏         | 119/5021 [00:00<00:24, 196.84it/s]
  7%|▋         | 339/5021 [00:01<00:24, 194.78it/s]
  3%|▎         | 139/5021 [00:00<00:25, 194.12it/s]
  7%|▋         | 359/5021 [00:01<00:23, 195.22it/s]
  3%|▎         | 159/5021 [00:00<00:25, 193.86it/s]
  8%|▊         | 379/5021 [00:01<00:23, 195.41it/s]
  4%|▎         | 179/5021 [00:00<00:24, 194.03it/s]
  8%|▊         | 399/5021 [00:02<00:23, 195.66it/s]
  4%|▍         | 199/5021 [00:01<00:24, 194.36it/s]
  8%|▊         | 419/5021 [00:02<00:23, 196.06it/s]
  4%|▍         | 219/5021 [00:01<00:24, 195.08it/s]
  9%|▊         | 439/5021 [00:02<00:23, 196.20it/s]
  5%|▍         | 239/5021 [00:01<00:24, 195.76it/s]
  9%|▉         | 459/5021 [00:02<00:23, 196.40it/s]
  5%|▌         | 259/5021 [00:01<00:24, 196.05it/s]
 10%|▉         | 479/5021 [00:02<00:23, 196.16it/s]
  6%|▌         | 279/5021 [00:01<00:24, 196.20it/s]
 10%|▉         | 499/5021 [00:02<00:22, 196.65it/s]
  6%|▌         | 299/5021 [00:01<00:23, 196.94it/s]
 10%|█         | 519/5021 [00:02<00:22, 196.67it/s]
  6%|▋         | 319/5021 [00:01<00:24, 195.61it/s]
 11%|█         | 539/5021 [00:02<00:22, 196.60it/s]
  7%|▋         | 339/5021 [00:01<00:23, 195.54it/s]
 11%|█         | 559/5021 [00:02<00:22, 196.36it/s]
  7%|▋         | 359/5021 [00:01<00:23, 196.44it/s]
 12%|█▏        | 579/5021 [00:02<00:22, 194.35it/s]
  8%|▊         | 379/5021 [00:01<00:23, 196.91it/s]
 12%|█▏        | 599/5021 [00:03<00:22, 194.88it/s]
  8%|▊         | 399/5021 [00:02<00:23, 197.05it/s]
 12%|█▏        | 619/5021 [00:03<00:22, 194.91it/s]
  8%|▊         | 419/5021 [00:02<00:23, 195.80it/s]
 13%|█▎        | 639/5021 [00:03<00:22, 194.95it/s]
  9%|▊         | 439/5021 [00:02<00:23, 195.65it/s]
 13%|█▎        | 659/5021 [00:03<00:22, 195.06it/s]
  9%|▉         | 459/5021 [00:02<00:23, 195.76it/s]
 14%|█▎        | 679/5021 [00:03<00:22, 195.43it/s]
 10%|▉         | 479/5021 [00:02<00:23, 196.23it/s]
 14%|█▍        | 699/5021 [00:03<00:22, 195.46it/s]
 10%|▉         | 499/5021 [00:02<00:23, 196.51it/s]
 14%|█▍        | 719/5021 [00:03<00:22, 195.45it/s]
 10%|█         | 519/5021 [00:02<00:22, 197.02it/s]
 15%|█▍        | 739/5021 [00:03<00:21, 195.51it/s]
 11%|█         | 539/5021 [00:02<00:22, 197.15it/s]
 15%|█▌        | 759/5021 [00:03<00:21, 194.95it/s]
 11%|█         | 559/5021 [00:02<00:22, 197.40it/s]
 16%|█▌        | 779/5021 [00:04<00:21, 194.92it/s]
 12%|█▏        | 579/5021 [00:02<00:22, 197.25it/s]
 16%|█▌        | 799/5021 [00:04<00:21, 194.95it/s]
 12%|█▏        | 599/5021 [00:03<00:22, 196.79it/s]
 16%|█▋        | 819/5021 [00:04<00:21, 194.80it/s]
 12%|█▏        | 619/5021 [00:03<00:22, 196.18it/s]
 17%|█▋        | 839/5021 [00:04<00:21, 194.64it/s]
 13%|█▎        | 639/5021 [00:03<00:22, 195.66it/s]
 17%|█▋        | 859/5021 [00:04<00:21, 194.30it/s]
 13%|█▎        | 659/5021 [00:03<00:22, 195.42it/s]
 18%|█▊        | 879/5021 [00:04<00:21, 194.72it/s]
 14%|█▎        | 679/5021 [00:03<00:22, 195.45it/s]
 18%|█▊        | 899/5021 [00:04<00:21, 192.20it/s]
 14%|█▍        | 699/5021 [00:03<00:22, 195.76it/s]
 18%|█▊        | 919/5021 [00:04<00:21, 191.87it/s]
 14%|█▍        | 719/5021 [00:03<00:21, 195.56it/s]
 19%|█▊        | 939/5021 [00:04<00:21, 192.14it/s]
 15%|█▍        | 739/5021 [00:03<00:21, 195.60it/s]
 19%|█▉        | 959/5021 [00:04<00:21, 193.00it/s]
 15%|█▌        | 759/5021 [00:03<00:21, 195.59it/s]
 19%|█▉        | 979/5021 [00:05<00:20, 193.33it/s]
 16%|█▌        | 779/5021 [00:03<00:21, 195.28it/s]
 20%|█▉        | 999/5021 [00:05<00:20, 192.56it/s]
 16%|█▌        | 799/5021 [00:04<00:21, 194.89it/s]
 20%|██        | 1019/5021 [00:05<00:20, 191.87it/s]
 16%|█▋        | 819/5021 [00:04<00:21, 194.82it/s]
 21%|██        | 1039/5021 [00:05<00:20, 191.35it/s]
 17%|█▋        | 839/5021 [00:04<00:21, 194.28it/s]
 21%|██        | 1059/5021 [00:05<00:20, 191.08it/s]
 17%|█▋        | 859/5021 [00:04<00:21, 194.13it/s]
 21%|██▏       | 1079/5021 [00:05<00:20, 190.94it/s]
 18%|█▊        | 879/5021 [00:04<00:21, 193.92it/s]
 22%|██▏       | 1099/5021 [00:05<00:20, 190.68it/s]
 18%|█▊        | 899/5021 [00:04<00:21, 190.70it/s]
 22%|██▏       | 1119/5021 [00:05<00:20, 190.20it/s]
 18%|█▊        | 919/5021 [00:04<00:21, 190.06it/s]
 23%|██▎       | 1139/5021 [00:05<00:20, 190.31it/s]
 19%|█▊        | 939/5021 [00:04<00:21, 190.34it/s]
 23%|██▎       | 1159/5021 [00:05<00:20, 190.67it/s]
 19%|█▉        | 959/5021 [00:04<00:21, 191.33it/s]
 23%|██▎       | 1179/5021 [00:06<00:20, 190.91it/s]
 19%|█▉        | 979/5021 [00:05<00:21, 191.87it/s]
 24%|██▍       | 1199/5021 [00:06<00:19, 191.95it/s]
 20%|█▉        | 999/5021 [00:05<00:20, 193.36it/s]
 24%|██▍       | 1219/5021 [00:06<00:19, 190.69it/s]
 20%|██        | 1019/5021 [00:05<00:20, 194.27it/s]
 25%|██▍       | 1239/5021 [00:06<00:19, 192.10it/s]
 21%|██        | 1039/5021 [00:05<00:20, 194.28it/s]
 25%|██▌       | 1259/5021 [00:06<00:19, 193.39it/s]
 21%|██        | 1059/5021 [00:05<00:20, 194.91it/s]
 25%|██▌       | 1279/5021 [00:06<00:19, 194.14it/s]
 21%|██▏       | 1079/5021 [00:05<00:20, 194.97it/s]
 26%|██▌       | 1299/5021 [00:06<00:19, 195.02it/s]
 22%|██▏       | 1099/5021 [00:05<00:20, 195.96it/s]
 26%|██▋       | 1319/5021 [00:06<00:18, 196.00it/s]
 22%|██▏       | 1119/5021 [00:05<00:19, 196.58it/s]
 27%|██▋       | 1339/5021 [00:06<00:18, 196.40it/s]
 23%|██▎       | 1139/5021 [00:05<00:19, 197.18it/s]
 27%|██▋       | 1359/5021 [00:07<00:18, 196.54it/s]
 23%|██▎       | 1159/5021 [00:05<00:19, 197.69it/s]
 27%|██▋       | 1379/5021 [00:07<00:18, 196.95it/s]
 23%|██▎       | 1179/5021 [00:06<00:19, 197.73it/s]
 28%|██▊       | 1399/5021 [00:07<00:18, 197.29it/s]
 24%|██▍       | 1199/5021 [00:06<00:19, 197.95it/s]
 28%|██▊       | 1419/5021 [00:07<00:18, 197.73it/s]
 24%|██▍       | 1219/5021 [00:06<00:19, 198.35it/s]
 29%|██▊       | 1439/5021 [00:07<00:18, 198.38it/s]
 25%|██▍       | 1239/5021 [00:06<00:19, 198.72it/s]
 29%|██▉       | 1459/5021 [00:07<00:17, 198.63it/s]
 25%|██▌       | 1259/5021 [00:06<00:19, 197.51it/s]
 29%|██▉       | 1479/5021 [00:07<00:17, 198.67it/s]
 25%|██▌       | 1279/5021 [00:06<00:19, 196.54it/s]
 30%|██▉       | 1499/5021 [00:07<00:17, 198.40it/s]
 26%|██▌       | 1300/5021 [00:06<00:18, 197.86it/s]
 30%|███       | 1519/5021 [00:07<00:17, 197.73it/s]
 26%|██▋       | 1321/5021 [00:06<00:18, 198.93it/s]
 31%|███       | 1539/5021 [00:07<00:17, 197.67it/s]
 27%|██▋       | 1342/5021 [00:06<00:18, 199.27it/s]
 31%|███       | 1559/5021 [00:08<00:17, 197.65it/s]
 27%|██▋       | 1363/5021 [00:06<00:18, 199.56it/s]
 31%|███▏      | 1579/5021 [00:08<00:17, 197.52it/s]
 28%|██▊       | 1383/5021 [00:07<00:18, 199.61it/s]
 32%|███▏      | 1599/5021 [00:08<00:17, 197.40it/s]
 28%|██▊       | 1403/5021 [00:07<00:18, 199.40it/s]
 32%|███▏      | 1619/5021 [00:08<00:17, 197.34it/s]
 28%|██▊       | 1423/5021 [00:07<00:18, 199.03it/s]
 33%|███▎      | 1639/5021 [00:08<00:17, 197.36it/s]
 29%|██▊       | 1443/5021 [00:07<00:18, 198.53it/s]
 33%|███▎      | 1659/5021 [00:08<00:17, 197.09it/s]
 29%|██▉       | 1463/5021 [00:07<00:17, 198.36it/s]
 33%|███▎      | 1679/5021 [00:08<00:16, 197.03it/s]
 30%|██▉       | 1483/5021 [00:07<00:17, 198.14it/s]
 34%|███▍      | 1699/5021 [00:08<00:16, 196.65it/s]
 30%|██▉       | 1503/5021 [00:07<00:17, 198.22it/s]
 34%|███▍      | 1719/5021 [00:08<00:16, 196.77it/s]
 30%|███       | 1523/5021 [00:07<00:17, 197.73it/s]
 35%|███▍      | 1739/5021 [00:08<00:16, 196.70it/s]
 31%|███       | 1543/5021 [00:07<00:17, 196.02it/s]
 35%|███▌      | 1759/5021 [00:09<00:16, 195.49it/s]
 31%|███       | 1563/5021 [00:07<00:17, 196.52it/s]
 35%|███▌      | 1779/5021 [00:09<00:16, 196.55it/s]
 32%|███▏      | 1584/5021 [00:08<00:17, 197.64it/s]
 36%|███▌      | 1799/5021 [00:09<00:16, 197.12it/s]
 32%|███▏      | 1604/5021 [00:08<00:17, 195.85it/s]
 36%|███▌      | 1819/5021 [00:09<00:16, 196.25it/s]
 32%|███▏      | 1624/5021 [00:08<00:17, 196.15it/s]
 37%|███▋      | 1839/5021 [00:09<00:16, 196.62it/s]
 33%|███▎      | 1644/5021 [00:08<00:17, 196.38it/s]
 37%|███▋      | 1859/5021 [00:09<00:16, 195.79it/s]
 33%|███▎      | 1664/5021 [00:08<00:17, 196.65it/s]
 37%|███▋      | 1879/5021 [00:09<00:15, 196.70it/s]
 34%|███▎      | 1684/5021 [00:08<00:16, 196.81it/s]
 38%|███▊      | 1899/5021 [00:09<00:15, 197.44it/s]
 34%|███▍      | 1704/5021 [00:08<00:16, 197.13it/s]
 38%|███▊      | 1919/5021 [00:09<00:15, 198.02it/s]
 34%|███▍      | 1724/5021 [00:08<00:16, 197.08it/s]
 39%|███▊      | 1939/5021 [00:09<00:15, 198.07it/s]
 35%|███▍      | 1744/5021 [00:08<00:16, 197.62it/s]
 39%|███▉      | 1959/5021 [00:10<00:15, 197.88it/s]
 35%|███▌      | 1764/5021 [00:09<00:16, 198.12it/s]
 39%|███▉      | 1979/5021 [00:10<00:15, 197.79it/s]
 36%|███▌      | 1784/5021 [00:09<00:16, 196.27it/s]
 40%|███▉      | 1999/5021 [00:10<00:15, 198.02it/s]
 36%|███▌      | 1804/5021 [00:09<00:16, 196.41it/s]
 40%|████      | 2019/5021 [00:10<00:15, 198.35it/s]
 36%|███▋      | 1824/5021 [00:09<00:16, 196.56it/s]
 41%|████      | 2039/5021 [00:10<00:15, 198.62it/s]
 37%|███▋      | 1844/5021 [00:09<00:16, 196.42it/s]
 41%|████      | 2059/5021 [00:10<00:14, 198.44it/s]
 37%|███▋      | 1864/5021 [00:09<00:16, 196.43it/s]
 41%|████▏     | 2079/5021 [00:10<00:14, 198.35it/s]
 38%|███▊      | 1884/5021 [00:09<00:16, 194.41it/s]
 42%|████▏     | 2099/5021 [00:10<00:14, 197.99it/s]
 38%|███▊      | 1904/5021 [00:09<00:15, 195.46it/s]
 42%|████▏     | 2119/5021 [00:10<00:14, 197.79it/s]
 38%|███▊      | 1924/5021 [00:09<00:16, 193.56it/s]
 43%|████▎     | 2139/5021 [00:10<00:14, 197.66it/s]
 39%|███▊      | 1944/5021 [00:09<00:15, 195.11it/s]
 43%|████▎     | 2159/5021 [00:11<00:14, 198.33it/s]
 39%|███▉      | 1964/5021 [00:10<00:15, 193.54it/s]
 43%|████▎     | 2180/5021 [00:11<00:14, 199.21it/s]
 40%|███▉      | 1984/5021 [00:10<00:15, 195.26it/s]
 44%|████▍     | 2201/5021 [00:11<00:14, 199.56it/s]
 40%|███▉      | 2004/5021 [00:10<00:15, 196.54it/s]
 44%|████▍     | 2221/5021 [00:11<00:14, 199.68it/s]
 40%|████      | 2024/5021 [00:10<00:15, 197.01it/s]
 45%|████▍     | 2242/5021 [00:11<00:13, 200.01it/s]
 41%|████      | 2044/5021 [00:10<00:15, 197.67it/s]
 45%|████▌     | 2263/5021 [00:11<00:13, 200.08it/s]
 41%|████      | 2065/5021 [00:10<00:14, 198.52it/s]
 45%|████▌     | 2284/5021 [00:11<00:13, 200.35it/s]
 42%|████▏     | 2085/5021 [00:10<00:14, 198.86it/s]
 46%|████▌     | 2305/5021 [00:11<00:13, 200.17it/s]
 42%|████▏     | 2105/5021 [00:10<00:14, 199.06it/s]
 46%|████▋     | 2326/5021 [00:11<00:13, 199.78it/s]
 42%|████▏     | 2126/5021 [00:10<00:14, 199.81it/s]
 47%|████▋     | 2346/5021 [00:11<00:13, 199.20it/s]
 43%|████▎     | 2147/5021 [00:10<00:14, 200.22it/s]
 47%|████▋     | 2366/5021 [00:12<00:13, 198.88it/s]
 43%|████▎     | 2168/5021 [00:11<00:14, 200.63it/s]
 48%|████▊     | 2386/5021 [00:12<00:13, 198.51it/s]
 44%|████▎     | 2189/5021 [00:11<00:14, 200.53it/s]
 48%|████▊     | 2406/5021 [00:12<00:13, 197.85it/s]
 44%|████▍     | 2210/5021 [00:11<00:14, 200.00it/s]
 48%|████▊     | 2426/5021 [00:12<00:13, 198.15it/s]
 44%|████▍     | 2230/5021 [00:11<00:13, 199.37it/s]
 49%|████▊     | 2446/5021 [00:12<00:12, 198.42it/s]
 45%|████▍     | 2250/5021 [00:11<00:14, 196.77it/s]
 49%|████▉     | 2466/5021 [00:12<00:12, 198.42it/s]
 45%|████▌     | 2270/5021 [00:11<00:13, 196.51it/s]
 50%|████▉     | 2486/5021 [00:12<00:12, 198.37it/s]
 46%|████▌     | 2290/5021 [00:11<00:13, 196.83it/s]
 50%|████▉     | 2506/5021 [00:12<00:12, 195.86it/s]
 46%|████▌     | 2310/5021 [00:11<00:13, 196.81it/s]
 50%|█████     | 2526/5021 [00:12<00:12, 195.09it/s]
 46%|████▋     | 2330/5021 [00:11<00:13, 197.12it/s]
 51%|█████     | 2546/5021 [00:13<00:12, 194.78it/s]
 47%|████▋     | 2350/5021 [00:11<00:13, 197.13it/s]
 51%|█████     | 2566/5021 [00:13<00:12, 194.33it/s]
 47%|████▋     | 2370/5021 [00:12<00:13, 196.62it/s]
 52%|█████▏    | 2586/5021 [00:13<00:12, 194.64it/s]
 48%|████▊     | 2390/5021 [00:12<00:13, 196.59it/s]
 52%|█████▏    | 2606/5021 [00:13<00:12, 195.27it/s]
 48%|████▊     | 2410/5021 [00:12<00:13, 196.79it/s]
 52%|█████▏    | 2626/5021 [00:13<00:12, 195.82it/s]
 48%|████▊     | 2430/5021 [00:12<00:13, 196.88it/s]
 53%|█████▎    | 2646/5021 [00:13<00:12, 195.74it/s]
 49%|████▉     | 2450/5021 [00:12<00:13, 197.65it/s]
 53%|█████▎    | 2666/5021 [00:13<00:11, 196.42it/s]
 49%|████▉     | 2471/5021 [00:12<00:12, 198.67it/s]
 53%|█████▎    | 2686/5021 [00:13<00:11, 195.96it/s]
 50%|████▉     | 2491/5021 [00:12<00:12, 198.56it/s]
 54%|█████▍    | 2706/5021 [00:13<00:11, 196.12it/s]
 50%|█████     | 2511/5021 [00:12<00:12, 198.72it/s]
 54%|█████▍    | 2726/5021 [00:13<00:11, 196.02it/s]
 50%|█████     | 2532/5021 [00:12<00:12, 199.32it/s]
 55%|█████▍    | 2746/5021 [00:14<00:11, 195.90it/s]
 51%|█████     | 2552/5021 [00:12<00:12, 199.47it/s]
 55%|█████▌    | 2766/5021 [00:14<00:11, 195.98it/s]
 51%|█████     | 2573/5021 [00:13<00:12, 199.80it/s]
 55%|█████▌    | 2786/5021 [00:14<00:11, 195.42it/s]
 52%|█████▏    | 2593/5021 [00:13<00:12, 199.84it/s]
 56%|█████▌    | 2806/5021 [00:14<00:11, 195.15it/s]
 52%|█████▏    | 2613/5021 [00:13<00:12, 199.52it/s]
 56%|█████▋    | 2826/5021 [00:14<00:11, 195.70it/s]
 52%|█████▏    | 2633/5021 [00:13<00:12, 198.78it/s]
 57%|█████▋    | 2846/5021 [00:14<00:11, 195.30it/s]
 53%|█████▎    | 2653/5021 [00:13<00:11, 198.32it/s]
 57%|█████▋    | 2866/5021 [00:14<00:11, 195.77it/s]
 53%|█████▎    | 2673/5021 [00:13<00:11, 198.21it/s]
 57%|█████▋    | 2886/5021 [00:14<00:10, 195.90it/s]
 54%|█████▎    | 2693/5021 [00:13<00:11, 197.98it/s]
 58%|█████▊    | 2906/5021 [00:14<00:10, 196.18it/s]
 54%|█████▍    | 2713/5021 [00:13<00:11, 197.57it/s]
 58%|█████▊    | 2926/5021 [00:14<00:10, 195.54it/s]
 54%|█████▍    | 2733/5021 [00:13<00:11, 197.18it/s]
 59%|█████▊    | 2946/5021 [00:15<00:10, 195.99it/s]
 55%|█████▍    | 2753/5021 [00:14<00:11, 195.98it/s]
 59%|█████▉    | 2966/5021 [00:15<00:10, 196.50it/s]
 55%|█████▌    | 2773/5021 [00:14<00:11, 195.87it/s]
 59%|█████▉    | 2986/5021 [00:15<00:10, 196.78it/s]
 56%|█████▌    | 2793/5021 [00:14<00:11, 196.06it/s]
 60%|█████▉    | 3006/5021 [00:15<00:10, 197.48it/s]
 56%|█████▌    | 2813/5021 [00:14<00:11, 195.77it/s]
 60%|██████    | 3026/5021 [00:15<00:10, 197.72it/s]
 56%|█████▋    | 2833/5021 [00:14<00:11, 195.42it/s]
 61%|██████    | 3046/5021 [00:15<00:09, 197.92it/s]
 57%|█████▋    | 2853/5021 [00:14<00:11, 195.69it/s]
 61%|██████    | 3066/5021 [00:15<00:09, 198.12it/s]
 57%|█████▋    | 2873/5021 [00:14<00:11, 195.11it/s]
 61%|██████▏   | 3086/5021 [00:15<00:09, 197.81it/s]
 58%|█████▊    | 2893/5021 [00:14<00:10, 196.10it/s]
 62%|██████▏   | 3106/5021 [00:15<00:09, 197.82it/s]
 58%|█████▊    | 2913/5021 [00:14<00:10, 196.72it/s]
 62%|██████▏   | 3126/5021 [00:15<00:09, 197.81it/s]
 58%|█████▊    | 2933/5021 [00:14<00:10, 197.64it/s]
 63%|██████▎   | 3146/5021 [00:16<00:09, 195.34it/s]
 59%|█████▉    | 2953/5021 [00:15<00:10, 198.04it/s]
 63%|██████▎   | 3166/5021 [00:16<00:09, 195.95it/s]
 59%|█████▉    | 2973/5021 [00:15<00:10, 198.34it/s]
 63%|██████▎   | 3186/5021 [00:16<00:09, 196.59it/s]
 60%|█████▉    | 2993/5021 [00:15<00:10, 198.42it/s]
 64%|██████▍   | 3206/5021 [00:16<00:09, 196.67it/s]
 60%|██████    | 3013/5021 [00:15<00:10, 198.73it/s]
 64%|██████▍   | 3226/5021 [00:16<00:09, 196.47it/s]
 60%|██████    | 3034/5021 [00:15<00:09, 199.22it/s]
 65%|██████▍   | 3246/5021 [00:16<00:09, 196.67it/s]
 61%|██████    | 3054/5021 [00:15<00:09, 199.34it/s]
 65%|██████▌   | 3266/5021 [00:16<00:08, 196.41it/s]
 61%|██████    | 3074/5021 [00:15<00:09, 198.90it/s]
 65%|██████▌   | 3286/5021 [00:16<00:08, 196.56it/s]
 62%|██████▏   | 3094/5021 [00:15<00:09, 198.35it/s]
 66%|██████▌   | 3306/5021 [00:16<00:08, 196.63it/s]
 62%|██████▏   | 3114/5021 [00:15<00:09, 198.56it/s]
 66%|██████▌   | 3326/5021 [00:16<00:08, 196.72it/s]
 62%|██████▏   | 3134/5021 [00:15<00:09, 198.19it/s]
 67%|██████▋   | 3346/5021 [00:17<00:08, 196.51it/s]
 63%|██████▎   | 3154/5021 [00:16<00:09, 197.26it/s]
 67%|██████▋   | 3366/5021 [00:17<00:08, 196.98it/s]
 63%|██████▎   | 3174/5021 [00:16<00:09, 197.26it/s]
 67%|██████▋   | 3386/5021 [00:17<00:08, 197.16it/s]
 64%|██████▎   | 3194/5021 [00:16<00:09, 196.83it/s]
 68%|██████▊   | 3406/5021 [00:17<00:08, 197.45it/s]
 64%|██████▍   | 3214/5021 [00:16<00:09, 197.16it/s]
 68%|██████▊   | 3426/5021 [00:17<00:08, 197.76it/s]
 64%|██████▍   | 3234/5021 [00:16<00:09, 197.51it/s]
 69%|██████▊   | 3446/5021 [00:17<00:07, 198.15it/s]
 65%|██████▍   | 3254/5021 [00:16<00:08, 197.17it/s]
 69%|██████▉   | 3466/5021 [00:17<00:07, 198.70it/s]
 65%|██████▌   | 3274/5021 [00:16<00:08, 197.45it/s]
 69%|██████▉   | 3486/5021 [00:17<00:07, 198.61it/s]
 66%|██████▌   | 3294/5021 [00:16<00:08, 197.84it/s]
 70%|██████▉   | 3506/5021 [00:17<00:07, 198.86it/s]
 66%|██████▌   | 3314/5021 [00:16<00:08, 198.14it/s]
 70%|███████   | 3526/5021 [00:17<00:07, 198.64it/s]
 66%|██████▋   | 3334/5021 [00:16<00:08, 198.24it/s]
 71%|███████   | 3546/5021 [00:18<00:07, 197.86it/s]
 67%|██████▋   | 3354/5021 [00:17<00:08, 198.55it/s]
 71%|███████   | 3566/5021 [00:18<00:07, 197.94it/s]
 67%|██████▋   | 3374/5021 [00:17<00:08, 198.66it/s]
 71%|███████▏  | 3586/5021 [00:18<00:07, 197.91it/s]
 68%|██████▊   | 3395/5021 [00:17<00:08, 199.25it/s]
 72%|███████▏  | 3606/5021 [00:18<00:07, 197.88it/s]
 68%|██████▊   | 3415/5021 [00:17<00:08, 198.55it/s]
 72%|███████▏  | 3626/5021 [00:18<00:07, 197.99it/s]
 68%|██████▊   | 3435/5021 [00:17<00:08, 197.97it/s]
 73%|███████▎  | 3646/5021 [00:18<00:06, 198.20it/s]
 69%|██████▉   | 3455/5021 [00:17<00:07, 197.59it/s]
 73%|███████▎  | 3666/5021 [00:18<00:06, 197.95it/s]
 69%|██████▉   | 3475/5021 [00:17<00:07, 197.75it/s]
 73%|███████▎  | 3686/5021 [00:18<00:06, 197.93it/s]
 70%|██████▉   | 3495/5021 [00:17<00:07, 197.60it/s]
 74%|███████▍  | 3706/5021 [00:18<00:06, 198.09it/s]
 70%|███████   | 3515/5021 [00:17<00:07, 197.69it/s]
 74%|███████▍  | 3726/5021 [00:19<00:06, 197.69it/s]
 70%|███████   | 3535/5021 [00:17<00:07, 197.67it/s]
 75%|███████▍  | 3746/5021 [00:19<00:06, 197.48it/s]
 71%|███████   | 3555/5021 [00:18<00:07, 197.60it/s]
 75%|███████▌  | 3766/5021 [00:19<00:06, 197.35it/s]
 71%|███████   | 3575/5021 [00:18<00:07, 197.54it/s]
 75%|███████▌  | 3786/5021 [00:19<00:06, 197.91it/s]
 72%|███████▏  | 3595/5021 [00:18<00:07, 197.51it/s]
 76%|███████▌  | 3806/5021 [00:19<00:06, 194.36it/s]
 72%|███████▏  | 3615/5021 [00:18<00:07, 197.50it/s]
 76%|███████▌  | 3826/5021 [00:19<00:06, 195.78it/s]
 72%|███████▏  | 3635/5021 [00:18<00:07, 197.58it/s]
 77%|███████▋  | 3847/5021 [00:19<00:05, 197.04it/s]
 73%|███████▎  | 3655/5021 [00:18<00:06, 197.77it/s]
 77%|███████▋  | 3867/5021 [00:19<00:05, 197.74it/s]
 73%|███████▎  | 3675/5021 [00:18<00:06, 198.03it/s]
 77%|███████▋  | 3887/5021 [00:19<00:05, 198.02it/s]
 74%|███████▎  | 3695/5021 [00:18<00:06, 197.80it/s]
 78%|███████▊  | 3907/5021 [00:19<00:05, 198.12it/s]
 74%|███████▍  | 3715/5021 [00:18<00:06, 198.11it/s]
 78%|███████▊  | 3927/5021 [00:20<00:05, 197.48it/s]
 74%|███████▍  | 3735/5021 [00:18<00:06, 198.08it/s]
 79%|███████▊  | 3947/5021 [00:20<00:05, 197.87it/s]
 75%|███████▍  | 3755/5021 [00:19<00:06, 198.59it/s]
 79%|███████▉  | 3967/5021 [00:20<00:05, 197.53it/s]
 75%|███████▌  | 3775/5021 [00:19<00:06, 198.59it/s]
 79%|███████▉  | 3987/5021 [00:20<00:05, 197.45it/s]
 76%|███████▌  | 3795/5021 [00:19<00:06, 198.72it/s]
 80%|███████▉  | 4007/5021 [00:20<00:05, 197.10it/s]
 76%|███████▌  | 3815/5021 [00:19<00:06, 198.59it/s]
 80%|████████  | 4027/5021 [00:20<00:05, 197.07it/s]
 76%|███████▋  | 3835/5021 [00:19<00:05, 198.18it/s]
 81%|████████  | 4047/5021 [00:20<00:04, 196.92it/s]
 77%|███████▋  | 3855/5021 [00:19<00:05, 196.29it/s]
 81%|████████  | 4067/5021 [00:20<00:04, 197.11it/s]
 77%|███████▋  | 3875/5021 [00:19<00:05, 196.71it/s]
 81%|████████▏ | 4087/5021 [00:20<00:04, 197.33it/s]
 78%|███████▊  | 3895/5021 [00:19<00:05, 196.84it/s]
 82%|████████▏ | 4107/5021 [00:20<00:04, 197.56it/s]
 78%|███████▊  | 3915/5021 [00:19<00:05, 196.92it/s]
 82%|████████▏ | 4127/5021 [00:21<00:04, 197.55it/s]
 78%|███████▊  | 3935/5021 [00:19<00:05, 196.59it/s]
 83%|████████▎ | 4147/5021 [00:21<00:04, 197.28it/s]
 79%|███████▉  | 3955/5021 [00:20<00:05, 196.54it/s]
 83%|████████▎ | 4167/5021 [00:21<00:04, 197.95it/s]
 79%|███████▉  | 3975/5021 [00:20<00:05, 196.59it/s]
 83%|████████▎ | 4188/5021 [00:21<00:04, 198.59it/s]
 80%|███████▉  | 3995/5021 [00:20<00:05, 197.29it/s]
 84%|████████▍ | 4209/5021 [00:21<00:04, 199.25it/s]
 80%|███████▉  | 4015/5021 [00:20<00:05, 197.21it/s]
 84%|████████▍ | 4230/5021 [00:21<00:03, 199.59it/s]
 80%|████████  | 4035/5021 [00:20<00:04, 197.39it/s]
 85%|████████▍ | 4250/5021 [00:21<00:03, 199.55it/s]
 81%|████████  | 4055/5021 [00:20<00:04, 197.55it/s]
 85%|████████▌ | 4270/5021 [00:21<00:03, 199.48it/s]
 81%|████████  | 4075/5021 [00:20<00:04, 198.06it/s]
 85%|████████▌ | 4291/5021 [00:21<00:03, 199.72it/s]
 82%|████████▏ | 4095/5021 [00:20<00:04, 198.23it/s]
 86%|████████▌ | 4311/5021 [00:21<00:03, 199.45it/s]
 82%|████████▏ | 4115/5021 [00:20<00:04, 197.53it/s]
 86%|████████▋ | 4331/5021 [00:22<00:03, 199.45it/s]
 82%|████████▏ | 4135/5021 [00:21<00:04, 196.89it/s]
 87%|████████▋ | 4351/5021 [00:22<00:03, 198.59it/s]
 83%|████████▎ | 4155/5021 [00:21<00:04, 197.41it/s]
 87%|████████▋ | 4371/5021 [00:22<00:03, 198.47it/s]
 83%|████████▎ | 4175/5021 [00:21<00:04, 197.07it/s]
 87%|████████▋ | 4391/5021 [00:22<00:03, 197.82it/s]
 84%|████████▎ | 4195/5021 [00:21<00:04, 196.30it/s]
 88%|████████▊ | 4411/5021 [00:22<00:03, 197.46it/s]
 84%|████████▍ | 4215/5021 [00:21<00:04, 195.98it/s]
 88%|████████▊ | 4431/5021 [00:22<00:03, 195.97it/s]
 84%|████████▍ | 4235/5021 [00:21<00:03, 196.87it/s]
 89%|████████▊ | 4451/5021 [00:22<00:02, 196.13it/s]
 85%|████████▍ | 4255/5021 [00:21<00:03, 196.79it/s]
 89%|████████▉ | 4471/5021 [00:22<00:02, 196.41it/s]
 85%|████████▌ | 4275/5021 [00:21<00:03, 197.38it/s]
 89%|████████▉ | 4491/5021 [00:22<00:02, 196.08it/s]
 86%|████████▌ | 4295/5021 [00:21<00:03, 197.39it/s]
 90%|████████▉ | 4511/5021 [00:22<00:02, 196.18it/s]
 86%|████████▌ | 4315/5021 [00:21<00:03, 197.37it/s]
 90%|█████████ | 4531/5021 [00:23<00:02, 196.43it/s]
 86%|████████▋ | 4335/5021 [00:22<00:03, 197.70it/s]
 91%|█████████ | 4551/5021 [00:23<00:02, 196.55it/s]
 87%|████████▋ | 4355/5021 [00:22<00:03, 197.73it/s]
 91%|█████████ | 4571/5021 [00:23<00:02, 196.90it/s]
 87%|████████▋ | 4375/5021 [00:22<00:03, 197.65it/s]
 91%|█████████▏| 4591/5021 [00:23<00:02, 197.70it/s]
 88%|████████▊ | 4395/5021 [00:22<00:03, 198.12it/s]
 92%|█████████▏| 4611/5021 [00:23<00:02, 198.16it/s]
 88%|████████▊ | 4415/5021 [00:22<00:03, 198.51it/s]
 92%|█████████▏| 4631/5021 [00:23<00:01, 198.22it/s]
 88%|████████▊ | 4435/5021 [00:22<00:02, 198.71it/s]
 93%|█████████▎| 4651/5021 [00:23<00:01, 198.35it/s]
 89%|████████▊ | 4455/5021 [00:22<00:02, 199.06it/s]
 93%|█████████▎| 4671/5021 [00:23<00:01, 198.54it/s]
 89%|████████▉ | 4475/5021 [00:22<00:02, 199.09it/s]
 93%|█████████▎| 4691/5021 [00:23<00:01, 198.97it/s]
 90%|████████▉ | 4495/5021 [00:22<00:02, 196.87it/s]
 94%|█████████▍| 4711/5021 [00:23<00:01, 198.90it/s]
 90%|████████▉ | 4515/5021 [00:22<00:02, 196.71it/s]
 94%|█████████▍| 4731/5021 [00:24<00:01, 198.83it/s]
 90%|█████████ | 4535/5021 [00:23<00:02, 196.42it/s]
 95%|█████████▍| 4751/5021 [00:24<00:01, 197.61it/s]
 91%|█████████ | 4555/5021 [00:23<00:02, 197.14it/s]
 95%|█████████▌| 4771/5021 [00:24<00:01, 197.17it/s]
 91%|█████████ | 4575/5021 [00:23<00:02, 197.41it/s]
 95%|█████████▌| 4791/5021 [00:24<00:01, 196.98it/s]
 92%|█████████▏| 4595/5021 [00:23<00:02, 197.25it/s]
 96%|█████████▌| 4811/5021 [00:24<00:01, 196.87it/s]
 92%|█████████▏| 4615/5021 [00:23<00:02, 197.12it/s]
 96%|█████████▌| 4831/5021 [00:24<00:00, 197.00it/s]
 92%|█████████▏| 4635/5021 [00:23<00:01, 196.75it/s]
 97%|█████████▋| 4851/5021 [00:24<00:00, 196.98it/s]
 93%|█████████▎| 4655/5021 [00:23<00:01, 196.72it/s]
 97%|█████████▋| 4871/5021 [00:24<00:00, 196.81it/s]
 93%|█████████▎| 4675/5021 [00:23<00:01, 196.40it/s]
 97%|█████████▋| 4891/5021 [00:24<00:00, 196.59it/s]
 94%|█████████▎| 4695/5021 [00:23<00:01, 195.97it/s]
 98%|█████████▊| 4911/5021 [00:25<00:00, 196.46it/s]
 94%|█████████▍| 4715/5021 [00:23<00:01, 195.66it/s]
 98%|█████████▊| 4931/5021 [00:25<00:00, 196.12it/s]
 94%|█████████▍| 4735/5021 [00:24<00:01, 195.32it/s]
 99%|█████████▊| 4951/5021 [00:25<00:00, 196.80it/s]
 95%|█████████▍| 4755/5021 [00:24<00:01, 195.42it/s]
 99%|█████████▉| 4971/5021 [00:25<00:00, 197.04it/s]
 95%|█████████▌| 4775/5021 [00:24<00:01, 195.32it/s]
 99%|█████████▉| 4991/5021 [00:25<00:00, 197.66it/s]
 95%|█████████▌| 4795/5021 [00:24<00:01, 195.72it/s]
100%|█████████▉| 5011/5021 [00:25<00:00, 197.92it/s]
 96%|█████████▌| 4815/5021 [00:24<00:01, 196.04it/s]
100%|██████████| 5021/5021 [00:25<00:00, 196.44it/s]

 96%|█████████▋| 4835/5021 [00:24<00:00, 194.53it/s]
 97%|█████████▋| 4855/5021 [00:24<00:00, 195.03it/s]
 97%|█████████▋| 4875/5021 [00:24<00:00, 195.46it/s]
 97%|█████████▋| 4895/5021 [00:24<00:00, 196.53it/s]
 98%|█████████▊| 4915/5021 [00:24<00:00, 197.27it/s]
 98%|█████████▊| 4935/5021 [00:25<00:00, 197.95it/s]
 99%|█████████▊| 4955/5021 [00:25<00:00, 192.24it/s]
 99%|█████████▉| 4975/5021 [00:25<00:00, 194.13it/s]
100%|█████████▉| 4996/5021 [00:25<00:00, 196.07it/s]
100%|█████████▉| 5016/5021 [00:25<00:00, 196.85it/s]
100%|██████████| 5021/5021 [00:25<00:00, 196.84it/s]
n136-128-154:2250935:2250935 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2250935:2250935 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2250935:2250935 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:2250935:2250935 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2250935:2250935 [0] NCCL INFO Comm config Blocking set to 1
n136-128-154:2250936:2250936 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:2250936:2250936 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2250936:2250936 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2250936:2250936 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2250936:2250936 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:2250936:2251207 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2250936:2251207 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2250935:2251206 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2250935:2251206 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2250936:2251207 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2250936:2251207 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2250936:2251207 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2250935:2251206 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2250935:2251206 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2250935:2251206 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2250936:2251207 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2250936:2251207 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2250936:2251207 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:2250936:2251207 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2250936:2251207 [1] NCCL INFO Using network IB
n136-128-154:2250935:2251206 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2250935:2251206 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2250935:2251206 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:2250935:2251206 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2250935:2251206 [0] NCCL INFO Using network IB
n136-128-154:2250936:2251207 [1] NCCL INFO ncclCommInitRankConfig comm 0x13dbbd10 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0x28bdcd3d8c23a13b - Init START
n136-128-154:2250935:2251206 [0] NCCL INFO ncclCommInitRankConfig comm 0x140bf390 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0x28bdcd3d8c23a13b - Init START
n136-128-154:2250936:2251207 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2250935:2251206 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2250936:2251207 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2250936:2251207 [1] NCCL INFO Retrieving state for IB
n136-128-154:2250936:2251207 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:2250936:2251207 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2250935:2251206 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2250935:2251206 [0] NCCL INFO Retrieving state for IB
n136-128-154:2250935:2251206 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:2250936:2251207 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2250935:2251206 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2250935:2251206 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2250936:2251207 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2250936:2251207 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2250935:2251206 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2250935:2251206 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2250935:2251206 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2250935:2251206 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2250936:2251207 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2250936:2251207 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2250936:2251207 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2250936:2251207 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2250936:2251207 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2250936:2251207 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2250936:2251207 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2250936:2251207 [1] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2250936:2251207 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2250936:2251207 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2250936:2251207 [1] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2250936:2251207 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2250936:2251207 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2250936:2251207 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2250936:2251207 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2250936:2251207 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2250936:2251207 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2250936:2251207 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2250936:2251207 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2250936:2251207 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2250936:2251207 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2250936:2251207 [1] NCCL INFO ==========================================
n136-128-154:2250936:2251207 [1] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2250936:2251207 [1] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2250936:2251207 [1] NCCL INFO Setting affinity for GPU 7 to 33-62,97-126
n136-128-154:2250935:2251206 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2250935:2251206 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2250935:2251206 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2250935:2251206 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2250935:2251206 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2250935:2251206 [0] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2250935:2251206 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2250935:2251206 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2250935:2251206 [0] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2250935:2251206 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2250935:2251206 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2250935:2251206 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2250935:2251206 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2250935:2251206 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2250935:2251206 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2250935:2251206 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2250935:2251206 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2250935:2251206 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2250935:2251206 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2250935:2251206 [0] NCCL INFO ==========================================
n136-128-154:2250935:2251206 [0] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2250935:2251206 [0] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2250935:2251206 [0] NCCL INFO Setting affinity for GPU 6 to 33-62,97-126
n136-128-154:2250936:2251207 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2250936:2251207 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250936:2251207 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250936:2251207 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250936:2251207 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250936:2251207 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250936:2251207 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250936:2251207 [1] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250936:2251207 [1] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250936:2251207 [1] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250936:2251207 [1] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250936:2251207 [1] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250936:2251207 [1] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250936:2251207 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2250936:2251207 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250936:2251207 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250936:2251207 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250936:2251207 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250936:2251207 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250936:2251207 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250936:2251207 [1] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2250936:2251207 [1] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2250936:2251207 [1] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2250936:2251207 [1] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2250936:2251207 [1] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2250936:2251207 [1] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2250935:2251206 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2250935:2251206 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250935:2251206 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250935:2251206 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250935:2251206 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250935:2251206 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250935:2251206 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250935:2251206 [0] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250935:2251206 [0] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250935:2251206 [0] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250935:2251206 [0] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250935:2251206 [0] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250935:2251206 [0] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250935:2251206 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2250935:2251206 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250935:2251206 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250935:2251206 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250935:2251206 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250935:2251206 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250935:2251206 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2250935:2251206 [0] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2250935:2251206 [0] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2250935:2251206 [0] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2250935:2251206 [0] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2250935:2251206 [0] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2250935:2251206 [0] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2250936:2251207 [1] NCCL INFO comm 0x13dbbd10 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:2250935:2251206 [0] NCCL INFO comm 0x140bf390 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:2250935:2251206 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:2250936:2251207 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:2250935:2251206 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:2250936:2251207 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:2250935:2251206 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:2250936:2251207 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:2250935:2251206 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:2250936:2251207 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:2250935:2251206 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:2250936:2251207 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:2250935:2251206 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:2250936:2251207 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:2250935:2251206 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:2250936:2251207 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:2250935:2251206 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:2250936:2251207 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:2250935:2251206 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:2250936:2251207 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:2250935:2251206 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:2250936:2251207 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:2250935:2251206 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:2250936:2251207 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:2250935:2251206 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:2250936:2251207 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:2250935:2251206 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:2250936:2251207 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:2250935:2251206 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:2250936:2251207 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:2250935:2251206 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:2250936:2251207 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:2250935:2251206 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:2250936:2251207 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:2250935:2251206 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:2250936:2251207 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:2250935:2251206 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:2250936:2251207 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:2250935:2251206 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:2250936:2251207 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:2250935:2251206 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:2250936:2251207 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:2250935:2251206 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:2250936:2251207 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:2250935:2251206 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:2250936:2251207 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:2250935:2251206 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:2250936:2251207 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:2250935:2251206 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:2250936:2251207 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:2250935:2251206 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:2250935:2251206 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:2250936:2251207 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:2250935:2251206 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:2250936:2251207 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:2250935:2251206 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:2250936:2251207 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:2250935:2251206 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:2250936:2251207 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:2250935:2251206 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:2250936:2251207 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:2250935:2251206 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:2250936:2251207 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:2250935:2251206 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:2250936:2251207 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:2250935:2251206 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:2250936:2251207 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:2250935:2251206 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:2250936:2251207 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:2250935:2251206 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:2250936:2251207 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:2250935:2251206 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:2250936:2251207 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:2250935:2251206 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:2250936:2251207 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:2250935:2251206 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:2250936:2251207 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:2250935:2251206 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:2250936:2251207 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:2250935:2251206 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:2250936:2251207 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:2250935:2251206 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:2250936:2251207 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:2250935:2251206 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:2250936:2251207 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:2250935:2251206 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:2250936:2251207 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:2250935:2251206 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:2250936:2251207 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:2250935:2251206 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:2250936:2251207 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:2250935:2251206 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:2250936:2251207 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:2250935:2251206 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:2250936:2251207 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:2250935:2251206 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:2250936:2251207 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:2250935:2251206 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:2250936:2251207 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:2250935:2251206 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:2250936:2251207 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:2250935:2251206 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:2250935:2251206 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:2250936:2251207 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2250935:2251206 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:2250935:2251206 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:2250935:2251206 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:2250935:2251206 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:2250935:2251206 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:2250935:2251206 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:2250935:2251206 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:2250935:2251206 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:2250935:2251206 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:2250935:2251206 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:2250935:2251206 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:2250935:2251206 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:2250935:2251206 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:2250935:2251206 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:2250935:2251206 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:2250935:2251206 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:2250935:2251206 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:2250935:2251206 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:2250935:2251206 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:2250935:2251206 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:2250935:2251206 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:2250935:2251206 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2250936:2251207 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2250936:2251220 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 102
n136-128-154:2250936:2251219 [1] NCCL INFO [Proxy Service] Device 1 CPU core 99
n136-128-154:2250935:2251206 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2250935:2251206 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:2250935:2251221 [0] NCCL INFO [Proxy Service] Device 0 CPU core 46
n136-128-154:2250935:2251222 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 47
n136-128-154:2250936:2251207 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2250936:2251207 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2250935:2251206 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2250935:2251206 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2250935:2251206 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:2250936:2251207 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2250936:2251207 [1] NCCL INFO ncclCommInitRankConfig comm 0x13dbbd10 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0x28bdcd3d8c23a13b - Init COMPLETE
n136-128-154:2250936:2251207 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.82 (kernels 0.14, alloc 0.44, bootstrap 0.00, allgathers 0.02, topo 0.08, graphs 0.00, connections 0.01, rest 0.14)
n136-128-154:2250935:2251206 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2250935:2251206 [0] NCCL INFO ncclCommInitRankConfig comm 0x140bf390 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0x28bdcd3d8c23a13b - Init COMPLETE
n136-128-154:2250935:2251206 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.83 (kernels 0.14, alloc 0.44, bootstrap 0.00, allgathers 0.02, topo 0.08, graphs 0.00, connections 0.01, rest 0.14)
n136-128-154:2250936:2251223 [1] NCCL INFO Channel 00/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2251223 [1] NCCL INFO Channel 01/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2251223 [1] NCCL INFO Channel 02/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2251223 [1] NCCL INFO Channel 03/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2251223 [1] NCCL INFO Channel 04/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2251223 [1] NCCL INFO Channel 05/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2251223 [1] NCCL INFO Channel 06/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2251223 [1] NCCL INFO Channel 07/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2251223 [1] NCCL INFO Channel 08/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2251223 [1] NCCL INFO Channel 09/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2251223 [1] NCCL INFO Channel 10/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2251223 [1] NCCL INFO Channel 11/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2251223 [1] NCCL INFO Channel 12/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2251223 [1] NCCL INFO Channel 13/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2251223 [1] NCCL INFO Channel 14/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250935:2251224 [0] NCCL INFO Channel 00/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250936:2251223 [1] NCCL INFO Channel 15/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250935:2251224 [0] NCCL INFO Channel 01/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250936:2251223 [1] NCCL INFO Channel 16/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250935:2251224 [0] NCCL INFO Channel 02/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250936:2251223 [1] NCCL INFO Channel 17/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250935:2251224 [0] NCCL INFO Channel 03/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250936:2251223 [1] NCCL INFO Channel 18/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250935:2251224 [0] NCCL INFO Channel 04/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250936:2251223 [1] NCCL INFO Channel 19/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250935:2251224 [0] NCCL INFO Channel 05/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250936:2251223 [1] NCCL INFO Channel 20/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250935:2251224 [0] NCCL INFO Channel 06/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250936:2251223 [1] NCCL INFO Channel 21/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250935:2251224 [0] NCCL INFO Channel 07/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250936:2251223 [1] NCCL INFO Channel 22/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250935:2251224 [0] NCCL INFO Channel 08/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250936:2251223 [1] NCCL INFO Channel 23/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250935:2251224 [0] NCCL INFO Channel 09/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250935:2251224 [0] NCCL INFO Channel 10/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250935:2251224 [0] NCCL INFO Channel 11/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250935:2251224 [0] NCCL INFO Channel 12/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250935:2251224 [0] NCCL INFO Channel 13/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250935:2251224 [0] NCCL INFO Channel 14/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250935:2251224 [0] NCCL INFO Channel 15/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250935:2251224 [0] NCCL INFO Channel 16/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250935:2251224 [0] NCCL INFO Channel 17/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250935:2251224 [0] NCCL INFO Channel 18/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250935:2251224 [0] NCCL INFO Channel 19/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250935:2251224 [0] NCCL INFO Channel 20/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250935:2251224 [0] NCCL INFO Channel 21/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250935:2251224 [0] NCCL INFO Channel 22/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250935:2251224 [0] NCCL INFO Channel 23/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2250936:2251223 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:2250935:2251224 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-09:20:10:44 INFO     [evaluator:559] Running loglikelihood requests
2025-12-09:20:10:44 INFO     [evaluator:559] Running loglikelihood requests
Passed argument batch_size = auto:1. Detecting largest batch size

Running loglikelihood requests:   0%|          | 0/20084 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 64
Determined largest batch size: 64

Running loglikelihood requests:   0%|          | 1/20084 [00:35<200:40:30, 35.97s/it]
Running loglikelihood requests:   0%|          | 65/20084 [00:42<2:42:42,  2.05it/s] 
Running loglikelihood requests:   1%|          | 129/20084 [00:48<1:25:13,  3.90it/s]
Running loglikelihood requests:   1%|          | 193/20084 [00:53<59:26,  5.58it/s]  
Running loglikelihood requests:   1%|▏         | 257/20084 [00:59<47:11,  7.00it/s]
Running loglikelihood requests:   2%|▏         | 321/20084 [01:04<40:20,  8.16it/s]
Running loglikelihood requests:   2%|▏         | 385/20084 [01:10<36:06,  9.09it/s]
Running loglikelihood requests:   2%|▏         | 449/20084 [01:15<33:20,  9.81it/s]
Running loglikelihood requests:   3%|▎         | 513/20084 [01:21<31:30, 10.35it/s]
Running loglikelihood requests:   3%|▎         | 577/20084 [01:26<30:09, 10.78it/s]
Running loglikelihood requests:   3%|▎         | 641/20084 [01:32<29:10, 11.11it/s]
Running loglikelihood requests:   4%|▎         | 705/20084 [01:37<28:26, 11.35it/s]
Running loglikelihood requests:   4%|▍         | 769/20084 [01:42<27:53, 11.54it/s]
Running loglikelihood requests:   4%|▍         | 833/20084 [01:48<27:28, 11.68it/s]
Running loglikelihood requests:   4%|▍         | 897/20084 [01:53<27:07, 11.79it/s]
Running loglikelihood requests:   5%|▍         | 961/20084 [01:58<26:49, 11.88it/s]
Running loglikelihood requests:   5%|▌         | 1025/20084 [02:03<26:34, 11.95it/s]
Running loglikelihood requests:   5%|▌         | 1089/20084 [02:09<26:21, 12.01it/s]
Running loglikelihood requests:   6%|▌         | 1153/20084 [02:14<26:09, 12.06it/s]
Running loglikelihood requests:   6%|▌         | 1217/20084 [02:19<25:58, 12.10it/s]
Running loglikelihood requests:   6%|▋         | 1281/20084 [02:24<25:48, 12.14it/s]
Running loglikelihood requests:   7%|▋         | 1345/20084 [02:30<25:38, 12.18it/s]
Running loglikelihood requests:   7%|▋         | 1409/20084 [02:35<25:28, 12.22it/s]
Running loglikelihood requests:   7%|▋         | 1473/20084 [02:40<25:19, 12.25it/s]
Running loglikelihood requests:   8%|▊         | 1537/20084 [02:45<25:10, 12.28it/s]
Running loglikelihood requests:   8%|▊         | 1601/20084 [02:50<25:02, 12.30it/s]
Running loglikelihood requests:   8%|▊         | 1665/20084 [02:56<24:54, 12.33it/s]
Running loglikelihood requests:   9%|▊         | 1729/20084 [03:01<24:45, 12.36it/s]
Running loglikelihood requests:   9%|▉         | 1793/20084 [03:06<24:37, 12.38it/s]
Running loglikelihood requests:   9%|▉         | 1857/20084 [03:11<24:28, 12.41it/s]
Running loglikelihood requests:  10%|▉         | 1921/20084 [03:16<24:21, 12.43it/s]
Running loglikelihood requests:  10%|▉         | 1985/20084 [03:21<24:13, 12.45it/s]
Running loglikelihood requests:  10%|█         | 2049/20084 [03:26<24:05, 12.48it/s]
Running loglikelihood requests:  11%|█         | 2113/20084 [03:32<23:57, 12.50it/s]
Running loglikelihood requests:  11%|█         | 2177/20084 [03:37<23:49, 12.52it/s]
Running loglikelihood requests:  11%|█         | 2241/20084 [03:42<23:42, 12.54it/s]
Running loglikelihood requests:  11%|█▏        | 2305/20084 [03:47<23:35, 12.56it/s]
Running loglikelihood requests:  12%|█▏        | 2369/20084 [03:52<23:28, 12.57it/s]
Running loglikelihood requests:  12%|█▏        | 2433/20084 [03:57<23:22, 12.58it/s]
Running loglikelihood requests:  12%|█▏        | 2497/20084 [04:02<23:15, 12.60it/s]
Running loglikelihood requests:  13%|█▎        | 2561/20084 [04:07<23:09, 12.61it/s]
Running loglikelihood requests:  13%|█▎        | 2625/20084 [04:12<23:02, 12.63it/s]
Running loglikelihood requests:  13%|█▎        | 2689/20084 [04:17<22:55, 12.64it/s]
Running loglikelihood requests:  14%|█▎        | 2753/20084 [04:22<22:49, 12.65it/s]
Running loglikelihood requests:  14%|█▍        | 2817/20084 [04:27<22:43, 12.66it/s]
Running loglikelihood requests:  14%|█▍        | 2881/20084 [04:32<22:35, 12.69it/s]
Running loglikelihood requests:  15%|█▍        | 2945/20084 [04:37<22:28, 12.71it/s]
Running loglikelihood requests:  15%|█▍        | 3009/20084 [04:42<22:22, 12.72it/s]
Running loglikelihood requests:  15%|█▌        | 3073/20084 [04:47<22:16, 12.73it/s]
Running loglikelihood requests:  16%|█▌        | 3137/20084 [04:52<22:08, 12.76it/s]
Running loglikelihood requests:  16%|█▌        | 3201/20084 [04:57<22:01, 12.78it/s]
Running loglikelihood requests:  16%|█▋        | 3265/20084 [05:02<21:55, 12.79it/s]
Running loglikelihood requests:  17%|█▋        | 3329/20084 [05:07<21:48, 12.81it/s]
Running loglikelihood requests:  17%|█▋        | 3393/20084 [05:12<21:41, 12.82it/s]
Running loglikelihood requests:  17%|█▋        | 3457/20084 [05:17<21:34, 12.84it/s]
Running loglikelihood requests:  18%|█▊        | 3521/20084 [05:22<21:28, 12.86it/s]
Running loglikelihood requests:  18%|█▊        | 3585/20084 [05:27<21:21, 12.87it/s]
Running loglikelihood requests:  18%|█▊        | 3649/20084 [05:32<21:14, 12.89it/s]
Running loglikelihood requests:  18%|█▊        | 3713/20084 [05:37<21:07, 12.91it/s]
Running loglikelihood requests:  19%|█▉        | 3777/20084 [05:42<21:01, 12.93it/s]
Running loglikelihood requests:  19%|█▉        | 3841/20084 [05:47<20:54, 12.94it/s]
Running loglikelihood requests:  19%|█▉        | 3905/20084 [05:52<20:49, 12.95it/s]
Running loglikelihood requests:  20%|█▉        | 3969/20084 [05:57<20:42, 12.97it/s]
Running loglikelihood requests:  20%|██        | 4033/20084 [06:02<20:36, 12.98it/s]
Running loglikelihood requests:  20%|██        | 4097/20084 [06:07<20:30, 13.00it/s]
Running loglikelihood requests:  21%|██        | 4161/20084 [06:11<20:24, 13.01it/s]
Running loglikelihood requests:  21%|██        | 4225/20084 [06:16<20:18, 13.02it/s]
Running loglikelihood requests:  21%|██▏       | 4289/20084 [06:21<20:11, 13.03it/s]
Running loglikelihood requests:  22%|██▏       | 4353/20084 [06:26<20:05, 13.05it/s]
Running loglikelihood requests:  22%|██▏       | 4417/20084 [06:31<19:59, 13.06it/s]
Running loglikelihood requests:  22%|██▏       | 4481/20084 [06:36<19:52, 13.08it/s]
Running loglikelihood requests:  23%|██▎       | 4545/20084 [06:41<19:45, 13.10it/s]
Running loglikelihood requests:  23%|██▎       | 4609/20084 [06:46<19:39, 13.12it/s]
Running loglikelihood requests:  23%|██▎       | 4673/20084 [06:51<19:33, 13.13it/s]
Running loglikelihood requests:  24%|██▎       | 4737/20084 [06:55<19:27, 13.14it/s]
Running loglikelihood requests:  24%|██▍       | 4801/20084 [07:00<19:21, 13.15it/s]
Running loglikelihood requests:  24%|██▍       | 4865/20084 [07:05<19:15, 13.17it/s]
Running loglikelihood requests:  25%|██▍       | 4929/20084 [07:10<19:09, 13.19it/s]
Running loglikelihood requests:  25%|██▍       | 4993/20084 [07:15<19:02, 13.21it/s]
Running loglikelihood requests:  25%|██▌       | 5057/20084 [07:20<18:57, 13.22it/s]
Running loglikelihood requests:  25%|██▌       | 5121/20084 [07:24<18:51, 13.22it/s]
Running loglikelihood requests:  26%|██▌       | 5185/20084 [07:29<18:45, 13.24it/s]
Running loglikelihood requests:  26%|██▌       | 5249/20084 [07:34<18:39, 13.25it/s]
Running loglikelihood requests:  26%|██▋       | 5313/20084 [07:39<18:33, 13.26it/s]
Running loglikelihood requests:  27%|██▋       | 5377/20084 [07:44<18:28, 13.27it/s]
Running loglikelihood requests:  27%|██▋       | 5441/20084 [07:48<18:21, 13.29it/s]
Running loglikelihood requests:  27%|██▋       | 5505/20084 [07:53<18:15, 13.31it/s]
Running loglikelihood requests:  28%|██▊       | 5569/20084 [07:58<18:09, 13.32it/s]
Running loglikelihood requests:  28%|██▊       | 5633/20084 [08:03<18:03, 13.34it/s]
Running loglikelihood requests:  28%|██▊       | 5697/20084 [08:08<17:54, 13.38it/s]
Running loglikelihood requests:  29%|██▊       | 5761/20084 [08:12<17:48, 13.40it/s]
Running loglikelihood requests:  29%|██▉       | 5825/20084 [08:17<17:41, 13.43it/s]
Running loglikelihood requests:  29%|██▉       | 5889/20084 [08:22<17:35, 13.45it/s]
Running loglikelihood requests:  30%|██▉       | 5953/20084 [08:27<17:30, 13.46it/s]
Running loglikelihood requests:  30%|██▉       | 6017/20084 [08:31<17:24, 13.47it/s]
Running loglikelihood requests:  30%|███       | 6081/20084 [08:36<17:18, 13.48it/s]
Running loglikelihood requests:  31%|███       | 6145/20084 [08:41<17:12, 13.50it/s]
Running loglikelihood requests:  31%|███       | 6209/20084 [08:46<17:06, 13.51it/s]
Running loglikelihood requests:  31%|███       | 6273/20084 [08:50<17:01, 13.53it/s]
Running loglikelihood requests:  32%|███▏      | 6337/20084 [08:55<16:55, 13.53it/s]
Running loglikelihood requests:  32%|███▏      | 6401/20084 [09:00<16:49, 13.55it/s]
Running loglikelihood requests:  32%|███▏      | 6465/20084 [09:04<16:44, 13.56it/s]
Running loglikelihood requests:  33%|███▎      | 6529/20084 [09:09<16:38, 13.57it/s]
Running loglikelihood requests:  33%|███▎      | 6593/20084 [09:14<16:36, 13.54it/s]
Running loglikelihood requests:  33%|███▎      | 6657/20084 [09:19<16:29, 13.57it/s]
Running loglikelihood requests:  33%|███▎      | 6721/20084 [09:23<16:23, 13.59it/s]
Running loglikelihood requests:  34%|███▍      | 6785/20084 [09:28<16:17, 13.61it/s]
Running loglikelihood requests:  34%|███▍      | 6849/20084 [09:33<16:11, 13.62it/s]
Running loglikelihood requests:  34%|███▍      | 6913/20084 [09:37<16:06, 13.63it/s]
Running loglikelihood requests:  35%|███▍      | 6977/20084 [09:42<16:00, 13.65it/s]
Running loglikelihood requests:  35%|███▌      | 7041/20084 [09:47<15:54, 13.67it/s]
Running loglikelihood requests:  35%|███▌      | 7105/20084 [09:51<15:50, 13.65it/s]
Running loglikelihood requests:  36%|███▌      | 7169/20084 [09:56<15:47, 13.64it/s]
Running loglikelihood requests:  36%|███▌      | 7233/20084 [10:01<15:40, 13.66it/s]
Running loglikelihood requests:  36%|███▋      | 7297/20084 [10:05<15:34, 13.68it/s]
Running loglikelihood requests:  37%|███▋      | 7361/20084 [10:10<15:28, 13.70it/s]
Running loglikelihood requests:  37%|███▋      | 7425/20084 [10:15<15:21, 13.73it/s]
Running loglikelihood requests:  37%|███▋      | 7489/20084 [10:19<15:16, 13.75it/s]
Running loglikelihood requests:  38%|███▊      | 7553/20084 [10:24<15:09, 13.77it/s]
Running loglikelihood requests:  38%|███▊      | 7617/20084 [10:29<15:03, 13.79it/s]
Running loglikelihood requests:  38%|███▊      | 7681/20084 [10:33<14:57, 13.82it/s]
Running loglikelihood requests:  39%|███▊      | 7745/20084 [10:38<14:52, 13.83it/s]
Running loglikelihood requests:  39%|███▉      | 7809/20084 [10:42<14:47, 13.84it/s]
Running loglikelihood requests:  39%|███▉      | 7873/20084 [10:47<14:41, 13.86it/s]
Running loglikelihood requests:  40%|███▉      | 7937/20084 [10:52<14:36, 13.86it/s]
Running loglikelihood requests:  40%|███▉      | 8001/20084 [10:56<14:31, 13.87it/s]
Running loglikelihood requests:  40%|████      | 8065/20084 [11:01<14:25, 13.89it/s]
Running loglikelihood requests:  40%|████      | 8129/20084 [11:05<14:20, 13.89it/s]
Running loglikelihood requests:  41%|████      | 8193/20084 [11:10<14:14, 13.91it/s]
Running loglikelihood requests:  41%|████      | 8257/20084 [11:15<14:09, 13.92it/s]
Running loglikelihood requests:  41%|████▏     | 8321/20084 [11:19<14:04, 13.94it/s]
Running loglikelihood requests:  42%|████▏     | 8385/20084 [11:24<13:58, 13.95it/s]
Running loglikelihood requests:  42%|████▏     | 8449/20084 [11:28<13:52, 13.97it/s]
Running loglikelihood requests:  42%|████▏     | 8513/20084 [11:33<13:47, 13.98it/s]
Running loglikelihood requests:  43%|████▎     | 8577/20084 [11:37<13:42, 13.99it/s]
Running loglikelihood requests:  43%|████▎     | 8641/20084 [11:42<13:37, 14.00it/s]
Running loglikelihood requests:  43%|████▎     | 8705/20084 [11:47<13:32, 14.01it/s]
Running loglikelihood requests:  44%|████▎     | 8769/20084 [11:51<13:27, 14.02it/s]
Running loglikelihood requests:  44%|████▍     | 8833/20084 [11:56<13:22, 14.03it/s]
Running loglikelihood requests:  44%|████▍     | 8897/20084 [12:00<13:17, 14.03it/s]
Running loglikelihood requests:  45%|████▍     | 8961/20084 [12:05<13:12, 14.04it/s]
Running loglikelihood requests:  45%|████▍     | 9025/20084 [12:09<13:11, 13.97it/s]
Running loglikelihood requests:  45%|████▌     | 9089/20084 [12:14<13:05, 13.99it/s]
Running loglikelihood requests:  46%|████▌     | 9153/20084 [12:19<12:59, 14.02it/s]
Running loglikelihood requests:  46%|████▌     | 9217/20084 [12:23<12:52, 14.06it/s]
Running loglikelihood requests:  46%|████▌     | 9281/20084 [12:28<12:46, 14.09it/s]
Running loglikelihood requests:  47%|████▋     | 9345/20084 [12:32<12:40, 14.11it/s]
Running loglikelihood requests:  47%|████▋     | 9409/20084 [12:37<12:35, 14.12it/s]
Running loglikelihood requests:  47%|████▋     | 9473/20084 [12:41<12:30, 14.13it/s]
Running loglikelihood requests:  47%|████▋     | 9537/20084 [12:46<12:25, 14.15it/s]
Running loglikelihood requests:  48%|████▊     | 9601/20084 [12:50<12:21, 14.14it/s]
Running loglikelihood requests:  48%|████▊     | 9665/20084 [12:55<12:15, 14.17it/s]
Running loglikelihood requests:  48%|████▊     | 9729/20084 [12:59<12:10, 14.18it/s]
Running loglikelihood requests:  49%|████▉     | 9793/20084 [13:04<12:04, 14.21it/s]
Running loglikelihood requests:  49%|████▉     | 9857/20084 [13:08<11:58, 14.23it/s]
Running loglikelihood requests:  49%|████▉     | 9921/20084 [13:13<11:53, 14.24it/s]
Running loglikelihood requests:  50%|████▉     | 9985/20084 [13:17<11:48, 14.25it/s]
Running loglikelihood requests:  50%|█████     | 10049/20084 [13:22<11:42, 14.28it/s]
Running loglikelihood requests:  50%|█████     | 10113/20084 [13:26<11:38, 14.28it/s]
Running loglikelihood requests:  51%|█████     | 10177/20084 [13:31<11:33, 14.29it/s]
Running loglikelihood requests:  51%|█████     | 10241/20084 [13:35<11:28, 14.29it/s]
Running loglikelihood requests:  51%|█████▏    | 10305/20084 [13:40<11:23, 14.30it/s]
Running loglikelihood requests:  52%|█████▏    | 10369/20084 [13:44<11:18, 14.31it/s]
Running loglikelihood requests:  52%|█████▏    | 10433/20084 [13:48<11:13, 14.32it/s]
Running loglikelihood requests:  52%|█████▏    | 10497/20084 [13:53<11:08, 14.34it/s]
Running loglikelihood requests:  53%|█████▎    | 10561/20084 [13:57<11:03, 14.35it/s]
Running loglikelihood requests:  53%|█████▎    | 10625/20084 [14:02<10:58, 14.36it/s]
Running loglikelihood requests:  53%|█████▎    | 10689/20084 [14:06<10:53, 14.37it/s]
Running loglikelihood requests:  54%|█████▎    | 10753/20084 [14:11<10:48, 14.39it/s]
Running loglikelihood requests:  54%|█████▍    | 10817/20084 [14:15<10:43, 14.41it/s]
Running loglikelihood requests:  54%|█████▍    | 10881/20084 [14:20<10:38, 14.42it/s]
Running loglikelihood requests:  54%|█████▍    | 10945/20084 [14:24<10:33, 14.43it/s]
Running loglikelihood requests:  55%|█████▍    | 11009/20084 [14:28<10:28, 14.45it/s]
Running loglikelihood requests:  55%|█████▌    | 11073/20084 [14:33<10:23, 14.46it/s]
Running loglikelihood requests:  55%|█████▌    | 11137/20084 [14:37<10:18, 14.46it/s]
Running loglikelihood requests:  56%|█████▌    | 11201/20084 [14:42<10:13, 14.47it/s]
Running loglikelihood requests:  56%|█████▌    | 11265/20084 [14:46<10:09, 14.47it/s]
Running loglikelihood requests:  56%|█████▋    | 11329/20084 [14:50<10:04, 14.48it/s]
Running loglikelihood requests:  57%|█████▋    | 11393/20084 [14:55<09:59, 14.50it/s]
Running loglikelihood requests:  57%|█████▋    | 11457/20084 [14:59<09:54, 14.52it/s]
Running loglikelihood requests:  57%|█████▋    | 11521/20084 [15:04<09:49, 14.54it/s]
Running loglikelihood requests:  58%|█████▊    | 11585/20084 [15:08<09:44, 14.55it/s]
Running loglikelihood requests:  58%|█████▊    | 11649/20084 [15:12<09:39, 14.56it/s]
Running loglikelihood requests:  58%|█████▊    | 11713/20084 [15:17<09:34, 14.58it/s]
Running loglikelihood requests:  59%|█████▊    | 11777/20084 [15:21<09:29, 14.59it/s]
Running loglikelihood requests:  59%|█████▉    | 11841/20084 [15:26<09:24, 14.61it/s]
Running loglikelihood requests:  59%|█████▉    | 11905/20084 [15:30<09:19, 14.61it/s]
Running loglikelihood requests:  60%|█████▉    | 11969/20084 [15:34<09:14, 14.63it/s]
Running loglikelihood requests:  60%|█████▉    | 12033/20084 [15:39<09:09, 14.64it/s]
Running loglikelihood requests:  60%|██████    | 12097/20084 [15:43<09:04, 14.66it/s]
Running loglikelihood requests:  61%|██████    | 12161/20084 [15:47<08:59, 14.69it/s]
Running loglikelihood requests:  61%|██████    | 12225/20084 [15:52<08:54, 14.70it/s]
Running loglikelihood requests:  61%|██████    | 12289/20084 [15:56<08:49, 14.72it/s]
Running loglikelihood requests:  62%|██████▏   | 12353/20084 [16:00<08:44, 14.73it/s]
Running loglikelihood requests:  62%|██████▏   | 12417/20084 [16:05<08:40, 14.74it/s]
Running loglikelihood requests:  62%|██████▏   | 12481/20084 [16:09<08:35, 14.76it/s]
Running loglikelihood requests:  62%|██████▏   | 12545/20084 [16:13<08:30, 14.77it/s]
Running loglikelihood requests:  63%|██████▎   | 12609/20084 [16:18<08:25, 14.79it/s]
Running loglikelihood requests:  63%|██████▎   | 12673/20084 [16:22<08:20, 14.80it/s]
Running loglikelihood requests:  63%|██████▎   | 12737/20084 [16:26<08:15, 14.81it/s]
Running loglikelihood requests:  64%|██████▎   | 12801/20084 [16:31<08:11, 14.83it/s]
Running loglikelihood requests:  64%|██████▍   | 12865/20084 [16:35<08:06, 14.84it/s]
Running loglikelihood requests:  64%|██████▍   | 12929/20084 [16:39<08:01, 14.86it/s]
Running loglikelihood requests:  65%|██████▍   | 12993/20084 [16:43<07:56, 14.87it/s]
Running loglikelihood requests:  65%|██████▌   | 13057/20084 [16:48<07:52, 14.89it/s]
Running loglikelihood requests:  65%|██████▌   | 13121/20084 [16:52<07:47, 14.89it/s]
Running loglikelihood requests:  66%|██████▌   | 13185/20084 [16:56<07:42, 14.91it/s]
Running loglikelihood requests:  66%|██████▌   | 13249/20084 [17:01<07:38, 14.92it/s]
Running loglikelihood requests:  66%|██████▋   | 13313/20084 [17:05<07:33, 14.93it/s]
Running loglikelihood requests:  67%|██████▋   | 13377/20084 [17:09<07:28, 14.94it/s]
Running loglikelihood requests:  67%|██████▋   | 13441/20084 [17:13<07:24, 14.95it/s]
Running loglikelihood requests:  67%|██████▋   | 13505/20084 [17:18<07:19, 14.97it/s]
Running loglikelihood requests:  68%|██████▊   | 13569/20084 [17:22<07:15, 14.97it/s]
Running loglikelihood requests:  68%|██████▊   | 13633/20084 [17:26<07:10, 14.99it/s]
Running loglikelihood requests:  68%|██████▊   | 13697/20084 [17:31<07:05, 15.00it/s]
Running loglikelihood requests:  69%|██████▊   | 13761/20084 [17:35<07:01, 15.02it/s]
Running loglikelihood requests:  69%|██████▉   | 13825/20084 [17:39<06:56, 15.02it/s]
Running loglikelihood requests:  69%|██████▉   | 13889/20084 [17:43<06:52, 15.03it/s]
Running loglikelihood requests:  69%|██████▉   | 13953/20084 [17:48<06:47, 15.05it/s]
Running loglikelihood requests:  70%|██████▉   | 14017/20084 [17:52<06:42, 15.06it/s]
Running loglikelihood requests:  70%|███████   | 14081/20084 [17:56<06:38, 15.08it/s]
Running loglikelihood requests:  70%|███████   | 14145/20084 [18:00<06:33, 15.09it/s]
Running loglikelihood requests:  71%|███████   | 14209/20084 [18:04<06:28, 15.11it/s]
Running loglikelihood requests:  71%|███████   | 14273/20084 [18:09<06:24, 15.13it/s]
Running loglikelihood requests:  71%|███████▏  | 14337/20084 [18:13<06:19, 15.13it/s]
Running loglikelihood requests:  72%|███████▏  | 14401/20084 [18:17<06:15, 15.15it/s]
Running loglikelihood requests:  72%|███████▏  | 14465/20084 [18:21<06:10, 15.17it/s]
Running loglikelihood requests:  72%|███████▏  | 14529/20084 [18:26<06:05, 15.19it/s]
Running loglikelihood requests:  73%|███████▎  | 14593/20084 [18:30<06:00, 15.22it/s]
Running loglikelihood requests:  73%|███████▎  | 14657/20084 [18:34<05:56, 15.23it/s]
Running loglikelihood requests:  73%|███████▎  | 14721/20084 [18:38<05:51, 15.24it/s]
Running loglikelihood requests:  74%|███████▎  | 14785/20084 [18:42<05:47, 15.26it/s]
Running loglikelihood requests:  74%|███████▍  | 14849/20084 [18:46<05:42, 15.28it/s]
Running loglikelihood requests:  74%|███████▍  | 14913/20084 [18:51<05:38, 15.30it/s]
Running loglikelihood requests:  75%|███████▍  | 14977/20084 [18:55<05:33, 15.30it/s]
Running loglikelihood requests:  75%|███████▍  | 15041/20084 [18:59<05:28, 15.33it/s]
Running loglikelihood requests:  75%|███████▌  | 15105/20084 [19:03<05:24, 15.33it/s]
Running loglikelihood requests:  76%|███████▌  | 15169/20084 [19:07<05:19, 15.36it/s]
Running loglikelihood requests:  76%|███████▌  | 15233/20084 [19:11<05:15, 15.38it/s]
Running loglikelihood requests:  76%|███████▌  | 15297/20084 [19:16<05:10, 15.40it/s]
Running loglikelihood requests:  76%|███████▋  | 15361/20084 [19:20<05:06, 15.41it/s]
Running loglikelihood requests:  77%|███████▋  | 15425/20084 [19:24<05:02, 15.42it/s]
Running loglikelihood requests:  77%|███████▋  | 15489/20084 [19:28<04:57, 15.43it/s]
Running loglikelihood requests:  77%|███████▋  | 15553/20084 [19:32<04:53, 15.44it/s]
Running loglikelihood requests:  78%|███████▊  | 15617/20084 [19:36<04:48, 15.51it/s]
Running loglikelihood requests:  78%|███████▊  | 15681/20084 [19:40<04:43, 15.53it/s]
Running loglikelihood requests:  78%|███████▊  | 15745/20084 [19:44<04:38, 15.56it/s]
Running loglikelihood requests:  79%|███████▊  | 15809/20084 [19:49<04:34, 15.58it/s]
Running loglikelihood requests:  79%|███████▉  | 15873/20084 [19:53<04:29, 15.61it/s]
Running loglikelihood requests:  79%|███████▉  | 15937/20084 [19:57<04:25, 15.63it/s]
Running loglikelihood requests:  80%|███████▉  | 16001/20084 [20:01<04:20, 15.65it/s]
Running loglikelihood requests:  80%|███████▉  | 16065/20084 [20:05<04:16, 15.67it/s]
Running loglikelihood requests:  80%|████████  | 16129/20084 [20:09<04:11, 15.70it/s]
Running loglikelihood requests:  81%|████████  | 16193/20084 [20:13<04:07, 15.72it/s]
Running loglikelihood requests:  81%|████████  | 16257/20084 [20:17<04:03, 15.74it/s]
Running loglikelihood requests:  81%|████████▏ | 16321/20084 [20:21<03:58, 15.76it/s]
Running loglikelihood requests:  82%|████████▏ | 16385/20084 [20:25<03:54, 15.78it/s]
Running loglikelihood requests:  82%|████████▏ | 16449/20084 [20:29<03:49, 15.81it/s]
Running loglikelihood requests:  82%|████████▏ | 16513/20084 [20:33<03:45, 15.84it/s]
Running loglikelihood requests:  83%|████████▎ | 16577/20084 [20:37<03:41, 15.86it/s]
Running loglikelihood requests:  83%|████████▎ | 16641/20084 [20:41<03:36, 15.88it/s]
Running loglikelihood requests:  83%|████████▎ | 16705/20084 [20:45<03:32, 15.91it/s]
Running loglikelihood requests:  83%|████████▎ | 16769/20084 [20:49<03:28, 15.93it/s]
Running loglikelihood requests:  84%|████████▍ | 16833/20084 [20:53<03:23, 15.96it/s]
Running loglikelihood requests:  84%|████████▍ | 16897/20084 [20:57<03:19, 15.98it/s]
Running loglikelihood requests:  84%|████████▍ | 16961/20084 [21:01<03:15, 16.01it/s]
Running loglikelihood requests:  85%|████████▍ | 17025/20084 [21:05<03:10, 16.03it/s]
Running loglikelihood requests:  85%|████████▌ | 17089/20084 [21:09<03:06, 16.06it/s]
Running loglikelihood requests:  85%|████████▌ | 17153/20084 [21:13<03:02, 16.08it/s]
Running loglikelihood requests:  86%|████████▌ | 17217/20084 [21:17<02:58, 16.10it/s]
Running loglikelihood requests:  86%|████████▌ | 17281/20084 [21:21<02:53, 16.13it/s]
Running loglikelihood requests:  86%|████████▋ | 17345/20084 [21:25<02:49, 16.15it/s]
Running loglikelihood requests:  87%|████████▋ | 17409/20084 [21:29<02:45, 16.18it/s]
Running loglikelihood requests:  87%|████████▋ | 17473/20084 [21:33<02:41, 16.18it/s]
Running loglikelihood requests:  87%|████████▋ | 17537/20084 [21:37<02:37, 16.21it/s]
Running loglikelihood requests:  88%|████████▊ | 17601/20084 [21:41<02:32, 16.24it/s]
Running loglikelihood requests:  88%|████████▊ | 17665/20084 [21:45<02:28, 16.27it/s]
Running loglikelihood requests:  88%|████████▊ | 17729/20084 [21:49<02:24, 16.31it/s]
Running loglikelihood requests:  89%|████████▊ | 17793/20084 [21:52<02:20, 16.34it/s]
Running loglikelihood requests:  89%|████████▉ | 17857/20084 [21:56<02:15, 16.38it/s]
Running loglikelihood requests:  89%|████████▉ | 17921/20084 [22:00<02:11, 16.41it/s]
Running loglikelihood requests:  90%|████████▉ | 17985/20084 [22:04<02:07, 16.45it/s]
Running loglikelihood requests:  90%|████████▉ | 18049/20084 [22:08<02:03, 16.48it/s]
Running loglikelihood requests:  90%|█████████ | 18113/20084 [22:12<01:59, 16.52it/s]
Running loglikelihood requests:  91%|█████████ | 18177/20084 [22:16<01:55, 16.55it/s]
Running loglikelihood requests:  91%|█████████ | 18241/20084 [22:19<01:51, 16.60it/s]
Running loglikelihood requests:  91%|█████████ | 18305/20084 [22:23<01:46, 16.64it/s]
Running loglikelihood requests:  91%|█████████▏| 18369/20084 [22:27<01:42, 16.68it/s]
Running loglikelihood requests:  92%|█████████▏| 18433/20084 [22:31<01:38, 16.72it/s]
Running loglikelihood requests:  92%|█████████▏| 18497/20084 [22:35<01:34, 16.76it/s]
Running loglikelihood requests:  92%|█████████▏| 18561/20084 [22:38<01:30, 16.82it/s]
Running loglikelihood requests:  93%|█████████▎| 18625/20084 [22:42<01:26, 16.86it/s]
Running loglikelihood requests:  93%|█████████▎| 18689/20084 [22:46<01:22, 16.90it/s]
Running loglikelihood requests:  93%|█████████▎| 18753/20084 [22:50<01:18, 16.95it/s]
Running loglikelihood requests:  94%|█████████▎| 18817/20084 [22:53<01:14, 17.02it/s]
Running loglikelihood requests:  94%|█████████▍| 18881/20084 [22:57<01:10, 17.08it/s]
Running loglikelihood requests:  94%|█████████▍| 18945/20084 [23:01<01:06, 17.12it/s]
Running loglikelihood requests:  95%|█████████▍| 19009/20084 [23:05<01:02, 17.17it/s]
Running loglikelihood requests:  95%|█████████▍| 19073/20084 [23:08<00:58, 17.25it/s]
Running loglikelihood requests:  95%|█████████▌| 19137/20084 [23:12<00:54, 17.34it/s]
Running loglikelihood requests:  96%|█████████▌| 19201/20084 [23:16<00:50, 17.43it/s]
Running loglikelihood requests:  96%|█████████▌| 19265/20084 [23:19<00:46, 17.52it/s]
Running loglikelihood requests:  96%|█████████▌| 19329/20084 [23:23<00:42, 17.61it/s]
Running loglikelihood requests:  97%|█████████▋| 19393/20084 [23:26<00:39, 17.71it/s]
Running loglikelihood requests:  97%|█████████▋| 19457/20084 [23:30<00:35, 17.79it/s]
Running loglikelihood requests:  97%|█████████▋| 19521/20084 [23:33<00:31, 17.88it/s]
Running loglikelihood requests:  98%|█████████▊| 19585/20084 [23:37<00:27, 17.98it/s]
Running loglikelihood requests:  98%|█████████▊| 19649/20084 [23:40<00:24, 18.08it/s]
Running loglikelihood requests:  98%|█████████▊| 19713/20084 [23:44<00:20, 18.23it/s]
Running loglikelihood requests:  98%|█████████▊| 19777/20084 [23:47<00:16, 18.41it/s]
Running loglikelihood requests:  99%|█████████▉| 19841/20084 [23:51<00:13, 18.63it/s]
Running loglikelihood requests:  99%|█████████▉| 19905/20084 [23:54<00:09, 18.84it/s]
Running loglikelihood requests:  99%|█████████▉| 19969/20084 [23:57<00:06, 19.16it/s]
Running loglikelihood requests: 100%|█████████▉| 20033/20084 [24:00<00:02, 20.68it/s]
Running loglikelihood requests: 100%|██████████| 20084/20084 [24:00<00:00, 13.95it/s]
n136-128-154:2250936:2253810 [1] NCCL INFO Channel 00/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2253810 [1] NCCL INFO Channel 01/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2253810 [1] NCCL INFO Channel 02/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2253810 [1] NCCL INFO Channel 03/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2253810 [1] NCCL INFO Channel 04/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2253810 [1] NCCL INFO Channel 05/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2253810 [1] NCCL INFO Channel 06/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2253810 [1] NCCL INFO Channel 07/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2253810 [1] NCCL INFO Channel 08/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2253810 [1] NCCL INFO Channel 09/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2253810 [1] NCCL INFO Channel 10/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2253810 [1] NCCL INFO Channel 11/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2253810 [1] NCCL INFO Channel 12/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2253810 [1] NCCL INFO Channel 13/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2253810 [1] NCCL INFO Channel 14/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2253810 [1] NCCL INFO Channel 15/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2253810 [1] NCCL INFO Channel 16/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2253810 [1] NCCL INFO Channel 17/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2253810 [1] NCCL INFO Channel 18/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2253810 [1] NCCL INFO Channel 19/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2253810 [1] NCCL INFO Channel 20/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2253810 [1] NCCL INFO Channel 21/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2253810 [1] NCCL INFO Channel 22/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2253810 [1] NCCL INFO Channel 23/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2253810 [1] NCCL INFO Channel 24/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2253810 [1] NCCL INFO Channel 25/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2253810 [1] NCCL INFO Channel 26/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2253810 [1] NCCL INFO Channel 27/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2253810 [1] NCCL INFO Channel 28/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2253810 [1] NCCL INFO Channel 29/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2253810 [1] NCCL INFO Channel 30/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2250936:2253810 [1] NCCL INFO Channel 31/1 : 1[7] -> 0[6] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:2250936:2253816 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2250936:2253816 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2250936:2253816 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2250936:2253816 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2250936:2253816 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2250936:2253816 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2250936:2251219 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-09:20:36:05 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 10, batch_size: auto (64)
|  Tasks  |Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|---------|------:|------|-----:|--------|---|-----:|---|-----:|
|hellaswag|      1|none  |    10|acc     |↑  |0.3834|±  |0.0049|
|         |       |none  |    10|acc_norm|↑  |0.5266|±  |0.0050|

[rank0]:[W1209 20:36:06.916712871 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:2250935:2253865 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2250935:2253865 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2250935:2253865 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2250935:2251221 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2250935:2253865 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2250935:2253865 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2250935:2253865 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2250936:2251219 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2250936:2253816 [1] NCCL INFO comm 0x13dbbd10 rank 1 nranks 2 cudaDev 1 busId c9000 - Abort COMPLETE
n136-128-154:2250935:2253865 [0] NCCL INFO comm 0x140bf390 rank 0 nranks 2 cudaDev 0 busId c5000 - Abort COMPLETE
任务 hellaswag 评估完成！
Execution time: 2748.52 seconds