nohup: ignoring input
Namespace(save_dir='/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg', self_attn_layer_to_quant='4 1 2 8 23', mlp_layer_to_quant='27 16 19 17 25', model_id='/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-7B', cuda_id=1)
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:07<00:21,  7.30s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:14<00:14,  7.10s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:21<00:07,  7.06s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:28<00:00,  7.16s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:28<00:00,  7.14s/it]
Qwen2ForCausalLM(
  (model): Qwen2Model(
    (embed_tokens): Embedding(152064, 3584)
    (layers): ModuleList(
      (0-27): 28 x Qwen2DecoderLayer(
        (self_attn): Qwen2Attention(
          (q_proj): Linear(in_features=3584, out_features=3584, bias=True)
          (k_proj): Linear(in_features=3584, out_features=512, bias=True)
          (v_proj): Linear(in_features=3584, out_features=512, bias=True)
          (o_proj): Linear(in_features=3584, out_features=3584, bias=False)
        )
        (mlp): Qwen2MLP(
          (gate_proj): Linear(in_features=3584, out_features=18944, bias=False)
          (up_proj): Linear(in_features=3584, out_features=18944, bias=False)
          (down_proj): Linear(in_features=18944, out_features=3584, bias=False)
          (act_fn): SiLUActivation()
        )
        (input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
        (post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
      )
    )
    (norm): Qwen2RMSNorm((3584,), eps=1e-06)
    (rotary_emb): Qwen2RotaryEmbedding()
  )
  (lm_head): Linear(in_features=3584, out_features=152064, bias=False)
)
Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation.
Once upon a time, there was a young girl named Lily who lived in a small village. She was a curious and adventurous girl who loved exploring the woods and discovering new things. One day, while wandering through the forest, she stumbled upon a magical tree. The tree had shimmering leaves and a trunk that glowed with a soft light. As she approached the tree, she noticed that the branches were adorned with tiny, glowing fruits. Lily couldn't resist the temptation and took one of the fruits into her hand.
quanting ... 
layers.0.self_attn 4 bit quant 
layers.0.mlp 4 bit quant 
layers.1.self_attn 2 bit quant 
layers.1.mlp 4 bit quant 
layers.2.self_attn 2 bit quant 
layers.2.mlp 4 bit quant 
layers.3.self_attn 4 bit quant 
layers.3.mlp 4 bit quant 
layers.4.self_attn 2 bit quant 
layers.4.mlp 4 bit quant 
layers.5.self_attn 4 bit quant 
layers.5.mlp 4 bit quant 
layers.6.self_attn 4 bit quant 
layers.6.mlp 4 bit quant 
layers.7.self_attn 4 bit quant 
layers.7.mlp 4 bit quant 
layers.8.self_attn 2 bit quant 
layers.8.mlp 4 bit quant 
layers.9.self_attn 4 bit quant 
layers.9.mlp 4 bit quant 
layers.10.self_attn 4 bit quant 
layers.10.mlp 4 bit quant 
layers.11.self_attn 4 bit quant 
layers.11.mlp 4 bit quant 
layers.12.self_attn 4 bit quant 
layers.12.mlp 4 bit quant 
layers.13.self_attn 4 bit quant 
layers.13.mlp 4 bit quant 
layers.14.self_attn 4 bit quant 
layers.14.mlp 4 bit quant 
layers.15.self_attn 4 bit quant 
layers.15.mlp 4 bit quant 
layers.16.self_attn 4 bit quant 
layers.16.mlp 2 bit quant 
layers.17.self_attn 4 bit quant 
layers.17.mlp 2 bit quant 
layers.18.self_attn 4 bit quant 
layers.18.mlp 4 bit quant 
layers.19.self_attn 4 bit quant 
layers.19.mlp 2 bit quant 
layers.20.self_attn 4 bit quant 
layers.20.mlp 4 bit quant 
layers.21.self_attn 4 bit quant 
layers.21.mlp 4 bit quant 
layers.22.self_attn 4 bit quant 
layers.22.mlp 4 bit quant 
layers.23.self_attn 2 bit quant 
layers.23.mlp 4 bit quant 
layers.24.self_attn 4 bit quant 
layers.24.mlp 4 bit quant 
layers.25.self_attn 4 bit quant 
layers.25.mlp 2 bit quant 
layers.26.self_attn 4 bit quant 
layers.26.mlp 4 bit quant 
layers.27.self_attn 4 bit quant 
layers.27.mlp 2 bit quant 
quanted
Qwen2ForCausalLM(
  (model): Qwen2Model(
    (embed_tokens): Embedding(152064, 3584)
    (layers): ModuleList(
      (0-27): 28 x Qwen2DecoderLayer(
        (self_attn): Qwen2Attention(
          (q_proj): W8A16Linear(3584, 3584, bias=True, weight_quant=per_channel)
          (k_proj): W8A16Linear(3584, 512, bias=True, weight_quant=per_channel)
          (v_proj): W8A16Linear(3584, 512, bias=True, weight_quant=per_channel)
          (o_proj): W8A16Linear(3584, 3584, bias=False, weight_quant=per_channel)
        )
        (mlp): Qwen2MLP(
          (gate_proj): W8A16Linear(3584, 18944, bias=False, weight_quant=per_channel)
          (up_proj): W8A16Linear(3584, 18944, bias=False, weight_quant=per_channel)
          (down_proj): W8A16Linear(18944, 3584, bias=False, weight_quant=per_channel)
          (act_fn): SiLUActivation()
        )
        (input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
        (post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
      )
    )
    (norm): Qwen2RMSNorm((3584,), eps=1e-06)
    (rotary_emb): Qwen2RotaryEmbedding()
  )
  (lm_head): Linear(in_features=3584, out_features=152064, bias=False)
)
Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation.
Once upon a time in a place called, "ret ret tret tret tret tret tret tret tret tret tret tret tret tret tret tret tret tret tret tret tret tret tret tret tret tret tret tret tret tret tret tret tret tret tret tret tret tret tret tret tret tret tret tret tret tret tret tret

==================================================
开始评估：任务=winogrande | 少样本数=5 | 模型=Qwen2.5-7B-quantization-fg
输出路径：results2/Qwen2.5-7B-quantization-fg/fg/winogrande.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-05:18:04:31 INFO     [__main__:440] Selected Tasks: ['winogrande']
2025-12-05:18:04:31 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-05:18:04:31 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-05:18:04:31 INFO     [__main__:440] Selected Tasks: ['winogrande']
2025-12-05:18:04:31 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-05:18:04:31 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-05:18:04:32 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-05:18:04:32 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-05:18:04:33 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.17it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.02it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.12it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.01it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.18it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.69it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.44it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.01s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.43it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.24it/s]
2025-12-05:18:04:52 WARNING  [evaluator:309] Overwriting default num_fewshot of winogrande from None to 5
2025-12-05:18:04:52 INFO     [api.task:434] Building contexts for winogrande on rank 1...

  0%|          | 0/633 [00:00<?, ?it/s]
100%|██████████| 633/633 [00:00<00:00, 28781.99it/s]
2025-12-05:18:05:03 WARNING  [evaluator:309] Overwriting default num_fewshot of winogrande from None to 5
2025-12-05:18:05:03 INFO     [api.task:434] Building contexts for winogrande on rank 0...

  0%|          | 0/634 [00:00<?, ?it/s]
100%|██████████| 634/634 [00:00<00:00, 29217.68it/s]
n136-128-154:1658159:1658159 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1658159:1658159 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1658159:1658159 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:1658159:1658159 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:1658159:1658159 [0] NCCL INFO Comm config Blocking set to 1
n136-128-154:1658160:1658160 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:1658160:1658160 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1658160:1658160 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1658160:1658160 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:1658160:1658160 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:1658159:1658564 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:1658159:1658564 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:1658160:1658565 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:1658160:1658565 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:1658159:1658564 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1658159:1658564 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1658159:1658564 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:1658160:1658565 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1658160:1658565 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1658160:1658565 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:1658160:1658565 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:1658160:1658565 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:1658160:1658565 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:1658159:1658564 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:1658159:1658564 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:1658159:1658564 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:1658160:1658565 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:1658160:1658565 [1] NCCL INFO Using network IB
n136-128-154:1658159:1658564 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:1658159:1658564 [0] NCCL INFO Using network IB
n136-128-154:1658159:1658564 [0] NCCL INFO ncclCommInitRankConfig comm 0x10d4cd80 rank 0 nranks 2 cudaDev 0 nvmlDev 1 busId 16000 commId 0x578a0970f19dc939 - Init START
n136-128-154:1658160:1658565 [1] NCCL INFO ncclCommInitRankConfig comm 0x10a6b100 rank 1 nranks 2 cudaDev 1 nvmlDev 2 busId 4a000 commId 0x578a0970f19dc939 - Init START
n136-128-154:1658160:1658565 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:1658159:1658564 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:1658160:1658565 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:1658160:1658565 [1] NCCL INFO Retrieving state for IB
n136-128-154:1658160:1658565 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:1658160:1658565 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:1658159:1658564 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:1658159:1658564 [0] NCCL INFO Retrieving state for IB
n136-128-154:1658159:1658564 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:1658160:1658565 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:1658159:1658564 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:1658160:1658565 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:1658159:1658564 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:1658160:1658565 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:1658159:1658564 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:1658159:1658564 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:1658159:1658564 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1658160:1658565 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1658159:1658564 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1658160:1658565 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1658159:1658564 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1658160:1658565 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1658159:1658564 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1658160:1658565 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1658159:1658564 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:1658159:1658564 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:1658159:1658564 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:1658159:1658564 [0] NCCL INFO               + PCI[24.0] - PCI/0-14000 (1000c01010de13b8)
n136-128-154:1658159:1658564 [0] NCCL INFO                             + PCI[24.0] - GPU/0-16000 (0)
n136-128-154:1658159:1658564 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1658159:1658564 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:1658159:1658564 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:1658159:1658564 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:1658159:1658564 [0] NCCL INFO               + PCI[24.0] - PCI/0-48000 (1000c01010de13b8)
n136-128-154:1658159:1658564 [0] NCCL INFO                             + PCI[24.0] - GPU/0-4a000 (1)
n136-128-154:1658159:1658564 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1658159:1658564 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:1658159:1658564 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:1658159:1658564 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:1658159:1658564 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:1658159:1658564 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:1658159:1658564 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:1658159:1658564 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:1658159:1658564 [0] NCCL INFO ==========================================
n136-128-154:1658159:1658564 [0] NCCL INFO GPU/0-16000 :GPU/0-16000 (0/5000.0/LOC) GPU/0-4a000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1658159:1658564 [0] NCCL INFO GPU/0-4a000 :GPU/0-16000 (2/240.0/NVL) GPU/0-4a000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1658159:1658564 [0] NCCL INFO Setting affinity for GPU 1 to 1-31,65-95
n136-128-154:1658160:1658565 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:1658160:1658565 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:1658160:1658565 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:1658160:1658565 [1] NCCL INFO               + PCI[24.0] - PCI/0-14000 (1000c01010de13b8)
n136-128-154:1658159:1658564 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:1658160:1658565 [1] NCCL INFO                             + PCI[24.0] - GPU/0-16000 (0)
n136-128-154:1658159:1658564 [0] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658160:1658565 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1658159:1658564 [0] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658160:1658565 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:1658159:1658564 [0] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658160:1658565 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:1658159:1658564 [0] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658160:1658565 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:1658159:1658564 [0] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658160:1658565 [1] NCCL INFO               + PCI[24.0] - PCI/0-48000 (1000c01010de13b8)
n136-128-154:1658159:1658564 [0] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658160:1658565 [1] NCCL INFO                             + PCI[24.0] - GPU/0-4a000 (1)
n136-128-154:1658159:1658564 [0] NCCL INFO  6 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658160:1658565 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1658159:1658564 [0] NCCL INFO  7 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658160:1658565 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:1658159:1658564 [0] NCCL INFO  8 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658160:1658565 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:1658159:1658564 [0] NCCL INFO  9 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658160:1658565 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:1658159:1658564 [0] NCCL INFO 10 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658160:1658565 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:1658159:1658564 [0] NCCL INFO 11 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658160:1658565 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:1658160:1658565 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:1658160:1658565 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:1658160:1658565 [1] NCCL INFO ==========================================
n136-128-154:1658160:1658565 [1] NCCL INFO GPU/0-16000 :GPU/0-16000 (0/5000.0/LOC) GPU/0-4a000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1658160:1658565 [1] NCCL INFO GPU/0-4a000 :GPU/0-16000 (2/240.0/NVL) GPU/0-4a000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1658160:1658565 [1] NCCL INFO Setting affinity for GPU 2 to 1-31,65-95
n136-128-154:1658159:1658564 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:1658159:1658564 [0] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658159:1658564 [0] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658159:1658564 [0] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658159:1658564 [0] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658159:1658564 [0] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658159:1658564 [0] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658159:1658564 [0] NCCL INFO  6 : GPU/0-4a000 GPU/0-16000
n136-128-154:1658159:1658564 [0] NCCL INFO  7 : GPU/0-4a000 GPU/0-16000
n136-128-154:1658159:1658564 [0] NCCL INFO  8 : GPU/0-4a000 GPU/0-16000
n136-128-154:1658159:1658564 [0] NCCL INFO  9 : GPU/0-4a000 GPU/0-16000
n136-128-154:1658159:1658564 [0] NCCL INFO 10 : GPU/0-4a000 GPU/0-16000
n136-128-154:1658159:1658564 [0] NCCL INFO 11 : GPU/0-4a000 GPU/0-16000
n136-128-154:1658160:1658565 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:1658160:1658565 [1] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658160:1658565 [1] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658160:1658565 [1] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658160:1658565 [1] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658160:1658565 [1] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658160:1658565 [1] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658160:1658565 [1] NCCL INFO  6 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658160:1658565 [1] NCCL INFO  7 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658160:1658565 [1] NCCL INFO  8 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658160:1658565 [1] NCCL INFO  9 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658160:1658565 [1] NCCL INFO 10 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658160:1658565 [1] NCCL INFO 11 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658160:1658565 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:1658160:1658565 [1] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658160:1658565 [1] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658160:1658565 [1] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658160:1658565 [1] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658160:1658565 [1] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658160:1658565 [1] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658160:1658565 [1] NCCL INFO  6 : GPU/0-4a000 GPU/0-16000
n136-128-154:1658160:1658565 [1] NCCL INFO  7 : GPU/0-4a000 GPU/0-16000
n136-128-154:1658160:1658565 [1] NCCL INFO  8 : GPU/0-4a000 GPU/0-16000
n136-128-154:1658160:1658565 [1] NCCL INFO  9 : GPU/0-4a000 GPU/0-16000
n136-128-154:1658160:1658565 [1] NCCL INFO 10 : GPU/0-4a000 GPU/0-16000
n136-128-154:1658160:1658565 [1] NCCL INFO 11 : GPU/0-4a000 GPU/0-16000
n136-128-154:1658159:1658564 [0] NCCL INFO comm 0x10d4cd80 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:1658160:1658565 [1] NCCL INFO comm 0x10a6b100 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:1658159:1658564 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:1658159:1658564 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:1658159:1658564 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:1658159:1658564 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:1658159:1658564 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:1658160:1658565 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:1658159:1658564 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:1658159:1658564 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:1658160:1658565 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:1658159:1658564 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:1658159:1658564 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:1658160:1658565 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:1658159:1658564 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:1658160:1658565 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:1658159:1658564 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:1658160:1658565 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:1658159:1658564 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:1658160:1658565 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:1658159:1658564 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:1658160:1658565 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:1658159:1658564 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:1658160:1658565 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:1658159:1658564 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:1658160:1658565 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:1658159:1658564 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:1658160:1658565 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:1658159:1658564 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:1658160:1658565 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:1658159:1658564 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:1658159:1658564 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:1658160:1658565 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:1658159:1658564 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:1658159:1658564 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:1658160:1658565 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:1658159:1658564 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:1658159:1658564 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:1658160:1658565 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:1658159:1658564 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:1658160:1658565 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:1658160:1658565 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:1658159:1658564 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:1658160:1658565 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:1658159:1658564 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:1658160:1658565 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:1658159:1658564 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:1658160:1658565 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:1658159:1658564 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:1658160:1658565 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:1658159:1658564 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:1658160:1658565 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:1658159:1658564 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:1658160:1658565 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:1658159:1658564 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:1658160:1658565 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:1658159:1658564 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:1658160:1658565 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:1658159:1658564 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:1658159:1658564 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:1658159:1658564 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:1658159:1658564 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:1658159:1658564 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:1658159:1658564 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:1658160:1658565 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:1658159:1658564 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:1658160:1658565 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:1658159:1658564 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:1658160:1658565 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:1658159:1658564 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:1658160:1658565 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:1658159:1658564 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:1658160:1658565 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:1658159:1658564 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:1658160:1658565 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:1658159:1658564 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:1658160:1658565 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:1658159:1658564 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:1658160:1658565 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:1658159:1658564 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:1658160:1658565 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:1658159:1658564 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:1658160:1658565 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:1658159:1658564 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:1658160:1658565 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:1658160:1658565 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:1658159:1658564 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:1658160:1658565 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:1658159:1658564 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:1658160:1658565 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:1658159:1658564 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:1658160:1658565 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:1658159:1658564 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:1658160:1658565 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:1658159:1658564 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:1658160:1658565 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:1658159:1658564 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:1658160:1658565 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:1658159:1658564 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:1658160:1658565 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:1658159:1658564 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:1658160:1658565 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:1658159:1658564 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:1658160:1658565 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:1658159:1658564 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:1658160:1658565 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:1658159:1658564 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:1658160:1658565 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:1658159:1658564 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:1658160:1658565 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:1658159:1658564 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:1658160:1658565 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:1658159:1658564 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:1658159:1658564 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:1658160:1658565 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:1658159:1658564 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:1658159:1658564 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:1658159:1658564 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:1658159:1658564 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:1658159:1658564 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:1658159:1658564 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:1658159:1658564 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:1658159:1658564 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:1658159:1658564 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:1658159:1658564 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:1658159:1658564 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:1658159:1658564 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:1658159:1658564 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:1658159:1658583 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 82
n136-128-154:1658159:1658582 [0] NCCL INFO [Proxy Service] Device 0 CPU core 81
n136-128-154:1658160:1658565 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:1658160:1658585 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 78
n136-128-154:1658160:1658584 [1] NCCL INFO [Proxy Service] Device 1 CPU core 77
n136-128-154:1658159:1658564 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:1658159:1658564 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:1658159:1658564 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:1658160:1658565 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:1658160:1658565 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:1658159:1658564 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:1658159:1658564 [0] NCCL INFO ncclCommInitRankConfig comm 0x10d4cd80 rank 0 nranks 2 cudaDev 0 nvmlDev 1 busId 16000 commId 0x578a0970f19dc939 - Init COMPLETE
n136-128-154:1658159:1658564 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 1.24 (kernels 0.24, alloc 0.70, bootstrap 0.00, allgathers 0.00, topo 0.08, graphs 0.00, connections 0.01, rest 0.20)
n136-128-154:1658160:1658565 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:1658160:1658565 [1] NCCL INFO ncclCommInitRankConfig comm 0x10a6b100 rank 1 nranks 2 cudaDev 1 nvmlDev 2 busId 4a000 commId 0x578a0970f19dc939 - Init COMPLETE
n136-128-154:1658160:1658565 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 1.24 (kernels 0.24, alloc 0.70, bootstrap 0.00, allgathers 0.00, topo 0.08, graphs 0.00, connections 0.01, rest 0.21)
n136-128-154:1658159:1658586 [0] NCCL INFO Channel 00/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658159:1658586 [0] NCCL INFO Channel 01/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658159:1658586 [0] NCCL INFO Channel 02/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658159:1658586 [0] NCCL INFO Channel 03/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658159:1658586 [0] NCCL INFO Channel 04/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658159:1658586 [0] NCCL INFO Channel 05/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658159:1658586 [0] NCCL INFO Channel 06/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658159:1658586 [0] NCCL INFO Channel 07/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658159:1658586 [0] NCCL INFO Channel 08/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658159:1658586 [0] NCCL INFO Channel 09/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658159:1658586 [0] NCCL INFO Channel 10/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658159:1658586 [0] NCCL INFO Channel 11/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658160:1658587 [1] NCCL INFO Channel 00/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658159:1658586 [0] NCCL INFO Channel 12/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658160:1658587 [1] NCCL INFO Channel 01/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658159:1658586 [0] NCCL INFO Channel 13/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658160:1658587 [1] NCCL INFO Channel 02/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658159:1658586 [0] NCCL INFO Channel 14/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658159:1658586 [0] NCCL INFO Channel 15/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658160:1658587 [1] NCCL INFO Channel 03/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658159:1658586 [0] NCCL INFO Channel 16/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658160:1658587 [1] NCCL INFO Channel 04/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658159:1658586 [0] NCCL INFO Channel 17/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658160:1658587 [1] NCCL INFO Channel 05/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658159:1658586 [0] NCCL INFO Channel 18/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658160:1658587 [1] NCCL INFO Channel 06/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658159:1658586 [0] NCCL INFO Channel 19/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658160:1658587 [1] NCCL INFO Channel 07/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658159:1658586 [0] NCCL INFO Channel 20/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658160:1658587 [1] NCCL INFO Channel 08/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658159:1658586 [0] NCCL INFO Channel 21/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658160:1658587 [1] NCCL INFO Channel 09/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658159:1658586 [0] NCCL INFO Channel 22/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658160:1658587 [1] NCCL INFO Channel 10/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658159:1658586 [0] NCCL INFO Channel 23/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658160:1658587 [1] NCCL INFO Channel 11/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658587 [1] NCCL INFO Channel 12/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658587 [1] NCCL INFO Channel 13/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658587 [1] NCCL INFO Channel 14/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658587 [1] NCCL INFO Channel 15/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658587 [1] NCCL INFO Channel 16/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658587 [1] NCCL INFO Channel 17/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658587 [1] NCCL INFO Channel 18/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658587 [1] NCCL INFO Channel 19/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658587 [1] NCCL INFO Channel 20/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658587 [1] NCCL INFO Channel 21/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658587 [1] NCCL INFO Channel 22/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658587 [1] NCCL INFO Channel 23/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658159:1658586 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:1658160:1658587 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-05:18:05:04 INFO     [evaluator:559] Running loglikelihood requests
2025-12-05:18:05:04 INFO     [evaluator:559] Running loglikelihood requests

Running loglikelihood requests:   0%|          | 0/1268 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 64
Determined largest batch size: 64

Running loglikelihood requests:   0%|          | 1/1268 [00:05<1:54:50,  5.44s/it]
Running loglikelihood requests:   5%|▌         | 65/1268 [00:06<01:27, 13.74it/s] 
Running loglikelihood requests:  10%|█         | 129/1268 [00:07<00:43, 26.13it/s]
Running loglikelihood requests:  15%|█▌        | 193/1268 [00:08<00:28, 37.08it/s]
Running loglikelihood requests:  20%|██        | 257/1268 [00:08<00:21, 46.23it/s]
Running loglikelihood requests:  25%|██▌       | 321/1268 [00:09<00:17, 53.60it/s]
Running loglikelihood requests:  30%|███       | 385/1268 [00:10<00:14, 59.53it/s]
Running loglikelihood requests:  35%|███▌      | 449/1268 [00:11<00:12, 63.76it/s]
Running loglikelihood requests:  40%|████      | 513/1268 [00:12<00:11, 67.17it/s]
Running loglikelihood requests:  46%|████▌     | 577/1268 [00:13<00:09, 69.89it/s]
Running loglikelihood requests:  51%|█████     | 641/1268 [00:13<00:08, 72.35it/s]
Running loglikelihood requests:  56%|█████▌    | 705/1268 [00:14<00:07, 73.98it/s]
Running loglikelihood requests:  61%|██████    | 769/1268 [00:15<00:06, 75.36it/s]
Running loglikelihood requests:  66%|██████▌   | 833/1268 [00:16<00:05, 76.35it/s]
Running loglikelihood requests:  71%|███████   | 897/1268 [00:17<00:04, 77.20it/s]
Running loglikelihood requests:  76%|███████▌  | 961/1268 [00:18<00:03, 77.77it/s]
Running loglikelihood requests:  81%|████████  | 1025/1268 [00:18<00:03, 78.73it/s]
Running loglikelihood requests:  86%|████████▌ | 1089/1268 [00:19<00:02, 79.63it/s]
Running loglikelihood requests:  91%|█████████ | 1153/1268 [00:20<00:01, 80.60it/s]
Running loglikelihood requests:  96%|█████████▌| 1217/1268 [00:21<00:00, 86.11it/s]
Running loglikelihood requests: 100%|██████████| 1268/1268 [00:21<00:00, 60.28it/s]
n136-128-154:1658160:1658630 [1] NCCL INFO Channel 00/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658630 [1] NCCL INFO Channel 01/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658630 [1] NCCL INFO Channel 02/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658630 [1] NCCL INFO Channel 03/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658630 [1] NCCL INFO Channel 04/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658630 [1] NCCL INFO Channel 05/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658630 [1] NCCL INFO Channel 06/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658630 [1] NCCL INFO Channel 07/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658630 [1] NCCL INFO Channel 08/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658630 [1] NCCL INFO Channel 09/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658630 [1] NCCL INFO Channel 10/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658630 [1] NCCL INFO Channel 11/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658630 [1] NCCL INFO Channel 12/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658630 [1] NCCL INFO Channel 13/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658630 [1] NCCL INFO Channel 14/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658630 [1] NCCL INFO Channel 15/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658630 [1] NCCL INFO Channel 16/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658630 [1] NCCL INFO Channel 17/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658630 [1] NCCL INFO Channel 18/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658630 [1] NCCL INFO Channel 19/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658630 [1] NCCL INFO Channel 20/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658630 [1] NCCL INFO Channel 21/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658630 [1] NCCL INFO Channel 22/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658630 [1] NCCL INFO Channel 23/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658630 [1] NCCL INFO Channel 24/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658630 [1] NCCL INFO Channel 25/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658630 [1] NCCL INFO Channel 26/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658630 [1] NCCL INFO Channel 27/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658630 [1] NCCL INFO Channel 28/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658630 [1] NCCL INFO Channel 29/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658630 [1] NCCL INFO Channel 30/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658160:1658630 [1] NCCL INFO Channel 31/1 : 1[2] -> 0[1] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:1658160:1658635 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1658160:1658635 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1658160:1658635 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1658160:1658635 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1658160:1658635 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1658160:1658635 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1658160:1658584 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-05:18:05:37 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: auto (64)
|  Tasks   |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|----------|------:|------|-----:|------|---|-----:|---|-----:|
|winogrande|      1|none  |     5|acc   |↑  |0.4996|±  |0.0141|

[rank0]:[W1205 18:05:38.848203292 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:1658159:1658688 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1658159:1658688 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1658159:1658688 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1658159:1658582 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:1658159:1658688 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1658159:1658688 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1658159:1658688 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1658160:1658584 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:1658160:1658635 [1] NCCL INFO comm 0x10a6b100 rank 1 nranks 2 cudaDev 1 busId 4a000 - Abort COMPLETE
n136-128-154:1658159:1658688 [0] NCCL INFO comm 0x10d4cd80 rank 0 nranks 2 cudaDev 0 busId 16000 - Abort COMPLETE
任务 winogrande 评估完成！

==================================================
开始评估：任务=gsm8k | 少样本数=4 | 模型=Qwen2.5-7B-quantization-fg
输出路径：results2/Qwen2.5-7B-quantization-fg/fg/gsm8k.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-05:18:07:02 INFO     [__main__:440] Selected Tasks: ['gsm8k']
2025-12-05:18:07:02 INFO     [__main__:440] Selected Tasks: ['gsm8k']
2025-12-05:18:07:02 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-05:18:07:02 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-05:18:07:02 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-05:18:07:02 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-05:18:07:03 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-05:18:07:04 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-05:18:07:04 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.07s/it]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.05s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.03s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.05s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.06it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.05it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.55it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.29it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.55it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.28it/s]
2025-12-05:18:07:21 INFO     [evaluator:290] gsm8k: Using gen_kwargs: {'until': ['Question:', '</s>', '<|im_end|>'], 'do_sample': False, 'temperature': 0.0}
2025-12-05:18:07:21 WARNING  [evaluator:309] Overwriting default num_fewshot of gsm8k from 5 to 4
2025-12-05:18:07:21 INFO     [api.task:434] Building contexts for gsm8k on rank 0...

  0%|          | 0/660 [00:00<?, ?it/s]
  6%|▋         | 42/660 [00:00<00:01, 413.65it/s]
 13%|█▎        | 84/660 [00:00<00:01, 413.38it/s]
 19%|█▉        | 126/660 [00:00<00:01, 411.86it/s]
 25%|██▌       | 168/660 [00:00<00:01, 413.75it/s]
 32%|███▏      | 210/660 [00:00<00:01, 415.36it/s]
 38%|███▊      | 252/660 [00:00<00:00, 416.00it/s]
 45%|████▍     | 294/660 [00:00<00:00, 410.92it/s]
 51%|█████     | 336/660 [00:00<00:00, 412.16it/s]
 57%|█████▋    | 378/660 [00:00<00:00, 411.16it/s]
 64%|██████▎   | 420/660 [00:01<00:00, 412.31it/s]
 70%|███████   | 462/660 [00:01<00:00, 412.99it/s]
 76%|███████▋  | 504/660 [00:01<00:00, 413.56it/s]
 83%|████████▎ | 546/660 [00:01<00:00, 410.80it/s]
 89%|████████▉ | 588/660 [00:01<00:00, 413.10it/s]
 95%|█████████▌| 630/660 [00:01<00:00, 414.24it/s]
100%|██████████| 660/660 [00:01<00:00, 413.14it/s]
n136-128-154:1658789:1658789 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1658789:1658789 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1658789:1658789 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:1658789:1658789 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:1658789:1658789 [0] NCCL INFO Comm config Blocking set to 1
n136-128-154:1658789:1659148 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:1658789:1659148 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:1658789:1659148 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1658789:1659148 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1658789:1659148 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:1658789:1659148 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:1658789:1659148 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:1658789:1659148 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:1658789:1659148 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:1658789:1659148 [0] NCCL INFO Using network IB
n136-128-154:1658789:1659148 [0] NCCL INFO ncclCommInitRankConfig comm 0x12b667f0 rank 0 nranks 2 cudaDev 0 nvmlDev 1 busId 16000 commId 0xf496d131a9623cc2 - Init START
2025-12-05:18:07:27 INFO     [evaluator:290] gsm8k: Using gen_kwargs: {'until': ['Question:', '</s>', '<|im_end|>'], 'do_sample': False, 'temperature': 0.0}
2025-12-05:18:07:27 WARNING  [evaluator:309] Overwriting default num_fewshot of gsm8k from 5 to 4
2025-12-05:18:07:27 INFO     [api.task:434] Building contexts for gsm8k on rank 1...

  0%|          | 0/659 [00:00<?, ?it/s]
  6%|▌         | 41/659 [00:00<00:01, 409.79it/s]
 13%|█▎        | 83/659 [00:00<00:01, 414.14it/s]
 19%|█▉        | 125/659 [00:00<00:01, 415.17it/s]
 25%|██▌       | 167/659 [00:00<00:01, 416.71it/s]
 32%|███▏      | 209/659 [00:00<00:01, 417.23it/s]
 38%|███▊      | 251/659 [00:00<00:00, 412.93it/s]
 44%|████▍     | 293/659 [00:00<00:00, 412.91it/s]
 51%|█████     | 335/659 [00:00<00:00, 414.84it/s]
 57%|█████▋    | 377/659 [00:00<00:00, 415.43it/s]
 64%|██████▎   | 419/659 [00:01<00:00, 414.16it/s]
 70%|██████▉   | 461/659 [00:01<00:00, 413.30it/s]
 76%|███████▋  | 504/659 [00:01<00:00, 415.48it/s]
 83%|████████▎ | 546/659 [00:01<00:00, 404.90it/s]
 89%|████████▉ | 588/659 [00:01<00:00, 407.24it/s]
 95%|█████████▌| 629/659 [00:01<00:00, 407.03it/s]
100%|██████████| 659/659 [00:01<00:00, 411.90it/s]
n136-128-154:1658790:1658790 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:1658790:1658790 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1658790:1658790 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1658790:1658790 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:1658790:1658790 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:1658790:1659164 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:1658790:1659164 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:1658790:1659164 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1658790:1659164 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1658790:1659164 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:1658790:1659164 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:1658790:1659164 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:1658790:1659164 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:1658790:1659164 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:1658790:1659164 [1] NCCL INFO Using network IB
n136-128-154:1658790:1659164 [1] NCCL INFO ncclCommInitRankConfig comm 0x10287fb0 rank 1 nranks 2 cudaDev 1 nvmlDev 2 busId 4a000 commId 0xf496d131a9623cc2 - Init START
n136-128-154:1658790:1659164 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:1658789:1659148 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:1658790:1659164 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:1658790:1659164 [1] NCCL INFO Retrieving state for IB
n136-128-154:1658790:1659164 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:1658790:1659164 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:1658790:1659164 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:1658790:1659164 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:1658789:1659148 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:1658789:1659148 [0] NCCL INFO Retrieving state for IB
n136-128-154:1658789:1659148 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:1658789:1659148 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:1658790:1659164 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:1658789:1659148 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:1658789:1659148 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:1658789:1659148 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:1658790:1659164 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1658790:1659164 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1658790:1659164 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1658790:1659164 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1658789:1659148 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1658789:1659148 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1658789:1659148 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1658789:1659148 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1658789:1659148 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:1658789:1659148 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:1658789:1659148 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:1658789:1659148 [0] NCCL INFO               + PCI[24.0] - PCI/0-14000 (1000c01010de13b8)
n136-128-154:1658789:1659148 [0] NCCL INFO                             + PCI[24.0] - GPU/0-16000 (0)
n136-128-154:1658789:1659148 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1658789:1659148 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:1658789:1659148 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:1658789:1659148 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:1658789:1659148 [0] NCCL INFO               + PCI[24.0] - PCI/0-48000 (1000c01010de13b8)
n136-128-154:1658789:1659148 [0] NCCL INFO                             + PCI[24.0] - GPU/0-4a000 (1)
n136-128-154:1658789:1659148 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1658789:1659148 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:1658789:1659148 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:1658789:1659148 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:1658789:1659148 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:1658789:1659148 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:1658789:1659148 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:1658789:1659148 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:1658789:1659148 [0] NCCL INFO ==========================================
n136-128-154:1658789:1659148 [0] NCCL INFO GPU/0-16000 :GPU/0-16000 (0/5000.0/LOC) GPU/0-4a000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1658790:1659164 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:1658789:1659148 [0] NCCL INFO GPU/0-4a000 :GPU/0-16000 (2/240.0/NVL) GPU/0-4a000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1658790:1659164 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:1658790:1659164 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:1658790:1659164 [1] NCCL INFO               + PCI[24.0] - PCI/0-14000 (1000c01010de13b8)
n136-128-154:1658790:1659164 [1] NCCL INFO                             + PCI[24.0] - GPU/0-16000 (0)
n136-128-154:1658789:1659148 [0] NCCL INFO Setting affinity for GPU 1 to 1-31,65-95
n136-128-154:1658790:1659164 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1658790:1659164 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:1658790:1659164 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:1658790:1659164 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:1658790:1659164 [1] NCCL INFO               + PCI[24.0] - PCI/0-48000 (1000c01010de13b8)
n136-128-154:1658790:1659164 [1] NCCL INFO                             + PCI[24.0] - GPU/0-4a000 (1)
n136-128-154:1658790:1659164 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1658790:1659164 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:1658790:1659164 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:1658790:1659164 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:1658790:1659164 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:1658790:1659164 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:1658790:1659164 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:1658790:1659164 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:1658790:1659164 [1] NCCL INFO ==========================================
n136-128-154:1658790:1659164 [1] NCCL INFO GPU/0-16000 :GPU/0-16000 (0/5000.0/LOC) GPU/0-4a000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1658790:1659164 [1] NCCL INFO GPU/0-4a000 :GPU/0-16000 (2/240.0/NVL) GPU/0-4a000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1658790:1659164 [1] NCCL INFO Setting affinity for GPU 2 to 1-31,65-95
n136-128-154:1658789:1659148 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:1658790:1659164 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:1658789:1659148 [0] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658790:1659164 [1] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658789:1659148 [0] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658790:1659164 [1] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658789:1659148 [0] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658790:1659164 [1] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658789:1659148 [0] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658790:1659164 [1] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658789:1659148 [0] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658790:1659164 [1] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658789:1659148 [0] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658790:1659164 [1] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658789:1659148 [0] NCCL INFO  6 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658790:1659164 [1] NCCL INFO  6 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658789:1659148 [0] NCCL INFO  7 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658790:1659164 [1] NCCL INFO  7 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658789:1659148 [0] NCCL INFO  8 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658790:1659164 [1] NCCL INFO  8 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658789:1659148 [0] NCCL INFO  9 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658790:1659164 [1] NCCL INFO  9 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658789:1659148 [0] NCCL INFO 10 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658790:1659164 [1] NCCL INFO 10 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658789:1659148 [0] NCCL INFO 11 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658790:1659164 [1] NCCL INFO 11 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658790:1659164 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:1658789:1659148 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:1658790:1659164 [1] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658789:1659148 [0] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658790:1659164 [1] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658789:1659148 [0] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658790:1659164 [1] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658789:1659148 [0] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658790:1659164 [1] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658789:1659148 [0] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658790:1659164 [1] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658789:1659148 [0] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658790:1659164 [1] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658789:1659148 [0] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1658790:1659164 [1] NCCL INFO  6 : GPU/0-4a000 GPU/0-16000
n136-128-154:1658789:1659148 [0] NCCL INFO  6 : GPU/0-4a000 GPU/0-16000
n136-128-154:1658790:1659164 [1] NCCL INFO  7 : GPU/0-4a000 GPU/0-16000
n136-128-154:1658789:1659148 [0] NCCL INFO  7 : GPU/0-4a000 GPU/0-16000
n136-128-154:1658790:1659164 [1] NCCL INFO  8 : GPU/0-4a000 GPU/0-16000
n136-128-154:1658789:1659148 [0] NCCL INFO  8 : GPU/0-4a000 GPU/0-16000
n136-128-154:1658790:1659164 [1] NCCL INFO  9 : GPU/0-4a000 GPU/0-16000
n136-128-154:1658789:1659148 [0] NCCL INFO  9 : GPU/0-4a000 GPU/0-16000
n136-128-154:1658790:1659164 [1] NCCL INFO 10 : GPU/0-4a000 GPU/0-16000
n136-128-154:1658789:1659148 [0] NCCL INFO 10 : GPU/0-4a000 GPU/0-16000
n136-128-154:1658790:1659164 [1] NCCL INFO 11 : GPU/0-4a000 GPU/0-16000
n136-128-154:1658789:1659148 [0] NCCL INFO 11 : GPU/0-4a000 GPU/0-16000
n136-128-154:1658790:1659164 [1] NCCL INFO comm 0x10287fb0 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:1658789:1659148 [0] NCCL INFO comm 0x12b667f0 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:1658790:1659164 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:1658790:1659164 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:1658789:1659148 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:1658790:1659164 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:1658789:1659148 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:1658790:1659164 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:1658789:1659148 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:1658790:1659164 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:1658789:1659148 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:1658790:1659164 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:1658789:1659148 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:1658790:1659164 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:1658789:1659148 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:1658790:1659164 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:1658789:1659148 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:1658790:1659164 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:1658789:1659148 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:1658790:1659164 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:1658789:1659148 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:1658790:1659164 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:1658789:1659148 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:1658790:1659164 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:1658789:1659148 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:1658790:1659164 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:1658789:1659148 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:1658790:1659164 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:1658789:1659148 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:1658790:1659164 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:1658789:1659148 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:1658790:1659164 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:1658789:1659148 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:1658790:1659164 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:1658789:1659148 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:1658790:1659164 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:1658790:1659164 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:1658789:1659148 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:1658790:1659164 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:1658789:1659148 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:1658790:1659164 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:1658789:1659148 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:1658790:1659164 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:1658789:1659148 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:1658790:1659164 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:1658789:1659148 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:1658790:1659164 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:1658789:1659148 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:1658789:1659148 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:1658789:1659148 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:1658790:1659164 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:1658789:1659148 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:1658790:1659164 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:1658789:1659148 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:1658790:1659164 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:1658789:1659148 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:1658790:1659164 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:1658789:1659148 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:1658790:1659164 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:1658789:1659148 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:1658790:1659164 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:1658789:1659148 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:1658790:1659164 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:1658789:1659148 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:1658790:1659164 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:1658789:1659148 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:1658790:1659164 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:1658789:1659148 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:1658790:1659164 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:1658789:1659148 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:1658790:1659164 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:1658789:1659148 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:1658790:1659164 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:1658789:1659148 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:1658790:1659164 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:1658789:1659148 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:1658790:1659164 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:1658789:1659148 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:1658790:1659164 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:1658789:1659148 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:1658790:1659164 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:1658789:1659148 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:1658790:1659164 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:1658789:1659148 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:1658790:1659164 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:1658789:1659148 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:1658790:1659164 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:1658789:1659148 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:1658790:1659164 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:1658789:1659148 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:1658790:1659164 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:1658789:1659148 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:1658790:1659164 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:1658789:1659148 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:1658790:1659164 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:1658789:1659148 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:1658790:1659164 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:1658789:1659148 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:1658790:1659164 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:1658789:1659148 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:1658790:1659164 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:1658789:1659148 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:1658789:1659148 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:1658789:1659148 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:1658789:1659148 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:1658789:1659148 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:1658789:1659148 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:1658789:1659148 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:1658789:1659148 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:1658789:1659148 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:1658789:1659148 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:1658789:1659148 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:1658789:1659148 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:1658789:1659148 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:1658789:1659148 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:1658789:1659148 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:1658789:1659148 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:1658789:1659148 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:1658789:1659148 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:1658789:1659148 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:1658789:1659148 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:1658789:1659148 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:1658789:1659148 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:1658789:1659148 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:1658789:1659148 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:1658789:1659148 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:1658790:1659164 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:1658790:1659172 [1] NCCL INFO [Proxy Service] Device 1 CPU core 81
n136-128-154:1658790:1659173 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 18
n136-128-154:1658789:1659148 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:1658789:1659148 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:1658789:1659175 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 67
n136-128-154:1658789:1659174 [0] NCCL INFO [Proxy Service] Device 0 CPU core 66
n136-128-154:1658790:1659164 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:1658790:1659164 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:1658789:1659148 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:1658789:1659148 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:1658789:1659148 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:1658790:1659164 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:1658790:1659164 [1] NCCL INFO ncclCommInitRankConfig comm 0x10287fb0 rank 1 nranks 2 cudaDev 1 nvmlDev 2 busId 4a000 commId 0xf496d131a9623cc2 - Init COMPLETE
n136-128-154:1658790:1659164 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.44 (kernels 0.13, alloc 0.12, bootstrap 0.00, allgathers 0.00, topo 0.07, graphs 0.00, connections 0.08, rest 0.04)
n136-128-154:1658789:1659148 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:1658789:1659148 [0] NCCL INFO ncclCommInitRankConfig comm 0x12b667f0 rank 0 nranks 2 cudaDev 0 nvmlDev 1 busId 16000 commId 0xf496d131a9623cc2 - Init COMPLETE
n136-128-154:1658789:1659148 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 5.76 (kernels 0.16, alloc 0.14, bootstrap 5.25, allgathers 0.00, topo 0.07, graphs 0.00, connections 0.08, rest 0.04)
n136-128-154:1658790:1659176 [1] NCCL INFO Channel 00/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1659176 [1] NCCL INFO Channel 01/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1659176 [1] NCCL INFO Channel 02/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1659176 [1] NCCL INFO Channel 03/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1659176 [1] NCCL INFO Channel 04/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1659176 [1] NCCL INFO Channel 05/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1659176 [1] NCCL INFO Channel 06/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1659176 [1] NCCL INFO Channel 07/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658789:1659177 [0] NCCL INFO Channel 00/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658790:1659176 [1] NCCL INFO Channel 08/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658789:1659177 [0] NCCL INFO Channel 01/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658790:1659176 [1] NCCL INFO Channel 09/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658789:1659177 [0] NCCL INFO Channel 02/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658790:1659176 [1] NCCL INFO Channel 10/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658789:1659177 [0] NCCL INFO Channel 03/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658790:1659176 [1] NCCL INFO Channel 11/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658789:1659177 [0] NCCL INFO Channel 04/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658790:1659176 [1] NCCL INFO Channel 12/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658789:1659177 [0] NCCL INFO Channel 05/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658790:1659176 [1] NCCL INFO Channel 13/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658789:1659177 [0] NCCL INFO Channel 06/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658790:1659176 [1] NCCL INFO Channel 14/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658789:1659177 [0] NCCL INFO Channel 07/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658790:1659176 [1] NCCL INFO Channel 15/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658789:1659177 [0] NCCL INFO Channel 08/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658790:1659176 [1] NCCL INFO Channel 16/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658789:1659177 [0] NCCL INFO Channel 09/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658790:1659176 [1] NCCL INFO Channel 17/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658789:1659177 [0] NCCL INFO Channel 10/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658790:1659176 [1] NCCL INFO Channel 18/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658789:1659177 [0] NCCL INFO Channel 11/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658790:1659176 [1] NCCL INFO Channel 19/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658789:1659177 [0] NCCL INFO Channel 12/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658790:1659176 [1] NCCL INFO Channel 20/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658789:1659177 [0] NCCL INFO Channel 13/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658790:1659176 [1] NCCL INFO Channel 21/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658789:1659177 [0] NCCL INFO Channel 14/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658790:1659176 [1] NCCL INFO Channel 22/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658789:1659177 [0] NCCL INFO Channel 15/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658790:1659176 [1] NCCL INFO Channel 23/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658789:1659177 [0] NCCL INFO Channel 16/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658789:1659177 [0] NCCL INFO Channel 17/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658789:1659177 [0] NCCL INFO Channel 18/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658789:1659177 [0] NCCL INFO Channel 19/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658789:1659177 [0] NCCL INFO Channel 20/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658789:1659177 [0] NCCL INFO Channel 21/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658789:1659177 [0] NCCL INFO Channel 22/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658789:1659177 [0] NCCL INFO Channel 23/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1658790:1659176 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:1658789:1659177 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-05:18:07:29 INFO     [evaluator:559] Running generate_until requests
Passed argument batch_size = auto. Detecting largest batch size
2025-12-05:18:07:29 INFO     [evaluator:559] Running generate_until requests

Running generate_until requests:   0%|          | 0/660 [00:00<?, ?it/s]Passed argument batch_size = auto. Detecting largest batch size
Determined Largest batch size: 1
Determined Largest batch size: 1
Both `max_new_tokens` (=2048) and `max_length`(=1578) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1605) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   0%|          | 1/660 [02:45<30:19:52, 165.69s/it]Both `max_new_tokens` (=2048) and `max_length`(=1567) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1554) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   0%|          | 2/660 [03:32<17:32:05, 95.94s/it] Both `max_new_tokens` (=2048) and `max_length`(=1557) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1539) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   0%|          | 3/660 [04:23<13:44:12, 75.27s/it]Both `max_new_tokens` (=2048) and `max_length`(=1519) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1536) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1484) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   1%|          | 4/660 [05:14<11:57:41, 65.64s/it]Both `max_new_tokens` (=2048) and `max_length`(=1508) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1483) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1459) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   1%|          | 5/660 [06:03<10:52:50, 59.80s/it]Both `max_new_tokens` (=2048) and `max_length`(=1494) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1456) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   1%|          | 6/660 [06:53<10:13:52, 56.32s/it]Both `max_new_tokens` (=2048) and `max_length`(=1480) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1446) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   1%|          | 7/660 [07:42<9:48:40, 54.09s/it] Both `max_new_tokens` (=2048) and `max_length`(=1468) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1440) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   1%|          | 8/660 [08:32<9:31:57, 52.63s/it]Both `max_new_tokens` (=2048) and `max_length`(=1459) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1440) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   1%|▏         | 9/660 [09:21<9:20:11, 51.63s/it]Both `max_new_tokens` (=2048) and `max_length`(=1451) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1431) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   2%|▏         | 10/660 [10:11<9:13:58, 51.14s/it]Both `max_new_tokens` (=2048) and `max_length`(=1435) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1427) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   2%|▏         | 11/660 [11:02<9:10:45, 50.92s/it]Both `max_new_tokens` (=2048) and `max_length`(=1431) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1423) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   2%|▏         | 12/660 [11:51<9:05:43, 50.53s/it]Both `max_new_tokens` (=2048) and `max_length`(=1429) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1410) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   2%|▏         | 13/660 [12:42<9:06:16, 50.66s/it]Both `max_new_tokens` (=2048) and `max_length`(=1393) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1398) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   2%|▏         | 14/660 [13:32<9:03:23, 50.47s/it]Both `max_new_tokens` (=2048) and `max_length`(=1387) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1397) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   2%|▏         | 15/660 [14:24<9:06:09, 50.80s/it]Both `max_new_tokens` (=2048) and `max_length`(=1386) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1396) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   2%|▏         | 16/660 [15:15<9:05:31, 50.82s/it]Both `max_new_tokens` (=2048) and `max_length`(=1382) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1384) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   3%|▎         | 17/660 [16:05<9:02:20, 50.61s/it]Both `max_new_tokens` (=2048) and `max_length`(=1379) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1382) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   3%|▎         | 18/660 [16:54<8:56:47, 50.17s/it]Both `max_new_tokens` (=2048) and `max_length`(=1377) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1382) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   3%|▎         | 19/660 [17:45<8:58:28, 50.40s/it]Both `max_new_tokens` (=2048) and `max_length`(=1377) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1373) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   3%|▎         | 20/660 [18:36<8:59:52, 50.61s/it]Both `max_new_tokens` (=2048) and `max_length`(=1372) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1357) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   3%|▎         | 21/660 [19:25<8:53:01, 50.05s/it]Both `max_new_tokens` (=2048) and `max_length`(=1365) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1353) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   3%|▎         | 22/660 [20:15<8:52:50, 50.11s/it]Both `max_new_tokens` (=2048) and `max_length`(=1365) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1352) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   3%|▎         | 23/660 [21:07<8:56:52, 50.57s/it]Both `max_new_tokens` (=2048) and `max_length`(=1353) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1351) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   4%|▎         | 24/660 [21:57<8:53:57, 50.37s/it]Both `max_new_tokens` (=2048) and `max_length`(=1352) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1349) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   4%|▍         | 25/660 [22:46<8:48:37, 49.95s/it]Both `max_new_tokens` (=2048) and `max_length`(=1342) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1338) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   4%|▍         | 26/660 [23:36<8:47:53, 49.96s/it]Both `max_new_tokens` (=2048) and `max_length`(=1341) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1329) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1328) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   4%|▍         | 27/660 [24:25<8:46:46, 49.93s/it]Both `max_new_tokens` (=2048) and `max_length`(=1340) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1327) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   4%|▍         | 28/660 [25:15<8:45:37, 49.90s/it]Both `max_new_tokens` (=2048) and `max_length`(=1337) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1322) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   4%|▍         | 29/660 [26:05<8:43:57, 49.82s/it]Both `max_new_tokens` (=2048) and `max_length`(=1323) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1318) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   5%|▍         | 30/660 [26:54<8:42:06, 49.72s/it]Both `max_new_tokens` (=2048) and `max_length`(=1322) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1317) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   5%|▍         | 31/660 [27:43<8:37:50, 49.40s/it]Both `max_new_tokens` (=2048) and `max_length`(=1321) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1317) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   5%|▍         | 32/660 [28:32<8:34:37, 49.17s/it]Both `max_new_tokens` (=2048) and `max_length`(=1318) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1317) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   5%|▌         | 33/660 [29:21<8:35:18, 49.31s/it]Both `max_new_tokens` (=2048) and `max_length`(=1317) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1317) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   5%|▌         | 34/660 [30:11<8:34:13, 49.29s/it]Both `max_new_tokens` (=2048) and `max_length`(=1313) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1310) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   5%|▌         | 35/660 [30:59<8:31:33, 49.11s/it]Both `max_new_tokens` (=2048) and `max_length`(=1313) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1307) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   5%|▌         | 36/660 [31:49<8:32:37, 49.29s/it]Both `max_new_tokens` (=2048) and `max_length`(=1312) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1306) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   6%|▌         | 37/660 [32:40<8:35:33, 49.65s/it]Both `max_new_tokens` (=2048) and `max_length`(=1306) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1302) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1298) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   6%|▌         | 38/660 [33:29<8:32:50, 49.47s/it]Both `max_new_tokens` (=2048) and `max_length`(=1303) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1295) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   6%|▌         | 39/660 [34:18<8:31:40, 49.44s/it]Both `max_new_tokens` (=2048) and `max_length`(=1300) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1292) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   6%|▌         | 40/660 [35:07<8:29:37, 49.32s/it]Both `max_new_tokens` (=2048) and `max_length`(=1297) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1290) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   6%|▌         | 41/660 [35:56<8:26:34, 49.10s/it]Both `max_new_tokens` (=2048) and `max_length`(=1297) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1289) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   6%|▋         | 42/660 [36:45<8:25:16, 49.06s/it]Both `max_new_tokens` (=2048) and `max_length`(=1296) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1288) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   7%|▋         | 43/660 [37:33<8:23:48, 48.99s/it]Both `max_new_tokens` (=2048) and `max_length`(=1293) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1282) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   7%|▋         | 44/660 [38:23<8:24:38, 49.15s/it]Both `max_new_tokens` (=2048) and `max_length`(=1290) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1275) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   7%|▋         | 45/660 [39:12<8:23:06, 49.08s/it]Both `max_new_tokens` (=2048) and `max_length`(=1282) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1273) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   7%|▋         | 46/660 [40:00<8:20:44, 48.93s/it]Both `max_new_tokens` (=2048) and `max_length`(=1278) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1270) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   7%|▋         | 47/660 [40:49<8:19:12, 48.86s/it]Both `max_new_tokens` (=2048) and `max_length`(=1277) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1270) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   7%|▋         | 48/660 [41:38<8:17:56, 48.82s/it]Both `max_new_tokens` (=2048) and `max_length`(=1274) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1268) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   7%|▋         | 49/660 [42:27<8:18:41, 48.97s/it]Both `max_new_tokens` (=2048) and `max_length`(=1270) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1267) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   8%|▊         | 50/660 [43:16<8:18:25, 49.03s/it]Both `max_new_tokens` (=2048) and `max_length`(=1269) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1256) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   8%|▊         | 51/660 [44:05<8:16:26, 48.91s/it]Both `max_new_tokens` (=2048) and `max_length`(=1269) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1255) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   8%|▊         | 52/660 [44:53<8:14:12, 48.77s/it]Both `max_new_tokens` (=2048) and `max_length`(=1265) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1255) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1255) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   8%|▊         | 53/660 [45:42<8:13:02, 48.73s/it]Both `max_new_tokens` (=2048) and `max_length`(=1264) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1252) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   8%|▊         | 54/660 [46:31<8:13:53, 48.90s/it]Both `max_new_tokens` (=2048) and `max_length`(=1262) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1251) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   8%|▊         | 55/660 [47:21<8:15:58, 49.19s/it]Both `max_new_tokens` (=2048) and `max_length`(=1258) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1251) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   8%|▊         | 56/660 [48:11<8:16:27, 49.32s/it]Both `max_new_tokens` (=2048) and `max_length`(=1257) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1250) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   9%|▊         | 57/660 [49:00<8:16:45, 49.43s/it]Both `max_new_tokens` (=2048) and `max_length`(=1256) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1247) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   9%|▉         | 58/660 [49:50<8:16:25, 49.48s/it]Both `max_new_tokens` (=2048) and `max_length`(=1254) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1247) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   9%|▉         | 59/660 [50:40<8:16:15, 49.54s/it]Both `max_new_tokens` (=2048) and `max_length`(=1251) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1246) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   9%|▉         | 60/660 [51:29<8:15:44, 49.57s/it]Both `max_new_tokens` (=2048) and `max_length`(=1249) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1245) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1243) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   9%|▉         | 61/660 [52:19<8:14:33, 49.54s/it]Both `max_new_tokens` (=2048) and `max_length`(=1246) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1241) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   9%|▉         | 62/660 [53:08<8:13:45, 49.54s/it]Both `max_new_tokens` (=2048) and `max_length`(=1245) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1241) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  10%|▉         | 63/660 [53:58<8:12:00, 49.45s/it]Both `max_new_tokens` (=2048) and `max_length`(=1245) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1239) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1238) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  10%|▉         | 64/660 [54:47<8:11:22, 49.47s/it]Both `max_new_tokens` (=2048) and `max_length`(=1245) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1236) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  10%|▉         | 65/660 [55:37<8:10:32, 49.47s/it]Both `max_new_tokens` (=2048) and `max_length`(=1241) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1236) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  10%|█         | 66/660 [56:26<8:09:32, 49.45s/it]Both `max_new_tokens` (=2048) and `max_length`(=1239) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1235) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  10%|█         | 67/660 [57:16<8:09:12, 49.50s/it]Both `max_new_tokens` (=2048) and `max_length`(=1238) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1235) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  10%|█         | 68/660 [58:05<8:08:55, 49.55s/it]Both `max_new_tokens` (=2048) and `max_length`(=1236) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1234) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  10%|█         | 69/660 [58:55<8:07:31, 49.50s/it]Both `max_new_tokens` (=2048) and `max_length`(=1234) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1233) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  11%|█         | 70/660 [59:44<8:06:42, 49.50s/it]Both `max_new_tokens` (=2048) and `max_length`(=1233) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1230) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  11%|█         | 71/660 [1:00:34<8:05:40, 49.47s/it]Both `max_new_tokens` (=2048) and `max_length`(=1233) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1229) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  11%|█         | 72/660 [1:01:23<8:05:28, 49.54s/it]Both `max_new_tokens` (=2048) and `max_length`(=1233) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1227) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  11%|█         | 73/660 [1:02:13<8:05:31, 49.63s/it]Both `max_new_tokens` (=2048) and `max_length`(=1231) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1226) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  11%|█         | 74/660 [1:03:03<8:04:21, 49.59s/it]Both `max_new_tokens` (=2048) and `max_length`(=1227) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1223) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  11%|█▏        | 75/660 [1:03:52<8:03:22, 49.58s/it]Both `max_new_tokens` (=2048) and `max_length`(=1227) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1221) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  12%|█▏        | 76/660 [1:04:42<8:02:15, 49.55s/it]Both `max_new_tokens` (=2048) and `max_length`(=1225) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1221) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  12%|█▏        | 77/660 [1:05:31<8:01:03, 49.51s/it]Both `max_new_tokens` (=2048) and `max_length`(=1225) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1220) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  12%|█▏        | 78/660 [1:06:21<8:00:05, 49.49s/it]Both `max_new_tokens` (=2048) and `max_length`(=1224) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1218) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  12%|█▏        | 79/660 [1:07:10<7:59:24, 49.51s/it]Both `max_new_tokens` (=2048) and `max_length`(=1223) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1215) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  12%|█▏        | 80/660 [1:07:59<7:57:57, 49.44s/it]Both `max_new_tokens` (=2048) and `max_length`(=1223) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1214) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  12%|█▏        | 81/660 [1:08:49<7:57:43, 49.51s/it]Both `max_new_tokens` (=2048) and `max_length`(=1222) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1214) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1213) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  12%|█▏        | 82/660 [1:09:39<7:58:35, 49.68s/it]Both `max_new_tokens` (=2048) and `max_length`(=1222) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1211) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  13%|█▎        | 83/660 [1:10:29<7:57:32, 49.66s/it]Both `max_new_tokens` (=2048) and `max_length`(=1221) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1208) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  13%|█▎        | 84/660 [1:11:18<7:56:21, 49.62s/it]Both `max_new_tokens` (=2048) and `max_length`(=1218) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1208) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  13%|█▎        | 85/660 [1:12:08<7:55:06, 49.58s/it]Both `max_new_tokens` (=2048) and `max_length`(=1217) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1207) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  13%|█▎        | 86/660 [1:12:57<7:53:35, 49.51s/it]Both `max_new_tokens` (=2048) and `max_length`(=1214) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1206) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  13%|█▎        | 87/660 [1:13:46<7:52:35, 49.49s/it]Both `max_new_tokens` (=2048) and `max_length`(=1214) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1203) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  13%|█▎        | 88/660 [1:14:36<7:51:53, 49.50s/it]Both `max_new_tokens` (=2048) and `max_length`(=1208) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1203) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  13%|█▎        | 89/660 [1:15:26<7:51:14, 49.52s/it]Both `max_new_tokens` (=2048) and `max_length`(=1207) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1202) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  14%|█▎        | 90/660 [1:16:16<7:53:18, 49.82s/it]Both `max_new_tokens` (=2048) and `max_length`(=1207) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1199) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  14%|█▍        | 91/660 [1:17:05<7:51:01, 49.67s/it]Both `max_new_tokens` (=2048) and `max_length`(=1206) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1199) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  14%|█▍        | 92/660 [1:17:55<7:49:56, 49.64s/it]Both `max_new_tokens` (=2048) and `max_length`(=1205) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1199) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  14%|█▍        | 93/660 [1:18:45<7:51:17, 49.87s/it]Both `max_new_tokens` (=2048) and `max_length`(=1205) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1198) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  14%|█▍        | 94/660 [1:19:35<7:50:12, 49.85s/it]Both `max_new_tokens` (=2048) and `max_length`(=1204) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1197) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  14%|█▍        | 95/660 [1:20:25<7:47:57, 49.69s/it]Both `max_new_tokens` (=2048) and `max_length`(=1203) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1197) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  15%|█▍        | 96/660 [1:21:13<7:44:26, 49.41s/it]Both `max_new_tokens` (=2048) and `max_length`(=1203) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1194) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  15%|█▍        | 97/660 [1:22:03<7:45:12, 49.58s/it]Both `max_new_tokens` (=2048) and `max_length`(=1202) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1193) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  15%|█▍        | 98/660 [1:22:53<7:45:33, 49.70s/it]Both `max_new_tokens` (=2048) and `max_length`(=1202) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1193) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  15%|█▌        | 99/660 [1:23:42<7:42:10, 49.43s/it]Both `max_new_tokens` (=2048) and `max_length`(=1202) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1192) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  15%|█▌        | 100/660 [1:24:31<7:39:10, 49.20s/it]Both `max_new_tokens` (=2048) and `max_length`(=1201) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1192) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1191) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  15%|█▌        | 101/660 [1:25:19<7:37:06, 49.06s/it]Both `max_new_tokens` (=2048) and `max_length`(=1200) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1191) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  15%|█▌        | 102/660 [1:26:08<7:35:13, 48.95s/it]Both `max_new_tokens` (=2048) and `max_length`(=1198) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1190) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  16%|█▌        | 103/660 [1:26:57<7:35:02, 49.02s/it]Both `max_new_tokens` (=2048) and `max_length`(=1197) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1187) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  16%|█▌        | 104/660 [1:27:48<7:39:17, 49.56s/it]Both `max_new_tokens` (=2048) and `max_length`(=1196) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1186) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1185) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  16%|█▌        | 105/660 [1:28:38<7:39:34, 49.68s/it]Both `max_new_tokens` (=2048) and `max_length`(=1196) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1185) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  16%|█▌        | 106/660 [1:29:28<7:38:09, 49.62s/it]Both `max_new_tokens` (=2048) and `max_length`(=1195) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1185) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  16%|█▌        | 107/660 [1:30:17<7:35:49, 49.46s/it]Both `max_new_tokens` (=2048) and `max_length`(=1193) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1184) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  16%|█▋        | 108/660 [1:31:06<7:33:22, 49.28s/it]Both `max_new_tokens` (=2048) and `max_length`(=1192) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1183) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  17%|█▋        | 109/660 [1:31:55<7:31:55, 49.21s/it]Both `max_new_tokens` (=2048) and `max_length`(=1192) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  17%|█▋        | 110/660 [1:31:56<5:18:56, 34.79s/it]Both `max_new_tokens` (=2048) and `max_length`(=1191) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1183) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1182) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  17%|█▋        | 111/660 [1:32:45<5:56:49, 39.00s/it]Both `max_new_tokens` (=2048) and `max_length`(=1191) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1181) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  17%|█▋        | 112/660 [1:33:34<6:23:40, 42.01s/it]Both `max_new_tokens` (=2048) and `max_length`(=1190) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1179) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  17%|█▋        | 113/660 [1:34:23<6:42:16, 44.13s/it]Both `max_new_tokens` (=2048) and `max_length`(=1189) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1179) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  17%|█▋        | 114/660 [1:35:11<6:54:12, 45.52s/it]Both `max_new_tokens` (=2048) and `max_length`(=1188) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1178) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  17%|█▋        | 115/660 [1:36:00<7:03:11, 46.59s/it]Both `max_new_tokens` (=2048) and `max_length`(=1187) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1178) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  18%|█▊        | 116/660 [1:36:50<7:09:07, 47.33s/it]Both `max_new_tokens` (=2048) and `max_length`(=1186) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1177) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  18%|█▊        | 117/660 [1:37:40<7:17:36, 48.35s/it]Both `max_new_tokens` (=2048) and `max_length`(=1185) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1176) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  18%|█▊        | 118/660 [1:38:31<7:22:46, 49.02s/it]Both `max_new_tokens` (=2048) and `max_length`(=1184) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1176) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  18%|█▊        | 119/660 [1:39:20<7:23:09, 49.15s/it]Both `max_new_tokens` (=2048) and `max_length`(=1184) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1176) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  18%|█▊        | 120/660 [1:40:10<7:23:05, 49.23s/it]Both `max_new_tokens` (=2048) and `max_length`(=1184) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1175) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  18%|█▊        | 121/660 [1:41:00<7:25:39, 49.61s/it]Both `max_new_tokens` (=2048) and `max_length`(=1184) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1174) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  18%|█▊        | 122/660 [1:41:51<7:26:49, 49.83s/it]Both `max_new_tokens` (=2048) and `max_length`(=1183) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1174) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  19%|█▊        | 123/660 [1:42:39<7:23:27, 49.55s/it]Both `max_new_tokens` (=2048) and `max_length`(=1183) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1173) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  19%|█▉        | 124/660 [1:43:30<7:25:59, 49.92s/it]Both `max_new_tokens` (=2048) and `max_length`(=1182) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1173) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  19%|█▉        | 125/660 [1:44:21<7:26:53, 50.12s/it]Both `max_new_tokens` (=2048) and `max_length`(=1182) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1171) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  19%|█▉        | 126/660 [1:45:09<7:22:09, 49.68s/it]Both `max_new_tokens` (=2048) and `max_length`(=1181) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1170) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  19%|█▉        | 127/660 [1:45:58<7:19:07, 49.43s/it]Both `max_new_tokens` (=2048) and `max_length`(=1181) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1169) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  19%|█▉        | 128/660 [1:46:47<7:16:16, 49.20s/it]Both `max_new_tokens` (=2048) and `max_length`(=1180) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1169) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  20%|█▉        | 129/660 [1:47:36<7:14:16, 49.07s/it]Both `max_new_tokens` (=2048) and `max_length`(=1178) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1167) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  20%|█▉        | 130/660 [1:48:25<7:12:42, 48.99s/it]Both `max_new_tokens` (=2048) and `max_length`(=1178) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1166) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1166) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  20%|█▉        | 131/660 [1:49:13<7:11:17, 48.92s/it]Both `max_new_tokens` (=2048) and `max_length`(=1178) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1164) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  20%|██        | 132/660 [1:50:02<7:10:24, 48.91s/it]Both `max_new_tokens` (=2048) and `max_length`(=1178) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1162) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  20%|██        | 133/660 [1:50:51<7:09:16, 48.87s/it]Both `max_new_tokens` (=2048) and `max_length`(=1173) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1161) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  20%|██        | 134/660 [1:51:40<7:07:55, 48.81s/it]Both `max_new_tokens` (=2048) and `max_length`(=1173) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1161) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  20%|██        | 135/660 [1:52:28<7:07:05, 48.81s/it]Both `max_new_tokens` (=2048) and `max_length`(=1173) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1160) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  21%|██        | 136/660 [1:53:17<7:05:39, 48.74s/it]Both `max_new_tokens` (=2048) and `max_length`(=1170) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1159) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1159) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  21%|██        | 137/660 [1:54:06<7:05:19, 48.79s/it]Both `max_new_tokens` (=2048) and `max_length`(=1170) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1158) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  21%|██        | 138/660 [1:54:55<7:05:07, 48.87s/it]Both `max_new_tokens` (=2048) and `max_length`(=1168) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1157) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  21%|██        | 139/660 [1:55:44<7:03:43, 48.80s/it]Both `max_new_tokens` (=2048) and `max_length`(=1168) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1153) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  21%|██        | 140/660 [1:56:32<7:02:50, 48.79s/it]Both `max_new_tokens` (=2048) and `max_length`(=1167) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1153) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  21%|██▏       | 141/660 [1:57:21<7:01:18, 48.71s/it]Both `max_new_tokens` (=2048) and `max_length`(=1165) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1153) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  22%|██▏       | 142/660 [1:58:09<7:00:02, 48.65s/it]Both `max_new_tokens` (=2048) and `max_length`(=1165) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1152) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  22%|██▏       | 143/660 [1:58:58<6:59:40, 48.71s/it]Both `max_new_tokens` (=2048) and `max_length`(=1164) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1152) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  22%|██▏       | 144/660 [1:59:47<6:58:45, 48.69s/it]Both `max_new_tokens` (=2048) and `max_length`(=1163) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1151) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  22%|██▏       | 145/660 [2:00:36<6:58:17, 48.73s/it]Both `max_new_tokens` (=2048) and `max_length`(=1163) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1151) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  22%|██▏       | 146/660 [2:01:25<6:57:51, 48.78s/it]Both `max_new_tokens` (=2048) and `max_length`(=1162) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1150) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  22%|██▏       | 147/660 [2:02:13<6:56:07, 48.67s/it]Both `max_new_tokens` (=2048) and `max_length`(=1161) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1150) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  22%|██▏       | 148/660 [2:03:02<6:55:32, 48.70s/it]Both `max_new_tokens` (=2048) and `max_length`(=1159) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1150) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  23%|██▎       | 149/660 [2:03:50<6:54:21, 48.65s/it]Both `max_new_tokens` (=2048) and `max_length`(=1159) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1149) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  23%|██▎       | 150/660 [2:04:39<6:53:31, 48.65s/it]Both `max_new_tokens` (=2048) and `max_length`(=1159) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1149) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  23%|██▎       | 151/660 [2:05:28<6:53:25, 48.73s/it]Both `max_new_tokens` (=2048) and `max_length`(=1157) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1149) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  23%|██▎       | 152/660 [2:06:17<6:52:38, 48.74s/it]Both `max_new_tokens` (=2048) and `max_length`(=1156) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1149) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  23%|██▎       | 153/660 [2:07:05<6:51:38, 48.71s/it]Both `max_new_tokens` (=2048) and `max_length`(=1156) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1148) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  23%|██▎       | 154/660 [2:07:54<6:51:09, 48.75s/it]Both `max_new_tokens` (=2048) and `max_length`(=1155) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1148) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  23%|██▎       | 155/660 [2:08:43<6:49:32, 48.66s/it]Both `max_new_tokens` (=2048) and `max_length`(=1155) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1148) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  24%|██▎       | 156/660 [2:09:31<6:48:54, 48.68s/it]Both `max_new_tokens` (=2048) and `max_length`(=1154) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1147) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  24%|██▍       | 157/660 [2:10:20<6:47:55, 48.66s/it]Both `max_new_tokens` (=2048) and `max_length`(=1154) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1147) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1146) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  24%|██▍       | 158/660 [2:11:09<6:47:16, 48.68s/it]Both `max_new_tokens` (=2048) and `max_length`(=1154) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1146) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  24%|██▍       | 159/660 [2:11:57<6:46:30, 48.68s/it]Both `max_new_tokens` (=2048) and `max_length`(=1153) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1146) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  24%|██▍       | 160/660 [2:12:46<6:45:46, 48.69s/it]Both `max_new_tokens` (=2048) and `max_length`(=1151) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1146) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  24%|██▍       | 161/660 [2:13:35<6:44:59, 48.70s/it]Both `max_new_tokens` (=2048) and `max_length`(=1151) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1145) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  25%|██▍       | 162/660 [2:14:24<6:44:36, 48.75s/it]Both `max_new_tokens` (=2048) and `max_length`(=1150) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1141) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  25%|██▍       | 163/660 [2:15:12<6:43:34, 48.72s/it]Both `max_new_tokens` (=2048) and `max_length`(=1150) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1141) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  25%|██▍       | 164/660 [2:16:01<6:42:50, 48.73s/it]Both `max_new_tokens` (=2048) and `max_length`(=1148) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1141) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  25%|██▌       | 165/660 [2:16:50<6:42:40, 48.81s/it]Both `max_new_tokens` (=2048) and `max_length`(=1148) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1140) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  25%|██▌       | 166/660 [2:17:39<6:41:46, 48.80s/it]Both `max_new_tokens` (=2048) and `max_length`(=1148) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1140) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  25%|██▌       | 167/660 [2:18:28<6:41:17, 48.84s/it]Both `max_new_tokens` (=2048) and `max_length`(=1146) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1140) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  25%|██▌       | 168/660 [2:19:17<6:40:26, 48.83s/it]Both `max_new_tokens` (=2048) and `max_length`(=1144) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1139) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  26%|██▌       | 169/660 [2:20:05<6:39:07, 48.77s/it]Both `max_new_tokens` (=2048) and `max_length`(=1144) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1139) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  26%|██▌       | 170/660 [2:20:54<6:38:29, 48.79s/it]Both `max_new_tokens` (=2048) and `max_length`(=1144) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1136) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  26%|██▌       | 171/660 [2:21:43<6:37:20, 48.75s/it]Both `max_new_tokens` (=2048) and `max_length`(=1144) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1136) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  26%|██▌       | 172/660 [2:22:32<6:37:22, 48.86s/it]Both `max_new_tokens` (=2048) and `max_length`(=1144) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1134) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  26%|██▌       | 173/660 [2:23:21<6:36:18, 48.83s/it]Both `max_new_tokens` (=2048) and `max_length`(=1143) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1134) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  26%|██▋       | 174/660 [2:24:09<6:34:59, 48.76s/it]Both `max_new_tokens` (=2048) and `max_length`(=1143) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1134) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  27%|██▋       | 175/660 [2:24:58<6:34:02, 48.75s/it]Both `max_new_tokens` (=2048) and `max_length`(=1143) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1134) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  27%|██▋       | 176/660 [2:25:47<6:34:15, 48.88s/it]Both `max_new_tokens` (=2048) and `max_length`(=1143) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1133) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  27%|██▋       | 177/660 [2:26:36<6:33:06, 48.83s/it]Both `max_new_tokens` (=2048) and `max_length`(=1142) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1133) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  27%|██▋       | 178/660 [2:27:25<6:32:03, 48.80s/it]Both `max_new_tokens` (=2048) and `max_length`(=1140) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1131) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  27%|██▋       | 179/660 [2:28:13<6:30:54, 48.76s/it]Both `max_new_tokens` (=2048) and `max_length`(=1139) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1130) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  27%|██▋       | 180/660 [2:29:02<6:29:55, 48.74s/it]Both `max_new_tokens` (=2048) and `max_length`(=1138) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1130) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1130) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  27%|██▋       | 181/660 [2:29:51<6:29:18, 48.77s/it]Both `max_new_tokens` (=2048) and `max_length`(=1137) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1129) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  28%|██▊       | 182/660 [2:30:39<6:28:09, 48.72s/it]Both `max_new_tokens` (=2048) and `max_length`(=1137) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1129) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  28%|██▊       | 183/660 [2:31:28<6:27:24, 48.73s/it]Both `max_new_tokens` (=2048) and `max_length`(=1136) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1129) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  28%|██▊       | 184/660 [2:32:18<6:28:45, 49.00s/it]Both `max_new_tokens` (=2048) and `max_length`(=1135) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1128) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  28%|██▊       | 185/660 [2:33:08<6:30:51, 49.37s/it]Both `max_new_tokens` (=2048) and `max_length`(=1135) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1128) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1125) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  28%|██▊       | 186/660 [2:33:57<6:30:19, 49.41s/it]Both `max_new_tokens` (=2048) and `max_length`(=1135) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1125) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  28%|██▊       | 187/660 [2:34:46<6:28:13, 49.25s/it]Both `max_new_tokens` (=2048) and `max_length`(=1135) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1125) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  28%|██▊       | 188/660 [2:35:35<6:25:41, 49.03s/it]Both `max_new_tokens` (=2048) and `max_length`(=1134) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1124) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  29%|██▊       | 189/660 [2:36:24<6:24:17, 48.95s/it]Both `max_new_tokens` (=2048) and `max_length`(=1132) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1123) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  29%|██▉       | 190/660 [2:37:12<6:22:24, 48.82s/it]Both `max_new_tokens` (=2048) and `max_length`(=1131) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1123) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  29%|██▉       | 191/660 [2:38:01<6:21:14, 48.77s/it]Both `max_new_tokens` (=2048) and `max_length`(=1131) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1122) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  29%|██▉       | 192/660 [2:38:50<6:20:36, 48.80s/it]Both `max_new_tokens` (=2048) and `max_length`(=1131) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1121) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  29%|██▉       | 193/660 [2:39:38<6:19:35, 48.77s/it]Both `max_new_tokens` (=2048) and `max_length`(=1131) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1119) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  29%|██▉       | 194/660 [2:40:27<6:18:55, 48.79s/it]Both `max_new_tokens` (=2048) and `max_length`(=1130) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1119) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  30%|██▉       | 195/660 [2:41:17<6:20:10, 49.06s/it]Both `max_new_tokens` (=2048) and `max_length`(=1129) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1118) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  30%|██▉       | 196/660 [2:42:08<6:23:24, 49.58s/it]Both `max_new_tokens` (=2048) and `max_length`(=1129) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1117) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  30%|██▉       | 197/660 [2:42:58<6:23:27, 49.69s/it]Both `max_new_tokens` (=2048) and `max_length`(=1126) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1117) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  30%|███       | 198/660 [2:43:47<6:20:43, 49.45s/it]Both `max_new_tokens` (=2048) and `max_length`(=1126) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1117) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  30%|███       | 199/660 [2:44:37<6:21:48, 49.69s/it]Both `max_new_tokens` (=2048) and `max_length`(=1125) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1117) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  30%|███       | 200/660 [2:45:27<6:22:41, 49.92s/it]Both `max_new_tokens` (=2048) and `max_length`(=1125) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1115) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  30%|███       | 201/660 [2:46:16<6:19:54, 49.66s/it]Both `max_new_tokens` (=2048) and `max_length`(=1124) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1113) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  31%|███       | 202/660 [2:47:05<6:17:28, 49.45s/it]Both `max_new_tokens` (=2048) and `max_length`(=1124) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1112) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  31%|███       | 203/660 [2:47:54<6:15:51, 49.35s/it]Both `max_new_tokens` (=2048) and `max_length`(=1122) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1112) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  31%|███       | 204/660 [2:48:43<6:14:03, 49.22s/it]Both `max_new_tokens` (=2048) and `max_length`(=1122) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1111) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  31%|███       | 205/660 [2:49:33<6:14:26, 49.38s/it]Both `max_new_tokens` (=2048) and `max_length`(=1121) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1111) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  31%|███       | 206/660 [2:50:23<6:15:54, 49.68s/it]Both `max_new_tokens` (=2048) and `max_length`(=1121) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1109) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  31%|███▏      | 207/660 [2:51:13<6:14:23, 49.59s/it]Both `max_new_tokens` (=2048) and `max_length`(=1119) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1108) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  32%|███▏      | 208/660 [2:52:02<6:12:23, 49.43s/it]Both `max_new_tokens` (=2048) and `max_length`(=1119) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1108) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  32%|███▏      | 209/660 [2:52:51<6:10:24, 49.28s/it]Both `max_new_tokens` (=2048) and `max_length`(=1119) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1108) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  32%|███▏      | 210/660 [2:52:54<4:25:47, 35.44s/it]Both `max_new_tokens` (=2048) and `max_length`(=1117) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1108) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  32%|███▏      | 211/660 [2:53:43<4:55:02, 39.43s/it]Both `max_new_tokens` (=2048) and `max_length`(=1117) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1108) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  32%|███▏      | 212/660 [2:54:32<5:16:00, 42.32s/it]Both `max_new_tokens` (=2048) and `max_length`(=1115) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1107) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  32%|███▏      | 213/660 [2:55:21<5:29:58, 44.29s/it]Both `max_new_tokens` (=2048) and `max_length`(=1114) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1107) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  32%|███▏      | 214/660 [2:56:10<5:39:31, 45.68s/it]Both `max_new_tokens` (=2048) and `max_length`(=1114) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1106) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  33%|███▎      | 215/660 [2:56:59<5:46:07, 46.67s/it]Both `max_new_tokens` (=2048) and `max_length`(=1114) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1106) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  33%|███▎      | 216/660 [2:57:47<5:50:05, 47.31s/it]Both `max_new_tokens` (=2048) and `max_length`(=1113) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1106) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  33%|███▎      | 217/660 [2:58:36<5:52:54, 47.80s/it]Both `max_new_tokens` (=2048) and `max_length`(=1113) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1104) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  33%|███▎      | 218/660 [2:59:25<5:54:40, 48.15s/it]Both `max_new_tokens` (=2048) and `max_length`(=1113) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1104) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  33%|███▎      | 219/660 [3:00:16<5:58:53, 48.83s/it]Both `max_new_tokens` (=2048) and `max_length`(=1112) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1101) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  33%|███▎      | 220/660 [3:01:06<6:01:58, 49.36s/it]Both `max_new_tokens` (=2048) and `max_length`(=1112) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1101) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  33%|███▎      | 221/660 [3:01:55<6:00:09, 49.22s/it]Both `max_new_tokens` (=2048) and `max_length`(=1110) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1100) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  34%|███▎      | 222/660 [3:02:44<5:58:51, 49.16s/it]Both `max_new_tokens` (=2048) and `max_length`(=1110) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1099) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1099) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  34%|███▍      | 223/660 [3:03:35<6:01:01, 49.57s/it]Both `max_new_tokens` (=2048) and `max_length`(=1108) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1098) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  34%|███▍      | 224/660 [3:04:25<6:02:34, 49.90s/it]Both `max_new_tokens` (=2048) and `max_length`(=1105) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1097) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  34%|███▍      | 225/660 [3:05:14<5:59:10, 49.54s/it]Both `max_new_tokens` (=2048) and `max_length`(=1105) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1097) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  34%|███▍      | 226/660 [3:06:05<6:00:48, 49.88s/it]Both `max_new_tokens` (=2048) and `max_length`(=1105) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1097) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  34%|███▍      | 227/660 [3:06:55<6:01:21, 50.07s/it]Both `max_new_tokens` (=2048) and `max_length`(=1104) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1096) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  35%|███▍      | 228/660 [3:07:44<5:58:16, 49.76s/it]Both `max_new_tokens` (=2048) and `max_length`(=1103) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1096) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  35%|███▍      | 229/660 [3:08:33<5:55:52, 49.54s/it]Both `max_new_tokens` (=2048) and `max_length`(=1103) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1096) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  35%|███▍      | 230/660 [3:09:22<5:54:00, 49.40s/it]Both `max_new_tokens` (=2048) and `max_length`(=1103) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1096) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  35%|███▌      | 231/660 [3:10:11<5:52:32, 49.31s/it]Both `max_new_tokens` (=2048) and `max_length`(=1102) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1095) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  35%|███▌      | 232/660 [3:11:01<5:51:24, 49.26s/it]Both `max_new_tokens` (=2048) and `max_length`(=1101) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1094) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  35%|███▌      | 233/660 [3:11:50<5:50:09, 49.20s/it]Both `max_new_tokens` (=2048) and `max_length`(=1101) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1094) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1094) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  35%|███▌      | 234/660 [3:12:39<5:49:39, 49.25s/it]Both `max_new_tokens` (=2048) and `max_length`(=1100) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1093) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  36%|███▌      | 235/660 [3:13:28<5:48:17, 49.17s/it]Both `max_new_tokens` (=2048) and `max_length`(=1099) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1092) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  36%|███▌      | 236/660 [3:14:17<5:47:03, 49.11s/it]Both `max_new_tokens` (=2048) and `max_length`(=1099) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1091) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  36%|███▌      | 237/660 [3:15:06<5:46:06, 49.09s/it]Both `max_new_tokens` (=2048) and `max_length`(=1098) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1091) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  36%|███▌      | 238/660 [3:15:55<5:44:45, 49.02s/it]Both `max_new_tokens` (=2048) and `max_length`(=1098) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1091) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  36%|███▌      | 239/660 [3:16:44<5:44:02, 49.03s/it]Both `max_new_tokens` (=2048) and `max_length`(=1097) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1090) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  36%|███▋      | 240/660 [3:17:33<5:43:16, 49.04s/it]Both `max_new_tokens` (=2048) and `max_length`(=1096) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1090) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  37%|███▋      | 241/660 [3:18:22<5:42:58, 49.11s/it]Both `max_new_tokens` (=2048) and `max_length`(=1096) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1089) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  37%|███▋      | 242/660 [3:19:11<5:42:03, 49.10s/it]Both `max_new_tokens` (=2048) and `max_length`(=1095) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1089) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  37%|███▋      | 243/660 [3:20:00<5:41:17, 49.11s/it]Both `max_new_tokens` (=2048) and `max_length`(=1095) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1089) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  37%|███▋      | 244/660 [3:20:49<5:40:11, 49.07s/it]Both `max_new_tokens` (=2048) and `max_length`(=1095) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1089) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  37%|███▋      | 245/660 [3:21:39<5:39:31, 49.09s/it]Both `max_new_tokens` (=2048) and `max_length`(=1094) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1088) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  37%|███▋      | 246/660 [3:22:28<5:39:40, 49.23s/it]Both `max_new_tokens` (=2048) and `max_length`(=1094) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1088) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  37%|███▋      | 247/660 [3:23:18<5:39:25, 49.31s/it]Both `max_new_tokens` (=2048) and `max_length`(=1093) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1088) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  38%|███▊      | 248/660 [3:24:07<5:39:16, 49.41s/it]Both `max_new_tokens` (=2048) and `max_length`(=1093) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1087) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  38%|███▊      | 249/660 [3:24:57<5:38:36, 49.43s/it]Both `max_new_tokens` (=2048) and `max_length`(=1093) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1086) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  38%|███▊      | 250/660 [3:25:46<5:38:02, 49.47s/it]Both `max_new_tokens` (=2048) and `max_length`(=1093) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1086) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  38%|███▊      | 251/660 [3:26:36<5:37:27, 49.51s/it]Both `max_new_tokens` (=2048) and `max_length`(=1093) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1085) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  38%|███▊      | 252/660 [3:27:25<5:36:42, 49.52s/it]Both `max_new_tokens` (=2048) and `max_length`(=1091) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1082) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1082) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  38%|███▊      | 253/660 [3:28:15<5:36:37, 49.63s/it]Both `max_new_tokens` (=2048) and `max_length`(=1091) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1082) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  38%|███▊      | 254/660 [3:29:05<5:35:44, 49.62s/it]Both `max_new_tokens` (=2048) and `max_length`(=1090) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1082) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  39%|███▊      | 255/660 [3:29:55<5:35:18, 49.68s/it]Both `max_new_tokens` (=2048) and `max_length`(=1089) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1081) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  39%|███▉      | 256/660 [3:30:44<5:34:27, 49.67s/it]Both `max_new_tokens` (=2048) and `max_length`(=1088) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1080) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  39%|███▉      | 257/660 [3:31:34<5:33:15, 49.62s/it]Both `max_new_tokens` (=2048) and `max_length`(=1088) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1080) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  39%|███▉      | 258/660 [3:32:23<5:32:16, 49.59s/it]Both `max_new_tokens` (=2048) and `max_length`(=1088) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1080) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  39%|███▉      | 259/660 [3:33:13<5:31:31, 49.60s/it]Both `max_new_tokens` (=2048) and `max_length`(=1087) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1080) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  39%|███▉      | 260/660 [3:34:03<5:30:26, 49.57s/it]Both `max_new_tokens` (=2048) and `max_length`(=1087) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  40%|███▉      | 261/660 [3:34:06<3:56:46, 35.60s/it]Both `max_new_tokens` (=2048) and `max_length`(=1086) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1080) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  40%|███▉      | 262/660 [3:34:55<4:23:52, 39.78s/it]Both `max_new_tokens` (=2048) and `max_length`(=1085) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1080) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  40%|███▉      | 263/660 [3:35:45<4:42:43, 42.73s/it]Both `max_new_tokens` (=2048) and `max_length`(=1084) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1079) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  40%|████      | 264/660 [3:36:34<4:55:26, 44.76s/it]Both `max_new_tokens` (=2048) and `max_length`(=1084) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1079) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  40%|████      | 265/660 [3:37:24<5:04:06, 46.19s/it]Both `max_new_tokens` (=2048) and `max_length`(=1083) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1079) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  40%|████      | 266/660 [3:38:13<5:09:46, 47.17s/it]Both `max_new_tokens` (=2048) and `max_length`(=1082) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1079) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  40%|████      | 267/660 [3:39:03<5:14:26, 48.01s/it]Both `max_new_tokens` (=2048) and `max_length`(=1082) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1078) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  41%|████      | 268/660 [3:39:53<5:16:58, 48.52s/it]Both `max_new_tokens` (=2048) and `max_length`(=1080) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1077) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  41%|████      | 269/660 [3:40:42<5:17:54, 48.78s/it]Both `max_new_tokens` (=2048) and `max_length`(=1079) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1076) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1076) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  41%|████      | 270/660 [3:41:32<5:18:34, 49.01s/it]Both `max_new_tokens` (=2048) and `max_length`(=1078) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1074) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  41%|████      | 271/660 [3:42:22<5:19:19, 49.25s/it]Both `max_new_tokens` (=2048) and `max_length`(=1078) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1074) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  41%|████      | 272/660 [3:43:11<5:19:24, 49.39s/it]Both `max_new_tokens` (=2048) and `max_length`(=1077) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1074) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  41%|████▏     | 273/660 [3:44:01<5:18:43, 49.41s/it]Both `max_new_tokens` (=2048) and `max_length`(=1077) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1073) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  42%|████▏     | 274/660 [3:44:50<5:17:30, 49.35s/it]Both `max_new_tokens` (=2048) and `max_length`(=1076) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1072) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  42%|████▏     | 275/660 [3:45:39<5:15:42, 49.20s/it]Both `max_new_tokens` (=2048) and `max_length`(=1076) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1072) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  42%|████▏     | 276/660 [3:46:28<5:14:27, 49.13s/it]Both `max_new_tokens` (=2048) and `max_length`(=1075) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1072) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  42%|████▏     | 277/660 [3:47:17<5:13:38, 49.14s/it]Both `max_new_tokens` (=2048) and `max_length`(=1074) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1072) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  42%|████▏     | 278/660 [3:48:06<5:12:23, 49.07s/it]Both `max_new_tokens` (=2048) and `max_length`(=1074) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1071) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  42%|████▏     | 279/660 [3:48:55<5:11:42, 49.09s/it]Both `max_new_tokens` (=2048) and `max_length`(=1074) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1070) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  42%|████▏     | 280/660 [3:49:44<5:11:03, 49.11s/it]Both `max_new_tokens` (=2048) and `max_length`(=1074) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1069) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  43%|████▎     | 281/660 [3:50:33<5:09:29, 49.00s/it]Both `max_new_tokens` (=2048) and `max_length`(=1074) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1069) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  43%|████▎     | 282/660 [3:51:22<5:08:24, 48.95s/it]Both `max_new_tokens` (=2048) and `max_length`(=1073) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1069) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  43%|████▎     | 283/660 [3:52:11<5:07:20, 48.91s/it]Both `max_new_tokens` (=2048) and `max_length`(=1073) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1068) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  43%|████▎     | 284/660 [3:53:00<5:07:38, 49.09s/it]Both `max_new_tokens` (=2048) and `max_length`(=1073) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1068) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  43%|████▎     | 285/660 [3:53:49<5:06:51, 49.10s/it]Both `max_new_tokens` (=2048) and `max_length`(=1072) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  43%|████▎     | 286/660 [3:53:52<3:39:55, 35.28s/it]Both `max_new_tokens` (=2048) and `max_length`(=1072) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1068) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  43%|████▎     | 287/660 [3:54:42<4:05:26, 39.48s/it]Both `max_new_tokens` (=2048) and `max_length`(=1072) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1067) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  44%|████▎     | 288/660 [3:55:32<4:25:34, 42.83s/it]Both `max_new_tokens` (=2048) and `max_length`(=1071) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1067) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  44%|████▍     | 289/660 [3:56:23<4:38:57, 45.12s/it]Both `max_new_tokens` (=2048) and `max_length`(=1071) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1067) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  44%|████▍     | 290/660 [3:57:11<4:44:47, 46.18s/it]Both `max_new_tokens` (=2048) and `max_length`(=1071) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1067) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  44%|████▍     | 291/660 [3:58:00<4:48:45, 46.95s/it]Both `max_new_tokens` (=2048) and `max_length`(=1071) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1066) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  44%|████▍     | 292/660 [3:58:49<4:51:23, 47.51s/it]Both `max_new_tokens` (=2048) and `max_length`(=1070) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1066) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  44%|████▍     | 293/660 [3:59:38<4:53:22, 47.96s/it]Both `max_new_tokens` (=2048) and `max_length`(=1070) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1066) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1065) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  45%|████▍     | 294/660 [4:00:27<4:54:30, 48.28s/it]Both `max_new_tokens` (=2048) and `max_length`(=1069) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1065) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  45%|████▍     | 295/660 [4:01:16<4:54:37, 48.43s/it]Both `max_new_tokens` (=2048) and `max_length`(=1069) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1064) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  45%|████▍     | 296/660 [4:02:04<4:54:26, 48.53s/it]Both `max_new_tokens` (=2048) and `max_length`(=1069) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1064) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  45%|████▌     | 297/660 [4:02:53<4:54:09, 48.62s/it]Both `max_new_tokens` (=2048) and `max_length`(=1069) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1063) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  45%|████▌     | 298/660 [4:03:43<4:55:21, 48.95s/it]Both `max_new_tokens` (=2048) and `max_length`(=1069) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1063) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  45%|████▌     | 299/660 [4:04:34<4:57:24, 49.43s/it]Both `max_new_tokens` (=2048) and `max_length`(=1068) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1062) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  45%|████▌     | 300/660 [4:05:25<5:00:54, 50.15s/it]Both `max_new_tokens` (=2048) and `max_length`(=1067) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1061) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  46%|████▌     | 301/660 [4:06:15<4:59:19, 50.03s/it]Both `max_new_tokens` (=2048) and `max_length`(=1066) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1060) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  46%|████▌     | 302/660 [4:07:05<4:58:07, 49.97s/it]Both `max_new_tokens` (=2048) and `max_length`(=1066) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1060) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  46%|████▌     | 303/660 [4:07:55<4:58:08, 50.11s/it]Both `max_new_tokens` (=2048) and `max_length`(=1066) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1059) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  46%|████▌     | 304/660 [4:08:45<4:56:08, 49.91s/it]Both `max_new_tokens` (=2048) and `max_length`(=1065) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1059) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  46%|████▌     | 305/660 [4:09:34<4:54:00, 49.69s/it]Both `max_new_tokens` (=2048) and `max_length`(=1064) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1058) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  46%|████▋     | 306/660 [4:10:23<4:52:30, 49.58s/it]Both `max_new_tokens` (=2048) and `max_length`(=1064) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1058) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  47%|████▋     | 307/660 [4:11:13<4:50:56, 49.45s/it]Both `max_new_tokens` (=2048) and `max_length`(=1063) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1057) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  47%|████▋     | 308/660 [4:12:02<4:49:59, 49.43s/it]Both `max_new_tokens` (=2048) and `max_length`(=1063) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1057) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  47%|████▋     | 309/660 [4:12:52<4:50:48, 49.71s/it]Both `max_new_tokens` (=2048) and `max_length`(=1063) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1057) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  47%|████▋     | 310/660 [4:13:43<4:52:08, 50.08s/it]Both `max_new_tokens` (=2048) and `max_length`(=1062) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1057) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  47%|████▋     | 311/660 [4:14:32<4:49:34, 49.78s/it]Both `max_new_tokens` (=2048) and `max_length`(=1062) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1057) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  47%|████▋     | 312/660 [4:15:21<4:47:03, 49.49s/it]Both `max_new_tokens` (=2048) and `max_length`(=1061) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1057) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  47%|████▋     | 313/660 [4:16:10<4:45:19, 49.33s/it]Both `max_new_tokens` (=2048) and `max_length`(=1061) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1056) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1056) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  48%|████▊     | 314/660 [4:16:59<4:44:07, 49.27s/it]Both `max_new_tokens` (=2048) and `max_length`(=1060) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1056) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  48%|████▊     | 315/660 [4:17:48<4:42:12, 49.08s/it]Both `max_new_tokens` (=2048) and `max_length`(=1059) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1056) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  48%|████▊     | 316/660 [4:18:37<4:40:58, 49.01s/it]Both `max_new_tokens` (=2048) and `max_length`(=1059) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1056) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  48%|████▊     | 317/660 [4:19:25<4:39:36, 48.91s/it]Both `max_new_tokens` (=2048) and `max_length`(=1059) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1055) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  48%|████▊     | 318/660 [4:20:14<4:38:04, 48.79s/it]Both `max_new_tokens` (=2048) and `max_length`(=1058) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1054) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  48%|████▊     | 319/660 [4:21:03<4:37:18, 48.79s/it]Both `max_new_tokens` (=2048) and `max_length`(=1058) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1053) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  48%|████▊     | 320/660 [4:21:51<4:36:05, 48.72s/it]Both `max_new_tokens` (=2048) and `max_length`(=1058) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1053) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  49%|████▊     | 321/660 [4:22:42<4:38:34, 49.31s/it]Both `max_new_tokens` (=2048) and `max_length`(=1057) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1053) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  49%|████▉     | 322/660 [4:23:32<4:39:29, 49.61s/it]Both `max_new_tokens` (=2048) and `max_length`(=1057) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1051) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  49%|████▉     | 323/660 [4:24:21<4:37:54, 49.48s/it]Both `max_new_tokens` (=2048) and `max_length`(=1056) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1051) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  49%|████▉     | 324/660 [4:25:10<4:35:56, 49.28s/it]Both `max_new_tokens` (=2048) and `max_length`(=1055) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1050) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  49%|████▉     | 325/660 [4:25:59<4:34:42, 49.20s/it]Both `max_new_tokens` (=2048) and `max_length`(=1055) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1050) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  49%|████▉     | 326/660 [4:26:50<4:36:33, 49.68s/it]Both `max_new_tokens` (=2048) and `max_length`(=1055) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1049) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  50%|████▉     | 327/660 [4:27:41<4:37:58, 50.08s/it]Both `max_new_tokens` (=2048) and `max_length`(=1054) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1049) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  50%|████▉     | 328/660 [4:28:30<4:35:43, 49.83s/it]Both `max_new_tokens` (=2048) and `max_length`(=1054) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1049) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  50%|████▉     | 329/660 [4:29:22<4:37:19, 50.27s/it]Both `max_new_tokens` (=2048) and `max_length`(=1053) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1049) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  50%|█████     | 330/660 [4:30:13<4:38:18, 50.60s/it]Both `max_new_tokens` (=2048) and `max_length`(=1053) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  50%|█████     | 331/660 [4:31:03<4:35:45, 50.29s/it]Both `max_new_tokens` (=2048) and `max_length`(=1053) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  50%|█████     | 332/660 [4:31:52<4:34:12, 50.16s/it]Both `max_new_tokens` (=2048) and `max_length`(=1053) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  50%|█████     | 333/660 [4:32:41<4:31:07, 49.75s/it]Both `max_new_tokens` (=2048) and `max_length`(=1052) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  51%|█████     | 334/660 [4:33:30<4:28:28, 49.41s/it]Both `max_new_tokens` (=2048) and `max_length`(=1052) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  51%|█████     | 335/660 [4:34:18<4:26:15, 49.16s/it]Both `max_new_tokens` (=2048) and `max_length`(=1052) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1047) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  51%|█████     | 336/660 [4:35:07<4:24:32, 48.99s/it]Both `max_new_tokens` (=2048) and `max_length`(=1051) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1047) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1047) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  51%|█████     | 337/660 [4:35:55<4:22:51, 48.83s/it]Both `max_new_tokens` (=2048) and `max_length`(=1051) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1046) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  51%|█████     | 338/660 [4:36:44<4:21:40, 48.76s/it]Both `max_new_tokens` (=2048) and `max_length`(=1051) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1045) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  51%|█████▏    | 339/660 [4:37:33<4:20:43, 48.73s/it]Both `max_new_tokens` (=2048) and `max_length`(=1051) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1045) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  52%|█████▏    | 340/660 [4:38:21<4:19:38, 48.68s/it]Both `max_new_tokens` (=2048) and `max_length`(=1050) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1044) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  52%|█████▏    | 341/660 [4:39:10<4:18:53, 48.69s/it]Both `max_new_tokens` (=2048) and `max_length`(=1049) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1044) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  52%|█████▏    | 342/660 [4:39:58<4:17:44, 48.63s/it]Both `max_new_tokens` (=2048) and `max_length`(=1049) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1044) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  52%|█████▏    | 343/660 [4:40:47<4:16:43, 48.59s/it]Both `max_new_tokens` (=2048) and `max_length`(=1049) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1044) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  52%|█████▏    | 344/660 [4:41:36<4:16:31, 48.71s/it]Both `max_new_tokens` (=2048) and `max_length`(=1049) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1044) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  52%|█████▏    | 345/660 [4:42:25<4:16:21, 48.83s/it]Both `max_new_tokens` (=2048) and `max_length`(=1048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1043) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  52%|█████▏    | 346/660 [4:43:14<4:16:13, 48.96s/it]Both `max_new_tokens` (=2048) and `max_length`(=1047) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1043) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  53%|█████▎    | 347/660 [4:44:04<4:15:51, 49.05s/it]Both `max_new_tokens` (=2048) and `max_length`(=1047) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1043) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  53%|█████▎    | 348/660 [4:44:53<4:14:57, 49.03s/it]Both `max_new_tokens` (=2048) and `max_length`(=1047) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1042) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  53%|█████▎    | 349/660 [4:45:42<4:14:25, 49.09s/it]Both `max_new_tokens` (=2048) and `max_length`(=1047) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1042) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  53%|█████▎    | 350/660 [4:46:31<4:13:49, 49.13s/it]Both `max_new_tokens` (=2048) and `max_length`(=1046) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1042) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  53%|█████▎    | 351/660 [4:47:20<4:12:57, 49.12s/it]Both `max_new_tokens` (=2048) and `max_length`(=1044) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1042) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  53%|█████▎    | 352/660 [4:48:09<4:12:17, 49.15s/it]Both `max_new_tokens` (=2048) and `max_length`(=1044) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1041) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  53%|█████▎    | 353/660 [4:48:59<4:11:35, 49.17s/it]Both `max_new_tokens` (=2048) and `max_length`(=1044) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1041) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  54%|█████▎    | 354/660 [4:49:48<4:11:18, 49.28s/it]Both `max_new_tokens` (=2048) and `max_length`(=1044) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1041) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  54%|█████▍    | 355/660 [4:50:37<4:10:38, 49.31s/it]Both `max_new_tokens` (=2048) and `max_length`(=1044) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1040) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  54%|█████▍    | 356/660 [4:51:27<4:09:50, 49.31s/it]Both `max_new_tokens` (=2048) and `max_length`(=1043) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1038) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  54%|█████▍    | 357/660 [4:52:16<4:09:22, 49.38s/it]Both `max_new_tokens` (=2048) and `max_length`(=1043) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1037) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  54%|█████▍    | 358/660 [4:53:06<4:08:34, 49.39s/it]Both `max_new_tokens` (=2048) and `max_length`(=1043) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1037) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  54%|█████▍    | 359/660 [4:53:55<4:07:13, 49.28s/it]Both `max_new_tokens` (=2048) and `max_length`(=1043) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1037) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  55%|█████▍    | 360/660 [4:54:44<4:06:24, 49.28s/it]Both `max_new_tokens` (=2048) and `max_length`(=1043) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1036) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  55%|█████▍    | 361/660 [4:55:33<4:05:28, 49.26s/it]Both `max_new_tokens` (=2048) and `max_length`(=1043) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1036) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1035) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  55%|█████▍    | 362/660 [4:56:22<4:04:21, 49.20s/it]Both `max_new_tokens` (=2048) and `max_length`(=1043) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1035) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  55%|█████▌    | 363/660 [4:57:12<4:03:46, 49.25s/it]Both `max_new_tokens` (=2048) and `max_length`(=1042) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  55%|█████▌    | 364/660 [4:57:13<2:51:26, 34.75s/it]Both `max_new_tokens` (=2048) and `max_length`(=1042) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1035) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  55%|█████▌    | 365/660 [4:58:02<3:12:18, 39.11s/it]Both `max_new_tokens` (=2048) and `max_length`(=1042) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1035) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  55%|█████▌    | 366/660 [4:58:51<3:26:22, 42.12s/it]Both `max_new_tokens` (=2048) and `max_length`(=1042) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1034) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  56%|█████▌    | 367/660 [4:59:40<3:36:13, 44.28s/it]Both `max_new_tokens` (=2048) and `max_length`(=1041) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1034) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  56%|█████▌    | 368/660 [5:00:29<3:42:35, 45.74s/it]Both `max_new_tokens` (=2048) and `max_length`(=1041) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1034) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  56%|█████▌    | 369/660 [5:01:19<3:47:10, 46.84s/it]Both `max_new_tokens` (=2048) and `max_length`(=1041) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1034) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  56%|█████▌    | 370/660 [5:02:08<3:50:09, 47.62s/it]Both `max_new_tokens` (=2048) and `max_length`(=1040) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1034) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  56%|█████▌    | 371/660 [5:02:57<3:51:35, 48.08s/it]Both `max_new_tokens` (=2048) and `max_length`(=1040) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  56%|█████▋    | 372/660 [5:02:58<2:42:29, 33.85s/it]Both `max_new_tokens` (=2048) and `max_length`(=1040) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1033) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  57%|█████▋    | 373/660 [5:03:47<3:04:06, 38.49s/it]Both `max_new_tokens` (=2048) and `max_length`(=1039) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1031) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  57%|█████▋    | 374/660 [5:04:37<3:19:03, 41.76s/it]Both `max_new_tokens` (=2048) and `max_length`(=1039) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1031) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  57%|█████▋    | 375/660 [5:05:26<3:28:57, 43.99s/it]Both `max_new_tokens` (=2048) and `max_length`(=1037) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1030) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  57%|█████▋    | 376/660 [5:06:15<3:35:59, 45.63s/it]Both `max_new_tokens` (=2048) and `max_length`(=1036) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1030) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  57%|█████▋    | 377/660 [5:07:06<3:42:01, 47.07s/it]Both `max_new_tokens` (=2048) and `max_length`(=1036) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1029) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  57%|█████▋    | 378/660 [5:07:55<3:44:06, 47.68s/it]Both `max_new_tokens` (=2048) and `max_length`(=1036) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1029) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  57%|█████▋    | 379/660 [5:08:44<3:44:57, 48.03s/it]Both `max_new_tokens` (=2048) and `max_length`(=1035) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1029) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  58%|█████▊    | 380/660 [5:09:33<3:45:25, 48.31s/it]Both `max_new_tokens` (=2048) and `max_length`(=1035) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1028) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  58%|█████▊    | 381/660 [5:10:21<3:45:06, 48.41s/it]Both `max_new_tokens` (=2048) and `max_length`(=1035) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1026) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  58%|█████▊    | 382/660 [5:11:10<3:44:40, 48.49s/it]Both `max_new_tokens` (=2048) and `max_length`(=1034) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1025) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  58%|█████▊    | 383/660 [5:11:59<3:44:02, 48.53s/it]Both `max_new_tokens` (=2048) and `max_length`(=1034) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1024) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  58%|█████▊    | 384/660 [5:12:47<3:43:24, 48.57s/it]Both `max_new_tokens` (=2048) and `max_length`(=1034) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1024) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  58%|█████▊    | 385/660 [5:13:36<3:42:44, 48.60s/it]Both `max_new_tokens` (=2048) and `max_length`(=1034) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1024) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  58%|█████▊    | 386/660 [5:14:25<3:41:54, 48.59s/it]Both `max_new_tokens` (=2048) and `max_length`(=1032) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1023) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  59%|█████▊    | 387/660 [5:15:13<3:41:17, 48.63s/it]Both `max_new_tokens` (=2048) and `max_length`(=1032) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1022) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  59%|█████▉    | 388/660 [5:16:02<3:40:21, 48.61s/it]Both `max_new_tokens` (=2048) and `max_length`(=1031) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1022) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  59%|█████▉    | 389/660 [5:16:51<3:39:31, 48.60s/it]Both `max_new_tokens` (=2048) and `max_length`(=1030) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1022) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  59%|█████▉    | 390/660 [5:17:40<3:39:16, 48.73s/it]Both `max_new_tokens` (=2048) and `max_length`(=1029) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1022) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  59%|█████▉    | 391/660 [5:18:30<3:41:27, 49.40s/it]Both `max_new_tokens` (=2048) and `max_length`(=1029) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1021) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  59%|█████▉    | 392/660 [5:19:21<3:41:59, 49.70s/it]Both `max_new_tokens` (=2048) and `max_length`(=1028) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1021) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  60%|█████▉    | 393/660 [5:20:11<3:41:02, 49.67s/it]Both `max_new_tokens` (=2048) and `max_length`(=1027) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1020) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  60%|█████▉    | 394/660 [5:21:00<3:39:59, 49.62s/it]Both `max_new_tokens` (=2048) and `max_length`(=1027) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1019) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  60%|█████▉    | 395/660 [5:21:50<3:39:02, 49.60s/it]Both `max_new_tokens` (=2048) and `max_length`(=1027) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1019) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  60%|██████    | 396/660 [5:22:40<3:39:02, 49.78s/it]Both `max_new_tokens` (=2048) and `max_length`(=1026) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1018) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1018) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  60%|██████    | 397/660 [5:23:29<3:37:24, 49.60s/it]Both `max_new_tokens` (=2048) and `max_length`(=1026) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1017) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  60%|██████    | 398/660 [5:24:18<3:36:03, 49.48s/it]Both `max_new_tokens` (=2048) and `max_length`(=1026) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1017) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1017) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  60%|██████    | 399/660 [5:25:08<3:35:05, 49.44s/it]Both `max_new_tokens` (=2048) and `max_length`(=1026) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1017) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  61%|██████    | 400/660 [5:25:57<3:34:32, 49.51s/it]Both `max_new_tokens` (=2048) and `max_length`(=1025) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1016) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  61%|██████    | 401/660 [5:26:47<3:33:35, 49.48s/it]Both `max_new_tokens` (=2048) and `max_length`(=1025) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1016) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  61%|██████    | 402/660 [5:27:37<3:33:30, 49.65s/it]Both `max_new_tokens` (=2048) and `max_length`(=1025) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1016) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  61%|██████    | 403/660 [5:28:27<3:33:56, 49.95s/it]Both `max_new_tokens` (=2048) and `max_length`(=1024) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1016) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  61%|██████    | 404/660 [5:29:17<3:32:41, 49.85s/it]Both `max_new_tokens` (=2048) and `max_length`(=1024) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1015) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  61%|██████▏   | 405/660 [5:30:07<3:31:58, 49.88s/it]Both `max_new_tokens` (=2048) and `max_length`(=1024) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1015) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  62%|██████▏   | 406/660 [5:30:58<3:32:16, 50.14s/it]Both `max_new_tokens` (=2048) and `max_length`(=1024) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1015) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  62%|██████▏   | 407/660 [5:31:48<3:31:30, 50.16s/it]Both `max_new_tokens` (=2048) and `max_length`(=1024) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1014) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  62%|██████▏   | 408/660 [5:32:37<3:29:03, 49.78s/it]Both `max_new_tokens` (=2048) and `max_length`(=1023) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1014) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  62%|██████▏   | 409/660 [5:33:26<3:27:28, 49.59s/it]Both `max_new_tokens` (=2048) and `max_length`(=1022) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1013) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  62%|██████▏   | 410/660 [5:34:15<3:25:51, 49.40s/it]Both `max_new_tokens` (=2048) and `max_length`(=1021) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1012) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  62%|██████▏   | 411/660 [5:35:04<3:24:33, 49.29s/it]Both `max_new_tokens` (=2048) and `max_length`(=1021) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1012) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  62%|██████▏   | 412/660 [5:35:55<3:25:28, 49.71s/it]Both `max_new_tokens` (=2048) and `max_length`(=1021) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1011) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  63%|██████▎   | 413/660 [5:36:44<3:24:47, 49.75s/it]Both `max_new_tokens` (=2048) and `max_length`(=1021) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1011) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  63%|██████▎   | 414/660 [5:37:33<3:23:01, 49.52s/it]Both `max_new_tokens` (=2048) and `max_length`(=1020) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1010) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  63%|██████▎   | 415/660 [5:38:22<3:21:37, 49.38s/it]Both `max_new_tokens` (=2048) and `max_length`(=1020) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1010) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  63%|██████▎   | 416/660 [5:39:11<3:20:12, 49.23s/it]Both `max_new_tokens` (=2048) and `max_length`(=1020) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1009) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  63%|██████▎   | 417/660 [5:40:00<3:19:01, 49.14s/it]Both `max_new_tokens` (=2048) and `max_length`(=1019) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1008) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  63%|██████▎   | 418/660 [5:40:49<3:17:51, 49.06s/it]Both `max_new_tokens` (=2048) and `max_length`(=1019) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1007) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  63%|██████▎   | 419/660 [5:41:38<3:16:35, 48.95s/it]Both `max_new_tokens` (=2048) and `max_length`(=1018) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1006) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  64%|██████▎   | 420/660 [5:42:27<3:15:38, 48.91s/it]Both `max_new_tokens` (=2048) and `max_length`(=1018) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1006) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  64%|██████▍   | 421/660 [5:43:15<3:14:36, 48.86s/it]Both `max_new_tokens` (=2048) and `max_length`(=1015) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1006) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  64%|██████▍   | 422/660 [5:44:04<3:13:40, 48.83s/it]Both `max_new_tokens` (=2048) and `max_length`(=1015) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1005) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1005) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  64%|██████▍   | 423/660 [5:44:53<3:13:04, 48.88s/it]Both `max_new_tokens` (=2048) and `max_length`(=1015) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1004) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  64%|██████▍   | 424/660 [5:45:44<3:14:33, 49.46s/it]Both `max_new_tokens` (=2048) and `max_length`(=1015) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1004) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  64%|██████▍   | 425/660 [5:46:34<3:14:14, 49.59s/it]Both `max_new_tokens` (=2048) and `max_length`(=1014) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1003) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  65%|██████▍   | 426/660 [5:47:23<3:12:44, 49.42s/it]Both `max_new_tokens` (=2048) and `max_length`(=1014) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=998) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  65%|██████▍   | 427/660 [5:48:12<3:11:08, 49.22s/it]Both `max_new_tokens` (=2048) and `max_length`(=1013) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=997) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  65%|██████▍   | 428/660 [5:49:02<3:11:39, 49.57s/it]Both `max_new_tokens` (=2048) and `max_length`(=1012) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=996) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  65%|██████▌   | 429/660 [5:49:53<3:12:36, 50.03s/it]Both `max_new_tokens` (=2048) and `max_length`(=1012) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=995) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  65%|██████▌   | 430/660 [5:50:42<3:10:22, 49.66s/it]Both `max_new_tokens` (=2048) and `max_length`(=1011) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=994) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  65%|██████▌   | 431/660 [5:51:32<3:10:07, 49.82s/it]Both `max_new_tokens` (=2048) and `max_length`(=1011) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=994) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  65%|██████▌   | 432/660 [5:52:23<3:10:50, 50.22s/it]Both `max_new_tokens` (=2048) and `max_length`(=1010) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=993) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  66%|██████▌   | 433/660 [5:53:14<3:10:46, 50.42s/it]Both `max_new_tokens` (=2048) and `max_length`(=1010) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=993) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  66%|██████▌   | 434/660 [5:54:04<3:09:01, 50.19s/it]Both `max_new_tokens` (=2048) and `max_length`(=1010) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=993) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  66%|██████▌   | 435/660 [5:54:54<3:08:01, 50.14s/it]Both `max_new_tokens` (=2048) and `max_length`(=1010) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=992) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  66%|██████▌   | 436/660 [5:55:44<3:06:50, 50.05s/it]Both `max_new_tokens` (=2048) and `max_length`(=1009) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=992) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  66%|██████▌   | 437/660 [5:56:33<3:05:32, 49.92s/it]Both `max_new_tokens` (=2048) and `max_length`(=1009) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=991) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  66%|██████▋   | 438/660 [5:57:23<3:04:43, 49.93s/it]Both `max_new_tokens` (=2048) and `max_length`(=1009) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=990) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  67%|██████▋   | 439/660 [5:58:13<3:03:25, 49.80s/it]Both `max_new_tokens` (=2048) and `max_length`(=1009) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=990) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  67%|██████▋   | 440/660 [5:59:02<3:02:10, 49.69s/it]Both `max_new_tokens` (=2048) and `max_length`(=1008) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=990) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  67%|██████▋   | 441/660 [5:59:52<3:01:07, 49.62s/it]Both `max_new_tokens` (=2048) and `max_length`(=1007) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=989) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  67%|██████▋   | 442/660 [6:00:41<3:00:07, 49.57s/it]Both `max_new_tokens` (=2048) and `max_length`(=1006) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=989) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  67%|██████▋   | 443/660 [6:01:30<2:59:10, 49.54s/it]Both `max_new_tokens` (=2048) and `max_length`(=1005) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=987) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  67%|██████▋   | 444/660 [6:02:20<2:58:23, 49.55s/it]Both `max_new_tokens` (=2048) and `max_length`(=1004) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=987) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  67%|██████▋   | 445/660 [6:03:10<2:57:31, 49.54s/it]Both `max_new_tokens` (=2048) and `max_length`(=1004) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=986) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=986) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  68%|██████▊   | 446/660 [6:03:59<2:56:45, 49.56s/it]Both `max_new_tokens` (=2048) and `max_length`(=1003) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=985) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  68%|██████▊   | 447/660 [6:04:49<2:56:04, 49.60s/it]Both `max_new_tokens` (=2048) and `max_length`(=1003) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=984) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  68%|██████▊   | 448/660 [6:05:38<2:55:07, 49.57s/it]Both `max_new_tokens` (=2048) and `max_length`(=1003) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=984) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  68%|██████▊   | 449/660 [6:06:28<2:54:09, 49.52s/it]Both `max_new_tokens` (=2048) and `max_length`(=1002) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=984) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  68%|██████▊   | 450/660 [6:07:17<2:53:20, 49.53s/it]Both `max_new_tokens` (=2048) and `max_length`(=1002) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=983) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  68%|██████▊   | 451/660 [6:08:07<2:52:35, 49.55s/it]Both `max_new_tokens` (=2048) and `max_length`(=1002) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  68%|██████▊   | 452/660 [6:08:09<2:02:36, 35.37s/it]Both `max_new_tokens` (=2048) and `max_length`(=1002) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=983) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=983) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  69%|██████▊   | 453/660 [6:08:59<2:17:10, 39.76s/it]Both `max_new_tokens` (=2048) and `max_length`(=1002) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=982) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  69%|██████▉   | 454/660 [6:09:49<2:27:12, 42.88s/it]Both `max_new_tokens` (=2048) and `max_length`(=1002) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=982) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  69%|██████▉   | 455/660 [6:10:39<2:33:52, 45.04s/it]Both `max_new_tokens` (=2048) and `max_length`(=1001) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=982) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  69%|██████▉   | 456/660 [6:11:29<2:37:58, 46.46s/it]Both `max_new_tokens` (=2048) and `max_length`(=1001) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=981) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  69%|██████▉   | 457/660 [6:12:19<2:40:31, 47.44s/it]Both `max_new_tokens` (=2048) and `max_length`(=1000) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=980) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  69%|██████▉   | 458/660 [6:13:09<2:41:58, 48.11s/it]Both `max_new_tokens` (=2048) and `max_length`(=998) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=979) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  70%|██████▉   | 459/660 [6:13:59<2:43:31, 48.81s/it]Both `max_new_tokens` (=2048) and `max_length`(=998) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=978) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  70%|██████▉   | 460/660 [6:14:50<2:45:06, 49.53s/it]Both `max_new_tokens` (=2048) and `max_length`(=998) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=978) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  70%|██████▉   | 461/660 [6:15:40<2:44:26, 49.58s/it]Both `max_new_tokens` (=2048) and `max_length`(=997) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=978) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  70%|███████   | 462/660 [6:16:30<2:43:43, 49.61s/it]Both `max_new_tokens` (=2048) and `max_length`(=997) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=977) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  70%|███████   | 463/660 [6:17:19<2:42:40, 49.55s/it]Both `max_new_tokens` (=2048) and `max_length`(=997) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=977) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  70%|███████   | 464/660 [6:18:08<2:41:03, 49.30s/it]Both `max_new_tokens` (=2048) and `max_length`(=997) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=976) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  70%|███████   | 465/660 [6:18:57<2:39:43, 49.14s/it]Both `max_new_tokens` (=2048) and `max_length`(=996) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=974) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  71%|███████   | 466/660 [6:19:45<2:38:18, 48.96s/it]Both `max_new_tokens` (=2048) and `max_length`(=996) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=974) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  71%|███████   | 467/660 [6:20:34<2:37:14, 48.88s/it]Both `max_new_tokens` (=2048) and `max_length`(=995) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=973) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  71%|███████   | 468/660 [6:21:23<2:36:14, 48.83s/it]Both `max_new_tokens` (=2048) and `max_length`(=994) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=973) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  71%|███████   | 469/660 [6:22:12<2:35:48, 48.95s/it]Both `max_new_tokens` (=2048) and `max_length`(=994) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=972) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  71%|███████   | 470/660 [6:23:01<2:35:25, 49.08s/it]Both `max_new_tokens` (=2048) and `max_length`(=993) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=972) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  71%|███████▏  | 471/660 [6:23:50<2:34:50, 49.16s/it]Both `max_new_tokens` (=2048) and `max_length`(=993) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=971) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  72%|███████▏  | 472/660 [6:24:40<2:34:03, 49.17s/it]Both `max_new_tokens` (=2048) and `max_length`(=990) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=970) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=970) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=970) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  72%|███████▏  | 473/660 [6:25:29<2:33:25, 49.23s/it]Both `max_new_tokens` (=2048) and `max_length`(=990) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=970) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  72%|███████▏  | 474/660 [6:26:18<2:32:33, 49.21s/it]Both `max_new_tokens` (=2048) and `max_length`(=989) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=969) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  72%|███████▏  | 475/660 [6:27:08<2:31:49, 49.24s/it]Both `max_new_tokens` (=2048) and `max_length`(=989) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=969) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  72%|███████▏  | 476/660 [6:27:57<2:31:11, 49.30s/it]Both `max_new_tokens` (=2048) and `max_length`(=988) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=968) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  72%|███████▏  | 477/660 [6:28:46<2:30:18, 49.28s/it]Both `max_new_tokens` (=2048) and `max_length`(=988) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=966) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  72%|███████▏  | 478/660 [6:29:36<2:29:36, 49.32s/it]Both `max_new_tokens` (=2048) and `max_length`(=987) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=966) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  73%|███████▎  | 479/660 [6:30:25<2:28:48, 49.33s/it]Both `max_new_tokens` (=2048) and `max_length`(=987) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=966) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  73%|███████▎  | 480/660 [6:31:14<2:27:49, 49.28s/it]Both `max_new_tokens` (=2048) and `max_length`(=987) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  73%|███████▎  | 481/660 [6:31:17<1:45:35, 35.39s/it]Both `max_new_tokens` (=2048) and `max_length`(=986) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=965) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  73%|███████▎  | 482/660 [6:32:06<1:56:56, 39.42s/it]Both `max_new_tokens` (=2048) and `max_length`(=984) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=965) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  73%|███████▎  | 483/660 [6:32:55<2:04:36, 42.24s/it]Both `max_new_tokens` (=2048) and `max_length`(=984) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=963) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=963) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  73%|███████▎  | 484/660 [6:33:43<2:09:30, 44.15s/it]Both `max_new_tokens` (=2048) and `max_length`(=984) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=963) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  73%|███████▎  | 485/660 [6:34:32<2:12:52, 45.56s/it]Both `max_new_tokens` (=2048) and `max_length`(=983) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=962) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  74%|███████▎  | 486/660 [6:35:21<2:14:45, 46.47s/it]Both `max_new_tokens` (=2048) and `max_length`(=983) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=962) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  74%|███████▍  | 487/660 [6:36:10<2:15:56, 47.15s/it]Both `max_new_tokens` (=2048) and `max_length`(=982) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=962) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  74%|███████▍  | 488/660 [6:36:58<2:16:36, 47.65s/it]Both `max_new_tokens` (=2048) and `max_length`(=979) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=962) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  74%|███████▍  | 489/660 [6:37:47<2:16:40, 47.96s/it]Both `max_new_tokens` (=2048) and `max_length`(=979) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=961) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  74%|███████▍  | 490/660 [6:38:36<2:16:25, 48.15s/it]Both `max_new_tokens` (=2048) and `max_length`(=977) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=961) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  74%|███████▍  | 491/660 [6:39:24<2:16:03, 48.30s/it]Both `max_new_tokens` (=2048) and `max_length`(=977) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=961) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  75%|███████▍  | 492/660 [6:40:13<2:15:39, 48.45s/it]Both `max_new_tokens` (=2048) and `max_length`(=976) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=960) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  75%|███████▍  | 493/660 [6:41:03<2:16:15, 48.96s/it]Both `max_new_tokens` (=2048) and `max_length`(=976) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=960) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  75%|███████▍  | 494/660 [6:41:52<2:15:28, 48.97s/it]Both `max_new_tokens` (=2048) and `max_length`(=975) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=959) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  75%|███████▌  | 495/660 [6:42:41<2:14:34, 48.94s/it]Both `max_new_tokens` (=2048) and `max_length`(=975) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=959) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  75%|███████▌  | 496/660 [6:43:30<2:13:39, 48.90s/it]Both `max_new_tokens` (=2048) and `max_length`(=975) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=959) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=958) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  75%|███████▌  | 497/660 [6:44:19<2:12:46, 48.87s/it]Both `max_new_tokens` (=2048) and `max_length`(=974) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=958) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  75%|███████▌  | 498/660 [6:45:07<2:11:49, 48.83s/it]Both `max_new_tokens` (=2048) and `max_length`(=974) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=958) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  76%|███████▌  | 499/660 [6:45:56<2:11:01, 48.83s/it]Both `max_new_tokens` (=2048) and `max_length`(=974) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=957) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  76%|███████▌  | 500/660 [6:46:45<2:10:06, 48.79s/it]Both `max_new_tokens` (=2048) and `max_length`(=974) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=957) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  76%|███████▌  | 501/660 [6:47:34<2:09:25, 48.84s/it]Both `max_new_tokens` (=2048) and `max_length`(=973) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=957) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  76%|███████▌  | 502/660 [6:48:23<2:08:41, 48.87s/it]Both `max_new_tokens` (=2048) and `max_length`(=971) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=956) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  76%|███████▌  | 503/660 [6:49:12<2:08:01, 48.93s/it]Both `max_new_tokens` (=2048) and `max_length`(=971) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=955) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  76%|███████▋  | 504/660 [6:50:02<2:08:01, 49.24s/it]Both `max_new_tokens` (=2048) and `max_length`(=971) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=955) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  77%|███████▋  | 505/660 [6:50:52<2:07:40, 49.42s/it]Both `max_new_tokens` (=2048) and `max_length`(=970) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=955) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  77%|███████▋  | 506/660 [6:51:41<2:06:41, 49.36s/it]Both `max_new_tokens` (=2048) and `max_length`(=970) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=953) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  77%|███████▋  | 507/660 [6:52:30<2:05:44, 49.31s/it]Both `max_new_tokens` (=2048) and `max_length`(=969) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=953) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  77%|███████▋  | 508/660 [6:53:20<2:05:41, 49.62s/it]Both `max_new_tokens` (=2048) and `max_length`(=969) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=953) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=953) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  77%|███████▋  | 509/660 [6:54:10<2:04:57, 49.65s/it]Both `max_new_tokens` (=2048) and `max_length`(=969) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=951) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  77%|███████▋  | 510/660 [6:54:59<2:03:20, 49.34s/it]Both `max_new_tokens` (=2048) and `max_length`(=967) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=950) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  77%|███████▋  | 511/660 [6:55:47<2:01:51, 49.07s/it]Both `max_new_tokens` (=2048) and `max_length`(=967) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=949) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  78%|███████▊  | 512/660 [6:56:36<2:00:47, 48.97s/it]Both `max_new_tokens` (=2048) and `max_length`(=967) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=949) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  78%|███████▊  | 513/660 [6:57:25<1:59:44, 48.88s/it]Both `max_new_tokens` (=2048) and `max_length`(=967) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=949) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  78%|███████▊  | 514/660 [6:58:15<1:59:51, 49.26s/it]Both `max_new_tokens` (=2048) and `max_length`(=966) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=949) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  78%|███████▊  | 515/660 [6:59:04<1:59:22, 49.39s/it]Both `max_new_tokens` (=2048) and `max_length`(=966) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=949) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  78%|███████▊  | 516/660 [6:59:53<1:58:05, 49.21s/it]Both `max_new_tokens` (=2048) and `max_length`(=965) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=948) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  78%|███████▊  | 517/660 [7:00:42<1:56:42, 48.97s/it]Both `max_new_tokens` (=2048) and `max_length`(=965) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=948) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  78%|███████▊  | 518/660 [7:01:30<1:55:31, 48.81s/it]Both `max_new_tokens` (=2048) and `max_length`(=965) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=948) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  79%|███████▊  | 519/660 [7:02:19<1:54:25, 48.69s/it]Both `max_new_tokens` (=2048) and `max_length`(=964) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=947) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  79%|███████▉  | 520/660 [7:03:07<1:53:29, 48.64s/it]Both `max_new_tokens` (=2048) and `max_length`(=964) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=947) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  79%|███████▉  | 521/660 [7:03:55<1:52:32, 48.58s/it]Both `max_new_tokens` (=2048) and `max_length`(=963) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=946) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  79%|███████▉  | 522/660 [7:04:44<1:51:33, 48.50s/it]Both `max_new_tokens` (=2048) and `max_length`(=962) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=946) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  79%|███████▉  | 523/660 [7:05:32<1:50:45, 48.51s/it]Both `max_new_tokens` (=2048) and `max_length`(=961) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=945) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  79%|███████▉  | 524/660 [7:06:21<1:49:51, 48.47s/it]Both `max_new_tokens` (=2048) and `max_length`(=961) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=944) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  80%|███████▉  | 525/660 [7:07:09<1:48:56, 48.42s/it]Both `max_new_tokens` (=2048) and `max_length`(=961) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=944) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  80%|███████▉  | 526/660 [7:07:59<1:49:01, 48.82s/it]Both `max_new_tokens` (=2048) and `max_length`(=960) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=944) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  80%|███████▉  | 527/660 [7:08:49<1:49:07, 49.23s/it]Both `max_new_tokens` (=2048) and `max_length`(=960) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=942) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  80%|████████  | 528/660 [7:09:38<1:48:09, 49.16s/it]Both `max_new_tokens` (=2048) and `max_length`(=959) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=941) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  80%|████████  | 529/660 [7:10:27<1:46:58, 49.00s/it]Both `max_new_tokens` (=2048) and `max_length`(=959) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=941) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  80%|████████  | 530/660 [7:11:15<1:45:42, 48.79s/it]Both `max_new_tokens` (=2048) and `max_length`(=959) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=940) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  80%|████████  | 531/660 [7:12:05<1:45:51, 49.24s/it]Both `max_new_tokens` (=2048) and `max_length`(=959) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=940) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=939) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  81%|████████  | 532/660 [7:12:56<1:45:51, 49.62s/it]Both `max_new_tokens` (=2048) and `max_length`(=957) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=937) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  81%|████████  | 533/660 [7:13:44<1:44:21, 49.30s/it]Both `max_new_tokens` (=2048) and `max_length`(=957) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=936) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  81%|████████  | 534/660 [7:14:34<1:44:07, 49.58s/it]Both `max_new_tokens` (=2048) and `max_length`(=957) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=936) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  81%|████████  | 535/660 [7:15:25<1:43:53, 49.87s/it]Both `max_new_tokens` (=2048) and `max_length`(=957) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=936) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  81%|████████  | 536/660 [7:16:14<1:42:29, 49.59s/it]Both `max_new_tokens` (=2048) and `max_length`(=956) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=935) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  81%|████████▏ | 537/660 [7:17:03<1:41:02, 49.29s/it]Both `max_new_tokens` (=2048) and `max_length`(=956) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=935) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  82%|████████▏ | 538/660 [7:17:51<1:39:48, 49.08s/it]Both `max_new_tokens` (=2048) and `max_length`(=953) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=935) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  82%|████████▏ | 539/660 [7:18:40<1:38:47, 48.99s/it]Both `max_new_tokens` (=2048) and `max_length`(=953) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=935) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  82%|████████▏ | 540/660 [7:19:29<1:37:55, 48.96s/it]Both `max_new_tokens` (=2048) and `max_length`(=952) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=934) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  82%|████████▏ | 541/660 [7:20:18<1:36:57, 48.89s/it]Both `max_new_tokens` (=2048) and `max_length`(=952) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=934) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  82%|████████▏ | 542/660 [7:21:06<1:36:06, 48.87s/it]Both `max_new_tokens` (=2048) and `max_length`(=952) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=933) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  82%|████████▏ | 543/660 [7:21:56<1:35:30, 48.98s/it]Both `max_new_tokens` (=2048) and `max_length`(=951) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=927) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  82%|████████▏ | 544/660 [7:22:45<1:34:46, 49.02s/it]Both `max_new_tokens` (=2048) and `max_length`(=950) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=927) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  83%|████████▎ | 545/660 [7:23:34<1:33:50, 48.96s/it]Both `max_new_tokens` (=2048) and `max_length`(=950) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=926) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  83%|████████▎ | 546/660 [7:24:22<1:32:48, 48.85s/it]Both `max_new_tokens` (=2048) and `max_length`(=949) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=926) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  83%|████████▎ | 547/660 [7:25:11<1:31:54, 48.80s/it]Both `max_new_tokens` (=2048) and `max_length`(=949) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  83%|████████▎ | 548/660 [7:25:12<1:04:18, 34.45s/it]Both `max_new_tokens` (=2048) and `max_length`(=947) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=926) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  83%|████████▎ | 549/660 [7:26:01<1:12:00, 38.92s/it]Both `max_new_tokens` (=2048) and `max_length`(=947) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=925) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  83%|████████▎ | 550/660 [7:26:50<1:16:42, 41.84s/it]Both `max_new_tokens` (=2048) and `max_length`(=946) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=925) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  83%|████████▎ | 551/660 [7:27:39<1:19:49, 43.94s/it]Both `max_new_tokens` (=2048) and `max_length`(=945) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=924) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  84%|████████▎ | 552/660 [7:28:27<1:21:42, 45.39s/it]Both `max_new_tokens` (=2048) and `max_length`(=945) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=922) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  84%|████████▍ | 553/660 [7:29:16<1:22:46, 46.41s/it]Both `max_new_tokens` (=2048) and `max_length`(=945) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=922) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  84%|████████▍ | 554/660 [7:30:05<1:23:21, 47.18s/it]Both `max_new_tokens` (=2048) and `max_length`(=944) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=921) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  84%|████████▍ | 555/660 [7:30:54<1:23:20, 47.62s/it]Both `max_new_tokens` (=2048) and `max_length`(=944) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=919) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  84%|████████▍ | 556/660 [7:31:43<1:23:06, 47.94s/it]Both `max_new_tokens` (=2048) and `max_length`(=944) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=918) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  84%|████████▍ | 557/660 [7:32:31<1:22:47, 48.23s/it]Both `max_new_tokens` (=2048) and `max_length`(=943) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=917) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  85%|████████▍ | 558/660 [7:33:20<1:22:20, 48.43s/it]Both `max_new_tokens` (=2048) and `max_length`(=942) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=916) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  85%|████████▍ | 559/660 [7:34:09<1:21:45, 48.57s/it]Both `max_new_tokens` (=2048) and `max_length`(=941) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=915) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=915) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  85%|████████▍ | 560/660 [7:34:58<1:21:02, 48.63s/it]Both `max_new_tokens` (=2048) and `max_length`(=940) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  85%|████████▌ | 561/660 [7:34:59<56:41, 34.36s/it]  Both `max_new_tokens` (=2048) and `max_length`(=938) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=914) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  85%|████████▌ | 562/660 [7:35:48<1:03:08, 38.66s/it]Both `max_new_tokens` (=2048) and `max_length`(=936) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=914) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=912) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=912) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  85%|████████▌ | 563/660 [7:36:37<1:07:39, 41.85s/it]Both `max_new_tokens` (=2048) and `max_length`(=936) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=910) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=910) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  85%|████████▌ | 564/660 [7:37:26<1:10:21, 43.97s/it]Both `max_new_tokens` (=2048) and `max_length`(=935) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=910) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  86%|████████▌ | 565/660 [7:38:15<1:11:55, 45.43s/it]Both `max_new_tokens` (=2048) and `max_length`(=932) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=910) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  86%|████████▌ | 566/660 [7:39:04<1:12:46, 46.46s/it]Both `max_new_tokens` (=2048) and `max_length`(=928) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=910) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  86%|████████▌ | 567/660 [7:39:52<1:13:06, 47.16s/it]Both `max_new_tokens` (=2048) and `max_length`(=928) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=909) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  86%|████████▌ | 568/660 [7:40:41<1:12:59, 47.61s/it]Both `max_new_tokens` (=2048) and `max_length`(=928) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=909) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  86%|████████▌ | 569/660 [7:41:30<1:12:46, 47.99s/it]Both `max_new_tokens` (=2048) and `max_length`(=927) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=905) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  86%|████████▋ | 570/660 [7:42:19<1:12:19, 48.22s/it]Both `max_new_tokens` (=2048) and `max_length`(=927) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=903) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  87%|████████▋ | 571/660 [7:43:08<1:11:49, 48.42s/it]Both `max_new_tokens` (=2048) and `max_length`(=925) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=902) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  87%|████████▋ | 572/660 [7:43:56<1:11:13, 48.56s/it]Both `max_new_tokens` (=2048) and `max_length`(=925) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=900) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  87%|████████▋ | 573/660 [7:44:45<1:10:29, 48.62s/it]Both `max_new_tokens` (=2048) and `max_length`(=925) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=900) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  87%|████████▋ | 574/660 [7:45:34<1:09:51, 48.74s/it]Both `max_new_tokens` (=2048) and `max_length`(=923) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  87%|████████▋ | 575/660 [7:45:36<49:03, 34.63s/it]  Both `max_new_tokens` (=2048) and `max_length`(=923) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=900) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  87%|████████▋ | 576/660 [7:46:25<54:27, 38.90s/it]Both `max_new_tokens` (=2048) and `max_length`(=923) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=899) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  87%|████████▋ | 577/660 [7:47:14<57:54, 41.86s/it]Both `max_new_tokens` (=2048) and `max_length`(=921) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=898) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  88%|████████▊ | 578/660 [7:48:02<1:00:03, 43.94s/it]Both `max_new_tokens` (=2048) and `max_length`(=921) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=896) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  88%|████████▊ | 579/660 [7:48:51<1:01:13, 45.35s/it]Both `max_new_tokens` (=2048) and `max_length`(=920) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=896) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  88%|████████▊ | 580/660 [7:49:40<1:02:00, 46.50s/it]Both `max_new_tokens` (=2048) and `max_length`(=919) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=896) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  88%|████████▊ | 581/660 [7:50:29<1:02:08, 47.20s/it]Both `max_new_tokens` (=2048) and `max_length`(=915) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=896) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  88%|████████▊ | 582/660 [7:51:18<1:02:07, 47.79s/it]Both `max_new_tokens` (=2048) and `max_length`(=914) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=892) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  88%|████████▊ | 583/660 [7:52:07<1:01:46, 48.14s/it]Both `max_new_tokens` (=2048) and `max_length`(=914) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=892) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  88%|████████▊ | 584/660 [7:52:56<1:01:16, 48.38s/it]Both `max_new_tokens` (=2048) and `max_length`(=911) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=891) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  89%|████████▊ | 585/660 [7:53:45<1:00:36, 48.49s/it]Both `max_new_tokens` (=2048) and `max_length`(=910) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=890) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  89%|████████▉ | 586/660 [7:54:34<59:54, 48.57s/it]  Both `max_new_tokens` (=2048) and `max_length`(=908) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=884) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  89%|████████▉ | 587/660 [7:55:22<59:07, 48.59s/it]Both `max_new_tokens` (=2048) and `max_length`(=907) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=884) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  89%|████████▉ | 588/660 [7:56:11<58:23, 48.66s/it]Both `max_new_tokens` (=2048) and `max_length`(=905) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=884) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  89%|████████▉ | 589/660 [7:57:00<57:36, 48.69s/it]Both `max_new_tokens` (=2048) and `max_length`(=903) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=883) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=882) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  89%|████████▉ | 590/660 [7:57:49<56:50, 48.72s/it]Both `max_new_tokens` (=2048) and `max_length`(=902) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=880) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  90%|████████▉ | 591/660 [7:58:38<56:07, 48.81s/it]Both `max_new_tokens` (=2048) and `max_length`(=902) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=878) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  90%|████████▉ | 592/660 [7:59:26<55:20, 48.83s/it]Both `max_new_tokens` (=2048) and `max_length`(=900) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=878) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  90%|████████▉ | 593/660 [8:00:15<54:29, 48.79s/it]Both `max_new_tokens` (=2048) and `max_length`(=899) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=876) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  90%|█████████ | 594/660 [8:01:04<53:40, 48.79s/it]Both `max_new_tokens` (=2048) and `max_length`(=899) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=874) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  90%|█████████ | 595/660 [8:01:53<52:48, 48.75s/it]Both `max_new_tokens` (=2048) and `max_length`(=899) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  90%|█████████ | 596/660 [8:01:54<36:55, 34.61s/it]Both `max_new_tokens` (=2048) and `max_length`(=898) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=872) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=870) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  90%|█████████ | 597/660 [8:02:43<40:46, 38.83s/it]Both `max_new_tokens` (=2048) and `max_length`(=898) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=870) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  91%|█████████ | 598/660 [8:03:33<43:30, 42.10s/it]Both `max_new_tokens` (=2048) and `max_length`(=897) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=869) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=868) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  91%|█████████ | 599/660 [8:04:23<45:16, 44.53s/it]Both `max_new_tokens` (=2048) and `max_length`(=897) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=863) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  91%|█████████ | 600/660 [8:05:12<45:55, 45.93s/it]Both `max_new_tokens` (=2048) and `max_length`(=897) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=862) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  91%|█████████ | 601/660 [8:06:01<46:06, 46.89s/it]Both `max_new_tokens` (=2048) and `max_length`(=897) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=860) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  91%|█████████ | 602/660 [8:06:50<45:54, 47.49s/it]Both `max_new_tokens` (=2048) and `max_length`(=896) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=857) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  91%|█████████▏| 603/660 [8:07:39<45:32, 47.94s/it]Both `max_new_tokens` (=2048) and `max_length`(=895) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=855) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  92%|█████████▏| 604/660 [8:08:28<45:05, 48.31s/it]Both `max_new_tokens` (=2048) and `max_length`(=894) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=848) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  92%|█████████▏| 605/660 [8:09:17<44:28, 48.53s/it]Both `max_new_tokens` (=2048) and `max_length`(=893) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=847) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  92%|█████████▏| 606/660 [8:10:06<43:51, 48.73s/it]Both `max_new_tokens` (=2048) and `max_length`(=892) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=846) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  92%|█████████▏| 607/660 [8:10:56<43:09, 48.86s/it]Both `max_new_tokens` (=2048) and `max_length`(=892) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=845) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  92%|█████████▏| 608/660 [8:11:45<42:23, 48.92s/it]Both `max_new_tokens` (=2048) and `max_length`(=892) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=844) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  92%|█████████▏| 609/660 [8:12:34<41:42, 49.07s/it]Both `max_new_tokens` (=2048) and `max_length`(=891) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=843) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  92%|█████████▏| 610/660 [8:13:25<41:26, 49.74s/it]Both `max_new_tokens` (=2048) and `max_length`(=890) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=842) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  93%|█████████▎| 611/660 [8:14:16<40:54, 50.10s/it]Both `max_new_tokens` (=2048) and `max_length`(=889) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=841) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  93%|█████████▎| 612/660 [8:15:06<39:59, 50.00s/it]Both `max_new_tokens` (=2048) and `max_length`(=887) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=841) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  93%|█████████▎| 613/660 [8:15:57<39:20, 50.21s/it]Both `max_new_tokens` (=2048) and `max_length`(=886) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=835) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  93%|█████████▎| 614/660 [8:16:48<38:36, 50.36s/it]Both `max_new_tokens` (=2048) and `max_length`(=886) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=834) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  93%|█████████▎| 615/660 [8:17:37<37:28, 49.97s/it]Both `max_new_tokens` (=2048) and `max_length`(=884) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=832) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  93%|█████████▎| 616/660 [8:18:25<36:23, 49.62s/it]Both `max_new_tokens` (=2048) and `max_length`(=883) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=831) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  93%|█████████▎| 617/660 [8:19:14<35:24, 49.40s/it]Both `max_new_tokens` (=2048) and `max_length`(=883) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=830) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  94%|█████████▎| 618/660 [8:20:03<34:29, 49.26s/it]Both `max_new_tokens` (=2048) and `max_length`(=882) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=826) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  94%|█████████▍| 619/660 [8:20:53<33:46, 49.43s/it]Both `max_new_tokens` (=2048) and `max_length`(=881) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=822) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  94%|█████████▍| 620/660 [8:21:43<33:09, 49.73s/it]Both `max_new_tokens` (=2048) and `max_length`(=879) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=813) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  94%|█████████▍| 621/660 [8:22:33<32:19, 49.74s/it]Both `max_new_tokens` (=2048) and `max_length`(=879) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=812) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  94%|█████████▍| 622/660 [8:23:22<31:21, 49.50s/it]Both `max_new_tokens` (=2048) and `max_length`(=877) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=809) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  94%|█████████▍| 623/660 [8:24:11<30:24, 49.32s/it]Both `max_new_tokens` (=2048) and `max_length`(=875) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=808) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=800) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  95%|█████████▍| 624/660 [8:25:00<29:29, 49.16s/it]Both `max_new_tokens` (=2048) and `max_length`(=873) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=799) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  95%|█████████▍| 625/660 [8:25:49<28:37, 49.06s/it]Both `max_new_tokens` (=2048) and `max_length`(=872) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=796) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=792) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  95%|█████████▍| 626/660 [8:26:38<27:45, 48.99s/it]Both `max_new_tokens` (=2048) and `max_length`(=871) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=680) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  95%|█████████▌| 627/660 [8:27:26<26:54, 48.92s/it]Both `max_new_tokens` (=2048) and `max_length`(=870) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  95%|█████████▌| 628/660 [8:28:15<26:05, 48.91s/it]Both `max_new_tokens` (=2048) and `max_length`(=865) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  95%|█████████▌| 629/660 [8:29:04<25:15, 48.89s/it]Both `max_new_tokens` (=2048) and `max_length`(=863) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  95%|█████████▌| 630/660 [8:29:53<24:26, 48.87s/it]Both `max_new_tokens` (=2048) and `max_length`(=861) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  96%|█████████▌| 631/660 [8:30:42<23:43, 49.09s/it]Both `max_new_tokens` (=2048) and `max_length`(=856) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  96%|█████████▌| 632/660 [8:31:33<23:11, 49.68s/it]Both `max_new_tokens` (=2048) and `max_length`(=855) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  96%|█████████▌| 633/660 [8:32:24<22:27, 49.91s/it]Both `max_new_tokens` (=2048) and `max_length`(=855) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  96%|█████████▌| 634/660 [8:33:13<21:33, 49.75s/it]Both `max_new_tokens` (=2048) and `max_length`(=854) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  96%|█████████▌| 635/660 [8:34:03<20:45, 49.81s/it]Both `max_new_tokens` (=2048) and `max_length`(=853) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  96%|█████████▋| 636/660 [8:34:55<20:11, 50.50s/it]Both `max_new_tokens` (=2048) and `max_length`(=853) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  97%|█████████▋| 637/660 [8:35:46<19:21, 50.48s/it]Both `max_new_tokens` (=2048) and `max_length`(=846) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  97%|█████████▋| 638/660 [8:36:37<18:32, 50.55s/it]Both `max_new_tokens` (=2048) and `max_length`(=844) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  97%|█████████▋| 639/660 [8:37:28<17:47, 50.85s/it]Both `max_new_tokens` (=2048) and `max_length`(=843) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  97%|█████████▋| 640/660 [8:38:18<16:50, 50.54s/it]Both `max_new_tokens` (=2048) and `max_length`(=842) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  97%|█████████▋| 641/660 [8:39:07<15:53, 50.18s/it]Both `max_new_tokens` (=2048) and `max_length`(=837) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  97%|█████████▋| 642/660 [8:39:57<15:01, 50.07s/it]Both `max_new_tokens` (=2048) and `max_length`(=836) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  97%|█████████▋| 643/660 [8:40:47<14:08, 49.91s/it]Both `max_new_tokens` (=2048) and `max_length`(=834) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  98%|█████████▊| 644/660 [8:41:36<13:15, 49.74s/it]Both `max_new_tokens` (=2048) and `max_length`(=825) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  98%|█████████▊| 645/660 [8:42:25<12:25, 49.67s/it]Both `max_new_tokens` (=2048) and `max_length`(=824) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  98%|█████████▊| 646/660 [8:42:27<08:11, 35.10s/it]Both `max_new_tokens` (=2048) and `max_length`(=823) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  98%|█████████▊| 647/660 [8:43:16<08:32, 39.42s/it]Both `max_new_tokens` (=2048) and `max_length`(=820) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  98%|█████████▊| 648/660 [8:44:05<08:28, 42.41s/it]Both `max_new_tokens` (=2048) and `max_length`(=817) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  98%|█████████▊| 649/660 [8:44:55<08:10, 44.57s/it]Both `max_new_tokens` (=2048) and `max_length`(=817) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  98%|█████████▊| 650/660 [8:45:44<07:39, 46.00s/it]Both `max_new_tokens` (=2048) and `max_length`(=816) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  99%|█████████▊| 651/660 [8:46:34<07:03, 47.05s/it]Both `max_new_tokens` (=2048) and `max_length`(=812) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  99%|█████████▉| 652/660 [8:47:23<06:22, 47.79s/it]Both `max_new_tokens` (=2048) and `max_length`(=802) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  99%|█████████▉| 653/660 [8:48:13<05:38, 48.29s/it]Both `max_new_tokens` (=2048) and `max_length`(=800) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  99%|█████████▉| 654/660 [8:49:02<04:52, 48.67s/it]Both `max_new_tokens` (=2048) and `max_length`(=794) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  99%|█████████▉| 655/660 [8:49:52<04:04, 48.92s/it]Both `max_new_tokens` (=2048) and `max_length`(=794) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  99%|█████████▉| 656/660 [8:50:41<03:16, 49.06s/it]Both `max_new_tokens` (=2048) and `max_length`(=792) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests: 100%|█████████▉| 657/660 [8:51:31<02:27, 49.20s/it]Both `max_new_tokens` (=2048) and `max_length`(=781) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests: 100%|█████████▉| 658/660 [8:52:20<01:38, 49.32s/it]Both `max_new_tokens` (=2048) and `max_length`(=760) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests: 100%|█████████▉| 659/660 [8:53:10<00:49, 49.39s/it]Both `max_new_tokens` (=2048) and `max_length`(=699) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests: 100%|██████████| 660/660 [8:54:00<00:00, 49.49s/it]
Running generate_until requests: 100%|██████████| 660/660 [8:54:00<00:00, 48.55s/it]
n136-128-154:1658790:1706819 [1] NCCL INFO Channel 00/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1706819 [1] NCCL INFO Channel 01/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1706819 [1] NCCL INFO Channel 02/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1706819 [1] NCCL INFO Channel 03/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1706819 [1] NCCL INFO Channel 04/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1706819 [1] NCCL INFO Channel 05/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1706819 [1] NCCL INFO Channel 06/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1706819 [1] NCCL INFO Channel 07/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1706819 [1] NCCL INFO Channel 08/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1706819 [1] NCCL INFO Channel 09/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1706819 [1] NCCL INFO Channel 10/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1706819 [1] NCCL INFO Channel 11/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1706819 [1] NCCL INFO Channel 12/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1706819 [1] NCCL INFO Channel 13/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1706819 [1] NCCL INFO Channel 14/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1706819 [1] NCCL INFO Channel 15/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1706819 [1] NCCL INFO Channel 16/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1706819 [1] NCCL INFO Channel 17/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1706819 [1] NCCL INFO Channel 18/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1706819 [1] NCCL INFO Channel 19/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1706819 [1] NCCL INFO Channel 20/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1706819 [1] NCCL INFO Channel 21/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1706819 [1] NCCL INFO Channel 22/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1706819 [1] NCCL INFO Channel 23/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1706819 [1] NCCL INFO Channel 24/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1706819 [1] NCCL INFO Channel 25/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1706819 [1] NCCL INFO Channel 26/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1706819 [1] NCCL INFO Channel 27/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1706819 [1] NCCL INFO Channel 28/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1706819 [1] NCCL INFO Channel 29/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1706819 [1] NCCL INFO Channel 30/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1658790:1706819 [1] NCCL INFO Channel 31/1 : 1[2] -> 0[1] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:1658790:1706824 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1658790:1706824 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1658790:1706824 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1658790:1706824 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1658790:1706824 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1658790:1706824 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1658790:1659172 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-06:03:01:40 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 4, batch_size: auto
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     4|exact_match|↑  |0.0190|±  |0.0038|
|     |       |strict-match    |     4|exact_match|↑  |0.0015|±  |0.0011|

[rank0]:[W1206 03:01:41.050700585 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:1658789:1706896 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1658789:1706896 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1658789:1706896 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1658789:1659174 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:1658789:1706896 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1658789:1706896 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1658789:1706896 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1658790:1659172 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:1658790:1706824 [1] NCCL INFO comm 0x10287fb0 rank 1 nranks 2 cudaDev 1 busId 4a000 - Abort COMPLETE
n136-128-154:1658789:1706896 [0] NCCL INFO comm 0x12b667f0 rank 0 nranks 2 cudaDev 0 busId 16000 - Abort COMPLETE
任务 gsm8k 评估完成！

==================================================
开始评估：任务=boolq | 少样本数=0 | 模型=Qwen2.5-7B-quantization-fg
输出路径：results2/Qwen2.5-7B-quantization-fg/fg/boolq.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-06:03:03:07 INFO     [__main__:440] Selected Tasks: ['boolq']
2025-12-06:03:03:07 INFO     [__main__:440] Selected Tasks: ['boolq']
2025-12-06:03:03:07 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-06:03:03:07 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-06:03:03:07 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-06:03:03:07 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-06:03:03:08 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-06:03:03:08 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
2025-12-06:03:03:08 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.16it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.01it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.17it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.01s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.17it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.68it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.45it/s]
2025-12-06:03:03:11 WARNING  [api.task:844] [Task: boolq] metric acc is defined, but aggregation is not. using default aggregation=mean
2025-12-06:03:03:11 WARNING  [api.task:856] [Task: boolq] metric acc is defined, but higher_is_better is not. using default higher_is_better=True

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.02it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.47it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.26it/s]
2025-12-06:03:03:12 WARNING  [api.task:844] [Task: boolq] metric acc is defined, but aggregation is not. using default aggregation=mean
2025-12-06:03:03:12 WARNING  [api.task:856] [Task: boolq] metric acc is defined, but higher_is_better is not. using default higher_is_better=True
2025-12-06:03:03:25 WARNING  [evaluator:309] Overwriting default num_fewshot of boolq from None to 0
2025-12-06:03:03:25 INFO     [api.task:434] Building contexts for boolq on rank 1...

  0%|          | 0/1635 [00:00<?, ?it/s]
 17%|█▋        | 272/1635 [00:00<00:00, 2715.55it/s]
 34%|███▍      | 554/1635 [00:00<00:00, 2775.00it/s]
 51%|█████     | 834/1635 [00:00<00:00, 2784.06it/s]
 68%|██████▊   | 1113/1635 [00:00<00:00, 2773.76it/s]
 85%|████████▌ | 1393/1635 [00:00<00:00, 2780.42it/s]
100%|██████████| 1635/1635 [00:00<00:00, 2759.95it/s]
2025-12-06:03:03:27 WARNING  [evaluator:309] Overwriting default num_fewshot of boolq from None to 0
2025-12-06:03:03:27 INFO     [api.task:434] Building contexts for boolq on rank 0...

  0%|          | 0/1635 [00:00<?, ?it/s]
 17%|█▋        | 281/1635 [00:00<00:00, 2804.67it/s]
 34%|███▍      | 564/1635 [00:00<00:00, 2816.43it/s]
 52%|█████▏    | 846/1635 [00:00<00:00, 2816.62it/s]
 69%|██████▉   | 1131/1635 [00:00<00:00, 2826.45it/s]
 86%|████████▋ | 1414/1635 [00:00<00:00, 2826.42it/s]
100%|██████████| 1635/1635 [00:00<00:00, 2807.40it/s]
n136-128-154:1706951:1706951 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1706951:1706951 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1706951:1706951 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:1706951:1706951 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:1706951:1706951 [0] NCCL INFO Comm config Blocking set to 1
n136-128-154:1706952:1706952 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:1706952:1706952 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1706952:1706952 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1706952:1706952 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:1706952:1706952 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:1706951:1707153 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:1706952:1707154 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:1706951:1707153 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:1706952:1707154 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:1706952:1707154 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1706952:1707154 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1706952:1707154 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:1706951:1707153 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1706951:1707153 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1706951:1707153 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:1706952:1707154 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:1706952:1707154 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:1706952:1707154 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:1706951:1707153 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:1706951:1707153 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:1706951:1707153 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:1706952:1707154 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:1706952:1707154 [1] NCCL INFO Using network IB
n136-128-154:1706951:1707153 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:1706951:1707153 [0] NCCL INFO Using network IB
n136-128-154:1706952:1707154 [1] NCCL INFO ncclCommInitRankConfig comm 0xe578900 rank 1 nranks 2 cudaDev 1 nvmlDev 2 busId 4a000 commId 0xe98a3522ac27b72b - Init START
n136-128-154:1706951:1707153 [0] NCCL INFO ncclCommInitRankConfig comm 0xeb759d0 rank 0 nranks 2 cudaDev 0 nvmlDev 1 busId 16000 commId 0xe98a3522ac27b72b - Init START
n136-128-154:1706951:1707153 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:1706952:1707154 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:1706952:1707154 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:1706952:1707154 [1] NCCL INFO Retrieving state for IB
n136-128-154:1706952:1707154 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:1706952:1707154 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:1706951:1707153 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:1706951:1707153 [0] NCCL INFO Retrieving state for IB
n136-128-154:1706951:1707153 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:1706952:1707154 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:1706951:1707153 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:1706952:1707154 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:1706951:1707153 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:1706952:1707154 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:1706951:1707153 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:1706951:1707153 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:1706951:1707153 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1706951:1707153 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1706951:1707153 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1706951:1707153 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1706952:1707154 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1706952:1707154 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1706952:1707154 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1706952:1707154 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1706951:1707153 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:1706951:1707153 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:1706951:1707153 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:1706951:1707153 [0] NCCL INFO               + PCI[24.0] - PCI/0-14000 (1000c01010de13b8)
n136-128-154:1706951:1707153 [0] NCCL INFO                             + PCI[24.0] - GPU/0-16000 (0)
n136-128-154:1706951:1707153 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1706951:1707153 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:1706951:1707153 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:1706951:1707153 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:1706951:1707153 [0] NCCL INFO               + PCI[24.0] - PCI/0-48000 (1000c01010de13b8)
n136-128-154:1706951:1707153 [0] NCCL INFO                             + PCI[24.0] - GPU/0-4a000 (1)
n136-128-154:1706951:1707153 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1706951:1707153 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:1706951:1707153 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:1706951:1707153 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:1706951:1707153 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:1706951:1707153 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:1706951:1707153 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:1706951:1707153 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:1706951:1707153 [0] NCCL INFO ==========================================
n136-128-154:1706951:1707153 [0] NCCL INFO GPU/0-16000 :GPU/0-16000 (0/5000.0/LOC) GPU/0-4a000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1706951:1707153 [0] NCCL INFO GPU/0-4a000 :GPU/0-16000 (2/240.0/NVL) GPU/0-4a000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1706951:1707153 [0] NCCL INFO Setting affinity for GPU 1 to 1-31,65-95
n136-128-154:1706951:1707153 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:1706951:1707153 [0] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1706951:1707153 [0] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1706951:1707153 [0] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1706951:1707153 [0] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1706951:1707153 [0] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1706951:1707153 [0] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1706951:1707153 [0] NCCL INFO  6 : GPU/0-16000 GPU/0-4a000
n136-128-154:1706951:1707153 [0] NCCL INFO  7 : GPU/0-16000 GPU/0-4a000
n136-128-154:1706951:1707153 [0] NCCL INFO  8 : GPU/0-16000 GPU/0-4a000
n136-128-154:1706951:1707153 [0] NCCL INFO  9 : GPU/0-16000 GPU/0-4a000
n136-128-154:1706951:1707153 [0] NCCL INFO 10 : GPU/0-16000 GPU/0-4a000
n136-128-154:1706951:1707153 [0] NCCL INFO 11 : GPU/0-16000 GPU/0-4a000
n136-128-154:1706951:1707153 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:1706951:1707153 [0] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1706951:1707153 [0] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1706951:1707153 [0] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1706951:1707153 [0] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1706951:1707153 [0] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1706951:1707153 [0] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1706951:1707153 [0] NCCL INFO  6 : GPU/0-4a000 GPU/0-16000
n136-128-154:1706951:1707153 [0] NCCL INFO  7 : GPU/0-4a000 GPU/0-16000
n136-128-154:1706951:1707153 [0] NCCL INFO  8 : GPU/0-4a000 GPU/0-16000
n136-128-154:1706952:1707154 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:1706951:1707153 [0] NCCL INFO  9 : GPU/0-4a000 GPU/0-16000
n136-128-154:1706952:1707154 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:1706951:1707153 [0] NCCL INFO 10 : GPU/0-4a000 GPU/0-16000
n136-128-154:1706952:1707154 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:1706951:1707153 [0] NCCL INFO 11 : GPU/0-4a000 GPU/0-16000
n136-128-154:1706952:1707154 [1] NCCL INFO               + PCI[24.0] - PCI/0-14000 (1000c01010de13b8)
n136-128-154:1706952:1707154 [1] NCCL INFO                             + PCI[24.0] - GPU/0-16000 (0)
n136-128-154:1706952:1707154 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1706952:1707154 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:1706952:1707154 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:1706952:1707154 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:1706952:1707154 [1] NCCL INFO               + PCI[24.0] - PCI/0-48000 (1000c01010de13b8)
n136-128-154:1706952:1707154 [1] NCCL INFO                             + PCI[24.0] - GPU/0-4a000 (1)
n136-128-154:1706952:1707154 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1706952:1707154 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:1706952:1707154 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:1706952:1707154 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:1706952:1707154 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:1706952:1707154 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:1706952:1707154 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:1706952:1707154 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:1706952:1707154 [1] NCCL INFO ==========================================
n136-128-154:1706952:1707154 [1] NCCL INFO GPU/0-16000 :GPU/0-16000 (0/5000.0/LOC) GPU/0-4a000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1706952:1707154 [1] NCCL INFO GPU/0-4a000 :GPU/0-16000 (2/240.0/NVL) GPU/0-4a000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1706952:1707154 [1] NCCL INFO Setting affinity for GPU 2 to 1-31,65-95
n136-128-154:1706952:1707154 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:1706952:1707154 [1] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1706952:1707154 [1] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1706952:1707154 [1] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1706952:1707154 [1] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1706952:1707154 [1] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1706952:1707154 [1] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1706952:1707154 [1] NCCL INFO  6 : GPU/0-16000 GPU/0-4a000
n136-128-154:1706952:1707154 [1] NCCL INFO  7 : GPU/0-16000 GPU/0-4a000
n136-128-154:1706952:1707154 [1] NCCL INFO  8 : GPU/0-16000 GPU/0-4a000
n136-128-154:1706952:1707154 [1] NCCL INFO  9 : GPU/0-16000 GPU/0-4a000
n136-128-154:1706952:1707154 [1] NCCL INFO 10 : GPU/0-16000 GPU/0-4a000
n136-128-154:1706952:1707154 [1] NCCL INFO 11 : GPU/0-16000 GPU/0-4a000
n136-128-154:1706952:1707154 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:1706952:1707154 [1] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1706952:1707154 [1] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1706952:1707154 [1] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1706952:1707154 [1] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1706952:1707154 [1] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1706952:1707154 [1] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1706952:1707154 [1] NCCL INFO  6 : GPU/0-4a000 GPU/0-16000
n136-128-154:1706952:1707154 [1] NCCL INFO  7 : GPU/0-4a000 GPU/0-16000
n136-128-154:1706952:1707154 [1] NCCL INFO  8 : GPU/0-4a000 GPU/0-16000
n136-128-154:1706952:1707154 [1] NCCL INFO  9 : GPU/0-4a000 GPU/0-16000
n136-128-154:1706952:1707154 [1] NCCL INFO 10 : GPU/0-4a000 GPU/0-16000
n136-128-154:1706952:1707154 [1] NCCL INFO 11 : GPU/0-4a000 GPU/0-16000
n136-128-154:1706951:1707153 [0] NCCL INFO comm 0xeb759d0 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:1706952:1707154 [1] NCCL INFO comm 0xe578900 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:1706951:1707153 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:1706952:1707154 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:1706951:1707153 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:1706952:1707154 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:1706951:1707153 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:1706952:1707154 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:1706951:1707153 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:1706952:1707154 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:1706951:1707153 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:1706952:1707154 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:1706951:1707153 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:1706952:1707154 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:1706951:1707153 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:1706952:1707154 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:1706951:1707153 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:1706952:1707154 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:1706951:1707153 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:1706952:1707154 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:1706951:1707153 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:1706952:1707154 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:1706951:1707153 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:1706952:1707154 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:1706951:1707153 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:1706952:1707154 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:1706951:1707153 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:1706952:1707154 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:1706951:1707153 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:1706952:1707154 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:1706951:1707153 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:1706952:1707154 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:1706952:1707154 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:1706951:1707153 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:1706952:1707154 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:1706951:1707153 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:1706952:1707154 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:1706951:1707153 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:1706952:1707154 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:1706951:1707153 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:1706952:1707154 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:1706951:1707153 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:1706952:1707154 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:1706951:1707153 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:1706952:1707154 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:1706951:1707153 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:1706952:1707154 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:1706951:1707153 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:1706952:1707154 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:1706951:1707153 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:1706951:1707153 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:1706951:1707153 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:1706951:1707153 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:1706952:1707154 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:1706951:1707153 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:1706952:1707154 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:1706951:1707153 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:1706952:1707154 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:1706951:1707153 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:1706952:1707154 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:1706951:1707153 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:1706952:1707154 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:1706951:1707153 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:1706952:1707154 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:1706951:1707153 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:1706952:1707154 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:1706951:1707153 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:1706952:1707154 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:1706951:1707153 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:1706952:1707154 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:1706951:1707153 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:1706952:1707154 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:1706951:1707153 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:1706952:1707154 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:1706951:1707153 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:1706952:1707154 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:1706951:1707153 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:1706952:1707154 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:1706951:1707153 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:1706952:1707154 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:1706951:1707153 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:1706952:1707154 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:1706951:1707153 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:1706952:1707154 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:1706951:1707153 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:1706952:1707154 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:1706951:1707153 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:1706952:1707154 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:1706951:1707153 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:1706952:1707154 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:1706951:1707153 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:1706952:1707154 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:1706951:1707153 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:1706952:1707154 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:1706951:1707153 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:1706952:1707154 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:1706951:1707153 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:1706952:1707154 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:1706951:1707153 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:1706952:1707154 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:1706951:1707153 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:1706952:1707154 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:1706951:1707153 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:1706952:1707154 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:1706951:1707153 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:1706951:1707153 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:1706951:1707153 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:1706951:1707153 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:1706951:1707153 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:1706951:1707153 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:1706951:1707153 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:1706951:1707153 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:1706951:1707153 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:1706951:1707153 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:1706951:1707153 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:1706951:1707153 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:1706951:1707153 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:1706951:1707153 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:1706951:1707153 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:1706951:1707153 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:1706951:1707153 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:1706951:1707153 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:1706951:1707153 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:1706951:1707153 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:1706951:1707153 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:1706951:1707153 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:1706951:1707153 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:1706951:1707153 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:1706951:1707166 [0] NCCL INFO [Proxy Service] Device 0 CPU core 68
n136-128-154:1706951:1707167 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 6
n136-128-154:1706952:1707154 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:1706952:1707168 [1] NCCL INFO [Proxy Service] Device 1 CPU core 72
n136-128-154:1706952:1707169 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 73
n136-128-154:1706951:1707153 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:1706951:1707153 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:1706952:1707154 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:1706952:1707154 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:1706951:1707153 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:1706951:1707153 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:1706951:1707153 [0] NCCL INFO ncclCommInitRankConfig comm 0xeb759d0 rank 0 nranks 2 cudaDev 0 nvmlDev 1 busId 16000 commId 0xe98a3522ac27b72b - Init COMPLETE
n136-128-154:1706951:1707153 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 1.02 (kernels 0.22, alloc 0.54, bootstrap 0.00, allgathers 0.01, topo 0.08, graphs 0.00, connections 0.04, rest 0.14)
n136-128-154:1706952:1707154 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:1706952:1707154 [1] NCCL INFO ncclCommInitRankConfig comm 0xe578900 rank 1 nranks 2 cudaDev 1 nvmlDev 2 busId 4a000 commId 0xe98a3522ac27b72b - Init COMPLETE
n136-128-154:1706952:1707154 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 1.01 (kernels 0.20, alloc 0.54, bootstrap 0.00, allgathers 0.01, topo 0.08, graphs 0.00, connections 0.03, rest 0.15)
n136-128-154:1706951:1707170 [0] NCCL INFO Channel 00/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1706951:1707170 [0] NCCL INFO Channel 01/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1706951:1707170 [0] NCCL INFO Channel 02/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1706951:1707170 [0] NCCL INFO Channel 03/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1706951:1707170 [0] NCCL INFO Channel 04/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1706951:1707170 [0] NCCL INFO Channel 05/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1706951:1707170 [0] NCCL INFO Channel 06/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1706951:1707170 [0] NCCL INFO Channel 07/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1706951:1707170 [0] NCCL INFO Channel 08/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1706951:1707170 [0] NCCL INFO Channel 09/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1706952:1707171 [1] NCCL INFO Channel 00/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706951:1707170 [0] NCCL INFO Channel 10/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1706952:1707171 [1] NCCL INFO Channel 01/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706951:1707170 [0] NCCL INFO Channel 11/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1706952:1707171 [1] NCCL INFO Channel 02/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706951:1707170 [0] NCCL INFO Channel 12/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1706952:1707171 [1] NCCL INFO Channel 03/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706951:1707170 [0] NCCL INFO Channel 13/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1706952:1707171 [1] NCCL INFO Channel 04/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706951:1707170 [0] NCCL INFO Channel 14/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1706952:1707171 [1] NCCL INFO Channel 05/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706951:1707170 [0] NCCL INFO Channel 15/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1706952:1707171 [1] NCCL INFO Channel 06/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706951:1707170 [0] NCCL INFO Channel 16/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1706952:1707171 [1] NCCL INFO Channel 07/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706951:1707170 [0] NCCL INFO Channel 17/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1706952:1707171 [1] NCCL INFO Channel 08/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706951:1707170 [0] NCCL INFO Channel 18/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1706952:1707171 [1] NCCL INFO Channel 09/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707171 [1] NCCL INFO Channel 10/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706951:1707170 [0] NCCL INFO Channel 19/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1706952:1707171 [1] NCCL INFO Channel 11/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706951:1707170 [0] NCCL INFO Channel 20/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1706952:1707171 [1] NCCL INFO Channel 12/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706951:1707170 [0] NCCL INFO Channel 21/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1706952:1707171 [1] NCCL INFO Channel 13/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706951:1707170 [0] NCCL INFO Channel 22/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1706952:1707171 [1] NCCL INFO Channel 14/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706951:1707170 [0] NCCL INFO Channel 23/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1706952:1707171 [1] NCCL INFO Channel 15/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707171 [1] NCCL INFO Channel 16/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707171 [1] NCCL INFO Channel 17/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707171 [1] NCCL INFO Channel 18/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707171 [1] NCCL INFO Channel 19/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707171 [1] NCCL INFO Channel 20/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707171 [1] NCCL INFO Channel 21/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707171 [1] NCCL INFO Channel 22/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707171 [1] NCCL INFO Channel 23/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706951:1707170 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:1706952:1707171 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-06:03:03:29 INFO     [evaluator:559] Running loglikelihood requests
2025-12-06:03:03:29 INFO     [evaluator:559] Running loglikelihood requests
Passed argument batch_size = auto:1. Detecting largest batch size

Running loglikelihood requests:   0%|          | 0/3270 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 45
Determined largest batch size: 45

Running loglikelihood requests:   0%|          | 1/3270 [00:55<49:58:45, 55.04s/it]
Running loglikelihood requests:   3%|▎         | 91/3270 [00:56<23:11,  2.28it/s]  
Running loglikelihood requests:   6%|▌         | 181/3270 [00:57<09:40,  5.32it/s]
Running loglikelihood requests:   8%|▊         | 271/3270 [00:58<05:21,  9.33it/s]
Running loglikelihood requests:  11%|█         | 361/3270 [00:59<03:20, 14.51it/s]
Running loglikelihood requests:  14%|█▍        | 451/3270 [01:00<02:14, 21.03it/s]
Running loglikelihood requests:  17%|█▋        | 541/3270 [01:00<01:34, 28.96it/s]
Running loglikelihood requests:  19%|█▉        | 631/3270 [01:01<01:08, 38.25it/s]
Running loglikelihood requests:  22%|██▏       | 721/3270 [01:02<00:52, 48.69it/s]
Running loglikelihood requests:  25%|██▍       | 811/3270 [01:03<00:41, 59.80it/s]
Running loglikelihood requests:  28%|██▊       | 901/3270 [01:03<00:33, 71.15it/s]
Running loglikelihood requests:  30%|███       | 991/3270 [01:04<00:27, 82.01it/s]
Running loglikelihood requests:  33%|███▎      | 1081/3270 [01:05<00:23, 92.09it/s]
Running loglikelihood requests:  36%|███▌      | 1171/3270 [01:06<00:20, 101.08it/s]
Running loglikelihood requests:  39%|███▊      | 1261/3270 [01:06<00:18, 109.54it/s]
Running loglikelihood requests:  41%|████▏     | 1351/3270 [01:07<00:16, 117.09it/s]
Running loglikelihood requests:  44%|████▍     | 1441/3270 [01:07<00:14, 123.57it/s]
Running loglikelihood requests:  47%|████▋     | 1531/3270 [01:08<00:13, 129.00it/s]
Running loglikelihood requests:  50%|████▉     | 1621/3270 [01:09<00:12, 133.55it/s]
Running loglikelihood requests:  52%|█████▏    | 1711/3270 [01:09<00:11, 138.50it/s]
Running loglikelihood requests:  55%|█████▌    | 1801/3270 [01:10<00:10, 143.19it/s]
Running loglikelihood requests:  58%|█████▊    | 1891/3270 [01:10<00:09, 147.32it/s]
Running loglikelihood requests:  61%|██████    | 1981/3270 [01:11<00:08, 151.12it/s]
Running loglikelihood requests:  63%|██████▎   | 2071/3270 [01:12<00:07, 154.48it/s]
Running loglikelihood requests:  66%|██████▌   | 2161/3270 [01:12<00:07, 157.53it/s]
Running loglikelihood requests:  69%|██████▉   | 2251/3270 [01:13<00:06, 162.03it/s]
Running loglikelihood requests:  72%|███████▏  | 2341/3270 [01:13<00:05, 166.08it/s]
Running loglikelihood requests:  74%|███████▍  | 2431/3270 [01:14<00:04, 170.40it/s]
Running loglikelihood requests:  77%|███████▋  | 2521/3270 [01:14<00:04, 174.31it/s]
Running loglikelihood requests:  80%|███████▉  | 2611/3270 [01:15<00:03, 177.94it/s]
Running loglikelihood requests:  83%|████████▎ | 2701/3270 [01:15<00:03, 182.15it/s]
Running loglikelihood requests:  85%|████████▌ | 2791/3270 [01:15<00:02, 188.11it/s]
Running loglikelihood requests:  88%|████████▊ | 2881/3270 [01:16<00:01, 195.06it/s]
Running loglikelihood requests:  91%|█████████ | 2971/3270 [01:16<00:01, 202.61it/s]
Running loglikelihood requests:  94%|█████████▎| 3061/3270 [01:17<00:00, 212.01it/s]
Running loglikelihood requests:  96%|█████████▋| 3151/3270 [01:17<00:00, 222.48it/s]
Running loglikelihood requests:  99%|█████████▉| 3241/3270 [01:17<00:00, 280.58it/s]
Running loglikelihood requests: 100%|██████████| 3270/3270 [01:17<00:00, 42.09it/s] 
n136-128-154:1706952:1707253 [1] NCCL INFO Channel 00/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707253 [1] NCCL INFO Channel 01/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707253 [1] NCCL INFO Channel 02/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707253 [1] NCCL INFO Channel 03/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707253 [1] NCCL INFO Channel 04/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707253 [1] NCCL INFO Channel 05/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707253 [1] NCCL INFO Channel 06/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707253 [1] NCCL INFO Channel 07/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707253 [1] NCCL INFO Channel 08/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707253 [1] NCCL INFO Channel 09/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707253 [1] NCCL INFO Channel 10/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707253 [1] NCCL INFO Channel 11/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707253 [1] NCCL INFO Channel 12/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707253 [1] NCCL INFO Channel 13/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707253 [1] NCCL INFO Channel 14/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707253 [1] NCCL INFO Channel 15/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707253 [1] NCCL INFO Channel 16/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707253 [1] NCCL INFO Channel 17/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707253 [1] NCCL INFO Channel 18/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707253 [1] NCCL INFO Channel 19/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707253 [1] NCCL INFO Channel 20/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707253 [1] NCCL INFO Channel 21/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707253 [1] NCCL INFO Channel 22/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707253 [1] NCCL INFO Channel 23/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707253 [1] NCCL INFO Channel 24/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707253 [1] NCCL INFO Channel 25/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707253 [1] NCCL INFO Channel 26/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707253 [1] NCCL INFO Channel 27/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707253 [1] NCCL INFO Channel 28/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707253 [1] NCCL INFO Channel 29/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707253 [1] NCCL INFO Channel 30/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1706952:1707253 [1] NCCL INFO Channel 31/1 : 1[2] -> 0[1] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:1706952:1707257 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1706952:1707257 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1706952:1707257 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1706952:1707257 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1706952:1707257 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1706952:1707257 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1706952:1707168 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-06:03:05:00 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: auto (45)
|Tasks|Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|-----|------:|------|-----:|------|---|-----:|---|-----:|
|boolq|      2|none  |     0|acc   |↑  |0.5046|±  |0.0087|

[rank0]:[W1206 03:05:00.143295240 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:1706951:1707309 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1706951:1707309 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1706951:1707309 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1706951:1707166 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:1706951:1707309 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1706951:1707309 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1706951:1707309 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1706952:1707168 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:1706952:1707257 [1] NCCL INFO comm 0xe578900 rank 1 nranks 2 cudaDev 1 busId 4a000 - Abort COMPLETE
n136-128-154:1706951:1707309 [0] NCCL INFO comm 0xeb759d0 rank 0 nranks 2 cudaDev 0 busId 16000 - Abort COMPLETE
任务 boolq 评估完成！

==================================================
开始评估：任务=arc_challenge | 少样本数=25 | 模型=Qwen2.5-7B-quantization-fg
输出路径：results2/Qwen2.5-7B-quantization-fg/fg/arc_challenge.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-06:03:06:16 INFO     [__main__:440] Selected Tasks: ['arc_challenge']
2025-12-06:03:06:16 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-06:03:06:16 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-06:03:06:17 INFO     [__main__:440] Selected Tasks: ['arc_challenge']
2025-12-06:03:06:17 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-06:03:06:17 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-06:03:06:17 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-06:03:06:18 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-06:03:06:18 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.07it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.04s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.03it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.05s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.07it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.54it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.32it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:00,  1.00it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.44it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.23it/s]
2025-12-06:03:06:40 WARNING  [evaluator:309] Overwriting default num_fewshot of arc_challenge from None to 25
2025-12-06:03:06:40 INFO     [api.task:434] Building contexts for arc_challenge on rank 0...

  0%|          | 0/586 [00:00<?, ?it/s]
  1%|          | 4/586 [00:00<00:16, 35.40it/s]
  1%|▏         | 8/586 [00:00<00:16, 35.94it/s]2025-12-06:03:06:40 WARNING  [evaluator:309] Overwriting default num_fewshot of arc_challenge from None to 25
2025-12-06:03:06:40 INFO     [api.task:434] Building contexts for arc_challenge on rank 1...

  0%|          | 0/586 [00:00<?, ?it/s]
  2%|▏         | 12/586 [00:00<00:15, 36.39it/s]
  3%|▎         | 16/586 [00:00<00:15, 36.63it/s]
  1%|          | 4/586 [00:00<00:16, 35.81it/s]
  3%|▎         | 20/586 [00:00<00:15, 36.86it/s]
  1%|▏         | 8/586 [00:00<00:15, 36.44it/s]
  4%|▍         | 24/586 [00:00<00:15, 36.91it/s]
  2%|▏         | 12/586 [00:00<00:15, 36.63it/s]
  3%|▎         | 16/586 [00:00<00:15, 36.79it/s]
  5%|▍         | 28/586 [00:00<00:15, 36.32it/s]
  3%|▎         | 20/586 [00:00<00:15, 36.74it/s]
  5%|▌         | 32/586 [00:00<00:15, 36.31it/s]
  4%|▍         | 24/586 [00:00<00:15, 36.69it/s]
  6%|▌         | 36/586 [00:00<00:15, 36.47it/s]
  5%|▍         | 28/586 [00:00<00:15, 36.68it/s]
  7%|▋         | 40/586 [00:01<00:15, 36.34it/s]
  5%|▌         | 32/586 [00:00<00:15, 36.76it/s]
  8%|▊         | 44/586 [00:01<00:14, 36.52it/s]
  6%|▌         | 36/586 [00:00<00:14, 36.80it/s]
  8%|▊         | 48/586 [00:01<00:14, 36.68it/s]
  7%|▋         | 40/586 [00:01<00:14, 36.85it/s]
  9%|▉         | 52/586 [00:01<00:14, 36.85it/s]
  8%|▊         | 44/586 [00:01<00:14, 36.88it/s]
 10%|▉         | 56/586 [00:01<00:14, 36.08it/s]
  8%|▊         | 48/586 [00:01<00:14, 36.61it/s]
 10%|█         | 60/586 [00:01<00:14, 36.35it/s]
  9%|▉         | 52/586 [00:01<00:14, 36.67it/s]
 11%|█         | 64/586 [00:01<00:14, 36.54it/s]
 10%|▉         | 56/586 [00:01<00:14, 36.56it/s]
 12%|█▏        | 68/586 [00:01<00:14, 36.66it/s]
 10%|█         | 60/586 [00:01<00:14, 36.51it/s]
 12%|█▏        | 72/586 [00:01<00:14, 36.71it/s]
 13%|█▎        | 76/586 [00:02<00:13, 36.82it/s]
 11%|█         | 64/586 [00:01<00:14, 35.84it/s]
 14%|█▎        | 80/586 [00:02<00:13, 36.84it/s]
 12%|█▏        | 68/586 [00:01<00:14, 36.11it/s]
 14%|█▍        | 84/586 [00:02<00:13, 36.91it/s]
 12%|█▏        | 72/586 [00:01<00:14, 35.93it/s]
 15%|█▌        | 88/586 [00:02<00:13, 37.01it/s]
 13%|█▎        | 76/586 [00:02<00:14, 36.18it/s]
 16%|█▌        | 92/586 [00:02<00:13, 37.07it/s]
 14%|█▎        | 80/586 [00:02<00:13, 36.36it/s]
 16%|█▋        | 96/586 [00:02<00:13, 37.13it/s]
 14%|█▍        | 84/586 [00:02<00:13, 36.48it/s]
 17%|█▋        | 100/586 [00:02<00:13, 37.18it/s]
 15%|█▌        | 88/586 [00:02<00:13, 36.61it/s]
 18%|█▊        | 104/586 [00:02<00:13, 36.69it/s]
 16%|█▌        | 92/586 [00:02<00:13, 36.68it/s]
 18%|█▊        | 108/586 [00:02<00:12, 36.80it/s]
 16%|█▋        | 96/586 [00:02<00:13, 36.74it/s]
 19%|█▉        | 112/586 [00:03<00:12, 36.95it/s]
 17%|█▋        | 100/586 [00:02<00:13, 36.80it/s]
 20%|█▉        | 116/586 [00:03<00:12, 37.02it/s]
 18%|█▊        | 104/586 [00:02<00:13, 36.82it/s]
 20%|██        | 120/586 [00:03<00:12, 36.43it/s]
 18%|█▊        | 108/586 [00:02<00:13, 35.70it/s]
 21%|██        | 124/586 [00:03<00:12, 36.44it/s]
 19%|█▉        | 112/586 [00:03<00:13, 36.02it/s]
 22%|██▏       | 128/586 [00:03<00:12, 36.59it/s]
 20%|█▉        | 116/586 [00:03<00:12, 36.36it/s]
 23%|██▎       | 132/586 [00:03<00:12, 36.76it/s]
 20%|██        | 120/586 [00:03<00:12, 36.13it/s]
 23%|██▎       | 136/586 [00:03<00:12, 36.49it/s]
 21%|██        | 124/586 [00:03<00:12, 36.31it/s]
 24%|██▍       | 140/586 [00:03<00:12, 36.66it/s]
 22%|██▏       | 128/586 [00:03<00:12, 36.47it/s]
 25%|██▍       | 144/586 [00:03<00:12, 36.79it/s]
 23%|██▎       | 132/586 [00:03<00:12, 36.66it/s]
 25%|██▌       | 148/586 [00:04<00:11, 36.88it/s]
 23%|██▎       | 136/586 [00:03<00:12, 36.81it/s]
 26%|██▌       | 152/586 [00:04<00:11, 36.91it/s]
 24%|██▍       | 140/586 [00:03<00:12, 36.87it/s]
 27%|██▋       | 156/586 [00:04<00:11, 37.00it/s]
 25%|██▍       | 144/586 [00:03<00:11, 36.93it/s]
 27%|██▋       | 160/586 [00:04<00:11, 37.05it/s]
 25%|██▌       | 148/586 [00:04<00:11, 36.95it/s]
 28%|██▊       | 164/586 [00:04<00:11, 37.11it/s]
 26%|██▌       | 152/586 [00:04<00:11, 36.62it/s]
 29%|██▊       | 168/586 [00:04<00:11, 37.13it/s]
 27%|██▋       | 156/586 [00:04<00:11, 36.74it/s]
 29%|██▉       | 172/586 [00:04<00:11, 37.21it/s]
 27%|██▋       | 160/586 [00:04<00:11, 36.85it/s]
 30%|███       | 176/586 [00:04<00:11, 37.19it/s]
 28%|██▊       | 164/586 [00:04<00:11, 36.92it/s]
 31%|███       | 180/586 [00:04<00:10, 37.13it/s]
 29%|██▊       | 168/586 [00:04<00:11, 36.97it/s]
 31%|███▏      | 184/586 [00:05<00:10, 37.18it/s]
 29%|██▉       | 172/586 [00:04<00:11, 36.89it/s]
 32%|███▏      | 188/586 [00:05<00:10, 37.22it/s]
 30%|███       | 176/586 [00:04<00:11, 36.96it/s]
 33%|███▎      | 192/586 [00:05<00:10, 37.20it/s]
 31%|███       | 180/586 [00:04<00:11, 36.80it/s]
 33%|███▎      | 196/586 [00:05<00:10, 36.80it/s]
 31%|███▏      | 184/586 [00:05<00:10, 36.68it/s]
 34%|███▍      | 200/586 [00:05<00:10, 36.89it/s]
 32%|███▏      | 188/586 [00:05<00:10, 36.52it/s]
 35%|███▍      | 204/586 [00:05<00:10, 36.99it/s]
 33%|███▎      | 192/586 [00:05<00:10, 36.67it/s]
 35%|███▌      | 208/586 [00:05<00:10, 37.10it/s]
 33%|███▎      | 196/586 [00:05<00:10, 36.79it/s]
 36%|███▌      | 212/586 [00:05<00:10, 37.19it/s]
 34%|███▍      | 200/586 [00:05<00:10, 36.86it/s]
 37%|███▋      | 216/586 [00:05<00:09, 37.25it/s]
 35%|███▍      | 204/586 [00:05<00:10, 36.92it/s]
 38%|███▊      | 220/586 [00:05<00:09, 37.25it/s]
 35%|███▌      | 208/586 [00:05<00:10, 36.95it/s]
 38%|███▊      | 224/586 [00:06<00:09, 37.26it/s]
 36%|███▌      | 212/586 [00:05<00:10, 36.88it/s]
 39%|███▉      | 228/586 [00:06<00:09, 36.72it/s]
 37%|███▋      | 216/586 [00:05<00:10, 36.92it/s]
 40%|███▉      | 232/586 [00:06<00:09, 36.85it/s]
 38%|███▊      | 220/586 [00:06<00:09, 36.61it/s]
 40%|████      | 236/586 [00:06<00:09, 36.94it/s]
 38%|███▊      | 224/586 [00:06<00:09, 36.82it/s]
 41%|████      | 240/586 [00:06<00:09, 36.97it/s]
 39%|███▉      | 228/586 [00:06<00:09, 36.98it/s]
 42%|████▏     | 244/586 [00:06<00:09, 37.00it/s]
 40%|███▉      | 232/586 [00:06<00:09, 37.08it/s]
 42%|████▏     | 248/586 [00:06<00:09, 37.04it/s]
 40%|████      | 236/586 [00:06<00:09, 37.05it/s]
 43%|████▎     | 252/586 [00:06<00:09, 37.04it/s]
 41%|████      | 240/586 [00:06<00:09, 36.66it/s]
 44%|████▎     | 256/586 [00:06<00:08, 37.04it/s]
 42%|████▏     | 244/586 [00:06<00:09, 36.69it/s]
 44%|████▍     | 260/586 [00:07<00:08, 37.04it/s]
 42%|████▏     | 248/586 [00:06<00:09, 36.74it/s]
 45%|████▌     | 264/586 [00:07<00:08, 37.04it/s]
 43%|████▎     | 252/586 [00:06<00:09, 36.84it/s]
 46%|████▌     | 268/586 [00:07<00:08, 37.10it/s]
 44%|████▎     | 256/586 [00:06<00:08, 36.83it/s]
 46%|████▋     | 272/586 [00:07<00:08, 36.76it/s]
 44%|████▍     | 260/586 [00:07<00:08, 36.71it/s]
 47%|████▋     | 276/586 [00:07<00:08, 36.89it/s]
 45%|████▌     | 264/586 [00:07<00:08, 36.76it/s]
 48%|████▊     | 280/586 [00:07<00:08, 37.02it/s]
 46%|████▌     | 268/586 [00:07<00:08, 36.79it/s]
 48%|████▊     | 284/586 [00:07<00:08, 37.12it/s]
 46%|████▋     | 272/586 [00:07<00:08, 36.87it/s]
 49%|████▉     | 288/586 [00:07<00:08, 36.75it/s]
 47%|████▋     | 276/586 [00:07<00:08, 36.90it/s]
 50%|████▉     | 292/586 [00:07<00:07, 36.88it/s]
 48%|████▊     | 280/586 [00:07<00:08, 36.93it/s]
 51%|█████     | 296/586 [00:08<00:07, 36.98it/s]
 48%|████▊     | 284/586 [00:07<00:08, 36.95it/s]
 51%|█████     | 300/586 [00:08<00:07, 37.08it/s]
 49%|████▉     | 288/586 [00:07<00:08, 37.01it/s]
 52%|█████▏    | 304/586 [00:08<00:07, 37.13it/s]
 50%|████▉     | 292/586 [00:07<00:07, 37.01it/s]
 53%|█████▎    | 308/586 [00:08<00:07, 37.14it/s]
 51%|█████     | 296/586 [00:08<00:07, 37.04it/s]
 53%|█████▎    | 312/586 [00:08<00:07, 37.19it/s]
 51%|█████     | 300/586 [00:08<00:07, 37.06it/s]
 54%|█████▍    | 316/586 [00:08<00:07, 37.25it/s]
 52%|█████▏    | 304/586 [00:08<00:07, 36.66it/s]
 55%|█████▍    | 320/586 [00:08<00:07, 37.28it/s]
 53%|█████▎    | 308/586 [00:08<00:07, 36.86it/s]
 55%|█████▌    | 324/586 [00:08<00:07, 37.25it/s]
 53%|█████▎    | 312/586 [00:08<00:07, 36.98it/s]
 56%|█████▌    | 328/586 [00:08<00:06, 37.29it/s]
 54%|█████▍    | 316/586 [00:08<00:07, 37.05it/s]
 57%|█████▋    | 332/586 [00:08<00:06, 37.28it/s]
 55%|█████▍    | 320/586 [00:08<00:07, 37.12it/s]
 57%|█████▋    | 336/586 [00:09<00:06, 37.31it/s]
 55%|█████▌    | 324/586 [00:08<00:07, 37.16it/s]
 58%|█████▊    | 340/586 [00:09<00:06, 37.36it/s]
 56%|█████▌    | 328/586 [00:08<00:06, 37.17it/s]
 59%|█████▊    | 344/586 [00:09<00:06, 37.38it/s]
 57%|█████▋    | 332/586 [00:09<00:06, 37.13it/s]
 59%|█████▉    | 348/586 [00:09<00:06, 37.32it/s]
 57%|█████▋    | 336/586 [00:09<00:06, 37.17it/s]
 60%|██████    | 352/586 [00:09<00:06, 37.36it/s]
 58%|█████▊    | 340/586 [00:09<00:06, 37.17it/s]
 61%|██████    | 356/586 [00:09<00:06, 37.36it/s]
 59%|█████▊    | 344/586 [00:09<00:06, 37.20it/s]
 61%|██████▏   | 360/586 [00:09<00:06, 37.33it/s]
 59%|█████▉    | 348/586 [00:09<00:06, 37.21it/s]
 62%|██████▏   | 364/586 [00:09<00:05, 37.32it/s]
 60%|██████    | 352/586 [00:09<00:06, 37.18it/s]
 63%|██████▎   | 368/586 [00:09<00:05, 37.34it/s]
 61%|██████    | 356/586 [00:09<00:06, 37.13it/s]
 63%|██████▎   | 372/586 [00:10<00:05, 37.33it/s]
 61%|██████▏   | 360/586 [00:09<00:06, 37.13it/s]
 64%|██████▍   | 376/586 [00:10<00:05, 37.32it/s]
 62%|██████▏   | 364/586 [00:09<00:05, 37.06it/s]
 65%|██████▍   | 380/586 [00:10<00:05, 37.33it/s]
 63%|██████▎   | 368/586 [00:10<00:05, 37.07it/s]
 66%|██████▌   | 384/586 [00:10<00:05, 37.30it/s]
 63%|██████▎   | 372/586 [00:10<00:05, 36.55it/s]
 66%|██████▌   | 388/586 [00:10<00:05, 37.24it/s]
 64%|██████▍   | 376/586 [00:10<00:05, 36.61it/s]
 67%|██████▋   | 392/586 [00:10<00:05, 37.22it/s]
 65%|██████▍   | 380/586 [00:10<00:05, 36.78it/s]
 68%|██████▊   | 396/586 [00:10<00:05, 37.18it/s]
 66%|██████▌   | 384/586 [00:10<00:05, 36.89it/s]
 68%|██████▊   | 400/586 [00:10<00:04, 37.20it/s]
 66%|██████▌   | 388/586 [00:10<00:05, 36.99it/s]
 69%|██████▉   | 404/586 [00:10<00:04, 37.19it/s]
 67%|██████▋   | 392/586 [00:10<00:05, 36.92it/s]
 70%|██████▉   | 408/586 [00:11<00:04, 37.20it/s]
 68%|██████▊   | 396/586 [00:10<00:05, 36.99it/s]
 70%|███████   | 412/586 [00:11<00:04, 37.22it/s]
 68%|██████▊   | 400/586 [00:10<00:05, 37.09it/s]
 71%|███████   | 416/586 [00:11<00:04, 37.22it/s]
 69%|██████▉   | 404/586 [00:10<00:04, 37.16it/s]
 72%|███████▏  | 420/586 [00:11<00:04, 37.13it/s]
 70%|██████▉   | 408/586 [00:11<00:04, 37.24it/s]
 72%|███████▏  | 424/586 [00:11<00:04, 37.12it/s]
 70%|███████   | 412/586 [00:11<00:04, 37.25it/s]
 73%|███████▎  | 428/586 [00:11<00:04, 37.09it/s]
 71%|███████   | 416/586 [00:11<00:04, 37.21it/s]
 74%|███████▎  | 432/586 [00:11<00:04, 37.09it/s]
 72%|███████▏  | 420/586 [00:11<00:04, 37.20it/s]
 74%|███████▍  | 436/586 [00:11<00:04, 37.05it/s]
 72%|███████▏  | 424/586 [00:11<00:04, 37.20it/s]
 75%|███████▌  | 440/586 [00:11<00:03, 37.03it/s]
 73%|███████▎  | 428/586 [00:11<00:04, 37.21it/s]
 76%|███████▌  | 444/586 [00:12<00:03, 36.97it/s]
 74%|███████▎  | 432/586 [00:11<00:04, 37.03it/s]
 76%|███████▋  | 448/586 [00:12<00:03, 37.03it/s]
 74%|███████▍  | 436/586 [00:11<00:04, 36.96it/s]
 77%|███████▋  | 452/586 [00:12<00:03, 37.05it/s]
 75%|███████▌  | 440/586 [00:11<00:03, 36.99it/s]
 78%|███████▊  | 456/586 [00:12<00:03, 37.04it/s]
 76%|███████▌  | 444/586 [00:12<00:03, 37.04it/s]
 78%|███████▊  | 460/586 [00:12<00:03, 37.04it/s]
 76%|███████▋  | 448/586 [00:12<00:03, 37.05it/s]
 79%|███████▉  | 464/586 [00:12<00:03, 37.01it/s]
 77%|███████▋  | 452/586 [00:12<00:03, 37.00it/s]
 80%|███████▉  | 468/586 [00:12<00:03, 36.92it/s]
 78%|███████▊  | 456/586 [00:12<00:03, 36.99it/s]
 81%|████████  | 472/586 [00:12<00:03, 36.94it/s]
 78%|███████▊  | 460/586 [00:12<00:03, 37.04it/s]
 81%|████████  | 476/586 [00:12<00:02, 36.95it/s]
 79%|███████▉  | 464/586 [00:12<00:03, 36.89it/s]
 82%|████████▏ | 480/586 [00:12<00:02, 36.91it/s]
 80%|███████▉  | 468/586 [00:12<00:03, 36.90it/s]
 83%|████████▎ | 484/586 [00:13<00:02, 36.88it/s]
 81%|████████  | 472/586 [00:12<00:03, 36.95it/s]
 83%|████████▎ | 488/586 [00:13<00:02, 36.89it/s]
 81%|████████  | 476/586 [00:12<00:02, 37.01it/s]
 84%|████████▍ | 492/586 [00:13<00:02, 36.90it/s]
 82%|████████▏ | 480/586 [00:13<00:02, 37.08it/s]
 85%|████████▍ | 496/586 [00:13<00:02, 36.91it/s]
 83%|████████▎ | 484/586 [00:13<00:02, 37.13it/s]
 85%|████████▌ | 500/586 [00:13<00:02, 36.90it/s]
 83%|████████▎ | 488/586 [00:13<00:02, 37.15it/s]
 86%|████████▌ | 504/586 [00:13<00:02, 36.93it/s]
 84%|████████▍ | 492/586 [00:13<00:02, 37.18it/s]
 87%|████████▋ | 508/586 [00:13<00:02, 36.98it/s]
 85%|████████▍ | 496/586 [00:13<00:02, 37.17it/s]
 87%|████████▋ | 512/586 [00:13<00:02, 36.96it/s]
 85%|████████▌ | 500/586 [00:13<00:02, 37.18it/s]
 88%|████████▊ | 516/586 [00:13<00:01, 36.96it/s]
 86%|████████▌ | 504/586 [00:13<00:02, 37.20it/s]
 89%|████████▊ | 520/586 [00:14<00:01, 37.00it/s]
 87%|████████▋ | 508/586 [00:13<00:02, 37.20it/s]
 89%|████████▉ | 524/586 [00:14<00:01, 37.01it/s]
 87%|████████▋ | 512/586 [00:13<00:01, 37.23it/s]
 90%|█████████ | 528/586 [00:14<00:01, 36.50it/s]
 88%|████████▊ | 516/586 [00:14<00:01, 37.24it/s]
 91%|█████████ | 532/586 [00:14<00:01, 36.64it/s]
 89%|████████▊ | 520/586 [00:14<00:01, 37.20it/s]
 91%|█████████▏| 536/586 [00:14<00:01, 36.79it/s]
 89%|████████▉ | 524/586 [00:14<00:01, 37.19it/s]
 92%|█████████▏| 540/586 [00:14<00:01, 36.87it/s]
 90%|█████████ | 528/586 [00:14<00:01, 37.17it/s]
 93%|█████████▎| 544/586 [00:14<00:01, 36.96it/s]
 91%|█████████ | 532/586 [00:14<00:01, 37.12it/s]
 94%|█████████▎| 548/586 [00:14<00:01, 36.99it/s]
 91%|█████████▏| 536/586 [00:14<00:01, 37.15it/s]
 94%|█████████▍| 552/586 [00:14<00:00, 37.06it/s]
 92%|█████████▏| 540/586 [00:14<00:01, 37.07it/s]
 95%|█████████▍| 556/586 [00:15<00:00, 37.11it/s]
 93%|█████████▎| 544/586 [00:14<00:01, 36.52it/s]
 96%|█████████▌| 560/586 [00:15<00:00, 37.12it/s]
 94%|█████████▎| 548/586 [00:14<00:01, 36.63it/s]
 96%|█████████▌| 564/586 [00:15<00:00, 36.96it/s]
 94%|█████████▍| 552/586 [00:14<00:00, 36.75it/s]
 97%|█████████▋| 568/586 [00:15<00:00, 36.93it/s]
 95%|█████████▍| 556/586 [00:15<00:00, 36.84it/s]
 98%|█████████▊| 572/586 [00:15<00:00, 36.95it/s]
 96%|█████████▌| 560/586 [00:15<00:00, 36.79it/s]
 98%|█████████▊| 576/586 [00:15<00:00, 37.01it/s]
 96%|█████████▌| 564/586 [00:15<00:00, 36.82it/s]
 99%|█████████▉| 580/586 [00:15<00:00, 37.07it/s]
 97%|█████████▋| 568/586 [00:15<00:00, 36.85it/s]
100%|█████████▉| 584/586 [00:15<00:00, 37.12it/s]
100%|██████████| 586/586 [00:15<00:00, 36.98it/s]

 98%|█████████▊| 572/586 [00:15<00:00, 36.93it/s]n136-128-154:1707362:1707362 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1707362:1707362 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1707362:1707362 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:1707362:1707362 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:1707362:1707362 [0] NCCL INFO Comm config Blocking set to 1

 98%|█████████▊| 576/586 [00:15<00:00, 36.86it/s]
 99%|█████████▉| 580/586 [00:15<00:00, 36.87it/s]n136-128-154:1707362:1707614 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:1707362:1707614 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.

100%|█████████▉| 584/586 [00:15<00:00, 36.84it/s]
100%|██████████| 586/586 [00:15<00:00, 36.86it/s]
n136-128-154:1707363:1707363 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:1707363:1707363 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1707363:1707363 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1707362:1707614 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1707362:1707614 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1707362:1707614 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:1707363:1707363 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:1707363:1707363 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:1707362:1707614 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:1707362:1707614 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:1707362:1707614 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:1707362:1707614 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:1707362:1707614 [0] NCCL INFO Using network IB
n136-128-154:1707362:1707614 [0] NCCL INFO ncclCommInitRankConfig comm 0xf300900 rank 0 nranks 2 cudaDev 0 nvmlDev 1 busId 16000 commId 0xfa97e203f5b44b91 - Init START
n136-128-154:1707363:1707615 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:1707363:1707615 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:1707363:1707615 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1707363:1707615 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1707363:1707615 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:1707363:1707615 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:1707363:1707615 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:1707363:1707615 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:1707363:1707615 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:1707363:1707615 [1] NCCL INFO Using network IB
n136-128-154:1707363:1707615 [1] NCCL INFO ncclCommInitRankConfig comm 0xf6d1700 rank 1 nranks 2 cudaDev 1 nvmlDev 2 busId 4a000 commId 0xfa97e203f5b44b91 - Init START
n136-128-154:1707363:1707615 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:1707362:1707614 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:1707363:1707615 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:1707363:1707615 [1] NCCL INFO Retrieving state for IB
n136-128-154:1707363:1707615 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:1707363:1707615 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:1707363:1707615 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:1707363:1707615 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:1707363:1707615 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:1707362:1707614 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:1707362:1707614 [0] NCCL INFO Retrieving state for IB
n136-128-154:1707362:1707614 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:1707362:1707614 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:1707362:1707614 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:1707362:1707614 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:1707362:1707614 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:1707363:1707615 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1707363:1707615 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1707363:1707615 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1707363:1707615 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1707363:1707615 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:1707363:1707615 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:1707363:1707615 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:1707363:1707615 [1] NCCL INFO               + PCI[24.0] - PCI/0-14000 (1000c01010de13b8)
n136-128-154:1707363:1707615 [1] NCCL INFO                             + PCI[24.0] - GPU/0-16000 (0)
n136-128-154:1707363:1707615 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1707363:1707615 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:1707363:1707615 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:1707363:1707615 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:1707363:1707615 [1] NCCL INFO               + PCI[24.0] - PCI/0-48000 (1000c01010de13b8)
n136-128-154:1707363:1707615 [1] NCCL INFO                             + PCI[24.0] - GPU/0-4a000 (1)
n136-128-154:1707363:1707615 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1707363:1707615 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:1707363:1707615 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:1707363:1707615 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:1707363:1707615 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:1707363:1707615 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:1707363:1707615 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:1707363:1707615 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:1707363:1707615 [1] NCCL INFO ==========================================
n136-128-154:1707362:1707614 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1707363:1707615 [1] NCCL INFO GPU/0-16000 :GPU/0-16000 (0/5000.0/LOC) GPU/0-4a000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1707362:1707614 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1707363:1707615 [1] NCCL INFO GPU/0-4a000 :GPU/0-16000 (2/240.0/NVL) GPU/0-4a000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1707362:1707614 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1707362:1707614 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1707363:1707615 [1] NCCL INFO Setting affinity for GPU 2 to 1-31,65-95
n136-128-154:1707362:1707614 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:1707362:1707614 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:1707362:1707614 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:1707362:1707614 [0] NCCL INFO               + PCI[24.0] - PCI/0-14000 (1000c01010de13b8)
n136-128-154:1707362:1707614 [0] NCCL INFO                             + PCI[24.0] - GPU/0-16000 (0)
n136-128-154:1707362:1707614 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1707362:1707614 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:1707362:1707614 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:1707362:1707614 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:1707362:1707614 [0] NCCL INFO               + PCI[24.0] - PCI/0-48000 (1000c01010de13b8)
n136-128-154:1707362:1707614 [0] NCCL INFO                             + PCI[24.0] - GPU/0-4a000 (1)
n136-128-154:1707362:1707614 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1707362:1707614 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:1707362:1707614 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:1707362:1707614 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:1707362:1707614 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:1707362:1707614 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:1707362:1707614 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:1707363:1707615 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:1707362:1707614 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:1707363:1707615 [1] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707362:1707614 [0] NCCL INFO ==========================================
n136-128-154:1707363:1707615 [1] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707363:1707615 [1] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707363:1707615 [1] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707362:1707614 [0] NCCL INFO GPU/0-16000 :GPU/0-16000 (0/5000.0/LOC) GPU/0-4a000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1707363:1707615 [1] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707363:1707615 [1] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707362:1707614 [0] NCCL INFO GPU/0-4a000 :GPU/0-16000 (2/240.0/NVL) GPU/0-4a000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1707363:1707615 [1] NCCL INFO  6 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707363:1707615 [1] NCCL INFO  7 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707363:1707615 [1] NCCL INFO  8 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707363:1707615 [1] NCCL INFO  9 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707362:1707614 [0] NCCL INFO Setting affinity for GPU 1 to 1-31,65-95
n136-128-154:1707363:1707615 [1] NCCL INFO 10 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707363:1707615 [1] NCCL INFO 11 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707363:1707615 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:1707363:1707615 [1] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707363:1707615 [1] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707363:1707615 [1] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707363:1707615 [1] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707363:1707615 [1] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707363:1707615 [1] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707363:1707615 [1] NCCL INFO  6 : GPU/0-4a000 GPU/0-16000
n136-128-154:1707363:1707615 [1] NCCL INFO  7 : GPU/0-4a000 GPU/0-16000
n136-128-154:1707363:1707615 [1] NCCL INFO  8 : GPU/0-4a000 GPU/0-16000
n136-128-154:1707363:1707615 [1] NCCL INFO  9 : GPU/0-4a000 GPU/0-16000
n136-128-154:1707363:1707615 [1] NCCL INFO 10 : GPU/0-4a000 GPU/0-16000
n136-128-154:1707363:1707615 [1] NCCL INFO 11 : GPU/0-4a000 GPU/0-16000
n136-128-154:1707362:1707614 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:1707362:1707614 [0] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707362:1707614 [0] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707362:1707614 [0] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707362:1707614 [0] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707362:1707614 [0] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707362:1707614 [0] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707362:1707614 [0] NCCL INFO  6 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707362:1707614 [0] NCCL INFO  7 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707362:1707614 [0] NCCL INFO  8 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707362:1707614 [0] NCCL INFO  9 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707362:1707614 [0] NCCL INFO 10 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707362:1707614 [0] NCCL INFO 11 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707362:1707614 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:1707362:1707614 [0] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707362:1707614 [0] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707362:1707614 [0] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707362:1707614 [0] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707362:1707614 [0] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707362:1707614 [0] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707362:1707614 [0] NCCL INFO  6 : GPU/0-4a000 GPU/0-16000
n136-128-154:1707362:1707614 [0] NCCL INFO  7 : GPU/0-4a000 GPU/0-16000
n136-128-154:1707362:1707614 [0] NCCL INFO  8 : GPU/0-4a000 GPU/0-16000
n136-128-154:1707362:1707614 [0] NCCL INFO  9 : GPU/0-4a000 GPU/0-16000
n136-128-154:1707362:1707614 [0] NCCL INFO 10 : GPU/0-4a000 GPU/0-16000
n136-128-154:1707362:1707614 [0] NCCL INFO 11 : GPU/0-4a000 GPU/0-16000
n136-128-154:1707363:1707615 [1] NCCL INFO comm 0xf6d1700 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:1707362:1707614 [0] NCCL INFO comm 0xf300900 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:1707363:1707615 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:1707363:1707615 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:1707363:1707615 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:1707362:1707614 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:1707363:1707615 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:1707362:1707614 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:1707363:1707615 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:1707362:1707614 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:1707363:1707615 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:1707362:1707614 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:1707363:1707615 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:1707362:1707614 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:1707363:1707615 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:1707362:1707614 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:1707363:1707615 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:1707362:1707614 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:1707363:1707615 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:1707362:1707614 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:1707363:1707615 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:1707362:1707614 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:1707363:1707615 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:1707362:1707614 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:1707363:1707615 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:1707362:1707614 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:1707363:1707615 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:1707362:1707614 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:1707363:1707615 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:1707362:1707614 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:1707363:1707615 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:1707362:1707614 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:1707363:1707615 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:1707362:1707614 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:1707363:1707615 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:1707362:1707614 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:1707363:1707615 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:1707362:1707614 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:1707363:1707615 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:1707362:1707614 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:1707363:1707615 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:1707362:1707614 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:1707363:1707615 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:1707362:1707614 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:1707363:1707615 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:1707362:1707614 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:1707363:1707615 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:1707362:1707614 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:1707362:1707614 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:1707362:1707614 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:1707363:1707615 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:1707362:1707614 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:1707363:1707615 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:1707362:1707614 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:1707363:1707615 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:1707362:1707614 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:1707363:1707615 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:1707362:1707614 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:1707363:1707615 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:1707362:1707614 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:1707363:1707615 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:1707362:1707614 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:1707363:1707615 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:1707362:1707614 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:1707363:1707615 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:1707362:1707614 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:1707363:1707615 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:1707362:1707614 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:1707363:1707615 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:1707362:1707614 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:1707363:1707615 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:1707362:1707614 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:1707363:1707615 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:1707362:1707614 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:1707363:1707615 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:1707362:1707614 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:1707363:1707615 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:1707362:1707614 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:1707363:1707615 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:1707362:1707614 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:1707363:1707615 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:1707362:1707614 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:1707363:1707615 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:1707362:1707614 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:1707363:1707615 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:1707362:1707614 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:1707363:1707615 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:1707362:1707614 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:1707363:1707615 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:1707362:1707614 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:1707363:1707615 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:1707362:1707614 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:1707363:1707615 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:1707362:1707614 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:1707363:1707615 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:1707362:1707614 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:1707363:1707615 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:1707362:1707614 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:1707363:1707615 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:1707362:1707614 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:1707363:1707615 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:1707362:1707614 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:1707362:1707614 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:1707362:1707614 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:1707362:1707614 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:1707362:1707614 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:1707362:1707614 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:1707362:1707614 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:1707362:1707614 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:1707362:1707614 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:1707362:1707614 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:1707362:1707614 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:1707362:1707614 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:1707362:1707614 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:1707362:1707614 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:1707362:1707614 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:1707362:1707614 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:1707362:1707614 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:1707362:1707614 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:1707362:1707614 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:1707362:1707614 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:1707362:1707614 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:1707362:1707614 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:1707362:1707614 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:1707362:1707614 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:1707362:1707614 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:1707363:1707615 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:1707363:1707627 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 28
n136-128-154:1707363:1707626 [1] NCCL INFO [Proxy Service] Device 1 CPU core 27
n136-128-154:1707362:1707614 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:1707362:1707614 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:1707362:1707628 [0] NCCL INFO [Proxy Service] Device 0 CPU core 67
n136-128-154:1707362:1707629 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 68
n136-128-154:1707363:1707615 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:1707363:1707615 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:1707362:1707614 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:1707362:1707614 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:1707362:1707614 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:1707363:1707615 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:1707363:1707615 [1] NCCL INFO ncclCommInitRankConfig comm 0xf6d1700 rank 1 nranks 2 cudaDev 1 nvmlDev 2 busId 4a000 commId 0xfa97e203f5b44b91 - Init COMPLETE
n136-128-154:1707363:1707615 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.57 (kernels 0.12, alloc 0.24, bootstrap 0.00, allgathers 0.00, topo 0.06, graphs 0.00, connections 0.01, rest 0.14)
n136-128-154:1707362:1707614 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:1707362:1707614 [0] NCCL INFO ncclCommInitRankConfig comm 0xf300900 rank 0 nranks 2 cudaDev 0 nvmlDev 1 busId 16000 commId 0xfa97e203f5b44b91 - Init COMPLETE
n136-128-154:1707362:1707614 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.95 (kernels 0.20, alloc 0.25, bootstrap 0.29, allgathers 0.00, topo 0.06, graphs 0.00, connections 0.01, rest 0.14)
n136-128-154:1707363:1707630 [1] NCCL INFO Channel 00/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707630 [1] NCCL INFO Channel 01/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707630 [1] NCCL INFO Channel 02/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707630 [1] NCCL INFO Channel 03/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707630 [1] NCCL INFO Channel 04/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707630 [1] NCCL INFO Channel 05/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707630 [1] NCCL INFO Channel 06/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707630 [1] NCCL INFO Channel 07/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707630 [1] NCCL INFO Channel 08/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707630 [1] NCCL INFO Channel 09/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707630 [1] NCCL INFO Channel 10/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707630 [1] NCCL INFO Channel 11/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707362:1707631 [0] NCCL INFO Channel 00/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707363:1707630 [1] NCCL INFO Channel 12/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707362:1707631 [0] NCCL INFO Channel 01/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707363:1707630 [1] NCCL INFO Channel 13/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707362:1707631 [0] NCCL INFO Channel 02/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707363:1707630 [1] NCCL INFO Channel 14/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707362:1707631 [0] NCCL INFO Channel 03/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707363:1707630 [1] NCCL INFO Channel 15/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707362:1707631 [0] NCCL INFO Channel 04/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707363:1707630 [1] NCCL INFO Channel 16/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707362:1707631 [0] NCCL INFO Channel 05/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707363:1707630 [1] NCCL INFO Channel 17/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707362:1707631 [0] NCCL INFO Channel 06/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707363:1707630 [1] NCCL INFO Channel 18/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707362:1707631 [0] NCCL INFO Channel 07/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707363:1707630 [1] NCCL INFO Channel 19/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707362:1707631 [0] NCCL INFO Channel 08/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707363:1707630 [1] NCCL INFO Channel 20/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707362:1707631 [0] NCCL INFO Channel 09/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707363:1707630 [1] NCCL INFO Channel 21/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707362:1707631 [0] NCCL INFO Channel 10/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707363:1707630 [1] NCCL INFO Channel 22/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707362:1707631 [0] NCCL INFO Channel 11/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707363:1707630 [1] NCCL INFO Channel 23/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707362:1707631 [0] NCCL INFO Channel 12/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707362:1707631 [0] NCCL INFO Channel 13/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707362:1707631 [0] NCCL INFO Channel 14/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707362:1707631 [0] NCCL INFO Channel 15/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707362:1707631 [0] NCCL INFO Channel 16/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707362:1707631 [0] NCCL INFO Channel 17/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707362:1707631 [0] NCCL INFO Channel 18/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707362:1707631 [0] NCCL INFO Channel 19/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707362:1707631 [0] NCCL INFO Channel 20/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707362:1707631 [0] NCCL INFO Channel 21/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707362:1707631 [0] NCCL INFO Channel 22/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707362:1707631 [0] NCCL INFO Channel 23/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707363:1707630 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:1707362:1707631 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-06:03:06:57 INFO     [evaluator:559] Running loglikelihood requests
2025-12-06:03:06:57 INFO     [evaluator:559] Running loglikelihood requests

Running loglikelihood requests:   0%|          | 0/2344 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 45
Determined largest batch size: 45

Running loglikelihood requests:   0%|          | 1/2344 [00:56<36:51:25, 56.63s/it]
Running loglikelihood requests:   2%|▏         | 46/2344 [01:00<36:27,  1.05it/s]  
Running loglikelihood requests:   4%|▍         | 94/2344 [01:04<16:02,  2.34it/s]
Running loglikelihood requests:   6%|▌         | 142/2344 [01:07<09:44,  3.77it/s]
Running loglikelihood requests:   8%|▊         | 189/2344 [01:11<06:52,  5.22it/s]
Running loglikelihood requests:  10%|█         | 242/2344 [01:15<05:01,  6.97it/s]
Running loglikelihood requests:  12%|█▏        | 290/2344 [01:18<04:06,  8.33it/s]
Running loglikelihood requests:  14%|█▍        | 336/2344 [01:22<03:33,  9.40it/s]
Running loglikelihood requests:  16%|█▋        | 381/2344 [01:25<03:11, 10.26it/s]
Running loglikelihood requests:  18%|█▊        | 429/2344 [01:29<02:50, 11.21it/s]
Running loglikelihood requests:  20%|██        | 480/2344 [01:32<02:32, 12.19it/s]
Running loglikelihood requests:  22%|██▏       | 526/2344 [01:35<02:24, 12.58it/s]
Running loglikelihood requests:  25%|██▍       | 580/2344 [01:39<02:10, 13.53it/s]
Running loglikelihood requests:  27%|██▋       | 626/2344 [01:42<02:06, 13.56it/s]
Running loglikelihood requests:  29%|██▉       | 674/2344 [01:45<02:01, 13.78it/s]
Running loglikelihood requests:  31%|███       | 719/2344 [01:49<01:58, 13.75it/s]
Running loglikelihood requests:  33%|███▎      | 765/2344 [01:52<01:54, 13.82it/s]
Running loglikelihood requests:  35%|███▍      | 813/2344 [01:55<01:49, 14.04it/s]
Running loglikelihood requests:  37%|███▋      | 863/2344 [01:59<01:42, 14.42it/s]
Running loglikelihood requests:  39%|███▉      | 914/2344 [02:02<01:36, 14.81it/s]
Running loglikelihood requests:  41%|████      | 959/2344 [02:05<01:35, 14.56it/s]
Running loglikelihood requests:  43%|████▎     | 1005/2344 [02:08<01:32, 14.46it/s]
Running loglikelihood requests:  45%|████▍     | 1052/2344 [02:11<01:29, 14.50it/s]
Running loglikelihood requests:  47%|████▋     | 1110/2344 [02:15<01:19, 15.61it/s]
Running loglikelihood requests:  49%|████▉     | 1159/2344 [02:18<01:16, 15.56it/s]
Running loglikelihood requests:  52%|█████▏    | 1208/2344 [02:21<01:13, 15.54it/s]
Running loglikelihood requests:  54%|█████▎    | 1255/2344 [02:24<01:10, 15.36it/s]
Running loglikelihood requests:  56%|█████▌    | 1303/2344 [02:27<01:07, 15.33it/s]
Running loglikelihood requests:  58%|█████▊    | 1356/2344 [02:30<01:02, 15.82it/s]
Running loglikelihood requests:  60%|█████▉    | 1401/2344 [02:34<01:01, 15.42it/s]
Running loglikelihood requests:  62%|██████▏   | 1454/2344 [02:37<00:55, 15.94it/s]
Running loglikelihood requests:  64%|██████▍   | 1504/2344 [02:40<00:52, 16.05it/s]
Running loglikelihood requests:  66%|██████▋   | 1553/2344 [02:43<00:49, 16.09it/s]
Running loglikelihood requests:  68%|██████▊   | 1601/2344 [02:46<00:46, 16.02it/s]
Running loglikelihood requests:  71%|███████   | 1653/2344 [02:49<00:42, 16.40it/s]
Running loglikelihood requests:  72%|███████▏  | 1698/2344 [02:52<00:40, 16.01it/s]
Running loglikelihood requests:  74%|███████▍  | 1745/2344 [02:55<00:37, 15.95it/s]
Running loglikelihood requests:  76%|███████▋  | 1793/2344 [02:58<00:34, 16.03it/s]
Running loglikelihood requests:  78%|███████▊  | 1840/2344 [03:01<00:31, 16.00it/s]
Running loglikelihood requests:  81%|████████  | 1891/2344 [03:04<00:27, 16.40it/s]
Running loglikelihood requests:  83%|████████▎ | 1938/2344 [03:06<00:24, 16.36it/s]
Running loglikelihood requests:  85%|████████▍ | 1983/2344 [03:09<00:22, 16.14it/s]
Running loglikelihood requests:  87%|████████▋ | 2032/2344 [03:12<00:19, 16.38it/s]
Running loglikelihood requests:  89%|████████▉ | 2089/2344 [03:15<00:14, 17.42it/s]
Running loglikelihood requests:  91%|█████████ | 2137/2344 [03:18<00:11, 17.27it/s]
Running loglikelihood requests:  93%|█████████▎| 2185/2344 [03:21<00:09, 17.21it/s]
Running loglikelihood requests:  96%|█████████▌| 2241/2344 [03:24<00:05, 18.07it/s]
Running loglikelihood requests:  98%|█████████▊| 2295/2344 [03:26<00:02, 19.30it/s]
Running loglikelihood requests: 100%|██████████| 2344/2344 [03:26<00:00, 11.36it/s]
n136-128-154:1707363:1707852 [1] NCCL INFO Channel 00/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707852 [1] NCCL INFO Channel 01/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707852 [1] NCCL INFO Channel 02/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707852 [1] NCCL INFO Channel 03/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707852 [1] NCCL INFO Channel 04/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707852 [1] NCCL INFO Channel 05/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707852 [1] NCCL INFO Channel 06/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707852 [1] NCCL INFO Channel 07/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707852 [1] NCCL INFO Channel 08/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707852 [1] NCCL INFO Channel 09/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707852 [1] NCCL INFO Channel 10/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707852 [1] NCCL INFO Channel 11/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707852 [1] NCCL INFO Channel 12/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707852 [1] NCCL INFO Channel 13/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707852 [1] NCCL INFO Channel 14/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707852 [1] NCCL INFO Channel 15/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707852 [1] NCCL INFO Channel 16/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707852 [1] NCCL INFO Channel 17/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707852 [1] NCCL INFO Channel 18/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707852 [1] NCCL INFO Channel 19/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707852 [1] NCCL INFO Channel 20/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707852 [1] NCCL INFO Channel 21/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707852 [1] NCCL INFO Channel 22/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707852 [1] NCCL INFO Channel 23/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707852 [1] NCCL INFO Channel 24/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707852 [1] NCCL INFO Channel 25/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707852 [1] NCCL INFO Channel 26/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707852 [1] NCCL INFO Channel 27/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707852 [1] NCCL INFO Channel 28/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707852 [1] NCCL INFO Channel 29/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707852 [1] NCCL INFO Channel 30/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707363:1707852 [1] NCCL INFO Channel 31/1 : 1[2] -> 0[1] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:1707363:1707858 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1707363:1707858 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1707363:1707858 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1707363:1707858 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1707363:1707858 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1707363:1707858 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1707363:1707626 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-06:03:10:45 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 25, batch_size: auto (45)
|    Tasks    |Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|-------------|------:|------|-----:|--------|---|-----:|---|-----:|
|arc_challenge|      1|none  |    25|acc     |↑  |0.2474|±  |0.0126|
|             |       |none  |    25|acc_norm|↑  |0.2688|±  |0.0130|

[rank0]:[W1206 03:10:46.573087668 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:1707362:1707907 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1707362:1707907 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1707362:1707907 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1707362:1707628 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:1707362:1707907 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1707362:1707907 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1707362:1707907 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1707363:1707626 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:1707363:1707858 [1] NCCL INFO comm 0xf6d1700 rank 1 nranks 2 cudaDev 1 busId 4a000 - Abort COMPLETE
n136-128-154:1707362:1707907 [0] NCCL INFO comm 0xf300900 rank 0 nranks 2 cudaDev 0 busId 16000 - Abort COMPLETE
任务 arc_challenge 评估完成！

==================================================
开始评估：任务=truthfulqa_mc1 | 少样本数=0 | 模型=Qwen2.5-7B-quantization-fg
输出路径：results2/Qwen2.5-7B-quantization-fg/fg/truthfulqa_mc1.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-06:03:12:08 INFO     [__main__:440] Selected Tasks: ['truthfulqa_mc1']
2025-12-06:03:12:08 INFO     [__main__:440] Selected Tasks: ['truthfulqa_mc1']
2025-12-06:03:12:08 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-06:03:12:08 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-06:03:12:08 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-06:03:12:08 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-06:03:12:09 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-06:03:12:09 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-06:03:12:09 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.03s/it]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.05s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.03s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.04s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.03it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.03it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.48it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.26it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.49it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.26it/s]
2025-12-06:03:12:53 INFO     [evaluator:305] num_fewshot has been set to 0 for truthfulqa_mc1 in its config. Manual configuration will be ignored.
2025-12-06:03:12:53 INFO     [api.task:434] Building contexts for truthfulqa_mc1 on rank 0...

  0%|          | 0/409 [00:00<?, ?it/s]
 25%|██▍       | 102/409 [00:00<00:00, 1014.06it/s]
 50%|████▉     | 204/409 [00:00<00:00, 997.35it/s] 
 75%|███████▍  | 306/409 [00:00<00:00, 1006.49it/s]
100%|██████████| 409/409 [00:00<00:00, 1013.99it/s]
100%|██████████| 409/409 [00:00<00:00, 1010.05it/s]
n136-128-154:1707960:1707960 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1707960:1707960 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1707960:1707960 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:1707960:1707960 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:1707960:1707960 [0] NCCL INFO Comm config Blocking set to 1
n136-128-154:1707960:1708200 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:1707960:1708200 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:1707960:1708200 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1707960:1708200 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1707960:1708200 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:1707960:1708200 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:1707960:1708200 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:1707960:1708200 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:1707960:1708200 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:1707960:1708200 [0] NCCL INFO Using network IB
n136-128-154:1707960:1708200 [0] NCCL INFO ncclCommInitRankConfig comm 0x10ceec80 rank 0 nranks 2 cudaDev 0 nvmlDev 1 busId 16000 commId 0xc58854ca40f60e9c - Init START
2025-12-06:03:13:04 INFO     [evaluator:305] num_fewshot has been set to 0 for truthfulqa_mc1 in its config. Manual configuration will be ignored.
2025-12-06:03:13:04 INFO     [api.task:434] Building contexts for truthfulqa_mc1 on rank 1...

  0%|          | 0/408 [00:00<?, ?it/s]
 25%|██▍       | 101/408 [00:00<00:00, 1005.80it/s]
 50%|█████     | 205/408 [00:00<00:00, 1021.13it/s]
 75%|███████▌  | 308/408 [00:00<00:00, 1021.89it/s]
100%|██████████| 408/408 [00:00<00:00, 1021.37it/s]
n136-128-154:1707961:1707961 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:1707961:1707961 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1707961:1707961 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1707961:1707961 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:1707961:1707961 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:1707961:1708219 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:1707961:1708219 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:1707961:1708219 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1707961:1708219 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1707961:1708219 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:1707961:1708219 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:1707961:1708219 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:1707961:1708219 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:1707961:1708219 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:1707961:1708219 [1] NCCL INFO Using network IB
n136-128-154:1707961:1708219 [1] NCCL INFO ncclCommInitRankConfig comm 0x10477620 rank 1 nranks 2 cudaDev 1 nvmlDev 2 busId 4a000 commId 0xc58854ca40f60e9c - Init START
n136-128-154:1707961:1708219 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:1707960:1708200 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:1707961:1708219 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:1707961:1708219 [1] NCCL INFO Retrieving state for IB
n136-128-154:1707961:1708219 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:1707961:1708219 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:1707961:1708219 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:1707961:1708219 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:1707961:1708219 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:1707960:1708200 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:1707960:1708200 [0] NCCL INFO Retrieving state for IB
n136-128-154:1707960:1708200 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:1707960:1708200 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:1707960:1708200 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:1707960:1708200 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:1707960:1708200 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:1707960:1708200 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1707960:1708200 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1707960:1708200 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1707960:1708200 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1707961:1708219 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1707961:1708219 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1707961:1708219 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1707961:1708219 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1707960:1708200 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:1707960:1708200 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:1707960:1708200 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:1707960:1708200 [0] NCCL INFO               + PCI[24.0] - PCI/0-14000 (1000c01010de13b8)
n136-128-154:1707960:1708200 [0] NCCL INFO                             + PCI[24.0] - GPU/0-16000 (0)
n136-128-154:1707960:1708200 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1707960:1708200 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:1707960:1708200 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:1707960:1708200 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:1707960:1708200 [0] NCCL INFO               + PCI[24.0] - PCI/0-48000 (1000c01010de13b8)
n136-128-154:1707960:1708200 [0] NCCL INFO                             + PCI[24.0] - GPU/0-4a000 (1)
n136-128-154:1707960:1708200 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1707960:1708200 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:1707960:1708200 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:1707960:1708200 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:1707960:1708200 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:1707960:1708200 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:1707960:1708200 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:1707960:1708200 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:1707960:1708200 [0] NCCL INFO ==========================================
n136-128-154:1707960:1708200 [0] NCCL INFO GPU/0-16000 :GPU/0-16000 (0/5000.0/LOC) GPU/0-4a000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1707960:1708200 [0] NCCL INFO GPU/0-4a000 :GPU/0-16000 (2/240.0/NVL) GPU/0-4a000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1707960:1708200 [0] NCCL INFO Setting affinity for GPU 1 to 1-31,65-95
n136-128-154:1707961:1708219 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:1707961:1708219 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:1707961:1708219 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:1707961:1708219 [1] NCCL INFO               + PCI[24.0] - PCI/0-14000 (1000c01010de13b8)
n136-128-154:1707961:1708219 [1] NCCL INFO                             + PCI[24.0] - GPU/0-16000 (0)
n136-128-154:1707961:1708219 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1707961:1708219 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:1707961:1708219 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:1707961:1708219 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:1707961:1708219 [1] NCCL INFO               + PCI[24.0] - PCI/0-48000 (1000c01010de13b8)
n136-128-154:1707961:1708219 [1] NCCL INFO                             + PCI[24.0] - GPU/0-4a000 (1)
n136-128-154:1707961:1708219 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1707961:1708219 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:1707961:1708219 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:1707961:1708219 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:1707961:1708219 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:1707961:1708219 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:1707961:1708219 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:1707961:1708219 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:1707961:1708219 [1] NCCL INFO ==========================================
n136-128-154:1707961:1708219 [1] NCCL INFO GPU/0-16000 :GPU/0-16000 (0/5000.0/LOC) GPU/0-4a000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1707961:1708219 [1] NCCL INFO GPU/0-4a000 :GPU/0-16000 (2/240.0/NVL) GPU/0-4a000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1707961:1708219 [1] NCCL INFO Setting affinity for GPU 2 to 1-31,65-95
n136-128-154:1707960:1708200 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:1707960:1708200 [0] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707960:1708200 [0] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707960:1708200 [0] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707960:1708200 [0] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707960:1708200 [0] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707960:1708200 [0] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707960:1708200 [0] NCCL INFO  6 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707960:1708200 [0] NCCL INFO  7 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707960:1708200 [0] NCCL INFO  8 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707960:1708200 [0] NCCL INFO  9 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707960:1708200 [0] NCCL INFO 10 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707960:1708200 [0] NCCL INFO 11 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707961:1708219 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:1707961:1708219 [1] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707961:1708219 [1] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707961:1708219 [1] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707961:1708219 [1] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707961:1708219 [1] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707961:1708219 [1] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707961:1708219 [1] NCCL INFO  6 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707961:1708219 [1] NCCL INFO  7 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707961:1708219 [1] NCCL INFO  8 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707961:1708219 [1] NCCL INFO  9 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707961:1708219 [1] NCCL INFO 10 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707961:1708219 [1] NCCL INFO 11 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707960:1708200 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:1707960:1708200 [0] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707960:1708200 [0] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707960:1708200 [0] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707960:1708200 [0] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707960:1708200 [0] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707960:1708200 [0] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707960:1708200 [0] NCCL INFO  6 : GPU/0-4a000 GPU/0-16000
n136-128-154:1707960:1708200 [0] NCCL INFO  7 : GPU/0-4a000 GPU/0-16000
n136-128-154:1707960:1708200 [0] NCCL INFO  8 : GPU/0-4a000 GPU/0-16000
n136-128-154:1707960:1708200 [0] NCCL INFO  9 : GPU/0-4a000 GPU/0-16000
n136-128-154:1707960:1708200 [0] NCCL INFO 10 : GPU/0-4a000 GPU/0-16000
n136-128-154:1707960:1708200 [0] NCCL INFO 11 : GPU/0-4a000 GPU/0-16000
n136-128-154:1707961:1708219 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:1707961:1708219 [1] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707961:1708219 [1] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707961:1708219 [1] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707961:1708219 [1] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707961:1708219 [1] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707961:1708219 [1] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1707961:1708219 [1] NCCL INFO  6 : GPU/0-4a000 GPU/0-16000
n136-128-154:1707961:1708219 [1] NCCL INFO  7 : GPU/0-4a000 GPU/0-16000
n136-128-154:1707961:1708219 [1] NCCL INFO  8 : GPU/0-4a000 GPU/0-16000
n136-128-154:1707961:1708219 [1] NCCL INFO  9 : GPU/0-4a000 GPU/0-16000
n136-128-154:1707961:1708219 [1] NCCL INFO 10 : GPU/0-4a000 GPU/0-16000
n136-128-154:1707961:1708219 [1] NCCL INFO 11 : GPU/0-4a000 GPU/0-16000
n136-128-154:1707961:1708219 [1] NCCL INFO comm 0x10477620 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:1707960:1708200 [0] NCCL INFO comm 0x10ceec80 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:1707961:1708219 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:1707960:1708200 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:1707961:1708219 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:1707960:1708200 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:1707961:1708219 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:1707960:1708200 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:1707961:1708219 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:1707960:1708200 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:1707961:1708219 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:1707960:1708200 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:1707961:1708219 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:1707960:1708200 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:1707961:1708219 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:1707960:1708200 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:1707961:1708219 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:1707960:1708200 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:1707961:1708219 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:1707960:1708200 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:1707961:1708219 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:1707960:1708200 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:1707961:1708219 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:1707960:1708200 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:1707961:1708219 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:1707960:1708200 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:1707961:1708219 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:1707960:1708200 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:1707961:1708219 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:1707960:1708200 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:1707961:1708219 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:1707960:1708200 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:1707961:1708219 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:1707960:1708200 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:1707961:1708219 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:1707960:1708200 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:1707961:1708219 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:1707960:1708200 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:1707961:1708219 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:1707960:1708200 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:1707961:1708219 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:1707960:1708200 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:1707961:1708219 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:1707960:1708200 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:1707961:1708219 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:1707960:1708200 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:1707961:1708219 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:1707960:1708200 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:1707961:1708219 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:1707960:1708200 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:1707960:1708200 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:1707960:1708200 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:1707961:1708219 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:1707960:1708200 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:1707961:1708219 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:1707960:1708200 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:1707961:1708219 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:1707960:1708200 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:1707961:1708219 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:1707960:1708200 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:1707961:1708219 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:1707960:1708200 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:1707961:1708219 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:1707960:1708200 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:1707961:1708219 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:1707960:1708200 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:1707961:1708219 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:1707960:1708200 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:1707961:1708219 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:1707960:1708200 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:1707961:1708219 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:1707960:1708200 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:1707961:1708219 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:1707960:1708200 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:1707961:1708219 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:1707960:1708200 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:1707961:1708219 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:1707960:1708200 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:1707961:1708219 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:1707960:1708200 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:1707961:1708219 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:1707960:1708200 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:1707961:1708219 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:1707960:1708200 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:1707961:1708219 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:1707960:1708200 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:1707961:1708219 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:1707960:1708200 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:1707961:1708219 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:1707960:1708200 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:1707961:1708219 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:1707960:1708200 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:1707961:1708219 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:1707960:1708200 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:1707961:1708219 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:1707960:1708200 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:1707961:1708219 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:1707960:1708200 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:1707961:1708219 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:1707960:1708200 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:1707961:1708219 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:1707960:1708200 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:1707960:1708200 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:1707961:1708219 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:1707960:1708200 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:1707960:1708200 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:1707960:1708200 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:1707960:1708200 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:1707960:1708200 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:1707960:1708200 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:1707960:1708200 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:1707960:1708200 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:1707960:1708200 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:1707960:1708200 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:1707960:1708200 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:1707960:1708200 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:1707960:1708200 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:1707960:1708200 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:1707960:1708200 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:1707960:1708200 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:1707960:1708200 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:1707960:1708200 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:1707960:1708200 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:1707960:1708200 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:1707960:1708200 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:1707960:1708200 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:1707960:1708200 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:1707960:1708200 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:1707960:1708228 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 3
n136-128-154:1707960:1708227 [0] NCCL INFO [Proxy Service] Device 0 CPU core 2
n136-128-154:1707961:1708219 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:1707961:1708229 [1] NCCL INFO [Proxy Service] Device 1 CPU core 87
n136-128-154:1707961:1708230 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 25
n136-128-154:1707961:1708219 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:1707961:1708219 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:1707960:1708200 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:1707960:1708200 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:1707960:1708200 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:1707961:1708219 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:1707961:1708219 [1] NCCL INFO ncclCommInitRankConfig comm 0x10477620 rank 1 nranks 2 cudaDev 1 nvmlDev 2 busId 4a000 commId 0xc58854ca40f60e9c - Init COMPLETE
n136-128-154:1707961:1708219 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.64 (kernels 0.13, alloc 0.22, bootstrap 0.00, allgathers 0.00, topo 0.06, graphs 0.00, connections 0.06, rest 0.17)
n136-128-154:1707960:1708200 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:1707960:1708200 [0] NCCL INFO ncclCommInitRankConfig comm 0x10ceec80 rank 0 nranks 2 cudaDev 0 nvmlDev 1 busId 16000 commId 0xc58854ca40f60e9c - Init COMPLETE
n136-128-154:1707960:1708200 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 12.30 (kernels 0.16, alloc 0.31, bootstrap 11.54, allgathers 0.00, topo 0.06, graphs 0.00, connections 0.06, rest 0.17)
n136-128-154:1707961:1708232 [1] NCCL INFO Channel 00/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708232 [1] NCCL INFO Channel 01/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708232 [1] NCCL INFO Channel 02/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708232 [1] NCCL INFO Channel 03/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708232 [1] NCCL INFO Channel 04/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708232 [1] NCCL INFO Channel 05/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708232 [1] NCCL INFO Channel 06/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708232 [1] NCCL INFO Channel 07/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708232 [1] NCCL INFO Channel 08/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707960:1708233 [0] NCCL INFO Channel 00/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707961:1708232 [1] NCCL INFO Channel 09/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707960:1708233 [0] NCCL INFO Channel 01/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707961:1708232 [1] NCCL INFO Channel 10/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707960:1708233 [0] NCCL INFO Channel 02/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707961:1708232 [1] NCCL INFO Channel 11/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707960:1708233 [0] NCCL INFO Channel 03/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707961:1708232 [1] NCCL INFO Channel 12/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707960:1708233 [0] NCCL INFO Channel 04/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707961:1708232 [1] NCCL INFO Channel 13/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707960:1708233 [0] NCCL INFO Channel 05/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707961:1708232 [1] NCCL INFO Channel 14/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707960:1708233 [0] NCCL INFO Channel 06/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707961:1708232 [1] NCCL INFO Channel 15/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707960:1708233 [0] NCCL INFO Channel 07/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707961:1708232 [1] NCCL INFO Channel 16/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708232 [1] NCCL INFO Channel 17/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707960:1708233 [0] NCCL INFO Channel 08/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707961:1708232 [1] NCCL INFO Channel 18/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707960:1708233 [0] NCCL INFO Channel 09/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707961:1708232 [1] NCCL INFO Channel 19/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707960:1708233 [0] NCCL INFO Channel 10/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707961:1708232 [1] NCCL INFO Channel 20/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708232 [1] NCCL INFO Channel 21/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707960:1708233 [0] NCCL INFO Channel 11/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707961:1708232 [1] NCCL INFO Channel 22/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707960:1708233 [0] NCCL INFO Channel 12/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707961:1708232 [1] NCCL INFO Channel 23/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707960:1708233 [0] NCCL INFO Channel 13/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707960:1708233 [0] NCCL INFO Channel 14/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707960:1708233 [0] NCCL INFO Channel 15/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707960:1708233 [0] NCCL INFO Channel 16/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707960:1708233 [0] NCCL INFO Channel 17/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707960:1708233 [0] NCCL INFO Channel 18/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707960:1708233 [0] NCCL INFO Channel 19/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707960:1708233 [0] NCCL INFO Channel 20/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707960:1708233 [0] NCCL INFO Channel 21/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707960:1708233 [0] NCCL INFO Channel 22/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707960:1708233 [0] NCCL INFO Channel 23/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1707960:1708233 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:1707961:1708232 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-06:03:13:06 INFO     [evaluator:559] Running loglikelihood requests
2025-12-06:03:13:06 INFO     [evaluator:559] Running loglikelihood requests

Running loglikelihood requests:   0%|          | 0/2066 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 64
Determined largest batch size: 64

Running loglikelihood requests:   0%|          | 1/2066 [00:07<4:07:12,  7.18s/it]
Running loglikelihood requests:   3%|▎         | 65/2066 [00:08<03:11, 10.45it/s] 
Running loglikelihood requests:   6%|▌         | 129/2066 [00:09<01:36, 20.05it/s]
Running loglikelihood requests:   9%|▉         | 193/2066 [00:10<01:05, 28.58it/s]
Running loglikelihood requests:  12%|█▏        | 257/2066 [00:11<00:50, 35.82it/s]
Running loglikelihood requests:  16%|█▌        | 321/2066 [00:12<00:41, 41.71it/s]
Running loglikelihood requests:  19%|█▊        | 385/2066 [00:13<00:36, 46.28it/s]
Running loglikelihood requests:  22%|██▏       | 449/2066 [00:14<00:32, 49.91it/s]
Running loglikelihood requests:  25%|██▍       | 513/2066 [00:15<00:29, 52.74it/s]
Running loglikelihood requests:  28%|██▊       | 577/2066 [00:17<00:27, 54.85it/s]
Running loglikelihood requests:  31%|███       | 641/2066 [00:18<00:25, 56.45it/s]
Running loglikelihood requests:  34%|███▍      | 705/2066 [00:19<00:23, 57.64it/s]
Running loglikelihood requests:  37%|███▋      | 769/2066 [00:20<00:22, 58.42it/s]
Running loglikelihood requests:  40%|████      | 833/2066 [00:21<00:20, 59.03it/s]
Running loglikelihood requests:  43%|████▎     | 897/2066 [00:22<00:19, 59.57it/s]
Running loglikelihood requests:  47%|████▋     | 961/2066 [00:23<00:18, 59.96it/s]
Running loglikelihood requests:  50%|████▉     | 1025/2066 [00:24<00:17, 60.17it/s]
Running loglikelihood requests:  53%|█████▎    | 1089/2066 [00:25<00:16, 60.40it/s]
Running loglikelihood requests:  56%|█████▌    | 1153/2066 [00:26<00:15, 60.76it/s]
Running loglikelihood requests:  59%|█████▉    | 1217/2066 [00:27<00:13, 60.89it/s]
Running loglikelihood requests:  62%|██████▏   | 1281/2066 [00:28<00:12, 61.10it/s]
Running loglikelihood requests:  65%|██████▌   | 1345/2066 [00:29<00:11, 61.21it/s]
Running loglikelihood requests:  68%|██████▊   | 1409/2066 [00:30<00:10, 61.42it/s]
Running loglikelihood requests:  71%|███████▏  | 1473/2066 [00:31<00:09, 61.47it/s]
Running loglikelihood requests:  74%|███████▍  | 1537/2066 [00:32<00:08, 61.58it/s]
Running loglikelihood requests:  77%|███████▋  | 1601/2066 [00:33<00:07, 61.67it/s]
Running loglikelihood requests:  81%|████████  | 1665/2066 [00:34<00:06, 61.83it/s]
Running loglikelihood requests:  84%|████████▎ | 1729/2066 [00:35<00:05, 61.92it/s]
Running loglikelihood requests:  87%|████████▋ | 1793/2066 [00:36<00:04, 61.75it/s]
Running loglikelihood requests:  90%|████████▉ | 1857/2066 [00:37<00:03, 61.95it/s]
Running loglikelihood requests:  93%|█████████▎| 1921/2066 [00:38<00:02, 62.13it/s]
Running loglikelihood requests:  96%|█████████▌| 1985/2066 [00:39<00:01, 62.70it/s]
Running loglikelihood requests:  99%|█████████▉| 2049/2066 [00:40<00:00, 79.64it/s]
Running loglikelihood requests: 100%|██████████| 2066/2066 [00:40<00:00, 51.35it/s]
n136-128-154:1707961:1708276 [1] NCCL INFO Channel 00/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708276 [1] NCCL INFO Channel 01/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708276 [1] NCCL INFO Channel 02/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708276 [1] NCCL INFO Channel 03/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708276 [1] NCCL INFO Channel 04/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708276 [1] NCCL INFO Channel 05/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708276 [1] NCCL INFO Channel 06/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708276 [1] NCCL INFO Channel 07/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708276 [1] NCCL INFO Channel 08/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708276 [1] NCCL INFO Channel 09/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708276 [1] NCCL INFO Channel 10/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708276 [1] NCCL INFO Channel 11/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708276 [1] NCCL INFO Channel 12/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708276 [1] NCCL INFO Channel 13/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708276 [1] NCCL INFO Channel 14/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708276 [1] NCCL INFO Channel 15/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708276 [1] NCCL INFO Channel 16/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708276 [1] NCCL INFO Channel 17/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708276 [1] NCCL INFO Channel 18/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708276 [1] NCCL INFO Channel 19/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708276 [1] NCCL INFO Channel 20/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708276 [1] NCCL INFO Channel 21/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708276 [1] NCCL INFO Channel 22/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708276 [1] NCCL INFO Channel 23/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708276 [1] NCCL INFO Channel 24/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708276 [1] NCCL INFO Channel 25/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708276 [1] NCCL INFO Channel 26/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708276 [1] NCCL INFO Channel 27/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708276 [1] NCCL INFO Channel 28/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708276 [1] NCCL INFO Channel 29/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708276 [1] NCCL INFO Channel 30/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1707961:1708276 [1] NCCL INFO Channel 31/1 : 1[2] -> 0[1] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:1707961:1708282 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1707961:1708282 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1707961:1708282 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1707961:1708282 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1707961:1708282 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1707961:1708282 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1707961:1708229 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-06:03:13:58 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: auto (64)
|    Tasks     |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|--------------|------:|------|-----:|------|---|-----:|---|-----:|
|truthfulqa_mc1|      2|none  |     0|acc   |↑  |0.2803|±  |0.0157|

[rank0]:[W1206 03:13:59.665614963 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:1707960:1708331 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1707960:1708331 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1707960:1708331 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1707960:1708227 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:1707960:1708331 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1707960:1708331 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1707960:1708331 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1707961:1708229 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:1707961:1708282 [1] NCCL INFO comm 0x10477620 rank 1 nranks 2 cudaDev 1 busId 4a000 - Abort COMPLETE
n136-128-154:1707960:1708331 [0] NCCL INFO comm 0x10ceec80 rank 0 nranks 2 cudaDev 0 busId 16000 - Abort COMPLETE
任务 truthfulqa_mc1 评估完成！

==================================================
开始评估：任务=piqa | 少样本数=0 | 模型=Qwen2.5-7B-quantization-fg
输出路径：results2/Qwen2.5-7B-quantization-fg/fg/piqa.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-06:03:15:20 INFO     [__main__:440] Selected Tasks: ['piqa']
2025-12-06:03:15:20 INFO     [__main__:440] Selected Tasks: ['piqa']
2025-12-06:03:15:20 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-06:03:15:20 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-06:03:15:20 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-06:03:15:20 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-06:03:15:21 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-06:03:15:22 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
`torch_dtype` is deprecated! Use `dtype` instead!
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-06:03:15:22 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.14it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.09it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.14it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.01it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.22it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.75it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.48it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.06it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.51it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.30it/s]
2025-12-06:03:15:34 WARNING  [evaluator:309] Overwriting default num_fewshot of piqa from None to 0
2025-12-06:03:15:34 INFO     [api.task:434] Building contexts for piqa on rank 1...

  0%|          | 0/919 [00:00<?, ?it/s]
 16%|█▋        | 151/919 [00:00<00:00, 1507.94it/s]
 33%|███▎      | 305/919 [00:00<00:00, 1526.01it/s]
 50%|████▉     | 459/919 [00:00<00:00, 1531.27it/s]2025-12-06:03:15:35 WARNING  [evaluator:309] Overwriting default num_fewshot of piqa from None to 0
2025-12-06:03:15:35 INFO     [api.task:434] Building contexts for piqa on rank 0...

  0%|          | 0/919 [00:00<?, ?it/s]
 67%|██████▋   | 615/919 [00:00<00:00, 1538.99it/s]
 17%|█▋        | 154/919 [00:00<00:00, 1531.17it/s]
 84%|████████▎ | 769/919 [00:00<00:00, 1528.89it/s]
 34%|███▎      | 310/919 [00:00<00:00, 1544.51it/s]
100%|██████████| 919/919 [00:00<00:00, 1531.70it/s]

 51%|█████     | 466/919 [00:00<00:00, 1548.15it/s]
 68%|██████▊   | 622/919 [00:00<00:00, 1549.44it/s]
 85%|████████▍ | 777/919 [00:00<00:00, 1534.49it/s]
100%|██████████| 919/919 [00:00<00:00, 1541.73it/s]
n136-128-154:1708385:1708385 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1708385:1708385 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1708385:1708385 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:1708385:1708385 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:1708385:1708385 [0] NCCL INFO Comm config Blocking set to 1
n136-128-154:1708386:1708386 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:1708386:1708386 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1708386:1708386 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1708386:1708386 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:1708386:1708386 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:1708385:1708579 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:1708386:1708580 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:1708385:1708579 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:1708386:1708580 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:1708385:1708579 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1708385:1708579 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1708386:1708580 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1708386:1708580 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1708385:1708579 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:1708386:1708580 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:1708386:1708580 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:1708386:1708580 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:1708386:1708580 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:1708385:1708579 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:1708385:1708579 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:1708385:1708579 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:1708386:1708580 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:1708386:1708580 [1] NCCL INFO Using network IB
n136-128-154:1708385:1708579 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:1708385:1708579 [0] NCCL INFO Using network IB
n136-128-154:1708385:1708579 [0] NCCL INFO ncclCommInitRankConfig comm 0x10cf2a90 rank 0 nranks 2 cudaDev 0 nvmlDev 1 busId 16000 commId 0x8d4536edfece5aa8 - Init START
n136-128-154:1708386:1708580 [1] NCCL INFO ncclCommInitRankConfig comm 0x10dd3860 rank 1 nranks 2 cudaDev 1 nvmlDev 2 busId 4a000 commId 0x8d4536edfece5aa8 - Init START
n136-128-154:1708386:1708580 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:1708385:1708579 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:1708386:1708580 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:1708386:1708580 [1] NCCL INFO Retrieving state for IB
n136-128-154:1708386:1708580 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:1708386:1708580 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:1708386:1708580 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:1708386:1708580 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:1708386:1708580 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:1708385:1708579 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:1708385:1708579 [0] NCCL INFO Retrieving state for IB
n136-128-154:1708385:1708579 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:1708385:1708579 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:1708385:1708579 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:1708385:1708579 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:1708385:1708579 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:1708385:1708579 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1708385:1708579 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1708385:1708579 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1708385:1708579 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1708386:1708580 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1708386:1708580 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1708386:1708580 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1708386:1708580 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1708385:1708579 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:1708385:1708579 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:1708385:1708579 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:1708385:1708579 [0] NCCL INFO               + PCI[24.0] - PCI/0-14000 (1000c01010de13b8)
n136-128-154:1708385:1708579 [0] NCCL INFO                             + PCI[24.0] - GPU/0-16000 (0)
n136-128-154:1708385:1708579 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1708385:1708579 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:1708385:1708579 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:1708385:1708579 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:1708385:1708579 [0] NCCL INFO               + PCI[24.0] - PCI/0-48000 (1000c01010de13b8)
n136-128-154:1708385:1708579 [0] NCCL INFO                             + PCI[24.0] - GPU/0-4a000 (1)
n136-128-154:1708385:1708579 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1708385:1708579 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:1708385:1708579 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:1708385:1708579 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:1708385:1708579 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:1708385:1708579 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:1708385:1708579 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:1708385:1708579 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:1708385:1708579 [0] NCCL INFO ==========================================
n136-128-154:1708385:1708579 [0] NCCL INFO GPU/0-16000 :GPU/0-16000 (0/5000.0/LOC) GPU/0-4a000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1708385:1708579 [0] NCCL INFO GPU/0-4a000 :GPU/0-16000 (2/240.0/NVL) GPU/0-4a000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1708385:1708579 [0] NCCL INFO Setting affinity for GPU 1 to 1-31,65-95
n136-128-154:1708386:1708580 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:1708386:1708580 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:1708386:1708580 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:1708386:1708580 [1] NCCL INFO               + PCI[24.0] - PCI/0-14000 (1000c01010de13b8)
n136-128-154:1708386:1708580 [1] NCCL INFO                             + PCI[24.0] - GPU/0-16000 (0)
n136-128-154:1708386:1708580 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1708386:1708580 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:1708386:1708580 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:1708386:1708580 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:1708386:1708580 [1] NCCL INFO               + PCI[24.0] - PCI/0-48000 (1000c01010de13b8)
n136-128-154:1708386:1708580 [1] NCCL INFO                             + PCI[24.0] - GPU/0-4a000 (1)
n136-128-154:1708386:1708580 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1708386:1708580 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:1708386:1708580 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:1708386:1708580 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:1708386:1708580 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:1708386:1708580 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:1708386:1708580 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:1708386:1708580 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:1708386:1708580 [1] NCCL INFO ==========================================
n136-128-154:1708386:1708580 [1] NCCL INFO GPU/0-16000 :GPU/0-16000 (0/5000.0/LOC) GPU/0-4a000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1708385:1708579 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:1708386:1708580 [1] NCCL INFO GPU/0-4a000 :GPU/0-16000 (2/240.0/NVL) GPU/0-4a000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1708385:1708579 [0] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708386:1708580 [1] NCCL INFO Setting affinity for GPU 2 to 1-31,65-95
n136-128-154:1708385:1708579 [0] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708385:1708579 [0] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708385:1708579 [0] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708385:1708579 [0] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708385:1708579 [0] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708385:1708579 [0] NCCL INFO  6 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708385:1708579 [0] NCCL INFO  7 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708385:1708579 [0] NCCL INFO  8 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708385:1708579 [0] NCCL INFO  9 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708385:1708579 [0] NCCL INFO 10 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708385:1708579 [0] NCCL INFO 11 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708385:1708579 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:1708385:1708579 [0] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708385:1708579 [0] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708385:1708579 [0] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708385:1708579 [0] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708385:1708579 [0] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708385:1708579 [0] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708385:1708579 [0] NCCL INFO  6 : GPU/0-4a000 GPU/0-16000
n136-128-154:1708385:1708579 [0] NCCL INFO  7 : GPU/0-4a000 GPU/0-16000
n136-128-154:1708385:1708579 [0] NCCL INFO  8 : GPU/0-4a000 GPU/0-16000
n136-128-154:1708385:1708579 [0] NCCL INFO  9 : GPU/0-4a000 GPU/0-16000
n136-128-154:1708385:1708579 [0] NCCL INFO 10 : GPU/0-4a000 GPU/0-16000
n136-128-154:1708385:1708579 [0] NCCL INFO 11 : GPU/0-4a000 GPU/0-16000
n136-128-154:1708386:1708580 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:1708386:1708580 [1] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708386:1708580 [1] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708386:1708580 [1] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708386:1708580 [1] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708386:1708580 [1] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708386:1708580 [1] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708386:1708580 [1] NCCL INFO  6 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708386:1708580 [1] NCCL INFO  7 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708386:1708580 [1] NCCL INFO  8 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708386:1708580 [1] NCCL INFO  9 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708386:1708580 [1] NCCL INFO 10 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708386:1708580 [1] NCCL INFO 11 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708386:1708580 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:1708386:1708580 [1] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708386:1708580 [1] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708386:1708580 [1] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708386:1708580 [1] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708386:1708580 [1] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708386:1708580 [1] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708386:1708580 [1] NCCL INFO  6 : GPU/0-4a000 GPU/0-16000
n136-128-154:1708386:1708580 [1] NCCL INFO  7 : GPU/0-4a000 GPU/0-16000
n136-128-154:1708386:1708580 [1] NCCL INFO  8 : GPU/0-4a000 GPU/0-16000
n136-128-154:1708386:1708580 [1] NCCL INFO  9 : GPU/0-4a000 GPU/0-16000
n136-128-154:1708386:1708580 [1] NCCL INFO 10 : GPU/0-4a000 GPU/0-16000
n136-128-154:1708386:1708580 [1] NCCL INFO 11 : GPU/0-4a000 GPU/0-16000
n136-128-154:1708386:1708580 [1] NCCL INFO comm 0x10dd3860 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:1708385:1708579 [0] NCCL INFO comm 0x10cf2a90 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:1708386:1708580 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:1708386:1708580 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:1708385:1708579 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:1708386:1708580 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:1708386:1708580 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:1708385:1708579 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:1708386:1708580 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:1708386:1708580 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:1708385:1708579 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:1708386:1708580 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:1708386:1708580 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:1708385:1708579 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:1708386:1708580 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:1708386:1708580 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:1708385:1708579 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:1708386:1708580 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:1708386:1708580 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:1708385:1708579 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:1708386:1708580 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:1708386:1708580 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:1708385:1708579 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:1708386:1708580 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:1708386:1708580 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:1708385:1708579 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:1708386:1708580 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:1708386:1708580 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:1708385:1708579 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:1708386:1708580 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:1708386:1708580 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:1708385:1708579 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:1708386:1708580 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:1708386:1708580 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:1708385:1708579 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:1708386:1708580 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:1708386:1708580 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:1708385:1708579 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:1708385:1708579 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:1708385:1708579 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:1708385:1708579 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:1708385:1708579 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:1708386:1708580 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:1708385:1708579 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:1708386:1708580 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:1708385:1708579 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:1708386:1708580 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:1708386:1708580 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:1708385:1708579 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:1708386:1708580 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:1708386:1708580 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:1708385:1708579 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:1708386:1708580 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:1708385:1708579 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:1708386:1708580 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:1708385:1708579 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:1708386:1708580 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:1708385:1708579 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:1708386:1708580 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:1708385:1708579 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:1708386:1708580 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:1708386:1708580 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:1708386:1708580 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:1708386:1708580 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:1708385:1708579 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:1708386:1708580 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:1708386:1708580 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:1708385:1708579 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:1708386:1708580 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:1708386:1708580 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:1708385:1708579 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:1708386:1708580 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:1708386:1708580 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:1708385:1708579 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:1708386:1708580 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:1708386:1708580 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:1708385:1708579 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:1708386:1708580 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:1708386:1708580 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:1708385:1708579 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:1708386:1708580 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:1708385:1708579 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:1708386:1708580 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:1708385:1708579 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:1708385:1708579 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:1708385:1708579 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:1708385:1708579 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:1708385:1708579 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:1708385:1708579 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:1708385:1708579 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:1708385:1708579 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:1708385:1708579 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:1708385:1708579 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:1708385:1708579 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:1708385:1708579 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:1708385:1708579 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:1708385:1708579 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:1708385:1708579 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:1708385:1708579 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:1708385:1708579 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:1708385:1708579 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:1708385:1708579 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:1708385:1708579 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:1708385:1708579 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:1708385:1708579 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:1708385:1708579 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:1708385:1708579 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:1708385:1708579 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:1708385:1708579 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:1708385:1708579 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:1708385:1708579 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:1708385:1708579 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:1708385:1708579 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:1708385:1708579 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:1708385:1708579 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:1708385:1708579 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:1708385:1708579 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:1708385:1708579 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:1708385:1708579 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:1708385:1708579 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:1708385:1708579 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:1708385:1708579 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:1708385:1708579 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:1708385:1708579 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:1708385:1708579 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:1708385:1708579 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:1708385:1708579 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:1708385:1708579 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:1708385:1708591 [0] NCCL INFO [Proxy Service] Device 0 CPU core 7
n136-128-154:1708385:1708592 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 72
n136-128-154:1708386:1708580 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:1708386:1708594 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 73
n136-128-154:1708386:1708593 [1] NCCL INFO [Proxy Service] Device 1 CPU core 4
n136-128-154:1708385:1708579 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:1708385:1708579 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:1708385:1708579 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:1708386:1708580 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:1708386:1708580 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:1708385:1708579 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:1708385:1708579 [0] NCCL INFO ncclCommInitRankConfig comm 0x10cf2a90 rank 0 nranks 2 cudaDev 0 nvmlDev 1 busId 16000 commId 0x8d4536edfece5aa8 - Init COMPLETE
n136-128-154:1708385:1708579 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.77 (kernels 0.24, alloc 0.22, bootstrap 0.01, allgathers 0.01, topo 0.12, graphs 0.00, connections 0.09, rest 0.08)
n136-128-154:1708386:1708580 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:1708386:1708580 [1] NCCL INFO ncclCommInitRankConfig comm 0x10dd3860 rank 1 nranks 2 cudaDev 1 nvmlDev 2 busId 4a000 commId 0x8d4536edfece5aa8 - Init COMPLETE
n136-128-154:1708386:1708580 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.74 (kernels 0.21, alloc 0.23, bootstrap 0.00, allgathers 0.01, topo 0.12, graphs 0.00, connections 0.08, rest 0.08)
n136-128-154:1708385:1708595 [0] NCCL INFO Channel 00/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708385:1708595 [0] NCCL INFO Channel 01/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708386:1708596 [1] NCCL INFO Channel 00/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708385:1708595 [0] NCCL INFO Channel 02/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708386:1708596 [1] NCCL INFO Channel 01/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708385:1708595 [0] NCCL INFO Channel 03/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708386:1708596 [1] NCCL INFO Channel 02/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708385:1708595 [0] NCCL INFO Channel 04/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708385:1708595 [0] NCCL INFO Channel 05/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708386:1708596 [1] NCCL INFO Channel 03/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708385:1708595 [0] NCCL INFO Channel 06/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708386:1708596 [1] NCCL INFO Channel 04/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708386:1708596 [1] NCCL INFO Channel 05/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708385:1708595 [0] NCCL INFO Channel 07/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708385:1708595 [0] NCCL INFO Channel 08/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708386:1708596 [1] NCCL INFO Channel 06/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708385:1708595 [0] NCCL INFO Channel 09/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708386:1708596 [1] NCCL INFO Channel 07/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708385:1708595 [0] NCCL INFO Channel 10/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708386:1708596 [1] NCCL INFO Channel 08/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708385:1708595 [0] NCCL INFO Channel 11/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708386:1708596 [1] NCCL INFO Channel 09/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708385:1708595 [0] NCCL INFO Channel 12/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708386:1708596 [1] NCCL INFO Channel 10/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708385:1708595 [0] NCCL INFO Channel 13/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708386:1708596 [1] NCCL INFO Channel 11/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708385:1708595 [0] NCCL INFO Channel 14/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708386:1708596 [1] NCCL INFO Channel 12/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708385:1708595 [0] NCCL INFO Channel 15/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708386:1708596 [1] NCCL INFO Channel 13/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708385:1708595 [0] NCCL INFO Channel 16/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708386:1708596 [1] NCCL INFO Channel 14/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708385:1708595 [0] NCCL INFO Channel 17/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708386:1708596 [1] NCCL INFO Channel 15/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708386:1708596 [1] NCCL INFO Channel 16/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708385:1708595 [0] NCCL INFO Channel 18/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708386:1708596 [1] NCCL INFO Channel 17/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708385:1708595 [0] NCCL INFO Channel 19/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708386:1708596 [1] NCCL INFO Channel 18/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708385:1708595 [0] NCCL INFO Channel 20/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708386:1708596 [1] NCCL INFO Channel 19/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708385:1708595 [0] NCCL INFO Channel 21/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708386:1708596 [1] NCCL INFO Channel 20/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708385:1708595 [0] NCCL INFO Channel 22/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708386:1708596 [1] NCCL INFO Channel 21/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708385:1708595 [0] NCCL INFO Channel 23/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708386:1708596 [1] NCCL INFO Channel 22/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708386:1708596 [1] NCCL INFO Channel 23/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708386:1708596 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:1708385:1708595 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-06:03:15:36 INFO     [evaluator:559] Running loglikelihood requests
2025-12-06:03:15:36 INFO     [evaluator:559] Running loglikelihood requests
Passed argument batch_size = auto:1. Detecting largest batch size

Running loglikelihood requests:   0%|          | 0/1838 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 64
Determined largest batch size: 64

Running loglikelihood requests:   0%|          | 1/1838 [00:07<3:55:46,  7.70s/it]
Running loglikelihood requests:   4%|▎         | 65/1838 [00:08<02:45, 10.71it/s] 
Running loglikelihood requests:   7%|▋         | 129/1838 [00:08<01:15, 22.72it/s]
Running loglikelihood requests:  11%|█         | 194/1838 [00:09<00:45, 36.27it/s]
Running loglikelihood requests:  14%|█▍        | 258/1838 [00:09<00:31, 50.57it/s]
Running loglikelihood requests:  18%|█▊        | 322/1838 [00:10<00:23, 65.46it/s]
Running loglikelihood requests:  21%|██        | 386/1838 [00:10<00:18, 79.88it/s]
Running loglikelihood requests:  25%|██▍       | 451/1838 [00:11<00:14, 94.11it/s]
Running loglikelihood requests:  28%|██▊       | 515/1838 [00:11<00:12, 107.09it/s]
Running loglikelihood requests:  32%|███▏      | 579/1838 [00:12<00:10, 118.68it/s]
Running loglikelihood requests:  35%|███▍      | 643/1838 [00:12<00:09, 128.74it/s]
Running loglikelihood requests:  38%|███▊      | 707/1838 [00:12<00:08, 134.40it/s]
Running loglikelihood requests:  42%|████▏     | 771/1838 [00:13<00:07, 142.39it/s]
Running loglikelihood requests:  45%|████▌     | 835/1838 [00:13<00:06, 148.65it/s]
Running loglikelihood requests:  49%|████▉     | 899/1838 [00:14<00:06, 154.80it/s]
Running loglikelihood requests:  52%|█████▏    | 964/1838 [00:14<00:05, 160.31it/s]
Running loglikelihood requests:  56%|█████▌    | 1028/1838 [00:14<00:04, 164.91it/s]
Running loglikelihood requests:  59%|█████▉    | 1092/1838 [00:15<00:04, 168.58it/s]
Running loglikelihood requests:  63%|██████▎   | 1157/1838 [00:15<00:03, 172.27it/s]
Running loglikelihood requests:  66%|██████▋   | 1221/1838 [00:15<00:03, 176.14it/s]
Running loglikelihood requests:  70%|██████▉   | 1286/1838 [00:16<00:03, 179.80it/s]
Running loglikelihood requests:  73%|███████▎  | 1350/1838 [00:16<00:02, 183.12it/s]
Running loglikelihood requests:  77%|███████▋  | 1414/1838 [00:16<00:02, 185.74it/s]
Running loglikelihood requests:  80%|████████  | 1478/1838 [00:17<00:01, 188.44it/s]
Running loglikelihood requests:  84%|████████▍ | 1542/1838 [00:17<00:01, 191.36it/s]
Running loglikelihood requests:  87%|████████▋ | 1606/1838 [00:17<00:01, 193.83it/s]
Running loglikelihood requests:  91%|█████████ | 1670/1838 [00:18<00:00, 197.82it/s]
Running loglikelihood requests:  94%|█████████▍| 1734/1838 [00:18<00:00, 201.91it/s]
Running loglikelihood requests:  98%|█████████▊| 1799/1838 [00:18<00:00, 228.85it/s]
Running loglikelihood requests: 100%|██████████| 1838/1838 [00:18<00:00, 98.15it/s] 
n136-128-154:1708386:1708616 [1] NCCL INFO Channel 00/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708386:1708616 [1] NCCL INFO Channel 01/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708386:1708616 [1] NCCL INFO Channel 02/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708386:1708616 [1] NCCL INFO Channel 03/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708386:1708616 [1] NCCL INFO Channel 04/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708386:1708616 [1] NCCL INFO Channel 05/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708386:1708616 [1] NCCL INFO Channel 06/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708386:1708616 [1] NCCL INFO Channel 07/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708386:1708616 [1] NCCL INFO Channel 08/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708386:1708616 [1] NCCL INFO Channel 09/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708386:1708616 [1] NCCL INFO Channel 10/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708386:1708616 [1] NCCL INFO Channel 11/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708386:1708616 [1] NCCL INFO Channel 12/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708386:1708616 [1] NCCL INFO Channel 13/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708386:1708616 [1] NCCL INFO Channel 14/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708386:1708616 [1] NCCL INFO Channel 15/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708386:1708616 [1] NCCL INFO Channel 16/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708386:1708616 [1] NCCL INFO Channel 17/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708386:1708616 [1] NCCL INFO Channel 18/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708386:1708616 [1] NCCL INFO Channel 19/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708386:1708616 [1] NCCL INFO Channel 20/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708386:1708616 [1] NCCL INFO Channel 21/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708386:1708616 [1] NCCL INFO Channel 22/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708386:1708616 [1] NCCL INFO Channel 23/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708386:1708616 [1] NCCL INFO Channel 24/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708386:1708616 [1] NCCL INFO Channel 25/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708386:1708616 [1] NCCL INFO Channel 26/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708386:1708616 [1] NCCL INFO Channel 27/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708386:1708616 [1] NCCL INFO Channel 28/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708386:1708616 [1] NCCL INFO Channel 29/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708386:1708616 [1] NCCL INFO Channel 30/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708386:1708616 [1] NCCL INFO Channel 31/1 : 1[2] -> 0[1] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:1708386:1708621 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1708386:1708621 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1708386:1708621 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1708386:1708621 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1708386:1708621 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1708386:1708621 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1708386:1708593 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-06:03:16:06 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: auto (64)
|Tasks|Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|-----|------:|------|-----:|--------|---|-----:|---|-----:|
|piqa |      1|none  |     0|acc     |↑  |0.6523|±  |0.0111|
|     |       |none  |     0|acc_norm|↑  |0.6567|±  |0.0111|

[rank0]:[W1206 03:16:07.675223866 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:1708385:1708672 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1708385:1708672 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1708385:1708672 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1708385:1708591 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:1708385:1708672 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1708385:1708672 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1708385:1708672 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1708386:1708593 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:1708386:1708621 [1] NCCL INFO comm 0x10dd3860 rank 1 nranks 2 cudaDev 1 busId 4a000 - Abort COMPLETE
n136-128-154:1708385:1708672 [0] NCCL INFO comm 0x10cf2a90 rank 0 nranks 2 cudaDev 0 busId 16000 - Abort COMPLETE
任务 piqa 评估完成！

==================================================
开始评估：任务=hellaswag | 少样本数=10 | 模型=Qwen2.5-7B-quantization-fg
输出路径：results2/Qwen2.5-7B-quantization-fg/fg/hellaswag.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-06:03:17:32 INFO     [__main__:440] Selected Tasks: ['hellaswag']
2025-12-06:03:17:32 INFO     [__main__:440] Selected Tasks: ['hellaswag']
2025-12-06:03:17:32 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-06:03:17:32 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-06:03:17:32 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-06:03:17:32 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-06:03:17:33 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-06:03:17:33 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-06:03:17:33 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
`torch_dtype` is deprecated! Use `dtype` instead!
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.18s/it]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.20s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.13s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.13s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.06s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.06s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.37it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.15it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.37it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.15it/s]
2025-12-06:03:17:50 WARNING  [evaluator:309] Overwriting default num_fewshot of hellaswag from None to 10
2025-12-06:03:17:50 INFO     [api.task:434] Building contexts for hellaswag on rank 0...
2025-12-06:03:17:51 WARNING  [evaluator:309] Overwriting default num_fewshot of hellaswag from None to 10
2025-12-06:03:17:51 INFO     [api.task:434] Building contexts for hellaswag on rank 1...

  0%|          | 0/5021 [00:00<?, ?it/s]
  0%|          | 19/5021 [00:00<00:27, 182.94it/s]
  1%|          | 39/5021 [00:00<00:26, 188.21it/s]
  1%|          | 59/5021 [00:00<00:25, 192.52it/s]
  0%|          | 0/5021 [00:00<?, ?it/s]
  2%|▏         | 79/5021 [00:00<00:25, 195.20it/s]
  0%|          | 19/5021 [00:00<00:26, 189.34it/s]
  2%|▏         | 99/5021 [00:00<00:25, 196.51it/s]
  1%|          | 39/5021 [00:00<00:25, 194.00it/s]
  2%|▏         | 119/5021 [00:00<00:24, 197.37it/s]
  1%|          | 59/5021 [00:00<00:25, 195.08it/s]
  3%|▎         | 139/5021 [00:00<00:24, 197.86it/s]
  2%|▏         | 79/5021 [00:00<00:25, 196.21it/s]
  3%|▎         | 159/5021 [00:00<00:24, 198.10it/s]
  2%|▏         | 99/5021 [00:00<00:25, 196.39it/s]
  4%|▎         | 179/5021 [00:00<00:24, 198.52it/s]
  2%|▏         | 119/5021 [00:00<00:24, 196.93it/s]
  4%|▍         | 199/5021 [00:01<00:24, 198.48it/s]
  3%|▎         | 139/5021 [00:00<00:24, 197.27it/s]
  4%|▍         | 219/5021 [00:01<00:24, 198.91it/s]
  3%|▎         | 159/5021 [00:00<00:24, 198.00it/s]
  5%|▍         | 240/5021 [00:01<00:23, 199.55it/s]
  4%|▎         | 179/5021 [00:00<00:24, 198.31it/s]
  5%|▌         | 260/5021 [00:01<00:23, 198.48it/s]
  4%|▍         | 199/5021 [00:01<00:24, 198.32it/s]
  6%|▌         | 280/5021 [00:01<00:23, 198.26it/s]
  4%|▍         | 219/5021 [00:01<00:24, 198.39it/s]
  6%|▌         | 300/5021 [00:01<00:23, 198.23it/s]
  5%|▍         | 239/5021 [00:01<00:24, 198.55it/s]
  6%|▋         | 320/5021 [00:01<00:23, 197.80it/s]
  5%|▌         | 259/5021 [00:01<00:23, 198.73it/s]
  7%|▋         | 340/5021 [00:01<00:23, 197.88it/s]
  6%|▌         | 279/5021 [00:01<00:23, 198.82it/s]
  7%|▋         | 360/5021 [00:01<00:23, 197.87it/s]
  6%|▌         | 299/5021 [00:01<00:24, 196.47it/s]
  8%|▊         | 380/5021 [00:01<00:23, 197.82it/s]
  6%|▋         | 319/5021 [00:01<00:23, 197.39it/s]
  8%|▊         | 400/5021 [00:02<00:23, 197.72it/s]
  7%|▋         | 339/5021 [00:01<00:23, 197.69it/s]
  8%|▊         | 420/5021 [00:02<00:23, 195.57it/s]
  7%|▋         | 359/5021 [00:01<00:23, 197.83it/s]
  9%|▉         | 440/5021 [00:02<00:23, 194.97it/s]
  8%|▊         | 379/5021 [00:01<00:23, 197.89it/s]
  8%|▊         | 399/5021 [00:02<00:23, 197.95it/s]
  9%|▉         | 460/5021 [00:02<00:23, 194.28it/s]
  8%|▊         | 419/5021 [00:02<00:23, 198.02it/s]
 10%|▉         | 480/5021 [00:02<00:23, 194.61it/s]
  9%|▊         | 439/5021 [00:02<00:23, 198.08it/s]
 10%|▉         | 500/5021 [00:02<00:23, 194.82it/s]
  9%|▉         | 459/5021 [00:02<00:23, 197.73it/s]
 10%|█         | 520/5021 [00:02<00:23, 195.08it/s]
 10%|▉         | 479/5021 [00:02<00:22, 197.59it/s]
 11%|█         | 540/5021 [00:02<00:22, 195.18it/s]
 10%|▉         | 499/5021 [00:02<00:22, 197.83it/s]
 11%|█         | 560/5021 [00:02<00:22, 195.66it/s]
 10%|█         | 519/5021 [00:02<00:22, 198.09it/s]
 12%|█▏        | 580/5021 [00:02<00:22, 195.10it/s]
 11%|█         | 539/5021 [00:02<00:22, 198.59it/s]
 12%|█▏        | 600/5021 [00:03<00:22, 192.55it/s]
 11%|█         | 559/5021 [00:02<00:22, 198.55it/s]
 12%|█▏        | 620/5021 [00:03<00:23, 190.44it/s]
 12%|█▏        | 579/5021 [00:02<00:22, 198.60it/s]
 13%|█▎        | 640/5021 [00:03<00:22, 191.51it/s]
 12%|█▏        | 599/5021 [00:03<00:22, 198.74it/s]
 13%|█▎        | 660/5021 [00:03<00:22, 192.01it/s]
 12%|█▏        | 619/5021 [00:03<00:22, 196.93it/s]
 14%|█▎        | 680/5021 [00:03<00:22, 193.31it/s]
 13%|█▎        | 639/5021 [00:03<00:22, 195.64it/s]
 14%|█▍        | 700/5021 [00:03<00:22, 194.16it/s]
 13%|█▎        | 659/5021 [00:03<00:22, 194.96it/s]
 14%|█▍        | 720/5021 [00:03<00:22, 194.67it/s]
 14%|█▎        | 679/5021 [00:03<00:22, 194.11it/s]
 15%|█▍        | 740/5021 [00:03<00:21, 195.18it/s]
 14%|█▍        | 699/5021 [00:03<00:22, 193.73it/s]
 15%|█▌        | 760/5021 [00:03<00:21, 195.75it/s]
 14%|█▍        | 719/5021 [00:03<00:22, 193.85it/s]
 16%|█▌        | 780/5021 [00:03<00:21, 196.22it/s]
 15%|█▍        | 739/5021 [00:03<00:22, 194.13it/s]
 16%|█▌        | 800/5021 [00:04<00:21, 196.61it/s]
 15%|█▌        | 759/5021 [00:03<00:21, 194.08it/s]
 16%|█▋        | 820/5021 [00:04<00:21, 197.03it/s]
 16%|█▌        | 779/5021 [00:03<00:22, 192.22it/s]
 17%|█▋        | 840/5021 [00:04<00:21, 197.04it/s]
 16%|█▌        | 799/5021 [00:04<00:21, 192.39it/s]
 17%|█▋        | 860/5021 [00:04<00:21, 197.03it/s]
 16%|█▋        | 819/5021 [00:04<00:21, 192.72it/s]
 18%|█▊        | 880/5021 [00:04<00:21, 196.84it/s]
 17%|█▋        | 839/5021 [00:04<00:21, 193.01it/s]
 18%|█▊        | 900/5021 [00:04<00:20, 196.66it/s]
 17%|█▋        | 859/5021 [00:04<00:21, 192.95it/s]
 18%|█▊        | 920/5021 [00:04<00:20, 195.47it/s]
 18%|█▊        | 879/5021 [00:04<00:21, 192.99it/s]
 19%|█▊        | 940/5021 [00:04<00:20, 195.90it/s]
 18%|█▊        | 899/5021 [00:04<00:21, 193.28it/s]
 19%|█▉        | 960/5021 [00:04<00:20, 196.11it/s]
 20%|█▉        | 980/5021 [00:05<00:20, 196.27it/s]
 18%|█▊        | 919/5021 [00:04<00:21, 193.73it/s]
 20%|█▉        | 1000/5021 [00:05<00:20, 196.74it/s]
 19%|█▊        | 939/5021 [00:04<00:21, 193.19it/s]
 20%|██        | 1020/5021 [00:05<00:20, 196.85it/s]
 19%|█▉        | 959/5021 [00:04<00:21, 193.30it/s]
 21%|██        | 1040/5021 [00:05<00:20, 196.98it/s]
 19%|█▉        | 979/5021 [00:04<00:20, 194.15it/s]
 21%|██        | 1060/5021 [00:05<00:20, 197.24it/s]
 20%|█▉        | 999/5021 [00:05<00:20, 194.69it/s]
 22%|██▏       | 1080/5021 [00:05<00:19, 197.49it/s]
 20%|██        | 1019/5021 [00:05<00:20, 195.29it/s]
 22%|██▏       | 1100/5021 [00:05<00:19, 197.86it/s]
 21%|██        | 1039/5021 [00:05<00:20, 195.98it/s]
 22%|██▏       | 1120/5021 [00:05<00:19, 197.52it/s]
 21%|██        | 1059/5021 [00:05<00:20, 196.56it/s]
 23%|██▎       | 1140/5021 [00:05<00:19, 197.92it/s]
 21%|██▏       | 1079/5021 [00:05<00:20, 196.58it/s]
 23%|██▎       | 1160/5021 [00:05<00:19, 198.41it/s]
 22%|██▏       | 1099/5021 [00:05<00:19, 196.40it/s]
 24%|██▎       | 1180/5021 [00:06<00:19, 198.29it/s]
 22%|██▏       | 1119/5021 [00:05<00:19, 196.40it/s]
 24%|██▍       | 1200/5021 [00:06<00:19, 198.61it/s]
 23%|██▎       | 1139/5021 [00:05<00:19, 196.35it/s]
 24%|██▍       | 1220/5021 [00:06<00:19, 198.58it/s]
 23%|██▎       | 1159/5021 [00:05<00:19, 196.55it/s]
 25%|██▍       | 1240/5021 [00:06<00:19, 198.50it/s]
 23%|██▎       | 1179/5021 [00:06<00:19, 196.44it/s]
 25%|██▌       | 1260/5021 [00:06<00:18, 198.90it/s]
 24%|██▍       | 1199/5021 [00:06<00:19, 196.33it/s]
 25%|██▌       | 1280/5021 [00:06<00:18, 199.07it/s]
 24%|██▍       | 1219/5021 [00:06<00:19, 196.22it/s]
 26%|██▌       | 1300/5021 [00:06<00:18, 199.33it/s]
 25%|██▍       | 1239/5021 [00:06<00:19, 196.37it/s]
 26%|██▋       | 1320/5021 [00:06<00:18, 198.68it/s]
 25%|██▌       | 1259/5021 [00:06<00:19, 196.39it/s]
 27%|██▋       | 1340/5021 [00:06<00:18, 198.38it/s]
 25%|██▌       | 1279/5021 [00:06<00:19, 196.62it/s]
 27%|██▋       | 1360/5021 [00:06<00:18, 198.28it/s]
 26%|██▌       | 1299/5021 [00:06<00:18, 196.80it/s]
 27%|██▋       | 1380/5021 [00:07<00:18, 198.14it/s]
 26%|██▋       | 1319/5021 [00:06<00:18, 196.64it/s]
 28%|██▊       | 1400/5021 [00:07<00:18, 197.90it/s]
 27%|██▋       | 1339/5021 [00:06<00:18, 196.87it/s]
 28%|██▊       | 1420/5021 [00:07<00:18, 197.55it/s]
 27%|██▋       | 1359/5021 [00:06<00:18, 196.83it/s]
 29%|██▊       | 1440/5021 [00:07<00:18, 197.39it/s]
 27%|██▋       | 1379/5021 [00:07<00:18, 197.00it/s]
 29%|██▉       | 1460/5021 [00:07<00:18, 197.76it/s]
 28%|██▊       | 1399/5021 [00:07<00:18, 197.07it/s]
 29%|██▉       | 1480/5021 [00:07<00:17, 197.40it/s]
 28%|██▊       | 1419/5021 [00:07<00:18, 197.04it/s]
 30%|██▉       | 1500/5021 [00:07<00:17, 197.23it/s]
 29%|██▊       | 1439/5021 [00:07<00:18, 197.18it/s]
 30%|███       | 1520/5021 [00:07<00:17, 197.01it/s]
 29%|██▉       | 1459/5021 [00:07<00:18, 197.36it/s]
 31%|███       | 1540/5021 [00:07<00:17, 197.46it/s]
 29%|██▉       | 1479/5021 [00:07<00:17, 197.44it/s]
 31%|███       | 1560/5021 [00:07<00:17, 196.87it/s]
 30%|██▉       | 1499/5021 [00:07<00:17, 197.54it/s]
 31%|███▏      | 1580/5021 [00:08<00:17, 195.93it/s]
 30%|███       | 1519/5021 [00:07<00:17, 197.94it/s]
 32%|███▏      | 1600/5021 [00:08<00:17, 196.00it/s]
 31%|███       | 1539/5021 [00:07<00:17, 198.12it/s]
 32%|███▏      | 1620/5021 [00:08<00:17, 196.06it/s]
 31%|███       | 1559/5021 [00:07<00:17, 198.10it/s]
 33%|███▎      | 1640/5021 [00:08<00:17, 196.48it/s]
 31%|███▏      | 1579/5021 [00:08<00:17, 196.77it/s]
 33%|███▎      | 1660/5021 [00:08<00:17, 196.64it/s]
 32%|███▏      | 1599/5021 [00:08<00:17, 196.79it/s]
 33%|███▎      | 1680/5021 [00:08<00:17, 196.32it/s]
 32%|███▏      | 1619/5021 [00:08<00:17, 196.58it/s]
 34%|███▍      | 1700/5021 [00:08<00:16, 196.90it/s]
 33%|███▎      | 1639/5021 [00:08<00:17, 196.36it/s]
 34%|███▍      | 1720/5021 [00:08<00:16, 197.24it/s]
 33%|███▎      | 1659/5021 [00:08<00:17, 196.22it/s]
 35%|███▍      | 1740/5021 [00:08<00:16, 197.34it/s]
 33%|███▎      | 1679/5021 [00:08<00:17, 196.03it/s]
 35%|███▌      | 1760/5021 [00:08<00:16, 196.97it/s]
 34%|███▍      | 1699/5021 [00:08<00:16, 195.97it/s]
 35%|███▌      | 1780/5021 [00:09<00:16, 196.83it/s]
 34%|███▍      | 1719/5021 [00:08<00:16, 196.32it/s]
 36%|███▌      | 1800/5021 [00:09<00:16, 196.68it/s]
 35%|███▍      | 1739/5021 [00:08<00:16, 196.37it/s]
 36%|███▌      | 1820/5021 [00:09<00:16, 196.67it/s]
 35%|███▌      | 1759/5021 [00:08<00:16, 196.52it/s]
 37%|███▋      | 1840/5021 [00:09<00:16, 196.68it/s]
 35%|███▌      | 1779/5021 [00:09<00:16, 196.56it/s]
 37%|███▋      | 1860/5021 [00:09<00:16, 196.22it/s]
 36%|███▌      | 1799/5021 [00:09<00:16, 196.00it/s]
 37%|███▋      | 1880/5021 [00:09<00:16, 193.66it/s]
 36%|███▌      | 1819/5021 [00:09<00:16, 196.13it/s]
 38%|███▊      | 1900/5021 [00:09<00:16, 193.94it/s]
 37%|███▋      | 1839/5021 [00:09<00:16, 196.39it/s]
 38%|███▊      | 1920/5021 [00:09<00:15, 194.21it/s]
 37%|███▋      | 1859/5021 [00:09<00:16, 196.70it/s]
 39%|███▊      | 1940/5021 [00:09<00:16, 192.13it/s]
 37%|███▋      | 1879/5021 [00:09<00:15, 196.70it/s]
 39%|███▉      | 1960/5021 [00:09<00:15, 191.98it/s]
 38%|███▊      | 1899/5021 [00:09<00:15, 195.76it/s]
 39%|███▉      | 1980/5021 [00:10<00:15, 192.67it/s]
 38%|███▊      | 1919/5021 [00:09<00:15, 196.28it/s]
 40%|███▉      | 2000/5021 [00:10<00:15, 193.25it/s]
 39%|███▊      | 1939/5021 [00:09<00:15, 196.12it/s]
 40%|████      | 2020/5021 [00:10<00:15, 193.60it/s]
 39%|███▉      | 1959/5021 [00:09<00:15, 196.32it/s]
 41%|████      | 2040/5021 [00:10<00:15, 193.90it/s]
 39%|███▉      | 1979/5021 [00:10<00:15, 196.36it/s]
 41%|████      | 2060/5021 [00:10<00:15, 194.52it/s]
 40%|███▉      | 1999/5021 [00:10<00:15, 196.54it/s]
 41%|████▏     | 2080/5021 [00:10<00:15, 194.79it/s]
 40%|████      | 2019/5021 [00:10<00:15, 196.55it/s]
 42%|████▏     | 2100/5021 [00:10<00:14, 195.71it/s]
 41%|████      | 2039/5021 [00:10<00:15, 196.47it/s]
 42%|████▏     | 2120/5021 [00:10<00:14, 196.75it/s]
 41%|████      | 2059/5021 [00:10<00:15, 196.29it/s]
 43%|████▎     | 2140/5021 [00:10<00:14, 197.11it/s]
 41%|████▏     | 2079/5021 [00:10<00:14, 196.27it/s]
 43%|████▎     | 2160/5021 [00:11<00:14, 197.22it/s]
 42%|████▏     | 2099/5021 [00:10<00:14, 195.80it/s]
 43%|████▎     | 2180/5021 [00:11<00:14, 197.12it/s]
 42%|████▏     | 2119/5021 [00:10<00:14, 195.57it/s]
 44%|████▍     | 2200/5021 [00:11<00:14, 196.68it/s]
 43%|████▎     | 2139/5021 [00:10<00:14, 195.66it/s]
 44%|████▍     | 2220/5021 [00:11<00:14, 196.77it/s]
 43%|████▎     | 2159/5021 [00:10<00:14, 196.07it/s]
 45%|████▍     | 2240/5021 [00:11<00:14, 196.15it/s]
 43%|████▎     | 2179/5021 [00:11<00:14, 195.94it/s]
 45%|████▌     | 2260/5021 [00:11<00:14, 196.03it/s]
 44%|████▍     | 2199/5021 [00:11<00:14, 195.96it/s]
 45%|████▌     | 2280/5021 [00:11<00:13, 195.90it/s]
 44%|████▍     | 2219/5021 [00:11<00:14, 194.70it/s]
 46%|████▌     | 2300/5021 [00:11<00:13, 195.98it/s]
 45%|████▍     | 2239/5021 [00:11<00:14, 194.48it/s]
 46%|████▌     | 2320/5021 [00:11<00:13, 195.90it/s]
 45%|████▍     | 2259/5021 [00:11<00:14, 194.53it/s]
 47%|████▋     | 2340/5021 [00:11<00:13, 195.19it/s]
 45%|████▌     | 2279/5021 [00:11<00:14, 194.80it/s]
 47%|████▋     | 2360/5021 [00:12<00:13, 195.57it/s]
 46%|████▌     | 2299/5021 [00:11<00:13, 194.99it/s]
 47%|████▋     | 2380/5021 [00:12<00:13, 196.25it/s]
 46%|████▌     | 2319/5021 [00:11<00:13, 194.99it/s]
 48%|████▊     | 2400/5021 [00:12<00:13, 196.68it/s]
 47%|████▋     | 2339/5021 [00:11<00:13, 195.34it/s]
 48%|████▊     | 2420/5021 [00:12<00:13, 196.95it/s]
 47%|████▋     | 2359/5021 [00:12<00:13, 195.72it/s]
 49%|████▊     | 2440/5021 [00:12<00:13, 197.06it/s]
 47%|████▋     | 2379/5021 [00:12<00:13, 195.94it/s]
 49%|████▉     | 2460/5021 [00:12<00:12, 197.20it/s]
 48%|████▊     | 2399/5021 [00:12<00:13, 195.99it/s]
 49%|████▉     | 2480/5021 [00:12<00:12, 197.33it/s]
 48%|████▊     | 2419/5021 [00:12<00:13, 195.69it/s]
 50%|████▉     | 2500/5021 [00:12<00:12, 197.73it/s]
 49%|████▊     | 2439/5021 [00:12<00:13, 195.69it/s]
 50%|█████     | 2520/5021 [00:12<00:12, 198.22it/s]
 49%|████▉     | 2459/5021 [00:12<00:13, 195.55it/s]
 51%|█████     | 2540/5021 [00:12<00:12, 198.27it/s]
 49%|████▉     | 2479/5021 [00:12<00:13, 195.41it/s]
 51%|█████     | 2560/5021 [00:13<00:12, 198.31it/s]
 50%|████▉     | 2499/5021 [00:12<00:12, 195.31it/s]
 51%|█████▏    | 2580/5021 [00:13<00:12, 198.52it/s]
 50%|█████     | 2519/5021 [00:12<00:12, 195.60it/s]
 52%|█████▏    | 2600/5021 [00:13<00:12, 198.44it/s]
 51%|█████     | 2539/5021 [00:12<00:12, 193.43it/s]
 52%|█████▏    | 2620/5021 [00:13<00:12, 198.07it/s]
 51%|█████     | 2559/5021 [00:13<00:12, 193.75it/s]
 53%|█████▎    | 2640/5021 [00:13<00:12, 197.80it/s]
 51%|█████▏    | 2579/5021 [00:13<00:12, 194.17it/s]
 53%|█████▎    | 2660/5021 [00:13<00:11, 197.54it/s]
 52%|█████▏    | 2599/5021 [00:13<00:12, 194.79it/s]
 53%|█████▎    | 2680/5021 [00:13<00:11, 197.21it/s]
 52%|█████▏    | 2619/5021 [00:13<00:12, 194.89it/s]
 54%|█████▍    | 2700/5021 [00:13<00:11, 197.52it/s]
 53%|█████▎    | 2639/5021 [00:13<00:12, 195.22it/s]
 54%|█████▍    | 2720/5021 [00:13<00:11, 196.99it/s]
 53%|█████▎    | 2659/5021 [00:13<00:12, 195.34it/s]
 55%|█████▍    | 2740/5021 [00:13<00:11, 197.06it/s]
 53%|█████▎    | 2679/5021 [00:13<00:11, 195.57it/s]
 55%|█████▍    | 2760/5021 [00:14<00:11, 196.64it/s]
 54%|█████▍    | 2699/5021 [00:13<00:11, 196.25it/s]
 55%|█████▌    | 2780/5021 [00:14<00:11, 196.78it/s]
 54%|█████▍    | 2719/5021 [00:13<00:11, 196.18it/s]
 56%|█████▌    | 2800/5021 [00:14<00:11, 197.01it/s]
 55%|█████▍    | 2739/5021 [00:13<00:11, 196.49it/s]
 56%|█████▌    | 2820/5021 [00:14<00:11, 196.93it/s]
 55%|█████▍    | 2759/5021 [00:14<00:11, 196.85it/s]
 57%|█████▋    | 2840/5021 [00:14<00:11, 197.52it/s]
 55%|█████▌    | 2779/5021 [00:14<00:11, 197.36it/s]
 57%|█████▋    | 2860/5021 [00:14<00:10, 197.43it/s]
 56%|█████▌    | 2799/5021 [00:14<00:11, 197.04it/s]
 57%|█████▋    | 2880/5021 [00:14<00:10, 197.97it/s]
 56%|█████▌    | 2819/5021 [00:14<00:11, 196.75it/s]
 58%|█████▊    | 2900/5021 [00:14<00:10, 198.41it/s]
 57%|█████▋    | 2839/5021 [00:14<00:11, 196.57it/s]
 58%|█████▊    | 2920/5021 [00:14<00:10, 198.62it/s]
 57%|█████▋    | 2859/5021 [00:14<00:11, 196.53it/s]
 59%|█████▊    | 2941/5021 [00:14<00:10, 198.99it/s]
 57%|█████▋    | 2879/5021 [00:14<00:10, 196.69it/s]
 59%|█████▉    | 2961/5021 [00:15<00:10, 199.06it/s]
 58%|█████▊    | 2899/5021 [00:14<00:10, 196.88it/s]
 59%|█████▉    | 2981/5021 [00:15<00:10, 198.67it/s]
 58%|█████▊    | 2919/5021 [00:14<00:10, 196.82it/s]
 60%|█████▉    | 3001/5021 [00:15<00:10, 198.36it/s]
 59%|█████▊    | 2939/5021 [00:14<00:10, 196.59it/s]
 60%|██████    | 3021/5021 [00:15<00:10, 198.26it/s]
 59%|█████▉    | 2959/5021 [00:15<00:10, 196.40it/s]
 61%|██████    | 3041/5021 [00:15<00:09, 198.58it/s]
 59%|█████▉    | 2979/5021 [00:15<00:10, 196.20it/s]
 61%|██████    | 3061/5021 [00:15<00:09, 198.46it/s]
 60%|█████▉    | 2999/5021 [00:15<00:10, 196.32it/s]
 61%|██████▏   | 3081/5021 [00:15<00:09, 198.53it/s]
 60%|██████    | 3019/5021 [00:15<00:10, 196.51it/s]
 62%|██████▏   | 3101/5021 [00:15<00:09, 198.27it/s]
 61%|██████    | 3039/5021 [00:15<00:10, 196.73it/s]
 62%|██████▏   | 3121/5021 [00:15<00:09, 198.39it/s]
 61%|██████    | 3059/5021 [00:15<00:09, 196.65it/s]
 63%|██████▎   | 3142/5021 [00:15<00:09, 198.84it/s]
 61%|██████▏   | 3079/5021 [00:15<00:09, 196.88it/s]
 63%|██████▎   | 3162/5021 [00:16<00:09, 198.54it/s]
 62%|██████▏   | 3099/5021 [00:15<00:09, 196.97it/s]
 63%|██████▎   | 3182/5021 [00:16<00:09, 198.21it/s]
 62%|██████▏   | 3119/5021 [00:15<00:09, 197.53it/s]
 64%|██████▍   | 3202/5021 [00:16<00:09, 198.32it/s]
 63%|██████▎   | 3139/5021 [00:15<00:09, 197.49it/s]
 64%|██████▍   | 3222/5021 [00:16<00:09, 198.42it/s]
 63%|██████▎   | 3159/5021 [00:16<00:09, 197.96it/s]
 65%|██████▍   | 3242/5021 [00:16<00:08, 198.83it/s]
 63%|██████▎   | 3179/5021 [00:16<00:09, 196.85it/s]
 65%|██████▍   | 3262/5021 [00:16<00:08, 198.98it/s]
 64%|██████▎   | 3199/5021 [00:16<00:09, 196.97it/s]
 65%|██████▌   | 3283/5021 [00:16<00:08, 199.31it/s]
 64%|██████▍   | 3219/5021 [00:16<00:09, 196.91it/s]
 66%|██████▌   | 3303/5021 [00:16<00:08, 199.20it/s]
 65%|██████▍   | 3239/5021 [00:16<00:09, 196.93it/s]
 66%|██████▌   | 3323/5021 [00:16<00:08, 195.82it/s]
 65%|██████▍   | 3259/5021 [00:16<00:08, 196.63it/s]
 67%|██████▋   | 3343/5021 [00:16<00:08, 196.73it/s]
 65%|██████▌   | 3279/5021 [00:16<00:08, 196.58it/s]
 67%|██████▋   | 3363/5021 [00:17<00:08, 197.67it/s]
 66%|██████▌   | 3299/5021 [00:16<00:08, 196.53it/s]
 67%|██████▋   | 3383/5021 [00:17<00:08, 198.19it/s]
 66%|██████▌   | 3319/5021 [00:16<00:08, 196.37it/s]
 68%|██████▊   | 3403/5021 [00:17<00:08, 198.49it/s]
 67%|██████▋   | 3339/5021 [00:17<00:08, 196.73it/s]
 68%|██████▊   | 3423/5021 [00:17<00:08, 197.89it/s]
 67%|██████▋   | 3359/5021 [00:17<00:08, 196.70it/s]
 69%|██████▊   | 3443/5021 [00:17<00:08, 196.80it/s]
 67%|██████▋   | 3379/5021 [00:17<00:08, 196.51it/s]
 69%|██████▉   | 3463/5021 [00:17<00:07, 195.49it/s]
 68%|██████▊   | 3399/5021 [00:17<00:08, 196.43it/s]
 69%|██████▉   | 3483/5021 [00:17<00:07, 195.66it/s]
 68%|██████▊   | 3419/5021 [00:17<00:08, 196.55it/s]
 70%|██████▉   | 3503/5021 [00:17<00:07, 196.06it/s]
 68%|██████▊   | 3439/5021 [00:17<00:08, 196.74it/s]
 70%|███████   | 3523/5021 [00:17<00:07, 196.57it/s]
 69%|██████▉   | 3459/5021 [00:17<00:07, 196.99it/s]
 71%|███████   | 3543/5021 [00:18<00:07, 196.41it/s]
 69%|██████▉   | 3479/5021 [00:17<00:07, 196.92it/s]
 71%|███████   | 3563/5021 [00:18<00:07, 196.25it/s]
 70%|██████▉   | 3499/5021 [00:17<00:07, 196.97it/s]
 71%|███████▏  | 3583/5021 [00:18<00:07, 195.85it/s]
 70%|███████   | 3519/5021 [00:17<00:07, 197.22it/s]
 72%|███████▏  | 3603/5021 [00:18<00:07, 196.50it/s]
 70%|███████   | 3539/5021 [00:18<00:07, 197.09it/s]
 72%|███████▏  | 3623/5021 [00:18<00:07, 197.21it/s]
 71%|███████   | 3559/5021 [00:18<00:07, 197.23it/s]
 73%|███████▎  | 3643/5021 [00:18<00:06, 198.02it/s]
 71%|███████▏  | 3579/5021 [00:18<00:07, 197.63it/s]
 73%|███████▎  | 3663/5021 [00:18<00:06, 198.48it/s]
 72%|███████▏  | 3599/5021 [00:18<00:07, 197.69it/s]
 73%|███████▎  | 3683/5021 [00:18<00:06, 198.71it/s]
 72%|███████▏  | 3619/5021 [00:18<00:07, 197.69it/s]
 74%|███████▍  | 3703/5021 [00:18<00:06, 198.93it/s]
 72%|███████▏  | 3639/5021 [00:18<00:06, 197.69it/s]
 74%|███████▍  | 3723/5021 [00:18<00:06, 198.09it/s]
 73%|███████▎  | 3659/5021 [00:18<00:06, 197.79it/s]
 75%|███████▍  | 3743/5021 [00:19<00:06, 198.44it/s]
 73%|███████▎  | 3679/5021 [00:18<00:06, 197.31it/s]
 75%|███████▍  | 3763/5021 [00:19<00:06, 198.54it/s]
 74%|███████▎  | 3699/5021 [00:18<00:06, 196.92it/s]
 75%|███████▌  | 3783/5021 [00:19<00:06, 198.62it/s]
 74%|███████▍  | 3719/5021 [00:18<00:06, 196.43it/s]
 76%|███████▌  | 3803/5021 [00:19<00:06, 197.88it/s]
 74%|███████▍  | 3739/5021 [00:19<00:06, 196.32it/s]
 76%|███████▌  | 3823/5021 [00:19<00:06, 197.62it/s]
 75%|███████▍  | 3759/5021 [00:19<00:06, 196.17it/s]
 77%|███████▋  | 3843/5021 [00:19<00:05, 197.65it/s]
 75%|███████▌  | 3779/5021 [00:19<00:06, 196.44it/s]
 77%|███████▋  | 3863/5021 [00:19<00:05, 198.11it/s]
 76%|███████▌  | 3799/5021 [00:19<00:06, 196.73it/s]
 77%|███████▋  | 3883/5021 [00:19<00:05, 197.70it/s]
 76%|███████▌  | 3819/5021 [00:19<00:06, 196.49it/s]
 78%|███████▊  | 3903/5021 [00:19<00:05, 197.78it/s]
 76%|███████▋  | 3839/5021 [00:19<00:06, 196.37it/s]
 78%|███████▊  | 3923/5021 [00:19<00:05, 197.18it/s]
 77%|███████▋  | 3859/5021 [00:19<00:05, 196.27it/s]
 79%|███████▊  | 3943/5021 [00:20<00:05, 197.01it/s]
 77%|███████▋  | 3879/5021 [00:19<00:05, 195.54it/s]
 79%|███████▉  | 3963/5021 [00:20<00:05, 197.31it/s]
 78%|███████▊  | 3899/5021 [00:19<00:05, 196.44it/s]
 79%|███████▉  | 3983/5021 [00:20<00:05, 197.64it/s]
 78%|███████▊  | 3919/5021 [00:19<00:05, 197.16it/s]
 80%|███████▉  | 4003/5021 [00:20<00:05, 197.47it/s]
 78%|███████▊  | 3939/5021 [00:20<00:05, 197.74it/s]
 80%|████████  | 4023/5021 [00:20<00:05, 197.50it/s]
 79%|███████▉  | 3959/5021 [00:20<00:05, 198.35it/s]
 81%|████████  | 4043/5021 [00:20<00:04, 198.13it/s]
 79%|███████▉  | 3979/5021 [00:20<00:05, 198.56it/s]
 81%|████████  | 4064/5021 [00:20<00:04, 198.92it/s]
 80%|███████▉  | 4000/5021 [00:20<00:05, 199.21it/s]
 81%|████████▏ | 4084/5021 [00:20<00:04, 198.98it/s]
 80%|████████  | 4020/5021 [00:20<00:05, 199.29it/s]
 82%|████████▏ | 4104/5021 [00:20<00:04, 198.76it/s]
 80%|████████  | 4040/5021 [00:20<00:04, 198.59it/s]
 82%|████████▏ | 4125/5021 [00:20<00:04, 199.50it/s]
 81%|████████  | 4060/5021 [00:20<00:04, 198.63it/s]
 83%|████████▎ | 4146/5021 [00:21<00:04, 199.59it/s]
 81%|████████▏ | 4080/5021 [00:20<00:04, 198.59it/s]
 83%|████████▎ | 4166/5021 [00:21<00:04, 199.48it/s]
 82%|████████▏ | 4100/5021 [00:20<00:04, 198.45it/s]
 83%|████████▎ | 4186/5021 [00:21<00:04, 198.63it/s]
 82%|████████▏ | 4120/5021 [00:20<00:04, 198.31it/s]
 84%|████████▍ | 4206/5021 [00:21<00:04, 198.72it/s]
 82%|████████▏ | 4140/5021 [00:21<00:04, 198.21it/s]
 84%|████████▍ | 4226/5021 [00:21<00:04, 198.37it/s]
 83%|████████▎ | 4160/5021 [00:21<00:04, 194.27it/s]
 85%|████████▍ | 4246/5021 [00:21<00:03, 194.09it/s]
 83%|████████▎ | 4180/5021 [00:21<00:04, 194.18it/s]
 85%|████████▍ | 4266/5021 [00:21<00:03, 195.17it/s]
 84%|████████▎ | 4200/5021 [00:21<00:04, 195.01it/s]
 85%|████████▌ | 4286/5021 [00:21<00:03, 195.77it/s]
 84%|████████▍ | 4220/5021 [00:21<00:04, 195.45it/s]
 86%|████████▌ | 4306/5021 [00:21<00:03, 196.37it/s]
 84%|████████▍ | 4240/5021 [00:21<00:03, 195.65it/s]
 86%|████████▌ | 4326/5021 [00:21<00:03, 196.74it/s]
 85%|████████▍ | 4260/5021 [00:21<00:03, 196.12it/s]
 87%|████████▋ | 4346/5021 [00:22<00:03, 197.46it/s]
 85%|████████▌ | 4280/5021 [00:21<00:03, 196.30it/s]
 87%|████████▋ | 4366/5021 [00:22<00:03, 197.80it/s]
 86%|████████▌ | 4300/5021 [00:21<00:03, 196.75it/s]
 87%|████████▋ | 4386/5021 [00:22<00:03, 197.36it/s]
 86%|████████▌ | 4320/5021 [00:21<00:03, 196.83it/s]
 88%|████████▊ | 4406/5021 [00:22<00:03, 197.62it/s]
 86%|████████▋ | 4340/5021 [00:22<00:03, 197.30it/s]
 88%|████████▊ | 4426/5021 [00:22<00:03, 197.89it/s]
 87%|████████▋ | 4360/5021 [00:22<00:03, 197.47it/s]
 89%|████████▊ | 4446/5021 [00:22<00:02, 198.38it/s]
 87%|████████▋ | 4380/5021 [00:22<00:03, 197.82it/s]
 89%|████████▉ | 4466/5021 [00:22<00:02, 198.53it/s]
 88%|████████▊ | 4400/5021 [00:22<00:03, 198.04it/s]
 89%|████████▉ | 4486/5021 [00:22<00:02, 198.83it/s]
 88%|████████▊ | 4420/5021 [00:22<00:03, 198.05it/s]
 90%|████████▉ | 4506/5021 [00:22<00:02, 198.88it/s]
 88%|████████▊ | 4440/5021 [00:22<00:02, 198.21it/s]
 90%|█████████ | 4526/5021 [00:22<00:02, 198.28it/s]
 89%|████████▉ | 4460/5021 [00:22<00:02, 196.28it/s]
 91%|█████████ | 4546/5021 [00:23<00:02, 198.70it/s]
 89%|████████▉ | 4480/5021 [00:22<00:02, 195.45it/s]
 91%|█████████ | 4566/5021 [00:23<00:02, 199.02it/s]
 90%|████████▉ | 4500/5021 [00:22<00:02, 195.40it/s]
 91%|█████████▏| 4586/5021 [00:23<00:02, 199.16it/s]
 90%|█████████ | 4520/5021 [00:23<00:02, 194.89it/s]
 92%|█████████▏| 4606/5021 [00:23<00:02, 199.00it/s]
 90%|█████████ | 4540/5021 [00:23<00:02, 194.99it/s]
 92%|█████████▏| 4626/5021 [00:23<00:01, 199.02it/s]
 91%|█████████ | 4560/5021 [00:23<00:02, 195.07it/s]
 93%|█████████▎| 4646/5021 [00:23<00:01, 198.67it/s]
 91%|█████████ | 4580/5021 [00:23<00:02, 194.71it/s]
 93%|█████████▎| 4666/5021 [00:23<00:01, 198.34it/s]
 92%|█████████▏| 4600/5021 [00:23<00:02, 194.90it/s]
 93%|█████████▎| 4686/5021 [00:23<00:01, 198.21it/s]
 92%|█████████▏| 4620/5021 [00:23<00:02, 194.86it/s]
 94%|█████████▎| 4706/5021 [00:23<00:01, 197.76it/s]
 92%|█████████▏| 4640/5021 [00:23<00:01, 194.66it/s]
 94%|█████████▍| 4726/5021 [00:23<00:01, 197.96it/s]
 93%|█████████▎| 4660/5021 [00:23<00:01, 195.00it/s]
 95%|█████████▍| 4746/5021 [00:24<00:01, 196.83it/s]
 93%|█████████▎| 4680/5021 [00:23<00:01, 195.57it/s]
 95%|█████████▍| 4766/5021 [00:24<00:01, 196.67it/s]
 94%|█████████▎| 4700/5021 [00:23<00:01, 196.11it/s]
 95%|█████████▌| 4786/5021 [00:24<00:01, 196.51it/s]
 94%|█████████▍| 4720/5021 [00:24<00:01, 196.25it/s]
 96%|█████████▌| 4806/5021 [00:24<00:01, 196.43it/s]
 94%|█████████▍| 4740/5021 [00:24<00:01, 196.39it/s]
 96%|█████████▌| 4826/5021 [00:24<00:00, 196.43it/s]
 95%|█████████▍| 4760/5021 [00:24<00:01, 196.86it/s]
 97%|█████████▋| 4846/5021 [00:24<00:00, 196.94it/s]
 95%|█████████▌| 4780/5021 [00:24<00:01, 197.18it/s]
 97%|█████████▋| 4866/5021 [00:24<00:00, 197.43it/s]
 96%|█████████▌| 4800/5021 [00:24<00:01, 197.19it/s]
 97%|█████████▋| 4886/5021 [00:24<00:00, 198.04it/s]
 96%|█████████▌| 4820/5021 [00:24<00:01, 197.24it/s]
 98%|█████████▊| 4906/5021 [00:24<00:00, 197.88it/s]
 96%|█████████▋| 4840/5021 [00:24<00:00, 197.23it/s]
 98%|█████████▊| 4926/5021 [00:24<00:00, 198.20it/s]
 97%|█████████▋| 4860/5021 [00:24<00:00, 197.14it/s]
 99%|█████████▊| 4946/5021 [00:25<00:00, 198.22it/s]
 97%|█████████▋| 4880/5021 [00:24<00:00, 197.01it/s]
 99%|█████████▉| 4966/5021 [00:25<00:00, 198.33it/s]
 98%|█████████▊| 4900/5021 [00:24<00:00, 196.67it/s]
 99%|█████████▉| 4986/5021 [00:25<00:00, 197.88it/s]
 98%|█████████▊| 4920/5021 [00:25<00:00, 196.52it/s]
100%|█████████▉| 5006/5021 [00:25<00:00, 197.51it/s]
 98%|█████████▊| 4940/5021 [00:25<00:00, 196.24it/s]
100%|██████████| 5021/5021 [00:25<00:00, 197.07it/s]
n136-128-154:1708728:1708728 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1708728:1708728 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1708728:1708728 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:1708728:1708728 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:1708728:1708728 [0] NCCL INFO Comm config Blocking set to 1

 99%|█████████▉| 4960/5021 [00:25<00:00, 195.40it/s]
 99%|█████████▉| 4980/5021 [00:25<00:00, 195.50it/s]n136-128-154:1708728:1708987 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:1708728:1708987 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:1708728:1708987 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1708728:1708987 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1708728:1708987 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:1708728:1708987 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:1708728:1708987 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:1708728:1708987 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:1708728:1708987 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:1708728:1708987 [0] NCCL INFO Using network IB

100%|█████████▉| 5000/5021 [00:25<00:00, 195.33it/s]n136-128-154:1708728:1708987 [0] NCCL INFO ncclCommInitRankConfig comm 0xf855840 rank 0 nranks 2 cudaDev 0 nvmlDev 1 busId 16000 commId 0x8e41aae0c4bf49e3 - Init START

100%|█████████▉| 5020/5021 [00:25<00:00, 195.61it/s]
100%|██████████| 5021/5021 [00:25<00:00, 196.36it/s]
n136-128-154:1708729:1708729 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:1708729:1708729 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1708729:1708729 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1708729:1708729 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:1708729:1708729 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:1708729:1708992 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:1708729:1708992 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:1708729:1708992 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1708729:1708992 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1708729:1708992 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:1708729:1708992 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:1708729:1708992 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:1708729:1708992 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:1708729:1708992 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:1708729:1708992 [1] NCCL INFO Using network IB
n136-128-154:1708729:1708992 [1] NCCL INFO ncclCommInitRankConfig comm 0xfc75670 rank 1 nranks 2 cudaDev 1 nvmlDev 2 busId 4a000 commId 0x8e41aae0c4bf49e3 - Init START
n136-128-154:1708729:1708992 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:1708728:1708987 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:1708729:1708992 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:1708729:1708992 [1] NCCL INFO Retrieving state for IB
n136-128-154:1708729:1708992 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:1708729:1708992 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:1708728:1708987 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:1708728:1708987 [0] NCCL INFO Retrieving state for IB
n136-128-154:1708728:1708987 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:1708729:1708992 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:1708728:1708987 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:1708729:1708992 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:1708728:1708987 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:1708729:1708992 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:1708728:1708987 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:1708728:1708987 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:1708728:1708987 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1708728:1708987 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1708728:1708987 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1708728:1708987 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1708729:1708992 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1708729:1708992 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1708729:1708992 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1708729:1708992 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1708728:1708987 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:1708728:1708987 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:1708728:1708987 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:1708728:1708987 [0] NCCL INFO               + PCI[24.0] - PCI/0-14000 (1000c01010de13b8)
n136-128-154:1708728:1708987 [0] NCCL INFO                             + PCI[24.0] - GPU/0-16000 (0)
n136-128-154:1708728:1708987 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1708728:1708987 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:1708728:1708987 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:1708728:1708987 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:1708728:1708987 [0] NCCL INFO               + PCI[24.0] - PCI/0-48000 (1000c01010de13b8)
n136-128-154:1708728:1708987 [0] NCCL INFO                             + PCI[24.0] - GPU/0-4a000 (1)
n136-128-154:1708728:1708987 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1708728:1708987 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:1708728:1708987 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:1708728:1708987 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:1708728:1708987 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:1708728:1708987 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:1708728:1708987 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:1708728:1708987 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:1708728:1708987 [0] NCCL INFO ==========================================
n136-128-154:1708728:1708987 [0] NCCL INFO GPU/0-16000 :GPU/0-16000 (0/5000.0/LOC) GPU/0-4a000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1708728:1708987 [0] NCCL INFO GPU/0-4a000 :GPU/0-16000 (2/240.0/NVL) GPU/0-4a000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1708728:1708987 [0] NCCL INFO Setting affinity for GPU 1 to 1-31,65-95
n136-128-154:1708729:1708992 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:1708729:1708992 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:1708729:1708992 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:1708729:1708992 [1] NCCL INFO               + PCI[24.0] - PCI/0-14000 (1000c01010de13b8)
n136-128-154:1708729:1708992 [1] NCCL INFO                             + PCI[24.0] - GPU/0-16000 (0)
n136-128-154:1708729:1708992 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1708729:1708992 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:1708729:1708992 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:1708729:1708992 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:1708729:1708992 [1] NCCL INFO               + PCI[24.0] - PCI/0-48000 (1000c01010de13b8)
n136-128-154:1708729:1708992 [1] NCCL INFO                             + PCI[24.0] - GPU/0-4a000 (1)
n136-128-154:1708729:1708992 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1708729:1708992 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:1708729:1708992 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:1708729:1708992 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:1708729:1708992 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:1708729:1708992 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:1708729:1708992 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:1708729:1708992 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:1708729:1708992 [1] NCCL INFO ==========================================
n136-128-154:1708729:1708992 [1] NCCL INFO GPU/0-16000 :GPU/0-16000 (0/5000.0/LOC) GPU/0-4a000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1708729:1708992 [1] NCCL INFO GPU/0-4a000 :GPU/0-16000 (2/240.0/NVL) GPU/0-4a000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1708729:1708992 [1] NCCL INFO Setting affinity for GPU 2 to 1-31,65-95
n136-128-154:1708728:1708987 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:1708728:1708987 [0] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708728:1708987 [0] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708728:1708987 [0] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708728:1708987 [0] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708728:1708987 [0] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708728:1708987 [0] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708728:1708987 [0] NCCL INFO  6 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708728:1708987 [0] NCCL INFO  7 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708728:1708987 [0] NCCL INFO  8 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708728:1708987 [0] NCCL INFO  9 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708728:1708987 [0] NCCL INFO 10 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708728:1708987 [0] NCCL INFO 11 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708729:1708992 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:1708729:1708992 [1] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708729:1708992 [1] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708729:1708992 [1] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708729:1708992 [1] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708729:1708992 [1] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708729:1708992 [1] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708729:1708992 [1] NCCL INFO  6 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708729:1708992 [1] NCCL INFO  7 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708729:1708992 [1] NCCL INFO  8 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708729:1708992 [1] NCCL INFO  9 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708729:1708992 [1] NCCL INFO 10 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708729:1708992 [1] NCCL INFO 11 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708728:1708987 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:1708728:1708987 [0] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708728:1708987 [0] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708728:1708987 [0] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708728:1708987 [0] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708728:1708987 [0] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708728:1708987 [0] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708728:1708987 [0] NCCL INFO  6 : GPU/0-4a000 GPU/0-16000
n136-128-154:1708728:1708987 [0] NCCL INFO  7 : GPU/0-4a000 GPU/0-16000
n136-128-154:1708728:1708987 [0] NCCL INFO  8 : GPU/0-4a000 GPU/0-16000
n136-128-154:1708728:1708987 [0] NCCL INFO  9 : GPU/0-4a000 GPU/0-16000
n136-128-154:1708728:1708987 [0] NCCL INFO 10 : GPU/0-4a000 GPU/0-16000
n136-128-154:1708728:1708987 [0] NCCL INFO 11 : GPU/0-4a000 GPU/0-16000
n136-128-154:1708729:1708992 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:1708729:1708992 [1] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708729:1708992 [1] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708729:1708992 [1] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708729:1708992 [1] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708729:1708992 [1] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708729:1708992 [1] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1708729:1708992 [1] NCCL INFO  6 : GPU/0-4a000 GPU/0-16000
n136-128-154:1708729:1708992 [1] NCCL INFO  7 : GPU/0-4a000 GPU/0-16000
n136-128-154:1708729:1708992 [1] NCCL INFO  8 : GPU/0-4a000 GPU/0-16000
n136-128-154:1708729:1708992 [1] NCCL INFO  9 : GPU/0-4a000 GPU/0-16000
n136-128-154:1708729:1708992 [1] NCCL INFO 10 : GPU/0-4a000 GPU/0-16000
n136-128-154:1708729:1708992 [1] NCCL INFO 11 : GPU/0-4a000 GPU/0-16000
n136-128-154:1708728:1708987 [0] NCCL INFO comm 0xf855840 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:1708729:1708992 [1] NCCL INFO comm 0xfc75670 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:1708729:1708992 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:1708728:1708987 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:1708729:1708992 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:1708728:1708987 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:1708729:1708992 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:1708728:1708987 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:1708729:1708992 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:1708728:1708987 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:1708729:1708992 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:1708728:1708987 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:1708729:1708992 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:1708728:1708987 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:1708729:1708992 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:1708728:1708987 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:1708729:1708992 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:1708728:1708987 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:1708729:1708992 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:1708728:1708987 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:1708729:1708992 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:1708728:1708987 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:1708729:1708992 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:1708728:1708987 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:1708729:1708992 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:1708728:1708987 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:1708729:1708992 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:1708728:1708987 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:1708729:1708992 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:1708728:1708987 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:1708729:1708992 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:1708728:1708987 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:1708729:1708992 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:1708728:1708987 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:1708729:1708992 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:1708728:1708987 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:1708729:1708992 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:1708728:1708987 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:1708729:1708992 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:1708728:1708987 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:1708729:1708992 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:1708728:1708987 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:1708729:1708992 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:1708728:1708987 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:1708729:1708992 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:1708728:1708987 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:1708729:1708992 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:1708728:1708987 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:1708729:1708992 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:1708728:1708987 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:1708728:1708987 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:1708728:1708987 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:1708729:1708992 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:1708728:1708987 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:1708729:1708992 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:1708728:1708987 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:1708729:1708992 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:1708728:1708987 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:1708729:1708992 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:1708728:1708987 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:1708729:1708992 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:1708728:1708987 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:1708729:1708992 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:1708728:1708987 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:1708729:1708992 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:1708728:1708987 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:1708729:1708992 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:1708728:1708987 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:1708729:1708992 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:1708728:1708987 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:1708729:1708992 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:1708728:1708987 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:1708729:1708992 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:1708728:1708987 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:1708729:1708992 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:1708728:1708987 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:1708729:1708992 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:1708728:1708987 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:1708729:1708992 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:1708728:1708987 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:1708729:1708992 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:1708728:1708987 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:1708729:1708992 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:1708728:1708987 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:1708729:1708992 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:1708728:1708987 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:1708729:1708992 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:1708728:1708987 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:1708729:1708992 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:1708728:1708987 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:1708729:1708992 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:1708728:1708987 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:1708729:1708992 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:1708728:1708987 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:1708729:1708992 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:1708728:1708987 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:1708729:1708992 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:1708729:1708992 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:1708728:1708987 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:1708729:1708992 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:1708728:1708987 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:1708729:1708992 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:1708728:1708987 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:1708728:1708987 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:1708728:1708987 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:1708728:1708987 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:1708728:1708987 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:1708728:1708987 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:1708728:1708987 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:1708728:1708987 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:1708728:1708987 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:1708728:1708987 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:1708728:1708987 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:1708728:1708987 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:1708728:1708987 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:1708728:1708987 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:1708728:1708987 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:1708728:1708987 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:1708728:1708987 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:1708728:1708987 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:1708728:1708987 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:1708728:1708987 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:1708728:1708987 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:1708728:1708987 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:1708728:1708987 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:1708728:1708987 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:1708729:1708992 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:1708729:1708999 [1] NCCL INFO [Proxy Service] Device 1 CPU core 29
n136-128-154:1708729:1709000 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 94
n136-128-154:1708728:1708987 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:1708728:1708987 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:1708728:1709001 [0] NCCL INFO [Proxy Service] Device 0 CPU core 10
n136-128-154:1708728:1709002 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 11
n136-128-154:1708729:1708992 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:1708729:1708992 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:1708728:1708987 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:1708728:1708987 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:1708728:1708987 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:1708728:1708987 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:1708728:1708987 [0] NCCL INFO ncclCommInitRankConfig comm 0xf855840 rank 0 nranks 2 cudaDev 0 nvmlDev 1 busId 16000 commId 0x8e41aae0c4bf49e3 - Init COMPLETE
n136-128-154:1708728:1708987 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.72 (kernels 0.19, alloc 0.12, bootstrap 0.32, allgathers 0.00, topo 0.05, graphs 0.00, connections 0.01, rest 0.02)
n136-128-154:1708729:1708992 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:1708729:1708992 [1] NCCL INFO ncclCommInitRankConfig comm 0xfc75670 rank 1 nranks 2 cudaDev 1 nvmlDev 2 busId 4a000 commId 0x8e41aae0c4bf49e3 - Init COMPLETE
n136-128-154:1708729:1708992 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.32 (kernels 0.12, alloc 0.11, bootstrap 0.00, allgathers 0.00, topo 0.05, graphs 0.00, connections 0.01, rest 0.02)
n136-128-154:1708728:1709003 [0] NCCL INFO Channel 00/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708729:1709004 [1] NCCL INFO Channel 00/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708728:1709003 [0] NCCL INFO Channel 01/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708729:1709004 [1] NCCL INFO Channel 01/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708728:1709003 [0] NCCL INFO Channel 02/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708729:1709004 [1] NCCL INFO Channel 02/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708728:1709003 [0] NCCL INFO Channel 03/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708729:1709004 [1] NCCL INFO Channel 03/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708728:1709003 [0] NCCL INFO Channel 04/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708729:1709004 [1] NCCL INFO Channel 04/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708728:1709003 [0] NCCL INFO Channel 05/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708729:1709004 [1] NCCL INFO Channel 05/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708728:1709003 [0] NCCL INFO Channel 06/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708729:1709004 [1] NCCL INFO Channel 06/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708728:1709003 [0] NCCL INFO Channel 07/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708729:1709004 [1] NCCL INFO Channel 07/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708728:1709003 [0] NCCL INFO Channel 08/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708729:1709004 [1] NCCL INFO Channel 08/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708728:1709003 [0] NCCL INFO Channel 09/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708729:1709004 [1] NCCL INFO Channel 09/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708728:1709003 [0] NCCL INFO Channel 10/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708729:1709004 [1] NCCL INFO Channel 10/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708728:1709003 [0] NCCL INFO Channel 11/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708729:1709004 [1] NCCL INFO Channel 11/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708728:1709003 [0] NCCL INFO Channel 12/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708729:1709004 [1] NCCL INFO Channel 12/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708728:1709003 [0] NCCL INFO Channel 13/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708729:1709004 [1] NCCL INFO Channel 13/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708728:1709003 [0] NCCL INFO Channel 14/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708729:1709004 [1] NCCL INFO Channel 14/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708728:1709003 [0] NCCL INFO Channel 15/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708729:1709004 [1] NCCL INFO Channel 15/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708728:1709003 [0] NCCL INFO Channel 16/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708729:1709004 [1] NCCL INFO Channel 16/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708728:1709003 [0] NCCL INFO Channel 17/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708729:1709004 [1] NCCL INFO Channel 17/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708728:1709003 [0] NCCL INFO Channel 18/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708729:1709004 [1] NCCL INFO Channel 18/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708728:1709003 [0] NCCL INFO Channel 19/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708729:1709004 [1] NCCL INFO Channel 19/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708728:1709003 [0] NCCL INFO Channel 20/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708729:1709004 [1] NCCL INFO Channel 20/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708728:1709003 [0] NCCL INFO Channel 21/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708729:1709004 [1] NCCL INFO Channel 21/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708728:1709003 [0] NCCL INFO Channel 22/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708729:1709004 [1] NCCL INFO Channel 22/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708728:1709003 [0] NCCL INFO Channel 23/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1708729:1709004 [1] NCCL INFO Channel 23/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708729:1709004 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:1708728:1709003 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-06:03:18:18 INFO     [evaluator:559] Running loglikelihood requests
2025-12-06:03:18:18 INFO     [evaluator:559] Running loglikelihood requests

Running loglikelihood requests:   0%|          | 0/20084 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 57
Determined largest batch size: 57

Running loglikelihood requests:   0%|          | 1/20084 [00:40<223:31:35, 40.07s/it]
Running loglikelihood requests:   0%|          | 58/20084 [00:45<3:11:46,  1.74it/s] 
Running loglikelihood requests:   1%|          | 115/20084 [00:49<1:35:30,  3.48it/s]
Running loglikelihood requests:   1%|          | 172/20084 [00:54<1:04:02,  5.18it/s]
Running loglikelihood requests:   1%|          | 229/20084 [00:59<49:05,  6.74it/s]  
Running loglikelihood requests:   1%|▏         | 286/20084 [01:03<40:44,  8.10it/s]
Running loglikelihood requests:   2%|▏         | 343/20084 [01:08<35:38,  9.23it/s]
Running loglikelihood requests:   2%|▏         | 400/20084 [01:12<32:19, 10.15it/s]
Running loglikelihood requests:   2%|▏         | 457/20084 [01:17<30:08, 10.85it/s]
Running loglikelihood requests:   3%|▎         | 514/20084 [01:21<28:35, 11.40it/s]
Running loglikelihood requests:   3%|▎         | 571/20084 [01:25<27:30, 11.82it/s]
Running loglikelihood requests:   3%|▎         | 628/20084 [01:30<26:40, 12.16it/s]
Running loglikelihood requests:   3%|▎         | 685/20084 [01:34<26:02, 12.41it/s]
Running loglikelihood requests:   4%|▎         | 742/20084 [01:38<25:34, 12.61it/s]
Running loglikelihood requests:   4%|▍         | 799/20084 [01:43<25:11, 12.75it/s]
Running loglikelihood requests:   4%|▍         | 856/20084 [01:47<24:54, 12.87it/s]
Running loglikelihood requests:   5%|▍         | 913/20084 [01:52<24:39, 12.96it/s]
Running loglikelihood requests:   5%|▍         | 970/20084 [01:56<24:27, 13.02it/s]
Running loglikelihood requests:   5%|▌         | 1027/20084 [02:00<24:16, 13.09it/s]
Running loglikelihood requests:   5%|▌         | 1084/20084 [02:04<24:05, 13.14it/s]
Running loglikelihood requests:   6%|▌         | 1141/20084 [02:09<23:56, 13.19it/s]
Running loglikelihood requests:   6%|▌         | 1198/20084 [02:13<23:47, 13.23it/s]
Running loglikelihood requests:   6%|▌         | 1255/20084 [02:17<23:39, 13.27it/s]
Running loglikelihood requests:   7%|▋         | 1312/20084 [02:22<23:32, 13.29it/s]
Running loglikelihood requests:   7%|▋         | 1369/20084 [02:26<23:25, 13.32it/s]
Running loglikelihood requests:   7%|▋         | 1426/20084 [02:30<23:17, 13.35it/s]
Running loglikelihood requests:   7%|▋         | 1483/20084 [02:34<23:10, 13.38it/s]
Running loglikelihood requests:   8%|▊         | 1540/20084 [02:39<23:03, 13.40it/s]
Running loglikelihood requests:   8%|▊         | 1597/20084 [02:43<22:56, 13.43it/s]
Running loglikelihood requests:   8%|▊         | 1654/20084 [02:47<22:49, 13.46it/s]
Running loglikelihood requests:   9%|▊         | 1711/20084 [02:51<22:43, 13.48it/s]
Running loglikelihood requests:   9%|▉         | 1768/20084 [02:55<22:37, 13.50it/s]
Running loglikelihood requests:   9%|▉         | 1825/20084 [03:00<22:30, 13.52it/s]
Running loglikelihood requests:   9%|▉         | 1882/20084 [03:04<22:24, 13.54it/s]
Running loglikelihood requests:  10%|▉         | 1939/20084 [03:08<22:18, 13.56it/s]
Running loglikelihood requests:  10%|▉         | 1996/20084 [03:12<22:12, 13.58it/s]
Running loglikelihood requests:  10%|█         | 2053/20084 [03:16<22:06, 13.59it/s]
Running loglikelihood requests:  11%|█         | 2110/20084 [03:20<22:00, 13.61it/s]
Running loglikelihood requests:  11%|█         | 2167/20084 [03:25<21:54, 13.63it/s]
Running loglikelihood requests:  11%|█         | 2224/20084 [03:29<21:48, 13.65it/s]
Running loglikelihood requests:  11%|█▏        | 2281/20084 [03:33<21:42, 13.67it/s]
Running loglikelihood requests:  12%|█▏        | 2338/20084 [03:37<21:36, 13.69it/s]
Running loglikelihood requests:  12%|█▏        | 2395/20084 [03:41<21:29, 13.71it/s]
Running loglikelihood requests:  12%|█▏        | 2452/20084 [03:45<21:23, 13.74it/s]
Running loglikelihood requests:  12%|█▏        | 2509/20084 [03:50<21:17, 13.75it/s]
Running loglikelihood requests:  13%|█▎        | 2566/20084 [03:54<21:12, 13.77it/s]
Running loglikelihood requests:  13%|█▎        | 2623/20084 [03:58<21:07, 13.78it/s]
Running loglikelihood requests:  13%|█▎        | 2680/20084 [04:02<21:01, 13.79it/s]
Running loglikelihood requests:  14%|█▎        | 2737/20084 [04:06<20:55, 13.82it/s]
Running loglikelihood requests:  14%|█▍        | 2794/20084 [04:10<20:49, 13.83it/s]
Running loglikelihood requests:  14%|█▍        | 2851/20084 [04:14<20:44, 13.85it/s]
Running loglikelihood requests:  14%|█▍        | 2908/20084 [04:18<20:40, 13.85it/s]
Running loglikelihood requests:  15%|█▍        | 2965/20084 [04:22<20:34, 13.87it/s]
Running loglikelihood requests:  15%|█▌        | 3022/20084 [04:27<20:28, 13.88it/s]
Running loglikelihood requests:  15%|█▌        | 3079/20084 [04:31<20:22, 13.91it/s]
Running loglikelihood requests:  16%|█▌        | 3136/20084 [04:35<20:16, 13.93it/s]
Running loglikelihood requests:  16%|█▌        | 3193/20084 [04:39<20:11, 13.95it/s]
Running loglikelihood requests:  16%|█▌        | 3250/20084 [04:43<20:05, 13.96it/s]
Running loglikelihood requests:  16%|█▋        | 3307/20084 [04:47<19:59, 13.99it/s]
Running loglikelihood requests:  17%|█▋        | 3364/20084 [04:51<19:55, 13.99it/s]
Running loglikelihood requests:  17%|█▋        | 3421/20084 [04:55<19:49, 14.01it/s]
Running loglikelihood requests:  17%|█▋        | 3478/20084 [04:59<19:43, 14.03it/s]
Running loglikelihood requests:  18%|█▊        | 3535/20084 [05:03<19:38, 14.05it/s]
Running loglikelihood requests:  18%|█▊        | 3592/20084 [05:07<19:33, 14.06it/s]
Running loglikelihood requests:  18%|█▊        | 3649/20084 [05:11<19:27, 14.08it/s]
Running loglikelihood requests:  18%|█▊        | 3706/20084 [05:15<19:22, 14.09it/s]
Running loglikelihood requests:  19%|█▊        | 3763/20084 [05:19<19:17, 14.11it/s]
Running loglikelihood requests:  19%|█▉        | 3820/20084 [05:23<19:11, 14.12it/s]
Running loglikelihood requests:  19%|█▉        | 3877/20084 [05:27<19:06, 14.13it/s]
Running loglikelihood requests:  20%|█▉        | 3934/20084 [05:31<19:01, 14.15it/s]
Running loglikelihood requests:  20%|█▉        | 3991/20084 [05:35<18:56, 14.16it/s]
Running loglikelihood requests:  20%|██        | 4048/20084 [05:39<18:51, 14.17it/s]
Running loglikelihood requests:  20%|██        | 4105/20084 [05:43<18:46, 14.19it/s]
Running loglikelihood requests:  21%|██        | 4162/20084 [05:47<18:41, 14.20it/s]
Running loglikelihood requests:  21%|██        | 4219/20084 [05:51<18:36, 14.20it/s]
Running loglikelihood requests:  21%|██▏       | 4276/20084 [05:55<18:32, 14.21it/s]
Running loglikelihood requests:  22%|██▏       | 4333/20084 [05:59<18:27, 14.22it/s]
Running loglikelihood requests:  22%|██▏       | 4390/20084 [06:03<18:22, 14.24it/s]
Running loglikelihood requests:  22%|██▏       | 4447/20084 [06:07<18:17, 14.25it/s]
Running loglikelihood requests:  22%|██▏       | 4504/20084 [06:11<18:11, 14.27it/s]
Running loglikelihood requests:  23%|██▎       | 4561/20084 [06:15<18:06, 14.29it/s]
Running loglikelihood requests:  23%|██▎       | 4618/20084 [06:19<17:57, 14.35it/s]
Running loglikelihood requests:  23%|██▎       | 4675/20084 [06:23<17:50, 14.39it/s]
Running loglikelihood requests:  24%|██▎       | 4732/20084 [06:27<17:43, 14.43it/s]
Running loglikelihood requests:  24%|██▍       | 4789/20084 [06:31<17:37, 14.46it/s]
Running loglikelihood requests:  24%|██▍       | 4846/20084 [06:35<17:31, 14.49it/s]
Running loglikelihood requests:  24%|██▍       | 4903/20084 [06:39<17:28, 14.47it/s]
Running loglikelihood requests:  25%|██▍       | 4960/20084 [06:43<17:26, 14.46it/s]
Running loglikelihood requests:  25%|██▍       | 5017/20084 [06:47<17:22, 14.45it/s]
Running loglikelihood requests:  25%|██▌       | 5074/20084 [06:51<17:17, 14.46it/s]
Running loglikelihood requests:  26%|██▌       | 5131/20084 [06:55<17:13, 14.47it/s]
Running loglikelihood requests:  26%|██▌       | 5188/20084 [06:59<17:08, 14.48it/s]
Running loglikelihood requests:  26%|██▌       | 5245/20084 [07:03<17:03, 14.49it/s]
Running loglikelihood requests:  26%|██▋       | 5302/20084 [07:06<16:59, 14.50it/s]
Running loglikelihood requests:  27%|██▋       | 5359/20084 [07:10<16:54, 14.52it/s]
Running loglikelihood requests:  27%|██▋       | 5416/20084 [07:14<16:50, 14.51it/s]
Running loglikelihood requests:  27%|██▋       | 5473/20084 [07:18<16:45, 14.52it/s]
Running loglikelihood requests:  28%|██▊       | 5530/20084 [07:22<16:41, 14.53it/s]
Running loglikelihood requests:  28%|██▊       | 5587/20084 [07:26<16:36, 14.55it/s]
Running loglikelihood requests:  28%|██▊       | 5644/20084 [07:30<16:31, 14.57it/s]
Running loglikelihood requests:  28%|██▊       | 5701/20084 [07:34<16:26, 14.57it/s]
Running loglikelihood requests:  29%|██▊       | 5758/20084 [07:38<16:22, 14.57it/s]
Running loglikelihood requests:  29%|██▉       | 5815/20084 [07:42<16:14, 14.64it/s]
Running loglikelihood requests:  29%|██▉       | 5872/20084 [07:46<16:08, 14.68it/s]
Running loglikelihood requests:  30%|██▉       | 5929/20084 [07:49<16:02, 14.70it/s]
Running loglikelihood requests:  30%|██▉       | 5986/20084 [07:53<15:57, 14.72it/s]
Running loglikelihood requests:  30%|███       | 6043/20084 [07:57<15:53, 14.73it/s]
Running loglikelihood requests:  30%|███       | 6100/20084 [08:01<15:48, 14.74it/s]
Running loglikelihood requests:  31%|███       | 6157/20084 [08:05<15:44, 14.74it/s]
Running loglikelihood requests:  31%|███       | 6214/20084 [08:09<15:40, 14.75it/s]
Running loglikelihood requests:  31%|███       | 6271/20084 [08:13<15:36, 14.75it/s]
Running loglikelihood requests:  32%|███▏      | 6328/20084 [08:16<15:32, 14.76it/s]
Running loglikelihood requests:  32%|███▏      | 6385/20084 [08:20<15:27, 14.77it/s]
Running loglikelihood requests:  32%|███▏      | 6442/20084 [08:24<15:23, 14.77it/s]
Running loglikelihood requests:  32%|███▏      | 6499/20084 [08:28<15:18, 14.79it/s]
Running loglikelihood requests:  33%|███▎      | 6556/20084 [08:32<15:14, 14.80it/s]
Running loglikelihood requests:  33%|███▎      | 6613/20084 [08:36<15:09, 14.80it/s]
Running loglikelihood requests:  33%|███▎      | 6670/20084 [08:39<15:05, 14.81it/s]
Running loglikelihood requests:  33%|███▎      | 6727/20084 [08:43<15:00, 14.83it/s]
Running loglikelihood requests:  34%|███▍      | 6784/20084 [08:47<14:56, 14.84it/s]
Running loglikelihood requests:  34%|███▍      | 6841/20084 [08:51<14:51, 14.85it/s]
Running loglikelihood requests:  34%|███▍      | 6898/20084 [08:55<14:46, 14.87it/s]
Running loglikelihood requests:  35%|███▍      | 6955/20084 [08:59<14:42, 14.87it/s]
Running loglikelihood requests:  35%|███▍      | 7012/20084 [09:02<14:38, 14.88it/s]
Running loglikelihood requests:  35%|███▌      | 7069/20084 [09:06<14:34, 14.89it/s]
Running loglikelihood requests:  35%|███▌      | 7126/20084 [09:10<14:26, 14.95it/s]
Running loglikelihood requests:  36%|███▌      | 7183/20084 [09:14<14:20, 14.99it/s]
Running loglikelihood requests:  36%|███▌      | 7240/20084 [09:18<14:15, 15.02it/s]
Running loglikelihood requests:  36%|███▋      | 7297/20084 [09:21<14:11, 15.01it/s]
Running loglikelihood requests:  37%|███▋      | 7354/20084 [09:25<14:08, 15.00it/s]
Running loglikelihood requests:  37%|███▋      | 7411/20084 [09:29<14:04, 15.00it/s]
Running loglikelihood requests:  37%|███▋      | 7468/20084 [09:33<14:01, 15.00it/s]
Running loglikelihood requests:  37%|███▋      | 7525/20084 [09:37<13:56, 15.01it/s]
Running loglikelihood requests:  38%|███▊      | 7582/20084 [09:40<13:53, 15.01it/s]
Running loglikelihood requests:  38%|███▊      | 7639/20084 [09:44<13:48, 15.02it/s]
Running loglikelihood requests:  38%|███▊      | 7696/20084 [09:48<13:44, 15.03it/s]
Running loglikelihood requests:  39%|███▊      | 7753/20084 [09:52<13:39, 15.05it/s]
Running loglikelihood requests:  39%|███▉      | 7810/20084 [09:56<13:35, 15.06it/s]
Running loglikelihood requests:  39%|███▉      | 7867/20084 [09:59<13:30, 15.08it/s]
Running loglikelihood requests:  39%|███▉      | 7924/20084 [10:03<13:26, 15.08it/s]
Running loglikelihood requests:  40%|███▉      | 7981/20084 [10:07<13:22, 15.09it/s]
Running loglikelihood requests:  40%|████      | 8038/20084 [10:11<13:17, 15.10it/s]
Running loglikelihood requests:  40%|████      | 8095/20084 [10:14<13:13, 15.12it/s]
Running loglikelihood requests:  41%|████      | 8152/20084 [10:18<13:08, 15.13it/s]
Running loglikelihood requests:  41%|████      | 8209/20084 [10:22<13:04, 15.15it/s]
Running loglikelihood requests:  41%|████      | 8266/20084 [10:26<12:59, 15.15it/s]
Running loglikelihood requests:  41%|████▏     | 8323/20084 [10:29<12:55, 15.17it/s]
Running loglikelihood requests:  42%|████▏     | 8380/20084 [10:33<12:51, 15.18it/s]
Running loglikelihood requests:  42%|████▏     | 8437/20084 [10:37<12:47, 15.18it/s]
Running loglikelihood requests:  42%|████▏     | 8494/20084 [10:41<12:43, 15.19it/s]
Running loglikelihood requests:  43%|████▎     | 8551/20084 [10:44<12:36, 15.25it/s]
Running loglikelihood requests:  43%|████▎     | 8608/20084 [10:48<12:30, 15.29it/s]
Running loglikelihood requests:  43%|████▎     | 8665/20084 [10:52<12:25, 15.32it/s]
Running loglikelihood requests:  43%|████▎     | 8722/20084 [10:56<12:22, 15.30it/s]
Running loglikelihood requests:  44%|████▎     | 8779/20084 [10:59<12:18, 15.30it/s]
Running loglikelihood requests:  44%|████▍     | 8836/20084 [11:03<12:15, 15.30it/s]
Running loglikelihood requests:  44%|████▍     | 8893/20084 [11:07<12:12, 15.28it/s]
Running loglikelihood requests:  45%|████▍     | 8950/20084 [11:10<12:08, 15.29it/s]
Running loglikelihood requests:  45%|████▍     | 9007/20084 [11:14<12:03, 15.30it/s]
Running loglikelihood requests:  45%|████▌     | 9064/20084 [11:18<11:59, 15.31it/s]
Running loglikelihood requests:  45%|████▌     | 9121/20084 [11:22<11:53, 15.37it/s]
Running loglikelihood requests:  46%|████▌     | 9178/20084 [11:25<11:48, 15.39it/s]
Running loglikelihood requests:  46%|████▌     | 9235/20084 [11:29<11:43, 15.42it/s]
Running loglikelihood requests:  46%|████▋     | 9292/20084 [11:33<11:40, 15.41it/s]
Running loglikelihood requests:  47%|████▋     | 9349/20084 [11:36<11:36, 15.41it/s]
Running loglikelihood requests:  47%|████▋     | 9406/20084 [11:40<11:32, 15.41it/s]
Running loglikelihood requests:  47%|████▋     | 9463/20084 [11:44<11:29, 15.41it/s]
Running loglikelihood requests:  47%|████▋     | 9520/20084 [11:47<11:24, 15.42it/s]
Running loglikelihood requests:  48%|████▊     | 9577/20084 [11:51<11:21, 15.43it/s]
Running loglikelihood requests:  48%|████▊     | 9634/20084 [11:55<11:17, 15.43it/s]
Running loglikelihood requests:  48%|████▊     | 9691/20084 [11:58<11:11, 15.49it/s]
Running loglikelihood requests:  49%|████▊     | 9748/20084 [12:02<11:05, 15.53it/s]
Running loglikelihood requests:  49%|████▉     | 9805/20084 [12:06<10:59, 15.57it/s]
Running loglikelihood requests:  49%|████▉     | 9862/20084 [12:09<10:55, 15.60it/s]
Running loglikelihood requests:  49%|████▉     | 9919/20084 [12:13<10:51, 15.61it/s]
Running loglikelihood requests:  50%|████▉     | 9976/20084 [12:17<10:48, 15.60it/s]
Running loglikelihood requests:  50%|████▉     | 10033/20084 [12:20<10:44, 15.59it/s]
Running loglikelihood requests:  50%|█████     | 10090/20084 [12:24<10:41, 15.58it/s]
Running loglikelihood requests:  51%|█████     | 10147/20084 [12:28<10:37, 15.58it/s]
Running loglikelihood requests:  51%|█████     | 10204/20084 [12:31<10:33, 15.58it/s]
Running loglikelihood requests:  51%|█████     | 10261/20084 [12:35<10:30, 15.59it/s]
Running loglikelihood requests:  51%|█████▏    | 10318/20084 [12:39<10:26, 15.59it/s]
Running loglikelihood requests:  52%|█████▏    | 10375/20084 [12:42<10:22, 15.59it/s]
Running loglikelihood requests:  52%|█████▏    | 10432/20084 [12:46<10:16, 15.65it/s]
Running loglikelihood requests:  52%|█████▏    | 10489/20084 [12:50<10:11, 15.69it/s]
Running loglikelihood requests:  53%|█████▎    | 10546/20084 [12:53<10:06, 15.71it/s]
Running loglikelihood requests:  53%|█████▎    | 10603/20084 [12:57<10:02, 15.74it/s]
Running loglikelihood requests:  53%|█████▎    | 10660/20084 [13:00<09:57, 15.76it/s]
Running loglikelihood requests:  53%|█████▎    | 10717/20084 [13:04<09:53, 15.78it/s]
Running loglikelihood requests:  54%|█████▎    | 10774/20084 [13:08<09:50, 15.78it/s]
Running loglikelihood requests:  54%|█████▍    | 10831/20084 [13:11<09:47, 15.76it/s]
Running loglikelihood requests:  54%|█████▍    | 10888/20084 [13:15<09:44, 15.75it/s]
Running loglikelihood requests:  54%|█████▍    | 10945/20084 [13:18<09:41, 15.72it/s]
Running loglikelihood requests:  55%|█████▍    | 11002/20084 [13:22<09:38, 15.71it/s]
Running loglikelihood requests:  55%|█████▌    | 11059/20084 [13:26<09:33, 15.73it/s]
Running loglikelihood requests:  55%|█████▌    | 11116/20084 [13:29<09:29, 15.74it/s]
Running loglikelihood requests:  56%|█████▌    | 11173/20084 [13:33<09:25, 15.75it/s]
Running loglikelihood requests:  56%|█████▌    | 11230/20084 [13:36<09:20, 15.81it/s]
Running loglikelihood requests:  56%|█████▌    | 11287/20084 [13:40<09:15, 15.85it/s]
Running loglikelihood requests:  56%|█████▋    | 11344/20084 [13:44<09:10, 15.87it/s]
Running loglikelihood requests:  57%|█████▋    | 11401/20084 [13:47<09:06, 15.88it/s]
Running loglikelihood requests:  57%|█████▋    | 11458/20084 [13:51<09:03, 15.87it/s]
Running loglikelihood requests:  57%|█████▋    | 11515/20084 [13:54<08:58, 15.90it/s]
Running loglikelihood requests:  58%|█████▊    | 11572/20084 [13:58<08:55, 15.89it/s]
Running loglikelihood requests:  58%|█████▊    | 11629/20084 [14:02<08:52, 15.87it/s]
Running loglikelihood requests:  58%|█████▊    | 11686/20084 [14:05<08:50, 15.84it/s]
Running loglikelihood requests:  58%|█████▊    | 11743/20084 [14:09<08:46, 15.84it/s]
Running loglikelihood requests:  59%|█████▉    | 11800/20084 [14:12<08:42, 15.85it/s]
Running loglikelihood requests:  59%|█████▉    | 11857/20084 [14:16<08:39, 15.84it/s]
Running loglikelihood requests:  59%|█████▉    | 11914/20084 [14:20<08:35, 15.84it/s]
Running loglikelihood requests:  60%|█████▉    | 11971/20084 [14:23<08:30, 15.88it/s]
Running loglikelihood requests:  60%|█████▉    | 12028/20084 [14:27<08:26, 15.92it/s]
Running loglikelihood requests:  60%|██████    | 12085/20084 [14:30<08:21, 15.93it/s]
Running loglikelihood requests:  60%|██████    | 12142/20084 [14:34<08:17, 15.96it/s]
Running loglikelihood requests:  61%|██████    | 12199/20084 [14:37<08:13, 15.99it/s]
Running loglikelihood requests:  61%|██████    | 12256/20084 [14:41<08:10, 15.97it/s]
Running loglikelihood requests:  61%|██████▏   | 12313/20084 [14:45<08:06, 15.97it/s]
Running loglikelihood requests:  62%|██████▏   | 12370/20084 [14:48<08:03, 15.97it/s]
Running loglikelihood requests:  62%|██████▏   | 12427/20084 [14:52<07:59, 15.98it/s]
Running loglikelihood requests:  62%|██████▏   | 12484/20084 [14:55<07:55, 15.99it/s]
Running loglikelihood requests:  62%|██████▏   | 12541/20084 [14:59<07:51, 15.99it/s]
Running loglikelihood requests:  63%|██████▎   | 12598/20084 [15:02<07:48, 15.98it/s]
Running loglikelihood requests:  63%|██████▎   | 12655/20084 [15:06<07:44, 15.99it/s]
Running loglikelihood requests:  63%|██████▎   | 12712/20084 [15:09<07:39, 16.06it/s]
Running loglikelihood requests:  64%|██████▎   | 12769/20084 [15:13<07:34, 16.09it/s]
Running loglikelihood requests:  64%|██████▍   | 12826/20084 [15:16<07:29, 16.14it/s]
Running loglikelihood requests:  64%|██████▍   | 12883/20084 [15:20<07:25, 16.16it/s]
Running loglikelihood requests:  64%|██████▍   | 12940/20084 [15:24<07:21, 16.17it/s]
Running loglikelihood requests:  65%|██████▍   | 12997/20084 [15:27<07:18, 16.16it/s]
Running loglikelihood requests:  65%|██████▍   | 13054/20084 [15:31<07:14, 16.16it/s]
Running loglikelihood requests:  65%|██████▌   | 13111/20084 [15:34<07:10, 16.21it/s]
Running loglikelihood requests:  66%|██████▌   | 13168/20084 [15:38<07:05, 16.25it/s]
Running loglikelihood requests:  66%|██████▌   | 13225/20084 [15:41<07:01, 16.28it/s]
Running loglikelihood requests:  66%|██████▌   | 13282/20084 [15:45<06:57, 16.28it/s]
Running loglikelihood requests:  66%|██████▋   | 13339/20084 [15:48<06:54, 16.29it/s]
Running loglikelihood requests:  67%|██████▋   | 13396/20084 [15:52<06:49, 16.32it/s]
Running loglikelihood requests:  67%|██████▋   | 13453/20084 [15:55<06:45, 16.34it/s]
Running loglikelihood requests:  67%|██████▋   | 13510/20084 [15:58<06:41, 16.36it/s]
Running loglikelihood requests:  68%|██████▊   | 13567/20084 [16:02<06:38, 16.37it/s]
Running loglikelihood requests:  68%|██████▊   | 13624/20084 [16:05<06:35, 16.34it/s]
Running loglikelihood requests:  68%|██████▊   | 13681/20084 [16:09<06:31, 16.34it/s]
Running loglikelihood requests:  68%|██████▊   | 13738/20084 [16:12<06:28, 16.34it/s]
Running loglikelihood requests:  69%|██████▊   | 13795/20084 [16:16<06:24, 16.34it/s]
Running loglikelihood requests:  69%|██████▉   | 13852/20084 [16:19<06:21, 16.34it/s]
Running loglikelihood requests:  69%|██████▉   | 13909/20084 [16:23<06:17, 16.34it/s]
Running loglikelihood requests:  70%|██████▉   | 13966/20084 [16:26<06:13, 16.40it/s]
Running loglikelihood requests:  70%|██████▉   | 14023/20084 [16:30<06:08, 16.44it/s]
Running loglikelihood requests:  70%|███████   | 14080/20084 [16:33<06:04, 16.47it/s]
Running loglikelihood requests:  70%|███████   | 14137/20084 [16:37<06:00, 16.50it/s]
Running loglikelihood requests:  71%|███████   | 14194/20084 [16:40<05:56, 16.53it/s]
Running loglikelihood requests:  71%|███████   | 14251/20084 [16:44<05:52, 16.56it/s]
Running loglikelihood requests:  71%|███████   | 14308/20084 [16:47<05:47, 16.60it/s]
Running loglikelihood requests:  72%|███████▏  | 14365/20084 [16:50<05:44, 16.62it/s]
Running loglikelihood requests:  72%|███████▏  | 14422/20084 [16:54<05:40, 16.63it/s]
Running loglikelihood requests:  72%|███████▏  | 14479/20084 [16:57<05:36, 16.63it/s]
Running loglikelihood requests:  72%|███████▏  | 14536/20084 [17:01<05:33, 16.66it/s]
Running loglikelihood requests:  73%|███████▎  | 14593/20084 [17:04<05:29, 16.67it/s]
Running loglikelihood requests:  73%|███████▎  | 14650/20084 [17:07<05:25, 16.67it/s]
Running loglikelihood requests:  73%|███████▎  | 14707/20084 [17:11<05:22, 16.65it/s]
Running loglikelihood requests:  74%|███████▎  | 14764/20084 [17:14<05:19, 16.64it/s]
Running loglikelihood requests:  74%|███████▍  | 14821/20084 [17:18<05:16, 16.65it/s]
Running loglikelihood requests:  74%|███████▍  | 14878/20084 [17:21<05:12, 16.65it/s]
Running loglikelihood requests:  74%|███████▍  | 14935/20084 [17:25<05:09, 16.66it/s]
Running loglikelihood requests:  75%|███████▍  | 14992/20084 [17:28<05:05, 16.67it/s]
Running loglikelihood requests:  75%|███████▍  | 15049/20084 [17:31<05:01, 16.69it/s]
Running loglikelihood requests:  75%|███████▌  | 15106/20084 [17:35<04:57, 16.73it/s]
Running loglikelihood requests:  75%|███████▌  | 15163/20084 [17:38<04:53, 16.79it/s]
Running loglikelihood requests:  76%|███████▌  | 15220/20084 [17:42<04:49, 16.82it/s]
Running loglikelihood requests:  76%|███████▌  | 15277/20084 [17:45<04:45, 16.85it/s]
Running loglikelihood requests:  76%|███████▋  | 15334/20084 [17:48<04:41, 16.87it/s]
Running loglikelihood requests:  77%|███████▋  | 15391/20084 [17:52<04:37, 16.90it/s]
Running loglikelihood requests:  77%|███████▋  | 15448/20084 [17:55<04:33, 16.92it/s]
Running loglikelihood requests:  77%|███████▋  | 15505/20084 [17:58<04:30, 16.94it/s]
Running loglikelihood requests:  77%|███████▋  | 15562/20084 [18:02<04:26, 16.95it/s]
Running loglikelihood requests:  78%|███████▊  | 15619/20084 [18:05<04:23, 16.97it/s]
Running loglikelihood requests:  78%|███████▊  | 15676/20084 [18:08<04:19, 16.98it/s]
Running loglikelihood requests:  78%|███████▊  | 15733/20084 [18:12<04:16, 16.99it/s]
Running loglikelihood requests:  79%|███████▊  | 15790/20084 [18:15<04:11, 17.05it/s]
Running loglikelihood requests:  79%|███████▉  | 15847/20084 [18:18<04:08, 17.06it/s]
Running loglikelihood requests:  79%|███████▉  | 15904/20084 [18:22<04:04, 17.08it/s]
Running loglikelihood requests:  79%|███████▉  | 15961/20084 [18:25<04:01, 17.06it/s]
Running loglikelihood requests:  80%|███████▉  | 16018/20084 [18:28<03:58, 17.05it/s]
Running loglikelihood requests:  80%|████████  | 16075/20084 [18:32<03:55, 17.06it/s]
Running loglikelihood requests:  80%|████████  | 16132/20084 [18:35<03:51, 17.06it/s]
Running loglikelihood requests:  81%|████████  | 16189/20084 [18:38<03:47, 17.11it/s]
Running loglikelihood requests:  81%|████████  | 16246/20084 [18:42<03:43, 17.15it/s]
Running loglikelihood requests:  81%|████████  | 16303/20084 [18:45<03:39, 17.21it/s]
Running loglikelihood requests:  81%|████████▏ | 16360/20084 [18:48<03:36, 17.24it/s]
Running loglikelihood requests:  82%|████████▏ | 16417/20084 [18:52<03:32, 17.28it/s]
Running loglikelihood requests:  82%|████████▏ | 16474/20084 [18:55<03:28, 17.31it/s]
Running loglikelihood requests:  82%|████████▏ | 16531/20084 [18:58<03:24, 17.35it/s]
Running loglikelihood requests:  83%|████████▎ | 16588/20084 [19:01<03:21, 17.39it/s]
Running loglikelihood requests:  83%|████████▎ | 16645/20084 [19:05<03:17, 17.41it/s]
Running loglikelihood requests:  83%|████████▎ | 16702/20084 [19:08<03:14, 17.43it/s]
Running loglikelihood requests:  83%|████████▎ | 16759/20084 [19:11<03:10, 17.46it/s]
Running loglikelihood requests:  84%|████████▎ | 16816/20084 [19:14<03:07, 17.48it/s]
Running loglikelihood requests:  84%|████████▍ | 16873/20084 [19:18<03:03, 17.49it/s]
Running loglikelihood requests:  84%|████████▍ | 16930/20084 [19:21<03:00, 17.47it/s]
Running loglikelihood requests:  85%|████████▍ | 16987/20084 [19:24<02:57, 17.48it/s]
Running loglikelihood requests:  85%|████████▍ | 17044/20084 [19:27<02:53, 17.49it/s]
Running loglikelihood requests:  85%|████████▌ | 17101/20084 [19:31<02:50, 17.53it/s]
Running loglikelihood requests:  85%|████████▌ | 17158/20084 [19:34<02:46, 17.57it/s]
Running loglikelihood requests:  86%|████████▌ | 17215/20084 [19:37<02:42, 17.63it/s]
Running loglikelihood requests:  86%|████████▌ | 17272/20084 [19:40<02:39, 17.67it/s]
Running loglikelihood requests:  86%|████████▋ | 17329/20084 [19:44<02:35, 17.69it/s]
Running loglikelihood requests:  87%|████████▋ | 17386/20084 [19:47<02:32, 17.71it/s]
Running loglikelihood requests:  87%|████████▋ | 17443/20084 [19:50<02:28, 17.74it/s]
Running loglikelihood requests:  87%|████████▋ | 17500/20084 [19:53<02:25, 17.77it/s]
Running loglikelihood requests:  87%|████████▋ | 17557/20084 [19:56<02:21, 17.80it/s]
Running loglikelihood requests:  88%|████████▊ | 17614/20084 [20:00<02:18, 17.82it/s]
Running loglikelihood requests:  88%|████████▊ | 17671/20084 [20:03<02:15, 17.86it/s]
Running loglikelihood requests:  88%|████████▊ | 17728/20084 [20:06<02:11, 17.88it/s]
Running loglikelihood requests:  89%|████████▊ | 17785/20084 [20:09<02:08, 17.90it/s]
Running loglikelihood requests:  89%|████████▉ | 17842/20084 [20:12<02:04, 17.96it/s]
Running loglikelihood requests:  89%|████████▉ | 17899/20084 [20:15<02:01, 18.00it/s]
Running loglikelihood requests:  89%|████████▉ | 17956/20084 [20:19<01:58, 18.03it/s]
Running loglikelihood requests:  90%|████████▉ | 18013/20084 [20:22<01:54, 18.05it/s]
Running loglikelihood requests:  90%|████████▉ | 18070/20084 [20:25<01:51, 18.05it/s]
Running loglikelihood requests:  90%|█████████ | 18127/20084 [20:28<01:48, 18.07it/s]
Running loglikelihood requests:  91%|█████████ | 18184/20084 [20:31<01:44, 18.12it/s]
Running loglikelihood requests:  91%|█████████ | 18241/20084 [20:34<01:41, 18.15it/s]
Running loglikelihood requests:  91%|█████████ | 18298/20084 [20:37<01:38, 18.17it/s]
Running loglikelihood requests:  91%|█████████▏| 18355/20084 [20:40<01:34, 18.23it/s]
Running loglikelihood requests:  92%|█████████▏| 18412/20084 [20:44<01:31, 18.30it/s]
Running loglikelihood requests:  92%|█████████▏| 18469/20084 [20:47<01:28, 18.33it/s]
Running loglikelihood requests:  92%|█████████▏| 18526/20084 [20:50<01:24, 18.39it/s]
Running loglikelihood requests:  93%|█████████▎| 18583/20084 [20:53<01:21, 18.43it/s]
Running loglikelihood requests:  93%|█████████▎| 18640/20084 [20:56<01:18, 18.45it/s]
Running loglikelihood requests:  93%|█████████▎| 18697/20084 [20:59<01:14, 18.53it/s]
Running loglikelihood requests:  93%|█████████▎| 18754/20084 [21:02<01:11, 18.57it/s]
Running loglikelihood requests:  94%|█████████▎| 18811/20084 [21:05<01:08, 18.63it/s]
Running loglikelihood requests:  94%|█████████▍| 18868/20084 [21:08<01:04, 18.72it/s]
Running loglikelihood requests:  94%|█████████▍| 18925/20084 [21:11<01:01, 18.81it/s]
Running loglikelihood requests:  95%|█████████▍| 18982/20084 [21:14<00:58, 18.85it/s]
Running loglikelihood requests:  95%|█████████▍| 19039/20084 [21:17<00:55, 18.91it/s]
Running loglikelihood requests:  95%|█████████▌| 19096/20084 [21:20<00:52, 18.97it/s]
Running loglikelihood requests:  95%|█████████▌| 19153/20084 [21:23<00:48, 19.05it/s]
Running loglikelihood requests:  96%|█████████▌| 19210/20084 [21:26<00:45, 19.11it/s]
Running loglikelihood requests:  96%|█████████▌| 19267/20084 [21:29<00:42, 19.19it/s]
Running loglikelihood requests:  96%|█████████▌| 19324/20084 [21:32<00:39, 19.26it/s]
Running loglikelihood requests:  96%|█████████▋| 19381/20084 [21:35<00:36, 19.34it/s]
Running loglikelihood requests:  97%|█████████▋| 19438/20084 [21:38<00:33, 19.42it/s]
Running loglikelihood requests:  97%|█████████▋| 19495/20084 [21:41<00:30, 19.50it/s]
Running loglikelihood requests:  97%|█████████▋| 19552/20084 [21:43<00:27, 19.60it/s]
Running loglikelihood requests:  98%|█████████▊| 19609/20084 [21:46<00:24, 19.71it/s]
Running loglikelihood requests:  98%|█████████▊| 19666/20084 [21:49<00:21, 19.82it/s]
Running loglikelihood requests:  98%|█████████▊| 19723/20084 [21:52<00:18, 19.98it/s]
Running loglikelihood requests:  98%|█████████▊| 19780/20084 [21:55<00:15, 20.16it/s]
Running loglikelihood requests:  99%|█████████▉| 19837/20084 [21:57<00:12, 20.40it/s]
Running loglikelihood requests:  99%|█████████▉| 19894/20084 [22:00<00:09, 20.61it/s]
Running loglikelihood requests:  99%|█████████▉| 19951/20084 [22:03<00:06, 20.90it/s]
Running loglikelihood requests: 100%|█████████▉| 20008/20084 [22:05<00:03, 21.26it/s]
Running loglikelihood requests: 100%|█████████▉| 20065/20084 [22:06<00:00, 26.73it/s]
Running loglikelihood requests: 100%|██████████| 20084/20084 [22:06<00:00, 15.14it/s]
n136-128-154:1708729:1710784 [1] NCCL INFO Channel 00/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708729:1710784 [1] NCCL INFO Channel 01/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708729:1710784 [1] NCCL INFO Channel 02/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708729:1710784 [1] NCCL INFO Channel 03/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708729:1710784 [1] NCCL INFO Channel 04/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708729:1710784 [1] NCCL INFO Channel 05/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708729:1710784 [1] NCCL INFO Channel 06/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708729:1710784 [1] NCCL INFO Channel 07/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708729:1710784 [1] NCCL INFO Channel 08/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708729:1710784 [1] NCCL INFO Channel 09/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708729:1710784 [1] NCCL INFO Channel 10/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708729:1710784 [1] NCCL INFO Channel 11/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708729:1710784 [1] NCCL INFO Channel 12/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708729:1710784 [1] NCCL INFO Channel 13/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708729:1710784 [1] NCCL INFO Channel 14/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708729:1710784 [1] NCCL INFO Channel 15/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708729:1710784 [1] NCCL INFO Channel 16/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708729:1710784 [1] NCCL INFO Channel 17/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708729:1710784 [1] NCCL INFO Channel 18/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708729:1710784 [1] NCCL INFO Channel 19/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708729:1710784 [1] NCCL INFO Channel 20/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708729:1710784 [1] NCCL INFO Channel 21/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708729:1710784 [1] NCCL INFO Channel 22/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708729:1710784 [1] NCCL INFO Channel 23/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708729:1710784 [1] NCCL INFO Channel 24/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708729:1710784 [1] NCCL INFO Channel 25/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708729:1710784 [1] NCCL INFO Channel 26/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708729:1710784 [1] NCCL INFO Channel 27/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708729:1710784 [1] NCCL INFO Channel 28/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708729:1710784 [1] NCCL INFO Channel 29/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708729:1710784 [1] NCCL INFO Channel 30/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1708729:1710784 [1] NCCL INFO Channel 31/1 : 1[2] -> 0[1] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:1708729:1710790 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1708729:1710790 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1708729:1710790 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1708729:1710790 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1708729:1710790 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1708729:1710790 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1708729:1708999 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-06:03:42:08 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 10, batch_size: auto (57)
|  Tasks  |Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|---------|------:|------|-----:|--------|---|-----:|---|-----:|
|hellaswag|      1|none  |    10|acc     |↑  |0.2825|±  |0.0045|
|         |       |none  |    10|acc_norm|↑  |0.3104|±  |0.0046|

[rank0]:[W1206 03:42:09.860326522 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:1708728:1710842 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1708728:1710842 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1708728:1710842 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1708728:1709001 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:1708728:1710842 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1708728:1710842 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1708728:1710842 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1708729:1708999 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:1708729:1710790 [1] NCCL INFO comm 0xfc75670 rank 1 nranks 2 cudaDev 1 busId 4a000 - Abort COMPLETE
n136-128-154:1708728:1710842 [0] NCCL INFO comm 0xf855840 rank 0 nranks 2 cudaDev 0 busId 16000 - Abort COMPLETE
任务 hellaswag 评估完成！
Execution time: 35039.5 seconds
Namespace(save_dir='/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg', self_attn_layer_to_quant='29 23 24 30 18', mlp_layer_to_quant='26 20 22 23 19', model_id='/mnt/bn/life-mllm/users/cxr/quantization/models/meta-llama/Llama-3.1-8B', cuda_id=1)
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:08<00:26,  8.78s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:17<00:17,  8.72s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:26<00:08,  8.67s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:28<00:00,  6.06s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:28<00:00,  7.03s/it]
LlamaForCausalLM(
  (model): LlamaModel(
    (embed_tokens): Embedding(128256, 4096)
    (layers): ModuleList(
      (0-31): 32 x LlamaDecoderLayer(
        (self_attn): LlamaAttention(
          (q_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (k_proj): Linear(in_features=4096, out_features=1024, bias=False)
          (v_proj): Linear(in_features=4096, out_features=1024, bias=False)
          (o_proj): Linear(in_features=4096, out_features=4096, bias=False)
        )
        (mlp): LlamaMLP(
          (gate_proj): Linear(in_features=4096, out_features=14336, bias=False)
          (up_proj): Linear(in_features=4096, out_features=14336, bias=False)
          (down_proj): Linear(in_features=14336, out_features=4096, bias=False)
          (act_fn): SiLUActivation()
        )
        (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
        (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
      )
    )
    (norm): LlamaRMSNorm((4096,), eps=1e-05)
    (rotary_emb): LlamaRotaryEmbedding()
  )
  (lm_head): Linear(in_features=4096, out_features=128256, bias=False)
)
Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.
Once upon a time, a long time ago, in a faraway land, a little girl was born. Her mother named her "Hope" and it was a name that suited her well. She was a happy child, always full of joy and hope. Her mother taught her to be grateful for all that she had, and she grew up to be a kind and caring young woman. She was always looking for ways to help others, and she always had a smile on her face. But one day, everything changed
quanting ... 
layers.0.self_attn 4 bit quant 
layers.0.mlp 4 bit quant 
layers.1.self_attn 4 bit quant 
layers.1.mlp 4 bit quant 
layers.2.self_attn 4 bit quant 
layers.2.mlp 4 bit quant 
layers.3.self_attn 4 bit quant 
layers.3.mlp 4 bit quant 
layers.4.self_attn 4 bit quant 
layers.4.mlp 4 bit quant 
layers.5.self_attn 4 bit quant 
layers.5.mlp 4 bit quant 
layers.6.self_attn 4 bit quant 
layers.6.mlp 4 bit quant 
layers.7.self_attn 4 bit quant 
layers.7.mlp 4 bit quant 
layers.8.self_attn 4 bit quant 
layers.8.mlp 4 bit quant 
layers.9.self_attn 4 bit quant 
layers.9.mlp 4 bit quant 
layers.10.self_attn 4 bit quant 
layers.10.mlp 4 bit quant 
layers.11.self_attn 4 bit quant 
layers.11.mlp 4 bit quant 
layers.12.self_attn 4 bit quant 
layers.12.mlp 4 bit quant 
layers.13.self_attn 4 bit quant 
layers.13.mlp 4 bit quant 
layers.14.self_attn 4 bit quant 
layers.14.mlp 4 bit quant 
layers.15.self_attn 4 bit quant 
layers.15.mlp 4 bit quant 
layers.16.self_attn 4 bit quant 
layers.16.mlp 4 bit quant 
layers.17.self_attn 4 bit quant 
layers.17.mlp 4 bit quant 
layers.18.self_attn 2 bit quant 
layers.18.mlp 4 bit quant 
layers.19.self_attn 4 bit quant 
layers.19.mlp 2 bit quant 
layers.20.self_attn 4 bit quant 
layers.20.mlp 2 bit quant 
layers.21.self_attn 4 bit quant 
layers.21.mlp 4 bit quant 
layers.22.self_attn 4 bit quant 
layers.22.mlp 2 bit quant 
layers.23.self_attn 2 bit quant 
layers.23.mlp 2 bit quant 
layers.24.self_attn 2 bit quant 
layers.24.mlp 4 bit quant 
layers.25.self_attn 4 bit quant 
layers.25.mlp 4 bit quant 
layers.26.self_attn 4 bit quant 
layers.26.mlp 2 bit quant 
layers.27.self_attn 4 bit quant 
layers.27.mlp 4 bit quant 
layers.28.self_attn 4 bit quant 
layers.28.mlp 4 bit quant 
layers.29.self_attn 2 bit quant 
layers.29.mlp 4 bit quant 
layers.30.self_attn 2 bit quant 
layers.30.mlp 4 bit quant 
layers.31.self_attn 4 bit quant 
layers.31.mlp 4 bit quant 
quanted
LlamaForCausalLM(
  (model): LlamaModel(
    (embed_tokens): Embedding(128256, 4096)
    (layers): ModuleList(
      (0-31): 32 x LlamaDecoderLayer(
        (self_attn): LlamaAttention(
          (q_proj): W8A16Linear(4096, 4096, bias=False, weight_quant=per_channel)
          (k_proj): W8A16Linear(4096, 1024, bias=False, weight_quant=per_channel)
          (v_proj): W8A16Linear(4096, 1024, bias=False, weight_quant=per_channel)
          (o_proj): W8A16Linear(4096, 4096, bias=False, weight_quant=per_channel)
        )
        (mlp): LlamaMLP(
          (gate_proj): W8A16Linear(4096, 14336, bias=False, weight_quant=per_channel)
          (up_proj): W8A16Linear(4096, 14336, bias=False, weight_quant=per_channel)
          (down_proj): W8A16Linear(14336, 4096, bias=False, weight_quant=per_channel)
          (act_fn): SiLUActivation()
        )
        (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
        (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
      )
    )
    (norm): LlamaRMSNorm((4096,), eps=1e-05)
    (rotary_emb): LlamaRotaryEmbedding()
  )
  (lm_head): Linear(in_features=4096, out_features=128256, bias=False)
)
Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.
Once upon a time, in the land of the head shop, a clever man called S.N.
Who do you choose for the 8.5?
Squatter’s Choice - 12
The 1981 pick-up
The 1999 pick-up
The 2004 pick-up
The 2007 pick-up
The 2012 (or 2013?) pick-up
The 2015 (or possibly the 2016?) pick-up
The 2017 (or possibly the

==================================================
开始评估：任务=winogrande | 少样本数=5 | 模型=Llama-3.1-8B-quantization-fg
输出路径：results2/Llama-3.1-8B-quantization-fg/fg/winogrande.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-06:03:48:48 INFO     [__main__:440] Selected Tasks: ['winogrande']
2025-12-06:03:48:48 INFO     [__main__:440] Selected Tasks: ['winogrande']
2025-12-06:03:48:48 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-06:03:48:48 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-06:03:48:48 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-06:03:48:48 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-06:03:48:49 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-06:03:48:49 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-06:03:48:49 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.03it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.10s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.13s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.29s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:00,  1.01it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.48it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.24it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.12s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.31it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.09it/s]
2025-12-06:03:49:04 WARNING  [evaluator:309] Overwriting default num_fewshot of winogrande from None to 5
2025-12-06:03:49:05 INFO     [api.task:434] Building contexts for winogrande on rank 0...
2025-12-06:03:49:05 WARNING  [evaluator:309] Overwriting default num_fewshot of winogrande from None to 5
2025-12-06:03:49:05 INFO     [api.task:434] Building contexts for winogrande on rank 1...

  0%|          | 0/634 [00:00<?, ?it/s]
100%|██████████| 634/634 [00:00<00:00, 26809.85it/s]

  0%|          | 0/633 [00:00<?, ?it/s]n136-128-154:1711814:1711814 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1711814:1711814 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1711814:1711814 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:1711814:1711814 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:1711814:1711814 [0] NCCL INFO Comm config Blocking set to 1

100%|██████████| 633/633 [00:00<00:00, 25033.18it/s]
n136-128-154:1711815:1711815 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:1711815:1711815 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1711815:1711815 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1711815:1711815 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:1711815:1711815 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:1711814:1712013 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:1711814:1712013 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:1711815:1712014 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:1711815:1712014 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:1711814:1712013 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1711814:1712013 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1711814:1712013 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:1711815:1712014 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1711815:1712014 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1711815:1712014 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:1711815:1712014 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:1711815:1712014 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:1711815:1712014 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:1711815:1712014 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:1711815:1712014 [1] NCCL INFO Using network IB
n136-128-154:1711814:1712013 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:1711814:1712013 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:1711814:1712013 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:1711814:1712013 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:1711814:1712013 [0] NCCL INFO Using network IB
n136-128-154:1711815:1712014 [1] NCCL INFO ncclCommInitRankConfig comm 0x153b72b0 rank 1 nranks 2 cudaDev 1 nvmlDev 2 busId 4a000 commId 0x9717e9b07b27bee8 - Init START
n136-128-154:1711814:1712013 [0] NCCL INFO ncclCommInitRankConfig comm 0x14c61800 rank 0 nranks 2 cudaDev 0 nvmlDev 1 busId 16000 commId 0x9717e9b07b27bee8 - Init START
n136-128-154:1711814:1712013 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:1711815:1712014 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:1711814:1712013 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:1711814:1712013 [0] NCCL INFO Retrieving state for IB
n136-128-154:1711814:1712013 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:1711814:1712013 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:1711815:1712014 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:1711815:1712014 [1] NCCL INFO Retrieving state for IB
n136-128-154:1711815:1712014 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:1711814:1712013 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:1711815:1712014 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:1711814:1712013 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:1711815:1712014 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:1711814:1712013 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:1711815:1712014 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:1711815:1712014 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:1711815:1712014 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1711814:1712013 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1711815:1712014 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1711814:1712013 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1711815:1712014 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1711814:1712013 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1711815:1712014 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1711814:1712013 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1711815:1712014 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:1711814:1712013 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:1711815:1712014 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:1711814:1712013 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:1711815:1712014 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:1711814:1712013 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:1711815:1712014 [1] NCCL INFO               + PCI[24.0] - PCI/0-14000 (1000c01010de13b8)
n136-128-154:1711814:1712013 [0] NCCL INFO               + PCI[24.0] - PCI/0-14000 (1000c01010de13b8)
n136-128-154:1711815:1712014 [1] NCCL INFO                             + PCI[24.0] - GPU/0-16000 (0)
n136-128-154:1711814:1712013 [0] NCCL INFO                             + PCI[24.0] - GPU/0-16000 (0)
n136-128-154:1711815:1712014 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1711814:1712013 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1711815:1712014 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:1711814:1712013 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:1711815:1712014 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:1711814:1712013 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:1711815:1712014 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:1711814:1712013 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:1711815:1712014 [1] NCCL INFO               + PCI[24.0] - PCI/0-48000 (1000c01010de13b8)
n136-128-154:1711814:1712013 [0] NCCL INFO               + PCI[24.0] - PCI/0-48000 (1000c01010de13b8)
n136-128-154:1711815:1712014 [1] NCCL INFO                             + PCI[24.0] - GPU/0-4a000 (1)
n136-128-154:1711814:1712013 [0] NCCL INFO                             + PCI[24.0] - GPU/0-4a000 (1)
n136-128-154:1711815:1712014 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1711814:1712013 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1711815:1712014 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:1711814:1712013 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:1711815:1712014 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:1711814:1712013 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:1711815:1712014 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:1711814:1712013 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:1711815:1712014 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:1711814:1712013 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:1711815:1712014 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:1711814:1712013 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:1711815:1712014 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:1711814:1712013 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:1711815:1712014 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:1711814:1712013 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:1711815:1712014 [1] NCCL INFO ==========================================
n136-128-154:1711814:1712013 [0] NCCL INFO ==========================================
n136-128-154:1711815:1712014 [1] NCCL INFO GPU/0-16000 :GPU/0-16000 (0/5000.0/LOC) GPU/0-4a000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1711814:1712013 [0] NCCL INFO GPU/0-16000 :GPU/0-16000 (0/5000.0/LOC) GPU/0-4a000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1711815:1712014 [1] NCCL INFO GPU/0-4a000 :GPU/0-16000 (2/240.0/NVL) GPU/0-4a000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1711814:1712013 [0] NCCL INFO GPU/0-4a000 :GPU/0-16000 (2/240.0/NVL) GPU/0-4a000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1711814:1712013 [0] NCCL INFO Setting affinity for GPU 1 to 1-31,65-95
n136-128-154:1711815:1712014 [1] NCCL INFO Setting affinity for GPU 2 to 1-31,65-95
n136-128-154:1711815:1712014 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:1711814:1712013 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:1711815:1712014 [1] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1711814:1712013 [0] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1711815:1712014 [1] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1711814:1712013 [0] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1711815:1712014 [1] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1711814:1712013 [0] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1711815:1712014 [1] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1711814:1712013 [0] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1711815:1712014 [1] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1711814:1712013 [0] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1711815:1712014 [1] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1711814:1712013 [0] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1711815:1712014 [1] NCCL INFO  6 : GPU/0-16000 GPU/0-4a000
n136-128-154:1711814:1712013 [0] NCCL INFO  6 : GPU/0-16000 GPU/0-4a000
n136-128-154:1711815:1712014 [1] NCCL INFO  7 : GPU/0-16000 GPU/0-4a000
n136-128-154:1711814:1712013 [0] NCCL INFO  7 : GPU/0-16000 GPU/0-4a000
n136-128-154:1711815:1712014 [1] NCCL INFO  8 : GPU/0-16000 GPU/0-4a000
n136-128-154:1711814:1712013 [0] NCCL INFO  8 : GPU/0-16000 GPU/0-4a000
n136-128-154:1711815:1712014 [1] NCCL INFO  9 : GPU/0-16000 GPU/0-4a000
n136-128-154:1711814:1712013 [0] NCCL INFO  9 : GPU/0-16000 GPU/0-4a000
n136-128-154:1711815:1712014 [1] NCCL INFO 10 : GPU/0-16000 GPU/0-4a000
n136-128-154:1711814:1712013 [0] NCCL INFO 10 : GPU/0-16000 GPU/0-4a000
n136-128-154:1711815:1712014 [1] NCCL INFO 11 : GPU/0-16000 GPU/0-4a000
n136-128-154:1711814:1712013 [0] NCCL INFO 11 : GPU/0-16000 GPU/0-4a000
n136-128-154:1711815:1712014 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:1711814:1712013 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:1711815:1712014 [1] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1711814:1712013 [0] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1711815:1712014 [1] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1711814:1712013 [0] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1711815:1712014 [1] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1711814:1712013 [0] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1711815:1712014 [1] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1711814:1712013 [0] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1711815:1712014 [1] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1711814:1712013 [0] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1711815:1712014 [1] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1711814:1712013 [0] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1711815:1712014 [1] NCCL INFO  6 : GPU/0-4a000 GPU/0-16000
n136-128-154:1711814:1712013 [0] NCCL INFO  6 : GPU/0-4a000 GPU/0-16000
n136-128-154:1711815:1712014 [1] NCCL INFO  7 : GPU/0-4a000 GPU/0-16000
n136-128-154:1711814:1712013 [0] NCCL INFO  7 : GPU/0-4a000 GPU/0-16000
n136-128-154:1711815:1712014 [1] NCCL INFO  8 : GPU/0-4a000 GPU/0-16000
n136-128-154:1711814:1712013 [0] NCCL INFO  8 : GPU/0-4a000 GPU/0-16000
n136-128-154:1711815:1712014 [1] NCCL INFO  9 : GPU/0-4a000 GPU/0-16000
n136-128-154:1711814:1712013 [0] NCCL INFO  9 : GPU/0-4a000 GPU/0-16000
n136-128-154:1711815:1712014 [1] NCCL INFO 10 : GPU/0-4a000 GPU/0-16000
n136-128-154:1711814:1712013 [0] NCCL INFO 10 : GPU/0-4a000 GPU/0-16000
n136-128-154:1711815:1712014 [1] NCCL INFO 11 : GPU/0-4a000 GPU/0-16000
n136-128-154:1711814:1712013 [0] NCCL INFO 11 : GPU/0-4a000 GPU/0-16000
n136-128-154:1711814:1712013 [0] NCCL INFO comm 0x14c61800 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:1711815:1712014 [1] NCCL INFO comm 0x153b72b0 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:1711814:1712013 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:1711815:1712014 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:1711814:1712013 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:1711815:1712014 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:1711814:1712013 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:1711815:1712014 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:1711814:1712013 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:1711815:1712014 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:1711814:1712013 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:1711815:1712014 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:1711814:1712013 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:1711815:1712014 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:1711814:1712013 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:1711815:1712014 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:1711814:1712013 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:1711815:1712014 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:1711814:1712013 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:1711815:1712014 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:1711814:1712013 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:1711815:1712014 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:1711814:1712013 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:1711815:1712014 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:1711814:1712013 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:1711815:1712014 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:1711814:1712013 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:1711815:1712014 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:1711814:1712013 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:1711815:1712014 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:1711814:1712013 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:1711815:1712014 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:1711814:1712013 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:1711815:1712014 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:1711814:1712013 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:1711815:1712014 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:1711814:1712013 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:1711815:1712014 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:1711814:1712013 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:1711815:1712014 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:1711814:1712013 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:1711815:1712014 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:1711814:1712013 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:1711815:1712014 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:1711814:1712013 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:1711815:1712014 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:1711814:1712013 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:1711815:1712014 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:1711814:1712013 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:1711815:1712014 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:1711814:1712013 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:1711814:1712013 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:1711815:1712014 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:1711814:1712013 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:1711815:1712014 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:1711814:1712013 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:1711815:1712014 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:1711814:1712013 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:1711815:1712014 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:1711814:1712013 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:1711815:1712014 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:1711814:1712013 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:1711815:1712014 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:1711814:1712013 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:1711815:1712014 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:1711814:1712013 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:1711815:1712014 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:1711814:1712013 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:1711815:1712014 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:1711814:1712013 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:1711815:1712014 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:1711814:1712013 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:1711815:1712014 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:1711814:1712013 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:1711815:1712014 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:1711814:1712013 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:1711815:1712014 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:1711814:1712013 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:1711815:1712014 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:1711814:1712013 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:1711815:1712014 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:1711814:1712013 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:1711815:1712014 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:1711814:1712013 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:1711815:1712014 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:1711814:1712013 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:1711815:1712014 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:1711814:1712013 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:1711815:1712014 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:1711814:1712013 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:1711815:1712014 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:1711814:1712013 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:1711815:1712014 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:1711814:1712013 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:1711815:1712014 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:1711814:1712013 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:1711815:1712014 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:1711814:1712013 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:1711815:1712014 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:1711814:1712013 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:1711815:1712014 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:1711814:1712013 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:1711815:1712014 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:1711814:1712013 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:1711814:1712013 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:1711814:1712013 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:1711814:1712013 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:1711814:1712013 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:1711814:1712013 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:1711814:1712013 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:1711814:1712013 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:1711814:1712013 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:1711814:1712013 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:1711814:1712013 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:1711814:1712013 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:1711814:1712013 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:1711814:1712013 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:1711814:1712013 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:1711814:1712013 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:1711814:1712013 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:1711814:1712013 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:1711814:1712013 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:1711814:1712013 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:1711814:1712013 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:1711814:1712013 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:1711814:1712013 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:1711815:1712014 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:1711815:1712026 [1] NCCL INFO [Proxy Service] Device 1 CPU core 2
n136-128-154:1711815:1712027 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 70
n136-128-154:1711814:1712013 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:1711814:1712013 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:1711814:1712028 [0] NCCL INFO [Proxy Service] Device 0 CPU core 71
n136-128-154:1711814:1712029 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 8
n136-128-154:1711815:1712014 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:1711815:1712014 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:1711814:1712013 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:1711814:1712013 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:1711814:1712013 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:1711815:1712014 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:1711815:1712014 [1] NCCL INFO ncclCommInitRankConfig comm 0x153b72b0 rank 1 nranks 2 cudaDev 1 nvmlDev 2 busId 4a000 commId 0x9717e9b07b27bee8 - Init COMPLETE
n136-128-154:1711815:1712014 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.59 (kernels 0.16, alloc 0.18, bootstrap 0.02, allgathers 0.00, topo 0.06, graphs 0.00, connections 0.12, rest 0.05)
n136-128-154:1711814:1712013 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:1711814:1712013 [0] NCCL INFO ncclCommInitRankConfig comm 0x14c61800 rank 0 nranks 2 cudaDev 0 nvmlDev 1 busId 16000 commId 0x9717e9b07b27bee8 - Init COMPLETE
n136-128-154:1711814:1712013 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.62 (kernels 0.20, alloc 0.19, bootstrap 0.00, allgathers 0.00, topo 0.06, graphs 0.00, connections 0.12, rest 0.05)
n136-128-154:1711815:1712030 [1] NCCL INFO Channel 00/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711815:1712030 [1] NCCL INFO Channel 01/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711815:1712030 [1] NCCL INFO Channel 02/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711815:1712030 [1] NCCL INFO Channel 03/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711815:1712030 [1] NCCL INFO Channel 04/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711815:1712030 [1] NCCL INFO Channel 05/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711814:1712031 [0] NCCL INFO Channel 00/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1711815:1712030 [1] NCCL INFO Channel 06/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711814:1712031 [0] NCCL INFO Channel 01/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1711815:1712030 [1] NCCL INFO Channel 07/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711814:1712031 [0] NCCL INFO Channel 02/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1711815:1712030 [1] NCCL INFO Channel 08/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711814:1712031 [0] NCCL INFO Channel 03/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1711815:1712030 [1] NCCL INFO Channel 09/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711814:1712031 [0] NCCL INFO Channel 04/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1711815:1712030 [1] NCCL INFO Channel 10/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711814:1712031 [0] NCCL INFO Channel 05/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1711815:1712030 [1] NCCL INFO Channel 11/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711814:1712031 [0] NCCL INFO Channel 06/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1711815:1712030 [1] NCCL INFO Channel 12/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711814:1712031 [0] NCCL INFO Channel 07/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1711815:1712030 [1] NCCL INFO Channel 13/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711814:1712031 [0] NCCL INFO Channel 08/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1711815:1712030 [1] NCCL INFO Channel 14/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711814:1712031 [0] NCCL INFO Channel 09/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1711815:1712030 [1] NCCL INFO Channel 15/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711814:1712031 [0] NCCL INFO Channel 10/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1711815:1712030 [1] NCCL INFO Channel 16/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711814:1712031 [0] NCCL INFO Channel 11/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1711815:1712030 [1] NCCL INFO Channel 17/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711814:1712031 [0] NCCL INFO Channel 12/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1711815:1712030 [1] NCCL INFO Channel 18/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711814:1712031 [0] NCCL INFO Channel 13/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1711815:1712030 [1] NCCL INFO Channel 19/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711814:1712031 [0] NCCL INFO Channel 14/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1711815:1712030 [1] NCCL INFO Channel 20/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711814:1712031 [0] NCCL INFO Channel 15/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1711815:1712030 [1] NCCL INFO Channel 21/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711814:1712031 [0] NCCL INFO Channel 16/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1711815:1712030 [1] NCCL INFO Channel 22/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711814:1712031 [0] NCCL INFO Channel 17/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1711814:1712031 [0] NCCL INFO Channel 18/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1711815:1712030 [1] NCCL INFO Channel 23/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711814:1712031 [0] NCCL INFO Channel 19/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1711814:1712031 [0] NCCL INFO Channel 20/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1711814:1712031 [0] NCCL INFO Channel 21/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1711814:1712031 [0] NCCL INFO Channel 22/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1711814:1712031 [0] NCCL INFO Channel 23/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1711815:1712030 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:1711814:1712031 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-06:03:49:05 INFO     [evaluator:559] Running loglikelihood requests
2025-12-06:03:49:05 INFO     [evaluator:559] Running loglikelihood requests

Running loglikelihood requests:   0%|          | 0/1268 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 64
Determined largest batch size: 64

Running loglikelihood requests:   0%|          | 1/1268 [00:06<2:07:31,  6.04s/it]
Running loglikelihood requests:   5%|▌         | 65/1268 [00:07<01:37, 12.40it/s] 
Running loglikelihood requests:  10%|█         | 129/1268 [00:08<00:48, 23.61it/s]
Running loglikelihood requests:  15%|█▌        | 193/1268 [00:08<00:32, 33.43it/s]
Running loglikelihood requests:  20%|██        | 257/1268 [00:09<00:24, 41.59it/s]
Running loglikelihood requests:  25%|██▌       | 321/1268 [00:10<00:19, 48.49it/s]
Running loglikelihood requests:  30%|███       | 385/1268 [00:11<00:16, 53.84it/s]
Running loglikelihood requests:  35%|███▌      | 449/1268 [00:12<00:14, 58.11it/s]
Running loglikelihood requests:  40%|████      | 513/1268 [00:13<00:12, 61.29it/s]
Running loglikelihood requests:  46%|████▌     | 577/1268 [00:14<00:10, 63.68it/s]
Running loglikelihood requests:  51%|█████     | 641/1268 [00:15<00:09, 65.74it/s]
Running loglikelihood requests:  56%|█████▌    | 705/1268 [00:16<00:08, 67.27it/s]
Running loglikelihood requests:  61%|██████    | 769/1268 [00:17<00:07, 68.46it/s]
Running loglikelihood requests:  66%|██████▌   | 833/1268 [00:18<00:06, 69.39it/s]
Running loglikelihood requests:  71%|███████   | 897/1268 [00:19<00:05, 70.31it/s]
Running loglikelihood requests:  76%|███████▌  | 961/1268 [00:19<00:04, 70.90it/s]
Running loglikelihood requests:  81%|████████  | 1025/1268 [00:20<00:03, 71.46it/s]
Running loglikelihood requests:  86%|████████▌ | 1089/1268 [00:21<00:02, 72.38it/s]
Running loglikelihood requests:  91%|█████████ | 1153/1268 [00:22<00:01, 73.12it/s]
Running loglikelihood requests:  96%|█████████▌| 1217/1268 [00:23<00:00, 78.32it/s]
Running loglikelihood requests: 100%|██████████| 1268/1268 [00:23<00:00, 54.65it/s]
n136-128-154:1711815:1712056 [1] NCCL INFO Channel 00/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711815:1712056 [1] NCCL INFO Channel 01/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711815:1712056 [1] NCCL INFO Channel 02/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711815:1712056 [1] NCCL INFO Channel 03/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711815:1712056 [1] NCCL INFO Channel 04/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711815:1712056 [1] NCCL INFO Channel 05/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711815:1712056 [1] NCCL INFO Channel 06/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711815:1712056 [1] NCCL INFO Channel 07/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711815:1712056 [1] NCCL INFO Channel 08/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711815:1712056 [1] NCCL INFO Channel 09/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711815:1712056 [1] NCCL INFO Channel 10/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711815:1712056 [1] NCCL INFO Channel 11/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711815:1712056 [1] NCCL INFO Channel 12/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711815:1712056 [1] NCCL INFO Channel 13/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711815:1712056 [1] NCCL INFO Channel 14/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711815:1712056 [1] NCCL INFO Channel 15/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711815:1712056 [1] NCCL INFO Channel 16/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711815:1712056 [1] NCCL INFO Channel 17/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711815:1712056 [1] NCCL INFO Channel 18/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711815:1712056 [1] NCCL INFO Channel 19/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711815:1712056 [1] NCCL INFO Channel 20/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711815:1712056 [1] NCCL INFO Channel 21/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711815:1712056 [1] NCCL INFO Channel 22/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711815:1712056 [1] NCCL INFO Channel 23/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711815:1712056 [1] NCCL INFO Channel 24/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711815:1712056 [1] NCCL INFO Channel 25/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711815:1712056 [1] NCCL INFO Channel 26/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711815:1712056 [1] NCCL INFO Channel 27/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711815:1712056 [1] NCCL INFO Channel 28/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711815:1712056 [1] NCCL INFO Channel 29/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711815:1712056 [1] NCCL INFO Channel 30/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1711815:1712056 [1] NCCL INFO Channel 31/1 : 1[2] -> 0[1] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:1711815:1712062 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1711815:1712062 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1711815:1712062 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1711815:1712062 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1711815:1712062 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1711815:1712062 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1711815:1712026 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-06:03:49:40 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: auto (64)
|  Tasks   |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|----------|------:|------|-----:|------|---|-----:|---|-----:|
|winogrande|      1|none  |     5|acc   |↑  |0.6851|±  |0.0131|

[rank0]:[W1206 03:49:41.739800330 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:1711814:1712119 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1711814:1712119 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1711814:1712119 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1711814:1712028 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:1711814:1712119 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1711814:1712119 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1711814:1712119 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1711815:1712026 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:1711815:1712062 [1] NCCL INFO comm 0x153b72b0 rank 1 nranks 2 cudaDev 1 busId 4a000 - Abort COMPLETE
n136-128-154:1711814:1712119 [0] NCCL INFO comm 0x14c61800 rank 0 nranks 2 cudaDev 0 busId 16000 - Abort COMPLETE
任务 winogrande 评估完成！

==================================================
开始评估：任务=gsm8k | 少样本数=4 | 模型=Llama-3.1-8B-quantization-fg
输出路径：results2/Llama-3.1-8B-quantization-fg/fg/gsm8k.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-06:03:51:04 INFO     [__main__:440] Selected Tasks: ['gsm8k']
2025-12-06:03:51:04 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-06:03:51:04 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-06:03:51:04 INFO     [__main__:440] Selected Tasks: ['gsm8k']
2025-12-06:03:51:04 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-06:03:51:04 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-06:03:51:06 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-06:03:51:06 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
`torch_dtype` is deprecated! Use `dtype` instead!
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-06:03:51:06 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.22s/it]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.22s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.23s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.23s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.25s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.25s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:04<00:00,  1.15it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:04<00:00,  1.15it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:04<00:00,  1.00s/it]

Loading checkpoint shards: 100%|██████████| 4/4 [00:04<00:00,  1.00s/it]
2025-12-06:03:51:21 INFO     [evaluator:290] gsm8k: Using gen_kwargs: {'until': ['Question:', '</s>', '<|im_end|>'], 'do_sample': False, 'temperature': 0.0}
2025-12-06:03:51:21 WARNING  [evaluator:309] Overwriting default num_fewshot of gsm8k from 5 to 4
2025-12-06:03:51:21 INFO     [api.task:434] Building contexts for gsm8k on rank 1...

  0%|          | 0/659 [00:00<?, ?it/s]
  6%|▌         | 41/659 [00:00<00:01, 406.58it/s]
 13%|█▎        | 83/659 [00:00<00:01, 411.46it/s]
 19%|█▉        | 125/659 [00:00<00:01, 411.57it/s]2025-12-06:03:51:21 INFO     [evaluator:290] gsm8k: Using gen_kwargs: {'until': ['Question:', '</s>', '<|im_end|>'], 'do_sample': False, 'temperature': 0.0}
2025-12-06:03:51:21 WARNING  [evaluator:309] Overwriting default num_fewshot of gsm8k from 5 to 4
2025-12-06:03:51:21 INFO     [api.task:434] Building contexts for gsm8k on rank 0...

 25%|██▌       | 167/659 [00:00<00:01, 407.82it/s]
  0%|          | 0/660 [00:00<?, ?it/s]
 32%|███▏      | 209/659 [00:00<00:01, 408.37it/s]
  6%|▌         | 40/660 [00:00<00:01, 399.64it/s]
 38%|███▊      | 250/659 [00:00<00:01, 406.79it/s]
 13%|█▎        | 83/660 [00:00<00:01, 413.15it/s]
 44%|████▍     | 291/659 [00:00<00:00, 405.61it/s]
 19%|█▉        | 125/660 [00:00<00:01, 414.00it/s]
 51%|█████     | 333/659 [00:00<00:00, 407.93it/s]
 25%|██▌       | 168/660 [00:00<00:01, 417.01it/s]
 57%|█████▋    | 374/659 [00:00<00:00, 407.88it/s]
 32%|███▏      | 211/660 [00:00<00:01, 420.32it/s]
 63%|██████▎   | 416/659 [00:01<00:00, 409.28it/s]
 38%|███▊      | 254/660 [00:00<00:00, 415.01it/s]
 69%|██████▉   | 457/659 [00:01<00:00, 407.18it/s]
 45%|████▌     | 297/660 [00:00<00:00, 416.99it/s]
 76%|███████▌  | 499/659 [00:01<00:00, 410.30it/s]
 52%|█████▏    | 340/660 [00:00<00:00, 418.88it/s]
 82%|████████▏ | 541/659 [00:01<00:00, 412.08it/s]
 58%|█████▊    | 383/660 [00:00<00:00, 420.23it/s]
 88%|████████▊ | 583/659 [00:01<00:00, 410.73it/s]
 65%|██████▍   | 426/660 [00:01<00:00, 421.83it/s]
 95%|█████████▍| 625/659 [00:01<00:00, 412.58it/s]
 71%|███████   | 469/660 [00:01<00:00, 421.27it/s]
100%|██████████| 659/659 [00:01<00:00, 410.11it/s]

 78%|███████▊  | 512/660 [00:01<00:00, 421.19it/s]
 84%|████████▍ | 555/660 [00:01<00:00, 421.73it/s]
 91%|█████████ | 598/660 [00:01<00:00, 421.94it/s]
 97%|█████████▋| 641/660 [00:01<00:00, 422.66it/s]
100%|██████████| 660/660 [00:01<00:00, 419.67it/s]
n136-128-154:1712228:1712228 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1712228:1712228 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1712228:1712228 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:1712228:1712228 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:1712228:1712228 [0] NCCL INFO Comm config Blocking set to 1
n136-128-154:1712229:1712229 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:1712229:1712229 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1712229:1712229 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1712229:1712229 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:1712229:1712229 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:1712228:1712563 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:1712228:1712563 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:1712229:1712564 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:1712229:1712564 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:1712228:1712563 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1712228:1712563 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1712229:1712564 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1712229:1712564 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1712228:1712563 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:1712229:1712564 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:1712228:1712563 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:1712228:1712563 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:1712228:1712563 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:1712229:1712564 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:1712229:1712564 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:1712229:1712564 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:1712228:1712563 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:1712228:1712563 [0] NCCL INFO Using network IB
n136-128-154:1712229:1712564 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:1712229:1712564 [1] NCCL INFO Using network IB
n136-128-154:1712229:1712564 [1] NCCL INFO ncclCommInitRankConfig comm 0xd8742a0 rank 1 nranks 2 cudaDev 1 nvmlDev 2 busId 4a000 commId 0x3e34cbe80a35aca4 - Init START
n136-128-154:1712228:1712563 [0] NCCL INFO ncclCommInitRankConfig comm 0x12f077a0 rank 0 nranks 2 cudaDev 0 nvmlDev 1 busId 16000 commId 0x3e34cbe80a35aca4 - Init START
n136-128-154:1712229:1712564 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:1712228:1712563 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:1712228:1712563 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:1712228:1712563 [0] NCCL INFO Retrieving state for IB
n136-128-154:1712228:1712563 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:1712228:1712563 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:1712229:1712564 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:1712229:1712564 [1] NCCL INFO Retrieving state for IB
n136-128-154:1712229:1712564 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:1712228:1712563 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:1712229:1712564 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:1712229:1712564 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:1712228:1712563 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:1712229:1712564 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:1712228:1712563 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:1712229:1712564 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:1712228:1712563 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1712228:1712563 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1712228:1712563 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1712228:1712563 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1712229:1712564 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1712229:1712564 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1712229:1712564 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1712229:1712564 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1712228:1712563 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:1712228:1712563 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:1712228:1712563 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:1712228:1712563 [0] NCCL INFO               + PCI[24.0] - PCI/0-14000 (1000c01010de13b8)
n136-128-154:1712228:1712563 [0] NCCL INFO                             + PCI[24.0] - GPU/0-16000 (0)
n136-128-154:1712228:1712563 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1712228:1712563 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:1712228:1712563 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:1712228:1712563 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:1712228:1712563 [0] NCCL INFO               + PCI[24.0] - PCI/0-48000 (1000c01010de13b8)
n136-128-154:1712228:1712563 [0] NCCL INFO                             + PCI[24.0] - GPU/0-4a000 (1)
n136-128-154:1712228:1712563 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1712229:1712564 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:1712228:1712563 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:1712229:1712564 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:1712228:1712563 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:1712228:1712563 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:1712229:1712564 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:1712228:1712563 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:1712229:1712564 [1] NCCL INFO               + PCI[24.0] - PCI/0-14000 (1000c01010de13b8)
n136-128-154:1712228:1712563 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:1712229:1712564 [1] NCCL INFO                             + PCI[24.0] - GPU/0-16000 (0)
n136-128-154:1712228:1712563 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:1712229:1712564 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1712228:1712563 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:1712229:1712564 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:1712228:1712563 [0] NCCL INFO ==========================================
n136-128-154:1712229:1712564 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:1712229:1712564 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:1712229:1712564 [1] NCCL INFO               + PCI[24.0] - PCI/0-48000 (1000c01010de13b8)
n136-128-154:1712228:1712563 [0] NCCL INFO GPU/0-16000 :GPU/0-16000 (0/5000.0/LOC) GPU/0-4a000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1712229:1712564 [1] NCCL INFO                             + PCI[24.0] - GPU/0-4a000 (1)
n136-128-154:1712229:1712564 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1712228:1712563 [0] NCCL INFO GPU/0-4a000 :GPU/0-16000 (2/240.0/NVL) GPU/0-4a000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1712229:1712564 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:1712229:1712564 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:1712229:1712564 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:1712229:1712564 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:1712228:1712563 [0] NCCL INFO Setting affinity for GPU 1 to 1-31,65-95
n136-128-154:1712229:1712564 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:1712229:1712564 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:1712229:1712564 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:1712229:1712564 [1] NCCL INFO ==========================================
n136-128-154:1712229:1712564 [1] NCCL INFO GPU/0-16000 :GPU/0-16000 (0/5000.0/LOC) GPU/0-4a000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1712229:1712564 [1] NCCL INFO GPU/0-4a000 :GPU/0-16000 (2/240.0/NVL) GPU/0-4a000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1712229:1712564 [1] NCCL INFO Setting affinity for GPU 2 to 1-31,65-95
n136-128-154:1712228:1712563 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:1712228:1712563 [0] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1712228:1712563 [0] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1712228:1712563 [0] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1712228:1712563 [0] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1712228:1712563 [0] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1712228:1712563 [0] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1712228:1712563 [0] NCCL INFO  6 : GPU/0-16000 GPU/0-4a000
n136-128-154:1712228:1712563 [0] NCCL INFO  7 : GPU/0-16000 GPU/0-4a000
n136-128-154:1712228:1712563 [0] NCCL INFO  8 : GPU/0-16000 GPU/0-4a000
n136-128-154:1712228:1712563 [0] NCCL INFO  9 : GPU/0-16000 GPU/0-4a000
n136-128-154:1712228:1712563 [0] NCCL INFO 10 : GPU/0-16000 GPU/0-4a000
n136-128-154:1712228:1712563 [0] NCCL INFO 11 : GPU/0-16000 GPU/0-4a000
n136-128-154:1712228:1712563 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:1712228:1712563 [0] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1712228:1712563 [0] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1712228:1712563 [0] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1712228:1712563 [0] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1712228:1712563 [0] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1712228:1712563 [0] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1712228:1712563 [0] NCCL INFO  6 : GPU/0-4a000 GPU/0-16000
n136-128-154:1712228:1712563 [0] NCCL INFO  7 : GPU/0-4a000 GPU/0-16000
n136-128-154:1712228:1712563 [0] NCCL INFO  8 : GPU/0-4a000 GPU/0-16000
n136-128-154:1712228:1712563 [0] NCCL INFO  9 : GPU/0-4a000 GPU/0-16000
n136-128-154:1712228:1712563 [0] NCCL INFO 10 : GPU/0-4a000 GPU/0-16000
n136-128-154:1712228:1712563 [0] NCCL INFO 11 : GPU/0-4a000 GPU/0-16000
n136-128-154:1712229:1712564 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:1712229:1712564 [1] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1712229:1712564 [1] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1712229:1712564 [1] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1712229:1712564 [1] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1712229:1712564 [1] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1712229:1712564 [1] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1712229:1712564 [1] NCCL INFO  6 : GPU/0-16000 GPU/0-4a000
n136-128-154:1712229:1712564 [1] NCCL INFO  7 : GPU/0-16000 GPU/0-4a000
n136-128-154:1712229:1712564 [1] NCCL INFO  8 : GPU/0-16000 GPU/0-4a000
n136-128-154:1712229:1712564 [1] NCCL INFO  9 : GPU/0-16000 GPU/0-4a000
n136-128-154:1712229:1712564 [1] NCCL INFO 10 : GPU/0-16000 GPU/0-4a000
n136-128-154:1712229:1712564 [1] NCCL INFO 11 : GPU/0-16000 GPU/0-4a000
n136-128-154:1712229:1712564 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:1712229:1712564 [1] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1712229:1712564 [1] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1712229:1712564 [1] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1712229:1712564 [1] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1712229:1712564 [1] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1712229:1712564 [1] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1712229:1712564 [1] NCCL INFO  6 : GPU/0-4a000 GPU/0-16000
n136-128-154:1712229:1712564 [1] NCCL INFO  7 : GPU/0-4a000 GPU/0-16000
n136-128-154:1712229:1712564 [1] NCCL INFO  8 : GPU/0-4a000 GPU/0-16000
n136-128-154:1712229:1712564 [1] NCCL INFO  9 : GPU/0-4a000 GPU/0-16000
n136-128-154:1712229:1712564 [1] NCCL INFO 10 : GPU/0-4a000 GPU/0-16000
n136-128-154:1712229:1712564 [1] NCCL INFO 11 : GPU/0-4a000 GPU/0-16000
n136-128-154:1712228:1712563 [0] NCCL INFO comm 0x12f077a0 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:1712229:1712564 [1] NCCL INFO comm 0xd8742a0 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:1712228:1712563 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:1712228:1712563 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:1712229:1712564 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:1712228:1712563 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:1712229:1712564 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:1712228:1712563 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:1712229:1712564 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:1712228:1712563 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:1712229:1712564 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:1712228:1712563 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:1712229:1712564 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:1712228:1712563 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:1712229:1712564 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:1712228:1712563 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:1712229:1712564 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:1712228:1712563 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:1712229:1712564 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:1712228:1712563 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:1712229:1712564 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:1712228:1712563 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:1712229:1712564 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:1712228:1712563 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:1712229:1712564 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:1712228:1712563 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:1712229:1712564 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:1712228:1712563 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:1712229:1712564 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:1712228:1712563 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:1712229:1712564 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:1712228:1712563 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:1712229:1712564 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:1712228:1712563 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:1712229:1712564 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:1712228:1712563 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:1712229:1712564 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:1712228:1712563 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:1712229:1712564 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:1712228:1712563 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:1712229:1712564 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:1712228:1712563 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:1712229:1712564 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:1712228:1712563 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:1712229:1712564 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:1712228:1712563 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:1712229:1712564 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:1712228:1712563 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:1712229:1712564 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:1712229:1712564 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:1712228:1712563 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:1712228:1712563 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:1712228:1712563 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:1712229:1712564 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:1712228:1712563 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:1712229:1712564 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:1712228:1712563 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:1712229:1712564 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:1712228:1712563 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:1712229:1712564 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:1712228:1712563 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:1712229:1712564 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:1712228:1712563 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:1712229:1712564 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:1712228:1712563 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:1712229:1712564 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:1712228:1712563 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:1712228:1712563 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:1712228:1712563 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:1712228:1712563 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:1712228:1712563 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:1712228:1712563 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:1712228:1712563 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:1712228:1712563 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:1712228:1712563 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:1712228:1712563 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:1712228:1712563 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:1712228:1712563 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:1712228:1712563 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:1712228:1712563 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:1712228:1712563 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:1712228:1712563 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:1712228:1712563 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:1712228:1712563 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:1712228:1712563 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:1712228:1712563 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:1712228:1712563 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:1712228:1712563 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:1712228:1712563 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:1712228:1712563 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:1712228:1712563 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:1712228:1712563 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:1712228:1712563 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:1712228:1712563 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:1712228:1712563 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:1712228:1712563 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:1712228:1712563 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:1712228:1712563 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:1712228:1712563 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:1712228:1712563 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:1712228:1712563 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:1712228:1712563 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:1712228:1712563 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:1712228:1712563 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:1712228:1712563 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:1712228:1712563 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:1712228:1712563 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:1712229:1712564 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:1712229:1712564 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:1712229:1712564 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:1712229:1712564 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:1712229:1712564 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:1712229:1712564 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:1712229:1712564 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:1712229:1712564 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:1712229:1712564 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:1712229:1712564 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:1712229:1712564 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:1712229:1712564 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:1712229:1712564 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:1712229:1712564 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:1712229:1712564 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:1712229:1712564 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:1712229:1712564 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:1712229:1712564 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:1712229:1712564 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:1712228:1712563 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:1712228:1712563 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:1712228:1712576 [0] NCCL INFO [Proxy Service] Device 0 CPU core 66
n136-128-154:1712228:1712577 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 68
n136-128-154:1712229:1712564 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:1712229:1712578 [1] NCCL INFO [Proxy Service] Device 1 CPU core 69
n136-128-154:1712229:1712579 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 74
n136-128-154:1712229:1712564 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:1712229:1712564 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:1712228:1712563 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:1712228:1712563 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:1712228:1712563 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:1712229:1712564 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:1712229:1712564 [1] NCCL INFO ncclCommInitRankConfig comm 0xd8742a0 rank 1 nranks 2 cudaDev 1 nvmlDev 2 busId 4a000 commId 0x3e34cbe80a35aca4 - Init COMPLETE
n136-128-154:1712229:1712564 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.63 (kernels 0.24, alloc 0.21, bootstrap 0.00, allgathers 0.00, topo 0.07, graphs 0.00, connections 0.06, rest 0.05)
n136-128-154:1712228:1712563 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:1712228:1712563 [0] NCCL INFO ncclCommInitRankConfig comm 0x12f077a0 rank 0 nranks 2 cudaDev 0 nvmlDev 1 busId 16000 commId 0x3e34cbe80a35aca4 - Init COMPLETE
n136-128-154:1712228:1712563 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.63 (kernels 0.25, alloc 0.21, bootstrap 0.00, allgathers 0.00, topo 0.07, graphs 0.00, connections 0.06, rest 0.04)
n136-128-154:1712229:1712580 [1] NCCL INFO Channel 00/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712228:1712581 [0] NCCL INFO Channel 00/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1712229:1712580 [1] NCCL INFO Channel 01/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712228:1712581 [0] NCCL INFO Channel 01/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1712229:1712580 [1] NCCL INFO Channel 02/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712228:1712581 [0] NCCL INFO Channel 02/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1712229:1712580 [1] NCCL INFO Channel 03/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712228:1712581 [0] NCCL INFO Channel 03/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1712229:1712580 [1] NCCL INFO Channel 04/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712228:1712581 [0] NCCL INFO Channel 04/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1712229:1712580 [1] NCCL INFO Channel 05/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712228:1712581 [0] NCCL INFO Channel 05/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1712229:1712580 [1] NCCL INFO Channel 06/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712228:1712581 [0] NCCL INFO Channel 06/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1712229:1712580 [1] NCCL INFO Channel 07/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712228:1712581 [0] NCCL INFO Channel 07/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1712229:1712580 [1] NCCL INFO Channel 08/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712228:1712581 [0] NCCL INFO Channel 08/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1712229:1712580 [1] NCCL INFO Channel 09/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712228:1712581 [0] NCCL INFO Channel 09/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1712229:1712580 [1] NCCL INFO Channel 10/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712228:1712581 [0] NCCL INFO Channel 10/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1712229:1712580 [1] NCCL INFO Channel 11/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712228:1712581 [0] NCCL INFO Channel 11/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1712229:1712580 [1] NCCL INFO Channel 12/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712228:1712581 [0] NCCL INFO Channel 12/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1712229:1712580 [1] NCCL INFO Channel 13/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712228:1712581 [0] NCCL INFO Channel 13/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1712229:1712580 [1] NCCL INFO Channel 14/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712228:1712581 [0] NCCL INFO Channel 14/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1712229:1712580 [1] NCCL INFO Channel 15/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712228:1712581 [0] NCCL INFO Channel 15/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1712229:1712580 [1] NCCL INFO Channel 16/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712228:1712581 [0] NCCL INFO Channel 16/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1712229:1712580 [1] NCCL INFO Channel 17/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712228:1712581 [0] NCCL INFO Channel 17/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1712229:1712580 [1] NCCL INFO Channel 18/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712228:1712581 [0] NCCL INFO Channel 18/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1712229:1712580 [1] NCCL INFO Channel 19/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712228:1712581 [0] NCCL INFO Channel 19/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1712229:1712580 [1] NCCL INFO Channel 20/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712228:1712581 [0] NCCL INFO Channel 20/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1712229:1712580 [1] NCCL INFO Channel 21/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712228:1712581 [0] NCCL INFO Channel 21/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1712229:1712580 [1] NCCL INFO Channel 22/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712228:1712581 [0] NCCL INFO Channel 22/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1712229:1712580 [1] NCCL INFO Channel 23/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712228:1712581 [0] NCCL INFO Channel 23/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1712228:1712581 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:1712229:1712580 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-06:03:51:24 INFO     [evaluator:559] Running generate_until requests
2025-12-06:03:51:24 INFO     [evaluator:559] Running generate_until requests

Running generate_until requests:   0%|          | 0/660 [00:00<?, ?it/s]Passed argument batch_size = auto. Detecting largest batch size
Passed argument batch_size = auto. Detecting largest batch size
Determined Largest batch size: 1
Determined Largest batch size: 1
The following generation flags are not valid and may be ignored: ['temperature', 'top_p']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
The following generation flags are not valid and may be ignored: ['temperature', 'top_p']. Set `TRANSFORMERS_VERBOSITY=info` for more details.

Running generate_until requests:   0%|          | 1/660 [02:27<26:57:53, 147.30s/it]
Running generate_until requests:   0%|          | 2/660 [02:33<11:46:46, 64.45s/it] 
Running generate_until requests:   0%|          | 3/660 [02:40<6:55:41, 37.96s/it] 
Running generate_until requests:   1%|          | 4/660 [02:42<4:20:51, 23.86s/it]
Running generate_until requests:   1%|          | 5/660 [02:48<3:11:46, 17.57s/it]
Running generate_until requests:   1%|          | 6/660 [02:55<2:30:13, 13.78s/it]
Running generate_until requests:   1%|          | 7/660 [03:01<2:03:44, 11.37s/it]
Running generate_until requests:   1%|          | 8/660 [03:08<1:46:16,  9.78s/it]
Running generate_until requests:   1%|▏         | 9/660 [03:14<1:34:31,  8.71s/it]
Running generate_until requests:   2%|▏         | 10/660 [03:20<1:26:35,  7.99s/it]
Running generate_until requests:   2%|▏         | 11/660 [03:27<1:21:08,  7.50s/it]
Running generate_until requests:   2%|▏         | 12/660 [03:33<1:17:34,  7.18s/it]
Running generate_until requests:   2%|▏         | 13/660 [03:40<1:14:52,  6.94s/it]
Running generate_until requests:   2%|▏         | 14/660 [03:46<1:13:02,  6.78s/it]
Running generate_until requests:   2%|▏         | 15/660 [03:52<1:11:40,  6.67s/it]
Running generate_until requests:   2%|▏         | 16/660 [03:59<1:10:52,  6.60s/it]
Running generate_until requests:   3%|▎         | 17/660 [04:05<1:10:02,  6.54s/it]
Running generate_until requests:   3%|▎         | 18/660 [04:12<1:09:30,  6.50s/it]
Running generate_until requests:   3%|▎         | 19/660 [04:18<1:09:00,  6.46s/it]
Running generate_until requests:   3%|▎         | 20/660 [04:24<1:08:43,  6.44s/it]
Running generate_until requests:   3%|▎         | 21/660 [04:31<1:08:57,  6.47s/it]
Running generate_until requests:   3%|▎         | 22/660 [04:37<1:08:51,  6.48s/it]
Running generate_until requests:   3%|▎         | 23/660 [04:44<1:08:31,  6.45s/it]
Running generate_until requests:   4%|▎         | 24/660 [04:50<1:08:25,  6.45s/it]
Running generate_until requests:   4%|▍         | 25/660 [04:54<58:41,  5.55s/it]  
Running generate_until requests:   4%|▍         | 26/660 [05:00<1:01:15,  5.80s/it]
Running generate_until requests:   4%|▍         | 27/660 [05:06<1:03:09,  5.99s/it]
Running generate_until requests:   4%|▍         | 28/660 [05:13<1:04:17,  6.10s/it]
Running generate_until requests:   4%|▍         | 29/660 [05:15<51:09,  4.87s/it]  
Running generate_until requests:   5%|▍         | 30/660 [05:21<55:51,  5.32s/it]
Running generate_until requests:   5%|▍         | 31/660 [05:28<59:05,  5.64s/it]
Running generate_until requests:   5%|▍         | 32/660 [05:34<1:01:13,  5.85s/it]
Running generate_until requests:   5%|▌         | 33/660 [05:40<1:02:59,  6.03s/it]
Running generate_until requests:   5%|▌         | 34/660 [05:47<1:04:05,  6.14s/it]
Running generate_until requests:   5%|▌         | 35/660 [05:53<1:04:57,  6.24s/it]
Running generate_until requests:   5%|▌         | 36/660 [06:00<1:05:15,  6.28s/it]
Running generate_until requests:   6%|▌         | 37/660 [06:06<1:05:34,  6.32s/it]
Running generate_until requests:   6%|▌         | 38/660 [06:12<1:05:35,  6.33s/it]
Running generate_until requests:   6%|▌         | 39/660 [06:19<1:05:37,  6.34s/it]
Running generate_until requests:   6%|▌         | 40/660 [06:21<52:50,  5.11s/it]  
Running generate_until requests:   6%|▌         | 41/660 [06:27<56:38,  5.49s/it]
Running generate_until requests:   6%|▋         | 42/660 [06:31<50:18,  4.88s/it]
Running generate_until requests:   7%|▋         | 43/660 [06:37<54:49,  5.33s/it]
Running generate_until requests:   7%|▋         | 44/660 [06:44<58:10,  5.67s/it]
Running generate_until requests:   7%|▋         | 45/660 [06:50<1:00:30,  5.90s/it]
Running generate_until requests:   7%|▋         | 46/660 [06:57<1:01:58,  6.06s/it]
Running generate_until requests:   7%|▋         | 47/660 [07:03<1:02:52,  6.15s/it]
Running generate_until requests:   7%|▋         | 48/660 [07:05<49:53,  4.89s/it]  
Running generate_until requests:   7%|▋         | 49/660 [07:11<54:23,  5.34s/it]
Running generate_until requests:   8%|▊         | 50/660 [07:16<53:24,  5.25s/it]
Running generate_until requests:   8%|▊         | 51/660 [07:23<56:43,  5.59s/it]
Running generate_until requests:   8%|▊         | 52/660 [07:29<59:07,  5.84s/it]
Running generate_until requests:   8%|▊         | 53/660 [07:35<1:00:39,  6.00s/it]
Running generate_until requests:   8%|▊         | 54/660 [07:42<1:01:57,  6.13s/it]
Running generate_until requests:   8%|▊         | 55/660 [07:48<1:02:42,  6.22s/it]
Running generate_until requests:   8%|▊         | 56/660 [07:55<1:03:11,  6.28s/it]
Running generate_until requests:   9%|▊         | 57/660 [07:59<55:46,  5.55s/it]  
Running generate_until requests:   9%|▉         | 58/660 [08:05<58:20,  5.82s/it]
Running generate_until requests:   9%|▉         | 59/660 [08:12<1:00:44,  6.06s/it]
Running generate_until requests:   9%|▉         | 60/660 [08:18<1:02:17,  6.23s/it]
Running generate_until requests:   9%|▉         | 61/660 [08:25<1:02:53,  6.30s/it]
Running generate_until requests:   9%|▉         | 62/660 [08:32<1:04:09,  6.44s/it]
Running generate_until requests:  10%|▉         | 63/660 [08:38<1:04:28,  6.48s/it]
Running generate_until requests:  10%|▉         | 64/660 [08:45<1:05:11,  6.56s/it]
Running generate_until requests:  10%|▉         | 65/660 [08:52<1:05:35,  6.61s/it]
Running generate_until requests:  10%|█         | 66/660 [08:58<1:05:31,  6.62s/it]
Running generate_until requests:  10%|█         | 67/660 [09:05<1:05:22,  6.62s/it]
Running generate_until requests:  10%|█         | 68/660 [09:11<1:05:12,  6.61s/it]
Running generate_until requests:  10%|█         | 69/660 [09:18<1:04:53,  6.59s/it]
Running generate_until requests:  11%|█         | 70/660 [09:25<1:04:50,  6.59s/it]
Running generate_until requests:  11%|█         | 71/660 [09:31<1:05:29,  6.67s/it]
Running generate_until requests:  11%|█         | 72/660 [09:38<1:04:30,  6.58s/it]
Running generate_until requests:  11%|█         | 73/660 [09:44<1:03:43,  6.51s/it]
Running generate_until requests:  11%|█         | 74/660 [09:50<1:03:08,  6.46s/it]
Running generate_until requests:  11%|█▏        | 75/660 [09:57<1:02:47,  6.44s/it]
Running generate_until requests:  12%|█▏        | 76/660 [10:03<1:02:27,  6.42s/it]
Running generate_until requests:  12%|█▏        | 77/660 [10:10<1:02:25,  6.42s/it]
Running generate_until requests:  12%|█▏        | 78/660 [10:16<1:02:19,  6.42s/it]
Running generate_until requests:  12%|█▏        | 79/660 [10:23<1:02:11,  6.42s/it]
Running generate_until requests:  12%|█▏        | 80/660 [10:29<1:01:50,  6.40s/it]
Running generate_until requests:  12%|█▏        | 81/660 [10:35<1:01:32,  6.38s/it]
Running generate_until requests:  12%|█▏        | 82/660 [10:42<1:01:20,  6.37s/it]
Running generate_until requests:  13%|█▎        | 83/660 [10:48<1:01:17,  6.37s/it]
Running generate_until requests:  13%|█▎        | 84/660 [10:54<1:01:09,  6.37s/it]
Running generate_until requests:  13%|█▎        | 85/660 [11:01<1:01:13,  6.39s/it]
Running generate_until requests:  13%|█▎        | 86/660 [11:07<1:01:08,  6.39s/it]
Running generate_until requests:  13%|█▎        | 87/660 [11:10<51:12,  5.36s/it]  
Running generate_until requests:  13%|█▎        | 88/660 [11:16<54:02,  5.67s/it]
Running generate_until requests:  13%|█▎        | 89/660 [11:23<56:01,  5.89s/it]
Running generate_until requests:  14%|█▎        | 90/660 [11:29<57:18,  6.03s/it]
Running generate_until requests:  14%|█▍        | 91/660 [11:36<58:24,  6.16s/it]
Running generate_until requests:  14%|█▍        | 92/660 [11:42<58:59,  6.23s/it]
Running generate_until requests:  14%|█▍        | 93/660 [11:49<1:00:16,  6.38s/it]
Running generate_until requests:  14%|█▍        | 94/660 [11:55<1:00:27,  6.41s/it]
Running generate_until requests:  14%|█▍        | 95/660 [12:02<1:00:50,  6.46s/it]
Running generate_until requests:  15%|█▍        | 96/660 [12:09<1:01:47,  6.57s/it]
Running generate_until requests:  15%|█▍        | 97/660 [12:15<1:01:26,  6.55s/it]
Running generate_until requests:  15%|█▍        | 98/660 [12:22<1:01:10,  6.53s/it]
Running generate_until requests:  15%|█▌        | 99/660 [12:28<1:01:14,  6.55s/it]
Running generate_until requests:  15%|█▌        | 100/660 [12:35<1:01:22,  6.58s/it]
Running generate_until requests:  15%|█▌        | 101/660 [12:41<1:00:57,  6.54s/it]
Running generate_until requests:  15%|█▌        | 102/660 [12:48<1:00:52,  6.55s/it]
Running generate_until requests:  16%|█▌        | 103/660 [12:54<1:00:41,  6.54s/it]
Running generate_until requests:  16%|█▌        | 104/660 [13:01<1:00:28,  6.53s/it]
Running generate_until requests:  16%|█▌        | 105/660 [13:07<1:00:09,  6.50s/it]
Running generate_until requests:  16%|█▌        | 106/660 [13:14<59:50,  6.48s/it]  
Running generate_until requests:  16%|█▌        | 107/660 [13:20<59:40,  6.47s/it]
Running generate_until requests:  16%|█▋        | 108/660 [13:27<59:35,  6.48s/it]
Running generate_until requests:  17%|█▋        | 109/660 [13:33<59:24,  6.47s/it]
Running generate_until requests:  17%|█▋        | 110/660 [13:40<59:11,  6.46s/it]
Running generate_until requests:  17%|█▋        | 111/660 [13:46<58:57,  6.44s/it]
Running generate_until requests:  17%|█▋        | 112/660 [13:52<58:47,  6.44s/it]
Running generate_until requests:  17%|█▋        | 113/660 [13:59<58:48,  6.45s/it]
Running generate_until requests:  17%|█▋        | 114/660 [14:05<58:54,  6.47s/it]
Running generate_until requests:  17%|█▋        | 115/660 [14:12<58:43,  6.47s/it]
Running generate_until requests:  18%|█▊        | 116/660 [14:19<59:11,  6.53s/it]
Running generate_until requests:  18%|█▊        | 117/660 [14:25<58:59,  6.52s/it]
Running generate_until requests:  18%|█▊        | 118/660 [14:26<42:22,  4.69s/it]
Running generate_until requests:  18%|█▊        | 119/660 [14:32<47:57,  5.32s/it]
Running generate_until requests:  18%|█▊        | 120/660 [14:39<51:08,  5.68s/it]
Running generate_until requests:  18%|█▊        | 121/660 [14:45<53:22,  5.94s/it]
Running generate_until requests:  18%|█▊        | 122/660 [14:52<55:00,  6.13s/it]
Running generate_until requests:  19%|█▊        | 123/660 [14:59<56:09,  6.27s/it]
Running generate_until requests:  19%|█▉        | 124/660 [15:05<56:59,  6.38s/it]
Running generate_until requests:  19%|█▉        | 125/660 [15:12<57:18,  6.43s/it]
Running generate_until requests:  19%|█▉        | 126/660 [15:18<57:37,  6.47s/it]
Running generate_until requests:  19%|█▉        | 127/660 [15:25<57:44,  6.50s/it]
Running generate_until requests:  19%|█▉        | 128/660 [15:31<57:38,  6.50s/it]
Running generate_until requests:  20%|█▉        | 129/660 [15:38<57:39,  6.52s/it]
Running generate_until requests:  20%|█▉        | 130/660 [15:39<43:40,  4.94s/it]
Running generate_until requests:  20%|█▉        | 131/660 [15:46<47:27,  5.38s/it]
Running generate_until requests:  20%|██        | 132/660 [15:52<50:01,  5.68s/it]
Running generate_until requests:  20%|██        | 133/660 [15:58<51:50,  5.90s/it]
Running generate_until requests:  20%|██        | 134/660 [16:05<53:02,  6.05s/it]
Running generate_until requests:  20%|██        | 135/660 [16:11<53:49,  6.15s/it]
Running generate_until requests:  21%|██        | 136/660 [16:18<54:28,  6.24s/it]
Running generate_until requests:  21%|██        | 137/660 [16:21<45:53,  5.26s/it]
Running generate_until requests:  21%|██        | 138/660 [16:27<48:54,  5.62s/it]
Running generate_until requests:  21%|██        | 139/660 [16:34<50:54,  5.86s/it]
Running generate_until requests:  21%|██        | 140/660 [16:40<52:11,  6.02s/it]
Running generate_until requests:  21%|██▏       | 141/660 [16:46<53:06,  6.14s/it]
Running generate_until requests:  22%|██▏       | 142/660 [16:53<53:41,  6.22s/it]
Running generate_until requests:  22%|██▏       | 143/660 [16:59<54:03,  6.27s/it]
Running generate_until requests:  22%|██▏       | 144/660 [17:06<54:12,  6.30s/it]
Running generate_until requests:  22%|██▏       | 145/660 [17:12<54:18,  6.33s/it]
Running generate_until requests:  22%|██▏       | 146/660 [17:18<54:28,  6.36s/it]
Running generate_until requests:  22%|██▏       | 147/660 [17:25<54:35,  6.38s/it]
Running generate_until requests:  22%|██▏       | 148/660 [17:31<54:41,  6.41s/it]
Running generate_until requests:  23%|██▎       | 149/660 [17:38<54:33,  6.41s/it]
Running generate_until requests:  23%|██▎       | 150/660 [17:40<43:38,  5.13s/it]
Running generate_until requests:  23%|██▎       | 151/660 [17:46<46:55,  5.53s/it]
Running generate_until requests:  23%|██▎       | 152/660 [17:53<49:01,  5.79s/it]
Running generate_until requests:  23%|██▎       | 153/660 [17:58<47:16,  5.59s/it]
Running generate_until requests:  23%|██▎       | 154/660 [18:04<49:13,  5.84s/it]
Running generate_until requests:  23%|██▎       | 155/660 [18:11<50:30,  6.00s/it]
Running generate_until requests:  24%|██▎       | 156/660 [18:17<51:28,  6.13s/it]
Running generate_until requests:  24%|██▍       | 157/660 [18:23<52:10,  6.22s/it]
Running generate_until requests:  24%|██▍       | 158/660 [18:30<52:39,  6.29s/it]
Running generate_until requests:  24%|██▍       | 159/660 [18:36<52:53,  6.34s/it]
Running generate_until requests:  24%|██▍       | 160/660 [18:43<53:00,  6.36s/it]
Running generate_until requests:  24%|██▍       | 161/660 [18:49<53:16,  6.41s/it]
Running generate_until requests:  25%|██▍       | 162/660 [18:56<53:10,  6.41s/it]
Running generate_until requests:  25%|██▍       | 163/660 [19:02<53:04,  6.41s/it]
Running generate_until requests:  25%|██▍       | 164/660 [19:08<52:56,  6.40s/it]
Running generate_until requests:  25%|██▌       | 165/660 [19:10<40:40,  4.93s/it]
Running generate_until requests:  25%|██▌       | 166/660 [19:16<44:19,  5.38s/it]
Running generate_until requests:  25%|██▌       | 167/660 [19:23<46:59,  5.72s/it]
Running generate_until requests:  25%|██▌       | 168/660 [19:29<48:35,  5.93s/it]
Running generate_until requests:  26%|██▌       | 169/660 [19:36<49:45,  6.08s/it]
Running generate_until requests:  26%|██▌       | 170/660 [19:42<50:35,  6.20s/it]
Running generate_until requests:  26%|██▌       | 171/660 [19:49<51:11,  6.28s/it]
Running generate_until requests:  26%|██▌       | 172/660 [19:55<51:22,  6.32s/it]
Running generate_until requests:  26%|██▌       | 173/660 [20:00<46:51,  5.77s/it]
Running generate_until requests:  26%|██▋       | 174/660 [20:06<48:18,  5.96s/it]
Running generate_until requests:  27%|██▋       | 175/660 [20:12<49:16,  6.10s/it]
Running generate_until requests:  27%|██▋       | 176/660 [20:19<50:06,  6.21s/it]
Running generate_until requests:  27%|██▋       | 177/660 [20:25<50:36,  6.29s/it]
Running generate_until requests:  27%|██▋       | 178/660 [20:32<50:55,  6.34s/it]
Running generate_until requests:  27%|██▋       | 179/660 [20:37<46:50,  5.84s/it]
Running generate_until requests:  27%|██▋       | 180/660 [20:43<48:12,  6.03s/it]
Running generate_until requests:  27%|██▋       | 181/660 [20:49<49:10,  6.16s/it]
Running generate_until requests:  28%|██▊       | 182/660 [20:56<49:38,  6.23s/it]
Running generate_until requests:  28%|██▊       | 183/660 [21:02<49:58,  6.29s/it]
Running generate_until requests:  28%|██▊       | 184/660 [21:09<50:09,  6.32s/it]
Running generate_until requests:  28%|██▊       | 185/660 [21:15<50:12,  6.34s/it]
Running generate_until requests:  28%|██▊       | 186/660 [21:22<50:22,  6.38s/it]
Running generate_until requests:  28%|██▊       | 187/660 [21:28<50:22,  6.39s/it]
Running generate_until requests:  28%|██▊       | 188/660 [21:34<50:18,  6.39s/it]
Running generate_until requests:  29%|██▊       | 189/660 [21:41<50:14,  6.40s/it]
Running generate_until requests:  29%|██▉       | 190/660 [21:47<50:14,  6.41s/it]
Running generate_until requests:  29%|██▉       | 191/660 [21:54<50:06,  6.41s/it]
Running generate_until requests:  29%|██▉       | 192/660 [22:00<49:55,  6.40s/it]
Running generate_until requests:  29%|██▉       | 193/660 [22:06<49:46,  6.39s/it]
Running generate_until requests:  29%|██▉       | 194/660 [22:13<49:39,  6.39s/it]
Running generate_until requests:  30%|██▉       | 195/660 [22:19<49:38,  6.41s/it]
Running generate_until requests:  30%|██▉       | 196/660 [22:26<49:38,  6.42s/it]
Running generate_until requests:  30%|██▉       | 197/660 [22:32<49:26,  6.41s/it]
Running generate_until requests:  30%|███       | 198/660 [22:38<49:24,  6.42s/it]
Running generate_until requests:  30%|███       | 199/660 [22:45<49:21,  6.42s/it]
Running generate_until requests:  30%|███       | 200/660 [22:51<49:17,  6.43s/it]
Running generate_until requests:  30%|███       | 201/660 [22:58<49:02,  6.41s/it]
Running generate_until requests:  31%|███       | 202/660 [23:04<48:52,  6.40s/it]
Running generate_until requests:  31%|███       | 203/660 [23:10<48:43,  6.40s/it]
Running generate_until requests:  31%|███       | 204/660 [23:12<37:40,  4.96s/it]
Running generate_until requests:  31%|███       | 205/660 [23:18<40:50,  5.39s/it]
Running generate_until requests:  31%|███       | 206/660 [23:22<36:57,  4.88s/it]
Running generate_until requests:  31%|███▏      | 207/660 [23:29<40:21,  5.34s/it]
Running generate_until requests:  32%|███▏      | 208/660 [23:35<42:37,  5.66s/it]
Running generate_until requests:  32%|███▏      | 209/660 [23:41<44:16,  5.89s/it]
Running generate_until requests:  32%|███▏      | 210/660 [23:48<45:33,  6.08s/it]
Running generate_until requests:  32%|███▏      | 211/660 [23:54<46:18,  6.19s/it]
Running generate_until requests:  32%|███▏      | 212/660 [24:01<46:55,  6.28s/it]
Running generate_until requests:  32%|███▏      | 213/660 [24:07<47:12,  6.34s/it]
Running generate_until requests:  32%|███▏      | 214/660 [24:14<47:26,  6.38s/it]
Running generate_until requests:  33%|███▎      | 215/660 [24:20<47:34,  6.42s/it]
Running generate_until requests:  33%|███▎      | 216/660 [24:27<47:32,  6.42s/it]
Running generate_until requests:  33%|███▎      | 217/660 [24:33<47:25,  6.42s/it]
Running generate_until requests:  33%|███▎      | 218/660 [24:40<47:25,  6.44s/it]
Running generate_until requests:  33%|███▎      | 219/660 [24:46<47:10,  6.42s/it]
Running generate_until requests:  33%|███▎      | 220/660 [24:52<47:00,  6.41s/it]
Running generate_until requests:  33%|███▎      | 221/660 [24:59<46:49,  6.40s/it]
Running generate_until requests:  34%|███▎      | 222/660 [25:05<46:40,  6.39s/it]
Running generate_until requests:  34%|███▍      | 223/660 [25:12<46:30,  6.39s/it]
Running generate_until requests:  34%|███▍      | 224/660 [25:18<46:24,  6.39s/it]
Running generate_until requests:  34%|███▍      | 225/660 [25:24<46:19,  6.39s/it]
Running generate_until requests:  34%|███▍      | 226/660 [25:31<46:18,  6.40s/it]
Running generate_until requests:  34%|███▍      | 227/660 [25:37<46:16,  6.41s/it]
Running generate_until requests:  35%|███▍      | 228/660 [25:44<46:14,  6.42s/it]
Running generate_until requests:  35%|███▍      | 229/660 [25:50<46:14,  6.44s/it]
Running generate_until requests:  35%|███▍      | 230/660 [25:57<46:03,  6.43s/it]
Running generate_until requests:  35%|███▌      | 231/660 [26:03<45:55,  6.42s/it]
Running generate_until requests:  35%|███▌      | 232/660 [26:09<45:42,  6.41s/it]
Running generate_until requests:  35%|███▌      | 233/660 [26:16<45:31,  6.40s/it]
Running generate_until requests:  35%|███▌      | 234/660 [26:22<45:22,  6.39s/it]
Running generate_until requests:  36%|███▌      | 235/660 [26:28<45:20,  6.40s/it]
Running generate_until requests:  36%|███▌      | 236/660 [26:35<45:14,  6.40s/it]
Running generate_until requests:  36%|███▌      | 237/660 [26:41<45:12,  6.41s/it]
Running generate_until requests:  36%|███▌      | 238/660 [26:48<45:08,  6.42s/it]
Running generate_until requests:  36%|███▌      | 239/660 [26:54<45:04,  6.42s/it]
Running generate_until requests:  36%|███▋      | 240/660 [26:57<36:42,  5.24s/it]
Running generate_until requests:  37%|███▋      | 241/660 [27:03<39:08,  5.61s/it]
Running generate_until requests:  37%|███▋      | 242/660 [27:10<40:45,  5.85s/it]
Running generate_until requests:  37%|███▋      | 243/660 [27:16<41:47,  6.01s/it]
Running generate_until requests:  37%|███▋      | 244/660 [27:22<42:30,  6.13s/it]
Running generate_until requests:  37%|███▋      | 245/660 [27:29<42:56,  6.21s/it]
Running generate_until requests:  37%|███▋      | 246/660 [27:35<43:13,  6.27s/it]
Running generate_until requests:  37%|███▋      | 247/660 [27:42<43:29,  6.32s/it]
Running generate_until requests:  38%|███▊      | 248/660 [27:48<43:30,  6.34s/it]
Running generate_until requests:  38%|███▊      | 249/660 [27:54<43:33,  6.36s/it]
Running generate_until requests:  38%|███▊      | 250/660 [28:01<43:29,  6.37s/it]
Running generate_until requests:  38%|███▊      | 251/660 [28:07<43:32,  6.39s/it]
Running generate_until requests:  38%|███▊      | 252/660 [28:14<43:28,  6.39s/it]
Running generate_until requests:  38%|███▊      | 253/660 [28:17<37:54,  5.59s/it]
Running generate_until requests:  38%|███▊      | 254/660 [28:24<39:29,  5.84s/it]
Running generate_until requests:  39%|███▊      | 255/660 [28:30<40:32,  6.01s/it]
Running generate_until requests:  39%|███▉      | 256/660 [28:37<41:17,  6.13s/it]
Running generate_until requests:  39%|███▉      | 257/660 [28:43<41:51,  6.23s/it]
Running generate_until requests:  39%|███▉      | 258/660 [28:49<42:07,  6.29s/it]
Running generate_until requests:  39%|███▉      | 259/660 [28:56<42:22,  6.34s/it]
Running generate_until requests:  39%|███▉      | 260/660 [29:02<42:27,  6.37s/it]
Running generate_until requests:  40%|███▉      | 261/660 [29:09<42:32,  6.40s/it]
Running generate_until requests:  40%|███▉      | 262/660 [29:15<42:25,  6.40s/it]
Running generate_until requests:  40%|███▉      | 263/660 [29:22<42:21,  6.40s/it]
Running generate_until requests:  40%|████      | 264/660 [29:28<42:15,  6.40s/it]
Running generate_until requests:  40%|████      | 265/660 [29:34<42:08,  6.40s/it]
Running generate_until requests:  40%|████      | 266/660 [29:41<42:06,  6.41s/it]
Running generate_until requests:  40%|████      | 267/660 [29:47<42:03,  6.42s/it]
Running generate_until requests:  41%|████      | 268/660 [29:54<42:03,  6.44s/it]
Running generate_until requests:  41%|████      | 269/660 [30:00<41:57,  6.44s/it]
Running generate_until requests:  41%|████      | 270/660 [30:07<41:44,  6.42s/it]
Running generate_until requests:  41%|████      | 271/660 [30:13<41:33,  6.41s/it]
Running generate_until requests:  41%|████      | 272/660 [30:19<41:34,  6.43s/it]
Running generate_until requests:  41%|████▏     | 273/660 [30:26<41:36,  6.45s/it]
Running generate_until requests:  42%|████▏     | 274/660 [30:28<32:12,  5.01s/it]
Running generate_until requests:  42%|████▏     | 275/660 [30:34<35:02,  5.46s/it]
Running generate_until requests:  42%|████▏     | 276/660 [30:37<30:12,  4.72s/it]
Running generate_until requests:  42%|████▏     | 277/660 [30:43<33:19,  5.22s/it]
Running generate_until requests:  42%|████▏     | 278/660 [30:50<35:35,  5.59s/it]
Running generate_until requests:  42%|████▏     | 279/660 [30:56<36:59,  5.83s/it]
Running generate_until requests:  42%|████▏     | 280/660 [31:02<35:43,  5.64s/it]
Running generate_until requests:  43%|████▎     | 281/660 [31:08<37:00,  5.86s/it]
Running generate_until requests:  43%|████▎     | 282/660 [31:14<38:00,  6.03s/it]
Running generate_until requests:  43%|████▎     | 283/660 [31:21<38:36,  6.14s/it]
Running generate_until requests:  43%|████▎     | 284/660 [31:27<38:57,  6.22s/it]
Running generate_until requests:  43%|████▎     | 285/660 [31:33<39:08,  6.26s/it]
Running generate_until requests:  43%|████▎     | 286/660 [31:40<39:16,  6.30s/it]
Running generate_until requests:  43%|████▎     | 287/660 [31:46<39:26,  6.35s/it]
Running generate_until requests:  44%|████▎     | 288/660 [31:53<39:37,  6.39s/it]
Running generate_until requests:  44%|████▍     | 289/660 [31:55<32:15,  5.22s/it]
Running generate_until requests:  44%|████▍     | 290/660 [32:02<34:21,  5.57s/it]
Running generate_until requests:  44%|████▍     | 291/660 [32:08<35:47,  5.82s/it]
Running generate_until requests:  44%|████▍     | 292/660 [32:15<36:51,  6.01s/it]
Running generate_until requests:  44%|████▍     | 293/660 [32:21<37:30,  6.13s/it]
Running generate_until requests:  45%|████▍     | 294/660 [32:27<37:50,  6.20s/it]
Running generate_until requests:  45%|████▍     | 295/660 [32:34<38:02,  6.25s/it]
Running generate_until requests:  45%|████▍     | 296/660 [32:40<38:09,  6.29s/it]
Running generate_until requests:  45%|████▌     | 297/660 [32:46<38:15,  6.33s/it]
Running generate_until requests:  45%|████▌     | 298/660 [32:53<38:23,  6.36s/it]
Running generate_until requests:  45%|████▌     | 299/660 [32:59<38:20,  6.37s/it]
Running generate_until requests:  45%|████▌     | 300/660 [33:06<38:20,  6.39s/it]
Running generate_until requests:  46%|████▌     | 301/660 [33:12<38:18,  6.40s/it]
Running generate_until requests:  46%|████▌     | 302/660 [33:19<38:16,  6.42s/it]
Running generate_until requests:  46%|████▌     | 303/660 [33:25<38:05,  6.40s/it]
Running generate_until requests:  46%|████▌     | 304/660 [33:31<37:56,  6.39s/it]
Running generate_until requests:  46%|████▌     | 305/660 [33:38<37:46,  6.38s/it]
Running generate_until requests:  46%|████▋     | 306/660 [33:44<37:40,  6.38s/it]
Running generate_until requests:  47%|████▋     | 307/660 [33:51<37:32,  6.38s/it]
Running generate_until requests:  47%|████▋     | 308/660 [33:57<37:29,  6.39s/it]
Running generate_until requests:  47%|████▋     | 309/660 [33:59<30:01,  5.13s/it]
Running generate_until requests:  47%|████▋     | 310/660 [34:06<32:12,  5.52s/it]
Running generate_until requests:  47%|████▋     | 311/660 [34:12<33:42,  5.80s/it]
Running generate_until requests:  47%|████▋     | 312/660 [34:18<34:44,  5.99s/it]
Running generate_until requests:  47%|████▋     | 313/660 [34:22<30:12,  5.22s/it]
Running generate_until requests:  48%|████▊     | 314/660 [34:28<32:06,  5.57s/it]
Running generate_until requests:  48%|████▊     | 315/660 [34:35<33:24,  5.81s/it]
Running generate_until requests:  48%|████▊     | 316/660 [34:41<34:19,  5.99s/it]
Running generate_until requests:  48%|████▊     | 317/660 [34:47<35:01,  6.13s/it]
Running generate_until requests:  48%|████▊     | 318/660 [34:54<35:27,  6.22s/it]
Running generate_until requests:  48%|████▊     | 319/660 [35:00<35:44,  6.29s/it]
Running generate_until requests:  48%|████▊     | 320/660 [35:02<27:16,  4.81s/it]
Running generate_until requests:  49%|████▊     | 321/660 [35:08<29:51,  5.28s/it]
Running generate_until requests:  49%|████▉     | 322/660 [35:14<31:37,  5.61s/it]
Running generate_until requests:  49%|████▉     | 323/660 [35:16<25:11,  4.48s/it]
Running generate_until requests:  49%|████▉     | 324/660 [35:23<28:23,  5.07s/it]
Running generate_until requests:  49%|████▉     | 325/660 [35:29<30:33,  5.47s/it]
Running generate_until requests:  49%|████▉     | 326/660 [35:36<31:58,  5.74s/it]
Running generate_until requests:  50%|████▉     | 327/660 [35:42<32:56,  5.94s/it]
Running generate_until requests:  50%|████▉     | 328/660 [35:48<33:37,  6.08s/it]
Running generate_until requests:  50%|████▉     | 329/660 [35:55<34:05,  6.18s/it]
Running generate_until requests:  50%|█████     | 330/660 [36:01<34:25,  6.26s/it]
Running generate_until requests:  50%|█████     | 331/660 [36:08<34:32,  6.30s/it]
Running generate_until requests:  50%|█████     | 332/660 [36:14<34:36,  6.33s/it]
Running generate_until requests:  50%|█████     | 333/660 [36:20<34:39,  6.36s/it]
Running generate_until requests:  51%|█████     | 334/660 [36:27<34:41,  6.38s/it]
Running generate_until requests:  51%|█████     | 335/660 [36:28<26:20,  4.86s/it]
Running generate_until requests:  51%|█████     | 336/660 [36:35<28:41,  5.31s/it]
Running generate_until requests:  51%|█████     | 337/660 [36:41<30:18,  5.63s/it]
Running generate_until requests:  51%|█████     | 338/660 [36:47<31:25,  5.85s/it]
Running generate_until requests:  51%|█████▏    | 339/660 [36:54<32:10,  6.01s/it]
Running generate_until requests:  52%|█████▏    | 340/660 [37:00<32:47,  6.15s/it]
Running generate_until requests:  52%|█████▏    | 341/660 [37:07<33:06,  6.23s/it]
Running generate_until requests:  52%|█████▏    | 342/660 [37:13<33:32,  6.33s/it]
Running generate_until requests:  52%|█████▏    | 343/660 [37:20<33:49,  6.40s/it]
Running generate_until requests:  52%|█████▏    | 344/660 [37:26<34:00,  6.46s/it]
Running generate_until requests:  52%|█████▏    | 345/660 [37:33<34:01,  6.48s/it]
Running generate_until requests:  52%|█████▏    | 346/660 [37:39<33:56,  6.48s/it]
Running generate_until requests:  53%|█████▎    | 347/660 [37:45<32:02,  6.14s/it]
Running generate_until requests:  53%|█████▎    | 348/660 [37:51<32:19,  6.22s/it]
Running generate_until requests:  53%|█████▎    | 349/660 [37:57<32:30,  6.27s/it]
Running generate_until requests:  53%|█████▎    | 350/660 [38:04<32:42,  6.33s/it]
Running generate_until requests:  53%|█████▎    | 351/660 [38:11<33:35,  6.52s/it]
Running generate_until requests:  53%|█████▎    | 352/660 [38:18<33:38,  6.55s/it]
Running generate_until requests:  53%|█████▎    | 353/660 [38:24<33:27,  6.54s/it]
Running generate_until requests:  54%|█████▎    | 354/660 [38:31<33:16,  6.53s/it]
Running generate_until requests:  54%|█████▍    | 355/660 [38:37<33:01,  6.50s/it]
Running generate_until requests:  54%|█████▍    | 356/660 [38:43<32:52,  6.49s/it]
Running generate_until requests:  54%|█████▍    | 357/660 [38:50<32:38,  6.46s/it]
Running generate_until requests:  54%|█████▍    | 358/660 [38:56<32:28,  6.45s/it]
Running generate_until requests:  54%|█████▍    | 359/660 [39:03<32:19,  6.44s/it]
Running generate_until requests:  55%|█████▍    | 360/660 [39:04<24:21,  4.87s/it]
Running generate_until requests:  55%|█████▍    | 361/660 [39:10<26:36,  5.34s/it]
Running generate_until requests:  55%|█████▍    | 362/660 [39:17<28:10,  5.67s/it]
Running generate_until requests:  55%|█████▌    | 363/660 [39:23<29:12,  5.90s/it]
Running generate_until requests:  55%|█████▌    | 364/660 [39:30<29:59,  6.08s/it]
Running generate_until requests:  55%|█████▌    | 365/660 [39:30<22:06,  4.50s/it]
Running generate_until requests:  55%|█████▌    | 366/660 [39:33<18:59,  3.88s/it]
Running generate_until requests:  56%|█████▌    | 367/660 [39:39<22:38,  4.64s/it]
Running generate_until requests:  56%|█████▌    | 368/660 [39:46<25:07,  5.16s/it]
Running generate_until requests:  56%|█████▌    | 369/660 [39:52<26:51,  5.54s/it]
Running generate_until requests:  56%|█████▌    | 370/660 [39:59<27:59,  5.79s/it]
Running generate_until requests:  56%|█████▌    | 371/660 [40:05<28:47,  5.98s/it]
Running generate_until requests:  56%|█████▋    | 372/660 [40:11<29:24,  6.13s/it]
Running generate_until requests:  57%|█████▋    | 373/660 [40:18<29:43,  6.21s/it]
Running generate_until requests:  57%|█████▋    | 374/660 [40:24<29:57,  6.29s/it]
Running generate_until requests:  57%|█████▋    | 375/660 [40:31<30:02,  6.33s/it]
Running generate_until requests:  57%|█████▋    | 376/660 [40:34<25:04,  5.30s/it]
Running generate_until requests:  57%|█████▋    | 377/660 [40:40<26:33,  5.63s/it]
Running generate_until requests:  57%|█████▋    | 378/660 [40:46<27:41,  5.89s/it]
Running generate_until requests:  57%|█████▋    | 379/660 [40:53<28:25,  6.07s/it]
Running generate_until requests:  58%|█████▊    | 380/660 [40:59<28:54,  6.20s/it]
Running generate_until requests:  58%|█████▊    | 381/660 [41:06<29:16,  6.29s/it]
Running generate_until requests:  58%|█████▊    | 382/660 [41:13<29:33,  6.38s/it]
Running generate_until requests:  58%|█████▊    | 383/660 [41:19<29:44,  6.44s/it]
Running generate_until requests:  58%|█████▊    | 384/660 [41:26<29:51,  6.49s/it]
Running generate_until requests:  58%|█████▊    | 385/660 [41:32<29:53,  6.52s/it]
Running generate_until requests:  58%|█████▊    | 386/660 [41:39<29:55,  6.55s/it]
Running generate_until requests:  59%|█████▊    | 387/660 [41:46<29:49,  6.55s/it]
Running generate_until requests:  59%|█████▉    | 388/660 [41:52<29:41,  6.55s/it]
Running generate_until requests:  59%|█████▉    | 389/660 [41:59<29:33,  6.55s/it]
Running generate_until requests:  59%|█████▉    | 390/660 [42:05<29:27,  6.55s/it]
Running generate_until requests:  59%|█████▉    | 391/660 [42:12<29:22,  6.55s/it]
Running generate_until requests:  59%|█████▉    | 392/660 [42:18<29:22,  6.58s/it]
Running generate_until requests:  60%|█████▉    | 393/660 [42:21<23:54,  5.37s/it]
Running generate_until requests:  60%|█████▉    | 394/660 [42:27<25:15,  5.70s/it]
Running generate_until requests:  60%|█████▉    | 395/660 [42:34<26:06,  5.91s/it]
Running generate_until requests:  60%|██████    | 396/660 [42:40<26:41,  6.06s/it]
Running generate_until requests:  60%|██████    | 397/660 [42:45<24:55,  5.69s/it]
Running generate_until requests:  60%|██████    | 398/660 [42:51<25:46,  5.90s/it]
Running generate_until requests:  60%|██████    | 399/660 [42:58<26:19,  6.05s/it]
Running generate_until requests:  61%|██████    | 400/660 [43:02<23:11,  5.35s/it]
Running generate_until requests:  61%|██████    | 401/660 [43:08<24:26,  5.66s/it]
Running generate_until requests:  61%|██████    | 402/660 [43:14<25:20,  5.89s/it]
Running generate_until requests:  61%|██████    | 403/660 [43:21<26:00,  6.07s/it]
Running generate_until requests:  61%|██████    | 404/660 [43:27<26:24,  6.19s/it]
Running generate_until requests:  61%|██████▏   | 405/660 [43:31<22:41,  5.34s/it]
Running generate_until requests:  62%|██████▏   | 406/660 [43:37<23:55,  5.65s/it]
Running generate_until requests:  62%|██████▏   | 407/660 [43:44<24:51,  5.90s/it]
Running generate_until requests:  62%|██████▏   | 408/660 [43:50<25:24,  6.05s/it]
Running generate_until requests:  62%|██████▏   | 409/660 [43:56<25:50,  6.18s/it]
Running generate_until requests:  62%|██████▏   | 410/660 [44:03<26:07,  6.27s/it]
Running generate_until requests:  62%|██████▏   | 411/660 [44:04<19:32,  4.71s/it]
Running generate_until requests:  62%|██████▏   | 412/660 [44:10<21:32,  5.21s/it]
Running generate_until requests:  63%|██████▎   | 413/660 [44:17<22:59,  5.58s/it]
Running generate_until requests:  63%|██████▎   | 414/660 [44:23<23:58,  5.85s/it]
Running generate_until requests:  63%|██████▎   | 415/660 [44:30<24:33,  6.01s/it]
Running generate_until requests:  63%|██████▎   | 416/660 [44:36<24:56,  6.13s/it]
Running generate_until requests:  63%|██████▎   | 417/660 [44:43<25:13,  6.23s/it]
Running generate_until requests:  63%|██████▎   | 418/660 [44:49<25:26,  6.31s/it]
Running generate_until requests:  63%|██████▎   | 419/660 [44:55<25:25,  6.33s/it]
Running generate_until requests:  64%|██████▎   | 420/660 [45:02<25:26,  6.36s/it]
Running generate_until requests:  64%|██████▍   | 421/660 [45:08<25:22,  6.37s/it]
Running generate_until requests:  64%|██████▍   | 422/660 [45:15<25:19,  6.39s/it]
Running generate_until requests:  64%|██████▍   | 423/660 [45:21<25:14,  6.39s/it]
Running generate_until requests:  64%|██████▍   | 424/660 [45:28<25:16,  6.43s/it]
Running generate_until requests:  64%|██████▍   | 425/660 [45:34<25:23,  6.48s/it]
Running generate_until requests:  65%|██████▍   | 426/660 [45:41<25:24,  6.52s/it]
Running generate_until requests:  65%|██████▍   | 427/660 [45:43<19:46,  5.09s/it]
Running generate_until requests:  65%|██████▍   | 428/660 [45:49<21:18,  5.51s/it]
Running generate_until requests:  65%|██████▌   | 429/660 [45:55<22:15,  5.78s/it]
Running generate_until requests:  65%|██████▌   | 430/660 [46:02<22:54,  5.98s/it]
Running generate_until requests:  65%|██████▌   | 431/660 [46:08<23:16,  6.10s/it]
Running generate_until requests:  65%|██████▌   | 432/660 [46:15<23:31,  6.19s/it]
Running generate_until requests:  66%|██████▌   | 433/660 [46:21<23:39,  6.25s/it]
Running generate_until requests:  66%|██████▌   | 434/660 [46:28<23:46,  6.31s/it]
Running generate_until requests:  66%|██████▌   | 435/660 [46:34<23:49,  6.35s/it]
Running generate_until requests:  66%|██████▌   | 436/660 [46:40<23:51,  6.39s/it]
Running generate_until requests:  66%|██████▌   | 437/660 [46:47<23:50,  6.41s/it]
Running generate_until requests:  66%|██████▋   | 438/660 [46:53<23:46,  6.42s/it]
Running generate_until requests:  67%|██████▋   | 439/660 [47:00<23:37,  6.41s/it]
Running generate_until requests:  67%|██████▋   | 440/660 [47:02<18:37,  5.08s/it]
Running generate_until requests:  67%|██████▋   | 441/660 [47:08<19:59,  5.48s/it]
Running generate_until requests:  67%|██████▋   | 442/660 [47:10<15:41,  4.32s/it]
Running generate_until requests:  67%|██████▋   | 443/660 [47:16<17:55,  4.95s/it]
Running generate_until requests:  67%|██████▋   | 444/660 [47:23<19:28,  5.41s/it]
Running generate_until requests:  67%|██████▋   | 445/660 [47:25<15:50,  4.42s/it]
Running generate_until requests:  68%|██████▊   | 446/660 [47:31<17:59,  5.05s/it]
Running generate_until requests:  68%|██████▊   | 447/660 [47:38<19:27,  5.48s/it]
Running generate_until requests:  68%|██████▊   | 448/660 [47:44<20:26,  5.78s/it]
Running generate_until requests:  68%|██████▊   | 449/660 [47:51<21:00,  5.98s/it]
Running generate_until requests:  68%|██████▊   | 450/660 [47:57<21:22,  6.11s/it]
Running generate_until requests:  68%|██████▊   | 451/660 [48:03<21:34,  6.19s/it]
Running generate_until requests:  68%|██████▊   | 452/660 [48:10<21:40,  6.25s/it]
Running generate_until requests:  69%|██████▊   | 453/660 [48:16<21:45,  6.31s/it]
Running generate_until requests:  69%|██████▉   | 454/660 [48:18<16:52,  4.92s/it]
Running generate_until requests:  69%|██████▉   | 455/660 [48:24<18:21,  5.37s/it]
Running generate_until requests:  69%|██████▉   | 456/660 [48:31<19:22,  5.70s/it]
Running generate_until requests:  69%|██████▉   | 457/660 [48:37<20:03,  5.93s/it]
Running generate_until requests:  69%|██████▉   | 458/660 [48:44<20:32,  6.10s/it]
Running generate_until requests:  70%|██████▉   | 459/660 [48:50<20:47,  6.21s/it]
Running generate_until requests:  70%|██████▉   | 460/660 [48:57<20:52,  6.26s/it]
Running generate_until requests:  70%|██████▉   | 461/660 [49:03<20:53,  6.30s/it]
Running generate_until requests:  70%|███████   | 462/660 [49:09<20:52,  6.33s/it]
Running generate_until requests:  70%|███████   | 463/660 [49:16<20:51,  6.36s/it]
Running generate_until requests:  70%|███████   | 464/660 [49:22<20:50,  6.38s/it]
Running generate_until requests:  70%|███████   | 465/660 [49:24<16:32,  5.09s/it]
Running generate_until requests:  71%|███████   | 466/660 [49:31<17:47,  5.50s/it]
Running generate_until requests:  71%|███████   | 467/660 [49:37<18:32,  5.77s/it]
Running generate_until requests:  71%|███████   | 468/660 [49:44<19:06,  5.97s/it]
Running generate_until requests:  71%|███████   | 469/660 [49:50<19:26,  6.11s/it]
Running generate_until requests:  71%|███████   | 470/660 [49:57<19:36,  6.19s/it]
Running generate_until requests:  71%|███████▏  | 471/660 [50:03<19:43,  6.26s/it]
Running generate_until requests:  72%|███████▏  | 472/660 [50:09<19:44,  6.30s/it]
Running generate_until requests:  72%|███████▏  | 473/660 [50:16<19:44,  6.33s/it]
Running generate_until requests:  72%|███████▏  | 474/660 [50:22<19:46,  6.38s/it]
Running generate_until requests:  72%|███████▏  | 475/660 [50:29<19:49,  6.43s/it]
Running generate_until requests:  72%|███████▏  | 476/660 [50:35<19:52,  6.48s/it]
Running generate_until requests:  72%|███████▏  | 477/660 [50:42<19:43,  6.47s/it]
Running generate_until requests:  72%|███████▏  | 478/660 [50:48<19:33,  6.45s/it]
Running generate_until requests:  73%|███████▎  | 479/660 [50:55<19:24,  6.43s/it]
Running generate_until requests:  73%|███████▎  | 480/660 [51:01<19:15,  6.42s/it]
Running generate_until requests:  73%|███████▎  | 481/660 [51:04<16:08,  5.41s/it]
Running generate_until requests:  73%|███████▎  | 482/660 [51:10<16:55,  5.71s/it]
Running generate_until requests:  73%|███████▎  | 483/660 [51:17<17:26,  5.91s/it]
Running generate_until requests:  73%|███████▎  | 484/660 [51:23<17:48,  6.07s/it]
Running generate_until requests:  73%|███████▎  | 485/660 [51:30<18:00,  6.17s/it]
Running generate_until requests:  74%|███████▎  | 486/660 [51:36<18:08,  6.26s/it]
Running generate_until requests:  74%|███████▍  | 487/660 [51:43<18:11,  6.31s/it]
Running generate_until requests:  74%|███████▍  | 488/660 [51:49<18:14,  6.36s/it]
Running generate_until requests:  74%|███████▍  | 489/660 [51:55<18:08,  6.37s/it]
Running generate_until requests:  74%|███████▍  | 490/660 [52:02<18:05,  6.39s/it]
Running generate_until requests:  74%|███████▍  | 491/660 [52:08<17:59,  6.39s/it]
Running generate_until requests:  75%|███████▍  | 492/660 [52:15<17:53,  6.39s/it]
Running generate_until requests:  75%|███████▍  | 493/660 [52:21<17:47,  6.39s/it]
Running generate_until requests:  75%|███████▍  | 494/660 [52:27<17:42,  6.40s/it]
Running generate_until requests:  75%|███████▌  | 495/660 [52:34<17:37,  6.41s/it]
Running generate_until requests:  75%|███████▌  | 496/660 [52:40<17:32,  6.42s/it]
Running generate_until requests:  75%|███████▌  | 497/660 [52:47<17:28,  6.43s/it]
Running generate_until requests:  75%|███████▌  | 498/660 [52:53<17:21,  6.43s/it]
Running generate_until requests:  76%|███████▌  | 499/660 [53:00<17:14,  6.43s/it]
Running generate_until requests:  76%|███████▌  | 500/660 [53:06<17:07,  6.42s/it]
Running generate_until requests:  76%|███████▌  | 501/660 [53:12<17:00,  6.42s/it]
Running generate_until requests:  76%|███████▌  | 502/660 [53:19<16:52,  6.41s/it]
Running generate_until requests:  76%|███████▌  | 503/660 [53:25<16:45,  6.40s/it]
Running generate_until requests:  76%|███████▋  | 504/660 [53:32<16:38,  6.40s/it]
Running generate_until requests:  77%|███████▋  | 505/660 [53:38<16:34,  6.42s/it]
Running generate_until requests:  77%|███████▋  | 506/660 [53:45<16:28,  6.42s/it]
Running generate_until requests:  77%|███████▋  | 507/660 [53:51<16:23,  6.43s/it]
Running generate_until requests:  77%|███████▋  | 508/660 [53:57<16:16,  6.42s/it]
Running generate_until requests:  77%|███████▋  | 509/660 [54:04<16:11,  6.43s/it]
Running generate_until requests:  77%|███████▋  | 510/660 [54:10<16:02,  6.42s/it]
Running generate_until requests:  77%|███████▋  | 511/660 [54:11<11:52,  4.79s/it]
Running generate_until requests:  78%|███████▊  | 512/660 [54:18<12:59,  5.26s/it]
Running generate_until requests:  78%|███████▊  | 513/660 [54:24<13:43,  5.60s/it]
Running generate_until requests:  78%|███████▊  | 514/660 [54:30<14:12,  5.84s/it]
Running generate_until requests:  78%|███████▊  | 515/660 [54:37<14:33,  6.02s/it]
Running generate_until requests:  78%|███████▊  | 516/660 [54:43<14:46,  6.16s/it]
Running generate_until requests:  78%|███████▊  | 517/660 [54:50<14:53,  6.25s/it]
Running generate_until requests:  78%|███████▊  | 518/660 [54:56<14:55,  6.31s/it]
Running generate_until requests:  79%|███████▊  | 519/660 [55:03<14:54,  6.35s/it]
Running generate_until requests:  79%|███████▉  | 520/660 [55:09<14:50,  6.36s/it]
Running generate_until requests:  79%|███████▉  | 521/660 [55:15<14:44,  6.36s/it]
Running generate_until requests:  79%|███████▉  | 522/660 [55:19<12:43,  5.54s/it]
Running generate_until requests:  79%|███████▉  | 523/660 [55:25<13:13,  5.79s/it]
Running generate_until requests:  79%|███████▉  | 524/660 [55:32<13:31,  5.97s/it]
Running generate_until requests:  80%|███████▉  | 525/660 [55:38<13:44,  6.11s/it]
Running generate_until requests:  80%|███████▉  | 526/660 [55:45<13:54,  6.23s/it]
Running generate_until requests:  80%|███████▉  | 527/660 [55:51<13:59,  6.31s/it]
Running generate_until requests:  80%|████████  | 528/660 [55:58<13:58,  6.35s/it]
Running generate_until requests:  80%|████████  | 529/660 [56:04<13:55,  6.38s/it]
Running generate_until requests:  80%|████████  | 530/660 [56:10<13:50,  6.39s/it]
Running generate_until requests:  80%|████████  | 531/660 [56:17<13:46,  6.41s/it]
Running generate_until requests:  81%|████████  | 532/660 [56:23<13:40,  6.41s/it]
Running generate_until requests:  81%|████████  | 533/660 [56:27<11:39,  5.51s/it]
Running generate_until requests:  81%|████████  | 534/660 [56:33<12:07,  5.78s/it]
Running generate_until requests:  81%|████████  | 535/660 [56:40<12:27,  5.98s/it]
Running generate_until requests:  81%|████████  | 536/660 [56:46<12:38,  6.12s/it]
Running generate_until requests:  81%|████████▏ | 537/660 [56:53<12:44,  6.22s/it]
Running generate_until requests:  82%|████████▏ | 538/660 [56:59<12:45,  6.28s/it]
Running generate_until requests:  82%|████████▏ | 539/660 [57:05<12:45,  6.32s/it]
Running generate_until requests:  82%|████████▏ | 540/660 [57:12<12:41,  6.34s/it]
Running generate_until requests:  82%|████████▏ | 541/660 [57:18<12:36,  6.36s/it]
Running generate_until requests:  82%|████████▏ | 542/660 [57:25<12:31,  6.37s/it]
Running generate_until requests:  82%|████████▏ | 543/660 [57:31<12:25,  6.38s/it]
Running generate_until requests:  82%|████████▏ | 544/660 [57:32<09:30,  4.92s/it]
Running generate_until requests:  83%|████████▎ | 545/660 [57:39<10:18,  5.38s/it]
Running generate_until requests:  83%|████████▎ | 546/660 [57:45<10:49,  5.70s/it]
Running generate_until requests:  83%|████████▎ | 547/660 [57:47<08:34,  4.56s/it]
Running generate_until requests:  83%|████████▎ | 548/660 [57:54<09:33,  5.12s/it]
Running generate_until requests:  83%|████████▎ | 549/660 [58:00<10:11,  5.51s/it]
Running generate_until requests:  83%|████████▎ | 550/660 [58:07<10:36,  5.79s/it]
Running generate_until requests:  83%|████████▎ | 551/660 [58:13<10:52,  5.99s/it]
Running generate_until requests:  84%|████████▎ | 552/660 [58:19<11:01,  6.12s/it]
Running generate_until requests:  84%|████████▍ | 553/660 [58:26<11:04,  6.21s/it]
Running generate_until requests:  84%|████████▍ | 554/660 [58:32<11:04,  6.27s/it]
Running generate_until requests:  84%|████████▍ | 555/660 [58:39<11:05,  6.33s/it]
Running generate_until requests:  84%|████████▍ | 556/660 [58:45<11:03,  6.38s/it]
Running generate_until requests:  84%|████████▍ | 557/660 [58:52<11:01,  6.42s/it]
Running generate_until requests:  85%|████████▍ | 558/660 [58:58<10:55,  6.42s/it]
Running generate_until requests:  85%|████████▍ | 559/660 [59:05<10:52,  6.46s/it]
Running generate_until requests:  85%|████████▍ | 560/660 [59:11<10:45,  6.46s/it]
Running generate_until requests:  85%|████████▌ | 561/660 [59:18<10:37,  6.44s/it]
Running generate_until requests:  85%|████████▌ | 562/660 [59:24<10:29,  6.42s/it]
Running generate_until requests:  85%|████████▌ | 563/660 [59:30<10:22,  6.42s/it]
Running generate_until requests:  85%|████████▌ | 564/660 [59:37<10:16,  6.42s/it]
Running generate_until requests:  86%|████████▌ | 565/660 [59:43<10:11,  6.44s/it]
Running generate_until requests:  86%|████████▌ | 566/660 [59:50<10:05,  6.44s/it]
Running generate_until requests:  86%|████████▌ | 567/660 [59:56<09:59,  6.45s/it]
Running generate_until requests:  86%|████████▌ | 568/660 [59:57<07:16,  4.75s/it]
Running generate_until requests:  86%|████████▌ | 569/660 [1:00:03<07:57,  5.25s/it]
Running generate_until requests:  86%|████████▋ | 570/660 [1:00:10<08:23,  5.60s/it]
Running generate_until requests:  87%|████████▋ | 571/660 [1:00:16<08:40,  5.84s/it]
Running generate_until requests:  87%|████████▋ | 572/660 [1:00:17<06:14,  4.26s/it]
Running generate_until requests:  87%|████████▋ | 573/660 [1:00:23<07:08,  4.92s/it]
Running generate_until requests:  87%|████████▋ | 574/660 [1:00:30<07:41,  5.37s/it]
Running generate_until requests:  87%|████████▋ | 575/660 [1:00:36<08:03,  5.69s/it]
Running generate_until requests:  87%|████████▋ | 576/660 [1:00:42<08:15,  5.90s/it]
Running generate_until requests:  87%|████████▋ | 577/660 [1:00:49<08:23,  6.07s/it]
Running generate_until requests:  88%|████████▊ | 578/660 [1:00:55<08:26,  6.18s/it]
Running generate_until requests:  88%|████████▊ | 579/660 [1:01:02<08:27,  6.27s/it]
Running generate_until requests:  88%|████████▊ | 580/660 [1:01:08<08:25,  6.31s/it]
Running generate_until requests:  88%|████████▊ | 581/660 [1:01:15<08:21,  6.35s/it]
Running generate_until requests:  88%|████████▊ | 582/660 [1:01:21<08:16,  6.37s/it]
Running generate_until requests:  88%|████████▊ | 583/660 [1:01:27<08:10,  6.37s/it]
Running generate_until requests:  88%|████████▊ | 584/660 [1:01:32<07:16,  5.75s/it]
Running generate_until requests:  89%|████████▊ | 585/660 [1:01:38<07:25,  5.94s/it]
Running generate_until requests:  89%|████████▉ | 586/660 [1:01:43<06:52,  5.58s/it]
Running generate_until requests:  89%|████████▉ | 587/660 [1:01:49<07:06,  5.84s/it]
Running generate_until requests:  89%|████████▉ | 588/660 [1:01:56<07:14,  6.03s/it]
Running generate_until requests:  89%|████████▉ | 589/660 [1:02:02<07:17,  6.16s/it]
Running generate_until requests:  89%|████████▉ | 590/660 [1:02:09<07:16,  6.23s/it]
Running generate_until requests:  90%|████████▉ | 591/660 [1:02:15<07:14,  6.30s/it]
Running generate_until requests:  90%|████████▉ | 592/660 [1:02:22<07:10,  6.33s/it]
Running generate_until requests:  90%|████████▉ | 593/660 [1:02:28<07:05,  6.35s/it]
Running generate_until requests:  90%|█████████ | 594/660 [1:02:34<06:59,  6.36s/it]
Running generate_until requests:  90%|█████████ | 595/660 [1:02:41<06:53,  6.36s/it]
Running generate_until requests:  90%|█████████ | 596/660 [1:02:47<06:48,  6.39s/it]
Running generate_until requests:  90%|█████████ | 597/660 [1:02:54<06:43,  6.41s/it]
Running generate_until requests:  91%|█████████ | 598/660 [1:03:00<06:36,  6.40s/it]
Running generate_until requests:  91%|█████████ | 599/660 [1:03:01<04:56,  4.86s/it]
Running generate_until requests:  91%|█████████ | 600/660 [1:03:08<05:21,  5.35s/it]
Running generate_until requests:  91%|█████████ | 601/660 [1:03:14<05:35,  5.69s/it]
Running generate_until requests:  91%|█████████ | 602/660 [1:03:21<05:44,  5.93s/it]
Running generate_until requests:  91%|█████████▏| 603/660 [1:03:27<05:52,  6.18s/it]
Running generate_until requests:  92%|█████████▏| 604/660 [1:03:28<04:11,  4.50s/it]
Running generate_until requests:  92%|█████████▏| 605/660 [1:03:28<02:57,  3.23s/it]
Running generate_until requests:  92%|█████████▏| 606/660 [1:03:35<03:47,  4.22s/it]
Running generate_until requests:  92%|█████████▏| 607/660 [1:03:41<04:19,  4.90s/it]
Running generate_until requests:  92%|█████████▏| 608/660 [1:03:44<03:45,  4.33s/it]
Running generate_until requests:  92%|█████████▏| 609/660 [1:03:51<04:14,  4.98s/it]
Running generate_until requests:  92%|█████████▏| 610/660 [1:03:57<04:31,  5.42s/it]
Running generate_until requests:  93%|█████████▎| 611/660 [1:04:04<04:42,  5.76s/it]
Running generate_until requests:  93%|█████████▎| 612/660 [1:04:10<04:47,  5.98s/it]
Running generate_until requests:  93%|█████████▎| 613/660 [1:04:17<04:48,  6.15s/it]
Running generate_until requests:  93%|█████████▎| 614/660 [1:04:23<04:46,  6.23s/it]
Running generate_until requests:  93%|█████████▎| 615/660 [1:04:30<04:43,  6.30s/it]
Running generate_until requests:  93%|█████████▎| 616/660 [1:04:35<04:18,  5.87s/it]
Running generate_until requests:  93%|█████████▎| 617/660 [1:04:41<04:18,  6.01s/it]
Running generate_until requests:  94%|█████████▎| 618/660 [1:04:47<04:18,  6.14s/it]
Running generate_until requests:  94%|█████████▍| 619/660 [1:04:54<04:15,  6.22s/it]
Running generate_until requests:  94%|█████████▍| 620/660 [1:05:00<04:11,  6.29s/it]
Running generate_until requests:  94%|█████████▍| 621/660 [1:05:07<04:06,  6.33s/it]
Running generate_until requests:  94%|█████████▍| 622/660 [1:05:12<03:45,  5.93s/it]
Running generate_until requests:  94%|█████████▍| 623/660 [1:05:18<03:44,  6.06s/it]
Running generate_until requests:  95%|█████████▍| 624/660 [1:05:24<03:41,  6.15s/it]
Running generate_until requests:  95%|█████████▍| 625/660 [1:05:26<02:47,  4.79s/it]
Running generate_until requests:  95%|█████████▍| 626/660 [1:05:32<02:58,  5.26s/it]
Running generate_until requests:  95%|█████████▌| 627/660 [1:05:39<03:04,  5.59s/it]
Running generate_until requests:  95%|█████████▌| 628/660 [1:05:43<02:47,  5.22s/it]
Running generate_until requests:  95%|█████████▌| 629/660 [1:05:50<02:53,  5.58s/it]
Running generate_until requests:  95%|█████████▌| 630/660 [1:05:56<02:54,  5.83s/it]
Running generate_until requests:  96%|█████████▌| 631/660 [1:06:02<02:54,  6.01s/it]
Running generate_until requests:  96%|█████████▌| 632/660 [1:06:09<02:51,  6.11s/it]
Running generate_until requests:  96%|█████████▌| 633/660 [1:06:15<02:47,  6.20s/it]
Running generate_until requests:  96%|█████████▌| 634/660 [1:06:22<02:42,  6.25s/it]
Running generate_until requests:  96%|█████████▌| 635/660 [1:06:28<02:37,  6.29s/it]
Running generate_until requests:  96%|█████████▋| 636/660 [1:06:34<02:31,  6.31s/it]
Running generate_until requests:  97%|█████████▋| 637/660 [1:06:41<02:25,  6.33s/it]
Running generate_until requests:  97%|█████████▋| 638/660 [1:06:47<02:19,  6.35s/it]
Running generate_until requests:  97%|█████████▋| 639/660 [1:06:53<02:13,  6.37s/it]
Running generate_until requests:  97%|█████████▋| 640/660 [1:06:55<01:39,  4.95s/it]
Running generate_until requests:  97%|█████████▋| 641/660 [1:07:02<01:42,  5.39s/it]
Running generate_until requests:  97%|█████████▋| 642/660 [1:07:03<01:15,  4.19s/it]
Running generate_until requests:  97%|█████████▋| 643/660 [1:07:09<01:22,  4.84s/it]
Running generate_until requests:  98%|█████████▊| 644/660 [1:07:16<01:24,  5.31s/it]
Running generate_until requests:  98%|█████████▊| 645/660 [1:07:22<01:24,  5.63s/it]
Running generate_until requests:  98%|█████████▊| 646/660 [1:07:28<01:21,  5.85s/it]
Running generate_until requests:  98%|█████████▊| 647/660 [1:07:31<01:01,  4.73s/it]
Running generate_until requests:  98%|█████████▊| 648/660 [1:07:37<01:02,  5.22s/it]
Running generate_until requests:  98%|█████████▊| 649/660 [1:07:38<00:43,  3.95s/it]
Running generate_until requests:  98%|█████████▊| 650/660 [1:07:44<00:46,  4.68s/it]
Running generate_until requests:  99%|█████████▊| 651/660 [1:07:51<00:46,  5.21s/it]
Running generate_until requests:  99%|█████████▉| 652/660 [1:07:57<00:44,  5.56s/it]
Running generate_until requests:  99%|█████████▉| 653/660 [1:08:03<00:40,  5.82s/it]
Running generate_until requests:  99%|█████████▉| 654/660 [1:08:06<00:28,  4.82s/it]
Running generate_until requests:  99%|█████████▉| 655/660 [1:08:07<00:19,  3.82s/it]
Running generate_until requests:  99%|█████████▉| 656/660 [1:08:10<00:13,  3.41s/it]
Running generate_until requests: 100%|█████████▉| 657/660 [1:08:16<00:12,  4.32s/it]
Running generate_until requests: 100%|█████████▉| 658/660 [1:08:21<00:08,  4.48s/it]
Running generate_until requests: 100%|█████████▉| 659/660 [1:08:28<00:05,  5.05s/it]
Running generate_until requests: 100%|██████████| 660/660 [1:08:34<00:00,  5.45s/it]
Running generate_until requests: 100%|██████████| 660/660 [1:08:34<00:00,  6.23s/it]
n136-128-154:1712229:1717984 [1] NCCL INFO Channel 00/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712229:1717984 [1] NCCL INFO Channel 01/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712229:1717984 [1] NCCL INFO Channel 02/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712229:1717984 [1] NCCL INFO Channel 03/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712229:1717984 [1] NCCL INFO Channel 04/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712229:1717984 [1] NCCL INFO Channel 05/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712229:1717984 [1] NCCL INFO Channel 06/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712229:1717984 [1] NCCL INFO Channel 07/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712229:1717984 [1] NCCL INFO Channel 08/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712229:1717984 [1] NCCL INFO Channel 09/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712229:1717984 [1] NCCL INFO Channel 10/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712229:1717984 [1] NCCL INFO Channel 11/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712229:1717984 [1] NCCL INFO Channel 12/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712229:1717984 [1] NCCL INFO Channel 13/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712229:1717984 [1] NCCL INFO Channel 14/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712229:1717984 [1] NCCL INFO Channel 15/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712229:1717984 [1] NCCL INFO Channel 16/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712229:1717984 [1] NCCL INFO Channel 17/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712229:1717984 [1] NCCL INFO Channel 18/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712229:1717984 [1] NCCL INFO Channel 19/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712229:1717984 [1] NCCL INFO Channel 20/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712229:1717984 [1] NCCL INFO Channel 21/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712229:1717984 [1] NCCL INFO Channel 22/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712229:1717984 [1] NCCL INFO Channel 23/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712229:1717984 [1] NCCL INFO Channel 24/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712229:1717984 [1] NCCL INFO Channel 25/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712229:1717984 [1] NCCL INFO Channel 26/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712229:1717984 [1] NCCL INFO Channel 27/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712229:1717984 [1] NCCL INFO Channel 28/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712229:1717984 [1] NCCL INFO Channel 29/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712229:1717984 [1] NCCL INFO Channel 30/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1712229:1717984 [1] NCCL INFO Channel 31/1 : 1[2] -> 0[1] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:1712229:1717988 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1712229:1717988 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1712229:1717988 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1712229:1717988 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1712229:1717988 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1712229:1717988 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1712229:1712578 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-06:05:00:09 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 4, batch_size: auto
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     4|exact_match|↑  |0.0106|±  |0.0028|
|     |       |strict-match    |     4|exact_match|↑  |0.0008|±  |0.0008|

[rank0]:[W1206 05:00:10.047535372 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:1712228:1718041 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1712228:1718041 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1712228:1718041 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1712228:1712576 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:1712228:1718041 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1712228:1718041 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1712228:1718041 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1712229:1712578 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:1712229:1717988 [1] NCCL INFO comm 0xd8742a0 rank 1 nranks 2 cudaDev 1 busId 4a000 - Abort COMPLETE
n136-128-154:1712228:1718041 [0] NCCL INFO comm 0x12f077a0 rank 0 nranks 2 cudaDev 0 busId 16000 - Abort COMPLETE
任务 gsm8k 评估完成！

==================================================
开始评估：任务=boolq | 少样本数=0 | 模型=Llama-3.1-8B-quantization-fg
输出路径：results2/Llama-3.1-8B-quantization-fg/fg/boolq.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-06:05:01:35 INFO     [__main__:440] Selected Tasks: ['boolq']
2025-12-06:05:01:35 INFO     [__main__:440] Selected Tasks: ['boolq']
2025-12-06:05:01:35 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-06:05:01:35 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-06:05:01:35 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-06:05:01:35 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-06:05:01:37 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-06:05:01:37 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-06:05:01:37 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.12it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.14it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.07it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.09it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.05s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.07s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.31it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.33it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.18it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.20it/s]
2025-12-06:05:01:41 WARNING  [api.task:844] [Task: boolq] metric acc is defined, but aggregation is not. using default aggregation=mean
2025-12-06:05:01:41 WARNING  [api.task:844] [Task: boolq] metric acc is defined, but aggregation is not. using default aggregation=mean
2025-12-06:05:01:41 WARNING  [api.task:856] [Task: boolq] metric acc is defined, but higher_is_better is not. using default higher_is_better=True
2025-12-06:05:01:41 WARNING  [api.task:856] [Task: boolq] metric acc is defined, but higher_is_better is not. using default higher_is_better=True
2025-12-06:05:02:01 WARNING  [evaluator:309] Overwriting default num_fewshot of boolq from None to 0
2025-12-06:05:02:01 INFO     [api.task:434] Building contexts for boolq on rank 1...
2025-12-06:05:02:01 WARNING  [evaluator:309] Overwriting default num_fewshot of boolq from None to 0
2025-12-06:05:02:01 INFO     [api.task:434] Building contexts for boolq on rank 0...

  0%|          | 0/1635 [00:00<?, ?it/s]
  0%|          | 0/1635 [00:00<?, ?it/s]
 17%|█▋        | 283/1635 [00:00<00:00, 2822.96it/s]
 17%|█▋        | 278/1635 [00:00<00:00, 2770.93it/s]
 35%|███▍      | 566/1635 [00:00<00:00, 2825.35it/s]
 34%|███▍      | 562/1635 [00:00<00:00, 2806.12it/s]
 52%|█████▏    | 852/1635 [00:00<00:00, 2838.57it/s]
 52%|█████▏    | 848/1635 [00:00<00:00, 2826.63it/s]
 69%|██████▉   | 1136/1635 [00:00<00:00, 2827.04it/s]
 69%|██████▉   | 1134/1635 [00:00<00:00, 2837.16it/s]
 87%|████████▋ | 1423/1635 [00:00<00:00, 2841.57it/s]
 87%|████████▋ | 1418/1635 [00:00<00:00, 2817.30it/s]
100%|██████████| 1635/1635 [00:00<00:00, 2834.37it/s]

100%|██████████| 1635/1635 [00:00<00:00, 2810.38it/s]
n136-128-154:1718093:1718093 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1718093:1718093 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1718093:1718093 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:1718093:1718093 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:1718093:1718093 [0] NCCL INFO Comm config Blocking set to 1
n136-128-154:1718094:1718094 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:1718094:1718094 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1718094:1718094 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1718094:1718094 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:1718094:1718094 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:1718093:1718328 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:1718093:1718328 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:1718094:1718329 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:1718094:1718329 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:1718093:1718328 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1718093:1718328 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1718093:1718328 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:1718094:1718329 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1718094:1718329 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1718094:1718329 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:1718093:1718328 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:1718093:1718328 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:1718093:1718328 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:1718094:1718329 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:1718094:1718329 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:1718094:1718329 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:1718093:1718328 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:1718093:1718328 [0] NCCL INFO Using network IB
n136-128-154:1718094:1718329 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:1718094:1718329 [1] NCCL INFO Using network IB
n136-128-154:1718093:1718328 [0] NCCL INFO ncclCommInitRankConfig comm 0x1182afd0 rank 0 nranks 2 cudaDev 0 nvmlDev 1 busId 16000 commId 0x2e657754adbae1ea - Init START
n136-128-154:1718094:1718329 [1] NCCL INFO ncclCommInitRankConfig comm 0xf232820 rank 1 nranks 2 cudaDev 1 nvmlDev 2 busId 4a000 commId 0x2e657754adbae1ea - Init START
n136-128-154:1718093:1718328 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:1718094:1718329 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:1718093:1718328 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:1718093:1718328 [0] NCCL INFO Retrieving state for IB
n136-128-154:1718093:1718328 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:1718093:1718328 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:1718094:1718329 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:1718094:1718329 [1] NCCL INFO Retrieving state for IB
n136-128-154:1718094:1718329 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:1718093:1718328 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:1718094:1718329 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:1718093:1718328 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:1718094:1718329 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:1718093:1718328 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:1718094:1718329 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:1718094:1718329 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:1718093:1718328 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1718093:1718328 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1718093:1718328 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1718093:1718328 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1718093:1718328 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:1718093:1718328 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:1718093:1718328 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:1718093:1718328 [0] NCCL INFO               + PCI[24.0] - PCI/0-14000 (1000c01010de13b8)
n136-128-154:1718093:1718328 [0] NCCL INFO                             + PCI[24.0] - GPU/0-16000 (0)
n136-128-154:1718093:1718328 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1718093:1718328 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:1718093:1718328 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:1718093:1718328 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:1718093:1718328 [0] NCCL INFO               + PCI[24.0] - PCI/0-48000 (1000c01010de13b8)
n136-128-154:1718093:1718328 [0] NCCL INFO                             + PCI[24.0] - GPU/0-4a000 (1)
n136-128-154:1718093:1718328 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1718093:1718328 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:1718093:1718328 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:1718093:1718328 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:1718093:1718328 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:1718093:1718328 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:1718093:1718328 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:1718093:1718328 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:1718093:1718328 [0] NCCL INFO ==========================================
n136-128-154:1718093:1718328 [0] NCCL INFO GPU/0-16000 :GPU/0-16000 (0/5000.0/LOC) GPU/0-4a000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1718093:1718328 [0] NCCL INFO GPU/0-4a000 :GPU/0-16000 (2/240.0/NVL) GPU/0-4a000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1718093:1718328 [0] NCCL INFO Setting affinity for GPU 1 to 1-31,65-95
n136-128-154:1718094:1718329 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1718093:1718328 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:1718094:1718329 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1718093:1718328 [0] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718094:1718329 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1718093:1718328 [0] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718094:1718329 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1718093:1718328 [0] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718093:1718328 [0] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718093:1718328 [0] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718093:1718328 [0] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718093:1718328 [0] NCCL INFO  6 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718093:1718328 [0] NCCL INFO  7 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718093:1718328 [0] NCCL INFO  8 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718093:1718328 [0] NCCL INFO  9 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718093:1718328 [0] NCCL INFO 10 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718093:1718328 [0] NCCL INFO 11 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718093:1718328 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:1718093:1718328 [0] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718093:1718328 [0] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718093:1718328 [0] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718093:1718328 [0] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718093:1718328 [0] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718093:1718328 [0] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718093:1718328 [0] NCCL INFO  6 : GPU/0-4a000 GPU/0-16000
n136-128-154:1718093:1718328 [0] NCCL INFO  7 : GPU/0-4a000 GPU/0-16000
n136-128-154:1718093:1718328 [0] NCCL INFO  8 : GPU/0-4a000 GPU/0-16000
n136-128-154:1718093:1718328 [0] NCCL INFO  9 : GPU/0-4a000 GPU/0-16000
n136-128-154:1718093:1718328 [0] NCCL INFO 10 : GPU/0-4a000 GPU/0-16000
n136-128-154:1718093:1718328 [0] NCCL INFO 11 : GPU/0-4a000 GPU/0-16000
n136-128-154:1718094:1718329 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:1718094:1718329 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:1718094:1718329 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:1718094:1718329 [1] NCCL INFO               + PCI[24.0] - PCI/0-14000 (1000c01010de13b8)
n136-128-154:1718094:1718329 [1] NCCL INFO                             + PCI[24.0] - GPU/0-16000 (0)
n136-128-154:1718094:1718329 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1718094:1718329 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:1718094:1718329 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:1718094:1718329 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:1718094:1718329 [1] NCCL INFO               + PCI[24.0] - PCI/0-48000 (1000c01010de13b8)
n136-128-154:1718094:1718329 [1] NCCL INFO                             + PCI[24.0] - GPU/0-4a000 (1)
n136-128-154:1718094:1718329 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1718094:1718329 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:1718094:1718329 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:1718094:1718329 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:1718094:1718329 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:1718094:1718329 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:1718094:1718329 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:1718094:1718329 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:1718094:1718329 [1] NCCL INFO ==========================================
n136-128-154:1718094:1718329 [1] NCCL INFO GPU/0-16000 :GPU/0-16000 (0/5000.0/LOC) GPU/0-4a000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1718094:1718329 [1] NCCL INFO GPU/0-4a000 :GPU/0-16000 (2/240.0/NVL) GPU/0-4a000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1718094:1718329 [1] NCCL INFO Setting affinity for GPU 2 to 1-31,65-95
n136-128-154:1718094:1718329 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:1718094:1718329 [1] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718094:1718329 [1] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718094:1718329 [1] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718094:1718329 [1] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718094:1718329 [1] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718094:1718329 [1] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718094:1718329 [1] NCCL INFO  6 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718094:1718329 [1] NCCL INFO  7 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718094:1718329 [1] NCCL INFO  8 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718094:1718329 [1] NCCL INFO  9 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718094:1718329 [1] NCCL INFO 10 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718094:1718329 [1] NCCL INFO 11 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718094:1718329 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:1718094:1718329 [1] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718094:1718329 [1] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718094:1718329 [1] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718094:1718329 [1] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718094:1718329 [1] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718094:1718329 [1] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718094:1718329 [1] NCCL INFO  6 : GPU/0-4a000 GPU/0-16000
n136-128-154:1718094:1718329 [1] NCCL INFO  7 : GPU/0-4a000 GPU/0-16000
n136-128-154:1718094:1718329 [1] NCCL INFO  8 : GPU/0-4a000 GPU/0-16000
n136-128-154:1718094:1718329 [1] NCCL INFO  9 : GPU/0-4a000 GPU/0-16000
n136-128-154:1718094:1718329 [1] NCCL INFO 10 : GPU/0-4a000 GPU/0-16000
n136-128-154:1718094:1718329 [1] NCCL INFO 11 : GPU/0-4a000 GPU/0-16000
n136-128-154:1718093:1718328 [0] NCCL INFO comm 0x1182afd0 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:1718094:1718329 [1] NCCL INFO comm 0xf232820 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:1718093:1718328 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:1718093:1718328 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:1718093:1718328 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:1718093:1718328 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:1718094:1718329 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:1718093:1718328 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:1718093:1718328 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:1718094:1718329 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:1718093:1718328 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:1718094:1718329 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:1718093:1718328 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:1718094:1718329 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:1718093:1718328 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:1718094:1718329 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:1718093:1718328 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:1718094:1718329 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:1718093:1718328 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:1718094:1718329 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:1718093:1718328 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:1718094:1718329 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:1718093:1718328 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:1718094:1718329 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:1718093:1718328 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:1718094:1718329 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:1718093:1718328 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:1718094:1718329 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:1718093:1718328 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:1718094:1718329 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:1718093:1718328 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:1718093:1718328 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:1718094:1718329 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:1718093:1718328 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:1718094:1718329 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:1718093:1718328 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:1718094:1718329 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:1718093:1718328 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:1718094:1718329 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:1718093:1718328 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:1718094:1718329 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:1718093:1718328 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:1718094:1718329 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:1718093:1718328 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:1718094:1718329 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:1718094:1718329 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:1718094:1718329 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:1718094:1718329 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:1718094:1718329 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:1718094:1718329 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:1718093:1718328 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:1718093:1718328 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:1718093:1718328 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:1718093:1718328 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:1718093:1718328 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:1718094:1718329 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:1718093:1718328 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:1718094:1718329 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:1718093:1718328 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:1718094:1718329 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:1718093:1718328 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:1718094:1718329 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:1718093:1718328 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:1718094:1718329 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:1718093:1718328 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:1718094:1718329 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:1718093:1718328 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:1718094:1718329 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:1718093:1718328 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:1718094:1718329 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:1718093:1718328 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:1718094:1718329 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:1718093:1718328 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:1718094:1718329 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:1718093:1718328 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:1718094:1718329 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:1718093:1718328 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:1718094:1718329 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:1718093:1718328 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:1718094:1718329 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:1718093:1718328 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:1718094:1718329 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:1718093:1718328 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:1718094:1718329 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:1718093:1718328 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:1718094:1718329 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:1718093:1718328 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:1718094:1718329 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:1718093:1718328 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:1718094:1718329 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:1718093:1718328 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:1718094:1718329 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:1718093:1718328 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:1718094:1718329 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:1718093:1718328 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:1718094:1718329 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:1718093:1718328 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:1718094:1718329 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:1718093:1718328 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:1718094:1718329 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:1718093:1718328 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:1718094:1718329 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:1718093:1718328 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:1718094:1718329 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:1718093:1718328 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:1718093:1718328 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:1718094:1718329 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:1718093:1718328 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:1718093:1718328 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:1718093:1718328 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:1718093:1718328 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:1718093:1718328 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:1718093:1718328 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:1718093:1718328 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:1718093:1718328 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:1718093:1718328 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:1718093:1718328 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:1718093:1718328 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:1718093:1718328 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:1718093:1718328 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:1718093:1718328 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:1718093:1718328 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:1718093:1718328 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:1718093:1718328 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:1718093:1718328 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:1718093:1718328 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:1718093:1718328 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:1718093:1718328 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:1718093:1718340 [0] NCCL INFO [Proxy Service] Device 0 CPU core 27
n136-128-154:1718093:1718341 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 28
n136-128-154:1718094:1718329 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:1718094:1718342 [1] NCCL INFO [Proxy Service] Device 1 CPU core 4
n136-128-154:1718094:1718343 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 6
n136-128-154:1718093:1718328 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:1718093:1718328 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:1718094:1718329 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:1718094:1718329 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:1718093:1718328 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:1718093:1718328 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:1718093:1718328 [0] NCCL INFO ncclCommInitRankConfig comm 0x1182afd0 rank 0 nranks 2 cudaDev 0 nvmlDev 1 busId 16000 commId 0x2e657754adbae1ea - Init COMPLETE
n136-128-154:1718093:1718328 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 1.02 (kernels 0.24, alloc 0.52, bootstrap 0.00, allgathers 0.00, topo 0.06, graphs 0.00, connections 0.06, rest 0.15)
n136-128-154:1718094:1718329 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:1718094:1718329 [1] NCCL INFO ncclCommInitRankConfig comm 0xf232820 rank 1 nranks 2 cudaDev 1 nvmlDev 2 busId 4a000 commId 0x2e657754adbae1ea - Init COMPLETE
n136-128-154:1718094:1718329 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 1.02 (kernels 0.23, alloc 0.52, bootstrap 0.00, allgathers 0.00, topo 0.06, graphs 0.00, connections 0.06, rest 0.15)
n136-128-154:1718093:1718345 [0] NCCL INFO Channel 00/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718093:1718345 [0] NCCL INFO Channel 01/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718093:1718345 [0] NCCL INFO Channel 02/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718094:1718346 [1] NCCL INFO Channel 00/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718093:1718345 [0] NCCL INFO Channel 03/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718094:1718346 [1] NCCL INFO Channel 01/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718093:1718345 [0] NCCL INFO Channel 04/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718094:1718346 [1] NCCL INFO Channel 02/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718093:1718345 [0] NCCL INFO Channel 05/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718094:1718346 [1] NCCL INFO Channel 03/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718093:1718345 [0] NCCL INFO Channel 06/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718094:1718346 [1] NCCL INFO Channel 04/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718093:1718345 [0] NCCL INFO Channel 07/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718094:1718346 [1] NCCL INFO Channel 05/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718093:1718345 [0] NCCL INFO Channel 08/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718094:1718346 [1] NCCL INFO Channel 06/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718093:1718345 [0] NCCL INFO Channel 09/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718094:1718346 [1] NCCL INFO Channel 07/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718093:1718345 [0] NCCL INFO Channel 10/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718094:1718346 [1] NCCL INFO Channel 08/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718093:1718345 [0] NCCL INFO Channel 11/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718094:1718346 [1] NCCL INFO Channel 09/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718093:1718345 [0] NCCL INFO Channel 12/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718094:1718346 [1] NCCL INFO Channel 10/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718093:1718345 [0] NCCL INFO Channel 13/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718094:1718346 [1] NCCL INFO Channel 11/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718093:1718345 [0] NCCL INFO Channel 14/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718094:1718346 [1] NCCL INFO Channel 12/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718093:1718345 [0] NCCL INFO Channel 15/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718094:1718346 [1] NCCL INFO Channel 13/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718093:1718345 [0] NCCL INFO Channel 16/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718094:1718346 [1] NCCL INFO Channel 14/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718093:1718345 [0] NCCL INFO Channel 17/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718094:1718346 [1] NCCL INFO Channel 15/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718093:1718345 [0] NCCL INFO Channel 18/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718094:1718346 [1] NCCL INFO Channel 16/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718093:1718345 [0] NCCL INFO Channel 19/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718094:1718346 [1] NCCL INFO Channel 17/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718093:1718345 [0] NCCL INFO Channel 20/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718094:1718346 [1] NCCL INFO Channel 18/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718093:1718345 [0] NCCL INFO Channel 21/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718094:1718346 [1] NCCL INFO Channel 19/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718093:1718345 [0] NCCL INFO Channel 22/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718094:1718346 [1] NCCL INFO Channel 20/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718093:1718345 [0] NCCL INFO Channel 23/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718094:1718346 [1] NCCL INFO Channel 21/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718094:1718346 [1] NCCL INFO Channel 22/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718094:1718346 [1] NCCL INFO Channel 23/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718094:1718346 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:1718093:1718345 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-06:05:02:02 INFO     [evaluator:559] Running loglikelihood requests
2025-12-06:05:02:02 INFO     [evaluator:559] Running loglikelihood requests
Passed argument batch_size = auto:1. Detecting largest batch size

Running loglikelihood requests:   0%|          | 0/3270 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 64
Determined largest batch size: 64

Running loglikelihood requests:   0%|          | 1/3270 [00:35<31:59:34, 35.23s/it]
Running loglikelihood requests:   4%|▍         | 129/3270 [00:36<10:42,  4.89it/s] 
Running loglikelihood requests:   8%|▊         | 257/3270 [00:38<04:34, 10.96it/s]
Running loglikelihood requests:  12%|█▏        | 385/3270 [00:39<02:36, 18.38it/s]
Running loglikelihood requests:  16%|█▌        | 513/3270 [00:40<01:41, 27.14it/s]
Running loglikelihood requests:  20%|█▉        | 641/3270 [00:42<01:10, 37.12it/s]
Running loglikelihood requests:  24%|██▎       | 769/3270 [00:43<00:52, 47.97it/s]
Running loglikelihood requests:  27%|██▋       | 897/3270 [00:44<00:40, 59.31it/s]
Running loglikelihood requests:  31%|███▏      | 1025/3270 [00:45<00:31, 70.71it/s]
Running loglikelihood requests:  35%|███▌      | 1153/3270 [00:46<00:25, 81.58it/s]
Running loglikelihood requests:  39%|███▉      | 1281/3270 [00:47<00:21, 91.82it/s]
Running loglikelihood requests:  43%|████▎     | 1409/3270 [00:48<00:18, 101.20it/s]
Running loglikelihood requests:  47%|████▋     | 1537/3270 [00:49<00:15, 109.47it/s]
Running loglikelihood requests:  51%|█████     | 1665/3270 [00:50<00:13, 117.49it/s]
Running loglikelihood requests:  55%|█████▍    | 1793/3270 [00:51<00:11, 124.19it/s]
Running loglikelihood requests:  59%|█████▊    | 1921/3270 [00:52<00:10, 130.96it/s]
Running loglikelihood requests:  63%|██████▎   | 2049/3270 [00:52<00:08, 136.54it/s]
Running loglikelihood requests:  67%|██████▋   | 2177/3270 [00:53<00:07, 142.76it/s]
Running loglikelihood requests:  70%|███████   | 2305/3270 [00:54<00:06, 148.07it/s]
Running loglikelihood requests:  74%|███████▍  | 2433/3270 [00:55<00:05, 153.27it/s]
Running loglikelihood requests:  78%|███████▊  | 2561/3270 [00:56<00:04, 159.23it/s]
Running loglikelihood requests:  82%|████████▏ | 2689/3270 [00:56<00:03, 165.12it/s]
Running loglikelihood requests:  86%|████████▌ | 2817/3270 [00:57<00:02, 172.57it/s]
Running loglikelihood requests:  90%|█████████ | 2945/3270 [00:58<00:01, 180.95it/s]
Running loglikelihood requests:  94%|█████████▍| 3073/3270 [00:58<00:01, 190.70it/s]
Running loglikelihood requests:  98%|█████████▊| 3201/3270 [00:58<00:00, 230.05it/s]
Running loglikelihood requests: 100%|██████████| 3270/3270 [00:58<00:00, 55.48it/s] 
n136-128-154:1718094:1718407 [1] NCCL INFO Channel 00/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718094:1718407 [1] NCCL INFO Channel 01/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718094:1718407 [1] NCCL INFO Channel 02/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718094:1718407 [1] NCCL INFO Channel 03/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718094:1718407 [1] NCCL INFO Channel 04/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718094:1718407 [1] NCCL INFO Channel 05/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718094:1718407 [1] NCCL INFO Channel 06/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718094:1718407 [1] NCCL INFO Channel 07/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718094:1718407 [1] NCCL INFO Channel 08/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718094:1718407 [1] NCCL INFO Channel 09/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718094:1718407 [1] NCCL INFO Channel 10/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718094:1718407 [1] NCCL INFO Channel 11/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718094:1718407 [1] NCCL INFO Channel 12/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718094:1718407 [1] NCCL INFO Channel 13/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718094:1718407 [1] NCCL INFO Channel 14/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718094:1718407 [1] NCCL INFO Channel 15/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718094:1718407 [1] NCCL INFO Channel 16/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718094:1718407 [1] NCCL INFO Channel 17/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718094:1718407 [1] NCCL INFO Channel 18/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718094:1718407 [1] NCCL INFO Channel 19/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718094:1718407 [1] NCCL INFO Channel 20/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718094:1718407 [1] NCCL INFO Channel 21/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718094:1718407 [1] NCCL INFO Channel 22/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718094:1718407 [1] NCCL INFO Channel 23/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718094:1718407 [1] NCCL INFO Channel 24/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718094:1718407 [1] NCCL INFO Channel 25/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718094:1718407 [1] NCCL INFO Channel 26/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718094:1718407 [1] NCCL INFO Channel 27/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718094:1718407 [1] NCCL INFO Channel 28/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718094:1718407 [1] NCCL INFO Channel 29/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718094:1718407 [1] NCCL INFO Channel 30/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718094:1718407 [1] NCCL INFO Channel 31/1 : 1[2] -> 0[1] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:1718094:1718413 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1718094:1718413 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1718094:1718413 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1718094:1718413 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1718094:1718413 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1718094:1718413 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1718094:1718342 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-06:05:03:13 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: auto (64)
|Tasks|Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|-----|------:|------|-----:|------|---|-----:|---|-----:|
|boolq|      2|none  |     0|acc   |↑  |0.7768|±  |0.0073|

[rank0]:[W1206 05:03:14.343996600 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:1718093:1718462 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1718093:1718462 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1718093:1718462 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1718093:1718340 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:1718093:1718462 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1718093:1718462 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1718093:1718462 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1718094:1718342 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:1718094:1718413 [1] NCCL INFO comm 0xf232820 rank 1 nranks 2 cudaDev 1 busId 4a000 - Abort COMPLETE
n136-128-154:1718093:1718462 [0] NCCL INFO comm 0x1182afd0 rank 0 nranks 2 cudaDev 0 busId 16000 - Abort COMPLETE
任务 boolq 评估完成！

==================================================
开始评估：任务=arc_challenge | 少样本数=25 | 模型=Llama-3.1-8B-quantization-fg
输出路径：results2/Llama-3.1-8B-quantization-fg/fg/arc_challenge.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-06:05:04:38 INFO     [__main__:440] Selected Tasks: ['arc_challenge']
2025-12-06:05:04:38 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-06:05:04:38 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-06:05:04:38 INFO     [__main__:440] Selected Tasks: ['arc_challenge']
2025-12-06:05:04:38 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-06:05:04:38 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-06:05:04:39 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-06:05:04:40 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-06:05:04:40 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
`torch_dtype` is deprecated! Use `dtype` instead!
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.15s/it]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.15s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.17s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.18s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.17s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.17s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.21it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.22it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.05it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.05it/s]

2025-12-06:05:04:52 WARNING  [evaluator:309] Overwriting default num_fewshot of arc_challenge from None to 25
2025-12-06:05:04:52 INFO     [api.task:434] Building contexts for arc_challenge on rank 1...

  0%|          | 0/586 [00:00<?, ?it/s]
  1%|          | 4/586 [00:00<00:16, 36.02it/s]
  1%|▏         | 8/586 [00:00<00:16, 35.56it/s]
  2%|▏         | 12/586 [00:00<00:16, 35.78it/s]
  3%|▎         | 16/586 [00:00<00:15, 36.08it/s]
  3%|▎         | 20/586 [00:00<00:15, 35.85it/s]
  4%|▍         | 24/586 [00:00<00:15, 35.94it/s]2025-12-06:05:04:53 WARNING  [evaluator:309] Overwriting default num_fewshot of arc_challenge from None to 25
2025-12-06:05:04:53 INFO     [api.task:434] Building contexts for arc_challenge on rank 0...

  5%|▍         | 28/586 [00:00<00:15, 36.15it/s]
  0%|          | 0/586 [00:00<?, ?it/s]
  5%|▌         | 32/586 [00:00<00:15, 35.42it/s]
  1%|          | 4/586 [00:00<00:16, 34.76it/s]
  1%|▏         | 8/586 [00:00<00:16, 35.30it/s]
  6%|▌         | 36/586 [00:01<00:15, 35.04it/s]
  7%|▋         | 40/586 [00:01<00:15, 35.56it/s]
  2%|▏         | 12/586 [00:00<00:16, 35.29it/s]
  8%|▊         | 44/586 [00:01<00:15, 35.90it/s]
  3%|▎         | 16/586 [00:00<00:16, 35.62it/s]
  8%|▊         | 48/586 [00:01<00:14, 35.98it/s]
  3%|▎         | 20/586 [00:00<00:16, 35.16it/s]
  9%|▉         | 52/586 [00:01<00:14, 36.16it/s]
  4%|▍         | 24/586 [00:00<00:15, 35.38it/s]
 10%|▉         | 56/586 [00:01<00:14, 35.89it/s]
  5%|▍         | 28/586 [00:00<00:15, 35.55it/s]
 10%|█         | 60/586 [00:01<00:14, 36.02it/s]
  5%|▌         | 32/586 [00:00<00:15, 35.54it/s]
 11%|█         | 64/586 [00:01<00:14, 36.21it/s]
  6%|▌         | 36/586 [00:01<00:15, 35.71it/s]
 12%|█▏        | 68/586 [00:01<00:14, 35.66it/s]
  7%|▋         | 40/586 [00:01<00:15, 35.66it/s]
 12%|█▏        | 72/586 [00:02<00:14, 35.88it/s]
  8%|▊         | 44/586 [00:01<00:15, 35.58it/s]
 13%|█▎        | 76/586 [00:02<00:14, 36.09it/s]
  8%|▊         | 48/586 [00:01<00:15, 35.31it/s]
 14%|█▎        | 80/586 [00:02<00:13, 36.29it/s]
  9%|▉         | 52/586 [00:01<00:15, 35.56it/s]
 14%|█▍        | 84/586 [00:02<00:13, 36.41it/s]
 10%|▉         | 56/586 [00:01<00:14, 35.64it/s]
 15%|█▌        | 88/586 [00:02<00:13, 36.51it/s]
 10%|█         | 60/586 [00:01<00:14, 35.81it/s]
 16%|█▌        | 92/586 [00:02<00:13, 36.53it/s]
 11%|█         | 64/586 [00:01<00:14, 35.84it/s]
 16%|█▋        | 96/586 [00:02<00:13, 36.60it/s]
 12%|█▏        | 68/586 [00:01<00:14, 35.41it/s]
 17%|█▋        | 100/586 [00:02<00:13, 36.72it/s]
 12%|█▏        | 72/586 [00:02<00:14, 35.59it/s]
 18%|█▊        | 104/586 [00:02<00:13, 36.76it/s]
 13%|█▎        | 76/586 [00:02<00:14, 35.62it/s]
 18%|█▊        | 108/586 [00:02<00:13, 36.72it/s]
 14%|█▎        | 80/586 [00:02<00:14, 35.83it/s]
 19%|█▉        | 112/586 [00:03<00:12, 36.65it/s]
 14%|█▍        | 84/586 [00:02<00:13, 36.05it/s]
 20%|█▉        | 116/586 [00:03<00:12, 36.54it/s]
 15%|█▌        | 88/586 [00:02<00:13, 35.93it/s]
 20%|██        | 120/586 [00:03<00:12, 36.50it/s]
 16%|█▌        | 92/586 [00:02<00:13, 35.99it/s]
 21%|██        | 124/586 [00:03<00:12, 36.49it/s]
 16%|█▋        | 96/586 [00:02<00:13, 36.12it/s]
 22%|██▏       | 128/586 [00:03<00:12, 36.51it/s]
 17%|█▋        | 100/586 [00:02<00:13, 36.22it/s]
 23%|██▎       | 132/586 [00:03<00:12, 36.53it/s]
 18%|█▊        | 104/586 [00:02<00:13, 36.14it/s]
 23%|██▎       | 136/586 [00:03<00:12, 36.62it/s]
 18%|█▊        | 108/586 [00:03<00:13, 36.14it/s]
 24%|██▍       | 140/586 [00:03<00:12, 36.71it/s]
 19%|█▉        | 112/586 [00:03<00:13, 36.14it/s]
 25%|██▍       | 144/586 [00:03<00:12, 36.65it/s]
 20%|█▉        | 116/586 [00:03<00:13, 36.08it/s]
 25%|██▌       | 148/586 [00:04<00:11, 36.73it/s]
 20%|██        | 120/586 [00:03<00:12, 36.01it/s]
 26%|██▌       | 152/586 [00:04<00:11, 36.77it/s]
 21%|██        | 124/586 [00:03<00:12, 35.64it/s]
 27%|██▋       | 156/586 [00:04<00:11, 36.81it/s]
 22%|██▏       | 128/586 [00:03<00:12, 35.56it/s]
 27%|██▋       | 160/586 [00:04<00:11, 36.89it/s]
 23%|██▎       | 132/586 [00:03<00:12, 35.51it/s]
 28%|██▊       | 164/586 [00:04<00:11, 36.69it/s]
 23%|██▎       | 136/586 [00:03<00:12, 35.55it/s]
 29%|██▊       | 168/586 [00:04<00:11, 36.81it/s]
 24%|██▍       | 140/586 [00:03<00:12, 35.61it/s]
 29%|██▉       | 172/586 [00:04<00:11, 36.87it/s]
 25%|██▍       | 144/586 [00:04<00:12, 35.67it/s]
 30%|███       | 176/586 [00:04<00:11, 36.89it/s]
 25%|██▌       | 148/586 [00:04<00:12, 35.70it/s]
 31%|███       | 180/586 [00:04<00:10, 36.92it/s]
 26%|██▌       | 152/586 [00:04<00:12, 35.83it/s]
 31%|███▏      | 184/586 [00:05<00:10, 36.95it/s]
 27%|██▋       | 156/586 [00:04<00:11, 35.89it/s]
 32%|███▏      | 188/586 [00:05<00:10, 36.95it/s]
 27%|██▋       | 160/586 [00:04<00:11, 36.02it/s]
 33%|███▎      | 192/586 [00:05<00:10, 37.00it/s]
 28%|██▊       | 164/586 [00:04<00:11, 35.64it/s]
 33%|███▎      | 196/586 [00:05<00:10, 37.02it/s]
 29%|██▊       | 168/586 [00:04<00:11, 35.81it/s]
 34%|███▍      | 200/586 [00:05<00:10, 36.55it/s]
 29%|██▉       | 172/586 [00:04<00:11, 35.72it/s]
 35%|███▍      | 204/586 [00:05<00:10, 36.56it/s]
 35%|███▌      | 208/586 [00:05<00:10, 36.62it/s]
 30%|███       | 176/586 [00:04<00:11, 35.55it/s]
 36%|███▌      | 212/586 [00:05<00:10, 36.79it/s]
 31%|███       | 180/586 [00:05<00:11, 35.41it/s]
 37%|███▋      | 216/586 [00:05<00:10, 36.85it/s]
 31%|███▏      | 184/586 [00:05<00:11, 34.99it/s]
 38%|███▊      | 220/586 [00:06<00:09, 36.84it/s]
 32%|███▏      | 188/586 [00:05<00:11, 34.63it/s]
 38%|███▊      | 224/586 [00:06<00:09, 36.85it/s]
 33%|███▎      | 192/586 [00:05<00:11, 34.90it/s]
 39%|███▉      | 228/586 [00:06<00:09, 36.84it/s]
 33%|███▎      | 196/586 [00:05<00:11, 35.02it/s]
 40%|███▉      | 232/586 [00:06<00:09, 36.85it/s]
 34%|███▍      | 200/586 [00:05<00:11, 35.08it/s]
 40%|████      | 236/586 [00:06<00:09, 36.88it/s]
 35%|███▍      | 204/586 [00:05<00:10, 35.25it/s]
 41%|████      | 240/586 [00:06<00:09, 36.94it/s]
 35%|███▌      | 208/586 [00:05<00:10, 35.33it/s]
 42%|████▏     | 244/586 [00:06<00:09, 36.85it/s]
 36%|███▌      | 212/586 [00:05<00:10, 35.29it/s]
 42%|████▏     | 248/586 [00:06<00:09, 36.87it/s]
 37%|███▋      | 216/586 [00:06<00:10, 35.27it/s]
 43%|████▎     | 252/586 [00:06<00:09, 36.93it/s]
 38%|███▊      | 220/586 [00:06<00:10, 35.21it/s]
 44%|████▎     | 256/586 [00:07<00:08, 36.94it/s]
 38%|███▊      | 224/586 [00:06<00:10, 35.26it/s]
 44%|████▍     | 260/586 [00:07<00:08, 36.95it/s]
 39%|███▉      | 228/586 [00:06<00:10, 35.31it/s]
 45%|████▌     | 264/586 [00:07<00:08, 36.85it/s]
 40%|███▉      | 232/586 [00:06<00:10, 35.24it/s]
 46%|████▌     | 268/586 [00:07<00:08, 36.86it/s]
 40%|████      | 236/586 [00:06<00:09, 35.26it/s]
 46%|████▋     | 272/586 [00:07<00:08, 36.83it/s]
 41%|████      | 240/586 [00:06<00:09, 35.23it/s]
 47%|████▋     | 276/586 [00:07<00:08, 36.83it/s]
 42%|████▏     | 244/586 [00:06<00:09, 35.31it/s]
 48%|████▊     | 280/586 [00:07<00:08, 36.86it/s]
 42%|████▏     | 248/586 [00:06<00:09, 35.37it/s]
 48%|████▊     | 284/586 [00:07<00:08, 36.89it/s]
 43%|████▎     | 252/586 [00:07<00:09, 35.42it/s]
 49%|████▉     | 288/586 [00:07<00:08, 36.93it/s]
 44%|████▎     | 256/586 [00:07<00:09, 35.44it/s]
 50%|████▉     | 292/586 [00:07<00:07, 36.94it/s]
 51%|█████     | 296/586 [00:08<00:07, 36.96it/s]
 44%|████▍     | 260/586 [00:07<00:09, 35.43it/s]
 51%|█████     | 300/586 [00:08<00:07, 36.93it/s]
 45%|████▌     | 264/586 [00:07<00:09, 35.53it/s]
 52%|█████▏    | 304/586 [00:08<00:07, 36.94it/s]
 46%|████▌     | 268/586 [00:07<00:08, 35.52it/s]
 53%|█████▎    | 308/586 [00:08<00:07, 36.95it/s]
 46%|████▋     | 272/586 [00:07<00:08, 35.42it/s]
 53%|█████▎    | 312/586 [00:08<00:07, 36.66it/s]
 47%|████▋     | 276/586 [00:07<00:08, 35.51it/s]
 54%|█████▍    | 316/586 [00:08<00:07, 36.19it/s]
 48%|████▊     | 280/586 [00:07<00:08, 35.59it/s]
 55%|█████▍    | 320/586 [00:08<00:07, 36.34it/s]
 48%|████▊     | 284/586 [00:07<00:08, 35.55it/s]
 55%|█████▌    | 324/586 [00:08<00:07, 36.48it/s]
 49%|████▉     | 288/586 [00:08<00:08, 35.64it/s]
 56%|█████▌    | 328/586 [00:08<00:07, 36.55it/s]
 50%|████▉     | 292/586 [00:08<00:08, 35.59it/s]
 57%|█████▋    | 332/586 [00:09<00:06, 36.62it/s]
 51%|█████     | 296/586 [00:08<00:08, 35.51it/s]
 57%|█████▋    | 336/586 [00:09<00:06, 36.06it/s]
 51%|█████     | 300/586 [00:08<00:08, 35.56it/s]
 58%|█████▊    | 340/586 [00:09<00:06, 35.56it/s]
 52%|█████▏    | 304/586 [00:08<00:07, 35.39it/s]
 59%|█████▊    | 344/586 [00:09<00:06, 35.55it/s]
 53%|█████▎    | 308/586 [00:08<00:07, 35.22it/s]
 59%|█████▉    | 348/586 [00:09<00:06, 35.49it/s]
 53%|█████▎    | 312/586 [00:08<00:07, 35.29it/s]
 60%|██████    | 352/586 [00:09<00:06, 35.07it/s]
 54%|█████▍    | 316/586 [00:08<00:07, 35.36it/s]
 61%|██████    | 356/586 [00:09<00:06, 34.87it/s]
 55%|█████▍    | 320/586 [00:09<00:07, 35.46it/s]
 61%|██████▏   | 360/586 [00:09<00:06, 34.76it/s]
 55%|█████▌    | 324/586 [00:09<00:07, 35.44it/s]
 62%|██████▏   | 364/586 [00:10<00:06, 34.82it/s]
 56%|█████▌    | 328/586 [00:09<00:07, 35.50it/s]
 63%|██████▎   | 368/586 [00:10<00:06, 34.45it/s]
 57%|█████▋    | 332/586 [00:09<00:07, 35.47it/s]
 63%|██████▎   | 372/586 [00:10<00:06, 34.23it/s]
 57%|█████▋    | 336/586 [00:09<00:07, 35.50it/s]
 58%|█████▊    | 340/586 [00:09<00:06, 35.48it/s]
 64%|██████▍   | 376/586 [00:10<00:06, 34.10it/s]
 59%|█████▊    | 344/586 [00:09<00:06, 34.90it/s]
 65%|██████▍   | 380/586 [00:10<00:06, 34.01it/s]
 59%|█████▉    | 348/586 [00:09<00:06, 34.76it/s]
 66%|██████▌   | 384/586 [00:10<00:05, 33.72it/s]
 60%|██████    | 352/586 [00:09<00:06, 34.81it/s]
 66%|██████▌   | 388/586 [00:10<00:05, 33.67it/s]
 61%|██████    | 356/586 [00:10<00:06, 34.86it/s]
 67%|██████▋   | 392/586 [00:10<00:05, 33.64it/s]
 61%|██████▏   | 360/586 [00:10<00:06, 35.12it/s]
 68%|██████▊   | 396/586 [00:10<00:05, 33.90it/s]
 62%|██████▏   | 364/586 [00:10<00:06, 35.26it/s]
 68%|██████▊   | 400/586 [00:11<00:05, 34.01it/s]
 63%|██████▎   | 368/586 [00:10<00:06, 35.36it/s]
 69%|██████▉   | 404/586 [00:11<00:05, 33.97it/s]
 63%|██████▎   | 372/586 [00:10<00:06, 35.44it/s]
 70%|██████▉   | 408/586 [00:11<00:05, 34.02it/s]
 64%|██████▍   | 376/586 [00:10<00:05, 35.50it/s]
 70%|███████   | 412/586 [00:11<00:05, 34.01it/s]
 65%|██████▍   | 380/586 [00:10<00:05, 35.63it/s]
 71%|███████   | 416/586 [00:11<00:04, 34.10it/s]
 66%|██████▌   | 384/586 [00:10<00:05, 35.70it/s]
 72%|███████▏  | 420/586 [00:11<00:04, 34.21it/s]
 66%|██████▌   | 388/586 [00:10<00:05, 35.78it/s]
 72%|███████▏  | 424/586 [00:11<00:04, 34.27it/s]
 67%|██████▋   | 392/586 [00:11<00:05, 35.73it/s]
 73%|███████▎  | 428/586 [00:11<00:04, 34.23it/s]
 68%|██████▊   | 396/586 [00:11<00:05, 35.70it/s]
 74%|███████▎  | 432/586 [00:12<00:04, 34.40it/s]
 68%|██████▊   | 400/586 [00:11<00:05, 35.72it/s]
 74%|███████▍  | 436/586 [00:12<00:04, 34.91it/s]
 69%|██████▉   | 404/586 [00:11<00:05, 35.68it/s]
 75%|███████▌  | 440/586 [00:12<00:04, 35.57it/s]
 70%|██████▉   | 408/586 [00:11<00:04, 35.65it/s]
 76%|███████▌  | 444/586 [00:12<00:03, 35.98it/s]
 70%|███████   | 412/586 [00:11<00:04, 35.64it/s]
 76%|███████▋  | 448/586 [00:12<00:03, 36.30it/s]
 71%|███████   | 416/586 [00:11<00:04, 35.54it/s]
 77%|███████▋  | 452/586 [00:12<00:03, 36.49it/s]
 72%|███████▏  | 420/586 [00:11<00:04, 35.50it/s]
 78%|███████▊  | 456/586 [00:12<00:03, 36.14it/s]
 72%|███████▏  | 424/586 [00:11<00:04, 35.52it/s]
 78%|███████▊  | 460/586 [00:12<00:03, 36.41it/s]
 73%|███████▎  | 428/586 [00:12<00:04, 35.53it/s]
 79%|███████▉  | 464/586 [00:12<00:03, 36.58it/s]
 74%|███████▎  | 432/586 [00:12<00:04, 35.50it/s]
 80%|███████▉  | 468/586 [00:12<00:03, 36.67it/s]
 74%|███████▍  | 436/586 [00:12<00:04, 35.51it/s]
 81%|████████  | 472/586 [00:13<00:03, 36.79it/s]
 75%|███████▌  | 440/586 [00:12<00:04, 35.57it/s]
 81%|████████  | 476/586 [00:13<00:02, 36.90it/s]
 76%|███████▌  | 444/586 [00:12<00:04, 35.43it/s]
 82%|████████▏ | 480/586 [00:13<00:02, 36.92it/s]
 76%|███████▋  | 448/586 [00:12<00:03, 35.24it/s]
 83%|████████▎ | 484/586 [00:13<00:02, 36.98it/s]
 77%|███████▋  | 452/586 [00:12<00:03, 35.18it/s]
 83%|████████▎ | 488/586 [00:13<00:02, 36.99it/s]
 84%|████████▍ | 492/586 [00:13<00:02, 36.99it/s]
 78%|███████▊  | 456/586 [00:12<00:03, 35.23it/s]
 85%|████████▍ | 496/586 [00:13<00:02, 36.91it/s]
 78%|███████▊  | 460/586 [00:12<00:03, 35.45it/s]
 85%|████████▌ | 500/586 [00:13<00:02, 36.91it/s]
 79%|███████▉  | 464/586 [00:13<00:03, 35.47it/s]
 86%|████████▌ | 504/586 [00:13<00:02, 36.88it/s]
 80%|███████▉  | 468/586 [00:13<00:03, 35.53it/s]
 87%|████████▋ | 508/586 [00:14<00:02, 36.93it/s]
 81%|████████  | 472/586 [00:13<00:03, 35.57it/s]
 87%|████████▋ | 512/586 [00:14<00:02, 36.94it/s]
 81%|████████  | 476/586 [00:13<00:03, 35.61it/s]
 88%|████████▊ | 516/586 [00:14<00:01, 36.97it/s]
 82%|████████▏ | 480/586 [00:13<00:02, 35.65it/s]
 89%|████████▊ | 520/586 [00:14<00:01, 36.93it/s]
 83%|████████▎ | 484/586 [00:13<00:02, 35.67it/s]
 89%|████████▉ | 524/586 [00:14<00:01, 36.35it/s]
 83%|████████▎ | 488/586 [00:13<00:02, 35.61it/s]
 90%|█████████ | 528/586 [00:14<00:01, 35.62it/s]
 84%|████████▍ | 492/586 [00:13<00:02, 35.52it/s]
 91%|█████████ | 532/586 [00:14<00:01, 35.22it/s]
 85%|████████▍ | 496/586 [00:13<00:02, 35.42it/s]
 91%|█████████▏| 536/586 [00:14<00:01, 35.05it/s]
 85%|████████▌ | 500/586 [00:14<00:02, 35.40it/s]
 92%|█████████▏| 540/586 [00:14<00:01, 34.98it/s]
 86%|████████▌ | 504/586 [00:14<00:02, 35.55it/s]
 93%|█████████▎| 544/586 [00:15<00:01, 34.94it/s]
 87%|████████▋ | 508/586 [00:14<00:02, 35.73it/s]
 94%|█████████▎| 548/586 [00:15<00:01, 34.78it/s]
 87%|████████▋ | 512/586 [00:14<00:02, 35.83it/s]
 88%|████████▊ | 516/586 [00:14<00:01, 36.00it/s]
 94%|█████████▍| 552/586 [00:15<00:00, 34.74it/s]
 89%|████████▊ | 520/586 [00:14<00:01, 36.02it/s]
 95%|█████████▍| 556/586 [00:15<00:00, 34.72it/s]
 89%|████████▉ | 524/586 [00:14<00:01, 35.87it/s]
 96%|█████████▌| 560/586 [00:15<00:00, 34.43it/s]
 90%|█████████ | 528/586 [00:14<00:01, 35.83it/s]
 96%|█████████▌| 564/586 [00:15<00:00, 34.33it/s]
 91%|█████████ | 532/586 [00:14<00:01, 35.77it/s]
 97%|█████████▋| 568/586 [00:15<00:00, 34.18it/s]
 91%|█████████▏| 536/586 [00:15<00:01, 35.89it/s]
 98%|█████████▊| 572/586 [00:15<00:00, 34.12it/s]
 92%|█████████▏| 540/586 [00:15<00:01, 35.84it/s]
 98%|█████████▊| 576/586 [00:16<00:00, 34.10it/s]
 93%|█████████▎| 544/586 [00:15<00:01, 35.85it/s]
 99%|█████████▉| 580/586 [00:16<00:00, 34.12it/s]
 94%|█████████▎| 548/586 [00:15<00:01, 35.74it/s]
100%|█████████▉| 584/586 [00:16<00:00, 34.12it/s]
100%|██████████| 586/586 [00:16<00:00, 35.93it/s]

 94%|█████████▍| 552/586 [00:15<00:00, 35.74it/s]
 95%|█████████▍| 556/586 [00:15<00:00, 35.91it/s]
 96%|█████████▌| 560/586 [00:15<00:00, 35.86it/s]
 96%|█████████▌| 564/586 [00:15<00:00, 35.94it/s]
 97%|█████████▋| 568/586 [00:15<00:00, 35.93it/s]
 98%|█████████▊| 572/586 [00:16<00:00, 35.72it/s]
 98%|█████████▊| 576/586 [00:16<00:00, 35.68it/s]
 99%|█████████▉| 580/586 [00:16<00:00, 35.56it/s]
100%|█████████▉| 584/586 [00:16<00:00, 35.44it/s]
100%|██████████| 586/586 [00:16<00:00, 35.54it/s]
n136-128-154:1718518:1718518 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1718518:1718518 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1718518:1718518 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:1718518:1718518 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:1718518:1718518 [0] NCCL INFO Comm config Blocking set to 1
n136-128-154:1718519:1718519 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:1718519:1718519 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1718519:1718519 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1718519:1718519 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:1718519:1718519 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:1718518:1718861 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:1718518:1718861 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:1718519:1718862 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:1718519:1718862 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:1718518:1718861 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1718518:1718861 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1718519:1718862 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1718519:1718862 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1718518:1718861 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:1718519:1718862 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:1718519:1718862 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:1718519:1718862 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:1718519:1718862 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:1718518:1718861 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:1718518:1718861 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:1718518:1718861 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:1718519:1718862 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:1718519:1718862 [1] NCCL INFO Using network IB
n136-128-154:1718518:1718861 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:1718518:1718861 [0] NCCL INFO Using network IB
n136-128-154:1718519:1718862 [1] NCCL INFO ncclCommInitRankConfig comm 0x11a60100 rank 1 nranks 2 cudaDev 1 nvmlDev 2 busId 4a000 commId 0xa268e5e053a110c3 - Init START
n136-128-154:1718518:1718861 [0] NCCL INFO ncclCommInitRankConfig comm 0x12d9fc50 rank 0 nranks 2 cudaDev 0 nvmlDev 1 busId 16000 commId 0xa268e5e053a110c3 - Init START
n136-128-154:1718519:1718862 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:1718518:1718861 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:1718518:1718861 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:1718518:1718861 [0] NCCL INFO Retrieving state for IB
n136-128-154:1718518:1718861 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:1718518:1718861 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:1718519:1718862 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:1718519:1718862 [1] NCCL INFO Retrieving state for IB
n136-128-154:1718519:1718862 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:1718518:1718861 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:1718519:1718862 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:1718518:1718861 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:1718519:1718862 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:1718518:1718861 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:1718519:1718862 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:1718519:1718862 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:1718518:1718861 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1718519:1718862 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1718518:1718861 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1718519:1718862 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1718518:1718861 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1718519:1718862 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1718518:1718861 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1718519:1718862 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1718518:1718861 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:1718518:1718861 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:1718518:1718861 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:1718518:1718861 [0] NCCL INFO               + PCI[24.0] - PCI/0-14000 (1000c01010de13b8)
n136-128-154:1718518:1718861 [0] NCCL INFO                             + PCI[24.0] - GPU/0-16000 (0)
n136-128-154:1718518:1718861 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1718518:1718861 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:1718519:1718862 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:1718518:1718861 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:1718518:1718861 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:1718519:1718862 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:1718518:1718861 [0] NCCL INFO               + PCI[24.0] - PCI/0-48000 (1000c01010de13b8)
n136-128-154:1718519:1718862 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:1718518:1718861 [0] NCCL INFO                             + PCI[24.0] - GPU/0-4a000 (1)
n136-128-154:1718519:1718862 [1] NCCL INFO               + PCI[24.0] - PCI/0-14000 (1000c01010de13b8)
n136-128-154:1718518:1718861 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1718519:1718862 [1] NCCL INFO                             + PCI[24.0] - GPU/0-16000 (0)
n136-128-154:1718519:1718862 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1718518:1718861 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:1718519:1718862 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:1718518:1718861 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:1718519:1718862 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:1718519:1718862 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:1718518:1718861 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:1718519:1718862 [1] NCCL INFO               + PCI[24.0] - PCI/0-48000 (1000c01010de13b8)
n136-128-154:1718518:1718861 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:1718519:1718862 [1] NCCL INFO                             + PCI[24.0] - GPU/0-4a000 (1)
n136-128-154:1718519:1718862 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1718518:1718861 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:1718519:1718862 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:1718519:1718862 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:1718518:1718861 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:1718519:1718862 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:1718519:1718862 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:1718518:1718861 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:1718519:1718862 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:1718518:1718861 [0] NCCL INFO ==========================================
n136-128-154:1718519:1718862 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:1718519:1718862 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:1718519:1718862 [1] NCCL INFO ==========================================
n136-128-154:1718518:1718861 [0] NCCL INFO GPU/0-16000 :GPU/0-16000 (0/5000.0/LOC) GPU/0-4a000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1718519:1718862 [1] NCCL INFO GPU/0-16000 :GPU/0-16000 (0/5000.0/LOC) GPU/0-4a000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1718519:1718862 [1] NCCL INFO GPU/0-4a000 :GPU/0-16000 (2/240.0/NVL) GPU/0-4a000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1718518:1718861 [0] NCCL INFO GPU/0-4a000 :GPU/0-16000 (2/240.0/NVL) GPU/0-4a000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1718519:1718862 [1] NCCL INFO Setting affinity for GPU 2 to 1-31,65-95
n136-128-154:1718518:1718861 [0] NCCL INFO Setting affinity for GPU 1 to 1-31,65-95
n136-128-154:1718518:1718861 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:1718518:1718861 [0] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718518:1718861 [0] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718518:1718861 [0] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718518:1718861 [0] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718518:1718861 [0] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718518:1718861 [0] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718518:1718861 [0] NCCL INFO  6 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718518:1718861 [0] NCCL INFO  7 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718518:1718861 [0] NCCL INFO  8 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718518:1718861 [0] NCCL INFO  9 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718518:1718861 [0] NCCL INFO 10 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718518:1718861 [0] NCCL INFO 11 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718519:1718862 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:1718519:1718862 [1] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718519:1718862 [1] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718519:1718862 [1] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718519:1718862 [1] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718519:1718862 [1] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718519:1718862 [1] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718519:1718862 [1] NCCL INFO  6 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718519:1718862 [1] NCCL INFO  7 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718519:1718862 [1] NCCL INFO  8 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718519:1718862 [1] NCCL INFO  9 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718519:1718862 [1] NCCL INFO 10 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718519:1718862 [1] NCCL INFO 11 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718518:1718861 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:1718518:1718861 [0] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718518:1718861 [0] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718518:1718861 [0] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718518:1718861 [0] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718518:1718861 [0] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718518:1718861 [0] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718518:1718861 [0] NCCL INFO  6 : GPU/0-4a000 GPU/0-16000
n136-128-154:1718518:1718861 [0] NCCL INFO  7 : GPU/0-4a000 GPU/0-16000
n136-128-154:1718518:1718861 [0] NCCL INFO  8 : GPU/0-4a000 GPU/0-16000
n136-128-154:1718518:1718861 [0] NCCL INFO  9 : GPU/0-4a000 GPU/0-16000
n136-128-154:1718518:1718861 [0] NCCL INFO 10 : GPU/0-4a000 GPU/0-16000
n136-128-154:1718519:1718862 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:1718518:1718861 [0] NCCL INFO 11 : GPU/0-4a000 GPU/0-16000
n136-128-154:1718519:1718862 [1] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718519:1718862 [1] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718519:1718862 [1] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718519:1718862 [1] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718519:1718862 [1] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718519:1718862 [1] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1718519:1718862 [1] NCCL INFO  6 : GPU/0-4a000 GPU/0-16000
n136-128-154:1718519:1718862 [1] NCCL INFO  7 : GPU/0-4a000 GPU/0-16000
n136-128-154:1718519:1718862 [1] NCCL INFO  8 : GPU/0-4a000 GPU/0-16000
n136-128-154:1718519:1718862 [1] NCCL INFO  9 : GPU/0-4a000 GPU/0-16000
n136-128-154:1718519:1718862 [1] NCCL INFO 10 : GPU/0-4a000 GPU/0-16000
n136-128-154:1718519:1718862 [1] NCCL INFO 11 : GPU/0-4a000 GPU/0-16000
n136-128-154:1718518:1718861 [0] NCCL INFO comm 0x12d9fc50 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:1718519:1718862 [1] NCCL INFO comm 0x11a60100 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:1718518:1718861 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:1718519:1718862 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:1718518:1718861 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:1718519:1718862 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:1718518:1718861 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:1718519:1718862 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:1718518:1718861 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:1718519:1718862 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:1718518:1718861 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:1718519:1718862 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:1718518:1718861 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:1718519:1718862 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:1718518:1718861 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:1718519:1718862 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:1718518:1718861 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:1718519:1718862 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:1718518:1718861 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:1718519:1718862 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:1718518:1718861 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:1718519:1718862 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:1718518:1718861 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:1718519:1718862 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:1718518:1718861 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:1718519:1718862 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:1718518:1718861 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:1718519:1718862 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:1718518:1718861 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:1718519:1718862 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:1718518:1718861 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:1718519:1718862 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:1718518:1718861 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:1718519:1718862 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:1718518:1718861 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:1718519:1718862 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:1718518:1718861 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:1718519:1718862 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:1718518:1718861 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:1718519:1718862 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:1718518:1718861 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:1718519:1718862 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:1718518:1718861 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:1718519:1718862 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:1718518:1718861 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:1718519:1718862 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:1718518:1718861 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:1718519:1718862 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:1718518:1718861 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:1718519:1718862 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:1718518:1718861 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:1718518:1718861 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:1718518:1718861 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:1718519:1718862 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:1718518:1718861 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:1718519:1718862 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:1718518:1718861 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:1718519:1718862 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:1718518:1718861 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:1718519:1718862 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:1718518:1718861 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:1718519:1718862 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:1718518:1718861 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:1718519:1718862 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:1718518:1718861 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:1718519:1718862 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:1718518:1718861 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:1718519:1718862 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:1718518:1718861 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:1718519:1718862 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:1718518:1718861 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:1718519:1718862 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:1718518:1718861 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:1718519:1718862 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:1718518:1718861 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:1718519:1718862 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:1718518:1718861 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:1718519:1718862 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:1718518:1718861 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:1718519:1718862 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:1718518:1718861 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:1718519:1718862 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:1718518:1718861 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:1718519:1718862 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:1718518:1718861 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:1718519:1718862 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:1718518:1718861 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:1718519:1718862 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:1718518:1718861 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:1718519:1718862 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:1718518:1718861 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:1718519:1718862 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:1718518:1718861 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:1718519:1718862 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:1718518:1718861 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:1718519:1718862 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:1718518:1718861 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:1718519:1718862 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:1718518:1718861 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:1718519:1718862 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:1718518:1718861 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:1718519:1718862 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:1718518:1718861 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:1718518:1718861 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:1718519:1718862 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:1718518:1718861 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:1718518:1718861 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:1718518:1718861 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:1718518:1718861 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:1718518:1718861 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:1718518:1718861 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:1718518:1718861 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:1718518:1718861 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:1718518:1718861 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:1718518:1718861 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:1718518:1718861 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:1718518:1718861 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:1718518:1718861 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:1718518:1718861 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:1718518:1718861 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:1718518:1718861 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:1718518:1718861 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:1718518:1718861 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:1718518:1718861 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:1718518:1718861 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:1718518:1718861 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:1718519:1718862 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:1718519:1718873 [1] NCCL INFO [Proxy Service] Device 1 CPU core 10
n136-128-154:1718519:1718874 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 13
n136-128-154:1718518:1718861 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:1718518:1718861 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:1718518:1718875 [0] NCCL INFO [Proxy Service] Device 0 CPU core 70
n136-128-154:1718518:1718876 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 71
n136-128-154:1718518:1718861 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:1718519:1718862 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:1718518:1718861 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:1718519:1718862 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:1718518:1718861 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:1718518:1718861 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:1718518:1718861 [0] NCCL INFO ncclCommInitRankConfig comm 0x12d9fc50 rank 0 nranks 2 cudaDev 0 nvmlDev 1 busId 16000 commId 0xa268e5e053a110c3 - Init COMPLETE
n136-128-154:1718518:1718861 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.60 (kernels 0.20, alloc 0.20, bootstrap 0.00, allgathers 0.00, topo 0.07, graphs 0.00, connections 0.06, rest 0.06)
n136-128-154:1718519:1718862 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:1718519:1718862 [1] NCCL INFO ncclCommInitRankConfig comm 0x11a60100 rank 1 nranks 2 cudaDev 1 nvmlDev 2 busId 4a000 commId 0xa268e5e053a110c3 - Init COMPLETE
n136-128-154:1718519:1718862 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.58 (kernels 0.19, alloc 0.20, bootstrap 0.00, allgathers 0.00, topo 0.07, graphs 0.00, connections 0.06, rest 0.06)
n136-128-154:1718518:1718877 [0] NCCL INFO Channel 00/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718519:1718878 [1] NCCL INFO Channel 00/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718518:1718877 [0] NCCL INFO Channel 01/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718519:1718878 [1] NCCL INFO Channel 01/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718518:1718877 [0] NCCL INFO Channel 02/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718519:1718878 [1] NCCL INFO Channel 02/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718518:1718877 [0] NCCL INFO Channel 03/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718519:1718878 [1] NCCL INFO Channel 03/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718518:1718877 [0] NCCL INFO Channel 04/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718519:1718878 [1] NCCL INFO Channel 04/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718518:1718877 [0] NCCL INFO Channel 05/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718519:1718878 [1] NCCL INFO Channel 05/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718518:1718877 [0] NCCL INFO Channel 06/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718519:1718878 [1] NCCL INFO Channel 06/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718518:1718877 [0] NCCL INFO Channel 07/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718519:1718878 [1] NCCL INFO Channel 07/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718518:1718877 [0] NCCL INFO Channel 08/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718519:1718878 [1] NCCL INFO Channel 08/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718518:1718877 [0] NCCL INFO Channel 09/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718519:1718878 [1] NCCL INFO Channel 09/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718518:1718877 [0] NCCL INFO Channel 10/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718519:1718878 [1] NCCL INFO Channel 10/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718518:1718877 [0] NCCL INFO Channel 11/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718519:1718878 [1] NCCL INFO Channel 11/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718518:1718877 [0] NCCL INFO Channel 12/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718519:1718878 [1] NCCL INFO Channel 12/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718518:1718877 [0] NCCL INFO Channel 13/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718519:1718878 [1] NCCL INFO Channel 13/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718518:1718877 [0] NCCL INFO Channel 14/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718519:1718878 [1] NCCL INFO Channel 14/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718518:1718877 [0] NCCL INFO Channel 15/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718519:1718878 [1] NCCL INFO Channel 15/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718518:1718877 [0] NCCL INFO Channel 16/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718519:1718878 [1] NCCL INFO Channel 16/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718518:1718877 [0] NCCL INFO Channel 17/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718519:1718878 [1] NCCL INFO Channel 17/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718518:1718877 [0] NCCL INFO Channel 18/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718519:1718878 [1] NCCL INFO Channel 18/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718518:1718877 [0] NCCL INFO Channel 19/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718519:1718878 [1] NCCL INFO Channel 19/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718518:1718877 [0] NCCL INFO Channel 20/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718519:1718878 [1] NCCL INFO Channel 20/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718518:1718877 [0] NCCL INFO Channel 21/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718519:1718878 [1] NCCL INFO Channel 21/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718518:1718877 [0] NCCL INFO Channel 22/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718519:1718878 [1] NCCL INFO Channel 22/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718518:1718877 [0] NCCL INFO Channel 23/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1718519:1718878 [1] NCCL INFO Channel 23/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718518:1718877 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:1718519:1718878 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-06:05:05:11 INFO     [evaluator:559] Running loglikelihood requests
2025-12-06:05:05:11 INFO     [evaluator:559] Running loglikelihood requests

Running loglikelihood requests:   0%|          | 0/2344 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 64
Determined largest batch size: 64

Running loglikelihood requests:   0%|          | 1/2344 [00:39<25:51:04, 39.72s/it]
Running loglikelihood requests:   3%|▎         | 65/2344 [00:46<20:09,  1.88it/s]  
Running loglikelihood requests:   6%|▌         | 135/2344 [00:52<09:30,  3.87it/s]
Running loglikelihood requests:   9%|▊         | 204/2344 [00:57<06:18,  5.66it/s]
Running loglikelihood requests:  12%|█▏        | 276/2344 [01:03<04:41,  7.34it/s]
Running loglikelihood requests:  15%|█▍        | 341/2344 [01:08<03:56,  8.46it/s]
Running loglikelihood requests:  17%|█▋        | 408/2344 [01:13<03:23,  9.49it/s]
Running loglikelihood requests:  20%|██        | 475/2344 [01:19<03:01, 10.32it/s]
Running loglikelihood requests:  24%|██▎       | 551/2344 [01:24<02:36, 11.45it/s]
Running loglikelihood requests:  26%|██▋       | 616/2344 [01:29<02:27, 11.72it/s]
Running loglikelihood requests:  29%|██▉       | 684/2344 [01:34<02:17, 12.11it/s]
Running loglikelihood requests:  32%|███▏      | 748/2344 [01:40<02:10, 12.22it/s]
Running loglikelihood requests:  35%|███▍      | 815/2344 [01:45<02:02, 12.48it/s]
Running loglikelihood requests:  38%|███▊      | 883/2344 [01:50<01:56, 12.59it/s]
Running loglikelihood requests:  41%|████      | 952/2344 [01:55<01:47, 12.92it/s]
Running loglikelihood requests:  44%|████▎     | 1022/2344 [02:00<01:39, 13.23it/s]
Running loglikelihood requests:  47%|████▋     | 1094/2344 [02:05<01:31, 13.61it/s]
Running loglikelihood requests:  50%|█████     | 1172/2344 [02:10<01:22, 14.25it/s]
Running loglikelihood requests:  53%|█████▎    | 1238/2344 [02:15<01:18, 14.02it/s]
Running loglikelihood requests:  56%|█████▌    | 1306/2344 [02:20<01:14, 13.99it/s]
Running loglikelihood requests:  59%|█████▊    | 1377/2344 [02:25<01:08, 14.20it/s]
Running loglikelihood requests:  62%|██████▏   | 1451/2344 [02:29<01:01, 14.56it/s]
Running loglikelihood requests:  65%|██████▍   | 1518/2344 [02:34<00:57, 14.44it/s]
Running loglikelihood requests:  68%|██████▊   | 1587/2344 [02:39<00:52, 14.52it/s]
Running loglikelihood requests:  71%|███████   | 1654/2344 [02:43<00:47, 14.47it/s]
Running loglikelihood requests:  74%|███████▎  | 1724/2344 [02:48<00:42, 14.66it/s]
Running loglikelihood requests:  76%|███████▋  | 1793/2344 [02:53<00:37, 14.79it/s]
Running loglikelihood requests:  79%|███████▉  | 1863/2344 [02:57<00:32, 14.98it/s]
Running loglikelihood requests:  82%|████████▏ | 1933/2344 [03:02<00:27, 15.13it/s]
Running loglikelihood requests:  85%|████████▌ | 2000/2344 [03:06<00:22, 15.07it/s]
Running loglikelihood requests:  88%|████████▊ | 2074/2344 [03:11<00:17, 15.54it/s]
Running loglikelihood requests:  92%|█████████▏| 2146/2344 [03:15<00:12, 15.79it/s]
Running loglikelihood requests:  94%|█████████▍| 2214/2344 [03:19<00:08, 15.78it/s]
Running loglikelihood requests:  98%|█████████▊| 2297/2344 [03:22<00:02, 19.21it/s]
Running loglikelihood requests: 100%|██████████| 2344/2344 [03:22<00:00, 11.59it/s]
n136-128-154:1718519:1719392 [1] NCCL INFO Channel 00/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718519:1719392 [1] NCCL INFO Channel 01/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718519:1719392 [1] NCCL INFO Channel 02/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718519:1719392 [1] NCCL INFO Channel 03/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718519:1719392 [1] NCCL INFO Channel 04/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718519:1719392 [1] NCCL INFO Channel 05/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718519:1719392 [1] NCCL INFO Channel 06/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718519:1719392 [1] NCCL INFO Channel 07/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718519:1719392 [1] NCCL INFO Channel 08/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718519:1719392 [1] NCCL INFO Channel 09/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718519:1719392 [1] NCCL INFO Channel 10/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718519:1719392 [1] NCCL INFO Channel 11/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718519:1719392 [1] NCCL INFO Channel 12/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718519:1719392 [1] NCCL INFO Channel 13/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718519:1719392 [1] NCCL INFO Channel 14/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718519:1719392 [1] NCCL INFO Channel 15/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718519:1719392 [1] NCCL INFO Channel 16/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718519:1719392 [1] NCCL INFO Channel 17/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718519:1719392 [1] NCCL INFO Channel 18/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718519:1719392 [1] NCCL INFO Channel 19/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718519:1719392 [1] NCCL INFO Channel 20/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718519:1719392 [1] NCCL INFO Channel 21/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718519:1719392 [1] NCCL INFO Channel 22/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718519:1719392 [1] NCCL INFO Channel 23/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718519:1719392 [1] NCCL INFO Channel 24/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718519:1719392 [1] NCCL INFO Channel 25/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718519:1719392 [1] NCCL INFO Channel 26/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718519:1719392 [1] NCCL INFO Channel 27/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718519:1719392 [1] NCCL INFO Channel 28/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718519:1719392 [1] NCCL INFO Channel 29/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718519:1719392 [1] NCCL INFO Channel 30/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1718519:1719392 [1] NCCL INFO Channel 31/1 : 1[2] -> 0[1] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:1718519:1719399 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1718519:1719399 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1718519:1719399 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1718519:1719399 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1718519:1719399 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1718519:1719399 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1718519:1718873 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-06:05:08:55 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 25, batch_size: auto (64)
|    Tasks    |Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|-------------|------:|------|-----:|--------|---|-----:|---|-----:|
|arc_challenge|      1|none  |    25|acc     |↑  |0.3737|±  |0.0141|
|             |       |none  |    25|acc_norm|↑  |0.4147|±  |0.0144|

[rank0]:[W1206 05:08:56.682272706 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:1718518:1719449 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1718518:1719449 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1718518:1719449 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1718518:1718875 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:1718518:1719449 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1718518:1719449 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1718518:1719449 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1718519:1718873 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:1718519:1719399 [1] NCCL INFO comm 0x11a60100 rank 1 nranks 2 cudaDev 1 busId 4a000 - Abort COMPLETE
n136-128-154:1718518:1719449 [0] NCCL INFO comm 0x12d9fc50 rank 0 nranks 2 cudaDev 0 busId 16000 - Abort COMPLETE
任务 arc_challenge 评估完成！

==================================================
开始评估：任务=truthfulqa_mc1 | 少样本数=0 | 模型=Llama-3.1-8B-quantization-fg
输出路径：results2/Llama-3.1-8B-quantization-fg/fg/truthfulqa_mc1.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-06:05:10:17 INFO     [__main__:440] Selected Tasks: ['truthfulqa_mc1']
2025-12-06:05:10:17 INFO     [__main__:440] Selected Tasks: ['truthfulqa_mc1']
2025-12-06:05:10:17 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-06:05:10:17 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-06:05:10:17 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-06:05:10:17 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-06:05:10:18 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-06:05:10:19 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-06:05:10:19 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.14it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.31it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.03it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.13it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.00it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.05s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.36it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.21it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.41it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.28it/s]
2025-12-06:05:10:33 INFO     [evaluator:305] num_fewshot has been set to 0 for truthfulqa_mc1 in its config. Manual configuration will be ignored.
2025-12-06:05:10:33 INFO     [api.task:434] Building contexts for truthfulqa_mc1 on rank 0...
2025-12-06:05:10:33 INFO     [evaluator:305] num_fewshot has been set to 0 for truthfulqa_mc1 in its config. Manual configuration will be ignored.
2025-12-06:05:10:33 INFO     [api.task:434] Building contexts for truthfulqa_mc1 on rank 1...

  0%|          | 0/409 [00:00<?, ?it/s]
  0%|          | 0/408 [00:00<?, ?it/s]
 23%|██▎       | 94/409 [00:00<00:00, 931.30it/s]
 25%|██▍       | 101/408 [00:00<00:00, 1008.33it/s]
 47%|████▋     | 194/409 [00:00<00:00, 965.98it/s]
 50%|████▉     | 203/408 [00:00<00:00, 1012.59it/s]
 73%|███████▎  | 298/409 [00:00<00:00, 998.17it/s]
 75%|███████▌  | 307/408 [00:00<00:00, 1021.55it/s]
 98%|█████████▊| 402/409 [00:00<00:00, 1012.34it/s]
100%|██████████| 409/409 [00:00<00:00, 998.34it/s] 
n136-128-154:1719504:1719504 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1719504:1719504 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1719504:1719504 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:1719504:1719504 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:1719504:1719504 [0] NCCL INFO Comm config Blocking set to 1

100%|██████████| 408/408 [00:00<00:00, 1016.88it/s]
n136-128-154:1719505:1719505 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:1719505:1719505 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1719505:1719505 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1719505:1719505 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:1719505:1719505 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:1719504:1719691 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:1719505:1719692 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:1719504:1719691 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:1719505:1719692 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:1719504:1719691 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1719504:1719691 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1719505:1719692 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1719505:1719692 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1719504:1719691 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:1719505:1719692 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:1719504:1719691 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:1719504:1719691 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:1719504:1719691 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:1719505:1719692 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:1719505:1719692 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:1719505:1719692 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:1719504:1719691 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:1719504:1719691 [0] NCCL INFO Using network IB
n136-128-154:1719505:1719692 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:1719505:1719692 [1] NCCL INFO Using network IB
n136-128-154:1719505:1719692 [1] NCCL INFO ncclCommInitRankConfig comm 0x162a5740 rank 1 nranks 2 cudaDev 1 nvmlDev 2 busId 4a000 commId 0x2229f7460bbce154 - Init START
n136-128-154:1719504:1719691 [0] NCCL INFO ncclCommInitRankConfig comm 0x15177fb0 rank 0 nranks 2 cudaDev 0 nvmlDev 1 busId 16000 commId 0x2229f7460bbce154 - Init START
n136-128-154:1719504:1719691 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:1719505:1719692 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:1719504:1719691 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:1719504:1719691 [0] NCCL INFO Retrieving state for IB
n136-128-154:1719504:1719691 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:1719504:1719691 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:1719504:1719691 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:1719504:1719691 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:1719504:1719691 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:1719505:1719692 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:1719505:1719692 [1] NCCL INFO Retrieving state for IB
n136-128-154:1719505:1719692 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:1719505:1719692 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:1719505:1719692 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:1719505:1719692 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:1719505:1719692 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:1719504:1719691 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1719504:1719691 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1719504:1719691 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1719505:1719692 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1719504:1719691 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1719505:1719692 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1719505:1719692 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1719505:1719692 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1719505:1719692 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:1719504:1719691 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:1719505:1719692 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:1719504:1719691 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:1719505:1719692 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:1719504:1719691 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:1719505:1719692 [1] NCCL INFO               + PCI[24.0] - PCI/0-14000 (1000c01010de13b8)
n136-128-154:1719504:1719691 [0] NCCL INFO               + PCI[24.0] - PCI/0-14000 (1000c01010de13b8)
n136-128-154:1719505:1719692 [1] NCCL INFO                             + PCI[24.0] - GPU/0-16000 (0)
n136-128-154:1719504:1719691 [0] NCCL INFO                             + PCI[24.0] - GPU/0-16000 (0)
n136-128-154:1719505:1719692 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1719504:1719691 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1719505:1719692 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:1719504:1719691 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:1719505:1719692 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:1719504:1719691 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:1719505:1719692 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:1719504:1719691 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:1719505:1719692 [1] NCCL INFO               + PCI[24.0] - PCI/0-48000 (1000c01010de13b8)
n136-128-154:1719504:1719691 [0] NCCL INFO               + PCI[24.0] - PCI/0-48000 (1000c01010de13b8)
n136-128-154:1719505:1719692 [1] NCCL INFO                             + PCI[24.0] - GPU/0-4a000 (1)
n136-128-154:1719504:1719691 [0] NCCL INFO                             + PCI[24.0] - GPU/0-4a000 (1)
n136-128-154:1719505:1719692 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1719504:1719691 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1719505:1719692 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:1719504:1719691 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:1719505:1719692 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:1719504:1719691 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:1719505:1719692 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:1719504:1719691 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:1719505:1719692 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:1719504:1719691 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:1719505:1719692 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:1719504:1719691 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:1719505:1719692 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:1719504:1719691 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:1719505:1719692 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:1719504:1719691 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:1719505:1719692 [1] NCCL INFO ==========================================
n136-128-154:1719504:1719691 [0] NCCL INFO ==========================================
n136-128-154:1719505:1719692 [1] NCCL INFO GPU/0-16000 :GPU/0-16000 (0/5000.0/LOC) GPU/0-4a000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1719504:1719691 [0] NCCL INFO GPU/0-16000 :GPU/0-16000 (0/5000.0/LOC) GPU/0-4a000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1719505:1719692 [1] NCCL INFO GPU/0-4a000 :GPU/0-16000 (2/240.0/NVL) GPU/0-4a000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1719504:1719691 [0] NCCL INFO GPU/0-4a000 :GPU/0-16000 (2/240.0/NVL) GPU/0-4a000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1719504:1719691 [0] NCCL INFO Setting affinity for GPU 1 to 1-31,65-95
n136-128-154:1719505:1719692 [1] NCCL INFO Setting affinity for GPU 2 to 1-31,65-95
n136-128-154:1719505:1719692 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:1719505:1719692 [1] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719505:1719692 [1] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719505:1719692 [1] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719505:1719692 [1] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719505:1719692 [1] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719505:1719692 [1] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719505:1719692 [1] NCCL INFO  6 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719505:1719692 [1] NCCL INFO  7 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719504:1719691 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:1719505:1719692 [1] NCCL INFO  8 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719504:1719691 [0] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719505:1719692 [1] NCCL INFO  9 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719504:1719691 [0] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719505:1719692 [1] NCCL INFO 10 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719504:1719691 [0] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719505:1719692 [1] NCCL INFO 11 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719504:1719691 [0] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719504:1719691 [0] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719504:1719691 [0] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719504:1719691 [0] NCCL INFO  6 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719504:1719691 [0] NCCL INFO  7 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719504:1719691 [0] NCCL INFO  8 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719504:1719691 [0] NCCL INFO  9 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719504:1719691 [0] NCCL INFO 10 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719504:1719691 [0] NCCL INFO 11 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719505:1719692 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:1719504:1719691 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:1719505:1719692 [1] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719504:1719691 [0] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719505:1719692 [1] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719504:1719691 [0] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719505:1719692 [1] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719504:1719691 [0] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719505:1719692 [1] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719504:1719691 [0] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719505:1719692 [1] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719504:1719691 [0] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719505:1719692 [1] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719504:1719691 [0] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719505:1719692 [1] NCCL INFO  6 : GPU/0-4a000 GPU/0-16000
n136-128-154:1719504:1719691 [0] NCCL INFO  6 : GPU/0-4a000 GPU/0-16000
n136-128-154:1719505:1719692 [1] NCCL INFO  7 : GPU/0-4a000 GPU/0-16000
n136-128-154:1719504:1719691 [0] NCCL INFO  7 : GPU/0-4a000 GPU/0-16000
n136-128-154:1719505:1719692 [1] NCCL INFO  8 : GPU/0-4a000 GPU/0-16000
n136-128-154:1719504:1719691 [0] NCCL INFO  8 : GPU/0-4a000 GPU/0-16000
n136-128-154:1719505:1719692 [1] NCCL INFO  9 : GPU/0-4a000 GPU/0-16000
n136-128-154:1719504:1719691 [0] NCCL INFO  9 : GPU/0-4a000 GPU/0-16000
n136-128-154:1719505:1719692 [1] NCCL INFO 10 : GPU/0-4a000 GPU/0-16000
n136-128-154:1719504:1719691 [0] NCCL INFO 10 : GPU/0-4a000 GPU/0-16000
n136-128-154:1719505:1719692 [1] NCCL INFO 11 : GPU/0-4a000 GPU/0-16000
n136-128-154:1719504:1719691 [0] NCCL INFO 11 : GPU/0-4a000 GPU/0-16000
n136-128-154:1719505:1719692 [1] NCCL INFO comm 0x162a5740 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:1719504:1719691 [0] NCCL INFO comm 0x15177fb0 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:1719505:1719692 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:1719504:1719691 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:1719505:1719692 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:1719504:1719691 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:1719505:1719692 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:1719504:1719691 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:1719505:1719692 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:1719504:1719691 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:1719505:1719692 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:1719504:1719691 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:1719505:1719692 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:1719504:1719691 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:1719505:1719692 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:1719504:1719691 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:1719505:1719692 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:1719504:1719691 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:1719505:1719692 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:1719504:1719691 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:1719505:1719692 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:1719504:1719691 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:1719505:1719692 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:1719504:1719691 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:1719505:1719692 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:1719504:1719691 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:1719505:1719692 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:1719504:1719691 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:1719505:1719692 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:1719504:1719691 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:1719505:1719692 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:1719504:1719691 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:1719505:1719692 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:1719504:1719691 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:1719505:1719692 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:1719504:1719691 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:1719505:1719692 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:1719504:1719691 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:1719505:1719692 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:1719504:1719691 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:1719505:1719692 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:1719504:1719691 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:1719505:1719692 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:1719504:1719691 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:1719505:1719692 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:1719504:1719691 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:1719505:1719692 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:1719504:1719691 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:1719505:1719692 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:1719504:1719691 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:1719504:1719691 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:1719504:1719691 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:1719505:1719692 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:1719504:1719691 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:1719505:1719692 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:1719504:1719691 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:1719505:1719692 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:1719504:1719691 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:1719505:1719692 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:1719504:1719691 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:1719505:1719692 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:1719504:1719691 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:1719505:1719692 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:1719504:1719691 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:1719505:1719692 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:1719504:1719691 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:1719505:1719692 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:1719504:1719691 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:1719505:1719692 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:1719504:1719691 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:1719505:1719692 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:1719504:1719691 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:1719505:1719692 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:1719504:1719691 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:1719505:1719692 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:1719504:1719691 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:1719505:1719692 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:1719504:1719691 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:1719505:1719692 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:1719504:1719691 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:1719505:1719692 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:1719504:1719691 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:1719505:1719692 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:1719504:1719691 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:1719505:1719692 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:1719504:1719691 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:1719505:1719692 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:1719504:1719691 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:1719505:1719692 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:1719504:1719691 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:1719505:1719692 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:1719504:1719691 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:1719505:1719692 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:1719504:1719691 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:1719505:1719692 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:1719504:1719691 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:1719505:1719692 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:1719504:1719691 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:1719505:1719692 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:1719504:1719691 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:1719505:1719692 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:1719504:1719691 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:1719504:1719691 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:1719505:1719692 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:1719504:1719691 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:1719504:1719691 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:1719504:1719691 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:1719504:1719691 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:1719504:1719691 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:1719504:1719691 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:1719504:1719691 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:1719504:1719691 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:1719504:1719691 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:1719504:1719691 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:1719504:1719691 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:1719504:1719691 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:1719504:1719691 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:1719504:1719691 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:1719504:1719691 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:1719504:1719691 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:1719504:1719691 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:1719504:1719691 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:1719504:1719691 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:1719504:1719691 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:1719504:1719691 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:1719504:1719691 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:1719505:1719692 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:1719505:1719703 [1] NCCL INFO [Proxy Service] Device 1 CPU core 73
n136-128-154:1719505:1719704 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 10
n136-128-154:1719504:1719691 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:1719504:1719691 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:1719504:1719706 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 66
n136-128-154:1719504:1719705 [0] NCCL INFO [Proxy Service] Device 0 CPU core 65
n136-128-154:1719505:1719692 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:1719505:1719692 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:1719504:1719691 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:1719504:1719691 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:1719504:1719691 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:1719505:1719692 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:1719505:1719692 [1] NCCL INFO ncclCommInitRankConfig comm 0x162a5740 rank 1 nranks 2 cudaDev 1 nvmlDev 2 busId 4a000 commId 0x2229f7460bbce154 - Init COMPLETE
n136-128-154:1719505:1719692 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.69 (kernels 0.27, alloc 0.33, bootstrap 0.00, allgathers 0.00, topo 0.05, graphs 0.00, connections 0.01, rest 0.02)
n136-128-154:1719504:1719691 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:1719504:1719691 [0] NCCL INFO ncclCommInitRankConfig comm 0x15177fb0 rank 0 nranks 2 cudaDev 0 nvmlDev 1 busId 16000 commId 0x2229f7460bbce154 - Init COMPLETE
n136-128-154:1719504:1719691 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.70 (kernels 0.28, alloc 0.33, bootstrap 0.00, allgathers 0.00, topo 0.05, graphs 0.00, connections 0.01, rest 0.02)
n136-128-154:1719505:1719707 [1] NCCL INFO Channel 00/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719707 [1] NCCL INFO Channel 01/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719707 [1] NCCL INFO Channel 02/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719707 [1] NCCL INFO Channel 03/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719707 [1] NCCL INFO Channel 04/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719707 [1] NCCL INFO Channel 05/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719707 [1] NCCL INFO Channel 06/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719707 [1] NCCL INFO Channel 07/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719504:1719708 [0] NCCL INFO Channel 00/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719505:1719707 [1] NCCL INFO Channel 08/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719707 [1] NCCL INFO Channel 09/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719504:1719708 [0] NCCL INFO Channel 01/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719505:1719707 [1] NCCL INFO Channel 10/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719504:1719708 [0] NCCL INFO Channel 02/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719505:1719707 [1] NCCL INFO Channel 11/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719504:1719708 [0] NCCL INFO Channel 03/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719505:1719707 [1] NCCL INFO Channel 12/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719504:1719708 [0] NCCL INFO Channel 04/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719505:1719707 [1] NCCL INFO Channel 13/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719504:1719708 [0] NCCL INFO Channel 05/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719505:1719707 [1] NCCL INFO Channel 14/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719504:1719708 [0] NCCL INFO Channel 06/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719505:1719707 [1] NCCL INFO Channel 15/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719504:1719708 [0] NCCL INFO Channel 07/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719505:1719707 [1] NCCL INFO Channel 16/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719504:1719708 [0] NCCL INFO Channel 08/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719505:1719707 [1] NCCL INFO Channel 17/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719707 [1] NCCL INFO Channel 18/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719504:1719708 [0] NCCL INFO Channel 09/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719505:1719707 [1] NCCL INFO Channel 19/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719504:1719708 [0] NCCL INFO Channel 10/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719505:1719707 [1] NCCL INFO Channel 20/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719504:1719708 [0] NCCL INFO Channel 11/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719505:1719707 [1] NCCL INFO Channel 21/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719504:1719708 [0] NCCL INFO Channel 12/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719505:1719707 [1] NCCL INFO Channel 22/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719504:1719708 [0] NCCL INFO Channel 13/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719505:1719707 [1] NCCL INFO Channel 23/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719504:1719708 [0] NCCL INFO Channel 14/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719504:1719708 [0] NCCL INFO Channel 15/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719504:1719708 [0] NCCL INFO Channel 16/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719504:1719708 [0] NCCL INFO Channel 17/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719504:1719708 [0] NCCL INFO Channel 18/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719504:1719708 [0] NCCL INFO Channel 19/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719504:1719708 [0] NCCL INFO Channel 20/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719504:1719708 [0] NCCL INFO Channel 21/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719504:1719708 [0] NCCL INFO Channel 22/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719504:1719708 [0] NCCL INFO Channel 23/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719504:1719708 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:1719505:1719707 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-06:05:10:34 INFO     [evaluator:559] Running loglikelihood requests
2025-12-06:05:10:34 INFO     [evaluator:559] Running loglikelihood requests
Passed argument batch_size = auto:1. Detecting largest batch size

Running loglikelihood requests:   0%|          | 0/2066 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 64
Determined largest batch size: 64

Running loglikelihood requests:   0%|          | 1/2066 [00:07<4:13:02,  7.35s/it]
Running loglikelihood requests:   3%|▎         | 65/2066 [00:08<03:17, 10.11it/s] 
Running loglikelihood requests:   6%|▌         | 129/2066 [00:09<01:40, 19.29it/s]
Running loglikelihood requests:   9%|▉         | 193/2066 [00:10<01:08, 27.35it/s]
Running loglikelihood requests:  12%|█▏        | 257/2066 [00:12<00:53, 34.01it/s]
Running loglikelihood requests:  16%|█▌        | 321/2066 [00:13<00:44, 39.39it/s]
Running loglikelihood requests:  19%|█▊        | 385/2066 [00:14<00:38, 43.68it/s]
Running loglikelihood requests:  22%|██▏       | 449/2066 [00:15<00:34, 46.86it/s]
Running loglikelihood requests:  25%|██▍       | 513/2066 [00:16<00:31, 49.33it/s]
Running loglikelihood requests:  28%|██▊       | 577/2066 [00:17<00:29, 51.26it/s]
Running loglikelihood requests:  31%|███       | 641/2066 [00:19<00:27, 52.77it/s]
Running loglikelihood requests:  34%|███▍      | 705/2066 [00:20<00:25, 53.97it/s]
Running loglikelihood requests:  37%|███▋      | 769/2066 [00:21<00:23, 54.88it/s]
Running loglikelihood requests:  40%|████      | 833/2066 [00:22<00:22, 55.49it/s]
Running loglikelihood requests:  43%|████▎     | 897/2066 [00:23<00:20, 55.95it/s]
Running loglikelihood requests:  47%|████▋     | 961/2066 [00:24<00:19, 56.29it/s]
Running loglikelihood requests:  50%|████▉     | 1025/2066 [00:25<00:18, 56.56it/s]
Running loglikelihood requests:  53%|█████▎    | 1089/2066 [00:26<00:17, 56.86it/s]
Running loglikelihood requests:  56%|█████▌    | 1153/2066 [00:28<00:15, 57.11it/s]
Running loglikelihood requests:  59%|█████▉    | 1217/2066 [00:29<00:14, 57.32it/s]
Running loglikelihood requests:  62%|██████▏   | 1281/2066 [00:30<00:13, 57.54it/s]
Running loglikelihood requests:  65%|██████▌   | 1345/2066 [00:31<00:12, 57.62it/s]
Running loglikelihood requests:  68%|██████▊   | 1409/2066 [00:32<00:11, 57.56it/s]
Running loglikelihood requests:  71%|███████▏  | 1473/2066 [00:33<00:10, 57.50it/s]
Running loglikelihood requests:  74%|███████▍  | 1537/2066 [00:34<00:09, 57.59it/s]
Running loglikelihood requests:  77%|███████▋  | 1601/2066 [00:35<00:08, 57.57it/s]
Running loglikelihood requests:  81%|████████  | 1665/2066 [00:36<00:06, 57.36it/s]
Running loglikelihood requests:  84%|████████▎ | 1729/2066 [00:37<00:05, 57.81it/s]
Running loglikelihood requests:  87%|████████▋ | 1793/2066 [00:39<00:04, 58.08it/s]
Running loglikelihood requests:  90%|████████▉ | 1857/2066 [00:40<00:03, 58.50it/s]
Running loglikelihood requests:  93%|█████████▎| 1921/2066 [00:41<00:02, 58.83it/s]
Running loglikelihood requests:  96%|█████████▌| 1985/2066 [00:42<00:01, 59.01it/s]
Running loglikelihood requests:  99%|█████████▉| 2049/2066 [00:42<00:00, 75.13it/s]
Running loglikelihood requests: 100%|██████████| 2066/2066 [00:42<00:00, 48.49it/s]
n136-128-154:1719505:1719753 [1] NCCL INFO Channel 00/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719753 [1] NCCL INFO Channel 01/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719753 [1] NCCL INFO Channel 02/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719753 [1] NCCL INFO Channel 03/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719753 [1] NCCL INFO Channel 04/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719753 [1] NCCL INFO Channel 05/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719753 [1] NCCL INFO Channel 06/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719753 [1] NCCL INFO Channel 07/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719753 [1] NCCL INFO Channel 08/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719753 [1] NCCL INFO Channel 09/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719753 [1] NCCL INFO Channel 10/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719753 [1] NCCL INFO Channel 11/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719753 [1] NCCL INFO Channel 12/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719753 [1] NCCL INFO Channel 13/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719753 [1] NCCL INFO Channel 14/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719753 [1] NCCL INFO Channel 15/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719753 [1] NCCL INFO Channel 16/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719753 [1] NCCL INFO Channel 17/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719753 [1] NCCL INFO Channel 18/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719753 [1] NCCL INFO Channel 19/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719753 [1] NCCL INFO Channel 20/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719753 [1] NCCL INFO Channel 21/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719753 [1] NCCL INFO Channel 22/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719753 [1] NCCL INFO Channel 23/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719753 [1] NCCL INFO Channel 24/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719753 [1] NCCL INFO Channel 25/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719753 [1] NCCL INFO Channel 26/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719753 [1] NCCL INFO Channel 27/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719753 [1] NCCL INFO Channel 28/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719753 [1] NCCL INFO Channel 29/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719753 [1] NCCL INFO Channel 30/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719505:1719753 [1] NCCL INFO Channel 31/1 : 1[2] -> 0[1] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:1719505:1719759 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1719505:1719759 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1719505:1719759 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1719505:1719759 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1719505:1719759 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1719505:1719759 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1719505:1719703 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-06:05:11:29 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: auto (64)
|    Tasks     |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|--------------|------:|------|-----:|------|---|-----:|---|-----:|
|truthfulqa_mc1|      2|none  |     0|acc   |↑  |0.2497|±  |0.0152|

[rank0]:[W1206 05:11:30.034795243 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:1719504:1719821 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1719504:1719821 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1719504:1719821 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1719504:1719705 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:1719504:1719821 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1719504:1719821 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1719504:1719821 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1719505:1719703 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:1719505:1719759 [1] NCCL INFO comm 0x162a5740 rank 1 nranks 2 cudaDev 1 busId 4a000 - Abort COMPLETE
n136-128-154:1719504:1719821 [0] NCCL INFO comm 0x15177fb0 rank 0 nranks 2 cudaDev 0 busId 16000 - Abort COMPLETE
任务 truthfulqa_mc1 评估完成！

==================================================
开始评估：任务=piqa | 少样本数=0 | 模型=Llama-3.1-8B-quantization-fg
输出路径：results2/Llama-3.1-8B-quantization-fg/fg/piqa.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-06:05:12:53 INFO     [__main__:440] Selected Tasks: ['piqa']
2025-12-06:05:12:53 INFO     [__main__:440] Selected Tasks: ['piqa']
2025-12-06:05:12:53 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-06:05:12:53 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-06:05:12:53 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-06:05:12:53 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-06:05:12:54 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-06:05:12:55 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-06:05:12:55 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.15s/it]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.15s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.20s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.20s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.22s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.22s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.17it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.02it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.17it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.02it/s]
2025-12-06:05:13:13 WARNING  [evaluator:309] Overwriting default num_fewshot of piqa from None to 0
2025-12-06:05:13:13 INFO     [api.task:434] Building contexts for piqa on rank 1...

  0%|          | 0/919 [00:00<?, ?it/s]
 16%|█▌        | 149/919 [00:00<00:00, 1489.72it/s]
 33%|███▎      | 305/919 [00:00<00:00, 1529.01it/s]
 50%|█████     | 461/919 [00:00<00:00, 1541.31it/s]
 67%|██████▋   | 618/919 [00:00<00:00, 1549.10it/s]
 84%|████████▍ | 773/919 [00:00<00:00, 1544.06it/s]
100%|██████████| 919/919 [00:00<00:00, 1544.56it/s]
2025-12-06:05:13:16 WARNING  [evaluator:309] Overwriting default num_fewshot of piqa from None to 0
2025-12-06:05:13:16 INFO     [api.task:434] Building contexts for piqa on rank 0...

  0%|          | 0/919 [00:00<?, ?it/s]
 16%|█▌        | 149/919 [00:00<00:00, 1487.58it/s]
 33%|███▎      | 303/919 [00:00<00:00, 1513.30it/s]
 50%|████▉     | 456/919 [00:00<00:00, 1519.06it/s]
 66%|██████▋   | 609/919 [00:00<00:00, 1519.76it/s]
 83%|████████▎ | 765/919 [00:00<00:00, 1532.03it/s]
100%|██████████| 919/919 [00:00<00:00, 1528.65it/s]
n136-128-154:1719950:1719950 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1719950:1719950 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1719950:1719950 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:1719950:1719950 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:1719950:1719950 [0] NCCL INFO Comm config Blocking set to 1
n136-128-154:1719951:1719951 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:1719951:1719951 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1719951:1719951 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1719951:1719951 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:1719951:1719951 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:1719950:1720290 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:1719950:1720290 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:1719951:1720291 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:1719951:1720291 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:1719950:1720290 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1719950:1720290 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1719950:1720290 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:1719951:1720291 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1719951:1720291 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1719951:1720291 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:1719951:1720291 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:1719951:1720291 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:1719951:1720291 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:1719950:1720290 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:1719950:1720290 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:1719950:1720290 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:1719951:1720291 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:1719951:1720291 [1] NCCL INFO Using network IB
n136-128-154:1719950:1720290 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:1719950:1720290 [0] NCCL INFO Using network IB
n136-128-154:1719950:1720290 [0] NCCL INFO ncclCommInitRankConfig comm 0x15aaccd0 rank 0 nranks 2 cudaDev 0 nvmlDev 1 busId 16000 commId 0xa16a9b90c264df48 - Init START
n136-128-154:1719951:1720291 [1] NCCL INFO ncclCommInitRankConfig comm 0x130515a0 rank 1 nranks 2 cudaDev 1 nvmlDev 2 busId 4a000 commId 0xa16a9b90c264df48 - Init START
n136-128-154:1719950:1720290 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:1719951:1720291 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:1719951:1720291 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:1719951:1720291 [1] NCCL INFO Retrieving state for IB
n136-128-154:1719951:1720291 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:1719951:1720291 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:1719950:1720290 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:1719950:1720290 [0] NCCL INFO Retrieving state for IB
n136-128-154:1719950:1720290 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:1719951:1720291 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:1719950:1720290 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:1719950:1720290 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:1719951:1720291 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:1719950:1720290 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:1719951:1720291 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:1719950:1720290 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:1719951:1720291 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1719951:1720291 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1719951:1720291 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1719951:1720291 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1719950:1720290 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1719950:1720290 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1719950:1720290 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1719950:1720290 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1719951:1720291 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:1719951:1720291 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:1719950:1720290 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:1719951:1720291 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:1719950:1720290 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:1719951:1720291 [1] NCCL INFO               + PCI[24.0] - PCI/0-14000 (1000c01010de13b8)
n136-128-154:1719950:1720290 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:1719951:1720291 [1] NCCL INFO                             + PCI[24.0] - GPU/0-16000 (0)
n136-128-154:1719950:1720290 [0] NCCL INFO               + PCI[24.0] - PCI/0-14000 (1000c01010de13b8)
n136-128-154:1719951:1720291 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1719950:1720290 [0] NCCL INFO                             + PCI[24.0] - GPU/0-16000 (0)
n136-128-154:1719951:1720291 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:1719950:1720290 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1719951:1720291 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:1719950:1720290 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:1719951:1720291 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:1719950:1720290 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:1719951:1720291 [1] NCCL INFO               + PCI[24.0] - PCI/0-48000 (1000c01010de13b8)
n136-128-154:1719950:1720290 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:1719951:1720291 [1] NCCL INFO                             + PCI[24.0] - GPU/0-4a000 (1)
n136-128-154:1719950:1720290 [0] NCCL INFO               + PCI[24.0] - PCI/0-48000 (1000c01010de13b8)
n136-128-154:1719951:1720291 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1719950:1720290 [0] NCCL INFO                             + PCI[24.0] - GPU/0-4a000 (1)
n136-128-154:1719951:1720291 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:1719950:1720290 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1719951:1720291 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:1719950:1720290 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:1719951:1720291 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:1719950:1720290 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:1719951:1720291 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:1719950:1720290 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:1719951:1720291 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:1719950:1720290 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:1719951:1720291 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:1719950:1720290 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:1719951:1720291 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:1719950:1720290 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:1719951:1720291 [1] NCCL INFO ==========================================
n136-128-154:1719950:1720290 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:1719950:1720290 [0] NCCL INFO ==========================================
n136-128-154:1719951:1720291 [1] NCCL INFO GPU/0-16000 :GPU/0-16000 (0/5000.0/LOC) GPU/0-4a000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1719950:1720290 [0] NCCL INFO GPU/0-16000 :GPU/0-16000 (0/5000.0/LOC) GPU/0-4a000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1719951:1720291 [1] NCCL INFO GPU/0-4a000 :GPU/0-16000 (2/240.0/NVL) GPU/0-4a000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1719950:1720290 [0] NCCL INFO GPU/0-4a000 :GPU/0-16000 (2/240.0/NVL) GPU/0-4a000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1719951:1720291 [1] NCCL INFO Setting affinity for GPU 2 to 1-31,65-95
n136-128-154:1719950:1720290 [0] NCCL INFO Setting affinity for GPU 1 to 1-31,65-95
n136-128-154:1719951:1720291 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:1719951:1720291 [1] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719951:1720291 [1] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719951:1720291 [1] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719951:1720291 [1] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719951:1720291 [1] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719951:1720291 [1] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719951:1720291 [1] NCCL INFO  6 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719951:1720291 [1] NCCL INFO  7 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719951:1720291 [1] NCCL INFO  8 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719951:1720291 [1] NCCL INFO  9 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719951:1720291 [1] NCCL INFO 10 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719951:1720291 [1] NCCL INFO 11 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719950:1720290 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:1719950:1720290 [0] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719950:1720290 [0] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719950:1720290 [0] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719950:1720290 [0] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719950:1720290 [0] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719950:1720290 [0] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719950:1720290 [0] NCCL INFO  6 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719950:1720290 [0] NCCL INFO  7 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719950:1720290 [0] NCCL INFO  8 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719950:1720290 [0] NCCL INFO  9 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719950:1720290 [0] NCCL INFO 10 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719950:1720290 [0] NCCL INFO 11 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719951:1720291 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:1719951:1720291 [1] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719951:1720291 [1] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719951:1720291 [1] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719951:1720291 [1] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719951:1720291 [1] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719951:1720291 [1] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719951:1720291 [1] NCCL INFO  6 : GPU/0-4a000 GPU/0-16000
n136-128-154:1719951:1720291 [1] NCCL INFO  7 : GPU/0-4a000 GPU/0-16000
n136-128-154:1719951:1720291 [1] NCCL INFO  8 : GPU/0-4a000 GPU/0-16000
n136-128-154:1719951:1720291 [1] NCCL INFO  9 : GPU/0-4a000 GPU/0-16000
n136-128-154:1719950:1720290 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:1719951:1720291 [1] NCCL INFO 10 : GPU/0-4a000 GPU/0-16000
n136-128-154:1719950:1720290 [0] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719951:1720291 [1] NCCL INFO 11 : GPU/0-4a000 GPU/0-16000
n136-128-154:1719950:1720290 [0] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719950:1720290 [0] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719950:1720290 [0] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719950:1720290 [0] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719950:1720290 [0] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1719950:1720290 [0] NCCL INFO  6 : GPU/0-4a000 GPU/0-16000
n136-128-154:1719950:1720290 [0] NCCL INFO  7 : GPU/0-4a000 GPU/0-16000
n136-128-154:1719950:1720290 [0] NCCL INFO  8 : GPU/0-4a000 GPU/0-16000
n136-128-154:1719950:1720290 [0] NCCL INFO  9 : GPU/0-4a000 GPU/0-16000
n136-128-154:1719950:1720290 [0] NCCL INFO 10 : GPU/0-4a000 GPU/0-16000
n136-128-154:1719950:1720290 [0] NCCL INFO 11 : GPU/0-4a000 GPU/0-16000
n136-128-154:1719951:1720291 [1] NCCL INFO comm 0x130515a0 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:1719950:1720290 [0] NCCL INFO comm 0x15aaccd0 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:1719951:1720291 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:1719950:1720290 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:1719951:1720291 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:1719950:1720290 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:1719951:1720291 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:1719950:1720290 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:1719951:1720291 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:1719950:1720290 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:1719951:1720291 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:1719950:1720290 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:1719951:1720291 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:1719950:1720290 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:1719951:1720291 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:1719950:1720290 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:1719951:1720291 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:1719950:1720290 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:1719951:1720291 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:1719950:1720290 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:1719951:1720291 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:1719950:1720290 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:1719951:1720291 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:1719950:1720290 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:1719951:1720291 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:1719950:1720290 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:1719951:1720291 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:1719950:1720290 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:1719951:1720291 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:1719950:1720290 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:1719951:1720291 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:1719950:1720290 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:1719951:1720291 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:1719950:1720290 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:1719951:1720291 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:1719950:1720290 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:1719951:1720291 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:1719950:1720290 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:1719951:1720291 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:1719950:1720290 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:1719951:1720291 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:1719950:1720290 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:1719951:1720291 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:1719950:1720290 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:1719951:1720291 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:1719950:1720290 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:1719951:1720291 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:1719950:1720290 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:1719951:1720291 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:1719950:1720290 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:1719950:1720290 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:1719950:1720290 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:1719951:1720291 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:1719950:1720290 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:1719951:1720291 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:1719950:1720290 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:1719951:1720291 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:1719950:1720290 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:1719951:1720291 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:1719950:1720290 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:1719951:1720291 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:1719950:1720290 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:1719951:1720291 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:1719950:1720290 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:1719951:1720291 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:1719950:1720290 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:1719951:1720291 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:1719950:1720290 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:1719951:1720291 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:1719950:1720290 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:1719951:1720291 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:1719950:1720290 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:1719951:1720291 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:1719950:1720290 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:1719951:1720291 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:1719950:1720290 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:1719951:1720291 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:1719950:1720290 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:1719951:1720291 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:1719950:1720290 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:1719951:1720291 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:1719950:1720290 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:1719951:1720291 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:1719950:1720290 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:1719951:1720291 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:1719950:1720290 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:1719951:1720291 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:1719950:1720290 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:1719951:1720291 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:1719950:1720290 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:1719951:1720291 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:1719950:1720290 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:1719951:1720291 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:1719950:1720290 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:1719951:1720291 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:1719950:1720290 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:1719951:1720291 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:1719950:1720290 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:1719951:1720291 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:1719950:1720290 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:1719951:1720291 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:1719950:1720290 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:1719950:1720290 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:1719951:1720291 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:1719950:1720290 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:1719950:1720290 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:1719950:1720290 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:1719950:1720290 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:1719950:1720290 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:1719950:1720290 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:1719950:1720290 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:1719950:1720290 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:1719950:1720290 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:1719950:1720290 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:1719950:1720290 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:1719950:1720290 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:1719950:1720290 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:1719950:1720290 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:1719950:1720290 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:1719950:1720290 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:1719950:1720290 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:1719950:1720290 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:1719950:1720290 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:1719950:1720290 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:1719950:1720290 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:1719950:1720290 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:1719951:1720291 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:1719951:1720304 [1] NCCL INFO [Proxy Service] Device 1 CPU core 67
n136-128-154:1719951:1720305 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 68
n136-128-154:1719950:1720290 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:1719950:1720290 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:1719950:1720306 [0] NCCL INFO [Proxy Service] Device 0 CPU core 69
n136-128-154:1719950:1720307 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 14
n136-128-154:1719951:1720291 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:1719951:1720291 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:1719950:1720290 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:1719950:1720290 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:1719950:1720290 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:1719950:1720290 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:1719950:1720290 [0] NCCL INFO ncclCommInitRankConfig comm 0x15aaccd0 rank 0 nranks 2 cudaDev 0 nvmlDev 1 busId 16000 commId 0xa16a9b90c264df48 - Init COMPLETE
n136-128-154:1719950:1720290 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.77 (kernels 0.24, alloc 0.37, bootstrap 0.00, allgathers 0.00, topo 0.05, graphs 0.00, connections 0.01, rest 0.09)
n136-128-154:1719951:1720291 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:1719951:1720291 [1] NCCL INFO ncclCommInitRankConfig comm 0x130515a0 rank 1 nranks 2 cudaDev 1 nvmlDev 2 busId 4a000 commId 0xa16a9b90c264df48 - Init COMPLETE
n136-128-154:1719951:1720291 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.77 (kernels 0.23, alloc 0.37, bootstrap 0.00, allgathers 0.00, topo 0.05, graphs 0.00, connections 0.01, rest 0.09)
n136-128-154:1719950:1720308 [0] NCCL INFO Channel 00/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719950:1720308 [0] NCCL INFO Channel 01/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719950:1720308 [0] NCCL INFO Channel 02/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719950:1720308 [0] NCCL INFO Channel 03/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719951:1720309 [1] NCCL INFO Channel 00/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719950:1720308 [0] NCCL INFO Channel 04/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719951:1720309 [1] NCCL INFO Channel 01/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719950:1720308 [0] NCCL INFO Channel 05/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719951:1720309 [1] NCCL INFO Channel 02/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719950:1720308 [0] NCCL INFO Channel 06/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719951:1720309 [1] NCCL INFO Channel 03/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719950:1720308 [0] NCCL INFO Channel 07/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719951:1720309 [1] NCCL INFO Channel 04/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719950:1720308 [0] NCCL INFO Channel 08/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719951:1720309 [1] NCCL INFO Channel 05/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719950:1720308 [0] NCCL INFO Channel 09/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719951:1720309 [1] NCCL INFO Channel 06/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719950:1720308 [0] NCCL INFO Channel 10/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719951:1720309 [1] NCCL INFO Channel 07/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719950:1720308 [0] NCCL INFO Channel 11/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719951:1720309 [1] NCCL INFO Channel 08/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719950:1720308 [0] NCCL INFO Channel 12/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719951:1720309 [1] NCCL INFO Channel 09/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719950:1720308 [0] NCCL INFO Channel 13/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719951:1720309 [1] NCCL INFO Channel 10/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719950:1720308 [0] NCCL INFO Channel 14/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719951:1720309 [1] NCCL INFO Channel 11/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719950:1720308 [0] NCCL INFO Channel 15/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719951:1720309 [1] NCCL INFO Channel 12/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719950:1720308 [0] NCCL INFO Channel 16/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719951:1720309 [1] NCCL INFO Channel 13/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719950:1720308 [0] NCCL INFO Channel 17/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719951:1720309 [1] NCCL INFO Channel 14/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719950:1720308 [0] NCCL INFO Channel 18/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719951:1720309 [1] NCCL INFO Channel 15/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719950:1720308 [0] NCCL INFO Channel 19/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719951:1720309 [1] NCCL INFO Channel 16/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719950:1720308 [0] NCCL INFO Channel 20/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719951:1720309 [1] NCCL INFO Channel 17/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719950:1720308 [0] NCCL INFO Channel 21/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719951:1720309 [1] NCCL INFO Channel 18/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719950:1720308 [0] NCCL INFO Channel 22/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719951:1720309 [1] NCCL INFO Channel 19/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719950:1720308 [0] NCCL INFO Channel 23/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1719951:1720309 [1] NCCL INFO Channel 20/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719951:1720309 [1] NCCL INFO Channel 21/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719951:1720309 [1] NCCL INFO Channel 22/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719951:1720309 [1] NCCL INFO Channel 23/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719951:1720309 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:1719950:1720308 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-06:05:13:17 INFO     [evaluator:559] Running loglikelihood requests
2025-12-06:05:13:17 INFO     [evaluator:559] Running loglikelihood requests

Running loglikelihood requests:   0%|          | 0/1838 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 64
Determined largest batch size: 64

Running loglikelihood requests:   0%|          | 1/1838 [00:08<4:15:18,  8.34s/it]
Running loglikelihood requests:   4%|▎         | 65/1838 [00:09<02:59,  9.88it/s] 
Running loglikelihood requests:   7%|▋         | 129/1838 [00:09<01:21, 20.92it/s]
Running loglikelihood requests:  11%|█         | 194/1838 [00:10<00:49, 33.43it/s]
Running loglikelihood requests:  14%|█▍        | 258/1838 [00:10<00:33, 46.70it/s]
Running loglikelihood requests:  18%|█▊        | 322/1838 [00:11<00:25, 60.31it/s]
Running loglikelihood requests:  21%|██        | 386/1838 [00:11<00:19, 73.70it/s]
Running loglikelihood requests:  25%|██▍       | 451/1838 [00:12<00:15, 86.97it/s]
Running loglikelihood requests:  28%|██▊       | 515/1838 [00:12<00:13, 97.25it/s]
Running loglikelihood requests:  32%|███▏      | 579/1838 [00:13<00:11, 107.60it/s]
Running loglikelihood requests:  35%|███▍      | 643/1838 [00:13<00:10, 116.92it/s]
Running loglikelihood requests:  38%|███▊      | 707/1838 [00:14<00:09, 124.50it/s]
Running loglikelihood requests:  42%|████▏     | 771/1838 [00:14<00:08, 130.92it/s]
Running loglikelihood requests:  45%|████▌     | 835/1838 [00:14<00:07, 136.48it/s]
Running loglikelihood requests:  49%|████▉     | 899/1838 [00:15<00:06, 141.47it/s]
Running loglikelihood requests:  52%|█████▏    | 964/1838 [00:15<00:05, 146.19it/s]
Running loglikelihood requests:  56%|█████▌    | 1028/1838 [00:16<00:05, 150.40it/s]
Running loglikelihood requests:  59%|█████▉    | 1092/1838 [00:16<00:04, 154.16it/s]
Running loglikelihood requests:  63%|██████▎   | 1156/1838 [00:16<00:04, 157.45it/s]
Running loglikelihood requests:  66%|██████▋   | 1221/1838 [00:17<00:03, 161.22it/s]
Running loglikelihood requests:  70%|██████▉   | 1286/1838 [00:17<00:03, 164.88it/s]
Running loglikelihood requests:  73%|███████▎  | 1350/1838 [00:18<00:02, 167.39it/s]
Running loglikelihood requests:  77%|███████▋  | 1414/1838 [00:18<00:02, 169.33it/s]
Running loglikelihood requests:  80%|████████  | 1478/1838 [00:18<00:02, 172.13it/s]
Running loglikelihood requests:  84%|████████▍ | 1542/1838 [00:19<00:01, 175.21it/s]
Running loglikelihood requests:  87%|████████▋ | 1606/1838 [00:19<00:01, 178.04it/s]
Running loglikelihood requests:  91%|█████████ | 1670/1838 [00:19<00:00, 181.35it/s]
Running loglikelihood requests:  94%|█████████▍| 1734/1838 [00:20<00:00, 185.18it/s]
Running loglikelihood requests:  98%|█████████▊| 1799/1838 [00:20<00:00, 200.85it/s]
Running loglikelihood requests: 100%|██████████| 1838/1838 [00:20<00:00, 89.98it/s] 
n136-128-154:1719951:1720350 [1] NCCL INFO Channel 00/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719951:1720350 [1] NCCL INFO Channel 01/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719951:1720350 [1] NCCL INFO Channel 02/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719951:1720350 [1] NCCL INFO Channel 03/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719951:1720350 [1] NCCL INFO Channel 04/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719951:1720350 [1] NCCL INFO Channel 05/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719951:1720350 [1] NCCL INFO Channel 06/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719951:1720350 [1] NCCL INFO Channel 07/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719951:1720350 [1] NCCL INFO Channel 08/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719951:1720350 [1] NCCL INFO Channel 09/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719951:1720350 [1] NCCL INFO Channel 10/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719951:1720350 [1] NCCL INFO Channel 11/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719951:1720350 [1] NCCL INFO Channel 12/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719951:1720350 [1] NCCL INFO Channel 13/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719951:1720350 [1] NCCL INFO Channel 14/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719951:1720350 [1] NCCL INFO Channel 15/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719951:1720350 [1] NCCL INFO Channel 16/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719951:1720350 [1] NCCL INFO Channel 17/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719951:1720350 [1] NCCL INFO Channel 18/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719951:1720350 [1] NCCL INFO Channel 19/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719951:1720350 [1] NCCL INFO Channel 20/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719951:1720350 [1] NCCL INFO Channel 21/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719951:1720350 [1] NCCL INFO Channel 22/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719951:1720350 [1] NCCL INFO Channel 23/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719951:1720350 [1] NCCL INFO Channel 24/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719951:1720350 [1] NCCL INFO Channel 25/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719951:1720350 [1] NCCL INFO Channel 26/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719951:1720350 [1] NCCL INFO Channel 27/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719951:1720350 [1] NCCL INFO Channel 28/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719951:1720350 [1] NCCL INFO Channel 29/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719951:1720350 [1] NCCL INFO Channel 30/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1719951:1720350 [1] NCCL INFO Channel 31/1 : 1[2] -> 0[1] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:1719951:1720356 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1719951:1720356 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1719951:1720356 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1719951:1720356 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1719951:1720356 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1719951:1720356 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1719951:1720304 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-06:05:13:49 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: auto (64)
|Tasks|Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|-----|------:|------|-----:|--------|---|-----:|---|-----:|
|piqa |      1|none  |     0|acc     |↑  |0.7008|±  |0.0107|
|     |       |none  |     0|acc_norm|↑  |0.7024|±  |0.0107|

[rank0]:[W1206 05:13:50.372810255 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:1719950:1720407 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1719950:1720407 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1719950:1720407 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1719950:1720306 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:1719951:1720304 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:1719950:1720407 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1719950:1720407 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1719950:1720407 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1719951:1720356 [1] NCCL INFO comm 0x130515a0 rank 1 nranks 2 cudaDev 1 busId 4a000 - Abort COMPLETE
n136-128-154:1719950:1720407 [0] NCCL INFO comm 0x15aaccd0 rank 0 nranks 2 cudaDev 0 busId 16000 - Abort COMPLETE
任务 piqa 评估完成！

==================================================
开始评估：任务=hellaswag | 少样本数=10 | 模型=Llama-3.1-8B-quantization-fg
输出路径：results2/Llama-3.1-8B-quantization-fg/fg/hellaswag.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-06:05:15:09 INFO     [__main__:440] Selected Tasks: ['hellaswag']
2025-12-06:05:15:09 INFO     [__main__:440] Selected Tasks: ['hellaswag']
2025-12-06:05:15:09 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-06:05:15:09 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-06:05:15:09 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-06:05:15:09 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg'}
2025-12-06:05:15:10 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-06:05:15:11 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-06:05:15:11 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.16it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.15it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.09it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:01<00:01,  1.09it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.07it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:02<00:00,  1.03it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.53it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.33it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.47it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.29it/s]
2025-12-06:05:15:33 WARNING  [evaluator:309] Overwriting default num_fewshot of hellaswag from None to 10
2025-12-06:05:15:33 INFO     [api.task:434] Building contexts for hellaswag on rank 0...

  0%|          | 0/5021 [00:00<?, ?it/s]
  0%|          | 20/5021 [00:00<00:26, 191.78it/s]
  1%|          | 40/5021 [00:00<00:26, 190.84it/s]
  1%|          | 60/5021 [00:00<00:25, 193.23it/s]
  2%|▏         | 80/5021 [00:00<00:25, 195.03it/s]2025-12-06:05:15:35 WARNING  [evaluator:309] Overwriting default num_fewshot of hellaswag from None to 10
2025-12-06:05:15:35 INFO     [api.task:434] Building contexts for hellaswag on rank 1...

  2%|▏         | 100/5021 [00:00<00:25, 196.24it/s]
  2%|▏         | 120/5021 [00:00<00:24, 197.10it/s]
  3%|▎         | 140/5021 [00:00<00:24, 197.45it/s]
  3%|▎         | 160/5021 [00:00<00:24, 197.40it/s]
  4%|▎         | 180/5021 [00:00<00:24, 197.64it/s]
  4%|▍         | 200/5021 [00:01<00:24, 197.65it/s]
  4%|▍         | 220/5021 [00:01<00:24, 195.45it/s]
  5%|▍         | 240/5021 [00:01<00:24, 195.31it/s]
  5%|▌         | 260/5021 [00:01<00:24, 195.23it/s]
  6%|▌         | 280/5021 [00:01<00:24, 194.98it/s]
  6%|▌         | 300/5021 [00:01<00:24, 195.31it/s]
  0%|          | 0/5021 [00:00<?, ?it/s]
  6%|▋         | 320/5021 [00:01<00:24, 195.77it/s]
  0%|          | 19/5021 [00:00<00:26, 189.87it/s]
  7%|▋         | 340/5021 [00:01<00:23, 196.13it/s]
  1%|          | 39/5021 [00:00<00:25, 194.47it/s]
  7%|▋         | 360/5021 [00:01<00:23, 196.76it/s]
  1%|          | 59/5021 [00:00<00:25, 195.87it/s]
  8%|▊         | 381/5021 [00:01<00:23, 197.85it/s]
  2%|▏         | 79/5021 [00:00<00:25, 196.07it/s]
  8%|▊         | 401/5021 [00:02<00:23, 198.29it/s]
  2%|▏         | 99/5021 [00:00<00:25, 196.63it/s]
  8%|▊         | 421/5021 [00:02<00:23, 198.11it/s]
  2%|▏         | 119/5021 [00:00<00:24, 196.38it/s]
  9%|▉         | 441/5021 [00:02<00:23, 198.03it/s]
  3%|▎         | 139/5021 [00:00<00:24, 196.94it/s]
  9%|▉         | 461/5021 [00:02<00:23, 198.18it/s]
  3%|▎         | 160/5021 [00:00<00:24, 198.17it/s]
 10%|▉         | 481/5021 [00:02<00:22, 198.22it/s]
  4%|▎         | 181/5021 [00:00<00:24, 199.14it/s]
 10%|▉         | 501/5021 [00:02<00:22, 198.35it/s]
  4%|▍         | 202/5021 [00:01<00:24, 199.90it/s]
 10%|█         | 521/5021 [00:02<00:22, 198.48it/s]
  4%|▍         | 223/5021 [00:01<00:23, 200.57it/s]
 11%|█         | 541/5021 [00:02<00:22, 196.84it/s]
  5%|▍         | 244/5021 [00:01<00:23, 199.85it/s]
 11%|█         | 561/5021 [00:02<00:22, 197.13it/s]
  5%|▌         | 264/5021 [00:01<00:23, 199.32it/s]
 12%|█▏        | 581/5021 [00:02<00:22, 197.11it/s]
  6%|▌         | 284/5021 [00:01<00:23, 199.33it/s]
 12%|█▏        | 601/5021 [00:03<00:22, 196.58it/s]
  6%|▌         | 304/5021 [00:01<00:23, 199.48it/s]
 12%|█▏        | 621/5021 [00:03<00:22, 196.63it/s]
  6%|▋         | 324/5021 [00:01<00:23, 197.90it/s]
 13%|█▎        | 641/5021 [00:03<00:22, 196.58it/s]
  7%|▋         | 344/5021 [00:01<00:23, 197.47it/s]
 13%|█▎        | 661/5021 [00:03<00:22, 196.91it/s]
  7%|▋         | 364/5021 [00:01<00:23, 197.44it/s]
 14%|█▎        | 681/5021 [00:03<00:22, 196.93it/s]
  8%|▊         | 384/5021 [00:01<00:23, 197.52it/s]
 14%|█▍        | 701/5021 [00:03<00:21, 197.24it/s]
  8%|▊         | 404/5021 [00:02<00:23, 197.80it/s]
 14%|█▍        | 721/5021 [00:03<00:21, 197.36it/s]
  8%|▊         | 424/5021 [00:02<00:23, 198.10it/s]
 15%|█▍        | 741/5021 [00:03<00:21, 197.27it/s]
  9%|▉         | 444/5021 [00:02<00:23, 198.33it/s]
 15%|█▌        | 761/5021 [00:03<00:21, 197.19it/s]
  9%|▉         | 464/5021 [00:02<00:22, 198.35it/s]
 16%|█▌        | 781/5021 [00:03<00:21, 196.70it/s]
 10%|▉         | 484/5021 [00:02<00:23, 195.90it/s]
 16%|█▌        | 801/5021 [00:04<00:21, 196.83it/s]
 10%|█         | 504/5021 [00:02<00:23, 196.04it/s]
 16%|█▋        | 821/5021 [00:04<00:21, 196.66it/s]
 10%|█         | 524/5021 [00:02<00:22, 196.01it/s]
 17%|█▋        | 841/5021 [00:04<00:21, 196.56it/s]
 11%|█         | 544/5021 [00:02<00:22, 195.91it/s]
 17%|█▋        | 861/5021 [00:04<00:21, 194.76it/s]
 11%|█         | 564/5021 [00:02<00:22, 196.04it/s]
 18%|█▊        | 881/5021 [00:04<00:21, 194.02it/s]
 12%|█▏        | 584/5021 [00:02<00:22, 196.16it/s]
 18%|█▊        | 901/5021 [00:04<00:21, 194.52it/s]
 12%|█▏        | 604/5021 [00:03<00:22, 195.81it/s]
 18%|█▊        | 921/5021 [00:04<00:21, 194.75it/s]
 12%|█▏        | 624/5021 [00:03<00:22, 196.73it/s]
 19%|█▊        | 941/5021 [00:04<00:20, 194.89it/s]
 13%|█▎        | 644/5021 [00:03<00:22, 197.05it/s]
 19%|█▉        | 961/5021 [00:04<00:20, 195.40it/s]
 13%|█▎        | 664/5021 [00:03<00:22, 195.09it/s]
 20%|█▉        | 981/5021 [00:04<00:20, 195.87it/s]
 14%|█▎        | 684/5021 [00:03<00:22, 195.06it/s]
 20%|█▉        | 1001/5021 [00:05<00:20, 196.01it/s]
 14%|█▍        | 704/5021 [00:03<00:22, 194.82it/s]
 20%|██        | 1021/5021 [00:05<00:20, 195.86it/s]
 14%|█▍        | 724/5021 [00:03<00:22, 194.65it/s]
 21%|██        | 1041/5021 [00:05<00:20, 195.72it/s]
 15%|█▍        | 744/5021 [00:03<00:21, 194.94it/s]
 21%|██        | 1061/5021 [00:05<00:20, 195.77it/s]
 15%|█▌        | 764/5021 [00:03<00:21, 195.44it/s]
 22%|██▏       | 1081/5021 [00:05<00:20, 195.53it/s]
 16%|█▌        | 784/5021 [00:03<00:21, 195.29it/s]
 22%|██▏       | 1101/5021 [00:05<00:20, 195.35it/s]
 16%|█▌        | 804/5021 [00:04<00:21, 195.51it/s]
 22%|██▏       | 1121/5021 [00:05<00:19, 195.43it/s]
 16%|█▋        | 824/5021 [00:04<00:21, 195.51it/s]
 23%|██▎       | 1141/5021 [00:05<00:19, 195.65it/s]
 17%|█▋        | 844/5021 [00:04<00:21, 195.33it/s]
 23%|██▎       | 1161/5021 [00:05<00:19, 195.81it/s]
 17%|█▋        | 864/5021 [00:04<00:21, 195.45it/s]
 24%|██▎       | 1181/5021 [00:06<00:19, 196.15it/s]
 18%|█▊        | 884/5021 [00:04<00:21, 195.32it/s]
 24%|██▍       | 1201/5021 [00:06<00:19, 194.09it/s]
 18%|█▊        | 904/5021 [00:04<00:21, 195.87it/s]
 24%|██▍       | 1221/5021 [00:06<00:19, 194.74it/s]
 18%|█▊        | 924/5021 [00:04<00:20, 195.68it/s]
 25%|██▍       | 1241/5021 [00:06<00:19, 195.08it/s]
 19%|█▉        | 944/5021 [00:04<00:20, 195.70it/s]
 25%|██▌       | 1261/5021 [00:06<00:19, 195.56it/s]
 19%|█▉        | 964/5021 [00:04<00:20, 195.77it/s]
 26%|██▌       | 1281/5021 [00:06<00:19, 195.85it/s]
 20%|█▉        | 984/5021 [00:05<00:20, 194.54it/s]
 26%|██▌       | 1301/5021 [00:06<00:18, 196.26it/s]
 20%|█▉        | 1004/5021 [00:05<00:20, 195.20it/s]
 26%|██▋       | 1321/5021 [00:06<00:18, 196.76it/s]
 20%|██        | 1024/5021 [00:05<00:20, 195.86it/s]
 27%|██▋       | 1341/5021 [00:06<00:18, 197.03it/s]
 21%|██        | 1044/5021 [00:05<00:20, 196.45it/s]
 27%|██▋       | 1361/5021 [00:06<00:18, 197.53it/s]
 21%|██        | 1064/5021 [00:05<00:20, 195.71it/s]
 28%|██▊       | 1381/5021 [00:07<00:18, 196.83it/s]
 22%|██▏       | 1084/5021 [00:05<00:20, 196.05it/s]
 28%|██▊       | 1401/5021 [00:07<00:18, 197.02it/s]
 22%|██▏       | 1104/5021 [00:05<00:19, 196.40it/s]
 28%|██▊       | 1421/5021 [00:07<00:18, 197.10it/s]
 22%|██▏       | 1124/5021 [00:05<00:19, 196.66it/s]
 29%|██▊       | 1441/5021 [00:07<00:18, 197.36it/s]
 23%|██▎       | 1144/5021 [00:05<00:19, 197.01it/s]
 29%|██▉       | 1461/5021 [00:07<00:18, 197.49it/s]
 23%|██▎       | 1164/5021 [00:05<00:19, 197.27it/s]
 29%|██▉       | 1481/5021 [00:07<00:17, 197.60it/s]
 24%|██▎       | 1184/5021 [00:06<00:19, 197.25it/s]
 30%|██▉       | 1501/5021 [00:07<00:17, 197.21it/s]
 24%|██▍       | 1204/5021 [00:06<00:19, 197.05it/s]
 30%|███       | 1521/5021 [00:07<00:17, 197.69it/s]
 24%|██▍       | 1224/5021 [00:06<00:19, 196.90it/s]
 31%|███       | 1541/5021 [00:07<00:17, 197.82it/s]
 25%|██▍       | 1244/5021 [00:06<00:19, 196.86it/s]
 31%|███       | 1561/5021 [00:07<00:17, 197.46it/s]
 25%|██▌       | 1264/5021 [00:06<00:19, 197.07it/s]
 31%|███▏      | 1581/5021 [00:08<00:17, 197.32it/s]
 26%|██▌       | 1284/5021 [00:06<00:18, 197.11it/s]
 32%|███▏      | 1601/5021 [00:08<00:17, 197.19it/s]
 26%|██▌       | 1304/5021 [00:06<00:18, 196.73it/s]
 32%|███▏      | 1621/5021 [00:08<00:17, 197.27it/s]
 26%|██▋       | 1324/5021 [00:06<00:18, 196.74it/s]
 33%|███▎      | 1641/5021 [00:08<00:17, 197.26it/s]
 27%|██▋       | 1344/5021 [00:06<00:18, 196.89it/s]
 33%|███▎      | 1661/5021 [00:08<00:17, 197.18it/s]
 27%|██▋       | 1364/5021 [00:06<00:18, 196.98it/s]
 33%|███▎      | 1681/5021 [00:08<00:16, 196.55it/s]
 28%|██▊       | 1384/5021 [00:07<00:19, 190.50it/s]
 34%|███▍      | 1701/5021 [00:08<00:17, 193.64it/s]
 28%|██▊       | 1404/5021 [00:07<00:18, 193.01it/s]
 34%|███▍      | 1721/5021 [00:08<00:16, 194.63it/s]
 28%|██▊       | 1424/5021 [00:07<00:18, 194.95it/s]
 35%|███▍      | 1741/5021 [00:08<00:16, 195.50it/s]
 29%|██▉       | 1444/5021 [00:07<00:18, 196.20it/s]
 35%|███▌      | 1761/5021 [00:08<00:16, 196.09it/s]
 29%|██▉       | 1464/5021 [00:07<00:18, 196.92it/s]
 35%|███▌      | 1781/5021 [00:09<00:16, 196.95it/s]
 30%|██▉       | 1484/5021 [00:07<00:17, 197.48it/s]
 36%|███▌      | 1801/5021 [00:09<00:16, 197.83it/s]
 30%|██▉       | 1504/5021 [00:07<00:17, 198.19it/s]
 36%|███▋      | 1821/5021 [00:09<00:16, 198.14it/s]
 30%|███       | 1524/5021 [00:07<00:17, 198.67it/s]
 37%|███▋      | 1841/5021 [00:09<00:16, 197.14it/s]
 31%|███       | 1545/5021 [00:07<00:17, 199.18it/s]
 37%|███▋      | 1861/5021 [00:09<00:16, 197.45it/s]
 31%|███       | 1566/5021 [00:07<00:17, 199.82it/s]
 37%|███▋      | 1881/5021 [00:09<00:15, 197.93it/s]
 32%|███▏      | 1587/5021 [00:08<00:17, 199.79it/s]
 38%|███▊      | 1901/5021 [00:09<00:15, 198.52it/s]
 32%|███▏      | 1607/5021 [00:08<00:17, 199.42it/s]
 38%|███▊      | 1921/5021 [00:09<00:15, 198.95it/s]
 32%|███▏      | 1627/5021 [00:08<00:17, 199.09it/s]
 39%|███▊      | 1942/5021 [00:09<00:15, 199.23it/s]
 33%|███▎      | 1647/5021 [00:08<00:16, 199.23it/s]
 39%|███▉      | 1962/5021 [00:09<00:15, 199.14it/s]
 33%|███▎      | 1667/5021 [00:08<00:16, 199.08it/s]
 39%|███▉      | 1982/5021 [00:10<00:15, 198.92it/s]
 34%|███▎      | 1687/5021 [00:08<00:16, 198.54it/s]
 40%|███▉      | 2002/5021 [00:10<00:15, 198.82it/s]
 34%|███▍      | 1707/5021 [00:08<00:16, 198.42it/s]
 40%|████      | 2022/5021 [00:10<00:15, 198.50it/s]
 34%|███▍      | 1727/5021 [00:08<00:16, 198.64it/s]
 41%|████      | 2042/5021 [00:10<00:15, 198.07it/s]
 35%|███▍      | 1747/5021 [00:08<00:16, 198.47it/s]
 41%|████      | 2062/5021 [00:10<00:14, 198.09it/s]
 35%|███▌      | 1767/5021 [00:08<00:16, 198.70it/s]
 41%|████▏     | 2082/5021 [00:10<00:14, 198.34it/s]
 36%|███▌      | 1787/5021 [00:09<00:16, 198.33it/s]
 42%|████▏     | 2102/5021 [00:10<00:14, 197.59it/s]
 36%|███▌      | 1808/5021 [00:09<00:16, 198.93it/s]
 42%|████▏     | 2122/5021 [00:10<00:14, 197.82it/s]
 36%|███▋      | 1828/5021 [00:09<00:16, 198.39it/s]
 43%|████▎     | 2142/5021 [00:10<00:14, 198.46it/s]
 37%|███▋      | 1848/5021 [00:09<00:16, 198.09it/s]
 43%|████▎     | 2163/5021 [00:10<00:14, 199.25it/s]
 37%|███▋      | 1868/5021 [00:09<00:15, 198.39it/s]
 43%|████▎     | 2183/5021 [00:11<00:14, 198.87it/s]
 38%|███▊      | 1888/5021 [00:09<00:15, 198.59it/s]
 44%|████▍     | 2203/5021 [00:11<00:14, 199.01it/s]
 38%|███▊      | 1908/5021 [00:09<00:15, 198.35it/s]
 44%|████▍     | 2223/5021 [00:11<00:14, 199.05it/s]
 38%|███▊      | 1929/5021 [00:09<00:15, 199.06it/s]
 45%|████▍     | 2243/5021 [00:11<00:13, 198.86it/s]
 39%|███▉      | 1949/5021 [00:09<00:15, 196.53it/s]
 45%|████▌     | 2263/5021 [00:11<00:13, 198.64it/s]
 39%|███▉      | 1969/5021 [00:09<00:15, 196.47it/s]
 45%|████▌     | 2283/5021 [00:11<00:13, 198.98it/s]
 40%|███▉      | 1989/5021 [00:10<00:15, 196.59it/s]
 46%|████▌     | 2303/5021 [00:11<00:13, 199.11it/s]
 40%|████      | 2009/5021 [00:10<00:15, 196.63it/s]
 46%|████▋     | 2323/5021 [00:11<00:13, 199.17it/s]
 40%|████      | 2029/5021 [00:10<00:15, 196.82it/s]
 47%|████▋     | 2343/5021 [00:11<00:13, 198.87it/s]
 41%|████      | 2049/5021 [00:10<00:15, 196.31it/s]
 47%|████▋     | 2363/5021 [00:11<00:13, 198.83it/s]
 41%|████      | 2069/5021 [00:10<00:15, 196.26it/s]
 47%|████▋     | 2383/5021 [00:12<00:13, 198.44it/s]
 42%|████▏     | 2089/5021 [00:10<00:14, 196.34it/s]
 48%|████▊     | 2403/5021 [00:12<00:13, 198.14it/s]
 42%|████▏     | 2109/5021 [00:10<00:14, 195.97it/s]
 48%|████▊     | 2423/5021 [00:12<00:13, 198.00it/s]
 42%|████▏     | 2129/5021 [00:10<00:14, 196.49it/s]
 49%|████▊     | 2443/5021 [00:12<00:13, 198.07it/s]
 43%|████▎     | 2149/5021 [00:10<00:14, 196.63it/s]
 49%|████▉     | 2463/5021 [00:12<00:12, 198.28it/s]
 43%|████▎     | 2169/5021 [00:11<00:14, 196.61it/s]
 49%|████▉     | 2483/5021 [00:12<00:12, 196.50it/s]
 44%|████▎     | 2189/5021 [00:11<00:14, 195.29it/s]
 50%|████▉     | 2503/5021 [00:12<00:12, 194.97it/s]
 44%|████▍     | 2209/5021 [00:11<00:14, 195.56it/s]
 50%|█████     | 2523/5021 [00:12<00:12, 195.02it/s]
 44%|████▍     | 2229/5021 [00:11<00:14, 196.10it/s]
 51%|█████     | 2543/5021 [00:12<00:12, 195.23it/s]
 45%|████▍     | 2249/5021 [00:11<00:14, 195.98it/s]
 51%|█████     | 2563/5021 [00:13<00:12, 195.35it/s]
 45%|████▌     | 2269/5021 [00:11<00:14, 194.38it/s]
 51%|█████▏    | 2583/5021 [00:13<00:12, 195.70it/s]
 46%|████▌     | 2289/5021 [00:11<00:14, 195.03it/s]
 52%|█████▏    | 2603/5021 [00:13<00:12, 195.92it/s]
 46%|████▌     | 2309/5021 [00:11<00:13, 195.62it/s]
 52%|█████▏    | 2623/5021 [00:13<00:12, 196.32it/s]
 46%|████▋     | 2329/5021 [00:11<00:13, 196.25it/s]
 53%|█████▎    | 2643/5021 [00:13<00:12, 196.45it/s]
 47%|████▋     | 2349/5021 [00:11<00:13, 196.62it/s]
 53%|█████▎    | 2663/5021 [00:13<00:11, 196.66it/s]
 47%|████▋     | 2369/5021 [00:12<00:13, 197.19it/s]
 53%|█████▎    | 2683/5021 [00:13<00:11, 196.84it/s]
 48%|████▊     | 2390/5021 [00:12<00:13, 197.95it/s]
 54%|█████▍    | 2703/5021 [00:13<00:11, 197.21it/s]
 48%|████▊     | 2410/5021 [00:12<00:13, 197.80it/s]
 54%|█████▍    | 2723/5021 [00:13<00:11, 196.98it/s]
 48%|████▊     | 2430/5021 [00:12<00:13, 197.64it/s]
 55%|█████▍    | 2743/5021 [00:13<00:11, 195.82it/s]
 49%|████▉     | 2450/5021 [00:12<00:13, 197.53it/s]
 55%|█████▌    | 2763/5021 [00:14<00:11, 195.00it/s]
 49%|████▉     | 2470/5021 [00:12<00:12, 197.51it/s]
 55%|█████▌    | 2783/5021 [00:14<00:11, 195.56it/s]
 50%|████▉     | 2490/5021 [00:12<00:12, 197.35it/s]
 56%|█████▌    | 2803/5021 [00:14<00:11, 195.96it/s]
 50%|████▉     | 2510/5021 [00:12<00:12, 197.04it/s]
 56%|█████▌    | 2823/5021 [00:14<00:11, 196.35it/s]
 50%|█████     | 2530/5021 [00:12<00:12, 197.15it/s]
 57%|█████▋    | 2843/5021 [00:14<00:11, 196.48it/s]
 51%|█████     | 2550/5021 [00:12<00:12, 197.06it/s]
 57%|█████▋    | 2863/5021 [00:14<00:10, 196.72it/s]
 51%|█████     | 2570/5021 [00:13<00:12, 196.86it/s]
 57%|█████▋    | 2883/5021 [00:14<00:10, 196.97it/s]
 52%|█████▏    | 2590/5021 [00:13<00:12, 196.93it/s]
 58%|█████▊    | 2903/5021 [00:14<00:10, 197.16it/s]
 52%|█████▏    | 2610/5021 [00:13<00:12, 196.92it/s]
 58%|█████▊    | 2923/5021 [00:14<00:10, 197.10it/s]
 52%|█████▏    | 2630/5021 [00:13<00:12, 197.02it/s]
 59%|█████▊    | 2943/5021 [00:14<00:10, 197.56it/s]
 53%|█████▎    | 2650/5021 [00:13<00:12, 197.19it/s]
 59%|█████▉    | 2963/5021 [00:15<00:10, 197.83it/s]
 53%|█████▎    | 2670/5021 [00:13<00:11, 197.45it/s]
 59%|█████▉    | 2983/5021 [00:15<00:10, 197.99it/s]
 54%|█████▎    | 2690/5021 [00:13<00:11, 197.85it/s]
 60%|█████▉    | 3003/5021 [00:15<00:10, 198.09it/s]
 54%|█████▍    | 2711/5021 [00:13<00:11, 198.54it/s]
 60%|██████    | 3023/5021 [00:15<00:10, 198.35it/s]
 54%|█████▍    | 2732/5021 [00:13<00:11, 199.65it/s]
 61%|██████    | 3044/5021 [00:15<00:09, 198.82it/s]
 55%|█████▍    | 2752/5021 [00:13<00:11, 199.65it/s]
 61%|██████    | 3065/5021 [00:15<00:09, 199.25it/s]
 55%|█████▌    | 2772/5021 [00:14<00:11, 199.60it/s]
 61%|██████▏   | 3085/5021 [00:15<00:09, 199.08it/s]
 56%|█████▌    | 2792/5021 [00:14<00:11, 199.58it/s]
 62%|██████▏   | 3106/5021 [00:15<00:09, 199.43it/s]
 56%|█████▌    | 2812/5021 [00:14<00:11, 199.54it/s]
 62%|██████▏   | 3126/5021 [00:15<00:09, 197.15it/s]
 56%|█████▋    | 2832/5021 [00:14<00:10, 199.58it/s]
 63%|██████▎   | 3146/5021 [00:15<00:09, 197.96it/s]
 57%|█████▋    | 2852/5021 [00:14<00:10, 199.50it/s]
 63%|██████▎   | 3166/5021 [00:16<00:09, 197.97it/s]
 57%|█████▋    | 2872/5021 [00:14<00:10, 198.95it/s]
 63%|██████▎   | 3186/5021 [00:16<00:09, 197.96it/s]
 58%|█████▊    | 2892/5021 [00:14<00:10, 198.28it/s]
 64%|██████▍   | 3206/5021 [00:16<00:09, 198.01it/s]
 58%|█████▊    | 2912/5021 [00:14<00:10, 196.91it/s]
 64%|██████▍   | 3226/5021 [00:16<00:09, 197.52it/s]
 58%|█████▊    | 2932/5021 [00:14<00:10, 197.69it/s]
 65%|██████▍   | 3246/5021 [00:16<00:08, 197.38it/s]
 59%|█████▉    | 2952/5021 [00:14<00:10, 198.35it/s]
 65%|██████▌   | 3266/5021 [00:16<00:08, 197.32it/s]
 59%|█████▉    | 2972/5021 [00:15<00:10, 198.01it/s]
 65%|██████▌   | 3286/5021 [00:16<00:08, 197.40it/s]
 60%|█████▉    | 2992/5021 [00:15<00:10, 197.83it/s]
 66%|██████▌   | 3306/5021 [00:16<00:08, 197.50it/s]
 60%|█████▉    | 3012/5021 [00:15<00:10, 198.28it/s]
 66%|██████▌   | 3326/5021 [00:16<00:08, 197.50it/s]
 60%|██████    | 3032/5021 [00:15<00:10, 197.97it/s]
 67%|██████▋   | 3346/5021 [00:16<00:08, 197.69it/s]
 61%|██████    | 3052/5021 [00:15<00:09, 198.19it/s]
 67%|██████▋   | 3366/5021 [00:17<00:08, 197.88it/s]
 61%|██████    | 3072/5021 [00:15<00:09, 198.20it/s]
 67%|██████▋   | 3386/5021 [00:17<00:08, 197.83it/s]
 62%|██████▏   | 3092/5021 [00:15<00:09, 198.48it/s]
 68%|██████▊   | 3406/5021 [00:17<00:08, 197.74it/s]
 62%|██████▏   | 3112/5021 [00:15<00:09, 198.02it/s]
 68%|██████▊   | 3426/5021 [00:17<00:08, 196.72it/s]
 62%|██████▏   | 3133/5021 [00:15<00:09, 198.88it/s]
 69%|██████▊   | 3446/5021 [00:17<00:07, 197.05it/s]
 63%|██████▎   | 3154/5021 [00:15<00:09, 199.29it/s]
 69%|██████▉   | 3466/5021 [00:17<00:07, 197.24it/s]
 63%|██████▎   | 3175/5021 [00:16<00:09, 200.45it/s]
 69%|██████▉   | 3486/5021 [00:17<00:07, 197.99it/s]
 64%|██████▎   | 3196/5021 [00:16<00:09, 201.13it/s]
 70%|██████▉   | 3507/5021 [00:17<00:07, 198.68it/s]
 64%|██████▍   | 3217/5021 [00:16<00:08, 201.66it/s]
 70%|███████   | 3527/5021 [00:17<00:07, 198.86it/s]
 64%|██████▍   | 3238/5021 [00:16<00:08, 201.39it/s]
 71%|███████   | 3548/5021 [00:18<00:07, 199.12it/s]
 65%|██████▍   | 3259/5021 [00:16<00:08, 200.92it/s]
 71%|███████   | 3569/5021 [00:18<00:07, 199.49it/s]
 65%|██████▌   | 3280/5021 [00:16<00:08, 200.31it/s]
 71%|███████▏  | 3589/5021 [00:18<00:07, 199.35it/s]
 66%|██████▌   | 3301/5021 [00:16<00:08, 199.72it/s]
 72%|███████▏  | 3609/5021 [00:18<00:07, 199.17it/s]
 66%|██████▌   | 3321/5021 [00:16<00:08, 199.35it/s]
 72%|███████▏  | 3629/5021 [00:18<00:07, 198.45it/s]
 67%|██████▋   | 3341/5021 [00:16<00:08, 199.44it/s]
 73%|███████▎  | 3649/5021 [00:18<00:06, 198.40it/s]
 67%|██████▋   | 3361/5021 [00:17<00:08, 199.39it/s]
 73%|███████▎  | 3669/5021 [00:18<00:06, 198.08it/s]
 73%|███████▎  | 3689/5021 [00:18<00:06, 198.06it/s]
 67%|██████▋   | 3382/5021 [00:17<00:08, 199.48it/s]
 74%|███████▍  | 3709/5021 [00:18<00:06, 198.38it/s]
 68%|██████▊   | 3403/5021 [00:17<00:08, 199.73it/s]
 74%|███████▍  | 3729/5021 [00:18<00:06, 198.54it/s]
 68%|██████▊   | 3423/5021 [00:17<00:08, 199.74it/s]
 75%|███████▍  | 3749/5021 [00:19<00:06, 198.62it/s]
 69%|██████▊   | 3443/5021 [00:17<00:07, 199.71it/s]
 75%|███████▌  | 3769/5021 [00:19<00:06, 198.47it/s]
 69%|██████▉   | 3464/5021 [00:17<00:07, 199.78it/s]
 75%|███████▌  | 3789/5021 [00:19<00:06, 198.28it/s]
 69%|██████▉   | 3485/5021 [00:17<00:07, 199.93it/s]
 76%|███████▌  | 3809/5021 [00:19<00:06, 198.13it/s]
 70%|██████▉   | 3506/5021 [00:17<00:07, 200.41it/s]
 76%|███████▋  | 3829/5021 [00:19<00:06, 198.21it/s]
 70%|███████   | 3527/5021 [00:17<00:07, 200.68it/s]
 77%|███████▋  | 3849/5021 [00:19<00:05, 198.22it/s]
 71%|███████   | 3548/5021 [00:17<00:07, 200.62it/s]
 77%|███████▋  | 3869/5021 [00:19<00:05, 198.45it/s]
 71%|███████   | 3569/5021 [00:18<00:07, 200.12it/s]
 77%|███████▋  | 3889/5021 [00:19<00:05, 198.71it/s]
 71%|███████▏  | 3590/5021 [00:18<00:07, 200.48it/s]
 78%|███████▊  | 3909/5021 [00:19<00:05, 198.94it/s]
 72%|███████▏  | 3611/5021 [00:18<00:07, 200.87it/s]
 78%|███████▊  | 3929/5021 [00:19<00:05, 198.99it/s]
 72%|███████▏  | 3632/5021 [00:18<00:06, 200.68it/s]
 79%|███████▊  | 3949/5021 [00:20<00:05, 198.60it/s]
 73%|███████▎  | 3653/5021 [00:18<00:06, 200.23it/s]
 79%|███████▉  | 3969/5021 [00:20<00:05, 198.28it/s]
 73%|███████▎  | 3674/5021 [00:18<00:06, 199.67it/s]
 79%|███████▉  | 3989/5021 [00:20<00:05, 197.99it/s]
 74%|███████▎  | 3694/5021 [00:18<00:06, 199.28it/s]
 80%|███████▉  | 4009/5021 [00:20<00:05, 198.07it/s]
 74%|███████▍  | 3714/5021 [00:18<00:06, 198.38it/s]
 80%|████████  | 4029/5021 [00:20<00:05, 197.80it/s]
 74%|███████▍  | 3735/5021 [00:18<00:06, 199.06it/s]
 81%|████████  | 4049/5021 [00:20<00:04, 197.95it/s]
 75%|███████▍  | 3756/5021 [00:19<00:06, 199.42it/s]
 81%|████████  | 4069/5021 [00:20<00:04, 197.80it/s]
 75%|███████▌  | 3776/5021 [00:19<00:06, 199.32it/s]
 81%|████████▏ | 4089/5021 [00:20<00:04, 197.85it/s]
 76%|███████▌  | 3797/5021 [00:19<00:06, 199.61it/s]
 82%|████████▏ | 4109/5021 [00:20<00:04, 197.99it/s]
 76%|███████▌  | 3817/5021 [00:19<00:06, 199.72it/s]
 82%|████████▏ | 4129/5021 [00:20<00:04, 198.12it/s]
 76%|███████▋  | 3837/5021 [00:19<00:05, 199.29it/s]
 83%|████████▎ | 4149/5021 [00:21<00:04, 197.97it/s]
 77%|███████▋  | 3857/5021 [00:19<00:05, 199.38it/s]
 83%|████████▎ | 4169/5021 [00:21<00:04, 198.21it/s]
 77%|███████▋  | 3878/5021 [00:19<00:05, 199.70it/s]
 83%|████████▎ | 4189/5021 [00:21<00:04, 198.46it/s]
 78%|███████▊  | 3898/5021 [00:19<00:05, 199.78it/s]
 84%|████████▍ | 4209/5021 [00:21<00:04, 198.66it/s]
 78%|███████▊  | 3919/5021 [00:19<00:05, 199.95it/s]
 84%|████████▍ | 4229/5021 [00:21<00:03, 198.38it/s]
 78%|███████▊  | 3940/5021 [00:19<00:05, 200.04it/s]
 85%|████████▍ | 4249/5021 [00:21<00:03, 198.37it/s]
 79%|███████▉  | 3961/5021 [00:20<00:05, 200.48it/s]
 85%|████████▌ | 4269/5021 [00:21<00:03, 198.31it/s]
 79%|███████▉  | 3982/5021 [00:20<00:05, 200.56it/s]
 85%|████████▌ | 4289/5021 [00:21<00:03, 198.05it/s]
 80%|███████▉  | 4003/5021 [00:20<00:05, 200.38it/s]
 86%|████████▌ | 4309/5021 [00:21<00:03, 198.31it/s]
 80%|████████  | 4024/5021 [00:20<00:04, 200.53it/s]
 86%|████████▌ | 4329/5021 [00:21<00:03, 198.30it/s]
 81%|████████  | 4045/5021 [00:20<00:04, 195.93it/s]
 87%|████████▋ | 4349/5021 [00:22<00:03, 198.59it/s]
 81%|████████  | 4065/5021 [00:20<00:04, 196.98it/s]
 87%|████████▋ | 4369/5021 [00:22<00:03, 198.40it/s]
 81%|████████▏ | 4085/5021 [00:20<00:04, 197.38it/s]
 87%|████████▋ | 4389/5021 [00:22<00:03, 198.50it/s]
 82%|████████▏ | 4105/5021 [00:20<00:04, 197.79it/s]
 88%|████████▊ | 4409/5021 [00:22<00:03, 197.10it/s]
 82%|████████▏ | 4125/5021 [00:20<00:04, 197.94it/s]
 88%|████████▊ | 4429/5021 [00:22<00:03, 197.09it/s]
 83%|████████▎ | 4146/5021 [00:20<00:04, 198.47it/s]
 89%|████████▊ | 4449/5021 [00:22<00:02, 197.31it/s]
 83%|████████▎ | 4167/5021 [00:21<00:04, 199.31it/s]
 89%|████████▉ | 4469/5021 [00:22<00:02, 197.90it/s]
 83%|████████▎ | 4187/5021 [00:21<00:04, 199.43it/s]
 89%|████████▉ | 4489/5021 [00:22<00:02, 197.81it/s]
 84%|████████▍ | 4207/5021 [00:21<00:04, 199.41it/s]
 90%|████████▉ | 4509/5021 [00:22<00:02, 198.22it/s]
 84%|████████▍ | 4227/5021 [00:21<00:03, 199.19it/s]
 90%|█████████ | 4529/5021 [00:22<00:02, 198.74it/s]
 85%|████████▍ | 4247/5021 [00:21<00:03, 199.18it/s]
 91%|█████████ | 4549/5021 [00:23<00:02, 198.53it/s]
 85%|████████▍ | 4267/5021 [00:21<00:03, 199.36it/s]
 91%|█████████ | 4569/5021 [00:23<00:02, 198.60it/s]
 85%|████████▌ | 4287/5021 [00:21<00:03, 199.50it/s]
 91%|█████████▏| 4589/5021 [00:23<00:02, 198.44it/s]
 92%|█████████▏| 4609/5021 [00:23<00:02, 198.59it/s]
 86%|████████▌ | 4308/5021 [00:21<00:03, 199.81it/s]
 92%|█████████▏| 4629/5021 [00:23<00:01, 198.85it/s]
 86%|████████▌ | 4329/5021 [00:21<00:03, 200.07it/s]
 93%|█████████▎| 4649/5021 [00:23<00:01, 198.66it/s]
 87%|████████▋ | 4350/5021 [00:21<00:03, 200.07it/s]
 93%|█████████▎| 4669/5021 [00:23<00:01, 198.99it/s]
 87%|████████▋ | 4371/5021 [00:22<00:03, 200.40it/s]
 93%|█████████▎| 4690/5021 [00:23<00:01, 199.53it/s]
 87%|████████▋ | 4392/5021 [00:22<00:03, 200.38it/s]
 94%|█████████▍| 4711/5021 [00:23<00:01, 199.89it/s]
 88%|████████▊ | 4413/5021 [00:22<00:03, 200.57it/s]
 94%|█████████▍| 4731/5021 [00:23<00:01, 199.81it/s]
 88%|████████▊ | 4434/5021 [00:22<00:02, 200.72it/s]
 95%|█████████▍| 4751/5021 [00:24<00:01, 199.51it/s]
 89%|████████▊ | 4455/5021 [00:22<00:02, 200.61it/s]
 95%|█████████▌| 4771/5021 [00:24<00:01, 199.34it/s]
 89%|████████▉ | 4476/5021 [00:22<00:02, 200.10it/s]
 95%|█████████▌| 4791/5021 [00:24<00:01, 199.46it/s]
 90%|████████▉ | 4497/5021 [00:22<00:02, 199.67it/s]
 96%|█████████▌| 4811/5021 [00:24<00:01, 199.03it/s]
 90%|████████▉ | 4517/5021 [00:22<00:02, 199.43it/s]
 96%|█████████▌| 4831/5021 [00:24<00:00, 198.46it/s]
 90%|█████████ | 4537/5021 [00:22<00:02, 199.04it/s]
 97%|█████████▋| 4851/5021 [00:24<00:00, 198.19it/s]
 91%|█████████ | 4557/5021 [00:23<00:02, 198.94it/s]
 97%|█████████▋| 4871/5021 [00:24<00:00, 198.12it/s]
 91%|█████████ | 4578/5021 [00:23<00:02, 199.37it/s]
 97%|█████████▋| 4891/5021 [00:24<00:00, 197.90it/s]
 92%|█████████▏| 4598/5021 [00:23<00:02, 199.43it/s]
 98%|█████████▊| 4911/5021 [00:24<00:00, 198.25it/s]
 92%|█████████▏| 4618/5021 [00:23<00:02, 199.53it/s]
 98%|█████████▊| 4931/5021 [00:24<00:00, 198.34it/s]
 92%|█████████▏| 4639/5021 [00:23<00:01, 199.62it/s]
 99%|█████████▊| 4951/5021 [00:25<00:00, 198.52it/s]
 93%|█████████▎| 4659/5021 [00:23<00:01, 199.45it/s]
 99%|█████████▉| 4971/5021 [00:25<00:00, 198.46it/s]
 93%|█████████▎| 4680/5021 [00:23<00:01, 200.23it/s]
 99%|█████████▉| 4991/5021 [00:25<00:00, 198.55it/s]
 94%|█████████▎| 4701/5021 [00:23<00:01, 200.64it/s]
100%|█████████▉| 5011/5021 [00:25<00:00, 198.67it/s]
 94%|█████████▍| 4722/5021 [00:23<00:01, 200.89it/s]
100%|██████████| 5021/5021 [00:25<00:00, 197.47it/s]
n136-128-154:1720460:1720460 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1720460:1720460 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1720460:1720460 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:1720460:1720460 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:1720460:1720460 [0] NCCL INFO Comm config Blocking set to 1

 94%|█████████▍| 4743/5021 [00:23<00:01, 200.72it/s]
 95%|█████████▍| 4764/5021 [00:24<00:01, 200.75it/s]n136-128-154:1720460:1720691 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:1720460:1720691 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:1720460:1720691 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1720460:1720691 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1720460:1720691 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:1720460:1720691 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:1720460:1720691 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:1720460:1720691 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:1720460:1720691 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:1720460:1720691 [0] NCCL INFO Using network IB

 95%|█████████▌| 4785/5021 [00:24<00:01, 200.43it/s]n136-128-154:1720460:1720691 [0] NCCL INFO ncclCommInitRankConfig comm 0x134d07d0 rank 0 nranks 2 cudaDev 0 nvmlDev 1 busId 16000 commId 0x114028e9049878c1 - Init START

 96%|█████████▌| 4806/5021 [00:24<00:01, 200.43it/s]
 96%|█████████▌| 4827/5021 [00:24<00:00, 200.41it/s]
 97%|█████████▋| 4848/5021 [00:24<00:00, 197.24it/s]
 97%|█████████▋| 4868/5021 [00:24<00:00, 196.83it/s]
 97%|█████████▋| 4888/5021 [00:24<00:00, 196.48it/s]
 98%|█████████▊| 4908/5021 [00:24<00:00, 195.98it/s]
 98%|█████████▊| 4928/5021 [00:24<00:00, 196.10it/s]
 99%|█████████▊| 4948/5021 [00:24<00:00, 196.27it/s]
 99%|█████████▉| 4968/5021 [00:25<00:00, 196.61it/s]
 99%|█████████▉| 4988/5021 [00:25<00:00, 197.07it/s]
100%|█████████▉| 5008/5021 [00:25<00:00, 197.31it/s]
100%|██████████| 5021/5021 [00:25<00:00, 198.03it/s]
n136-128-154:1720461:1720461 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:1720461:1720461 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1720461:1720461 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1720461:1720461 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:1720461:1720461 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:1720461:1720697 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:1720461:1720697 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:1720461:1720697 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:1720461:1720697 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:1720461:1720697 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:1720461:1720697 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:1720461:1720697 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:1720461:1720697 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:1720461:1720697 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:1720461:1720697 [1] NCCL INFO Using network IB
n136-128-154:1720461:1720697 [1] NCCL INFO ncclCommInitRankConfig comm 0x15f07470 rank 1 nranks 2 cudaDev 1 nvmlDev 2 busId 4a000 commId 0x114028e9049878c1 - Init START
n136-128-154:1720461:1720697 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:1720460:1720691 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:1720460:1720691 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:1720460:1720691 [0] NCCL INFO Retrieving state for IB
n136-128-154:1720460:1720691 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:1720460:1720691 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:1720461:1720697 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:1720461:1720697 [1] NCCL INFO Retrieving state for IB
n136-128-154:1720461:1720697 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:1720460:1720691 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:1720461:1720697 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:1720461:1720697 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:1720460:1720691 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:1720461:1720697 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:1720460:1720691 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:1720461:1720697 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:1720460:1720691 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1720461:1720697 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1720460:1720691 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1720460:1720691 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1720461:1720697 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 0 (distance 5 <= 5), read 0 mode Default
n136-128-154:1720460:1720691 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1720461:1720697 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1720461:1720697 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 1 (distance 5 <= 5), read 0 mode Default
n136-128-154:1720460:1720691 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:1720460:1720691 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:1720460:1720691 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:1720460:1720691 [0] NCCL INFO               + PCI[24.0] - PCI/0-14000 (1000c01010de13b8)
n136-128-154:1720460:1720691 [0] NCCL INFO                             + PCI[24.0] - GPU/0-16000 (0)
n136-128-154:1720460:1720691 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1720460:1720691 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:1720460:1720691 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:1720460:1720691 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:1720460:1720691 [0] NCCL INFO               + PCI[24.0] - PCI/0-48000 (1000c01010de13b8)
n136-128-154:1720460:1720691 [0] NCCL INFO                             + PCI[24.0] - GPU/0-4a000 (1)
n136-128-154:1720460:1720691 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1720460:1720691 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:1720460:1720691 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:1720460:1720691 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:1720460:1720691 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:1720460:1720691 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:1720460:1720691 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:1720460:1720691 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:1720460:1720691 [0] NCCL INFO ==========================================
n136-128-154:1720460:1720691 [0] NCCL INFO GPU/0-16000 :GPU/0-16000 (0/5000.0/LOC) GPU/0-4a000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1720460:1720691 [0] NCCL INFO GPU/0-4a000 :GPU/0-16000 (2/240.0/NVL) GPU/0-4a000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1720460:1720691 [0] NCCL INFO Setting affinity for GPU 1 to 1-31,65-95
n136-128-154:1720461:1720697 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:1720461:1720697 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:1720461:1720697 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:1720461:1720697 [1] NCCL INFO               + PCI[24.0] - PCI/0-14000 (1000c01010de13b8)
n136-128-154:1720461:1720697 [1] NCCL INFO                             + PCI[24.0] - GPU/0-16000 (0)
n136-128-154:1720461:1720697 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1720461:1720697 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:1720461:1720697 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:1720461:1720697 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:1720461:1720697 [1] NCCL INFO               + PCI[24.0] - PCI/0-48000 (1000c01010de13b8)
n136-128-154:1720461:1720697 [1] NCCL INFO                             + PCI[24.0] - GPU/0-4a000 (1)
n136-128-154:1720461:1720697 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:1720461:1720697 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:1720461:1720697 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:1720461:1720697 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:1720461:1720697 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:1720461:1720697 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:1720461:1720697 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:1720461:1720697 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:1720461:1720697 [1] NCCL INFO ==========================================
n136-128-154:1720461:1720697 [1] NCCL INFO GPU/0-16000 :GPU/0-16000 (0/5000.0/LOC) GPU/0-4a000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1720461:1720697 [1] NCCL INFO GPU/0-4a000 :GPU/0-16000 (2/240.0/NVL) GPU/0-4a000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-0 (3/24.0/PHB) CPU/0-1 (4/10.0/SYS) 
n136-128-154:1720461:1720697 [1] NCCL INFO Setting affinity for GPU 2 to 1-31,65-95
n136-128-154:1720460:1720691 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:1720460:1720691 [0] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1720460:1720691 [0] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1720460:1720691 [0] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1720460:1720691 [0] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1720460:1720691 [0] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1720460:1720691 [0] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1720460:1720691 [0] NCCL INFO  6 : GPU/0-16000 GPU/0-4a000
n136-128-154:1720460:1720691 [0] NCCL INFO  7 : GPU/0-16000 GPU/0-4a000
n136-128-154:1720460:1720691 [0] NCCL INFO  8 : GPU/0-16000 GPU/0-4a000
n136-128-154:1720460:1720691 [0] NCCL INFO  9 : GPU/0-16000 GPU/0-4a000
n136-128-154:1720460:1720691 [0] NCCL INFO 10 : GPU/0-16000 GPU/0-4a000
n136-128-154:1720460:1720691 [0] NCCL INFO 11 : GPU/0-16000 GPU/0-4a000
n136-128-154:1720461:1720697 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:1720461:1720697 [1] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1720461:1720697 [1] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1720461:1720697 [1] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1720461:1720697 [1] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1720461:1720697 [1] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1720461:1720697 [1] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1720461:1720697 [1] NCCL INFO  6 : GPU/0-16000 GPU/0-4a000
n136-128-154:1720461:1720697 [1] NCCL INFO  7 : GPU/0-16000 GPU/0-4a000
n136-128-154:1720461:1720697 [1] NCCL INFO  8 : GPU/0-16000 GPU/0-4a000
n136-128-154:1720461:1720697 [1] NCCL INFO  9 : GPU/0-16000 GPU/0-4a000
n136-128-154:1720461:1720697 [1] NCCL INFO 10 : GPU/0-16000 GPU/0-4a000
n136-128-154:1720461:1720697 [1] NCCL INFO 11 : GPU/0-16000 GPU/0-4a000
n136-128-154:1720460:1720691 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:1720460:1720691 [0] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1720460:1720691 [0] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1720460:1720691 [0] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1720460:1720691 [0] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1720460:1720691 [0] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1720460:1720691 [0] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1720460:1720691 [0] NCCL INFO  6 : GPU/0-4a000 GPU/0-16000
n136-128-154:1720460:1720691 [0] NCCL INFO  7 : GPU/0-4a000 GPU/0-16000
n136-128-154:1720460:1720691 [0] NCCL INFO  8 : GPU/0-4a000 GPU/0-16000
n136-128-154:1720460:1720691 [0] NCCL INFO  9 : GPU/0-4a000 GPU/0-16000
n136-128-154:1720460:1720691 [0] NCCL INFO 10 : GPU/0-4a000 GPU/0-16000
n136-128-154:1720460:1720691 [0] NCCL INFO 11 : GPU/0-4a000 GPU/0-16000
n136-128-154:1720461:1720697 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:1720461:1720697 [1] NCCL INFO  0 : GPU/0-16000 GPU/0-4a000
n136-128-154:1720461:1720697 [1] NCCL INFO  1 : GPU/0-16000 GPU/0-4a000
n136-128-154:1720461:1720697 [1] NCCL INFO  2 : GPU/0-16000 GPU/0-4a000
n136-128-154:1720461:1720697 [1] NCCL INFO  3 : GPU/0-16000 GPU/0-4a000
n136-128-154:1720461:1720697 [1] NCCL INFO  4 : GPU/0-16000 GPU/0-4a000
n136-128-154:1720461:1720697 [1] NCCL INFO  5 : GPU/0-16000 GPU/0-4a000
n136-128-154:1720461:1720697 [1] NCCL INFO  6 : GPU/0-4a000 GPU/0-16000
n136-128-154:1720461:1720697 [1] NCCL INFO  7 : GPU/0-4a000 GPU/0-16000
n136-128-154:1720461:1720697 [1] NCCL INFO  8 : GPU/0-4a000 GPU/0-16000
n136-128-154:1720461:1720697 [1] NCCL INFO  9 : GPU/0-4a000 GPU/0-16000
n136-128-154:1720461:1720697 [1] NCCL INFO 10 : GPU/0-4a000 GPU/0-16000
n136-128-154:1720461:1720697 [1] NCCL INFO 11 : GPU/0-4a000 GPU/0-16000
n136-128-154:1720460:1720691 [0] NCCL INFO comm 0x134d07d0 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:1720461:1720697 [1] NCCL INFO comm 0x15f07470 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:1720460:1720691 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:1720461:1720697 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:1720460:1720691 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:1720461:1720697 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:1720460:1720691 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:1720461:1720697 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:1720460:1720691 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:1720461:1720697 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:1720460:1720691 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:1720461:1720697 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:1720460:1720691 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:1720461:1720697 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:1720460:1720691 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:1720461:1720697 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:1720460:1720691 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:1720461:1720697 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:1720460:1720691 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:1720461:1720697 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:1720460:1720691 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:1720461:1720697 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:1720460:1720691 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:1720461:1720697 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:1720460:1720691 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:1720461:1720697 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:1720460:1720691 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:1720461:1720697 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:1720460:1720691 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:1720461:1720697 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:1720460:1720691 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:1720461:1720697 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:1720460:1720691 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:1720461:1720697 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:1720460:1720691 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:1720461:1720697 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:1720460:1720691 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:1720461:1720697 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:1720460:1720691 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:1720461:1720697 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:1720460:1720691 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:1720461:1720697 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:1720460:1720691 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:1720461:1720697 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:1720460:1720691 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:1720461:1720697 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:1720460:1720691 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:1720461:1720697 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:1720460:1720691 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:1720461:1720697 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:1720460:1720691 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:1720460:1720691 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:1720461:1720697 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:1720460:1720691 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:1720461:1720697 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:1720460:1720691 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:1720461:1720697 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:1720460:1720691 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:1720461:1720697 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:1720460:1720691 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:1720461:1720697 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:1720460:1720691 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:1720461:1720697 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:1720460:1720691 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:1720461:1720697 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:1720460:1720691 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:1720461:1720697 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:1720460:1720691 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:1720461:1720697 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:1720460:1720691 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:1720461:1720697 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:1720460:1720691 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:1720461:1720697 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:1720460:1720691 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:1720461:1720697 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:1720460:1720691 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:1720461:1720697 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:1720460:1720691 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:1720461:1720697 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:1720460:1720691 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:1720461:1720697 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:1720460:1720691 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:1720461:1720697 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:1720460:1720691 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:1720461:1720697 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:1720460:1720691 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:1720461:1720697 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:1720460:1720691 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:1720461:1720697 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:1720460:1720691 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:1720461:1720697 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:1720460:1720691 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:1720461:1720697 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:1720460:1720691 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:1720461:1720697 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:1720460:1720691 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:1720461:1720697 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:1720460:1720691 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:1720461:1720697 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:1720460:1720691 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:1720461:1720697 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:1720460:1720691 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:1720460:1720691 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:1720461:1720697 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:1720460:1720691 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:1720460:1720691 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:1720460:1720691 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:1720460:1720691 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:1720460:1720691 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:1720460:1720691 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:1720460:1720691 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:1720460:1720691 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:1720460:1720691 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:1720460:1720691 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:1720460:1720691 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:1720460:1720691 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:1720460:1720691 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:1720460:1720691 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:1720460:1720691 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:1720460:1720691 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:1720460:1720691 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:1720460:1720691 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:1720460:1720691 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:1720460:1720691 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:1720460:1720691 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:1720460:1720691 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:1720461:1720697 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:1720461:1720705 [1] NCCL INFO [Proxy Service] Device 1 CPU core 29
n136-128-154:1720461:1720706 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 92
n136-128-154:1720460:1720691 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:1720460:1720691 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:1720460:1720707 [0] NCCL INFO [Proxy Service] Device 0 CPU core 70
n136-128-154:1720460:1720708 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 7
n136-128-154:1720461:1720697 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:1720461:1720697 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:1720460:1720691 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:1720460:1720691 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:1720460:1720691 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:1720461:1720697 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:1720461:1720697 [1] NCCL INFO ncclCommInitRankConfig comm 0x15f07470 rank 1 nranks 2 cudaDev 1 nvmlDev 2 busId 4a000 commId 0x114028e9049878c1 - Init COMPLETE
n136-128-154:1720461:1720697 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.63 (kernels 0.12, alloc 0.25, bootstrap 0.00, allgathers 0.01, topo 0.08, graphs 0.00, connections 0.01, rest 0.16)
n136-128-154:1720460:1720691 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:1720460:1720691 [0] NCCL INFO ncclCommInitRankConfig comm 0x134d07d0 rank 0 nranks 2 cudaDev 0 nvmlDev 1 busId 16000 commId 0x114028e9049878c1 - Init COMPLETE
n136-128-154:1720460:1720691 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 2.14 (kernels 0.17, alloc 0.14, bootstrap 1.57, allgathers 0.01, topo 0.08, graphs 0.00, connections 0.01, rest 0.17)
n136-128-154:1720461:1720709 [1] NCCL INFO Channel 00/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1720709 [1] NCCL INFO Channel 01/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1720709 [1] NCCL INFO Channel 02/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1720709 [1] NCCL INFO Channel 03/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1720709 [1] NCCL INFO Channel 04/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1720709 [1] NCCL INFO Channel 05/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1720709 [1] NCCL INFO Channel 06/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720460:1720710 [0] NCCL INFO Channel 00/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1720461:1720709 [1] NCCL INFO Channel 07/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720460:1720710 [0] NCCL INFO Channel 01/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1720461:1720709 [1] NCCL INFO Channel 08/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720460:1720710 [0] NCCL INFO Channel 02/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1720461:1720709 [1] NCCL INFO Channel 09/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720460:1720710 [0] NCCL INFO Channel 03/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1720461:1720709 [1] NCCL INFO Channel 10/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720460:1720710 [0] NCCL INFO Channel 04/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1720461:1720709 [1] NCCL INFO Channel 11/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720460:1720710 [0] NCCL INFO Channel 05/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1720461:1720709 [1] NCCL INFO Channel 12/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720460:1720710 [0] NCCL INFO Channel 06/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1720461:1720709 [1] NCCL INFO Channel 13/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720460:1720710 [0] NCCL INFO Channel 07/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1720461:1720709 [1] NCCL INFO Channel 14/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720460:1720710 [0] NCCL INFO Channel 08/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1720461:1720709 [1] NCCL INFO Channel 15/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720460:1720710 [0] NCCL INFO Channel 09/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1720461:1720709 [1] NCCL INFO Channel 16/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720460:1720710 [0] NCCL INFO Channel 10/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1720461:1720709 [1] NCCL INFO Channel 17/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720460:1720710 [0] NCCL INFO Channel 11/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1720461:1720709 [1] NCCL INFO Channel 18/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720460:1720710 [0] NCCL INFO Channel 12/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1720461:1720709 [1] NCCL INFO Channel 19/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720460:1720710 [0] NCCL INFO Channel 13/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1720461:1720709 [1] NCCL INFO Channel 20/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720460:1720710 [0] NCCL INFO Channel 14/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1720461:1720709 [1] NCCL INFO Channel 21/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720460:1720710 [0] NCCL INFO Channel 15/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1720461:1720709 [1] NCCL INFO Channel 22/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720460:1720710 [0] NCCL INFO Channel 16/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1720461:1720709 [1] NCCL INFO Channel 23/0 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720460:1720710 [0] NCCL INFO Channel 17/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1720460:1720710 [0] NCCL INFO Channel 18/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1720460:1720710 [0] NCCL INFO Channel 19/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1720460:1720710 [0] NCCL INFO Channel 20/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1720460:1720710 [0] NCCL INFO Channel 21/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1720460:1720710 [0] NCCL INFO Channel 22/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1720460:1720710 [0] NCCL INFO Channel 23/0 : 0[1] -> 1[2] via P2P/CUMEM/read
n136-128-154:1720461:1720709 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:1720460:1720710 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-06:05:16:02 INFO     [evaluator:559] Running loglikelihood requests
2025-12-06:05:16:02 INFO     [evaluator:559] Running loglikelihood requests

Running loglikelihood requests:   0%|          | 0/20084 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 64
Determined largest batch size: 64

Running loglikelihood requests:   0%|          | 1/20084 [00:35<200:43:55, 35.98s/it]
Running loglikelihood requests:   0%|          | 65/20084 [00:42<2:42:29,  2.05it/s] 
Running loglikelihood requests:   1%|          | 129/20084 [00:48<1:25:02,  3.91it/s]
Running loglikelihood requests:   1%|          | 193/20084 [00:53<59:14,  5.60it/s]  
Running loglikelihood requests:   1%|▏         | 257/20084 [00:59<46:59,  7.03it/s]
Running loglikelihood requests:   2%|▏         | 321/20084 [01:04<40:07,  8.21it/s]
Running loglikelihood requests:   2%|▏         | 385/20084 [01:10<35:53,  9.15it/s]
Running loglikelihood requests:   2%|▏         | 449/20084 [01:15<33:08,  9.87it/s]
Running loglikelihood requests:   3%|▎         | 513/20084 [01:21<31:18, 10.42it/s]
Running loglikelihood requests:   3%|▎         | 577/20084 [01:26<29:57, 10.86it/s]
Running loglikelihood requests:   3%|▎         | 641/20084 [01:31<28:58, 11.19it/s]
Running loglikelihood requests:   4%|▎         | 705/20084 [01:37<28:14, 11.44it/s]
Running loglikelihood requests:   4%|▍         | 769/20084 [01:42<27:41, 11.63it/s]
Running loglikelihood requests:   4%|▍         | 833/20084 [01:47<27:15, 11.77it/s]
Running loglikelihood requests:   4%|▍         | 897/20084 [01:52<26:54, 11.88it/s]
Running loglikelihood requests:   5%|▍         | 961/20084 [01:58<26:36, 11.98it/s]
Running loglikelihood requests:   5%|▌         | 1025/20084 [02:03<26:21, 12.05it/s]
Running loglikelihood requests:   5%|▌         | 1089/20084 [02:08<26:07, 12.11it/s]
Running loglikelihood requests:   6%|▌         | 1153/20084 [02:13<25:55, 12.17it/s]
Running loglikelihood requests:   6%|▌         | 1217/20084 [02:18<25:45, 12.21it/s]
Running loglikelihood requests:   6%|▋         | 1281/20084 [02:24<25:35, 12.24it/s]
Running loglikelihood requests:   7%|▋         | 1345/20084 [02:29<25:26, 12.27it/s]
Running loglikelihood requests:   7%|▋         | 1409/20084 [02:34<25:17, 12.31it/s]
Running loglikelihood requests:   7%|▋         | 1473/20084 [02:39<25:09, 12.33it/s]
Running loglikelihood requests:   8%|▊         | 1537/20084 [02:44<25:01, 12.35it/s]
Running loglikelihood requests:   8%|▊         | 1601/20084 [02:49<24:54, 12.37it/s]
Running loglikelihood requests:   8%|▊         | 1665/20084 [02:55<24:45, 12.40it/s]
Running loglikelihood requests:   9%|▊         | 1729/20084 [03:00<24:37, 12.43it/s]
Running loglikelihood requests:   9%|▉         | 1793/20084 [03:05<24:27, 12.46it/s]
Running loglikelihood requests:   9%|▉         | 1857/20084 [03:10<24:18, 12.50it/s]
Running loglikelihood requests:  10%|▉         | 1921/20084 [03:15<24:10, 12.52it/s]
Running loglikelihood requests:  10%|▉         | 1985/20084 [03:20<24:03, 12.54it/s]
Running loglikelihood requests:  10%|█         | 2049/20084 [03:25<23:55, 12.56it/s]
Running loglikelihood requests:  11%|█         | 2113/20084 [03:30<23:48, 12.58it/s]
Running loglikelihood requests:  11%|█         | 2177/20084 [03:35<23:40, 12.61it/s]
Running loglikelihood requests:  11%|█         | 2241/20084 [03:40<23:32, 12.63it/s]
Running loglikelihood requests:  11%|█▏        | 2305/20084 [03:45<23:26, 12.64it/s]
Running loglikelihood requests:  12%|█▏        | 2369/20084 [03:50<23:19, 12.65it/s]
Running loglikelihood requests:  12%|█▏        | 2433/20084 [03:55<23:12, 12.67it/s]
Running loglikelihood requests:  12%|█▏        | 2497/20084 [04:01<23:06, 12.68it/s]
Running loglikelihood requests:  13%|█▎        | 2561/20084 [04:06<22:59, 12.70it/s]
Running loglikelihood requests:  13%|█▎        | 2625/20084 [04:11<22:52, 12.72it/s]
Running loglikelihood requests:  13%|█▎        | 2689/20084 [04:16<22:45, 12.74it/s]
Running loglikelihood requests:  14%|█▎        | 2753/20084 [04:21<22:39, 12.75it/s]
Running loglikelihood requests:  14%|█▍        | 2817/20084 [04:26<22:33, 12.76it/s]
Running loglikelihood requests:  14%|█▍        | 2881/20084 [04:31<22:26, 12.77it/s]
Running loglikelihood requests:  15%|█▍        | 2945/20084 [04:36<22:19, 12.79it/s]
Running loglikelihood requests:  15%|█▍        | 3009/20084 [04:41<22:13, 12.80it/s]
Running loglikelihood requests:  15%|█▌        | 3073/20084 [04:46<22:06, 12.82it/s]
Running loglikelihood requests:  16%|█▌        | 3137/20084 [04:50<22:00, 12.84it/s]
Running loglikelihood requests:  16%|█▌        | 3201/20084 [04:55<21:54, 12.85it/s]
Running loglikelihood requests:  16%|█▋        | 3265/20084 [05:00<21:47, 12.86it/s]
Running loglikelihood requests:  17%|█▋        | 3329/20084 [05:05<21:40, 12.88it/s]
Running loglikelihood requests:  17%|█▋        | 3393/20084 [05:10<21:34, 12.90it/s]
Running loglikelihood requests:  17%|█▋        | 3457/20084 [05:15<21:28, 12.91it/s]
Running loglikelihood requests:  18%|█▊        | 3521/20084 [05:20<21:22, 12.91it/s]
Running loglikelihood requests:  18%|█▊        | 3585/20084 [05:25<21:18, 12.91it/s]
Running loglikelihood requests:  18%|█▊        | 3649/20084 [05:30<21:11, 12.93it/s]
Running loglikelihood requests:  18%|█▊        | 3713/20084 [05:35<21:05, 12.94it/s]
Running loglikelihood requests:  19%|█▉        | 3777/20084 [05:40<20:59, 12.95it/s]
Running loglikelihood requests:  19%|█▉        | 3841/20084 [05:45<20:53, 12.96it/s]
Running loglikelihood requests:  19%|█▉        | 3905/20084 [05:50<20:46, 12.98it/s]
Running loglikelihood requests:  20%|█▉        | 3969/20084 [05:55<20:39, 13.00it/s]
Running loglikelihood requests:  20%|██        | 4033/20084 [06:00<20:33, 13.01it/s]
Running loglikelihood requests:  20%|██        | 4097/20084 [06:05<20:26, 13.03it/s]
Running loglikelihood requests:  21%|██        | 4161/20084 [06:09<20:20, 13.04it/s]
Running loglikelihood requests:  21%|██        | 4225/20084 [06:14<20:14, 13.06it/s]
Running loglikelihood requests:  21%|██▏       | 4289/20084 [06:19<20:07, 13.08it/s]
Running loglikelihood requests:  22%|██▏       | 4353/20084 [06:24<20:00, 13.10it/s]
Running loglikelihood requests:  22%|██▏       | 4417/20084 [06:29<19:53, 13.12it/s]
Running loglikelihood requests:  22%|██▏       | 4481/20084 [06:34<19:47, 13.14it/s]
Running loglikelihood requests:  23%|██▎       | 4545/20084 [06:39<19:40, 13.16it/s]
Running loglikelihood requests:  23%|██▎       | 4609/20084 [06:43<19:35, 13.17it/s]
Running loglikelihood requests:  23%|██▎       | 4673/20084 [06:48<19:28, 13.19it/s]
Running loglikelihood requests:  24%|██▎       | 4737/20084 [06:53<19:22, 13.20it/s]
Running loglikelihood requests:  24%|██▍       | 4801/20084 [06:58<19:16, 13.21it/s]
Running loglikelihood requests:  24%|██▍       | 4865/20084 [07:03<19:10, 13.23it/s]
Running loglikelihood requests:  25%|██▍       | 4929/20084 [07:08<19:04, 13.24it/s]
Running loglikelihood requests:  25%|██▍       | 4993/20084 [07:12<18:58, 13.26it/s]
Running loglikelihood requests:  25%|██▌       | 5057/20084 [07:17<18:52, 13.26it/s]
Running loglikelihood requests:  25%|██▌       | 5121/20084 [07:22<18:47, 13.27it/s]
Running loglikelihood requests:  26%|██▌       | 5185/20084 [07:27<18:41, 13.28it/s]
Running loglikelihood requests:  26%|██▌       | 5249/20084 [07:32<18:35, 13.29it/s]
Running loglikelihood requests:  26%|██▋       | 5313/20084 [07:37<18:30, 13.30it/s]
Running loglikelihood requests:  27%|██▋       | 5377/20084 [07:41<18:26, 13.29it/s]
Running loglikelihood requests:  27%|██▋       | 5441/20084 [07:46<18:20, 13.31it/s]
Running loglikelihood requests:  27%|██▋       | 5505/20084 [07:51<18:13, 13.33it/s]
Running loglikelihood requests:  28%|██▊       | 5569/20084 [07:56<18:07, 13.34it/s]
Running loglikelihood requests:  28%|██▊       | 5633/20084 [08:00<18:01, 13.36it/s]
Running loglikelihood requests:  28%|██▊       | 5697/20084 [08:05<17:53, 13.41it/s]
Running loglikelihood requests:  29%|██▊       | 5761/20084 [08:10<17:45, 13.44it/s]
Running loglikelihood requests:  29%|██▉       | 5825/20084 [08:15<17:39, 13.46it/s]
Running loglikelihood requests:  29%|██▉       | 5889/20084 [08:19<17:32, 13.48it/s]
Running loglikelihood requests:  30%|██▉       | 5953/20084 [08:24<17:28, 13.48it/s]
Running loglikelihood requests:  30%|██▉       | 6017/20084 [08:29<17:22, 13.50it/s]
Running loglikelihood requests:  30%|███       | 6081/20084 [08:34<17:17, 13.50it/s]
Running loglikelihood requests:  31%|███       | 6145/20084 [08:38<17:10, 13.52it/s]
Running loglikelihood requests:  31%|███       | 6209/20084 [08:43<17:06, 13.52it/s]
Running loglikelihood requests:  31%|███       | 6273/20084 [08:48<16:59, 13.54it/s]
Running loglikelihood requests:  32%|███▏      | 6337/20084 [08:52<16:53, 13.56it/s]
Running loglikelihood requests:  32%|███▏      | 6401/20084 [08:57<16:47, 13.58it/s]
Running loglikelihood requests:  32%|███▏      | 6465/20084 [09:02<16:41, 13.59it/s]
Running loglikelihood requests:  33%|███▎      | 6529/20084 [09:07<16:36, 13.61it/s]
Running loglikelihood requests:  33%|███▎      | 6593/20084 [09:11<16:30, 13.62it/s]
Running loglikelihood requests:  33%|███▎      | 6657/20084 [09:16<16:24, 13.64it/s]
Running loglikelihood requests:  33%|███▎      | 6721/20084 [09:21<16:18, 13.66it/s]
Running loglikelihood requests:  34%|███▍      | 6785/20084 [09:25<16:13, 13.67it/s]
Running loglikelihood requests:  34%|███▍      | 6849/20084 [09:30<16:07, 13.67it/s]
Running loglikelihood requests:  34%|███▍      | 6913/20084 [09:35<16:02, 13.69it/s]
Running loglikelihood requests:  35%|███▍      | 6977/20084 [09:39<15:56, 13.70it/s]
Running loglikelihood requests:  35%|███▌      | 7041/20084 [09:44<15:51, 13.71it/s]
Running loglikelihood requests:  35%|███▌      | 7105/20084 [09:49<15:46, 13.71it/s]
Running loglikelihood requests:  36%|███▌      | 7169/20084 [09:53<15:41, 13.72it/s]
Running loglikelihood requests:  36%|███▌      | 7233/20084 [09:58<15:35, 13.74it/s]
Running loglikelihood requests:  36%|███▋      | 7297/20084 [10:03<15:35, 13.67it/s]
Running loglikelihood requests:  37%|███▋      | 7361/20084 [10:07<15:34, 13.61it/s]
Running loglikelihood requests:  37%|███▋      | 7425/20084 [10:12<15:26, 13.67it/s]
Running loglikelihood requests:  37%|███▋      | 7489/20084 [10:17<15:18, 13.71it/s]
Running loglikelihood requests:  38%|███▊      | 7553/20084 [10:21<15:16, 13.67it/s]
Running loglikelihood requests:  38%|███▊      | 7617/20084 [10:26<15:16, 13.60it/s]
Running loglikelihood requests:  38%|███▊      | 7681/20084 [10:31<15:06, 13.68it/s]
Running loglikelihood requests:  39%|███▊      | 7745/20084 [10:35<14:57, 13.74it/s]
Running loglikelihood requests:  39%|███▉      | 7809/20084 [10:40<14:50, 13.79it/s]
Running loglikelihood requests:  39%|███▉      | 7873/20084 [10:45<14:43, 13.82it/s]
Running loglikelihood requests:  40%|███▉      | 7937/20084 [10:49<14:36, 13.85it/s]
Running loglikelihood requests:  40%|███▉      | 8001/20084 [10:54<14:30, 13.88it/s]
Running loglikelihood requests:  40%|████      | 8065/20084 [10:58<14:25, 13.89it/s]
Running loglikelihood requests:  40%|████      | 8129/20084 [11:03<14:19, 13.90it/s]
Running loglikelihood requests:  41%|████      | 8193/20084 [11:08<14:13, 13.92it/s]
Running loglikelihood requests:  41%|████      | 8257/20084 [11:12<14:07, 13.95it/s]
Running loglikelihood requests:  41%|████▏     | 8321/20084 [11:17<14:02, 13.96it/s]
Running loglikelihood requests:  42%|████▏     | 8385/20084 [11:21<13:57, 13.96it/s]
Running loglikelihood requests:  42%|████▏     | 8449/20084 [11:26<13:52, 13.98it/s]
Running loglikelihood requests:  42%|████▏     | 8513/20084 [11:30<13:46, 13.99it/s]
Running loglikelihood requests:  43%|████▎     | 8577/20084 [11:35<13:41, 14.01it/s]
Running loglikelihood requests:  43%|████▎     | 8641/20084 [11:40<13:36, 14.02it/s]
Running loglikelihood requests:  43%|████▎     | 8705/20084 [11:44<13:31, 14.03it/s]
Running loglikelihood requests:  44%|████▎     | 8769/20084 [11:49<13:26, 14.03it/s]
Running loglikelihood requests:  44%|████▍     | 8833/20084 [11:53<13:21, 14.04it/s]
Running loglikelihood requests:  44%|████▍     | 8897/20084 [11:58<13:15, 14.06it/s]
Running loglikelihood requests:  45%|████▍     | 8961/20084 [12:02<13:09, 14.08it/s]
Running loglikelihood requests:  45%|████▍     | 9025/20084 [12:07<13:04, 14.09it/s]
Running loglikelihood requests:  45%|████▌     | 9089/20084 [12:11<13:00, 14.09it/s]
Running loglikelihood requests:  46%|████▌     | 9153/20084 [12:16<12:55, 14.10it/s]
Running loglikelihood requests:  46%|████▌     | 9217/20084 [12:20<12:50, 14.11it/s]
Running loglikelihood requests:  46%|████▌     | 9281/20084 [12:25<12:44, 14.13it/s]
Running loglikelihood requests:  47%|████▋     | 9345/20084 [12:29<12:39, 14.14it/s]
Running loglikelihood requests:  47%|████▋     | 9409/20084 [12:34<12:34, 14.14it/s]
Running loglikelihood requests:  47%|████▋     | 9473/20084 [12:38<12:29, 14.16it/s]
Running loglikelihood requests:  47%|████▋     | 9537/20084 [12:43<12:24, 14.17it/s]
Running loglikelihood requests:  48%|████▊     | 9601/20084 [12:47<12:18, 14.19it/s]
Running loglikelihood requests:  48%|████▊     | 9665/20084 [12:52<12:13, 14.21it/s]
Running loglikelihood requests:  48%|████▊     | 9729/20084 [12:56<12:08, 14.22it/s]
Running loglikelihood requests:  49%|████▉     | 9793/20084 [13:01<12:03, 14.23it/s]
Running loglikelihood requests:  49%|████▉     | 9857/20084 [13:05<11:58, 14.24it/s]
Running loglikelihood requests:  49%|████▉     | 9921/20084 [13:10<11:52, 14.26it/s]
Running loglikelihood requests:  50%|████▉     | 9985/20084 [13:14<11:47, 14.27it/s]
Running loglikelihood requests:  50%|█████     | 10049/20084 [13:19<11:42, 14.28it/s]
Running loglikelihood requests:  50%|█████     | 10113/20084 [13:23<11:38, 14.28it/s]
Running loglikelihood requests:  51%|█████     | 10177/20084 [13:28<11:33, 14.28it/s]
Running loglikelihood requests:  51%|█████     | 10241/20084 [13:32<11:28, 14.29it/s]
Running loglikelihood requests:  51%|█████▏    | 10305/20084 [13:37<11:23, 14.30it/s]
Running loglikelihood requests:  52%|█████▏    | 10369/20084 [13:41<11:18, 14.32it/s]
Running loglikelihood requests:  52%|█████▏    | 10433/20084 [13:46<11:13, 14.33it/s]
Running loglikelihood requests:  52%|█████▏    | 10497/20084 [13:50<11:08, 14.35it/s]
Running loglikelihood requests:  53%|█████▎    | 10561/20084 [13:55<11:03, 14.35it/s]
Running loglikelihood requests:  53%|█████▎    | 10625/20084 [13:59<10:57, 14.38it/s]
Running loglikelihood requests:  53%|█████▎    | 10689/20084 [14:03<10:52, 14.40it/s]
Running loglikelihood requests:  54%|█████▎    | 10753/20084 [14:08<10:47, 14.41it/s]
Running loglikelihood requests:  54%|█████▍    | 10817/20084 [14:12<10:42, 14.43it/s]
Running loglikelihood requests:  54%|█████▍    | 10881/20084 [14:17<10:36, 14.45it/s]
Running loglikelihood requests:  54%|█████▍    | 10945/20084 [14:21<10:31, 14.46it/s]
Running loglikelihood requests:  55%|█████▍    | 11009/20084 [14:26<10:27, 14.47it/s]
Running loglikelihood requests:  55%|█████▌    | 11073/20084 [14:30<10:21, 14.49it/s]
Running loglikelihood requests:  55%|█████▌    | 11137/20084 [14:34<10:17, 14.49it/s]
Running loglikelihood requests:  56%|█████▌    | 11201/20084 [14:39<10:12, 14.51it/s]
Running loglikelihood requests:  56%|█████▌    | 11265/20084 [14:43<10:07, 14.52it/s]
Running loglikelihood requests:  56%|█████▋    | 11329/20084 [14:48<10:02, 14.52it/s]
Running loglikelihood requests:  57%|█████▋    | 11393/20084 [14:52<09:58, 14.53it/s]
Running loglikelihood requests:  57%|█████▋    | 11457/20084 [14:56<09:53, 14.54it/s]
Running loglikelihood requests:  57%|█████▋    | 11521/20084 [15:01<09:48, 14.54it/s]
Running loglikelihood requests:  58%|█████▊    | 11585/20084 [15:05<09:44, 14.55it/s]
Running loglikelihood requests:  58%|█████▊    | 11649/20084 [15:10<09:39, 14.57it/s]
Running loglikelihood requests:  58%|█████▊    | 11713/20084 [15:14<09:34, 14.58it/s]
Running loglikelihood requests:  59%|█████▊    | 11777/20084 [15:18<09:29, 14.59it/s]
Running loglikelihood requests:  59%|█████▉    | 11841/20084 [15:23<09:24, 14.61it/s]
Running loglikelihood requests:  59%|█████▉    | 11905/20084 [15:27<09:19, 14.62it/s]
Running loglikelihood requests:  60%|█████▉    | 11969/20084 [15:31<09:14, 14.63it/s]
Running loglikelihood requests:  60%|█████▉    | 12033/20084 [15:36<09:09, 14.66it/s]
Running loglikelihood requests:  60%|██████    | 12097/20084 [15:40<09:04, 14.66it/s]
Running loglikelihood requests:  61%|██████    | 12161/20084 [15:44<08:59, 14.68it/s]
Running loglikelihood requests:  61%|██████    | 12225/20084 [15:49<08:54, 14.69it/s]
Running loglikelihood requests:  61%|██████    | 12289/20084 [15:53<08:50, 14.70it/s]
Running loglikelihood requests:  62%|██████▏   | 12353/20084 [15:57<08:45, 14.71it/s]
Running loglikelihood requests:  62%|██████▏   | 12417/20084 [16:02<08:40, 14.73it/s]
Running loglikelihood requests:  62%|██████▏   | 12481/20084 [16:06<08:35, 14.74it/s]
Running loglikelihood requests:  62%|██████▏   | 12545/20084 [16:10<08:30, 14.76it/s]
Running loglikelihood requests:  63%|██████▎   | 12609/20084 [16:15<08:25, 14.79it/s]
Running loglikelihood requests:  63%|██████▎   | 12673/20084 [16:19<08:20, 14.80it/s]
Running loglikelihood requests:  63%|██████▎   | 12737/20084 [16:23<08:15, 14.81it/s]
Running loglikelihood requests:  64%|██████▎   | 12801/20084 [16:28<08:11, 14.82it/s]
Running loglikelihood requests:  64%|██████▍   | 12865/20084 [16:32<08:06, 14.85it/s]
Running loglikelihood requests:  64%|██████▍   | 12929/20084 [16:36<08:01, 14.86it/s]
Running loglikelihood requests:  65%|██████▍   | 12993/20084 [16:41<07:56, 14.88it/s]
Running loglikelihood requests:  65%|██████▌   | 13057/20084 [16:45<07:51, 14.90it/s]
Running loglikelihood requests:  65%|██████▌   | 13121/20084 [16:49<07:46, 14.91it/s]
Running loglikelihood requests:  66%|██████▌   | 13185/20084 [16:53<07:41, 14.94it/s]
Running loglikelihood requests:  66%|██████▌   | 13249/20084 [16:58<07:37, 14.95it/s]
Running loglikelihood requests:  66%|██████▋   | 13313/20084 [17:02<07:32, 14.97it/s]
Running loglikelihood requests:  67%|██████▋   | 13377/20084 [17:06<07:27, 14.98it/s]
Running loglikelihood requests:  67%|██████▋   | 13441/20084 [17:10<07:22, 15.00it/s]
Running loglikelihood requests:  67%|██████▋   | 13505/20084 [17:15<07:18, 15.01it/s]
Running loglikelihood requests:  68%|██████▊   | 13569/20084 [17:19<07:13, 15.02it/s]
Running loglikelihood requests:  68%|██████▊   | 13633/20084 [17:23<07:08, 15.04it/s]
Running loglikelihood requests:  68%|██████▊   | 13697/20084 [17:27<07:04, 15.06it/s]
Running loglikelihood requests:  69%|██████▊   | 13761/20084 [17:32<06:59, 15.08it/s]
Running loglikelihood requests:  69%|██████▉   | 13825/20084 [17:36<06:54, 15.09it/s]
Running loglikelihood requests:  69%|██████▉   | 13889/20084 [17:40<06:50, 15.10it/s]
Running loglikelihood requests:  69%|██████▉   | 13953/20084 [17:44<06:45, 15.12it/s]
Running loglikelihood requests:  70%|██████▉   | 14017/20084 [17:49<06:41, 15.13it/s]
Running loglikelihood requests:  70%|███████   | 14081/20084 [17:53<06:36, 15.14it/s]
Running loglikelihood requests:  70%|███████   | 14145/20084 [17:57<06:31, 15.16it/s]
Running loglikelihood requests:  71%|███████   | 14209/20084 [18:01<06:26, 15.18it/s]
Running loglikelihood requests:  71%|███████   | 14273/20084 [18:05<06:22, 15.21it/s]
Running loglikelihood requests:  71%|███████▏  | 14337/20084 [18:10<06:17, 15.23it/s]
Running loglikelihood requests:  72%|███████▏  | 14401/20084 [18:14<06:12, 15.24it/s]
Running loglikelihood requests:  72%|███████▏  | 14465/20084 [18:18<06:08, 15.26it/s]
Running loglikelihood requests:  72%|███████▏  | 14529/20084 [18:22<06:03, 15.27it/s]
Running loglikelihood requests:  73%|███████▎  | 14593/20084 [18:26<05:58, 15.30it/s]
Running loglikelihood requests:  73%|███████▎  | 14657/20084 [18:31<05:54, 15.31it/s]
Running loglikelihood requests:  73%|███████▎  | 14721/20084 [18:35<05:50, 15.32it/s]
Running loglikelihood requests:  74%|███████▎  | 14785/20084 [18:39<05:45, 15.34it/s]
Running loglikelihood requests:  74%|███████▍  | 14849/20084 [18:43<05:41, 15.35it/s]
Running loglikelihood requests:  74%|███████▍  | 14913/20084 [18:47<05:36, 15.37it/s]
Running loglikelihood requests:  75%|███████▍  | 14977/20084 [18:51<05:32, 15.38it/s]
Running loglikelihood requests:  75%|███████▍  | 15041/20084 [18:55<05:27, 15.39it/s]
Running loglikelihood requests:  75%|███████▌  | 15105/20084 [19:00<05:23, 15.41it/s]
Running loglikelihood requests:  76%|███████▌  | 15169/20084 [19:04<05:18, 15.43it/s]
Running loglikelihood requests:  76%|███████▌  | 15233/20084 [19:08<05:14, 15.44it/s]
Running loglikelihood requests:  76%|███████▌  | 15297/20084 [19:12<05:09, 15.46it/s]
Running loglikelihood requests:  76%|███████▋  | 15361/20084 [19:16<05:05, 15.48it/s]
Running loglikelihood requests:  77%|███████▋  | 15425/20084 [19:20<05:00, 15.49it/s]
Running loglikelihood requests:  77%|███████▋  | 15489/20084 [19:24<04:56, 15.51it/s]
Running loglikelihood requests:  77%|███████▋  | 15553/20084 [19:28<04:51, 15.52it/s]
Running loglikelihood requests:  78%|███████▊  | 15617/20084 [19:33<04:46, 15.58it/s]
Running loglikelihood requests:  78%|███████▊  | 15681/20084 [19:37<04:42, 15.61it/s]
Running loglikelihood requests:  78%|███████▊  | 15745/20084 [19:41<04:37, 15.64it/s]
Running loglikelihood requests:  79%|███████▊  | 15809/20084 [19:45<04:32, 15.67it/s]
Running loglikelihood requests:  79%|███████▉  | 15873/20084 [19:49<04:28, 15.69it/s]
Running loglikelihood requests:  79%|███████▉  | 15937/20084 [19:53<04:23, 15.71it/s]
Running loglikelihood requests:  80%|███████▉  | 16001/20084 [19:57<04:19, 15.73it/s]
Running loglikelihood requests:  80%|███████▉  | 16065/20084 [20:01<04:15, 15.76it/s]
Running loglikelihood requests:  80%|████████  | 16129/20084 [20:05<04:10, 15.78it/s]
Running loglikelihood requests:  81%|████████  | 16193/20084 [20:09<04:06, 15.80it/s]
Running loglikelihood requests:  81%|████████  | 16257/20084 [20:13<04:01, 15.83it/s]
Running loglikelihood requests:  81%|████████▏ | 16321/20084 [20:17<03:57, 15.84it/s]
Running loglikelihood requests:  82%|████████▏ | 16385/20084 [20:21<03:53, 15.87it/s]
Running loglikelihood requests:  82%|████████▏ | 16449/20084 [20:25<03:48, 15.90it/s]
Running loglikelihood requests:  82%|████████▏ | 16513/20084 [20:29<03:44, 15.92it/s]
Running loglikelihood requests:  83%|████████▎ | 16577/20084 [20:33<03:39, 15.95it/s]
Running loglikelihood requests:  83%|████████▎ | 16641/20084 [20:37<03:35, 15.98it/s]
Running loglikelihood requests:  83%|████████▎ | 16705/20084 [20:41<03:31, 15.99it/s]
Running loglikelihood requests:  83%|████████▎ | 16769/20084 [20:45<03:26, 16.02it/s]
Running loglikelihood requests:  84%|████████▍ | 16833/20084 [20:49<03:22, 16.05it/s]
Running loglikelihood requests:  84%|████████▍ | 16897/20084 [20:53<03:18, 16.07it/s]
Running loglikelihood requests:  84%|████████▍ | 16961/20084 [20:57<03:14, 16.09it/s]
Running loglikelihood requests:  85%|████████▍ | 17025/20084 [21:01<03:09, 16.12it/s]
Running loglikelihood requests:  85%|████████▌ | 17089/20084 [21:05<03:05, 16.14it/s]
Running loglikelihood requests:  85%|████████▌ | 17153/20084 [21:09<03:01, 16.17it/s]
Running loglikelihood requests:  86%|████████▌ | 17217/20084 [21:13<02:56, 16.20it/s]
Running loglikelihood requests:  86%|████████▌ | 17281/20084 [21:17<02:52, 16.22it/s]
Running loglikelihood requests:  86%|████████▋ | 17345/20084 [21:21<02:48, 16.24it/s]
Running loglikelihood requests:  87%|████████▋ | 17409/20084 [21:25<02:44, 16.27it/s]
Running loglikelihood requests:  87%|████████▋ | 17473/20084 [21:29<02:40, 16.30it/s]
Running loglikelihood requests:  87%|████████▋ | 17537/20084 [21:32<02:36, 16.32it/s]
Running loglikelihood requests:  88%|████████▊ | 17601/20084 [21:36<02:31, 16.35it/s]
Running loglikelihood requests:  88%|████████▊ | 17665/20084 [21:40<02:27, 16.39it/s]
Running loglikelihood requests:  88%|████████▊ | 17729/20084 [21:44<02:23, 16.42it/s]
Running loglikelihood requests:  89%|████████▊ | 17793/20084 [21:48<02:19, 16.45it/s]
Running loglikelihood requests:  89%|████████▉ | 17857/20084 [21:52<02:15, 16.48it/s]
Running loglikelihood requests:  89%|████████▉ | 17921/20084 [21:56<02:10, 16.51it/s]
Running loglikelihood requests:  90%|████████▉ | 17985/20084 [22:00<02:06, 16.55it/s]
Running loglikelihood requests:  90%|████████▉ | 18049/20084 [22:03<02:02, 16.58it/s]
Running loglikelihood requests:  90%|█████████ | 18113/20084 [22:07<01:58, 16.61it/s]
Running loglikelihood requests:  91%|█████████ | 18177/20084 [22:11<01:54, 16.64it/s]
Running loglikelihood requests:  91%|█████████ | 18241/20084 [22:15<01:50, 16.68it/s]
Running loglikelihood requests:  91%|█████████ | 18305/20084 [22:19<01:46, 16.73it/s]
Running loglikelihood requests:  91%|█████████▏| 18369/20084 [22:22<01:42, 16.78it/s]
Running loglikelihood requests:  92%|█████████▏| 18433/20084 [22:26<01:38, 16.81it/s]
Running loglikelihood requests:  92%|█████████▏| 18497/20084 [22:30<01:34, 16.86it/s]
Running loglikelihood requests:  92%|█████████▏| 18561/20084 [22:34<01:30, 16.92it/s]
Running loglikelihood requests:  93%|█████████▎| 18625/20084 [22:37<01:25, 16.98it/s]
Running loglikelihood requests:  93%|█████████▎| 18689/20084 [22:41<01:21, 17.02it/s]
Running loglikelihood requests:  93%|█████████▎| 18753/20084 [22:45<01:17, 17.07it/s]
Running loglikelihood requests:  94%|█████████▎| 18817/20084 [22:49<01:13, 17.13it/s]
Running loglikelihood requests:  94%|█████████▍| 18881/20084 [22:52<01:10, 17.18it/s]
Running loglikelihood requests:  94%|█████████▍| 18945/20084 [22:56<01:06, 17.24it/s]
Running loglikelihood requests:  95%|█████████▍| 19009/20084 [23:00<01:02, 17.28it/s]
Running loglikelihood requests:  95%|█████████▍| 19073/20084 [23:03<00:58, 17.34it/s]
Running loglikelihood requests:  95%|█████████▌| 19137/20084 [23:07<00:54, 17.45it/s]
Running loglikelihood requests:  96%|█████████▌| 19201/20084 [23:11<00:50, 17.54it/s]
Running loglikelihood requests:  96%|█████████▌| 19265/20084 [23:14<00:46, 17.62it/s]
Running loglikelihood requests:  96%|█████████▌| 19329/20084 [23:18<00:42, 17.71it/s]
Running loglikelihood requests:  97%|█████████▋| 19393/20084 [23:21<00:38, 17.79it/s]
Running loglikelihood requests:  97%|█████████▋| 19457/20084 [23:25<00:35, 17.87it/s]
Running loglikelihood requests:  97%|█████████▋| 19521/20084 [23:28<00:31, 17.95it/s]
Running loglikelihood requests:  98%|█████████▊| 19585/20084 [23:32<00:27, 18.05it/s]
Running loglikelihood requests:  98%|█████████▊| 19649/20084 [23:35<00:23, 18.16it/s]
Running loglikelihood requests:  98%|█████████▊| 19713/20084 [23:39<00:20, 18.31it/s]
Running loglikelihood requests:  98%|█████████▊| 19777/20084 [23:42<00:16, 18.48it/s]
Running loglikelihood requests:  99%|█████████▉| 19841/20084 [23:46<00:12, 18.70it/s]
Running loglikelihood requests:  99%|█████████▉| 19905/20084 [23:49<00:09, 18.92it/s]
Running loglikelihood requests:  99%|█████████▉| 19969/20084 [23:52<00:05, 19.23it/s]
Running loglikelihood requests: 100%|█████████▉| 20033/20084 [23:54<00:02, 20.79it/s]
Running loglikelihood requests: 100%|██████████| 20084/20084 [23:55<00:00, 14.00it/s]
n136-128-154:1720461:1723019 [1] NCCL INFO Channel 00/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1723019 [1] NCCL INFO Channel 01/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1723019 [1] NCCL INFO Channel 02/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1723019 [1] NCCL INFO Channel 03/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1723019 [1] NCCL INFO Channel 04/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1723019 [1] NCCL INFO Channel 05/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1723019 [1] NCCL INFO Channel 06/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1723019 [1] NCCL INFO Channel 07/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1723019 [1] NCCL INFO Channel 08/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1723019 [1] NCCL INFO Channel 09/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1723019 [1] NCCL INFO Channel 10/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1723019 [1] NCCL INFO Channel 11/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1723019 [1] NCCL INFO Channel 12/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1723019 [1] NCCL INFO Channel 13/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1723019 [1] NCCL INFO Channel 14/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1723019 [1] NCCL INFO Channel 15/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1723019 [1] NCCL INFO Channel 16/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1723019 [1] NCCL INFO Channel 17/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1723019 [1] NCCL INFO Channel 18/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1723019 [1] NCCL INFO Channel 19/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1723019 [1] NCCL INFO Channel 20/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1723019 [1] NCCL INFO Channel 21/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1723019 [1] NCCL INFO Channel 22/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1723019 [1] NCCL INFO Channel 23/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1723019 [1] NCCL INFO Channel 24/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1723019 [1] NCCL INFO Channel 25/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1723019 [1] NCCL INFO Channel 26/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1723019 [1] NCCL INFO Channel 27/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1723019 [1] NCCL INFO Channel 28/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1723019 [1] NCCL INFO Channel 29/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1723019 [1] NCCL INFO Channel 30/1 : 1[2] -> 0[1] via P2P/CUMEM/read
n136-128-154:1720461:1723019 [1] NCCL INFO Channel 31/1 : 1[2] -> 0[1] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:1720461:1723024 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1720461:1723024 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1720461:1723024 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1720461:1723024 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1720461:1723024 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1720461:1723024 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1720461:1720705 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-06:05:41:33 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Llama-3.1-8B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 10, batch_size: auto (64)
|  Tasks  |Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|---------|------:|------|-----:|--------|---|-----:|---|-----:|
|hellaswag|      1|none  |    10|acc     |↑  |0.4510|±  |0.0050|
|         |       |none  |    10|acc_norm|↑  |0.6418|±  |0.0048|

[rank0]:[W1206 05:41:34.198573901 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:1720460:1723080 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1720460:1723080 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1720460:1723080 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1720460:1720707 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:1720460:1723080 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:1720460:1723080 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:1720460:1723080 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:1720461:1720705 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:1720461:1723024 [1] NCCL INFO comm 0x15f07470 rank 1 nranks 2 cudaDev 1 busId 4a000 - Abort COMPLETE
n136-128-154:1720460:1723080 [0] NCCL INFO comm 0x134d07d0 rank 0 nranks 2 cudaDev 0 busId 16000 - Abort COMPLETE
任务 hellaswag 评估完成！
Execution time: 7164.04 seconds