nohup: ignoring input
Namespace(save_dir='/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg', self_attn_layer_to_quant='23 22 25 24 26', mlp_layer_to_quant='27', model_id='/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-7B', cuda_id=6)
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:07<00:21,  7.23s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:14<00:14,  7.05s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:20<00:06,  6.94s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:27<00:00,  6.82s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:27<00:00,  6.90s/it]
Qwen2ForCausalLM(
  (model): Qwen2Model(
    (embed_tokens): Embedding(152064, 3584)
    (layers): ModuleList(
      (0-27): 28 x Qwen2DecoderLayer(
        (self_attn): Qwen2Attention(
          (q_proj): Linear(in_features=3584, out_features=3584, bias=True)
          (k_proj): Linear(in_features=3584, out_features=512, bias=True)
          (v_proj): Linear(in_features=3584, out_features=512, bias=True)
          (o_proj): Linear(in_features=3584, out_features=3584, bias=False)
        )
        (mlp): Qwen2MLP(
          (gate_proj): Linear(in_features=3584, out_features=18944, bias=False)
          (up_proj): Linear(in_features=3584, out_features=18944, bias=False)
          (down_proj): Linear(in_features=18944, out_features=3584, bias=False)
          (act_fn): SiLUActivation()
        )
        (input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
        (post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
      )
    )
    (norm): Qwen2RMSNorm((3584,), eps=1e-06)
    (rotary_emb): Qwen2RotaryEmbedding()
  )
  (lm_head): Linear(in_features=3584, out_features=152064, bias=False)
)
Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation.
Once upon a time there was a poor woodcutter who had a young son. One day the woodcutter went out into the forest to cut wood. He was very tired, and he sat down under a tree to rest.
As he sat there, he heard a voice. It said, “Woodcutter, you are too tired. Why don’t you take a nap?”
The woodcutter was very surprised, but he lay down under the tree and soon fell asleep.
When he woke up,
quanting ... 
layers.0.self_attn 4 bit quant 
layers.0.mlp 4 bit quant 
layers.1.self_attn 4 bit quant 
layers.1.mlp 4 bit quant 
layers.2.self_attn 4 bit quant 
layers.2.mlp 4 bit quant 
layers.3.self_attn 4 bit quant 
layers.3.mlp 4 bit quant 
layers.4.self_attn 4 bit quant 
layers.4.mlp 4 bit quant 
layers.5.self_attn 4 bit quant 
layers.5.mlp 4 bit quant 
layers.6.self_attn 4 bit quant 
layers.6.mlp 4 bit quant 
layers.7.self_attn 4 bit quant 
layers.7.mlp 4 bit quant 
layers.8.self_attn 4 bit quant 
layers.8.mlp 4 bit quant 
layers.9.self_attn 4 bit quant 
layers.9.mlp 4 bit quant 
layers.10.self_attn 4 bit quant 
layers.10.mlp 4 bit quant 
layers.11.self_attn 4 bit quant 
layers.11.mlp 4 bit quant 
layers.12.self_attn 4 bit quant 
layers.12.mlp 4 bit quant 
layers.13.self_attn 4 bit quant 
layers.13.mlp 4 bit quant 
layers.14.self_attn 4 bit quant 
layers.14.mlp 4 bit quant 
layers.15.self_attn 4 bit quant 
layers.15.mlp 4 bit quant 
layers.16.self_attn 4 bit quant 
layers.16.mlp 4 bit quant 
layers.17.self_attn 4 bit quant 
layers.17.mlp 4 bit quant 
layers.18.self_attn 4 bit quant 
layers.18.mlp 4 bit quant 
layers.19.self_attn 4 bit quant 
layers.19.mlp 4 bit quant 
layers.20.self_attn 4 bit quant 
layers.20.mlp 4 bit quant 
layers.21.self_attn 4 bit quant 
layers.21.mlp 4 bit quant 
layers.22.self_attn 2 bit quant 
layers.22.mlp 4 bit quant 
layers.23.self_attn 2 bit quant 
layers.23.mlp 4 bit quant 
layers.24.self_attn 2 bit quant 
layers.24.mlp 4 bit quant 
layers.25.self_attn 2 bit quant 
layers.25.mlp 4 bit quant 
layers.26.self_attn 2 bit quant 
layers.26.mlp 4 bit quant 
layers.27.self_attn 4 bit quant 
layers.27.mlp 2 bit quant 
quanted
Qwen2ForCausalLM(
  (model): Qwen2Model(
    (embed_tokens): Embedding(152064, 3584)
    (layers): ModuleList(
      (0-27): 28 x Qwen2DecoderLayer(
        (self_attn): Qwen2Attention(
          (q_proj): W8A16Linear(3584, 3584, bias=True, weight_quant=per_channel)
          (k_proj): W8A16Linear(3584, 512, bias=True, weight_quant=per_channel)
          (v_proj): W8A16Linear(3584, 512, bias=True, weight_quant=per_channel)
          (o_proj): W8A16Linear(3584, 3584, bias=False, weight_quant=per_channel)
        )
        (mlp): Qwen2MLP(
          (gate_proj): W8A16Linear(3584, 18944, bias=False, weight_quant=per_channel)
          (up_proj): W8A16Linear(3584, 18944, bias=False, weight_quant=per_channel)
          (down_proj): W8A16Linear(18944, 3584, bias=False, weight_quant=per_channel)
          (act_fn): SiLUActivation()
        )
        (input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
        (post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
      )
    )
    (norm): Qwen2RMSNorm((3584,), eps=1e-06)
    (rotary_emb): Qwen2RotaryEmbedding()
  )
  (lm_head): Linear(in_features=3584, out_features=152064, bias=False)
)
Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation.
Once upon a time, in a land not so different from our own, a man of means has just ret ret ret ret ret retired from a life of the, um, fulltime. He has just retired from a life of   1  . He has a bit of money set aside for the first year of retirement, and a plan for the rest of his days. He has a plan of a sort. He has a plan to the  2 . He has a plan to do what he wants

==================================================
开始评估：任务=winogrande | 少样本数=5 | 模型=Qwen2.5-7B-quantization-fg
输出路径：results2/Qwen2.5-7B-quantization-fg/fg4/winogrande.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-09:12:13:56 INFO     [__main__:440] Selected Tasks: ['winogrande']
2025-12-09:12:13:56 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:12:13:56 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-09:12:13:56 INFO     [__main__:440] Selected Tasks: ['winogrande']
2025-12-09:12:13:56 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:12:13:56 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-09:12:13:57 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:12:13:57 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:12:13:58 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.18it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.04it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:03,  1.55s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:03<00:03,  1.60s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.18s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.25it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.04it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.27s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:04<00:00,  1.14it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:04<00:00,  1.04s/it]
2025-12-09:12:14:21 WARNING  [evaluator:309] Overwriting default num_fewshot of winogrande from None to 5
2025-12-09:12:14:21 INFO     [api.task:434] Building contexts for winogrande on rank 1...

  0%|          | 0/633 [00:00<?, ?it/s]
100%|██████████| 633/633 [00:00<00:00, 26473.96it/s]
2025-12-09:12:14:24 WARNING  [evaluator:309] Overwriting default num_fewshot of winogrande from None to 5
2025-12-09:12:14:24 INFO     [api.task:434] Building contexts for winogrande on rank 0...

  0%|          | 0/634 [00:00<?, ?it/s]
100%|██████████| 634/634 [00:00<00:00, 25153.13it/s]
n136-128-154:2176198:2176198 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2176198:2176198 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2176198:2176198 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:2176198:2176198 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2176198:2176198 [0] NCCL INFO Comm config Blocking set to 1
n136-128-154:2176200:2176200 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:2176200:2176200 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2176200:2176200 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2176200:2176200 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2176200:2176200 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:2176198:2176460 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2176198:2176460 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2176200:2176461 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2176200:2176461 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2176198:2176460 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2176198:2176460 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2176198:2176460 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2176200:2176461 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2176200:2176461 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2176200:2176461 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2176198:2176460 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2176198:2176460 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2176198:2176460 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:2176198:2176460 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2176198:2176460 [0] NCCL INFO Using network IB
n136-128-154:2176200:2176461 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2176200:2176461 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2176200:2176461 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:2176200:2176461 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2176200:2176461 [1] NCCL INFO Using network IB
n136-128-154:2176198:2176460 [0] NCCL INFO ncclCommInitRankConfig comm 0x10f213c0 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0xfd365c632a5270ba - Init START
n136-128-154:2176200:2176461 [1] NCCL INFO ncclCommInitRankConfig comm 0x109a6240 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0xfd365c632a5270ba - Init START
n136-128-154:2176200:2176461 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2176198:2176460 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2176200:2176461 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2176200:2176461 [1] NCCL INFO Retrieving state for IB
n136-128-154:2176200:2176461 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:2176200:2176461 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2176200:2176461 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2176200:2176461 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2176200:2176461 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2176198:2176460 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2176198:2176460 [0] NCCL INFO Retrieving state for IB
n136-128-154:2176198:2176460 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:2176198:2176460 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2176198:2176460 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2176198:2176460 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2176198:2176460 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2176200:2176461 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2176200:2176461 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2176198:2176460 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2176198:2176460 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2176198:2176460 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2176198:2176460 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2176198:2176460 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2176198:2176460 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2176198:2176460 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2176198:2176460 [0] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2176198:2176460 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2176198:2176460 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2176198:2176460 [0] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2176198:2176460 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2176198:2176460 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2176198:2176460 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2176198:2176460 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2176198:2176460 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2176198:2176460 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2176198:2176460 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2176198:2176460 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2176198:2176460 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2176198:2176460 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2176198:2176460 [0] NCCL INFO ==========================================
n136-128-154:2176198:2176460 [0] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2176198:2176460 [0] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2176198:2176460 [0] NCCL INFO Setting affinity for GPU 6 to 33-62,97-126
n136-128-154:2176200:2176461 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2176200:2176461 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2176200:2176461 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2176200:2176461 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2176200:2176461 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2176200:2176461 [1] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2176200:2176461 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2176200:2176461 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2176200:2176461 [1] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2176200:2176461 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2176200:2176461 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2176200:2176461 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2176200:2176461 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2176200:2176461 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2176200:2176461 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2176200:2176461 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2176200:2176461 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2176200:2176461 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2176200:2176461 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2176200:2176461 [1] NCCL INFO ==========================================
n136-128-154:2176200:2176461 [1] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2176200:2176461 [1] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2176200:2176461 [1] NCCL INFO Setting affinity for GPU 7 to 33-62,97-126
n136-128-154:2176198:2176460 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2176198:2176460 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176198:2176460 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176198:2176460 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176198:2176460 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176198:2176460 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176198:2176460 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176198:2176460 [0] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176198:2176460 [0] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176198:2176460 [0] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176198:2176460 [0] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176198:2176460 [0] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176198:2176460 [0] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176198:2176460 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2176198:2176460 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176198:2176460 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176198:2176460 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176198:2176460 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176198:2176460 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176198:2176460 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176200:2176461 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2176198:2176460 [0] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2176198:2176460 [0] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2176200:2176461 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176198:2176460 [0] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2176198:2176460 [0] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2176200:2176461 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176198:2176460 [0] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2176198:2176460 [0] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2176200:2176461 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176200:2176461 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176200:2176461 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176200:2176461 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176200:2176461 [1] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176200:2176461 [1] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176200:2176461 [1] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176200:2176461 [1] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176200:2176461 [1] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176200:2176461 [1] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176200:2176461 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2176200:2176461 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176200:2176461 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176200:2176461 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176200:2176461 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176200:2176461 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176200:2176461 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176200:2176461 [1] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2176200:2176461 [1] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2176200:2176461 [1] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2176200:2176461 [1] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2176200:2176461 [1] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2176200:2176461 [1] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2176198:2176460 [0] NCCL INFO comm 0x10f213c0 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:2176200:2176461 [1] NCCL INFO comm 0x109a6240 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:2176198:2176460 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:2176198:2176460 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:2176200:2176461 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:2176198:2176460 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:2176200:2176461 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:2176198:2176460 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:2176200:2176461 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:2176198:2176460 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:2176200:2176461 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:2176198:2176460 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:2176200:2176461 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:2176198:2176460 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:2176200:2176461 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:2176198:2176460 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:2176200:2176461 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:2176198:2176460 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:2176200:2176461 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:2176198:2176460 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:2176200:2176461 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:2176198:2176460 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:2176200:2176461 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:2176198:2176460 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:2176200:2176461 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:2176198:2176460 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:2176200:2176461 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:2176198:2176460 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:2176200:2176461 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:2176198:2176460 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:2176200:2176461 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:2176198:2176460 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:2176200:2176461 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:2176198:2176460 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:2176200:2176461 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:2176198:2176460 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:2176200:2176461 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:2176198:2176460 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:2176200:2176461 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:2176198:2176460 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:2176200:2176461 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:2176198:2176460 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:2176200:2176461 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:2176198:2176460 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:2176200:2176461 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:2176198:2176460 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:2176200:2176461 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:2176198:2176460 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:2176200:2176461 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:2176200:2176461 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:2176198:2176460 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:2176198:2176460 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:2176198:2176460 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:2176200:2176461 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:2176198:2176460 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:2176200:2176461 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:2176198:2176460 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:2176200:2176461 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:2176198:2176460 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:2176200:2176461 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:2176198:2176460 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:2176200:2176461 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:2176198:2176460 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:2176200:2176461 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:2176198:2176460 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:2176200:2176461 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:2176198:2176460 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:2176200:2176461 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:2176198:2176460 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:2176200:2176461 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:2176198:2176460 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:2176200:2176461 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:2176198:2176460 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:2176200:2176461 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:2176198:2176460 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:2176200:2176461 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:2176198:2176460 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:2176200:2176461 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:2176198:2176460 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:2176200:2176461 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:2176198:2176460 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:2176200:2176461 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:2176198:2176460 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:2176200:2176461 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:2176198:2176460 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:2176200:2176461 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:2176198:2176460 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:2176200:2176461 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:2176198:2176460 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:2176198:2176460 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:2176200:2176461 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:2176198:2176460 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:2176200:2176461 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:2176198:2176460 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:2176200:2176461 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:2176198:2176460 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:2176200:2176461 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:2176198:2176460 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:2176200:2176461 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:2176198:2176460 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:2176200:2176461 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:2176198:2176460 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:2176200:2176461 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:2176198:2176460 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:2176200:2176461 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2176198:2176460 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:2176198:2176460 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:2176198:2176460 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:2176198:2176460 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:2176198:2176460 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:2176198:2176460 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:2176198:2176460 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:2176198:2176460 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:2176198:2176460 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:2176198:2176460 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:2176198:2176460 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:2176198:2176460 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:2176198:2176460 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:2176198:2176460 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:2176198:2176460 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:2176198:2176460 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:2176198:2176460 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:2176198:2176460 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:2176198:2176460 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:2176198:2176460 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:2176198:2176460 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2176200:2176461 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2176200:2176474 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 55
n136-128-154:2176200:2176473 [1] NCCL INFO [Proxy Service] Device 1 CPU core 118
n136-128-154:2176198:2176460 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2176198:2176460 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:2176198:2176476 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 35
n136-128-154:2176198:2176475 [0] NCCL INFO [Proxy Service] Device 0 CPU core 98
n136-128-154:2176200:2176461 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2176200:2176461 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2176198:2176460 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2176198:2176460 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2176198:2176460 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:2176200:2176461 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2176200:2176461 [1] NCCL INFO ncclCommInitRankConfig comm 0x109a6240 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0xfd365c632a5270ba - Init COMPLETE
n136-128-154:2176200:2176461 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.75 (kernels 0.11, alloc 0.36, bootstrap 0.00, allgathers 0.00, topo 0.06, graphs 0.00, connections 0.09, rest 0.12)
n136-128-154:2176198:2176460 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2176198:2176460 [0] NCCL INFO ncclCommInitRankConfig comm 0x10f213c0 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0xfd365c632a5270ba - Init COMPLETE
n136-128-154:2176198:2176460 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.76 (kernels 0.12, alloc 0.37, bootstrap 0.00, allgathers 0.00, topo 0.06, graphs 0.00, connections 0.10, rest 0.12)
n136-128-154:2176200:2176478 [1] NCCL INFO Channel 00/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176200:2176478 [1] NCCL INFO Channel 01/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176200:2176478 [1] NCCL INFO Channel 02/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176200:2176478 [1] NCCL INFO Channel 03/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176200:2176478 [1] NCCL INFO Channel 04/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176200:2176478 [1] NCCL INFO Channel 05/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176198:2176479 [0] NCCL INFO Channel 00/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176200:2176478 [1] NCCL INFO Channel 06/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176198:2176479 [0] NCCL INFO Channel 01/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176200:2176478 [1] NCCL INFO Channel 07/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176198:2176479 [0] NCCL INFO Channel 02/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176200:2176478 [1] NCCL INFO Channel 08/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176198:2176479 [0] NCCL INFO Channel 03/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176200:2176478 [1] NCCL INFO Channel 09/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176198:2176479 [0] NCCL INFO Channel 04/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176200:2176478 [1] NCCL INFO Channel 10/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176198:2176479 [0] NCCL INFO Channel 05/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176200:2176478 [1] NCCL INFO Channel 11/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176198:2176479 [0] NCCL INFO Channel 06/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176200:2176478 [1] NCCL INFO Channel 12/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176198:2176479 [0] NCCL INFO Channel 07/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176200:2176478 [1] NCCL INFO Channel 13/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176198:2176479 [0] NCCL INFO Channel 08/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176200:2176478 [1] NCCL INFO Channel 14/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176198:2176479 [0] NCCL INFO Channel 09/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176200:2176478 [1] NCCL INFO Channel 15/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176198:2176479 [0] NCCL INFO Channel 10/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176200:2176478 [1] NCCL INFO Channel 16/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176198:2176479 [0] NCCL INFO Channel 11/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176200:2176478 [1] NCCL INFO Channel 17/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176198:2176479 [0] NCCL INFO Channel 12/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176200:2176478 [1] NCCL INFO Channel 18/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176198:2176479 [0] NCCL INFO Channel 13/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176200:2176478 [1] NCCL INFO Channel 19/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176198:2176479 [0] NCCL INFO Channel 14/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176200:2176478 [1] NCCL INFO Channel 20/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176198:2176479 [0] NCCL INFO Channel 15/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176200:2176478 [1] NCCL INFO Channel 21/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176198:2176479 [0] NCCL INFO Channel 16/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176200:2176478 [1] NCCL INFO Channel 22/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176198:2176479 [0] NCCL INFO Channel 17/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176200:2176478 [1] NCCL INFO Channel 23/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176198:2176479 [0] NCCL INFO Channel 18/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176198:2176479 [0] NCCL INFO Channel 19/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176198:2176479 [0] NCCL INFO Channel 20/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176198:2176479 [0] NCCL INFO Channel 21/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176198:2176479 [0] NCCL INFO Channel 22/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176198:2176479 [0] NCCL INFO Channel 23/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176198:2176479 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:2176200:2176478 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-09:12:14:25 INFO     [evaluator:559] Running loglikelihood requests
2025-12-09:12:14:25 INFO     [evaluator:559] Running loglikelihood requests

Running loglikelihood requests:   0%|          | 0/1268 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 64
Determined largest batch size: 64

Running loglikelihood requests:   0%|          | 1/1268 [00:05<1:56:08,  5.50s/it]
Running loglikelihood requests:   5%|▌         | 65/1268 [00:06<01:28, 13.58it/s] 
Running loglikelihood requests:  10%|█         | 129/1268 [00:07<00:44, 25.81it/s]
Running loglikelihood requests:  15%|█▌        | 193/1268 [00:08<00:29, 36.62it/s]
Running loglikelihood requests:  20%|██        | 257/1268 [00:09<00:22, 45.63it/s]
Running loglikelihood requests:  25%|██▌       | 321/1268 [00:09<00:17, 52.93it/s]
Running loglikelihood requests:  30%|███       | 385/1268 [00:10<00:15, 58.80it/s]
Running loglikelihood requests:  35%|███▌      | 449/1268 [00:11<00:12, 63.33it/s]
Running loglikelihood requests:  40%|████      | 513/1268 [00:12<00:11, 66.51it/s]
Running loglikelihood requests:  46%|████▌     | 577/1268 [00:13<00:09, 69.32it/s]
Running loglikelihood requests:  51%|█████     | 641/1268 [00:14<00:08, 71.58it/s]
Running loglikelihood requests:  56%|█████▌    | 705/1268 [00:14<00:07, 73.21it/s]
Running loglikelihood requests:  61%|██████    | 769/1268 [00:15<00:06, 74.58it/s]
Running loglikelihood requests:  66%|██████▌   | 833/1268 [00:16<00:05, 75.56it/s]
Running loglikelihood requests:  71%|███████   | 897/1268 [00:17<00:04, 76.41it/s]
Running loglikelihood requests:  76%|███████▌  | 961/1268 [00:18<00:03, 77.05it/s]
Running loglikelihood requests:  81%|████████  | 1025/1268 [00:19<00:03, 77.91it/s]
Running loglikelihood requests:  86%|████████▌ | 1089/1268 [00:19<00:02, 78.75it/s]
Running loglikelihood requests:  91%|█████████ | 1153/1268 [00:20<00:01, 79.77it/s]
Running loglikelihood requests:  96%|█████████▌| 1217/1268 [00:21<00:00, 85.05it/s]
Running loglikelihood requests: 100%|██████████| 1268/1268 [00:21<00:00, 59.61it/s]
n136-128-154:2176200:2176506 [1] NCCL INFO Channel 00/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176200:2176506 [1] NCCL INFO Channel 01/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176200:2176506 [1] NCCL INFO Channel 02/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176200:2176506 [1] NCCL INFO Channel 03/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176200:2176506 [1] NCCL INFO Channel 04/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176200:2176506 [1] NCCL INFO Channel 05/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176200:2176506 [1] NCCL INFO Channel 06/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176200:2176506 [1] NCCL INFO Channel 07/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176200:2176506 [1] NCCL INFO Channel 08/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176200:2176506 [1] NCCL INFO Channel 09/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176200:2176506 [1] NCCL INFO Channel 10/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176200:2176506 [1] NCCL INFO Channel 11/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176200:2176506 [1] NCCL INFO Channel 12/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176200:2176506 [1] NCCL INFO Channel 13/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176200:2176506 [1] NCCL INFO Channel 14/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176200:2176506 [1] NCCL INFO Channel 15/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176200:2176506 [1] NCCL INFO Channel 16/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176200:2176506 [1] NCCL INFO Channel 17/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176200:2176506 [1] NCCL INFO Channel 18/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176200:2176506 [1] NCCL INFO Channel 19/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176200:2176506 [1] NCCL INFO Channel 20/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176200:2176506 [1] NCCL INFO Channel 21/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176200:2176506 [1] NCCL INFO Channel 22/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176200:2176506 [1] NCCL INFO Channel 23/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176200:2176506 [1] NCCL INFO Channel 24/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176200:2176506 [1] NCCL INFO Channel 25/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176200:2176506 [1] NCCL INFO Channel 26/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176200:2176506 [1] NCCL INFO Channel 27/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176200:2176506 [1] NCCL INFO Channel 28/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176200:2176506 [1] NCCL INFO Channel 29/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176200:2176506 [1] NCCL INFO Channel 30/1 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176200:2176506 [1] NCCL INFO Channel 31/1 : 1[7] -> 0[6] via P2P/CUMEM/read
fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization'
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization
n136-128-154:2176200:2176512 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2176200:2176512 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2176200:2176512 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2176200:2176512 [1] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2176200:2176512 [1] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2176200:2176512 [1] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2176200:2176473 [1] NCCL INFO misc/socket.cc:915 -> 3
2025-12-09:12:14:58 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: auto (64)
|  Tasks   |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|----------|------:|------|-----:|------|---|-----:|---|-----:|
|winogrande|      1|none  |     5|acc   |↑  |0.6006|±  |0.0138|

[rank0]:[W1209 12:14:59.438047570 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
n136-128-154:2176198:2176562 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2176198:2176562 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2176198:2176562 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2176198:2176475 [0] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2176198:2176562 [0] NCCL INFO misc/socket.cc:64 -> 3
n136-128-154:2176198:2176562 [0] NCCL INFO misc/socket.cc:81 -> 3
n136-128-154:2176200:2176473 [1] NCCL INFO misc/socket.cc:915 -> 3
n136-128-154:2176198:2176562 [0] NCCL INFO misc/socket.cc:863 -> 3
n136-128-154:2176200:2176512 [1] NCCL INFO comm 0x109a6240 rank 1 nranks 2 cudaDev 1 busId c9000 - Abort COMPLETE
n136-128-154:2176198:2176562 [0] NCCL INFO comm 0x10f213c0 rank 0 nranks 2 cudaDev 0 busId c5000 - Abort COMPLETE
任务 winogrande 评估完成！

==================================================
开始评估：任务=gsm8k | 少样本数=4 | 模型=Qwen2.5-7B-quantization-fg
输出路径：results2/Qwen2.5-7B-quantization-fg/fg4/gsm8k.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-09:12:16:25 INFO     [__main__:440] Selected Tasks: ['gsm8k']
2025-12-09:12:16:25 INFO     [__main__:440] Selected Tasks: ['gsm8k']
2025-12-09:12:16:25 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:12:16:25 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-09:12:16:25 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:12:16:25 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-09:12:16:26 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:12:16:27 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:12:16:27 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.25s/it]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.25s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.25s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.25s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.20s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.20s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.20it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.20it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.02it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.02it/s]
2025-12-09:12:16:50 INFO     [evaluator:290] gsm8k: Using gen_kwargs: {'until': ['Question:', '</s>', '<|im_end|>'], 'do_sample': False, 'temperature': 0.0}
2025-12-09:12:16:50 WARNING  [evaluator:309] Overwriting default num_fewshot of gsm8k from 5 to 4
2025-12-09:12:16:50 INFO     [api.task:434] Building contexts for gsm8k on rank 1...

  0%|          | 0/659 [00:00<?, ?it/s]
  6%|▌         | 41/659 [00:00<00:01, 404.83it/s]
 13%|█▎        | 83/659 [00:00<00:01, 409.88it/s]
 19%|█▉        | 124/659 [00:00<00:01, 401.86it/s]
 25%|██▌       | 166/659 [00:00<00:01, 405.21it/s]
 32%|███▏      | 208/659 [00:00<00:01, 407.74it/s]
 38%|███▊      | 250/659 [00:00<00:01, 408.59it/s]
 44%|████▍     | 292/659 [00:00<00:00, 410.18it/s]
 51%|█████     | 334/659 [00:00<00:00, 405.56it/s]
 57%|█████▋    | 375/659 [00:00<00:00, 406.82it/s]
 63%|██████▎   | 416/659 [00:01<00:00, 406.17it/s]
 69%|██████▉   | 457/659 [00:01<00:00, 396.52it/s]
 76%|███████▌  | 498/659 [00:01<00:00, 396.78it/s]
 82%|████████▏ | 539/659 [00:01<00:00, 398.00it/s]
 88%|████████▊ | 579/659 [00:01<00:00, 392.73it/s]
 94%|█████████▍| 621/659 [00:01<00:00, 397.93it/s]
100%|██████████| 659/659 [00:01<00:00, 402.39it/s]
2025-12-09:12:16:57 INFO     [evaluator:290] gsm8k: Using gen_kwargs: {'until': ['Question:', '</s>', '<|im_end|>'], 'do_sample': False, 'temperature': 0.0}
2025-12-09:12:16:57 WARNING  [evaluator:309] Overwriting default num_fewshot of gsm8k from 5 to 4
2025-12-09:12:16:57 INFO     [api.task:434] Building contexts for gsm8k on rank 0...

  0%|          | 0/660 [00:00<?, ?it/s]
  6%|▋         | 42/660 [00:00<00:01, 415.48it/s]
 13%|█▎        | 84/660 [00:00<00:01, 417.55it/s]
 19%|█▉        | 127/660 [00:00<00:01, 420.27it/s]
 26%|██▌       | 170/660 [00:00<00:01, 416.86it/s]
 32%|███▏      | 213/660 [00:00<00:01, 418.04it/s]
 39%|███▊      | 255/660 [00:00<00:00, 417.25it/s]
 45%|████▌     | 298/660 [00:00<00:00, 418.85it/s]
 52%|█████▏    | 341/660 [00:00<00:00, 419.98it/s]
 58%|█████▊    | 384/660 [00:00<00:00, 420.63it/s]
 65%|██████▍   | 427/660 [00:01<00:00, 417.94it/s]
 71%|███████   | 469/660 [00:01<00:00, 417.58it/s]
 78%|███████▊  | 512/660 [00:01<00:00, 418.98it/s]
 84%|████████▍ | 555/660 [00:01<00:00, 419.99it/s]
 91%|█████████ | 598/660 [00:01<00:00, 421.09it/s]
 97%|█████████▋| 641/660 [00:01<00:00, 417.38it/s]
100%|██████████| 660/660 [00:01<00:00, 418.45it/s]
n136-128-154:2176617:2176617 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2176617:2176617 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2176617:2176617 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:2176617:2176617 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2176617:2176617 [0] NCCL INFO Comm config Blocking set to 1
n136-128-154:2176618:2176618 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:2176618:2176618 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2176618:2176618 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2176618:2176618 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2176618:2176618 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:2176618:2176879 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2176618:2176879 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2176617:2176878 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2176617:2176878 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2176618:2176879 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2176618:2176879 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2176618:2176879 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2176617:2176878 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2176617:2176878 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2176617:2176878 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2176617:2176878 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2176617:2176878 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2176617:2176878 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:2176617:2176878 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2176617:2176878 [0] NCCL INFO Using network IB
n136-128-154:2176618:2176879 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2176618:2176879 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2176618:2176879 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:2176618:2176879 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2176618:2176879 [1] NCCL INFO Using network IB
n136-128-154:2176617:2176878 [0] NCCL INFO ncclCommInitRankConfig comm 0xf900720 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0x65c92858ccf27062 - Init START
n136-128-154:2176618:2176879 [1] NCCL INFO ncclCommInitRankConfig comm 0xfe20660 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0x65c92858ccf27062 - Init START
n136-128-154:2176618:2176879 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2176617:2176878 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2176617:2176878 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2176617:2176878 [0] NCCL INFO Retrieving state for IB
n136-128-154:2176617:2176878 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:2176617:2176878 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2176618:2176879 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2176618:2176879 [1] NCCL INFO Retrieving state for IB
n136-128-154:2176618:2176879 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:2176618:2176879 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2176617:2176878 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2176618:2176879 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2176617:2176878 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2176618:2176879 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2176617:2176878 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2176618:2176879 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2176617:2176878 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2176618:2176879 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2176617:2176878 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2176618:2176879 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2176617:2176878 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2176617:2176878 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2176617:2176878 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2176617:2176878 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2176617:2176878 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2176618:2176879 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2176617:2176878 [0] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2176617:2176878 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2176618:2176879 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2176617:2176878 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2176618:2176879 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2176617:2176878 [0] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2176618:2176879 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2176617:2176878 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2176617:2176878 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2176618:2176879 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2176617:2176878 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2176618:2176879 [1] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2176617:2176878 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2176617:2176878 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2176618:2176879 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2176617:2176878 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2176618:2176879 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2176617:2176878 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2176617:2176878 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2176618:2176879 [1] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2176617:2176878 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2176618:2176879 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2176617:2176878 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2176617:2176878 [0] NCCL INFO ==========================================
n136-128-154:2176618:2176879 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2176618:2176879 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2176617:2176878 [0] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2176618:2176879 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2176618:2176879 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2176617:2176878 [0] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2176618:2176879 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2176618:2176879 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2176618:2176879 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2176617:2176878 [0] NCCL INFO Setting affinity for GPU 6 to 33-62,97-126
n136-128-154:2176618:2176879 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2176618:2176879 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2176618:2176879 [1] NCCL INFO ==========================================
n136-128-154:2176618:2176879 [1] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2176618:2176879 [1] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2176618:2176879 [1] NCCL INFO Setting affinity for GPU 7 to 33-62,97-126
n136-128-154:2176617:2176878 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2176617:2176878 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176617:2176878 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176617:2176878 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176617:2176878 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176617:2176878 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176617:2176878 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176617:2176878 [0] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176617:2176878 [0] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176617:2176878 [0] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176617:2176878 [0] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176617:2176878 [0] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176617:2176878 [0] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176617:2176878 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2176617:2176878 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176617:2176878 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176617:2176878 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176617:2176878 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176617:2176878 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176617:2176878 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176617:2176878 [0] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2176617:2176878 [0] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2176617:2176878 [0] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2176617:2176878 [0] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2176617:2176878 [0] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2176617:2176878 [0] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2176618:2176879 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2176618:2176879 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176618:2176879 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176618:2176879 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176618:2176879 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176618:2176879 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176618:2176879 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176618:2176879 [1] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176618:2176879 [1] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176618:2176879 [1] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176618:2176879 [1] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176618:2176879 [1] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176618:2176879 [1] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176618:2176879 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2176618:2176879 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176618:2176879 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176618:2176879 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176618:2176879 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176618:2176879 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176618:2176879 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2176618:2176879 [1] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2176618:2176879 [1] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2176618:2176879 [1] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2176618:2176879 [1] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2176618:2176879 [1] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2176618:2176879 [1] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2176617:2176878 [0] NCCL INFO comm 0xf900720 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:2176617:2176878 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:2176617:2176878 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:2176617:2176878 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:2176617:2176878 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:2176617:2176878 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:2176617:2176878 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:2176617:2176878 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:2176617:2176878 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:2176617:2176878 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:2176617:2176878 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:2176617:2176878 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:2176617:2176878 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:2176617:2176878 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:2176617:2176878 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:2176617:2176878 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:2176617:2176878 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:2176617:2176878 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:2176617:2176878 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:2176617:2176878 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:2176617:2176878 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:2176617:2176878 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:2176617:2176878 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:2176617:2176878 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:2176617:2176878 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:2176617:2176878 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:2176617:2176878 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:2176617:2176878 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:2176617:2176878 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:2176617:2176878 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:2176617:2176878 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:2176617:2176878 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:2176617:2176878 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:2176617:2176878 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:2176617:2176878 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:2176617:2176878 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:2176617:2176878 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:2176617:2176878 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:2176617:2176878 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:2176617:2176878 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:2176617:2176878 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:2176617:2176878 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:2176617:2176878 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:2176617:2176878 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:2176617:2176878 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:2176617:2176878 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:2176617:2176878 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:2176617:2176878 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:2176617:2176878 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:2176617:2176878 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:2176617:2176878 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:2176617:2176878 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:2176617:2176878 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:2176617:2176878 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:2176617:2176878 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:2176617:2176878 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:2176617:2176878 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:2176617:2176878 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:2176617:2176878 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:2176617:2176878 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:2176617:2176878 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:2176617:2176878 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:2176617:2176878 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:2176617:2176878 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:2176617:2176878 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:2176617:2176878 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:2176617:2176878 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:2176617:2176878 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:2176617:2176878 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:2176617:2176878 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:2176617:2176878 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:2176617:2176878 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:2176617:2176878 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:2176617:2176878 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:2176617:2176878 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2176618:2176879 [1] NCCL INFO comm 0xfe20660 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:2176618:2176879 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:2176618:2176879 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:2176618:2176879 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:2176618:2176879 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:2176618:2176879 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:2176618:2176879 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:2176618:2176879 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:2176618:2176879 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:2176618:2176879 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:2176618:2176879 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:2176618:2176879 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:2176618:2176879 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:2176618:2176879 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:2176618:2176879 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:2176618:2176879 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:2176618:2176879 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:2176618:2176879 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:2176618:2176879 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:2176618:2176879 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:2176618:2176879 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:2176618:2176879 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:2176618:2176879 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:2176618:2176879 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:2176618:2176879 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:2176618:2176879 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:2176618:2176879 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:2176618:2176879 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:2176618:2176879 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:2176618:2176879 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:2176618:2176879 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:2176618:2176879 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:2176618:2176879 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:2176618:2176879 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:2176618:2176879 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:2176618:2176879 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:2176618:2176879 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:2176618:2176879 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:2176618:2176879 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:2176618:2176879 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:2176618:2176879 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:2176618:2176879 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:2176618:2176879 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:2176618:2176879 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:2176618:2176879 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:2176618:2176879 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:2176618:2176879 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:2176618:2176879 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:2176618:2176879 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:2176618:2176879 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:2176618:2176879 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2176617:2176878 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2176617:2176878 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:2176617:2176890 [0] NCCL INFO [Proxy Service] Device 0 CPU core 98
n136-128-154:2176617:2176891 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 35
n136-128-154:2176618:2176879 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2176618:2176892 [1] NCCL INFO [Proxy Service] Device 1 CPU core 37
n136-128-154:2176618:2176893 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 102
n136-128-154:2176617:2176878 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2176617:2176878 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2176617:2176878 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:2176618:2176879 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2176618:2176879 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2176618:2176879 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2176618:2176879 [1] NCCL INFO ncclCommInitRankConfig comm 0xfe20660 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0x65c92858ccf27062 - Init COMPLETE
n136-128-154:2176618:2176879 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.78 (kernels 0.23, alloc 0.37, bootstrap 0.00, allgathers 0.01, topo 0.07, graphs 0.00, connections 0.01, rest 0.09)
n136-128-154:2176617:2176878 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2176617:2176878 [0] NCCL INFO ncclCommInitRankConfig comm 0xf900720 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0x65c92858ccf27062 - Init COMPLETE
n136-128-154:2176617:2176878 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.79 (kernels 0.23, alloc 0.36, bootstrap 0.01, allgathers 0.01, topo 0.07, graphs 0.00, connections 0.01, rest 0.09)
n136-128-154:2176618:2176894 [1] NCCL INFO Channel 00/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176618:2176894 [1] NCCL INFO Channel 01/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176618:2176894 [1] NCCL INFO Channel 02/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176617:2176895 [0] NCCL INFO Channel 00/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176618:2176894 [1] NCCL INFO Channel 03/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176617:2176895 [0] NCCL INFO Channel 01/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176618:2176894 [1] NCCL INFO Channel 04/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176617:2176895 [0] NCCL INFO Channel 02/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176618:2176894 [1] NCCL INFO Channel 05/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176617:2176895 [0] NCCL INFO Channel 03/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176618:2176894 [1] NCCL INFO Channel 06/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176617:2176895 [0] NCCL INFO Channel 04/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176618:2176894 [1] NCCL INFO Channel 07/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176617:2176895 [0] NCCL INFO Channel 05/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176618:2176894 [1] NCCL INFO Channel 08/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176617:2176895 [0] NCCL INFO Channel 06/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176618:2176894 [1] NCCL INFO Channel 09/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176617:2176895 [0] NCCL INFO Channel 07/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176618:2176894 [1] NCCL INFO Channel 10/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176617:2176895 [0] NCCL INFO Channel 08/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176618:2176894 [1] NCCL INFO Channel 11/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176617:2176895 [0] NCCL INFO Channel 09/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176618:2176894 [1] NCCL INFO Channel 12/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176617:2176895 [0] NCCL INFO Channel 10/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176618:2176894 [1] NCCL INFO Channel 13/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176617:2176895 [0] NCCL INFO Channel 11/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176618:2176894 [1] NCCL INFO Channel 14/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176617:2176895 [0] NCCL INFO Channel 12/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176618:2176894 [1] NCCL INFO Channel 15/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176617:2176895 [0] NCCL INFO Channel 13/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176618:2176894 [1] NCCL INFO Channel 16/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176617:2176895 [0] NCCL INFO Channel 14/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176618:2176894 [1] NCCL INFO Channel 17/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176617:2176895 [0] NCCL INFO Channel 15/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176618:2176894 [1] NCCL INFO Channel 18/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176617:2176895 [0] NCCL INFO Channel 16/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176618:2176894 [1] NCCL INFO Channel 19/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176617:2176895 [0] NCCL INFO Channel 17/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176618:2176894 [1] NCCL INFO Channel 20/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176617:2176895 [0] NCCL INFO Channel 18/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176618:2176894 [1] NCCL INFO Channel 21/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176617:2176895 [0] NCCL INFO Channel 19/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176618:2176894 [1] NCCL INFO Channel 22/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176617:2176895 [0] NCCL INFO Channel 20/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176618:2176894 [1] NCCL INFO Channel 23/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2176617:2176895 [0] NCCL INFO Channel 21/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176617:2176895 [0] NCCL INFO Channel 22/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176617:2176895 [0] NCCL INFO Channel 23/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2176617:2176895 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:2176618:2176894 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-09:12:17:00 INFO     [evaluator:559] Running generate_until requests
2025-12-09:12:17:00 INFO     [evaluator:559] Running generate_until requests

Running generate_until requests:   0%|          | 0/660 [00:00<?, ?it/s]Passed argument batch_size = auto. Detecting largest batch size
Passed argument batch_size = auto. Detecting largest batch size
Determined Largest batch size: 1
Determined Largest batch size: 1
Both `max_new_tokens` (=2048) and `max_length`(=1578) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1605) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   0%|          | 1/660 [02:47<30:36:12, 167.18s/it]Both `max_new_tokens` (=2048) and `max_length`(=1567) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1554) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   0%|          | 2/660 [03:36<17:52:49, 97.83s/it] Both `max_new_tokens` (=2048) and `max_length`(=1557) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1539) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   0%|          | 3/660 [04:25<13:48:56, 75.70s/it]Both `max_new_tokens` (=2048) and `max_length`(=1519) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1536) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   1%|          | 4/660 [05:15<11:55:48, 65.47s/it]Both `max_new_tokens` (=2048) and `max_length`(=1508) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1484) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   1%|          | 5/660 [06:05<10:51:51, 59.71s/it]Both `max_new_tokens` (=2048) and `max_length`(=1494) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1483) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   1%|          | 6/660 [06:53<10:09:37, 55.93s/it]Both `max_new_tokens` (=2048) and `max_length`(=1480) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1459) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   1%|          | 7/660 [07:42<9:42:32, 53.53s/it] Both `max_new_tokens` (=2048) and `max_length`(=1468) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1456) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   1%|          | 8/660 [08:32<9:29:33, 52.41s/it]Both `max_new_tokens` (=2048) and `max_length`(=1459) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1446) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   1%|▏         | 9/660 [09:24<9:26:20, 52.20s/it]Both `max_new_tokens` (=2048) and `max_length`(=1451) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1440) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   2%|▏         | 10/660 [10:13<9:17:14, 51.44s/it]Both `max_new_tokens` (=2048) and `max_length`(=1435) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1440) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   2%|▏         | 11/660 [11:04<9:14:47, 51.29s/it]Both `max_new_tokens` (=2048) and `max_length`(=1431) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1431) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   2%|▏         | 12/660 [11:56<9:14:32, 51.35s/it]Both `max_new_tokens` (=2048) and `max_length`(=1429) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1427) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   2%|▏         | 13/660 [12:45<9:08:01, 50.82s/it]Both `max_new_tokens` (=2048) and `max_length`(=1393) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1423) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   2%|▏         | 14/660 [13:35<9:03:15, 50.46s/it]Both `max_new_tokens` (=2048) and `max_length`(=1387) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1410) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   2%|▏         | 15/660 [14:24<8:59:18, 50.17s/it]Both `max_new_tokens` (=2048) and `max_length`(=1386) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1398) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   2%|▏         | 16/660 [15:14<8:56:12, 49.96s/it]Both `max_new_tokens` (=2048) and `max_length`(=1382) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1397) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   3%|▎         | 17/660 [16:03<8:53:43, 49.80s/it]Both `max_new_tokens` (=2048) and `max_length`(=1379) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1396) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   3%|▎         | 18/660 [16:53<8:51:28, 49.67s/it]Both `max_new_tokens` (=2048) and `max_length`(=1377) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1384) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   3%|▎         | 19/660 [17:42<8:50:04, 49.62s/it]Both `max_new_tokens` (=2048) and `max_length`(=1377) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1382) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   3%|▎         | 20/660 [18:32<8:49:35, 49.65s/it]Both `max_new_tokens` (=2048) and `max_length`(=1372) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1382) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   3%|▎         | 21/660 [19:22<8:49:05, 49.68s/it]Both `max_new_tokens` (=2048) and `max_length`(=1365) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1373) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   3%|▎         | 22/660 [20:11<8:47:32, 49.61s/it]Both `max_new_tokens` (=2048) and `max_length`(=1365) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1357) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   3%|▎         | 23/660 [21:01<8:45:59, 49.54s/it]Both `max_new_tokens` (=2048) and `max_length`(=1353) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1353) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   4%|▎         | 24/660 [21:50<8:44:16, 49.46s/it]Both `max_new_tokens` (=2048) and `max_length`(=1352) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1352) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   4%|▍         | 25/660 [22:40<8:44:17, 49.54s/it]Both `max_new_tokens` (=2048) and `max_length`(=1342) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1351) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   4%|▍         | 26/660 [23:29<8:43:23, 49.53s/it]Both `max_new_tokens` (=2048) and `max_length`(=1341) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1349) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   4%|▍         | 27/660 [24:18<8:42:08, 49.49s/it]Both `max_new_tokens` (=2048) and `max_length`(=1340) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1338) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   4%|▍         | 28/660 [25:08<8:41:18, 49.49s/it]Both `max_new_tokens` (=2048) and `max_length`(=1337) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1329) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   4%|▍         | 29/660 [25:57<8:40:19, 49.48s/it]Both `max_new_tokens` (=2048) and `max_length`(=1323) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1328) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   5%|▍         | 30/660 [26:47<8:39:33, 49.48s/it]Both `max_new_tokens` (=2048) and `max_length`(=1322) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1327) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   5%|▍         | 31/660 [27:36<8:39:04, 49.51s/it]Both `max_new_tokens` (=2048) and `max_length`(=1321) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1322) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   5%|▍         | 32/660 [28:26<8:38:19, 49.52s/it]Both `max_new_tokens` (=2048) and `max_length`(=1318) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1318) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   5%|▌         | 33/660 [29:15<8:36:54, 49.46s/it]Both `max_new_tokens` (=2048) and `max_length`(=1317) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1317) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   5%|▌         | 34/660 [30:05<8:35:48, 49.44s/it]Both `max_new_tokens` (=2048) and `max_length`(=1313) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1317) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   5%|▌         | 35/660 [30:54<8:34:50, 49.42s/it]Both `max_new_tokens` (=2048) and `max_length`(=1313) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1317) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   5%|▌         | 36/660 [31:44<8:34:02, 49.43s/it]Both `max_new_tokens` (=2048) and `max_length`(=1312) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1317) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   6%|▌         | 37/660 [32:33<8:33:17, 49.43s/it]Both `max_new_tokens` (=2048) and `max_length`(=1306) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1310) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   6%|▌         | 38/660 [33:22<8:32:16, 49.42s/it]Both `max_new_tokens` (=2048) and `max_length`(=1303) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1307) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   6%|▌         | 39/660 [34:12<8:31:27, 49.42s/it]Both `max_new_tokens` (=2048) and `max_length`(=1300) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1306) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   6%|▌         | 40/660 [35:01<8:30:49, 49.43s/it]Both `max_new_tokens` (=2048) and `max_length`(=1297) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1302) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   6%|▌         | 41/660 [35:51<8:29:49, 49.42s/it]Both `max_new_tokens` (=2048) and `max_length`(=1297) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1298) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   6%|▋         | 42/660 [36:40<8:29:16, 49.44s/it]Both `max_new_tokens` (=2048) and `max_length`(=1296) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1295) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   7%|▋         | 43/660 [37:30<8:28:30, 49.45s/it]Both `max_new_tokens` (=2048) and `max_length`(=1293) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1292) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   7%|▋         | 44/660 [38:19<8:27:33, 49.44s/it]Both `max_new_tokens` (=2048) and `max_length`(=1290) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1290) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   7%|▋         | 45/660 [39:09<8:27:00, 49.46s/it]Both `max_new_tokens` (=2048) and `max_length`(=1282) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1289) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   7%|▋         | 46/660 [39:58<8:26:02, 49.45s/it]Both `max_new_tokens` (=2048) and `max_length`(=1278) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1288) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   7%|▋         | 47/660 [40:48<8:25:44, 49.50s/it]Both `max_new_tokens` (=2048) and `max_length`(=1277) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1282) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   7%|▋         | 48/660 [41:37<8:25:49, 49.59s/it]Both `max_new_tokens` (=2048) and `max_length`(=1274) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1275) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   7%|▋         | 49/660 [42:27<8:25:17, 49.62s/it]Both `max_new_tokens` (=2048) and `max_length`(=1270) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1273) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   8%|▊         | 50/660 [43:17<8:24:27, 49.62s/it]Both `max_new_tokens` (=2048) and `max_length`(=1269) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1270) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   8%|▊         | 51/660 [44:06<8:23:32, 49.61s/it]Both `max_new_tokens` (=2048) and `max_length`(=1269) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1270) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   8%|▊         | 52/660 [44:56<8:22:49, 49.62s/it]Both `max_new_tokens` (=2048) and `max_length`(=1265) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1268) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   8%|▊         | 53/660 [45:45<8:21:48, 49.60s/it]Both `max_new_tokens` (=2048) and `max_length`(=1264) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1267) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   8%|▊         | 54/660 [46:35<8:20:38, 49.57s/it]Both `max_new_tokens` (=2048) and `max_length`(=1262) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1256) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   8%|▊         | 55/660 [47:24<8:19:01, 49.49s/it]Both `max_new_tokens` (=2048) and `max_length`(=1258) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1255) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   8%|▊         | 56/660 [48:14<8:18:04, 49.48s/it]Both `max_new_tokens` (=2048) and `max_length`(=1257) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1255) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   9%|▊         | 57/660 [49:03<8:17:28, 49.50s/it]Both `max_new_tokens` (=2048) and `max_length`(=1256) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1255) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   9%|▉         | 58/660 [49:53<8:16:14, 49.46s/it]Both `max_new_tokens` (=2048) and `max_length`(=1254) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1252) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   9%|▉         | 59/660 [50:42<8:14:58, 49.42s/it]Both `max_new_tokens` (=2048) and `max_length`(=1251) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1251) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   9%|▉         | 60/660 [51:31<8:13:59, 49.40s/it]Both `max_new_tokens` (=2048) and `max_length`(=1249) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1251) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   9%|▉         | 61/660 [52:21<8:12:46, 49.36s/it]Both `max_new_tokens` (=2048) and `max_length`(=1246) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1250) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:   9%|▉         | 62/660 [53:10<8:12:04, 49.37s/it]Both `max_new_tokens` (=2048) and `max_length`(=1245) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1247) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  10%|▉         | 63/660 [53:59<8:11:15, 49.37s/it]Both `max_new_tokens` (=2048) and `max_length`(=1245) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1247) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  10%|▉         | 64/660 [54:49<8:10:32, 49.38s/it]Both `max_new_tokens` (=2048) and `max_length`(=1245) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1246) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  10%|▉         | 65/660 [55:38<8:09:56, 49.41s/it]Both `max_new_tokens` (=2048) and `max_length`(=1241) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1245) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  10%|█         | 66/660 [56:27<8:08:40, 49.36s/it]Both `max_new_tokens` (=2048) and `max_length`(=1239) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1243) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  10%|█         | 67/660 [57:17<8:07:54, 49.37s/it]Both `max_new_tokens` (=2048) and `max_length`(=1238) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1241) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  10%|█         | 68/660 [58:06<8:06:54, 49.35s/it]Both `max_new_tokens` (=2048) and `max_length`(=1236) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1241) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  10%|█         | 69/660 [58:55<8:05:56, 49.33s/it]Both `max_new_tokens` (=2048) and `max_length`(=1234) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1239) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  11%|█         | 70/660 [59:45<8:04:42, 49.29s/it]Both `max_new_tokens` (=2048) and `max_length`(=1233) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1238) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  11%|█         | 71/660 [1:00:34<8:03:07, 49.21s/it]Both `max_new_tokens` (=2048) and `max_length`(=1233) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1236) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  11%|█         | 72/660 [1:01:23<8:01:25, 49.13s/it]Both `max_new_tokens` (=2048) and `max_length`(=1233) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1236) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  11%|█         | 73/660 [1:02:11<7:58:48, 48.94s/it]Both `max_new_tokens` (=2048) and `max_length`(=1231) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1235) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  11%|█         | 74/660 [1:03:00<7:56:49, 48.82s/it]Both `max_new_tokens` (=2048) and `max_length`(=1227) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  11%|█▏        | 75/660 [1:03:00<5:34:29, 34.31s/it]Both `max_new_tokens` (=2048) and `max_length`(=1227) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1235) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  12%|█▏        | 76/660 [1:03:49<6:15:25, 38.57s/it]Both `max_new_tokens` (=2048) and `max_length`(=1225) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1234) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  12%|█▏        | 77/660 [1:04:37<6:44:04, 41.59s/it]Both `max_new_tokens` (=2048) and `max_length`(=1225) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1233) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  12%|█▏        | 78/660 [1:05:26<7:04:06, 43.72s/it]Both `max_new_tokens` (=2048) and `max_length`(=1224) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1230) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  12%|█▏        | 79/660 [1:06:15<7:17:49, 45.21s/it]Both `max_new_tokens` (=2048) and `max_length`(=1223) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1229) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  12%|█▏        | 80/660 [1:07:03<7:26:49, 46.22s/it]Both `max_new_tokens` (=2048) and `max_length`(=1223) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1227) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  12%|█▏        | 81/660 [1:07:52<7:33:42, 47.02s/it]Both `max_new_tokens` (=2048) and `max_length`(=1222) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1226) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  12%|█▏        | 82/660 [1:08:42<7:40:38, 47.82s/it]Both `max_new_tokens` (=2048) and `max_length`(=1222) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1223) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  13%|█▎        | 83/660 [1:09:32<7:47:34, 48.62s/it]Both `max_new_tokens` (=2048) and `max_length`(=1221) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1221) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  13%|█▎        | 84/660 [1:10:23<7:51:49, 49.15s/it]Both `max_new_tokens` (=2048) and `max_length`(=1218) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1221) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  13%|█▎        | 85/660 [1:11:12<7:51:55, 49.24s/it]Both `max_new_tokens` (=2048) and `max_length`(=1217) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1220) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  13%|█▎        | 86/660 [1:12:02<7:52:02, 49.34s/it]Both `max_new_tokens` (=2048) and `max_length`(=1214) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1218) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  13%|█▎        | 87/660 [1:12:52<7:55:14, 49.76s/it]Both `max_new_tokens` (=2048) and `max_length`(=1214) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1215) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  13%|█▎        | 88/660 [1:13:43<7:57:44, 50.11s/it]Both `max_new_tokens` (=2048) and `max_length`(=1208) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1214) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  13%|█▎        | 89/660 [1:14:33<7:55:48, 50.00s/it]Both `max_new_tokens` (=2048) and `max_length`(=1207) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1214) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  14%|█▎        | 90/660 [1:15:23<7:54:10, 49.91s/it]Both `max_new_tokens` (=2048) and `max_length`(=1207) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1213) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  14%|█▍        | 91/660 [1:16:12<7:52:37, 49.84s/it]Both `max_new_tokens` (=2048) and `max_length`(=1206) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1211) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  14%|█▍        | 92/660 [1:17:02<7:51:48, 49.84s/it]Both `max_new_tokens` (=2048) and `max_length`(=1205) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1208) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  14%|█▍        | 93/660 [1:17:53<7:53:45, 50.13s/it]Both `max_new_tokens` (=2048) and `max_length`(=1205) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1208) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  14%|█▍        | 94/660 [1:18:43<7:52:32, 50.09s/it]Both `max_new_tokens` (=2048) and `max_length`(=1204) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1207) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  14%|█▍        | 95/660 [1:19:32<7:48:32, 49.76s/it]Both `max_new_tokens` (=2048) and `max_length`(=1203) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1206) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  15%|█▍        | 96/660 [1:20:21<7:45:20, 49.50s/it]Both `max_new_tokens` (=2048) and `max_length`(=1203) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1203) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  15%|█▍        | 97/660 [1:21:10<7:42:43, 49.31s/it]Both `max_new_tokens` (=2048) and `max_length`(=1202) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1203) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  15%|█▍        | 98/660 [1:21:59<7:41:08, 49.23s/it]Both `max_new_tokens` (=2048) and `max_length`(=1202) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1202) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  15%|█▌        | 99/660 [1:22:48<7:39:25, 49.14s/it]Both `max_new_tokens` (=2048) and `max_length`(=1202) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1199) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  15%|█▌        | 100/660 [1:23:37<7:37:50, 49.05s/it]Both `max_new_tokens` (=2048) and `max_length`(=1201) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1199) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  15%|█▌        | 101/660 [1:24:26<7:36:26, 48.99s/it]Both `max_new_tokens` (=2048) and `max_length`(=1200) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1199) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  15%|█▌        | 102/660 [1:25:14<7:34:47, 48.90s/it]Both `max_new_tokens` (=2048) and `max_length`(=1198) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1198) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  16%|█▌        | 103/660 [1:26:03<7:34:07, 48.92s/it]Both `max_new_tokens` (=2048) and `max_length`(=1197) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1197) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  16%|█▌        | 104/660 [1:26:52<7:33:31, 48.94s/it]Both `max_new_tokens` (=2048) and `max_length`(=1196) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  16%|█▌        | 105/660 [1:27:42<7:36:14, 49.32s/it]Both `max_new_tokens` (=2048) and `max_length`(=1196) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1197) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  16%|█▌        | 106/660 [1:28:32<7:36:45, 49.47s/it]Both `max_new_tokens` (=2048) and `max_length`(=1195) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1194) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  16%|█▌        | 107/660 [1:29:21<7:35:03, 49.37s/it]Both `max_new_tokens` (=2048) and `max_length`(=1193) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1193) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  16%|█▋        | 108/660 [1:30:11<7:34:11, 49.37s/it]Both `max_new_tokens` (=2048) and `max_length`(=1192) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1193) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  17%|█▋        | 109/660 [1:31:01<7:37:13, 49.79s/it]Both `max_new_tokens` (=2048) and `max_length`(=1192) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1192) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  17%|█▋        | 110/660 [1:31:51<7:36:42, 49.82s/it]Both `max_new_tokens` (=2048) and `max_length`(=1191) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1192) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  17%|█▋        | 111/660 [1:32:40<7:33:05, 49.52s/it]Both `max_new_tokens` (=2048) and `max_length`(=1191) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1191) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  17%|█▋        | 112/660 [1:33:30<7:32:55, 49.59s/it]Both `max_new_tokens` (=2048) and `max_length`(=1190) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1191) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  17%|█▋        | 113/660 [1:34:20<7:34:33, 49.86s/it]Both `max_new_tokens` (=2048) and `max_length`(=1189) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1190) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  17%|█▋        | 114/660 [1:35:09<7:31:35, 49.63s/it]Both `max_new_tokens` (=2048) and `max_length`(=1188) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1187) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  17%|█▋        | 115/660 [1:35:59<7:29:37, 49.50s/it]Both `max_new_tokens` (=2048) and `max_length`(=1187) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1186) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  18%|█▊        | 116/660 [1:36:48<7:28:07, 49.43s/it]Both `max_new_tokens` (=2048) and `max_length`(=1186) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1185) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  18%|█▊        | 117/660 [1:37:37<7:26:52, 49.38s/it]Both `max_new_tokens` (=2048) and `max_length`(=1185) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1185) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  18%|█▊        | 118/660 [1:38:27<7:25:52, 49.36s/it]Both `max_new_tokens` (=2048) and `max_length`(=1184) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1185) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  18%|█▊        | 119/660 [1:39:16<7:24:39, 49.31s/it]Both `max_new_tokens` (=2048) and `max_length`(=1184) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1184) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  18%|█▊        | 120/660 [1:40:05<7:23:39, 49.30s/it]Both `max_new_tokens` (=2048) and `max_length`(=1184) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1183) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  18%|█▊        | 121/660 [1:40:54<7:23:03, 49.32s/it]Both `max_new_tokens` (=2048) and `max_length`(=1184) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1183) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  18%|█▊        | 122/660 [1:41:44<7:22:15, 49.32s/it]Both `max_new_tokens` (=2048) and `max_length`(=1183) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1182) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  19%|█▊        | 123/660 [1:42:33<7:21:30, 49.33s/it]Both `max_new_tokens` (=2048) and `max_length`(=1183) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1181) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  19%|█▉        | 124/660 [1:43:22<7:20:19, 49.29s/it]Both `max_new_tokens` (=2048) and `max_length`(=1182) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1179) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  19%|█▉        | 125/660 [1:44:11<7:19:14, 49.26s/it]Both `max_new_tokens` (=2048) and `max_length`(=1182) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1179) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  19%|█▉        | 126/660 [1:44:19<5:28:19, 36.89s/it]Both `max_new_tokens` (=2048) and `max_length`(=1181) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1178) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  19%|█▉        | 127/660 [1:45:09<6:00:31, 40.58s/it]Both `max_new_tokens` (=2048) and `max_length`(=1181) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1178) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  19%|█▉        | 128/660 [1:45:58<6:23:14, 43.22s/it]Both `max_new_tokens` (=2048) and `max_length`(=1180) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1177) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  20%|█▉        | 129/660 [1:46:47<6:38:36, 45.04s/it]Both `max_new_tokens` (=2048) and `max_length`(=1178) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  20%|█▉        | 130/660 [1:47:37<6:48:57, 46.30s/it]Both `max_new_tokens` (=2048) and `max_length`(=1178) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1176) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  20%|█▉        | 131/660 [1:48:26<6:55:35, 47.14s/it]Both `max_new_tokens` (=2048) and `max_length`(=1178) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1176) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  20%|██        | 132/660 [1:49:15<7:00:19, 47.76s/it]Both `max_new_tokens` (=2048) and `max_length`(=1178) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1176) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  20%|██        | 133/660 [1:50:04<7:03:21, 48.20s/it]Both `max_new_tokens` (=2048) and `max_length`(=1175) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1173) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  20%|██        | 134/660 [1:50:53<7:05:22, 48.52s/it]Both `max_new_tokens` (=2048) and `max_length`(=1173) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1174) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  20%|██        | 135/660 [1:51:43<7:07:03, 48.81s/it]Both `max_new_tokens` (=2048) and `max_length`(=1173) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1174) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  21%|██        | 136/660 [1:52:33<7:10:00, 49.24s/it]Both `max_new_tokens` (=2048) and `max_length`(=1170) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1173) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  21%|██        | 137/660 [1:53:23<7:10:17, 49.37s/it]Both `max_new_tokens` (=2048) and `max_length`(=1170) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1173) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  21%|██        | 138/660 [1:54:12<7:10:00, 49.43s/it]Both `max_new_tokens` (=2048) and `max_length`(=1168) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1171) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  21%|██        | 139/660 [1:55:02<7:09:07, 49.42s/it]Both `max_new_tokens` (=2048) and `max_length`(=1168) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1170) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  21%|██        | 140/660 [1:55:51<7:07:47, 49.36s/it]Both `max_new_tokens` (=2048) and `max_length`(=1167) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1169) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  21%|██▏       | 141/660 [1:56:40<7:06:39, 49.33s/it]Both `max_new_tokens` (=2048) and `max_length`(=1165) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1169) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  22%|██▏       | 142/660 [1:57:29<7:05:21, 49.27s/it]Both `max_new_tokens` (=2048) and `max_length`(=1165) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1167) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  22%|██▏       | 143/660 [1:58:19<7:04:17, 49.24s/it]Both `max_new_tokens` (=2048) and `max_length`(=1164) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1166) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  22%|██▏       | 144/660 [1:59:08<7:03:06, 49.20s/it]Both `max_new_tokens` (=2048) and `max_length`(=1163) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1166) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  22%|██▏       | 145/660 [1:59:57<7:02:05, 49.18s/it]Both `max_new_tokens` (=2048) and `max_length`(=1163) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1164) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  22%|██▏       | 146/660 [2:00:48<7:05:49, 49.71s/it]Both `max_new_tokens` (=2048) and `max_length`(=1162) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1162) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  22%|██▏       | 147/660 [2:01:39<7:08:28, 50.11s/it]Both `max_new_tokens` (=2048) and `max_length`(=1161) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1161) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  22%|██▏       | 148/660 [2:02:28<7:05:39, 49.88s/it]Both `max_new_tokens` (=2048) and `max_length`(=1159) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1161) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  23%|██▎       | 149/660 [2:03:17<7:01:47, 49.52s/it]Both `max_new_tokens` (=2048) and `max_length`(=1159) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1160) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  23%|██▎       | 150/660 [2:04:07<7:02:34, 49.71s/it]Both `max_new_tokens` (=2048) and `max_length`(=1159) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1159) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  23%|██▎       | 151/660 [2:04:58<7:04:23, 50.03s/it]Both `max_new_tokens` (=2048) and `max_length`(=1157) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1159) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  23%|██▎       | 152/660 [2:05:47<7:01:38, 49.80s/it]Both `max_new_tokens` (=2048) and `max_length`(=1156) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1158) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  23%|██▎       | 153/660 [2:06:36<6:57:51, 49.45s/it]Both `max_new_tokens` (=2048) and `max_length`(=1156) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1157) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  23%|██▎       | 154/660 [2:07:24<6:55:00, 49.21s/it]Both `max_new_tokens` (=2048) and `max_length`(=1155) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1153) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  23%|██▎       | 155/660 [2:08:13<6:53:03, 49.08s/it]Both `max_new_tokens` (=2048) and `max_length`(=1155) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1153) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  24%|██▎       | 156/660 [2:09:04<6:56:45, 49.61s/it]Both `max_new_tokens` (=2048) and `max_length`(=1154) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1153) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  24%|██▍       | 157/660 [2:09:54<6:57:48, 49.84s/it]Both `max_new_tokens` (=2048) and `max_length`(=1154) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1152) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  24%|██▍       | 158/660 [2:10:43<6:54:18, 49.52s/it]Both `max_new_tokens` (=2048) and `max_length`(=1154) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1152) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  24%|██▍       | 159/660 [2:11:33<6:54:49, 49.68s/it]Both `max_new_tokens` (=2048) and `max_length`(=1153) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1151) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  24%|██▍       | 160/660 [2:12:24<6:57:12, 50.07s/it]Both `max_new_tokens` (=2048) and `max_length`(=1151) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1151) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  24%|██▍       | 161/660 [2:13:13<6:53:12, 49.68s/it]Both `max_new_tokens` (=2048) and `max_length`(=1151) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1150) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  25%|██▍       | 162/660 [2:14:04<6:55:29, 50.06s/it]Both `max_new_tokens` (=2048) and `max_length`(=1150) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1150) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  25%|██▍       | 163/660 [2:14:54<6:55:29, 50.16s/it]Both `max_new_tokens` (=2048) and `max_length`(=1150) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1150) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  25%|██▍       | 164/660 [2:15:43<6:52:06, 49.85s/it]Both `max_new_tokens` (=2048) and `max_length`(=1148) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1149) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  25%|██▌       | 165/660 [2:16:32<6:49:26, 49.63s/it]Both `max_new_tokens` (=2048) and `max_length`(=1148) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1149) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  25%|██▌       | 166/660 [2:17:21<6:46:46, 49.41s/it]Both `max_new_tokens` (=2048) and `max_length`(=1148) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1149) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  25%|██▌       | 167/660 [2:18:10<6:44:14, 49.20s/it]Both `max_new_tokens` (=2048) and `max_length`(=1146) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  25%|██▌       | 168/660 [2:18:11<4:44:09, 34.65s/it]Both `max_new_tokens` (=2048) and `max_length`(=1144) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1149) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  26%|██▌       | 169/660 [2:19:00<5:18:29, 38.92s/it]Both `max_new_tokens` (=2048) and `max_length`(=1144) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1148) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  26%|██▌       | 170/660 [2:19:48<5:42:19, 41.92s/it]Both `max_new_tokens` (=2048) and `max_length`(=1144) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1148) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  26%|██▌       | 171/660 [2:20:37<5:58:20, 43.97s/it]Both `max_new_tokens` (=2048) and `max_length`(=1144) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1148) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  26%|██▌       | 172/660 [2:21:26<6:10:14, 45.52s/it]Both `max_new_tokens` (=2048) and `max_length`(=1144) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1147) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  26%|██▌       | 173/660 [2:22:16<6:19:32, 46.76s/it]Both `max_new_tokens` (=2048) and `max_length`(=1143) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1147) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  26%|██▋       | 174/660 [2:23:06<6:26:13, 47.68s/it]Both `max_new_tokens` (=2048) and `max_length`(=1143) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1146) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  27%|██▋       | 175/660 [2:23:55<6:29:01, 48.13s/it]Both `max_new_tokens` (=2048) and `max_length`(=1143) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1146) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  27%|██▋       | 176/660 [2:24:44<6:30:45, 48.44s/it]Both `max_new_tokens` (=2048) and `max_length`(=1143) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Both `max_new_tokens` (=2048) and `max_length`(=1146) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  27%|██▋       | 177/660 [2:25:33<6:31:58, 48.69s/it]Both `max_new_tokens` (=2048) and `max_length`(=1142) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
W1209 14:43:04.603000 2176566 site-packages/torch/distributed/elastic/agent/server/api.py:725] Received 15 death signal, shutting down workers
W1209 14:43:04.606000 2176566 site-packages/torch/distributed/elastic/multiprocessing/api.py:908] Sending process 2176617 closing signal SIGTERM
W1209 14:43:04.607000 2176566 site-packages/torch/distributed/elastic/multiprocessing/api.py:908] Sending process 2176618 closing signal SIGTERM
Traceback (most recent call last):
  File "/mnt/bn/life-mllm/users/cxr/zhy/miniconda3/envs/lm-evaluation-harness/bin/accelerate", line 7, in <module>
    sys.exit(main())
             ^^^^^^
  File "/mnt/bn/life-mllm/users/cxr/zhy/miniconda3/envs/lm-evaluation-harness/lib/python3.11/site-packages/accelerate/commands/accelerate_cli.py", line 50, in main
    args.func(args)
  File "/mnt/bn/life-mllm/users/cxr/zhy/miniconda3/envs/lm-evaluation-harness/lib/python3.11/site-packages/accelerate/commands/launch.py", line 1272, in launch_command
    multi_gpu_launcher(args)
  File "/mnt/bn/life-mllm/users/cxr/zhy/miniconda3/envs/lm-evaluation-harness/lib/python3.11/site-packages/accelerate/commands/launch.py", line 899, in multi_gpu_launcher
    distrib_run.run(args)
  File "/mnt/bn/life-mllm/users/cxr/zhy/miniconda3/envs/lm-evaluation-harness/lib/python3.11/site-packages/torch/distributed/run.py", line 927, in run
    elastic_launch(
  File "/mnt/bn/life-mllm/users/cxr/zhy/miniconda3/envs/lm-evaluation-harness/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 156, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/bn/life-mllm/users/cxr/zhy/miniconda3/envs/lm-evaluation-harness/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 284, in launch_agent
    result = agent.run()
             ^^^^^^^^^^^
  File "/mnt/bn/life-mllm/users/cxr/zhy/miniconda3/envs/lm-evaluation-harness/lib/python3.11/site-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
    result = f(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^
  File "/mnt/bn/life-mllm/users/cxr/zhy/miniconda3/envs/lm-evaluation-harness/lib/python3.11/site-packages/torch/distributed/elastic/agent/server/api.py", line 717, in run
    result = self._invoke_run(role)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/bn/life-mllm/users/cxr/zhy/miniconda3/envs/lm-evaluation-harness/lib/python3.11/site-packages/torch/distributed/elastic/agent/server/api.py", line 881, in _invoke_run
    time.sleep(monitor_interval)
  File "/mnt/bn/life-mllm/users/cxr/zhy/miniconda3/envs/lm-evaluation-harness/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 85, in _terminate_process_handler
    raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
torch.distributed.elastic.multiprocessing.api.SignalException: Process 2176566 got signal: 15
警告：任务 gsm8k 执行失败！

==================================================
开始评估：任务=boolq | 少样本数=0 | 模型=Qwen2.5-7B-quantization-fg
输出路径：results2/Qwen2.5-7B-quantization-fg/fg4/boolq.json
==================================================
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-12-09:14:44:25 INFO     [__main__:440] Selected Tasks: ['boolq']
2025-12-09:14:44:25 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:14:44:25 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-09:14:44:25 INFO     [__main__:440] Selected Tasks: ['boolq']
2025-12-09:14:44:25 INFO     [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-12-09:14:44:25 INFO     [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg'}
2025-12-09:14:44:26 WARNING  [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:14:44:26 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]The tokenizer you are loading from '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen2.5-7B-quantization-fg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
2025-12-09:14:44:27 INFO     [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.37it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:02,  1.42it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:03<00:03,  1.66s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:03<00:03,  1.86s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.34s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:04<00:01,  1.45s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:04<00:00,  1.03it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:04<00:00,  1.11it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:04<00:00,  1.06s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:04<00:00,  1.14s/it]

2025-12-09:14:44:31 WARNING  [api.task:844] [Task: boolq] metric acc is defined, but aggregation is not. using default aggregation=mean
2025-12-09:14:44:31 WARNING  [api.task:844] [Task: boolq] metric acc is defined, but aggregation is not. using default aggregation=mean
2025-12-09:14:44:31 WARNING  [api.task:856] [Task: boolq] metric acc is defined, but higher_is_better is not. using default higher_is_better=True
2025-12-09:14:44:31 WARNING  [api.task:856] [Task: boolq] metric acc is defined, but higher_is_better is not. using default higher_is_better=True
2025-12-09:14:44:54 WARNING  [evaluator:309] Overwriting default num_fewshot of boolq from None to 0
2025-12-09:14:44:54 INFO     [api.task:434] Building contexts for boolq on rank 1...

  0%|          | 0/1635 [00:00<?, ?it/s]
 11%|█▏        | 188/1635 [00:00<00:00, 1879.61it/s]
 23%|██▎       | 376/1635 [00:00<00:00, 1748.43it/s]
 37%|███▋      | 602/1635 [00:00<00:00, 1969.01it/s]
 49%|████▉     | 801/1635 [00:00<00:00, 1950.24it/s]
 64%|██████▍   | 1046/1635 [00:00<00:00, 2123.65it/s]
 77%|███████▋  | 1260/1635 [00:00<00:00, 2058.99it/s]
 94%|█████████▍| 1541/1635 [00:00<00:00, 2295.10it/s]
100%|██████████| 1635/1635 [00:00<00:00, 2148.84it/s]
2025-12-09:14:44:56 WARNING  [evaluator:309] Overwriting default num_fewshot of boolq from None to 0
2025-12-09:14:44:56 INFO     [api.task:434] Building contexts for boolq on rank 0...

  0%|          | 0/1635 [00:00<?, ?it/s]
 16%|█▌        | 263/1635 [00:00<00:00, 2620.57it/s]
 33%|███▎      | 535/1635 [00:00<00:00, 2673.35it/s]
 49%|████▉     | 803/1635 [00:00<00:00, 2602.47it/s]
 65%|██████▌   | 1064/1635 [00:00<00:00, 2581.58it/s]
 81%|████████  | 1323/1635 [00:00<00:00, 2583.41it/s]
 97%|█████████▋| 1582/1635 [00:00<00:00, 2581.28it/s]
100%|██████████| 1635/1635 [00:00<00:00, 2594.13it/s]
n136-128-154:2192560:2192560 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2192560:2192560 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2192560:2192560 [0] NCCL INFO cudaDriverVersion 12040
n136-128-154:2192560:2192560 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2192560:2192560 [0] NCCL INFO Comm config Blocking set to 1
n136-128-154:2192561:2192561 [1] NCCL INFO cudaDriverVersion 12040
n136-128-154:2192561:2192561 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2192561:2192561 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2192561:2192561 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
n136-128-154:2192561:2192561 [1] NCCL INFO Comm config Blocking set to 1
n136-128-154:2192560:2193418 [0] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2192560:2193418 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2192561:2193419 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. 
n136-128-154:2192561:2193419 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
n136-128-154:2192560:2193418 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2192560:2193418 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2192560:2193418 [0] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2192561:2193419 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
n136-128-154:2192561:2193419 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
n136-128-154:2192561:2193419 [1] NCCL INFO NCCL_IB_HCA set to mlx5
n136-128-154:2192560:2193418 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2192560:2193418 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2192560:2193418 [0] NCCL INFO Initialized NET plugin IB
n136-128-154:2192561:2193419 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
n136-128-154:2192561:2193419 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0>
n136-128-154:2192561:2193419 [1] NCCL INFO Initialized NET plugin IB
n136-128-154:2192560:2193418 [0] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2192560:2193418 [0] NCCL INFO Using network IB
n136-128-154:2192561:2193419 [1] NCCL INFO Assigned NET plugin IB to comm
n136-128-154:2192561:2193419 [1] NCCL INFO Using network IB
n136-128-154:2192560:2193418 [0] NCCL INFO ncclCommInitRankConfig comm 0xfa092e0 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0x2a690c9d1b8a03ae - Init START
n136-128-154:2192561:2193419 [1] NCCL INFO ncclCommInitRankConfig comm 0xf94e740 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0x2a690c9d1b8a03ae - Init START
n136-128-154:2192560:2193418 [0] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2192561:2193419 [1] NCCL INFO RAS client listening socket at ::1<28028>
n136-128-154:2192560:2193418 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2192560:2193418 [0] NCCL INFO Retrieving state for IB
n136-128-154:2192560:2193418 [0] NCCL INFO Initialized state 0 for IB
n136-128-154:2192560:2193418 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2192561:2193419 [1] NCCL INFO TOPO/NET : Importing network plugins to topology
n136-128-154:2192561:2193419 [1] NCCL INFO Retrieving state for IB
n136-128-154:2192561:2193419 [1] NCCL INFO Initialized state 0 for IB
n136-128-154:2192561:2193419 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null)
n136-128-154:2192560:2193418 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2192561:2193419 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null)
n136-128-154:2192560:2193418 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2192561:2193419 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null)
n136-128-154:2192560:2193418 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2192561:2193419 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null)
n136-128-154:2192560:2193418 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2192560:2193418 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2192561:2193419 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2192561:2193419 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default
n136-128-154:2192560:2193418 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2192560:2193418 [0] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2192560:2193418 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2192560:2193418 [0] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2192560:2193418 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2192560:2193418 [0] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2192560:2193418 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2192560:2193418 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2192560:2193418 [0] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2192560:2193418 [0] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2192560:2193418 [0] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2192560:2193418 [0] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2192560:2193418 [0] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2192560:2193418 [0] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2192560:2193418 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2192560:2193418 [0] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2192560:2193418 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2192560:2193418 [0] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2192560:2193418 [0] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2192560:2193418 [0] NCCL INFO ==========================================
n136-128-154:2192560:2193418 [0] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2192560:2193418 [0] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2192560:2193418 [0] NCCL INFO Setting affinity for GPU 6 to 33-62,97-126
n136-128-154:2192560:2193418 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2192560:2193418 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2192560:2193418 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2192560:2193418 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2192560:2193418 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2192560:2193418 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2192560:2193418 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2192560:2193418 [0] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2192560:2193418 [0] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2192560:2193418 [0] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2192560:2193418 [0] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2192560:2193418 [0] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2192560:2193418 [0] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2192560:2193418 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2192560:2193418 [0] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2192560:2193418 [0] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2192560:2193418 [0] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2192560:2193418 [0] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2192560:2193418 [0] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2192560:2193418 [0] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2192560:2193418 [0] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2192560:2193418 [0] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2192560:2193418 [0] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2192560:2193418 [0] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2192560:2193418 [0] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2192560:2193418 [0] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2192561:2193419 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 ===
n136-128-154:2192561:2193419 [1] NCCL INFO CPU/0-1 (1/1/2)
n136-128-154:2192561:2193419 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff)
n136-128-154:2192561:2193419 [1] NCCL INFO               + PCI[24.0] - NIC/0-95000
n136-128-154:2192561:2193419 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff)
n136-128-154:2192561:2193419 [1] NCCL INFO               + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8)
n136-128-154:2192561:2193419 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c5000 (0)
n136-128-154:2192561:2193419 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2192561:2193419 [1] NCCL INFO               + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8)
n136-128-154:2192561:2193419 [1] NCCL INFO                             + PCI[24.0] - GPU/0-c9000 (1)
n136-128-154:2192561:2193419 [1] NCCL INFO                                           + NVL[240.0] - NVS/0-0
n136-128-154:2192561:2193419 [1] NCCL INFO               + PCI[24.0] - NIC/0-cc000
n136-128-154:2192561:2193419 [1] NCCL INFO + SYS[10.0] - CPU/0-0
n136-128-154:2192561:2193419 [1] NCCL INFO CPU/0-0 (1/1/2)
n136-128-154:2192561:2193419 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff)
n136-128-154:2192561:2193419 [1] NCCL INFO               + PCI[24.0] - NIC/0-1d000
n136-128-154:2192561:2193419 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff)
n136-128-154:2192561:2193419 [1] NCCL INFO               + PCI[24.0] - NIC/0-60000
n136-128-154:2192561:2193419 [1] NCCL INFO + SYS[10.0] - CPU/0-1
n136-128-154:2192561:2193419 [1] NCCL INFO ==========================================
n136-128-154:2192561:2193419 [1] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2192561:2193419 [1] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) 
n136-128-154:2192561:2193419 [1] NCCL INFO Setting affinity for GPU 7 to 33-62,97-126
n136-128-154:2192561:2193419 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1
n136-128-154:2192561:2193419 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2192561:2193419 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2192561:2193419 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2192561:2193419 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2192561:2193419 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2192561:2193419 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2192561:2193419 [1] NCCL INFO  6 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2192561:2193419 [1] NCCL INFO  7 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2192561:2193419 [1] NCCL INFO  8 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2192561:2193419 [1] NCCL INFO  9 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2192561:2193419 [1] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2192561:2193419 [1] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2192561:2193419 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0
n136-128-154:2192561:2193419 [1] NCCL INFO  0 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2192561:2193419 [1] NCCL INFO  1 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2192561:2193419 [1] NCCL INFO  2 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2192561:2193419 [1] NCCL INFO  3 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2192561:2193419 [1] NCCL INFO  4 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2192561:2193419 [1] NCCL INFO  5 : GPU/0-c5000 GPU/0-c9000
n136-128-154:2192561:2193419 [1] NCCL INFO  6 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2192561:2193419 [1] NCCL INFO  7 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2192561:2193419 [1] NCCL INFO  8 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2192561:2193419 [1] NCCL INFO  9 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2192561:2193419 [1] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2192561:2193419 [1] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000
n136-128-154:2192560:2193418 [0] NCCL INFO comm 0xfa092e0 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
n136-128-154:2192561:2193419 [1] NCCL INFO comm 0xf94e740 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
n136-128-154:2192560:2193418 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
n136-128-154:2192560:2193418 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
n136-128-154:2192561:2193419 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
n136-128-154:2192560:2193418 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
n136-128-154:2192561:2193419 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1
n136-128-154:2192560:2193418 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
n136-128-154:2192561:2193419 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1
n136-128-154:2192560:2193418 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
n136-128-154:2192561:2193419 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1
n136-128-154:2192560:2193418 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
n136-128-154:2192561:2193419 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1
n136-128-154:2192560:2193418 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
n136-128-154:2192561:2193419 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1
n136-128-154:2192560:2193418 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
n136-128-154:2192561:2193419 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1
n136-128-154:2192560:2193418 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
n136-128-154:2192561:2193419 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1
n136-128-154:2192560:2193418 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
n136-128-154:2192561:2193419 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1
n136-128-154:2192560:2193418 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
n136-128-154:2192561:2193419 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1
n136-128-154:2192560:2193418 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
n136-128-154:2192561:2193419 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1
n136-128-154:2192560:2193418 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1
n136-128-154:2192561:2193419 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1
n136-128-154:2192560:2193418 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1
n136-128-154:2192561:2193419 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1
n136-128-154:2192560:2193418 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1
n136-128-154:2192561:2193419 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1
n136-128-154:2192560:2193418 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1
n136-128-154:2192561:2193419 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1
n136-128-154:2192560:2193418 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1
n136-128-154:2192561:2193419 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1
n136-128-154:2192560:2193418 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1
n136-128-154:2192561:2193419 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1
n136-128-154:2192560:2193418 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1
n136-128-154:2192561:2193419 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1
n136-128-154:2192560:2193418 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1
n136-128-154:2192561:2193419 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1
n136-128-154:2192560:2193418 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1
n136-128-154:2192561:2193419 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1
n136-128-154:2192560:2193418 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1
n136-128-154:2192561:2193419 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1
n136-128-154:2192560:2193418 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1
n136-128-154:2192561:2193419 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1
n136-128-154:2192560:2193418 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1
n136-128-154:2192561:2193419 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1
n136-128-154:2192561:2193419 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1
n136-128-154:2192560:2193418 [0] NCCL INFO Channel 00/24 : 0 1
n136-128-154:2192560:2193418 [0] NCCL INFO Channel 01/24 : 0 1
n136-128-154:2192560:2193418 [0] NCCL INFO Channel 02/24 : 0 1
n136-128-154:2192560:2193418 [0] NCCL INFO Channel 03/24 : 0 1
n136-128-154:2192561:2193419 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0
n136-128-154:2192560:2193418 [0] NCCL INFO Channel 04/24 : 0 1
n136-128-154:2192561:2193419 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0
n136-128-154:2192560:2193418 [0] NCCL INFO Channel 05/24 : 0 1
n136-128-154:2192561:2193419 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0
n136-128-154:2192560:2193418 [0] NCCL INFO Channel 06/24 : 0 1
n136-128-154:2192561:2193419 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0
n136-128-154:2192560:2193418 [0] NCCL INFO Channel 07/24 : 0 1
n136-128-154:2192561:2193419 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0
n136-128-154:2192560:2193418 [0] NCCL INFO Channel 08/24 : 0 1
n136-128-154:2192561:2193419 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0
n136-128-154:2192560:2193418 [0] NCCL INFO Channel 09/24 : 0 1
n136-128-154:2192561:2193419 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0
n136-128-154:2192560:2193418 [0] NCCL INFO Channel 10/24 : 0 1
n136-128-154:2192561:2193419 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0
n136-128-154:2192560:2193418 [0] NCCL INFO Channel 11/24 : 0 1
n136-128-154:2192561:2193419 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0
n136-128-154:2192560:2193418 [0] NCCL INFO Channel 12/24 : 0 1
n136-128-154:2192561:2193419 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0
n136-128-154:2192560:2193418 [0] NCCL INFO Channel 13/24 : 0 1
n136-128-154:2192561:2193419 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0
n136-128-154:2192560:2193418 [0] NCCL INFO Channel 14/24 : 0 1
n136-128-154:2192561:2193419 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0
n136-128-154:2192560:2193418 [0] NCCL INFO Channel 15/24 : 0 1
n136-128-154:2192561:2193419 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0
n136-128-154:2192560:2193418 [0] NCCL INFO Channel 16/24 : 0 1
n136-128-154:2192561:2193419 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0
n136-128-154:2192560:2193418 [0] NCCL INFO Channel 17/24 : 0 1
n136-128-154:2192561:2193419 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0
n136-128-154:2192560:2193418 [0] NCCL INFO Channel 18/24 : 0 1
n136-128-154:2192561:2193419 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0
n136-128-154:2192560:2193418 [0] NCCL INFO Channel 19/24 : 0 1
n136-128-154:2192561:2193419 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0
n136-128-154:2192560:2193418 [0] NCCL INFO Channel 20/24 : 0 1
n136-128-154:2192561:2193419 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0
n136-128-154:2192560:2193418 [0] NCCL INFO Channel 21/24 : 0 1
n136-128-154:2192561:2193419 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0
n136-128-154:2192560:2193418 [0] NCCL INFO Channel 22/24 : 0 1
n136-128-154:2192561:2193419 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0
n136-128-154:2192560:2193418 [0] NCCL INFO Channel 23/24 : 0 1
n136-128-154:2192561:2193419 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0
n136-128-154:2192560:2193418 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
n136-128-154:2192561:2193419 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0
n136-128-154:2192560:2193418 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
n136-128-154:2192561:2193419 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0
n136-128-154:2192560:2193418 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1
n136-128-154:2192561:2193419 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0
n136-128-154:2192560:2193418 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1
n136-128-154:2192561:2193419 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1
n136-128-154:2192560:2193418 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1
n136-128-154:2192560:2193418 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1
n136-128-154:2192561:2193419 [1] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2192560:2193418 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1
n136-128-154:2192560:2193418 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1
n136-128-154:2192560:2193418 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1
n136-128-154:2192560:2193418 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1
n136-128-154:2192560:2193418 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1
n136-128-154:2192560:2193418 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1
n136-128-154:2192560:2193418 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1
n136-128-154:2192560:2193418 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1
n136-128-154:2192560:2193418 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1
n136-128-154:2192560:2193418 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1
n136-128-154:2192560:2193418 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1
n136-128-154:2192560:2193418 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1
n136-128-154:2192560:2193418 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1
n136-128-154:2192560:2193418 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1
n136-128-154:2192560:2193418 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1
n136-128-154:2192560:2193418 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1
n136-128-154:2192560:2193418 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1
n136-128-154:2192560:2193418 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1
n136-128-154:2192560:2193418 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1
n136-128-154:2192560:2193418 [0] NCCL INFO P2P Chunksize set to 524288
n136-128-154:2192560:2193418 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2192560:2193418 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
n136-128-154:2192560:2193491 [0] NCCL INFO [Proxy Service] Device 0 CPU core 33
n136-128-154:2192560:2193492 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 99
n136-128-154:2192561:2193419 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
n136-128-154:2192561:2193493 [1] NCCL INFO [Proxy Service] Device 1 CPU core 60
n136-128-154:2192561:2193494 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 61
n136-128-154:2192561:2193419 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2192561:2193419 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2192560:2193418 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
n136-128-154:2192560:2193418 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
n136-128-154:2192560:2193418 [0] NCCL INFO CC Off, workFifoBytes 1048576
n136-128-154:2192561:2193419 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2192561:2193419 [1] NCCL INFO ncclCommInitRankConfig comm 0xf94e740 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0x2a690c9d1b8a03ae - Init COMPLETE
n136-128-154:2192561:2193419 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.76 (kernels 0.21, alloc 0.30, bootstrap 0.00, allgathers 0.04, topo 0.15, graphs 0.00, connections 0.01, rest 0.05)
n136-128-154:2192560:2193418 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
n136-128-154:2192560:2193418 [0] NCCL INFO ncclCommInitRankConfig comm 0xfa092e0 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0x2a690c9d1b8a03ae - Init COMPLETE
n136-128-154:2192560:2193418 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.76 (kernels 0.22, alloc 0.30, bootstrap 0.00, allgathers 0.04, topo 0.15, graphs 0.00, connections 0.02, rest 0.04)
n136-128-154:2192561:2193495 [1] NCCL INFO Channel 00/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2192561:2193495 [1] NCCL INFO Channel 01/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2192561:2193495 [1] NCCL INFO Channel 02/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2192561:2193495 [1] NCCL INFO Channel 03/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2192561:2193495 [1] NCCL INFO Channel 04/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2192561:2193495 [1] NCCL INFO Channel 05/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2192561:2193495 [1] NCCL INFO Channel 06/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2192560:2193496 [0] NCCL INFO Channel 00/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2192561:2193495 [1] NCCL INFO Channel 07/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2192560:2193496 [0] NCCL INFO Channel 01/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2192561:2193495 [1] NCCL INFO Channel 08/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2192560:2193496 [0] NCCL INFO Channel 02/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2192561:2193495 [1] NCCL INFO Channel 09/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2192560:2193496 [0] NCCL INFO Channel 03/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2192561:2193495 [1] NCCL INFO Channel 10/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2192560:2193496 [0] NCCL INFO Channel 04/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2192561:2193495 [1] NCCL INFO Channel 11/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2192560:2193496 [0] NCCL INFO Channel 05/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2192561:2193495 [1] NCCL INFO Channel 12/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2192560:2193496 [0] NCCL INFO Channel 06/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2192561:2193495 [1] NCCL INFO Channel 13/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2192560:2193496 [0] NCCL INFO Channel 07/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2192561:2193495 [1] NCCL INFO Channel 14/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2192560:2193496 [0] NCCL INFO Channel 08/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2192561:2193495 [1] NCCL INFO Channel 15/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2192560:2193496 [0] NCCL INFO Channel 09/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2192561:2193495 [1] NCCL INFO Channel 16/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2192560:2193496 [0] NCCL INFO Channel 10/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2192561:2193495 [1] NCCL INFO Channel 17/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2192561:2193495 [1] NCCL INFO Channel 18/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2192560:2193496 [0] NCCL INFO Channel 11/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2192561:2193495 [1] NCCL INFO Channel 19/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2192560:2193496 [0] NCCL INFO Channel 12/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2192561:2193495 [1] NCCL INFO Channel 20/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2192560:2193496 [0] NCCL INFO Channel 13/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2192561:2193495 [1] NCCL INFO Channel 21/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2192560:2193496 [0] NCCL INFO Channel 14/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2192561:2193495 [1] NCCL INFO Channel 22/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2192560:2193496 [0] NCCL INFO Channel 15/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2192561:2193495 [1] NCCL INFO Channel 23/0 : 1[7] -> 0[6] via P2P/CUMEM/read
n136-128-154:2192560:2193496 [0] NCCL INFO Channel 16/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2192560:2193496 [0] NCCL INFO Channel 17/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2192560:2193496 [0] NCCL INFO Channel 18/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2192560:2193496 [0] NCCL INFO Channel 19/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2192560:2193496 [0] NCCL INFO Channel 20/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2192560:2193496 [0] NCCL INFO Channel 21/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2192560:2193496 [0] NCCL INFO Channel 22/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2192560:2193496 [0] NCCL INFO Channel 23/0 : 0[6] -> 1[7] via P2P/CUMEM/read
n136-128-154:2192560:2193496 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
n136-128-154:2192561:2193495 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
2025-12-09:14:44:58 INFO     [evaluator:559] Running loglikelihood requests
2025-12-09:14:44:58 INFO     [evaluator:559] Running loglikelihood requests
Passed argument batch_size = auto:1. Detecting largest batch size

Running loglikelihood requests:   0%|          | 0/3270 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
W1209 14:45:01.416000 2192335 site-packages/torch/distributed/elastic/agent/server/api.py:725] Received 15 death signal, shutting down workers
W1209 14:45:01.417000 2192335 site-packages/torch/distributed/elastic/multiprocessing/api.py:908] Sending process 2192560 closing signal SIGTERM
W1209 14:45:01.417000 2192335 site-packages/torch/distributed/elastic/multiprocessing/api.py:908] Sending process 2192561 closing signal SIGTERM
Traceback (most recent call last):
  File "/mnt/bn/life-mllm/users/cxr/zhy/miniconda3/envs/lm-evaluation-harness/bin/accelerate", line 7, in <module>
    sys.exit(main())
             ^^^^^^
  File "/mnt/bn/life-mllm/users/cxr/zhy/miniconda3/envs/lm-evaluation-harness/lib/python3.11/site-packages/accelerate/commands/accelerate_cli.py", line 50, in main
    args.func(args)
  File "/mnt/bn/life-mllm/users/cxr/zhy/miniconda3/envs/lm-evaluation-harness/lib/python3.11/site-packages/accelerate/commands/launch.py", line 1272, in launch_command
    multi_gpu_launcher(args)
  File "/mnt/bn/life-mllm/users/cxr/zhy/miniconda3/envs/lm-evaluation-harness/lib/python3.11/site-packages/accelerate/commands/launch.py", line 899, in multi_gpu_launcher
    distrib_run.run(args)
  File "/mnt/bn/life-mllm/users/cxr/zhy/miniconda3/envs/lm-evaluation-harness/lib/python3.11/site-packages/torch/distributed/run.py", line 927, in run
    elastic_launch(
  File "/mnt/bn/life-mllm/users/cxr/zhy/miniconda3/envs/lm-evaluation-harness/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 156, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/bn/life-mllm/users/cxr/zhy/miniconda3/envs/lm-evaluation-harness/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 284, in launch_agent
    result = agent.run()
             ^^^^^^^^^^^
  File "/mnt/bn/life-mllm/users/cxr/zhy/miniconda3/envs/lm-evaluation-harness/lib/python3.11/site-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
    result = f(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^
  File "/mnt/bn/life-mllm/users/cxr/zhy/miniconda3/envs/lm-evaluation-harness/lib/python3.11/site-packages/torch/distributed/elastic/agent/server/api.py", line 717, in run
    result = self._invoke_run(role)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/bn/life-mllm/users/cxr/zhy/miniconda3/envs/lm-evaluation-harness/lib/python3.11/site-packages/torch/distributed/elastic/agent/server/api.py", line 881, in _invoke_run
    time.sleep(monitor_interval)
  File "/mnt/bn/life-mllm/users/cxr/zhy/miniconda3/envs/lm-evaluation-harness/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 85, in _terminate_process_handler
    raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
torch.distributed.elastic.multiprocessing.api.SignalException: Process 2192335 got signal: 15