Example code from README output is nonsense
Running the example code in README generate nonsense like this:
the a- \n in. the. and,,..,\n to, a the a, the. and - to.\n, in to and\n,, the a\n\n,\n a the to and - the-., ,,
Tried several prompts, but all gives the same nonsense.
Any idea?
The example code:
from transformers import AutoTokenizer, AutoModelForCausalLM
import transformers
import torch
model = "tiiuae/falcon-40b"
tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline("text-generation", model=model, tokenizer=tokenizer, torch_dtype=torch.bfloat16, trust_remote_code=True, device_map="auto",)
sequences = pipeline("Anything here ...", max_length=200, do_sample=True, top_k=10, num_return_sequences=1, eos_token_id=tokenizer.eos_token_id,)
for seq in sequences:
print(f"Result: {seq['generated_text']}")
Running on 4 GPUs (Quadro RTX 8000)
conda env:
python 3.11.3
transformers 4.29.2 py311h06a4308_0
pytorch 2.0.1 py3.11_cuda11.8_cudnn8.7.0_0 pytorch
Problem solved.
Tried 4 X A100 GPU and the problem didn't happen.
It could be that Quadro-8000 does not support well BF16
Or, the model was corrupted when downloaded on the first GPU.