Text Generation
Transformers
Safetensors
falcon
conversational
custom_code
text-generation-inference
Instructions to use tiiuae/falcon-11B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use tiiuae/falcon-11B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="tiiuae/falcon-11B", trust_remote_code=True) messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("tiiuae/falcon-11B", trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained("tiiuae/falcon-11B", trust_remote_code=True) messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=40) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps
- vLLM
How to use tiiuae/falcon-11B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "tiiuae/falcon-11B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "tiiuae/falcon-11B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/tiiuae/falcon-11B
- SGLang
How to use tiiuae/falcon-11B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "tiiuae/falcon-11B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "tiiuae/falcon-11B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "tiiuae/falcon-11B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "tiiuae/falcon-11B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use tiiuae/falcon-11B with Docker Model Runner:
docker model run hf.co/tiiuae/falcon-11B
no idea what went wrong
#12
by eightzerofoursix - opened
i was using falcon2 via command line, and noticed some odd behaviour, some strange answers. eg when i asked it about itself, it said "I'm currently running as>>ABSTRACT<< =>>ABSTRACT<</Users"
i decided to give it a one-script python app i made to see how it would debug/critique it, and it sorta broke, in a similar style
output varies, but all looks a bit like this:
#>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<">>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT#>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRAC<<">>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<t
>>ABSTRACT<<>>ABSTRACT<<t t
>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<< m 0
0>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<t
>>ABSTRACT<<>>ABSTRACT<<t;>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT>>ABSTRACT<<>>ABSTRACT<<t;>>ABSTRACT<<>>ABSTRCT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<what>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTR<>>ABSTRACT<<what>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<">>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRCT<<">>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<, #
>>ABSTRACT<<>>ABSTRACT<<t (>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<
>>ABSTRACT<<t
>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<t[>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<t[>>ABSTRACT<<>>ABSTRAT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<< &
>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<< >>ABSTRACT<< "t' 0 0
0 0 0 0 0 " 0 0 0 0 0 " 0x 0 , 0x 0x (v** >>ABSTRACT<<s [x": >>ABSTRACT<<
>>ABSTRACT<< >>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<< >>ABSTRACT<< 0x "
0x0x 0x 0x 0
* " >>ABSTRACT<< & * & " " " ,What " " , " " , " " , 3 , >>ABSTRACT<< & ,
>>ABSTRACT<<>>ABSTRACT<< >>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<< >>ABSTRACT<< . ,
>>ABSTRACT<< , , , , , , ,
0 0 " 2 " " " " " " " >>ABSTRACT<< s
0 ' , " * " 3 v 3 "x [d" "d"d d " " " " " " " . >>ABSTRACT<< " 2 0 0 0 0 " ( " " "
" " " 3 5 d
d " , 3 0 0 0 0
0x0x>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<
[
any idea why?
did i do that?
Same here, I noticed the issue when the prompt is a bit long (for example with few-shot learning).
>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<0>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<< , >>ABSTRACT<< 1>>ABSTRACT<< 1>>ABSTRACT<<1 0 0>>ABSTRACT<<1>>ABSTRACT<<1>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<111 & , , 1 , 1 , , 11, , 0 1 1 1 1 0 1 1 1 000>>ABSTRACT<< 1111 1 3 \n , 0 111 , , 1 1 0 0 0 , \n 0 >>ABSTRACT<<1110 0 1 \nM , 1111000>>ABSTRACT<<1011 0000011110 0 1 1111111111111110>>ABSTRACT<<111100110001111000000,00110000011110001111111100 1111000000010011010001111111110000110000111000000001111000000000000000111100,1110000000000000000011111110000000000001100000000011110 0000201111111111100010000111100000000000,000000000001111000001011000.\n\nA>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<,\nA.\nA.\n>>ABSTRACT<<>>ABSTRACT<<.\nA.\nA.\nA>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<.\nA.\nA.\nA.\nA.\nA.\n>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<.\nA.\nA.>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<.\nA>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<.\nA.\nA.\nA.\n>>ABSTRACT<<>>ABSTRACT<<.\n>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<.\nA.>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<.\nA>>ABSTRACT<<>>ABSTRACT<<.\nA.>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<.\nA.\nA.>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<.\nA.\n>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<.\nA.\nA.\nA>>ABSTRACT<<>>ABSTRACT<<.\nA.\n>>ABSTRACT<<>>ABSTRACT<<.\nA.\n>>ABSTRACT<<>>ABSTRACT<<.\nA.\nA.\nA.>>ABSTRACT<<.\nA.\nA.\nA.\n>>ABSTRACT<<.\nA.\nA.>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<.\nA.>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<.\nA.>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<.\nA.\nA.\nA.\nA.\n>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<\nA.\n>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<\nA.>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<\nA.\n>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<\nA.\nA>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<\nA.\nA.\n>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<0s0t>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<0e0>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<, 0c>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<\n>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<\nT >>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<t3d>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<\n0t>>ABSTRACT<<>>ABSTRACT<<0t0t0t0t0t>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<0s 0s>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<0t 0t 0t >>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<0s0>>ABSTRACT<<>>ABSTRACT<<0s0s0>>ABSTRACT<<,0s0s0s0s0s0s0s0s0s0s0s>>ABSTRACT<<>>ABSTRACT<<>>ABSTRACT<<e0s0s0s0s0s0s0s0s>>ABSTRACT<<0s0s0s0s0s>>ABSTRACT<<>>ABSTRACT<<0s0s0s0s0s0s>>ABSTRACT<<,\n2, 0>>ABSTRACT<<0s0s0s0s0s0s0s0s>>ABSTRACT<<>>ABSTRACT<<0s0s0s0s0s0s0s0s0>>ABSTRACT<<>>ABSTRACT<<0s0s0s0s0s0s0s0d0e0s0e0m0e00e0e0e0e3e0e0e0e0e0e2e0e0e0e0e00000003000000000000000000000000030003000003003303000003000 0 3s0400000000000000030000003000 00s0e0s00s0s0s0s0 0s00s00000333 (0s0s0s00m004d00,0000000000000000000000000000000