Instructions to use Qwen/Qwen2.5-Coder-7B-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Qwen/Qwen2.5-Coder-7B-Instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Qwen/Qwen2.5-Coder-7B-Instruct")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use Qwen/Qwen2.5-Coder-7B-Instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Qwen/Qwen2.5-Coder-7B-Instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen2.5-Coder-7B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Qwen/Qwen2.5-Coder-7B-Instruct

SGLang

How to use Qwen/Qwen2.5-Coder-7B-Instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Qwen/Qwen2.5-Coder-7B-Instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen2.5-Coder-7B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Qwen/Qwen2.5-Coder-7B-Instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen2.5-Coder-7B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Qwen/Qwen2.5-Coder-7B-Instruct with Docker Model Runner:
```
docker model run hf.co/Qwen/Qwen2.5-Coder-7B-Instruct
```

Size mismatch

#10

by neighborwang - opened Oct 5, 2024

Discussion

neighborwang

Oct 5, 2024

•

edited Oct 10, 2024

The same error happens like this https://github.com/huggingface/autotrain-advanced/issues/487 when i'm trying to merge my adapter (finetuned LoRA model - https://huggingface.co/neighborwang/ModeliCo-7B) into the base model (Qwen2.5-Coder-7B-Instruct ).

RuntimeError: Error(s) in loading state_dict for PeftModelForCausalLM:
size mismatch for base_model.model.model.embed_tokens.weight: 
copying a param with shape torch.Size([151665, 3584]) from checkpoint, 
the shape in current model is torch.Size([152064, 3584]).

I faced the same issue with Llama 3.1 but i solved it use specific transformers version, so I tried for my adapter and Qwen2.5-Coder-7B-Instruct the following transformers versions:

v4.45.1
v4.45.0
v4.44.0
v4.43.0
v4.37.0

But nothing works... I need some help. Also in the GitHub issue I mentioned above, other people are also facing this issue.

Thanks a lot in advance!

jklj077

Qwen org Oct 8, 2024

Hi, it appeared the the embed_tokens and the lm_head had different shapes from the base model. Please try padding the tensors from the adapter model manually or truncating the tensors from the base model.

FYI: the size of the vocabulary (151665) is different from the size of the embed_tokens and the lm_head (depending on the size of the model, for 7B, it is 152064, the vocab_size in the config.json). Normally, the size from the config.json is used.

neighborwang

Oct 10, 2024

•

edited Oct 10, 2024

Hi Xuancheng, thanks a lot for your answer!

I used AutoTrain, I think with that the parameters are supposed to be matched automatically. Is this a problem with AutoTrain? Or what was the reason of this mismatch?

Thx!

abhishek

Oct 11, 2024

when using the merge_adapter param to true in AutoTrain, the model seems to be merging fine.

neighborwang

Oct 11, 2024

Ok, I will try, thank you very much Abhishek!

neighborwang changed discussion status to closed Oct 11, 2024

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment