Instructions to use QuantTrio/GLM-5-AWQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use QuantTrio/GLM-5-AWQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="QuantTrio/GLM-5-AWQ")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("QuantTrio/GLM-5-AWQ")
model = AutoModelForCausalLM.from_pretrained("QuantTrio/GLM-5-AWQ")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use QuantTrio/GLM-5-AWQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "QuantTrio/GLM-5-AWQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantTrio/GLM-5-AWQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/QuantTrio/GLM-5-AWQ

SGLang

How to use QuantTrio/GLM-5-AWQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "QuantTrio/GLM-5-AWQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantTrio/GLM-5-AWQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "QuantTrio/GLM-5-AWQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantTrio/GLM-5-AWQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use QuantTrio/GLM-5-AWQ with Docker Model Runner:
```
docker model run hf.co/QuantTrio/GLM-5-AWQ
```

Great work

by JoeyHwong - opened Feb 28

Discussion

JoeyHwong

Feb 28

Thank you very much for your contributions. I’m reaching out to kindly ask if you could share the details of your inference environment.

I’m currently stuck on Sparse MLA / DSA compatibility issue when trying to run the model with vLLM, which is blocking my usage.

tclf90

QuantTrio org Feb 28

testing env:

System: Ubuntu24LTS 
Driver: 580.82.07
Graphics Card: 8xH200
Docker Image: Ubuntu22LTS Python3.12 Cuda12.8

install & run

python3.12 -m venv venv
source venv/bin/activate
pip install -U vllm --pre --index-url https://pypi.org/simple --extra-index-url https://wheels.vllm.ai/nightly --force-reinstall 
pip install git+https://github.com/huggingface/transformers.git
pip install git+https://github.com/deepseek-ai/DeepGEMM.git@v2.1.1.post3 --no-build-isolation

vllm serve ...

R-omk

Feb 28

•

edited Feb 28

@tclf90

Thank you for your work! I'd like to point out that not all files have been uploaded to the repository yet, almost a third of the volume is missing. I'd like to ask you to check and upload the missing files.

tclf90

QuantTrio org Feb 28

@tclf90

Thank you for your work! I'd like to point out that not all files have been uploaded to the repository yet, almost a third of the volume is missing. I'd like to ask you to check and upload the missing files.

@JunHowie HELP!! 👀

JunHowie

QuantTrio org Feb 28

@tclf90

Thank you for your work! I'd like to point out that not all files have been uploaded to the repository yet, almost a third of the volume is missing. I'd like to ask you to check and upload the missing files.

@JunHowie HELP!! 👀
My Bad

JunHowie

QuantTrio org Feb 28

@R-omk
Due to an unknown issue, the upload process showed as completed but resulted in missing weights. I have re-uploaded the files and manually verified their integrity. Please download the latest weights. If there are any further issues, please continue reporting them to our team.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment