Instructions to use invocation02/test_9_resid_post with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use invocation02/test_9_resid_post with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="invocation02/test_9_resid_post")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("invocation02/test_9_resid_post")
model = AutoModelForCausalLM.from_pretrained("invocation02/test_9_resid_post")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use invocation02/test_9_resid_post with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "invocation02/test_9_resid_post"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "invocation02/test_9_resid_post",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/invocation02/test_9_resid_post

SGLang

How to use invocation02/test_9_resid_post with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "invocation02/test_9_resid_post" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "invocation02/test_9_resid_post",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "invocation02/test_9_resid_post" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "invocation02/test_9_resid_post",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use invocation02/test_9_resid_post with Docker Model Runner:
```
docker model run hf.co/invocation02/test_9_resid_post
```

Upload folder using huggingface_hub

by invocation02 - opened May 15, 2024

base: refs/heads/main

←

from: refs/pr/3

Discussion Files changed

-0

invocation02

Owner May 15, 2024

•

edited May 15, 2024

Upload folder using huggingface_hub

Multi commit ID: 5558fe3ac1798ae99cbd0c850f2e8a9a5fe4966cac8141863c02e1b28ab553f7

Scheduled commits:

Upload 4 file(s) totalling 24.9K (3fd1a3f4e6b3b92e1d3e3946a8082ba267b2c072901eb146d8a7a9249b02335c)

This is a PR opened using the huggingface_hub library in the context of a multi-commit. PR can be commented as a usual PR. However, please be aware that manually updating the PR description, changing the PR status, or pushing new commits, is not recommended as it might corrupt the commit process. Learn more about multi-commits in this guide.

invocation02

Owner May 15, 2024

Multi-commit is now completed! You can ping the repo owner to review the changes. This PR can now be commented or modified without risking to corrupt it.

This is a comment posted using the huggingface_hub library in the context of a multi-commit. Learn more about multi-commits in this guide.

invocation02 changed pull request status to open May 15, 2024

invocation02 changed pull request title from [WIP] Upload folder using huggingface_hub (multi-commit 5558fe3ac1798ae99cbd0c850f2e8a9a5fe4966cac8141863c02e1b28ab553f7) to Upload folder using huggingface_hub May 15, 2024

3fd1a3f4e6b3b92e1d3e3946a8082ba267b2c072901eb146d8a7a9249b02335c6be4ce1f

invocation02

Owner May 15, 2024

Cannot merge Pull Requests as no changes are associated. This PR will be closed automatically.

This is a comment posted using the huggingface_hub library in the context of a multi-commit. Learn more about multi-commits in this guide.

invocation02 changed pull request status to closed May 15, 2024

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment