Instructions to use metacortex-ai/metacortex-models with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use metacortex-ai/metacortex-models with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="metacortex-ai/metacortex-models",
	filename="Qwen3.5-27B-Q4_K_M.gguf",
)

output = llm(
	"Once upon a time,",
	max_tokens=512,
	echo=True
)
print(output)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use metacortex-ai/metacortex-models with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf metacortex-ai/metacortex-models:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf metacortex-ai/metacortex-models:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf metacortex-ai/metacortex-models:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf metacortex-ai/metacortex-models:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf metacortex-ai/metacortex-models:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf metacortex-ai/metacortex-models:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf metacortex-ai/metacortex-models:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf metacortex-ai/metacortex-models:Q4_K_M

Use Docker

docker model run hf.co/metacortex-ai/metacortex-models:Q4_K_M

LM Studio
Jan
Ollama
How to use metacortex-ai/metacortex-models with Ollama:
```
ollama run hf.co/metacortex-ai/metacortex-models:Q4_K_M
```

Unsloth Studio new

How to use metacortex-ai/metacortex-models with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for metacortex-ai/metacortex-models to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for metacortex-ai/metacortex-models to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for metacortex-ai/metacortex-models to start chatting

Docker Model Runner
How to use metacortex-ai/metacortex-models with Docker Model Runner:
```
docker model run hf.co/metacortex-ai/metacortex-models:Q4_K_M
```

Lemonade

How to use metacortex-ai/metacortex-models with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull metacortex-ai/metacortex-models:Q4_K_M

Run and chat with the model

lemonade run user.metacortex-models-Q4_K_M

List all available models

lemonade list

metacortex-models / README.md

metacortex-ai

Upload README.md with huggingface_hub

9986e00 verified 28 days ago

preview code

raw

history blame contribute delete

3.38 kB

	---
	license: apache-2.0
	tags:
	- gguf
	- metacortex-ai
	- on-device
	- receipts
	---

	# metacortex-models

	GGUF models used by [metacortex-ai](https://github.com/nicaibutou1993/metacortex-ai) for on-device AI with receipt-based attestation.

	These files are hosted here to provide authoritative SHA-256 reference hashes for the [receipt model verification](https://github.com/nicaibutou1993/metacortex-ai/blob/main/docs/receipts-spec.md#model-verification-planned) feature.

	## Models

	\| File \| Parameters \| Quantization \| Size \| Upstream Source \|
	\|------\|-----------\|-------------\|------\|----------------\|
	\| `Qwen3.5-2B-Q4_K_M.gguf` \| 2B \| Q4_K_M \| 1.2 GB \| [unsloth/Qwen3.5-2B-GGUF](https://huggingface.co/unsloth/Qwen3.5-2B-GGUF) \|
	\| `Qwen3.5-2B-Q8_0.gguf` \| 2B \| Q8_0 \| 2.0 GB \| [unsloth/Qwen3.5-2B-GGUF](https://huggingface.co/unsloth/Qwen3.5-2B-GGUF) \|
	\| `Qwen3.5-4B-Q4_K_M.gguf` \| 4B \| Q4_K_M \| 2.6 GB \| [unsloth/Qwen3.5-4B-GGUF](https://huggingface.co/unsloth/Qwen3.5-4B-GGUF) \|
	\| `Qwen3.5-4B-Q8_0.gguf` \| 4B \| Q8_0 \| 4.2 GB \| [unsloth/Qwen3.5-4B-GGUF](https://huggingface.co/unsloth/Qwen3.5-4B-GGUF) \|
	\| `Qwen3.5-9B-Q4_K_M.gguf` \| 9B \| Q4_K_M \| 5.3 GB \| [unsloth/Qwen3.5-9B-GGUF](https://huggingface.co/unsloth/Qwen3.5-9B-GGUF) \|
	\| `Qwen3.5-9B-Q8_0.gguf` \| 9B \| Q8_0 \| 8.9 GB \| [unsloth/Qwen3.5-9B-GGUF](https://huggingface.co/unsloth/Qwen3.5-9B-GGUF) \|
	\| `Qwen3.5-27B-Q4_K_M.gguf` \| 27B \| Q4_K_M \| 16 GB \| [unsloth/Qwen3.5-27B-GGUF](https://huggingface.co/unsloth/Qwen3.5-27B-GGUF) \|
	\| `Qwen3.5-27B-Q8_0.gguf` \| 27B \| Q8_0 \| 27 GB \| [unsloth/Qwen3.5-27B-GGUF](https://huggingface.co/unsloth/Qwen3.5-27B-GGUF) \|
	\| `embeddinggemma-300m-qat-Q8_0.gguf` \| 300M \| Q8_0 \| 313 MB \| [ggml-org/embeddinggemma-300m-qat-q8_0-GGUF](https://huggingface.co/ggml-org/embeddinggemma-300m-qat-q8_0-GGUF) \|

	## SHA-256 Checksums

	```
	aaf42c8b7c3cab2bf3d69c355048d4a0ee9973d48f16c731c0520ee914699223 Qwen3.5-2B-Q4_K_M.gguf
	1b04acba824817554f4ce23639bc8495ff70453b8fcb047900c731521021f2c1 Qwen3.5-2B-Q8_0.gguf
	00fe7986ff5f6b463e62455821146049db6f9313603938a70800d1fb69ef11a4 Qwen3.5-4B-Q4_K_M.gguf
	10cc391b403021dd11c614679d2fd92f611c3681d29e29651b717316965d61e1 Qwen3.5-4B-Q8_0.gguf
	03b74727a860a56338e042c4420bb3f04b2fec5734175f4cb9fa853daf52b7e8 Qwen3.5-9B-Q4_K_M.gguf
	809626574d0cb43d4becfa56169980da2bb448f2299270f7be443cb89d0a6ae4 Qwen3.5-9B-Q8_0.gguf
	84b5f7f112156d63836a01a69dc3f11a6ba63b10a23b8ca7a7efaf52d5a2d806 Qwen3.5-27B-Q4_K_M.gguf
	6b0a101b0a86697fe11eabcc1a7db72699a9f3d4b18b6a1ac75ea3fb2c26c450 Qwen3.5-27B-Q8_0.gguf
	6fa0c02a9c302be6f977521d399b4de3a46310a4f2621ee0063747881b673f67 embeddinggemma-300m-qat-Q8_0.gguf
	```

	## Purpose

	Each metacortex-ai receipt includes a `gguf_sha256` field (SHA-256 of the local model file). Users can compare this against the hashes published here to verify the model file on disk is genuine and unmodified.

	This does not prove that this specific model generated the response -- only that an unmodified copy of the model exists on the device. See the [receipts spec](https://github.com/nicaibutou1993/metacortex-ai/blob/main/docs/receipts-spec.md) for the full trust model.

	## Usage with llama-server

	```bash
	# Chat model
	llama-server --model Qwen3.5-9B-Q4_K_M.gguf --jinja --reasoning-format deepseek -ngl 99

	# Embedding model
	llama-server --model embeddinggemma-300m-qat-Q8_0.gguf --embedding --pooling mean -c 2048
	```