|
|
--- |
|
|
license: cc-by-nc-4.0 |
|
|
language: |
|
|
- en |
|
|
pipeline_tag: image-text-to-text |
|
|
--- |
|
|
# Isaac-0.2-2B by Perceptron |
|
|
|
|
|
Introducing the 2B parameter variant of Isaac-0.2, the hybrid-reasoning vision-language model. |
|
|
|
|
|
This release brings major upgrades — optional reasoning via thinking traces, perceptive tool calling (including our new Focus system), stronger grounding, better OCR, better desktop use, and improved structured output — while remaining fast, compact, and deployable. |
|
|
|
|
|
[Try it on our demo! 🚀](https://www.perceptron.inc/demo) - [API Docs 📘](https://docs.perceptron.inc/) - [Discord 💬](https://discord.gg/fgBeaACQzE) |
|
|
|
|
|
## Extending the efficient frontier of perception |
|
|
|
|
|
Isaac 0.2 extends what we started with Isaac 0.1: small models that outperform systems 10× larger on visual reasoning and perception tasks, all running on commodity GPUs or edge devices. |
|
|
From robotics to media search to industrial inspection, Isaac 0.2 delivers high-accuracy perception without the heavy compute footprint. |
|
|
|
|
|
 |
|
|
|
|
|
## What's New in Isaac 0.2 |
|
|
|
|
|
* **Reasoning via Thinking Traces**: Short, structured reasoning traces improve multi-step decisions, small-object understanding, and ambiguous spatial tasks. |
|
|
|
|
|
* **Perceptive Tool Calling + Focus (Zoom & Crop)**: Isaac 0.2 can trigger tool calls to focus (i.e., zoom and crop) and re-query the model on a smaller region — dramatically improving fine-grained perception. |
|
|
|
|
|
* **Structured Outputs**: More reliable structured output generation for consistent JSON and predictable downstream integration. |
|
|
|
|
|
* **Complex OCR**: Improved text recognition across cluttered, low-resolution, or distorted regions — enabling accurate extraction from documents, diagrams, labels, screens, and dense real-world scenes. |
|
|
|
|
|
* **Desktop Use**: Better performance on everyday desktop and mobile workflows such as UI understanding and navigation, making Isaac faster and more capable for agentic use cases. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Performance Benchmarks |
|
|
|
|
|
 |
|
|
|
|
|
## Chatting with Isaac in 🤗 Transformers |
|
|
Learn more at our [Huggingface Example Repo](https://github.com/perceptron-ai-inc/perceptron/tree/main/huggingface), where we demo extracting and rendering points. |
|
|
|
|
|
```bash |
|
|
pip install perceptron |
|
|
``` |
|
|
|
|
|
### Usage |
|
|
|
|
|
```python |
|
|
import torch |
|
|
from transformers import AutoModelForCausalLM, AutoProcessor |
|
|
from transformers.image_utils import load_image |
|
|
from transformers.utils.import_utils import is_torch_cuda_available |
|
|
|
|
|
def document_to_messages(document: list[dict]): |
|
|
messages, images = [], [] |
|
|
for item in document: |
|
|
if not (content := item.get("content")): |
|
|
continue |
|
|
role = item.get("role", "user") |
|
|
if item.get("type") == "image": |
|
|
images.append(load_image(content)) |
|
|
messages.append({"role": role, "content": "<image>"}) |
|
|
elif item.get("type") == "text": |
|
|
messages.append({"role": role, "content": content}) |
|
|
return messages, images |
|
|
|
|
|
hf_path = "PerceptronAI/Isaac-0.2-2B-Preview" |
|
|
device, dtype = ("cuda",torch.bfloat16) if is_torch_cuda_available() else ("cpu",torch.float32) |
|
|
|
|
|
# Load model/processor from the checkpoint |
|
|
processor = AutoProcessor.from_pretrained(hf_path, trust_remote_code=True) |
|
|
model = AutoModelForCausalLM.from_pretrained( |
|
|
hf_path, trust_remote_code=True, vision_attn_implementation="flash_attention_2" |
|
|
) |
|
|
model = model.to(device=device, dtype=dtype) |
|
|
model.eval() |
|
|
|
|
|
# Prepare input for generation |
|
|
document = [ |
|
|
{ |
|
|
"type": "text", |
|
|
"content": "<hint>BOX</hint>", |
|
|
"role": "user", |
|
|
}, |
|
|
{ |
|
|
"type": "image", |
|
|
"content": "https://raw.githubusercontent.com/perceptron-ai-inc/perceptron/refs/heads/main/huggingface/assets/example.webp", |
|
|
"role": "user", |
|
|
}, |
|
|
{ |
|
|
"type": "text", |
|
|
"content": "Determine whether it is safe to cross the street. Look for signage and moving traffic.", |
|
|
"role": "user", |
|
|
}, |
|
|
] |
|
|
messages, images = document_to_messages(document) |
|
|
text = processor.apply_chat_template( |
|
|
messages, tokenize=False, add_generation_prompt=True |
|
|
) |
|
|
inputs = processor(text=text, images=images, return_tensors="pt") |
|
|
|
|
|
# Generate text using the model |
|
|
generated_ids = model.generate( |
|
|
tensor_stream=inputs["tensor_stream"].to(next(model.parameters()).device), |
|
|
max_new_tokens=256, |
|
|
do_sample=False, |
|
|
) |
|
|
generated_text = processor.tokenizer.decode( |
|
|
generated_ids[0], skip_special_tokens=False |
|
|
) |
|
|
print(f"\nFull generated output:\n{generated_text}") |
|
|
``` |
