MOSS-VL-Realtime-SGLANG

This repository packages MOSS-VL-Realtime for Transformers 5.12.1 and the MOSS-VL SGLang-Omni realtime backend. It is a compatibility release of the original streaming checkpoint, not a newly trained or quantized model.

Resource	Purpose
OpenMOSS-Team/MOSS-VL-Realtime-SGLANG	This Transformers 5.12.1-compatible checkpoint and custom code
OpenMOSS-Team/MOSS-VL-Realtime	Original checkpoint and Transformers 4.57-series reference code
fnlp-vision/sglang-omni-realtime	Specialized realtime serving backend, developed on SGLang-Omni
fnlp-vision/MOSS-VL-Realtime_Demo	Separate Demo and gateway integration

The custom Python files and configuration in this repository must be used together. Do not replace them with the original repository's 4.57-series files or assume that installing the official sglang-omni PyPI package includes the specialized backend.

Compatibility and RoPE Fixes

The validated stack uses Transformers 5.12.1, SGLang 0.5.16, and PyTorch 2.11.0 on NVIDIA CUDA. The package keeps the 5.12.1 adaptations for configuration/RoPE APIs, output recording, chat-template return values, and attention-mask/cache interfaces.

Two targeted fixes are included:

• Cross-attention Query RoPE: every newly computed text query is rotated, including steps that reuse cached visual KV without new visual input. Visual keys are rotated only when newly computed; cached keys are not rotated again. The equivalent narrow fix is also published in the original reference repository.
• Vision rotary frequencies under Transformers 5.12.1: non-persistent frequency buffers may be rematerialized during model loading. This implementation reconstructs the canonical FP32 frequencies on the active device rather than trusting potentially invalid buffer contents. This is a 5.12.1 compatibility fix, not a claim that the original 4.57 environment has the same loading problem.

The five BF16 weight shards, tokenizer and vocabulary are unchanged from the original checkpoint. The compatibility changes are in code and configuration, not model training. Pin the model revision and backend commit together when reproducing results. CUDA validation does not imply NPU support, nor should the latest 4.57-series source changes be assumed to be present in this separate compatibility branch.

MOSS-VL is an open vision-language model family from OpenMOSS, supporting image understanding, long-video understanding, and realtime streaming interaction.

Technical Report: https://arxiv.org/pdf/2608.15045

Overview

MOSS-VL-Realtime is the realtime streaming checkpoint of the MOSS-VL release, part of the OpenMOSS ecosystem for open visual understanding.

Unlike offline video-language models that first read a complete video and then answer, MOSS-VL-Realtime is designed for continuous video streams. It perceives incoming frames and generates text in parallel, supports questions at arbitrary moments in the stream, and can decide whether to respond or keep observing when the visual evidence is insufficient.

This release keeps the MOSS-VL cross-attention design and a 256K text context window while adding realtime streaming data and an inference interface for timestamped frame-by-frame input.

Key Features

• Realtime streaming understanding: processes incoming frames continuously instead of waiting for a complete video.
• Interruptible interaction: users can ask questions at any timestamp in a running stream, and the model answers based on the frames observed so far.
• Proactive silence: the model can emit <|silence|> and continue observing when there is no meaningful visual update or the context is not sufficient.
• Dynamic correction: as new frames arrive, the model can revise earlier responses instead of being locked to an initial interpretation.
• Timestamp-aware frames: each streamed frame is associated with an absolute timestamp, helping the model reason about event order, duration, pacing, and fine-grained temporal localization.
• Unified MOSS-VL family: released together with MOSS-VL-Instruct and MOSS-VL-Base for offline use, continued pretraining, fine-tuning, and applied research.

Model Design

Architecture

MOSS-VL-Realtime adopts a cross-attention-based vision-language architecture that decouples visual encoding from language reasoning. This design is important for realtime usage because incoming visual content can be integrated into the running generation context without forcing the model into a strictly offline "load all frames, then answer" workflow.

Timestamp-aware Video Encoding

For video and realtime frame inputs, MOSS-VL injects absolute timestamps alongside sampled frames. This helps the model reason about when an event happens, how long it lasts, and how the scene changes over time instead of relying only on frame order.

MOSS-VL also uses Cross-attention Rotary Position Embedding (XRoPE), which maps text tokens and visual patches into a unified three-dimensional coordinate space defined by Time (t), Height (h), and Width (w). This gives the model a consistent positional representation for image, offline video, and realtime streaming video reasoning.

Configuration

Item	Value
Parameters	11B
Tensor type	BF16
Context length	256K
Vision patch size	16
Temporal patch size	1
Default video FPS	1.0
Default max video frames	256
Realtime frame format	PIL-compatible image plus timestamp
Direct Transformers session scope	One active realtime session per model instance
SGLang-Omni session scope	Configurable with --max-running-requests; subject to KV capacity

Performance

MOSS-VL-Realtime is designed for streaming video understanding benchmarks where questions can arrive before a full video has been observed and correct answers may change as the scene evolves. It targets realtime interaction quality, proactive silence, and dynamic response updates in addition to standard video understanding accuracy.

Detailed benchmark tables and comparisons for this release will be maintained in the MOSS-VL project resources.

Quickstart

Installation

For the specialized backend, install its source and pinned dependencies in a fresh, compatible CUDA environment. Install uv first if it is not already available:

git clone https://github.com/fnlp-vision/sglang-omni-realtime.git
cd sglang-omni-realtime
uv venv .venv -p 3.12
source .venv/bin/activate
uv pip install -e .

See the backend README for CUDA/toolchain prerequisites and deployment details. This is not a universal installation recipe for arbitrary CUDA versions. For direct Transformers inference, use the same compatible Transformers 5.12.1 environment and the example below; an SGLang server does not need to be running.

Load the Model

import torch
from transformers import AutoModelForCausalLM, AutoProcessor

checkpoint = "OpenMOSS-Team/MOSS-VL-Realtime-SGLANG"

processor = AutoProcessor.from_pretrained(
    checkpoint,
    trust_remote_code=True,
    frame_extract_num_threads=1,
)
model = AutoModelForCausalLM.from_pretrained(
    checkpoint,
    trust_remote_code=True,
    device_map="auto",
    dtype=torch.bfloat16,
    attn_implementation="eager",
)
model.eval()

This direct-Transformers example uses eager attention. The SGLang-Omni backend has its own attention configuration; do not confuse the two execution paths.

Start the SGLang-Omni Backend

Download this checkpoint to a directory outside the backend source tree, then launch from the backend repository root:

hf download OpenMOSS-Team/MOSS-VL-Realtime-SGLANG \
  --local-dir /path/to/moss-vl-realtime-sglang

python examples/run_moss_vl_realtime_server.py \
  --model-path /path/to/moss-vl-realtime-sglang \
  --gpu 0 --host 127.0.0.1 --port 8000 \
  --context-length 131072 \
  --mem-fraction-static 0.60 \
  --max-running-requests 1

Access requires authorization while the repository is private. The command uses a 128K context as an explicit deployment example; the model's 256K capacity and launcher defaults do not guarantee sufficient KV memory on every device. Tune context, memory fraction and concurrency for your hardware. The backend endpoint is /v1/video/realtime; its WebSocket protocol and TP options are documented in the Realtime Cookbook.

This model repository contains weights and custom model/processor code, not the backend scheduler, Demo frontend, customer authentication or billing service.

Inference Examples

Online Inference

Session-style Online Inference

The recommended direct API is create_realtime_session(...). A service or application owns the video capture pipeline, converts camera, screen, or video-file input into PIL-compatible frames, and pushes each frame with a non-decreasing timestamp.

Common session operations:

• session.push_frame(image, timestamp=...) appends one visual frame.
• session.push_prompt("...") appends a user question while the stream is running.
• session.push_prompt_frame(prompt, image, timestamp=...) aligns a prompt with a specific frame.
• session.poll_output(...) or session.stream_outputs(...) returns incremental text chunks.

system_prompt and initial_prompt are tokenized as the initial system/user turns before the first frame arrives. Subsequent user turns can be appended with push_prompt(...) while the same session continues observing frames.

For complete real-time inference usage, including local-video replay and service deployment, see realtime_inference in the MOSS-VL GitHub repository.

import time
from PIL import Image

session = model.create_realtime_session(
    processor,
    initial_prompt=(
        "As the video streams frame by frame, describe important changes as they happen. "
        "Stay silent when there is no relevant update."
    ),
    frame_queue_size=256,
    max_tokens_per_turn=12,
    max_new_tokens=4096,
    do_sample=False,
)

frame_paths = [
    "data/frame_0001.jpg",
    "data/frame_0002.jpg",
    "data/frame_0003.jpg",
]

try:
    session.start()

    for index, frame_path in enumerate(frame_paths):
        image = Image.open(frame_path).convert("RGB")
        session.push_frame(image, timestamp=index / 1.0)

        while True:
            chunk = session.poll_output(timeout=0.0)
            if chunk is None:
                break
            print(chunk, end="", flush=True)

        time.sleep(1.0)

    session.push_prompt("What changed in the latest frames?")

    # Realtime sessions stay alive waiting for future input, so use a bounded
    # drain window and close the session explicitly when the producer is done.
    drain_deadline = time.monotonic() + 5.0
    while time.monotonic() < drain_deadline:
        chunk = session.poll_output(timeout=0.1)
        if chunk is not None:
            print(chunk, end="", flush=True)
finally:
    session.close()

Frame timestamps are measured in seconds and must be non-decreasing within a session. The input producer can be a camera, screen capture, decoded video file, browser frame sampler, or any other source that yields images with timestamps.

Queue-style Online Inference

online_generate(...) is useful for backend systems that separate frame production and model inference through queues. It accepts dictionaries containing frames, prompts, events, reset controls, and stop controls.

import queue
import threading
from PIL import Image

input_queue = queue.Queue()
output_queue = queue.Queue()

worker = threading.Thread(
    target=model.online_generate,
    args=(processor, input_queue, output_queue),
    kwargs={
        "frame_queue_size": 256,
        "max_tokens_per_turn": 12,
        "max_new_tokens": 4096,
        "do_sample": False,
    },
    daemon=True,
)
worker.start()

input_queue.put({
    "initial_prompt": "Answer only when the streamed video provides enough evidence.",
})

input_queue.put({"frame": Image.open("data/frame_0001.jpg").convert("RGB"), "timestamp": 0.0})
input_queue.put({"frame": Image.open("data/frame_0002.jpg").convert("RGB"), "timestamp": 1.0})
input_queue.put({"prompt": "What is happening now?"})

try:
    while True:
        chunk = output_queue.get(timeout=0.5)
        print(chunk, end="", flush=True)
except queue.Empty:
    pass

input_queue.put({"stop_online_generate": True})
worker.join()

Each queue item can contain frame or image, timestamp, prompt, frames, event, events, initial_prompt, system_prompt, generate_kwargs, reset_session, or stop controls such as stop_online_generate.

Offline Inference

MOSS-VL-Realtime also keeps the offline helper APIs for image and video prompts. For purely offline use, MOSS-VL-Instruct is usually the preferred checkpoint, but the realtime checkpoint can still process complete image and video inputs.

Single-video Offline Inference

video_path = "data/example_video.mp4"
prompt = "Describe this video."

text = model.offline_video_generate(
    processor,
    prompt=prompt,
    video=video_path,
    shortest_edge=4096,
    longest_edge=16777216,
    video_max_pixels=201326592,
    patch_size=16,
    temporal_patch_size=1,
    merge_size=2,
    video_fps=1.0,
    min_frames=1,
    max_frames=256,
    num_extract_threads=4,
    image_mean=[0.5, 0.5, 0.5],
    image_std=[0.5, 0.5, 0.5],
    max_new_tokens=256,
    temperature=1.0,
    top_k=50,
    top_p=1.0,
    repetition_penalty=1.0,
    do_sample=False,
    vision_chunked_length=64,
)

print(text)

Batched Offline Inference

offline_batch_generate accepts independent image/video/text queries. Queries in the same batch should share the same media_kwargs and generate_kwargs.

queries = [
    {
        "prompt": "Describe sample A.",
        "images": [],
        "videos": ["data/sample_a.mp4"],
        "media_kwargs": {
            "video_fps": 1.0,
            "min_frames": 8,
            "max_frames": 256,
        },
        "generate_kwargs": {
            "temperature": 1.0,
            "top_k": 50,
            "top_p": 1.0,
            "max_new_tokens": 256,
            "repetition_penalty": 1.0,
            "do_sample": False,
        },
    },
    {
        "prompt": "Describe sample B.",
        "images": [],
        "videos": ["data/sample_b.mp4"],
        "media_kwargs": {
            "video_fps": 1.0,
            "min_frames": 8,
            "max_frames": 256,
        },
        "generate_kwargs": {
            "temperature": 1.0,
            "top_k": 50,
            "top_p": 1.0,
            "max_new_tokens": 256,
            "repetition_penalty": 1.0,
            "do_sample": False,
        },
    },
]

with torch.no_grad():
    result = model.offline_batch_generate(
        processor,
        queries,
        vision_chunked_length=64,
    )

texts = [item["text"] for item in result["results"]]
print(texts)

Related Checkpoints

Model	Parameters	Context	Usage	Hugging Face
MOSS-VL-Realtime-SGLANG	11B	256K	Transformers 5.12.1 / specialized SGLang-Omni compatibility package	https://huggingface.co/OpenMOSS-Team/MOSS-VL-Realtime-SGLANG
MOSS-VL-Realtime	11B	256K	Realtime streaming video interaction	https://huggingface.co/OpenMOSS-Team/MOSS-VL-Realtime
MOSS-VL-Instruct	11B	256K	Offline multimodal instruction following	https://huggingface.co/OpenMOSS-Team/MOSS-VL-Instruct
MOSS-VL-Base	11B	256K	Continued pretraining and fine-tuning	https://huggingface.co/OpenMOSS-Team/MOSS-VL-Base
MOSS-VL-Instruct-0408	11B	256K	Previous instruction-tuned checkpoint	https://huggingface.co/OpenMOSS-Team/MOSS-VL-Instruct-0408
MOSS-VL-Base-0408	11B	256K	Previous base checkpoint	https://huggingface.co/OpenMOSS-Team/MOSS-VL-Base-0408

Validation of This Compatibility Package

On 2026-09-06, the Query RoPE and vision-frequency regression suite passed 17 tests, including CUDA dtype/autocast and non-persistent-buffer rematerialization cases. An isolated H200 smoke test with Transformers 5.12.1 / PyTorch 2.11.0 loaded the full checkpoint, completed an image forward pass and generated a finite natural-language answer with eager attention.

These checks cover the targeted fixes and basic model execution. They do not certify every live-stream schedule, full NPU compatibility, or a production latency/concurrency SLA. The current Transformers runtime still emits deprecation warnings for processor aliases and cache_position; future Transformers releases require separate adaptation rather than an unqualified dependency upgrade.

Limitations and Roadmap

MOSS-VL-Realtime is optimized for timestamped frame-by-frame streaming, but production latency depends on GPU hardware, frame sampling rate, transport overhead, and decoding speed. The direct Transformers API supports one active realtime session per model instance, and its bounded frame queue may drop older pending frames when overloaded. The specialized SGLang-Omni backend supports configured multi-session serving and uses protocol backpressure; these are different runtime contracts.

The 1 FPS workflow is the semantic validation baseline. Real-time input arrival, sampling, session stop rules and optional vision-KV eviction can change outputs even when fixed-boundary comparisons agree. Neither the RoPE fixes nor a successful smoke test imply bitwise equality for every live HF/SGLang session or a production SLA. Optional frame pooling is not required for this checkpoint and remains disabled by default in the specialized backend.

The model may emit realtime control tokens such as <|silence|>, <|round_start|>, and <|round_end|> depending on the application protocol. Downstream services should filter or render these tokens according to their UI needs.

We are continuing to improve realtime response timing, dynamic correction, broader streaming evaluations, RL post-training, and task-specific deployment recipes for future MOSS-VL releases.

Citation

@misc{mossvl,
  title         = {MOSS-VL Technical Report},
  author        = {Wang, Pengyu and Tan, Chenkun and Zhou, Shaojun and Zhou, Qirui and Chen, Yanxin and He, Xingyang and Zeng, Huazheng and Cheng, Jijun and Wang, Chenghao and Qian, Xiaomeng and Wang, Pengfei and Huang, Zhan and Gao, Shanqing and Huang, Wei and Cao, Longjun and Ran, Wu and Liu, Jie and Zhu, Changtai and Wang, Hongkai and Tian, Yixian and Liu, Chenghao and Ye, Zhen and Wang, Xinghao and Jiang, Botian and Feng, Guoguo and Fei, Zhaoye and Li, Ruixiao and Chen, Mingshu and Gao, Yang and Cheng, Qinyuan and Li, Shimin and Qiu, Xipeng},
  year          = {2026},
  eprint        = {2608.15045},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2608.15045}
}

@misc{mossvideopreview,
  title         = {{MOSS-Video-Preview: Toward Real-Time Video Understanding via Cross-Attention}},
  author        = {Pengyu Wang and Chenkun Tan and Shaojun Zhou and Wei Huang and Qirui Zhou and Zhan Huang and Zhen Ye and Jijun Cheng and Xiaomeng Qian and Yanxin Chen and Xingyang He and Huazheng Zeng and Chenghao Wang and Pengfei Wang and Hongkai Wang and Shanqing Gao and Yixian Tian and Chenghao Liu and Xinghao Wang and Botian Jiang and Xipeng Qiu},
  year          = {2026},
  eprint        = {2606.07639},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2606.07639}
}