litert-community/MoViNet-A0-Stream-LiteRT

MoViNet-A0 Stream — LiteRT (on-device video action recognition, GPU)

On-device streaming video action recognition: recognises human actions across a stream of camera frames — one frame at a time, constant memory, real-time — running fully on the LiteRT CompiledModel GPU delegate (no CPU fallback).

• Architecture: MoViNet-A0 streaming variant (Google Research) — a causal 2+1D CNN.
• Task: Kinetics-600 — 600 action classes.
• Weights: ported PyTorch checkpoint from Atze00/MoViNet-pytorch.
• Size: 15 MB · ~3.75 M params · input frame 172×172.

How the streaming graph works

MoViNet's temporal convolutions and global-average-pools each keep a small buffer of the recent past, so the network can be fed one frame at a time and its prediction sharpens as more frames of the same action arrive. The stock streaming graph carries that history in 5D state tensors [1, T, H, W, C], which a GPU delegate cannot compile (all tensors must be ≤ 4D). This model is re-authored as a single-frame, 4D-only functional forward (47 inputs / 28 outputs) with the recurrent state threaded explicitly through the graph I/O:

I/O slot	count	shape	meaning
input[0]	1	[1,3,172,172]	current RGB frame (NCHW, 0..1)
input[1..28]	28	[1,C,H,W]	temporal-conv stream buffers (11 convs)
input[29..44]	16	[1,C,1,1]	streaming avg-pool running sums (15 SE + head)
input[45]	1	[1,1,1,1]	inv_count = 1 / current frame number
input[46]	1	[1,1,1,1]	constant 1.0 (Mali output decoupler)
output[0]	1	[1,600]	Kinetics-600 logits
output[1..11]	11	[1,C,H,W]	current per-temporal-conv frame
output[12..27]	16	[1,C,1,1]	fresh per-frame spatial means

The stream-buffer shift register and pool running-sum accumulation are done host-side: each frame you run once, shift each stream buffer (drop oldest, append the emitted current frame), accumulate running_sum += emitted_mean, and feed both back as inputs. The converted graph is all float32, 0 tensors of rank > 4, 0 GPU-incompatible ops and matches the original PyTorch model bit-for-bit (correlation 0.99999999999, top-5 identical; device GPU on a Pixel 8a locks onto "jumping jacks" within a few frames). Keeping the state in-graph tripped three silent Mali CompiledModel bugs, which is why the state plumbing is host-side.

Minimal usage

Python (LiteRT / ai-edge-litert, frame-by-frame)

from ai_edge_litert.interpreter import Interpreter
import numpy as np

it = Interpreter(model_path="movinet_a0_stream.tflite"); it.allocate_tensors()
inp, out = it.get_input_details(), it.get_output_details()

DIMS = [2, 2, 2, 4, 2, 2, 4, 2, 2, 2, 4]        # temporal-conv buffer depths
offs, o = [], 0
for d in DIMS: offs.append(o); o += d
hist = [[np.zeros(inp[1 + offs[c] + i]["shape"], np.float32) for i in range(DIMS[c])]
        for c in range(11)]                      # host-side shift registers
psum = [np.zeros(inp[29 + i]["shape"], np.float32) for i in range(16)]  # running sums

for n, frame in enumerate(video_frames, start=1):   # frame: [1,3,172,172], RGB, 0..1
    it.set_tensor(inp[0]["index"], frame.astype(np.float32))
    for c in range(11):
        for i in range(DIMS[c]): it.set_tensor(inp[1 + offs[c] + i]["index"], hist[c][i])
    for i in range(16): it.set_tensor(inp[29 + i]["index"], psum[i])
    it.set_tensor(inp[45]["index"], np.full((1, 1, 1, 1), 1.0 / n, np.float32))  # inv_count
    it.set_tensor(inp[46]["index"], np.ones((1, 1, 1, 1), np.float32))           # decoupler
    it.invoke()
    logits = it.get_tensor(out[0]["index"])[0]      # [600]
    for c in range(11):                             # shift: drop oldest, append current
        hist[c] = hist[c][1:] + [it.get_tensor(out[1 + c]["index"]).copy()]
    for i in range(16):                             # accumulate running sum
        psum[i] = psum[i] + it.get_tensor(out[12 + i]["index"])

print("top-1:", int(logits.argmax()))

Kotlin (Android, LiteRT CompiledModel GPU)

val options = CompiledModel.Options(Accelerator.GPU)
val model = CompiledModel.create(context.assets, "movinet_a0_stream.tflite", options, null)
val inBufs = model.createInputBuffers()    // [0]=frame, [1..28]=stream, [29..44]=pool sums, [45]=inv_count, [46]=1.0
val outBufs = model.createOutputBuffers()  // [0]=logits, [1..11]=current frames, [12..27]=fresh means

inBufs[46].writeFloat(floatArrayOf(1f))    // constant decoupler
// reset recurrent state (zeros) at the start of a clip; keep host-side stream buffers + pool sums

for ((n, frameNCHW) in videoFrames.withIndex()) {           // frame: [1,3,172,172], RGB, 0..1
    inBufs[0].writeFloat(frameNCHW)
    inBufs[45].writeFloat(floatArrayOf(1f / (n + 1)))       // inv_count
    // (stream inputs 1..28 and pool inputs 29..44 already staged from the previous frame)
    model.run(inBufs, outBufs)
    val logits = outBufs[0].readFloat()                     // [600] Kinetics-600
    // host-side: shift each stream buffer with the emitted current frame (outBufs[1..11]),
    // and accumulate poolSum[i] += outBufs[12+i], then write both back to inBufs for the next frame.
}

A full implementation (camera → per-frame → top-5, with the host-side shift register and pool accumulation) is in the sample app's ActionRecognizer.kt.

Conversion

Re-authored and converted with litert-torch. See the sample app and build script: build_movinet.py + stream_model.py.

License

Apache-2.0 (MoViNet / Atze00/MoViNet-pytorch). Kinetics-600 label taxonomy from the DeepMind Kinetics dataset.