EchoFM: A Video Vision Foundation Model for Echocardiography

ViT-L video masked autoencoder pretrained on ~41k apical echocardiogram clips with a cardiac-cycle-aware objective:

L = L_recon (norm-pix, 75% spatio-temporally consistent masking) + L_triplet + L_cycle-KL

• triplet: positives/negatives chosen by a pixel-space cycle-similarity prior (hard mining, cosine margin 0.2)
• cycle-KL: per-anchor embedding-similarity distributions distilled toward the pixel prior with the static (anatomy) component removed — this makes the embeddings cardiac-phase-aware

Code, training pipeline, and diagnostics: https://github.com/SekeunKim/EchoFM

Checkpoint

echofm_vitl.pth — final model, 200 epochs. {"model": state_dict, "model_args": dict} (1.4 GB).

Validation on held-out clips (final):

metric	value
embedding-vs-pixel cycle correlation r	0.991
phase contrast (same-phase minus opposite-phase similarity)	0.90 (positive on 100% of clips)
masked PSNR (75% masking)	24.7 dB

Usage

import torch
from huggingface_hub import hf_hub_download
from EchoFM import models_mae  # from the GitHub repo

weights = hf_hub_download(repo_id="sekeun/EchoFM", filename="echofm_vitl.pth")
ckpt = torch.load(weights, map_location="cpu")
model = models_mae.mae_vit_large_patch16(**{
    k: ckpt["model_args"][k] for k in
    ["num_frames", "t_patch_size", "pred_t_dim", "sep_pos_embed", "cls_embed", "norm_pix_loss"]
})
model.load_state_dict(ckpt["model"], strict=False)
model.eval()

# imgs: [B, 3, 32, 224, 224] in [0, 1]
latent, _, _ = model.forward_encoder(imgs, mask_ratio=0.0)      # [B, 8*196, 1024] tokens
cls_stack = torch.stack(model.forward_prj(latent), dim=1)       # [B, 8, 1024] per-frame (phase) embeddings
video_emb = latent.mean(dim=1)                                  # [B, 1024] video embedding

ED/ES and cardiac-cycle extraction

echofm_phase.py (in this repo) provides ready-to-use phase utilities — cycle length, heart rate, ED/ES frame detection (no model needed), and embedding-based same-phase retrieval:

from huggingface_hub import hf_hub_download
import importlib.util

spec = importlib.util.spec_from_file_location(
    "echofm_phase", hf_hub_download("sekeun/EchoFM", "echofm_phase.py"))
phase = importlib.util.module_from_spec(spec); spec.loader.exec_module(phase)

# clip: float tensor [3, T, H, W] in [0, 1]
info = phase.detect_ed_es(clip, fps=30)
# {'ed_frames': [6, 30], 'es_frame': 13, 'cycle_frames': 24, 'hr_bpm': 75.0, ...}

z = phase.phase_embeddings(model, clip)         # [8, 1024] per-timestep phase embeddings
match = phase.same_phase_frame(model, clip, frame=info["ed_frames"][0])
# {'match_frame': 30, ...} — retrieves the same phase in the next cycle

See notebooks/echofm_usage.ipynb in the GitHub repo for more examples.