FrameMorrow selects historical frames for what comes next. Prospective tokens predict useful history from observed frames and the current condition. Future observations supervise training only.

1. Clone the repository

git clone https://github.com/YinBo0927/FrameMorrow.git
cd FrameMorrow

2. Install the environment (Linux, Python 3.12, CUDA 12.8)

conda create -n framemorrow python=3.12 -y
conda activate framemorrow
conda install -c conda-forge ffmpeg -y
python -m pip install --upgrade pip setuptools wheel
python -m pip install -r requirements.txt
python -m pip install -e . --no-deps

Run the following steps in order from the repository root. The default configuration uses all OpenVidHD videos.

export DATA_DIR=data/openvidhd
export RUN_DIR=runs/openvidhd

1. Download and annotate

python download_data.py plan "$DATA_DIR/raw"
python download_data.py fetch "$DATA_DIR/raw/download_plan.json"
python annotate.py "$DATA_DIR/raw/videos.jsonl" "$DATA_DIR/annotations" \
  --model Qwen/Qwen3-VL-8B-Instruct --device cuda --condition-seconds 4

Source: OpenVidHD. Rerun fetch to resume downloads. Add --resume to continue annotation with the same settings. If Hugging Face is unreachable, set export HF_ENDPOINT=https://hf-mirror.com before downloading.

2. Prepare frames and features

python prepare.py "$DATA_DIR/annotations/videos.jsonl" "$DATA_DIR/tuples" --seed 0
python encode.py "$DATA_DIR/tuples/tuples.jsonl" "$DATA_DIR/features" \
  --modality text --device cuda --max-condition-tokens 256

Features use frozen DINOv2 ViT-B/14 and UMT5-XXL. Models download automatically. Clips without valid training windows are listed in preparation_report.json.

3. Train the selector

python train.py "$DATA_DIR/features/features.jsonl" \
  --config configs/text.json --output "$RUN_DIR"

Saves best.pt and last.pt. Add --resume "$RUN_DIR/last.pt" to resume the same run.

4. Select frames

python select_frames.py "$RUN_DIR/best.pt" \
  --input "$DATA_DIR/tuples/observed.json" \
  --output outputs/selected.json --device cuda

Returns frame paths, timestamps, indices, and scores. Custom inputs contain history, recent, and prompt. Frame paths are relative to the input JSON. Future frames are excluded at inference. Selector weights are not bundled.

Edit configs/default.json. CLI flags override JSON settings. Both plan and train.py accept --settings FILE.

For a small run, add --count 20 --min-seconds 12 to plan and --epochs 1 --batch-size 16 to training. Model architecture is configured in configs/text.json. Use new output directories for separate runs.

Download the backends and weights, then run either generator:

python examples/setup_generators.py

# longlive
python examples/longlive.py --selector-checkpoint "$RUN_DIR/best.pt" --output outputs/longlive

# self-forcing
# python examples/self_forcing.py --selector-checkpoint "$RUN_DIR/best.pt" --output outputs/self_forcing

Uses 2 historical frames by default. Change this with --k. Each example saves conditioned.mp4 in its output directory.

Long video generation on MovieGenBench.

Interactive video generation under single-shot and multi-shot conditions.

Action-conditioned interactive world models.

@article{yin2026framemorrow,
  title={FrameMorrow: Future-guided Frame Selection with Prospective Tokens for Long-Horizon Video Generation},
  author={Yin, Bo and Hu, Xiaobin and Zhao, Jiaqi and Yan, Shuicheng},
  journal={arXiv preprint arXiv:2609.38839},
  year={2026}
}