Mungyeom Kim^1* · Minkyeong Jeon^1* · Honggyu An^1* · Jaewoo Jung¹ · Hyunah Ko¹ · Jisang Han¹ · Hyeonseo Yu¹ · Donghwan Shin¹ · Sunghwan Hong² · Takuya Narihira³ · Kazumi Fukuda³ · Yuki Mitsufuji^3,4† · Seungryong Kim^1†

^*Co-first author, †Co-corresponding author

We present T4GS (Temporally Aware Compact 4D Gaussian Splatting), a feed-forward 4D reconstruction framework built upon a compact set of timestamp-conditioned learnable query tokens. Each token aggregates features across the full temporal context and decodes a 3D Gaussian whose position is modulated by the target timestamp, enabling globally coherent motion modeling without input camera poses. The resulting Gaussians serve as a motion-aware anchor for camera-controllable video generation and support 4D feature lifting for point tracking and dynamic scene understanding.

• Training code for T4GS 4D Gaussian reconstruction.
  
• Camera-controllable video generation (VDM) code.
  
• 4D feature lifting code.
  
• Pretrained T4GS weights (paper setting).
  

Our code is developed based on PyTorch 2.2.0, CUDA 12.1, and Python 3.10.

We recommend using conda for installation:

git clone --recursive https://github.com/cvlab-kaist/T4GS.git
cd T4GS
# (if already cloned without --recursive: git submodule update --init --recursive)

conda create -n t4gs python=3.10
conda activate t4gs
bash scripts/install.sh

scripts/install.sh installs the Python requirements and builds the Gaussian rasterizer in submodules/diff_gaussian_rasterization_w_pose (it needs the GLM submodule fetched above).

T4GS is trained on a mixture of four datasets (sampling ratio in parentheses). All dataset paths in config/ are placeholders written as /path/to/...; set them to your local copies (or override them on the command line, e.g. dataset.kubric.roots=[/data/kubric_root]).

Spring needs its GT depth / scene flow converted once into per-frame motion maps (depth.npy, flow.npy, depth_next.npy, mask.npy, valid.npy per scene):

python scripts/preprocess_spring_motion.py --root /path/to/Spring/train --out /path/to/spring_motion_224

Checkpoints are available in the T4GS Hugging Face repository:

The checkpoint stores the encoder weights under state_dict with the encoder. prefix, so it can be used directly to initialize training or fine-tuning:

bash scripts/train.sh model.encoder.pretrained_weights=/path/to/t4gs.ckpt

The reconstruction model runs standalone; the paper's novel-view-synthesis and tracking results use it without the video diffusion module, which is only needed for camera-controllable video generation.

scripts/render_qualitative_davis.py shows how to run the model on monocular videos (DAVIS-style frame folders): it builds the encoder/decoder, loads t4gs.ckpt, estimates pseudo cameras with Depth Anything 3 (install it separately) and renders novel views.

T4GS initializes its VGGT backbone and Gaussian decoder from the C3G static model. Download gaussian_decoder.ckpt from the C3G Hugging Face repository and set model.encoder.pretrained_weights in config/training/t4gs.yaml to its path.

The frozen models used for supervision are downloaded automatically on first use: MoGe-2 (Ruicheng/moge-2-vitl-normal, depth / normal pseudo-labels), CoWTracker (tracking-consistency loss, included as a submodule) and torchvision RAFT (online flow pseudo-labels for EgoExo4D).

bash scripts/train.sh

This runs python -m src.main +training=t4gs with the paper recipe in config/training/t4gs.yaml: 4 GPUs × batch 1, 30k steps, AdamW lr 1e-5 (backbone ×0.01) with 1k warm-up and a 10k-step cosine period (optimizer.cosine_period; the LR decays, rises and decays again over the 30k steps, as for the released checkpoint), relative time embedding, and the photometric, depth/normal, tracking and motion losses described in the paper. Any value can be overridden on the command line, e.g.

bash scripts/train.sh trainer.max_steps=10000 wandb.mode=online

Logging to wandb is disabled by default (wandb.mode=disabled).

T4GS renders of a monocular video serve as a motion-aware anchor for camera-controllable video generation with a video diffusion model (Wan2.1-VACE-1.3B). The code is included under submodules/DiffSynth-Studio; please follow submodules/DiffSynth-Studio/README.md for training and inference.

@article{kim2026learning,
  title={Learning Global Motion with Compact Gaussians for Feed-Forward 4D Reconstruction},
  author={Kim, Mungyeom and Jeon, Minkyeong and An, Honggyu and Jung, Jaewoo and Ko, Hyuna and Han, Jisang and Yu, Hyeonseo and Shin, Donghwan and Hong, Sunghwan and Narihira, Takuya and others},
  journal={arXiv preprint arXiv:2605.31595},
  year={2026}
}

The code written for this project is released under the MIT License. Third-party code included in this repository (VGGT, MoGe, DINOv2, CroCo utilities, CoWTracker, the Gaussian rasterizer, DiffSynth-Studio) remains under its own license, several of which are non-commercial; see THIRD_PARTY_LICENSES.md.

The released weights are initialized from C3G, whose backbone comes from VGGT-1B (CC BY-NC 4.0), and are released under CC BY-NC 4.0 (non-commercial use only).

We thank the authors of VGGT, MoGe, CoWTracker, 3D Gaussian Splatting, and DiffSynth-Studio for their excellent work and code.