VQ Transplant

ICLR 2026

VQ-Transplant

Efficient VQ-Module Integration
for Pre-trained Visual Tokenizers

Xianghong Fang1 · Yuan Yuan2 · Dehan Kong1 · Tim G. J. Rudner1,3

1University of Toronto   2Boston College   3Vijil

Read paper ↗ Code ↗ BibTeX ↓
Can we develop new VQ methods without retrainingan entire visual tokenizer?

In our paper, we show that a new quantizer can be transplanted into a frozen, pretrained tokenizer, then aligned with just five epochs of decoder adaptation on ImageNet-1K.

95%lower training cost
21.8×faster than VAR training
0.81ImageNet rFID at 5 epochs
100%MMD-VAR codebook use
01 / Motivation

Change the quantizer.
Keep the tokenizer.

Modern visual tokenizers couple the encoder, quantizer, and decoder in one expensive training loop. Testing a new VQ method therefore means repeating adversarial training at scale.

VQ-Transplant separates quantization research from tokenizer training. It reuses the encoder and decoder of a strong pretrained model, replaces only the native VQ module, and repairs the resulting decoder mismatch with a short adaptation phase.

From scratchEncoder + VQ + DecoderOpenImages · 16 A100s · 60 hours for VAR
VQ-TransplantNew VQ + adapted DecoderImageNet-1K · 2 A100s · 22 hours
02 / Method

A two-stage transplant

The framework preserves the pretrained representation, learns a compatible codebook, and adapts only the component that must read the new latent space.

VQ-Transplant methodology pipeline
Figure 1. A pretrained tokenizer contains an encoder, a native VQ module, and a decoder. Stage I substitutes and trains the VQ module while freezing the encoder and decoder. Stage II freezes the encoder and transplanted quantizer, then adapts the decoder to the new quantized space.
Stage I

VQ module substitution

Encode images with the frozen encoder and train the new quantizer to minimize quantization error plus its method-specific uniqueness objective.

Stage II

Decoder adaptation

Freeze the encoder and new quantizer. Update the pretrained decoder with reconstruction, perceptual, and adversarial losses for five epochs.

03 / MMD-VQ

Match distributions,
not just moments.

MMD-VQ aligns encoder features and codebook vectors with Maximum Mean Discrepancy. A characteristic multi-Gaussian kernel matches the full distributions without assuming Gaussian features.

D2MMD(X, Y) = E[k(x,x′)] + E[k(y,y′)] − 2E[k(x,y)]

Wasserstein-VQ uses a Gaussian closed form that aligns first and second moments. MMD-VQ remains expressive for multimodal, heavy-tailed, and other non-Gaussian latent distributions, while retaining high codebook utilization.

04 / Results

Less training, stronger reconstruction

MMD-VAR with 8,192 codes improves on the pretrained VAR tokenizer after five adaptation epochs, using 22 total training hours.

VAR tokenizer0.92rFID · 4,096 codes
MMD-VAR0.81rFID · 8,192 codes · 5 epochs
MMD-VAR extended0.74rFID · 8,192 codes · 20 epochs
Decoder refinement

Adaptation restores high-frequency detail.

VQ substitution reduces quantization error, but the frozen decoder still expects its original latent distribution. A short adaptation phase resolves this mismatch and sharpens the reconstruction.

ImageNet-1K reconstructions before and after decoder adaptation
Figure 2. MMD-VAR reconstructions on ImageNet-1K. Top: VQ module substitution stage. Bottom: post-decoder adaptation phase.
05 / Generalization

One framework, three domains

Fixed-scale Wasserstein-VQ and MMD-VQ transplant cleanly across faces and churches, datasets structurally distinct from the tokenizer's OpenImages pretraining data.

FFHQ256 × 256
Original FFHQ inputs and Wasserstein-VQ and MMD-VQ reconstructions
Original inputs, Wasserstein-VQ reconstructions, and MMD-VQ reconstructions.
CelebA-HQ256 × 256
Original CelebA-HQ inputs and Wasserstein-VQ and MMD-VQ reconstructions
Original inputs, Wasserstein-VQ reconstructions, and MMD-VQ reconstructions.
LSUN-Churches256 × 256
Original LSUN-Churches inputs and Wasserstein-VQ and MMD-VQ reconstructions
Original inputs, Wasserstein-VQ reconstructions, and MMD-VQ reconstructions.

Across all three datasets, distribution-aligned quantizers achieve near-complete codebook utilization and preserve fine identity, texture, and structure after transplantation.

06 / Takeaways
  1. VQ-Transplant replaces a quantizer without repeating full tokenizer training.
  2. Five epochs of decoder adaptation align pretrained decoder priors with the new latent space.
  3. MMD-VQ matches non-Gaussian feature and codebook distributions without parametric assumptions.
  4. MMD-VAR reaches 0.81 rFID with 100% codebook utilization at 21.8× faster training than VAR.
07 / Citation

Build on our work

View code ↗
@inproceedings{fang2026vqtransplant,
  title     = {VQ-Transplant: Efficient VQ-Module Integration for Pre-trained Visual Tokenizers},
  author    = {Fang, Xianghong and Yuan, Yuan and Kong, Dehan and Rudner, Tim G. J.},
  booktitle = {International Conference on Learning Representations},
  year      = {2026}
}