A Latent Distribution Perspective on Evaluating and Improving Latent Generative Models

Xianghong Fang^1,* · Wenjie Shu^1,* · Tongda Xu² · Wenlong Mou¹ · Dehan Kong¹ · Tim G. J. Rudner^1,3

¹University of Toronto ²Independent ³Vijil ^*Equal contribution

Reconstruction quality need not track generative performance: the same decoder receives encoder latents during reconstruction and generator-produced latents during generation. Generation-aware reconstruction (GAR) probes decoder behavior along the transition between these distributions. Intermediate GAR latents retain source correspondence, providing paired, generation-aware supervision for decoder adaptation.

01 / Motivation

Two pathways.
Two latent distributions.

Standard reconstruction maps an image to an encoder latent z_e ∼ P_e, then asks the decoder to recover the input. Generation starts from noise and produces a generative latent z_g ∼ P_g. The same decoder receives inputs from different distributions.

rFID measures decoder behavior under P_e, while gFID depends on behavior under P_g. This latent distribution mismatch provides a perspective on why strong reconstruction can coexist with weak generation.

Overview of the reconstruction and generation mismatch, the generative trajectory, and the correlations of rFID and GAR-FID with gFID. — **Figure 1. Generation-aware reconstruction exposes the reconstruction–generation latent transition.** (a) Reconstruction decodes encoder latents z_e ∼ P_e, while generation decodes generative latents z_g ∼ P_g. (b) GAR constructs a trajectory of intermediate distributions P_g^t from reconstruction toward generation. (c) In this setting, rFID is weakly correlated with gFID, with Spearman ρ = −0.34. (d) GAR-FID probes decoder behavior on intermediate latents and reaches ρ = 0.99 at η_t = 0.8.

Comparing reconstruction and generation alone leaves the transition between them unobserved. GAR reveals how decoder behavior changes as its inputs move from encoder latents toward generation-time distributions.

02 / Method

Reconstruction along
the generative trajectory

GAR constructs a controlled trajectory for diagnosing decoder behavior. It perturbs encoder latents, denoises them with the frozen generative model, and probes the decoder on the resulting intermediate latents.

The noise level η_t controls the position along the trajectory: η_t = 0 recovers standard reconstruction, and η_t = 1 recovers generation from pure noise. As η_t increases, correspondence with the source image gradually weakens.

GAR pipeline with encoder, noise injection, generative model, and decoder.

01 / Encode

Start from paired data

Map each image x to an encoder latent z_e ∼ P_e.

02 / Perturb

Choose a noise level

Construct z_noisy^t = a_tz_e + b_tε.

03 / Denoise

Enter the trajectory

Apply the generative model to obtain z_g^t ∼ P_g^t.

04 / Evaluate

Measure GAR-FID

Compute FID between decoded GAR outputs and the source image set. Vary the noise level to track decoder behavior along the trajectory.

03 / Distributional gap

A measurable path from P_e to P_g

Latent-FD measures the gap between latent distributions in a shared latent space. In the SD-VAE and iMF-B/2 experiment, increasing η_t moves P_g^t away from P_e and closer to P_g.

The two distances trace opposite directions across the same GAR trajectory.

Latent-FD curves comparing intermediate GAR distributions with encoder and generative distributions. — Figure 4. Latent-FD from intermediate GAR distributions P_g^t to the reconstruction endpoint P_e and generation endpoint P_g. The distances show the progressive shift toward the generation-time regime.

04 / Evaluation

Tracking decoder behavior
toward generation

GAR-FID tracks decoder behavior along the latent trajectory. Across 13 VAEs and two SiT generator scales, its empirical correlation with gFID strengthens as the trajectory approaches P_g, while rFID remains negatively correlated.

Table 1

Correlation with generative performance

Metrics	gFID SiT-B				gFID SiT-XL				gFID SiT-mixed
	w/o CFG		w/ CFG		w/o CFG		w/ CFG		w/o CFG		w/ CFG
	PCC ↑	SRCC ↑	PCC ↑	SRCC ↑	PCC ↑	SRCC ↑	PCC ↑	SRCC ↑	PCC ↑	SRCC ↑	PCC ↑	SRCC ↑
rFID^★	−0.11	−0.36	−0.11	−0.30	−0.13	−0.34	−0.15	−0.39	−0.10	−0.30	−0.10	−0.28
iFID^★	0.86	0.83	0.80	0.79	0.90	0.88	0.88	0.90	0.72	0.77	0.64	0.70
GAR-FID, η_t = 0.6	0.81	0.79	0.77	0.78	0.79	0.84	0.80	0.79	0.86	0.89	0.84	0.85
GAR-FID, η_t = 0.7	0.93	0.90	0.89	0.87	0.91	0.90	0.91	0.86	0.95	0.96	0.93	0.91
GAR-FID, η_t = 0.8	0.98	0.96	0.96	0.94	0.98	0.99	0.97	0.96	0.99	0.99	0.97	0.95
GAR-FID, η_t = 0.9	1.00	0.99	0.98	0.98	1.00	1.00	0.98	0.96	1.00	1.00	0.98	0.97
GAR-FID, η_t = 1.0	1.00	0.99	0.98	0.99	1.00	1.00	0.98	0.96	1.00	1.00	0.98	0.97

PCC denotes Pearson correlation and SRCC denotes Spearman rank correlation. Higher is better. ★ marks our reproduced results. Bold values indicate the best results.

At η_t = 0.8, GAR-FID reaches SRCCs of 0.94–0.99 across the reported settings, including 0.99 for SiT-XL and the mixed setting without CFG. These results support GAR-FID as a trajectory-based diagnostic; direct gFID evaluation remains necessary when final sample quality is the target.

05 / Structural fidelity

How does source structure change?

GAR outputs reveal the progressive loss of source structure as the trajectory moves from reconstruction toward generation.

GAR outputs at increasing noise levels showing progressive loss of source structure. — Figure 8. Progressive loss of source structure along the GAR trajectory. Each row shows an input followed by GAR outputs at η_t ∈ {0.0, 0.1, 0.2, 0.4, 0.6, 0.8, 1.0}. Fine structure and local texture gradually degrade. At high noise levels (e.g., η_t ≥ 0.6 in these examples), outputs can remain semantically plausible while no longer preserving the source structure.

This progression exposes the trade-off between generation awareness and source correspondence. Intermediate noise levels provide generation-relevant latents while retaining meaningful paired supervision for decoder adaptation.

06 / Decoder adaptation

From diagnosis to
decoder adaptation

Intermediate GAR latents move toward generation-time distributions while retaining correspondence with their source images. Decoder adaptation uses this paired supervision to fine-tune only the decoder, with the encoder and generative model frozen and no additional inference cost.

We compare DA w/o CFG, trained for 10 epochs on GAR latents constructed without classifier-free guidance (CFG), with CFG-aware DA, trained for one epoch on GAR latents constructed with CFG. In both settings, only the decoder is updated.

Table 2

Effect of decoder adaptation on gFID

Protocol / adaptation setting	gFID w/o CFG ↓				gFID w/ CFG ↓
Protocol / adaptation setting	iMF-B/2	iMF-M/2	iMF-L/2	iMF-XL/2	iMF-B/2	iMF-M/2	iMF-L/2	iMF-XL/2
Reported iMFWithout DA^†	—	—	—	—	3.39	2.27	1.86	1.72
Official iMFWithout DA^‡	16.58	11.92	9.28	9.78	3.37	2.27	1.86	1.73
Official iMFDA w/o CFG (10 epochs)^‡	12.21↓4.37	9.52↓2.40	7.34↓1.94	7.47↓2.31	2.90↓0.47	2.12↓0.15	1.65↓0.21	1.56↓0.17
Official iMFCFG-aware DA (1 epoch)^‡	—	—	—	—	2.81↓0.56	1.98↓0.29	1.58↓0.28	1.50↓0.23
OpenAIWithout DA^★	16.41	11.92	9.26	9.41	3.47	2.40	1.90	1.80
OpenAIDA w/o CFG (10 epochs)^★	12.41↓4.00	9.66↓2.26	7.41↓1.85	7.61↓1.80	3.12↓0.35	2.30↓0.10	1.80↓0.10	1.73↓0.07
OpenAICFG-aware DA (1 epoch)^★	—	—	—	—	3.01↓0.46	2.19↓0.21	1.70↓0.20	1.61↓0.19

Lower gFID is better. † denotes results reported in iMF; ‡ denotes our reproduction with the official iMF protocol; ★ denotes our reproduction with the OpenAI protocol. ↓ shows the reduction from the corresponding protocol's reproduced baseline without DA. — denotes results not reported in the paper. Column groups indicate CFG at evaluation; row labels indicate CFG during adaptation.

One epoch of CFG-aware DA outperforms 10 epochs of DA w/o CFG on guided generation at every tested iMF scale under both protocols. For iMF-XL/2, guided gFID improves from 1.73 to 1.50 under the official iMF protocol and from 1.80 to 1.61 under the OpenAI protocol, with no additional inference cost.

The same one-epoch CFG-aware adaptation also improves MeanFlow-XL/2 from 3.43 to 2.96 and DMF-XL/2+ from 2.16 to 1.90 under the official iMF evaluation protocol.

Qualitative comparison

Identical latents, adapted decoder

Each column uses the same generative latent. The top row uses the original decoder; the bottom row uses the adapted decoder.

Indigo bunting samples before and after decoder adaptation. — Class 14: indigo bunting Top: before. Bottom: after.

Ptarmigan samples before and after decoder adaptation. — Class 81: ptarmigan Top: before. Bottom: after.

Flamingo samples before and after decoder adaptation. — Class 130: flamingo Top: before. Bottom: after.

Sports car samples before and after decoder adaptation. — Class 817: sports car Top: before. Bottom: after.

Adaptation sharpens fine details, reduces artifacts, and improves semantic consistency without changing the generative latent.

07 / Takeaways

Reconstruction and generation use the same decoder under different latent distributions, P_e and P_g, helping explain their performance discrepancy.
GAR constructs and probes intermediate distributions P_g^t, making decoder behavior along the reconstruction–generation transition observable and diagnosable.
GAR-FID provides a trajectory-based diagnostic with strong empirical correlation with gFID near the generation endpoint across the evaluated tokenizers and generator scales.
Intermediate GAR latents retain source correspondence, enabling paired supervision for decoder adaptation. One epoch of CFG-aware adaptation improves guided gFID at every tested iMF scale without additional inference cost.

08 / Citation

Build on our work

View code ↗

@article{fang2026latent,
  title   = {A Latent Distribution Perspective on Evaluating and Improving Latent Generative Models},
  author  = {Fang, Xianghong and Shu, Wenjie and Xu, Tongda and Mou, Wenlong and Kong, Dehan and Rudner, Tim G. J.},
  journal = {arXiv},
  year    = {2026}
}