AdvFD: Boosting Visual Generation via Adversarial Fréchet Distance Loss

Abstract

Fréchet distance has recently emerged as an effective distribution-level objective for generator post-training. However, directly optimizing it can cause Fréchet hacking: the target metric keeps improving while visual quality and Fréchet alignment in other feature spaces stagnate or deteriorate. We attribute this to the static pretrained feature spaces used by existing Fréchet losses, which give incomplete and fixed views of the gap between real and generated distributions. We propose Adversarial Fréchet Distance (AdvFD), which complements those static targets with a learnable representation that adversarially maximizes the Fréchet discrepancy while the generator minimizes it in the resulting adaptive space. To stop the representation from inflating the objective through feature amplification, we add real-feature whitening, normalizing its scale and covariance geometry to stabilize the min–max optimization. AdvFD consistently improves one-step generator post-training across JiT and pMF backbones and model scales.

Fréchet Hacking

Implicit assumption in FD-Loss: aligning generated and real samples in a static pretrained feature space could improve visual quality.

This does not always hold. Supervised only by static feature spaces, the generator corrects the discrepancies those representations detect and leaves the rest weakly constrained, so the optimized FD keeps falling while visual quality degrades. Adding more frozen encoders widens the supervision but the comparison spaces stay static, which motivates a representation that evolves with the generated distribution.

Fréchet hacking under a static Inception representation. Left: a universal perturbation optimized only for Inception FID adds visible artifacts yet cuts FID from 3.25 to 2.56, exposing a blind direction of the static representation. Right: during JiT-B post-training from 50k to 75k steps, FD-r-Inception drops 29.4% while FD-r-CLIP rises 8.5%.

Method

With \(p\) the real distribution, \(q_{\theta}\) the generator distribution, and \(\mathcal{F}\) a set of frozen encoders (SigLIP, MAE, Inception), FD-Loss defines the static objective \(D_{\mathrm{static}}(p,q_{\theta}) = \sum_{\phi \in \mathcal{F}} \lambda_{\phi} D_{\mathrm{FD}}^{\phi}(p, q_{\theta})\). We add a trainable representation \(\psi_{\omega}\), initialized from a pretrained encoder, and the generator minimizes

\[ \mathcal{L}_{t}(\theta; \omega_t) = D_{\mathrm{static}}(p, q_{\theta}) + \lambda_{\mathrm{adv}}\, D_{\mathrm{adv}}(p, q_{\theta}; \omega_t). \]

The two updates alternate: the generator reduces the discrepancy the current representation measures, and the representation raises \(D_{\mathrm{adv}}\) to expose what remains. This is a GAN-like game in feature space.

Generator_θ updating

↓

Generated

Real

↓

Static repr. φ frozen encoders

Adverse repr. ψ_ω learned

↓

real-feature whitening μ → 0, Σ → I

feature space of ψ_ω

real generated

Our adaptive training, one step at a time. In the G-step the generator minimizes the static and adaptive Fréchet objectives while both representations stay frozen, pulling the two distributions together. In the D-step the generator is frozen and the adverse representation maximizes the Fréchet discrepancy, pushing them apart to re-expose what the static encoders miss.

Real-Feature Whitening

Raw FD in a trainable feature space admits a trivial scale direction, since \(D_{\mathrm{FD}}^{c\psi_{\omega}} = c^{2} D_{\mathrm{FD}}^{\psi_{\omega}}\) for \(c>0\). The representation could then raise FD just by enlarging its feature norm. We remove this degeneracy by whitening against the real-feature statistics \(\mu_{p}^{\psi}, \Sigma_{p}^{\psi}\):

\[ \bar{\psi}_{\omega}(x) = \left(\psi_{\omega}(x) - \mu_{p}^{\psi}\right) \left(\Sigma_{p}^{\psi} + \epsilon I\right)^{-1/2}, \qquad D_{\mathrm{adv}} = D_{\mathrm{FD}}^{\bar{\psi}_{\omega}}(p, q_{\theta}). \]

Under exact whitening a common rescaling of \(\psi_{\omega}\) cancels against the change in real-feature statistics, so \(D_{\mathrm{adv}}\) cannot be increased by global rescaling alone.

Effect of feature whitening on adversarial feature stability.

Effect of whitening on feature stability. (a) Without whitening, samples degrade severely. (b) Pre-whitening RMS norm ratio of adaptive to frozen-reference features, log scale. Whitening suppresses feature-norm inflation.

Results

Class-conditional generation on ImageNet 256×256. Lower is better for all metrics.

Method	NFE	Space	#Params	FID ↓	FD-r6 ↓	FD-r3 ↓
reference (real images)
50k validation images	N/A	N/A	N/A	1.68	1.00	1.00
discrete-space models
VAR-d30	10×2	discrete	2B	1.97	6.70	6.77
BAR-L	256×2×4	discrete	1.1B	1.01	3.57	3.35
latent-space models, multi-step
without semantic distillation
SiT-XL/2	250×2	latent	675M	2.12	8.44	9.20
MAR-L	256×2×100	latent	478M	1.80	6.68	7.32
FlowAR-H	50×2^†	latent	1.9B	1.68	6.13	6.16
MAR-H	256×2×100	latent	942M	1.56	5.61	6.31
MAR-L, DeTok	256×2×100	latent	478M	1.39	5.49	6.05
with semantic distillation
REG	250×2^†	latent	685M	1.54	4.64	5.15
SiT-XL/2-REPA	250×2^†	latent	675M	1.42	5.45	6.05
LightningDiT	250×2	latent	675M	1.42	4.57	5.02
DDT-XL	250×2	latent	675M	1.26	5.70	6.38
REPA-E	250×2^†	latent	676M	1.17	3.04	3.33
RAE-XL	50×2^†	latent	839M	1.16	3.26	3.92
latent-space models, one-step
Drift-L (latent)	1	latent	463M	1.53	10.92	11.32
iMF-XL	1	latent	610M	1.82	8.39	8.72
iMF-XL	2	latent	610M	1.61	7.48	7.79
pixel-space models, multi-step backbones
JiT-B	50×2×2^†	pixel	131M	3.71	15.65	15.06
+ FD-loss	1	pixel	131M	1.00	5.53	8.45
+ AdvFD (Ours)	1	pixel	131M	0.79↓21.0%	3.92↓29.1%	6.03↓28.6%
JiT-L	50×2×2^†	pixel	459M	2.59	10.73	10.27
+ FD-loss	1	pixel	459M	0.77	3.24	5.46
+ AdvFD (Ours)	1	pixel	459M	0.73↓5.2%	2.01↓38.0%	3.20↓41.4%
JiT-H	50×2×2^†	pixel	953M	1.97	7.66	9.07
+ FD-loss	1	pixel	953M	0.75	2.65	4.44
+ AdvFD (Ours)	1	pixel	953M	0.72↓4.0%	1.80↓32.1%	2.93↓34.0%
pixel-space models, one-step
Drift-L (pixel)	1	pixel	465M	1.43	10.51	11.18
pMF-B	1	pixel	118M	3.31	13.70	11.82
+ FD-loss	1	pixel	118M	0.85	3.50	4.49
+ AdvFD (Ours)	1	pixel	118M	0.81↓4.7%	3.32↓5.1%	4.22↓6.0%
pMF-L	1	pixel	410M	2.72	9.09	7.62
+ FD-loss	1	pixel	410M	0.78	2.09	2.72
+ AdvFD (Ours)	1	pixel	410M	0.77↓1.3%	1.89↓9.6%	2.57↓5.5%
pMF-H	1	pixel	935M	2.29	6.87	6.09
+ FD-loss	1	pixel	935M	0.77	1.89	2.69
+ AdvFD (Ours)	1	pixel	935M	0.74↓3.9%	1.74↓7.9%	2.50↓7.1%

FD-loss and AdvFD use the SIM representations (SigLIP, Inception, MAE). FD-r3 excludes those training representations, so it measures generalization beyond the optimized set. Bold is best within each backbone block; percentages are relative to the FD-loss baseline. ^†For methods using interval CFG, the full-CFG NFE upper bound is reported, following FD-loss.

Qualitative comparison

Drag any handle, or focus one and use the arrow keys.

FD-Loss vs. AdvFD on ImageNet 256×256. Four representative samples per backbone, paired under the same 1-NFE budget: FD-loss on the left of each handle, AdvFD on the right. AdvFD gives cleaner textures, more coherent object structure, and fewer artifacts on both backbones.

Ablations

All on JiT-B with the SIM FD objective.

Method	FID ↓	FD-r6 ↓	FD-r3 ↓
FD-loss	1.00	5.53	8.45
FD-loss + PatchGAN	0.90	5.57	8.90
FD-loss + DMD	1.77	5.90	6.75
AdvFD	0.79	3.92	6.03

(a) Alternative remedies. External guidance is inconsistent; AdvFD improves all metrics.

Method	Branch	FID ↓	FD-r6 ↓	FD-r3 ↓
FD-loss	–	1.00	5.53	8.45
AdvFD	None	10.69	58.50	40.54
	Static	13.49	28.32	21.66
	Adaptive	0.79	3.92	6.03

(b) Whitening location. Whitening the adverse representation is required for stability.

Method	Adv. Repr.	Initialization	FID ↓	FD-r6 ↓	FD-r3 ↓
FD-loss (Baseline)	–	–	1.00	5.53	8.45
AdvFD	SigLIP	Pretrained	0.89	4.55	7.40
AdvFD	MAE	Pretrained	0.90	4.73	7.42
AdvFD	Inception	Random	0.85	5.69	7.63
AdvFD	Inception	Pretrained	0.79	3.92	6.03

(c) Adverse representation backbone and initialization. SigLIP and MAE use rank-16 LoRA; Inception is fully fine-tuned. Pretrained Inception wins on all metrics. Bold best, underline second.

BibTeX

@article{gao2026advfd,
  title         = {AdvFD: Boosting Visual Generation via Adversarial Fr\'echet Distance Loss},
  author        = {Gao, Mingju and Zhou, Jingkai and Gai, Kun and Yu, Changqian and Tang, Hao},
  journal       = {arXiv preprint arXiv:2608.11205},
  year          = {2026},
  eprint        = {2608.11205},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2608.11205}
}