Most Fréchet-distance based post-training objectives rely on fixed, pretrained feature spaces. That static view can be exploited: target metrics improve while perceptual quality or alignment under other representations stalls or worsens (a phenomenon the authors term "Fréchet hacking"). AdvFD responds by making the representation adaptive — adversarially learned to expose the largest Fréchet discrepancy — while the generator is trained to minimize that discrepancy in the resulting feature space, with a whitening constraint on real features to prevent trivial feature-amplification and to stabilize the min–max game.
Key findings
- Adversarial representation complements static targets: adding a learnable representation uncovers distribution gaps that frozen features miss, so the generator improves in more representation-diverse evaluations.
- Real-feature whitening prevents degenerate adversary behavior: normalizing scale and covariance geometry stops the adversarial head from trivially inflating distances, making optimization stable in practice.
- Consistent gains across backbones and scales: post-training with AdvFD improves one-step generator quality for JiT and pMF backbones and across model sizes, enabling stronger one-step performance without distillation or adversarial image-level losses.
How it works
AdvFD augments a conventional FD-loss by (1) maintaining one or more static pretrained feature spaces as reference statistics, and (2) training a lightweight representation network adversarially to maximize Fréchet discrepancy between real and generated feature distributions. The generator minimizes the same discrepancy computed in the combined (static + adversarial) feature space. To avoid the adversary trivially increasing norms/covariances, the method whitens real features before computing the Fréchet distance, constraining the adversary to find semantically meaningful discrepancies rather than pathological scalings.
Who it's for and tradeoffs
Great fit if you need a distribution-level post-training objective to (re)purpose multi-step generators into stronger one-step samplers, or to robustly improve visual fidelity beyond what static-FD post-training delivers. Look elsewhere if you cannot afford the extra min–max optimization complexity or if you require strictly deterministic, single-objective training pipelines; AdvFD introduces adversarial dynamics that need careful tuning and slightly more compute during post-training.