Given a target dataset, such as faces with eyeglasses, and a background dataset, such as faces without, contrastive analysis separates salient factors specific to the target from common content shared by both. We aim for salient representations that capture target-specific detail in each image, such as the shape, color, and position of the glasses, so that they reveal subtypes without subtype labels and guide the generation of new examples of a discovered subtype, even one with no name or text description.
We introduce SAGE, which learns both factors directly in the high-dimensional spatial latent of a frozen representation autoencoder and conditions a diffusion transformer on the learned salient representation of a reference image. On Digits-ImageNet and FFHQ eyeglasses, SAGE combines high-fidelity reconstruction (rFID below 2) with unsupervised subtype discovery, recovering the digits better than baselines (probe accuracy 0.950 vs. at most 0.281) and revealing eyewear types, finer sunglasses styles, and mislabeled images; salient-conditioned generation raises Digits-ImageNet subtype accuracy over the unfactorized latent (90.5% vs. 27.7%) and diversity on both datasets. On retinal OCT, SAGE's salient space separates three diseases using only normal/disease labels.
The approach
Learn the difference between two datasets, then use that difference to generate new examples.
A frozen representation autoencoder provides a rich spatial latent. SAGE learns separate common and salient encoders in this space: common factors capture shared content, while salient factors capture what is specific to the target dataset. Reconstruction, adversarial swapping, sparsity, and cycle consistency encourage the two factors to stay informative and distinct.
The learned salient representation of a reference image is globally max-pooled and used to condition a diffusion transformer. This guides generation toward the reference's digit or eyewear subtype while allowing the surrounding scene or face to vary.
Experiments
Evaluated on Digits-ImageNet, FFHQ eyeglasses, and retinal OCT.
| Method | Digits-ImageNet | FFHQ | ||||
|---|---|---|---|---|---|---|
| rFID ↓ | Probe acc. ↑ | ARI / NMI ↑ | rFID ↓ | Probe acc. ↑ | ARI / NMI ↑ | |
| cVAE | 151.0 | 0.135 | 0.001 / 0.003 | 188.2 | 0.898 | 0.001 / 0.001 |
| SepVAE | 162.5 | 0.145 | 0.009 / 0.019 | 197.4 | 0.897 | −0.001 / 0.000 |
| Double-InfoGAN | 171.4 | 0.281 | 0.001 / 0.003 | 122.5 | 0.964 | 0.298 / 0.299 |
| SepCLR | N/A | 0.180 | 0.000 / 0.001 | N/A | 0.970 | 0.822 / 0.696 |
| DINOv3 + SepCLR | N/A | 0.148 | 0.000 / 0.001 | N/A | 0.976 | 0.915 / 0.825 |
| SAGE (Ours) | 1.78 | 0.950 | 0.337 / 0.472 | 1.65 | 0.983 | 0.939 / 0.865 |
Lower rFID indicates better reconstruction. Probe accuracy, ARI, and NMI evaluate subtype information in the salient representation. FFHQ labels are audit-corrected. Double-InfoGAN uses 128 × 128 images; other methods use 256 × 256. N/A indicates no decoder. See the paper for the complete evaluation protocol.
SAGE recovers digit identity and eyewear subtypes using only background/target dataset membership during training. The salient space also reveals finer sunglasses styles and a cluster of mislabeled FFHQ images.
Exploring the factors
Manipulate salient attributes while retaining the common content of an image.
From discovery to generation
Conditioning on the separated salient factor encourages diverse samples that share the reference's target attribute.
| Dataset | Conditioning | Subtype acc. ↑ | Vendi ↑ | Cos-to-ref ↓ |
|---|---|---|---|---|
| Digits-ImageNet | Raw DINOv3 GMP | 27.70% | 2.65 | 0.794 |
| Digits-ImageNet | SAGE Salient GMP | 90.48% | 12.50 | 0.116 |
| FFHQ | Raw DINOv3 GMP | 98.5% | 2.32 | 0.779 |
| FFHQ | SAGE Salient GMP | 96.6% | 6.96 | 0.470 |
GMP: global max pooling. Vendi measures diversity across 16 samples per reference; cosine similarity to the reference uses DINOv2-L features. On FFHQ, SAGE increases diversity while subtype accuracy is slightly lower than raw DINOv3 conditioning. Full results, including generation FID, are in the paper.
Beyond natural images
Using only normal/disease training labels, SAGE's salient space separates CNV, DME, and DRUSEN.
@misc{liang2026sage,
title={{SAGE}: Salient Factor Discovery and Generation with Visual Foundation Representations},
author={Liang, Shuang and Liao, Lejun and Zhang, Shiyuan and Zhang, Max C. and Luo, Xiaolong and Wang, Han and Anzellotti, Stefano and Yuan, Yuan},
year={2026},
eprint={2609.39635},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2609.39635}
}