DREAM: Unifying Contrastive and Generative Objectives

The challenge

Two objectives, opposing masking regimes

Text-image contrastive learning and masked generative modeling are typically trained in separate models. Unifying them in a single end‑to‑end model runs into a fundamental conflict over the masking ratio.

Contrastive alignment

Requires near-complete visibility

Image-text alignment degrades as the masking ratio grows. Heavy masking starves the encoder of the global semantics that contrastive learning depends on.

vs.

Masked generative modeling

Requires heavy corruption

Generative models such as MAR train on masking ratios well above 50%. Lightly masked inputs provide too weak a reconstruction signal.

Our approach

Masking Warmup

Rather than committing to one regime, DREAM samples masking ratios from a broad distribution whose mean moves from fully visible to fully masked over training. Low and high ratios coexist at every step. This co-exposure stabilizes joint optimization and lets one encoder serve both objectives. By contrast, a fixed high masking ratio collapses ImageNet linear probing to 4.6%.

As training progresses, the distribution slides from fully visible to fully masked but stays wide, so every batch mixes lightly and heavily masked images. Lightly masked samples (≤ 75%) train the contrastive loss, and heavily masked samples (> 50%) train the diffusion loss. Details: truncated Gaussian with σ = 0.45, whose mean rises linearly over the first 200K steps.

Method

A unified encoder–decoder over continuous latents

Hover over a step to see how that stage is built up.

1

Tokenize & mask

Images are encoded into continuous latents by the Stable Diffusion VAE. Tokens are masked at ratios drawn from the Masking Warmup schedule, and masked positions are dropped.

2

Contrastive alignment

A MAR-style ViT encoder processes only the visible tokens (plus learnable buffer tokens). Its mean-pooled embedding is aligned with a CLIP text encoder through a symmetric InfoNCE loss.

3

Diffusion reconstruction

A FLUID-style decoder cross-attends to frozen T5-XXL embeddings and predicts the masked latents with a per-token MLP diffusion head. Text conditioning is confined to the decoder, so the encoder cannot rely on language shortcuts.

Semantically Aligned Decoding

Trajectory selection from 12.5% of the image

Joint optimization enables an inference-time capability that decoupled models cannot replicate. Because the text encoder is trained against visual embeddings across the full range of masking ratios, it can reliably score partially decoded images.

Semantically Aligned Decoding spawns K candidate trajectories, decodes each to an intermediate step, scores their sparse visual embeddings against the prompt, and completes only the top-ranked candidate. External CLIP rerankers are trained only on fully visible images, so they must generate every candidate to completion before selecting one.

Hover over a step to see how that stage is built up.

1

Spawn & partially decode

The decoder runs K candidate trajectories for the same prompt in parallel and halts each at an intermediate step, with as little as 12.5% of tokens decoded.

2

Score with the joint encoder

DREAM's vision encoder embeds each candidate from its decoded tokens alone. Cosine similarity with the prompt's CLIP-style text embedding yields an alignment score.

3

Select & complete

Only the top-scoring trajectory is decoded to completion, conditioned on T5-XXL, and detokenized by the VAE. The remaining K−1 are discarded before incurring full decoding cost.

7.0%

FID improvement over standard decoding

+5.6%

FID vs. external CLIP reranking

+10.1%

throughput vs. external CLIP reranking

"Watercolor dog portrait"

Fewer artifacts

"Sunset over ocean waves"

More defined waves

"Busy market street in sunlight"

Cleaner details, less blur

Qualitative comparison of DREAM samples with and without Semantically Aligned Decoding (SD) for identical prompts.

Results

Synergy, not trade-off

Joint training improves both understanding and generation over single-objective baselines.

In a controlled comparison on CC12M, where all models share the same encoder–decoder architecture and differ only in objective, DREAM (yellow) outperforms CLIP, MAR, FLUID and REPA on every axis: linear probing, fine-tuning, few-shot transfer, semantic segmentation, FID and CLIP Score.

The representation gains extend to dense prediction, with +1.9% on ADE20K segmentation and +6.25% on NYU depth estimation over CLIP.

Representations that transfer

The encoder's gains are not confined to ImageNet. In the same controlled setting, DREAM outperforms every baseline on few-shot transfer and on robustness after fine-tuning, and matches or beats them on dense prediction with a frozen backbone. Each chart cycles through the baselines, and you can also pick one.

Few-shot transfer

5-way 5-shot accuracy, DREAM minus each baseline (points)

DREAM leads every baseline on average: +4.1 over CLIP, +18.4 over REPA, +30.0 over FLUID and +31.0 over MAR. Against the strongest baseline, CLIP, it matches or wins on 13 of 14 datasets, with the largest gains on scene and object recognition (VOC2007), remote sensing (EuroSAT, UC Merced) and textures (DTD).

On fine-grained recognition it improves on CLIP for Stanford Dogs (+3.0) and Food-101 (+0.9), and trails only slightly on Flowers (−0.3). It outperforms REPA, FLUID and MAR on all 14 datasets.

Robustness after fine-tuning

Top-1 accuracy after ImageNet fine-tuning, DREAM minus each baseline (points)

After fine-tuning on ImageNet, DREAM generalizes best under distribution shift, beating every baseline on each benchmark shown. Averaged over all 9 benchmarks it leads CLIP by +2.8, REPA by +2.4, FLUID by +6.1 and MAR by +6.0, with the largest gains on the hardest shift, ImageNet-A.

Model	ADE20K mIoU ↑	NYU-v2 RMSE ↓
MAR	23.4	0.75
FLUID	22.1	0.76
CLIP	34.9	0.64
REPA	32.7	0.60
DREAM	36.8	0.60

Dense prediction

Frozen backbone, linear-probe protocol

The representations also carry spatial detail. With the encoder frozen, DREAM gives the best ADE20K segmentation (+1.9 mIoU over CLIP) and ties REPA for the best NYU-v2 depth estimation.

Robust to missing information, and it scales

Because it is trained across the full range of masking ratios, DREAM's encoder still recognizes images when most of the input is missing. Understanding and generation also improve together as the model grows.

Robustness to masking

Zero-shot accuracy (%) as the input is increasingly masked

Zero-shot accuracy versus masking ratio. At 80% masking, DREAM achieves 6.2× the accuracy of CLIP (21.6% vs. 3.5%). DREAM also outperforms CLIP trained with the same schedule (CLIP-M) at every masking level, which points to the locally grounded representations learned through diffusion reconstruction.

Scaling behavior

ImageNet linear probing (%) ↑

CC12M FID ↓

Both metrics improve consistently from DREAM-B to DREAM-G (σ = 0.45). DREAM overtakes CLIP on linear probing from the L scale onward, and FLUID stays far below both (40.7–50.2%). With Semantically Aligned Decoding, FID improves from 5.56 to 3.62, the best result at every model size.

Gallery

Text-to-image samples

Samples from DREAM at three model scales, trained on CC12M. The prompts are identical across scales, so switching between them shows how generation quality improves with model size. Prompts are CC12M captions (abbreviated). CFG = 5.0.

Church of the Good Shepherd at night with the Milky Way

Hunas Falls: the starting point of the waterfall

Egg drop soup topped with scallions and fried noodles

Fruit still life with two peaches and grapes on the vine

A snow-covered abandoned mall, forgotten by time

Abstract

Unifying text-image contrastive learning and text-to-image (T2I) generation in a single end-to-end model is challenging because the two objectives demand opposing masking regimes: contrastive alignment needs near-complete visible tokens, while masked generative modeling needs heavy corruption. We introduce DREAM, a unified framework that resolves this conflict through Masking Warmup, a schedule that shifts the center of the masking distribution over training, so low and high masking ratios coexist at every step. This co-exposure lets a single jointly-trained encoder serve both objectives. The resulting stable optimization unlocks Semantically Aligned Decoding at inference: the text encoder, trained against visual embeddings at all masking ratios, can score partially generated images and select the best trajectory with as little as 12.5% of the image decoded, improving both FID and throughput. DREAM outperforms its single-objective baselines, CLIP and FLUID: on ImageNet linear-probing (+1.1%), 5-shot transfer (+4.1%), ADE20K segmentation (+1.9%), and NYU depth estimation (+6.25%) over CLIP, and on CC12M FID (+6.2%) over FLUID while maintaining CLIP Score. Together, these gains show that text-image contrastive and generative objectives, when properly unified, are synergistic rather than competing.

BibTeX

@misc{li2026unifyingcontrastivegenerativeobjectives,
  title         = {Unifying Contrastive and Generative Objectives for
                   Visual Understanding and Text-to-Image Generation},
  author        = {Chao Li and Tianhong Li and Sai Vidyaranya Nuthalapati and
                   Hong-You Chen and Satya Narayan Shukla and Jianpeng Cheng and
                   Yonghuan Yang and Jun Xiao and Xiangjun Fan and Aashu Singh and
                   Dina Katabi and Shlok Kumar Mishra},
  year          = {2026},
  eprint        = {2603.02667},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2603.02667}
}