inferwire
/
AI·4 min read

ParetoSlider: One Diffusion Model, a Slider for Competing Goals

Researchers train a single image or video model that lets users shift the balance between competing goals at generation time, instead of fixing one trade-off during training.

TL;DR

  • ParetoSlider fine-tunes one diffusion model so users can set the balance between competing rewards, such as edit strength versus source fidelity, at generation time [^1].
  • On Stable Diffusion 3.5, FluxKontext and LTX-2, the single model matched or beat baselines trained separately for each fixed trade-off, the authors report [^1].

Background

Image and video generators are often fine-tuned with reinforcement learning (RL): the model produces samples, reward models score them, and training pushes the model toward higher scores [^1]. When several criteria matter, common practice merges the rewards into one fixed weighted sum, which the authors call "early scalarization" [^1]. That locks in a single trade-off at training time, for example between following an edit instruction and staying faithful to the source image, with no way to change it later [^1].

What happened

Researchers from Tel Aviv University, Lightricks and Adobe Research present ParetoSlider, which trains one diffusion model to approximate the whole Pareto front: the set of trade-offs where no criterion can be improved without degrading another [^1]. During training, each prompt gets a randomly drawn preference vector, a set of non-negative weights that sum to one, and the model receives it as an extra input. The model then generates a group of samples, and every reward model scores each one [^1]. Rewards with naturally large values or variance could drown out the others, a problem the authors call "reward hijacking". To prevent it, scores are normalized separately for each reward and combined with the weights only at the final loss step, which the paper calls late scalarization [^1]. The method builds on DiffusionNFT, an existing RL fine-tuning framework, and adds small conditioning modules trained alongside LoRA adapters [^1].

The team tested three backbones: Stable Diffusion 3.5 for text-to-image, FluxKontext for instruction-based image editing and LTX-2 for text-to-video [^1]. The trade-offs included photorealism versus styles such as sketch, watercolor or anime, source preservation versus instruction adherence, and animation versus photorealism in video [^1]. Baselines included Fixed-Weights, which needs a separate training run for each trade-off point, Flow-Multi, which learns one static policy, prompt rewriting by a language model and, for editing, sweeps of the text and image guidance scales [^1].

On the photorealism-versus-sketch task, ParetoSlider's front consistently dominated all baselines, and Fixed-Weights tended to collapse toward the dominant reward [^1]. In the supplementary hypervolume tables, a standard measure of how well a set of results covers the trade-off space, ParetoSlider scored 0.870 against 0.827 for prompt rewriting, 0.683 for Flow-Multi and 0.435 for Fixed-Weights [^1]. For image editing, it scored 0.574 against 0.561 for text guidance and 0.395 for image guidance [^1].

Why it matters

The practical gain is one checkpoint instead of many. Other approaches to inference-time control interpolate between separately trained models, so storage grows linearly with the number of objectives, or steer generation without training at a heavy per-step cost [^1]. ParetoSlider needs no retraining, model interpolation or per-step gradient guidance at inference: the user sets the weights and the model samples as usual [^1]. The authors see this as a path to control interfaces that let non-expert users navigate reward trade-offs [^1].

The margins vary, though. In editing, the hypervolume lead over plain text guidance is small, and in the ablations early and late scalarization recovered a similar overall front; late scalarization's advantage was smoother intermediate steps, where the alternatives tended to collapse toward photorealism [^1]. The method also inherits its reward models' flaws. The authors give the example of a sketch reward that responds to grayscale images rather than genuine sketch style, and warn that one poorly specified reward can distort the entire trade-off surface [^1].

Practical example

Say Noa runs a small studio that turns client portraits into fantasy characters with an image editing model. One client wants to stay recognizable; another wants the full armored-warrior look. With a model fine-tuned on one fixed weighting, Noa gets a single balance between keeping the face and following the instruction. Getting a different one means another fine-tuning run; the authors trained five separate models for five points in their own comparison [^1].

Turning the usual dials has side effects. Raising the image guidance scale preserves more of the source but adds visual artifacts, and raising the text guidance scale tends to make images look more saturated [^1].

With a ParetoSlider-trained model, Noa loads one checkpoint and sets a preference value per job [^1]. Near full preservation, fine identity details such as freckles stay intact; as the weight moves toward instruction adherence, the identity shifts slightly and those details fade under the stronger edit, much like the paper's warrior example [^1]. For the first client, Noa stays near the balanced midpoint; for the second, Noa slides toward the edit. Noa still checks each result, since the slider is only as good as the reward models it was trained on [^1].

Related gear

We recommend this book because it explains reinforcement learning from the ground up, the training method ParetoSlider uses to fine-tune image and video models.

AdvertisementAmazon

Reinforcement Learning: An Introduction

Sources

  1. [1]arXiv — ParetoSlider: Diffusion Models Post-Training for Continuous Reward Control