PreResQ-R1: Towards Fine-Grained Rank-and-Score Reinforcement Learning for Image Quality Assessment via Preference-Response Disentangled Policy Optimization

Zehui Feng1, Tian Qiu1, Tong Wu1, Huayuan Xu1, Juanxuan Li1, Ting Han1,2†
1Shanghai Jiao Tong University, 2Zhejiang University, †Corresponding Author
Paper Code

🤗

 Checkpoints
data-overview

Overall Performance.
(a) Existing score/ranking reward function assign minimal difference, which results in distribution fall or robustness fail.
(b) PreResQ-R1 focus on fine-grained response-ranking reward balance and preference.
(c) PreResQ-R1 enables state-of-the-art performance and stable image quality assessment with discriminative reward.
(d) typical qualitative and quantitative example comparison between VisualQuality-R1 and PreResQ-R1, which demonstrates superior performance on image quality describe and score.

Abstract

Visual Quality Assessment (QA) seeks to predict human perceptual judgments of visual fidelity. While recent multimodal large language models (MLLMs) show promise in reasoning about image and video quality, existing approaches mainly rely on supervised fine-tuning or rank-only objectives, resulting in shallow reasoning, poor score calibration, and limited cross-domain generalization. We propose PreResQ-R1, a Preference–Response Disentangled Reinforcement Learning framework that unifies absolute score regression and relative ranking consistency within a single reasoning-driven optimization scheme. Unlike prior QA methods, PreResQ-R1 introduces a dual-branch reward formulation that separately models intra-sample response coherence and inter-sample preference alignment, optimized via Group Relative Policy Optimization (GRPO). This design encourages fine-grained, stable, and interpretable chain-of-thought reasoning about perceptual quality. To extend beyond static imagery, we further design a global–temporal and local–spatial data flow strategy for Video Quality Assessment. Remarkably, with reinforcement fine-tuning on only 6K images and 28K videos, PreResQ-R1 achieves state-of-the-art results across 10 IQA and 5 VQA benchmarks under both PLCC and SRCC metrics. Beyond quantitative gains, it produces human-aligned reasoning traces that reveal the perceptual cues underlying quality judgments. Code and model are available.

Method Overview

pipeline

🌟🌟🌟 Overall training framework of PreResQ-R1 via RL2RS. Given an sample batch with a shared text prompt, PreResQ-R1 generates K responses.
🌟🌟🌟 To quickly activate CoT differences and then access generation stability, we introduce the response penalty and fine-grained triplet-response balance reward.
🌟🌟🌟 To jointly enhance the robustness of ranking and score ability, we introduce the preference pairwise-and-triplet score-and-ranking reward for GRPO.

PRPO

pipeline

🌟🌟🌟 Pipeline of the Preference-Response Disentangled Policy Optimization (PRPO), which applies response ranking response balance reward, and preference pairwise score and ranking reward, and preference triplet ranking reward to optimize group policy learning.

Qualitative Performance Cases

pipeline

Answer Distribution

🌟🌟🌟 Comparison between PreResQ-R1 and VisualQuality-R1 on distribution of difference between answer and ground truth. The horizontal axis represents the error, and the vertical axis represents the relative proportion. The closer the distribution is to 0, the better the model performance is. Blue and Orange represents PreResQ-R1 and VisualQuality-R1.

pipeline

Data Flow

pipeline

More Qualitative IQA Cases on Comparison.

pipeline

More Qualitative IQA Cases on Ablation.

pipeline

More Qualitative IQA Cases on Training Steps.

pipeline

More Qualitative VQA Cases on Comparison.

pipeline

More Qualitative VQA Cases on Ablation.

pipeline

More Qualitative VQA Cases on Training Steps.

pipeline

BibTeX



@article{Bratrix,
  title={PreResQ-R1: Towards Fine-Grained Rank-and-Score Reinforcement Learning for Image Quality Assessment via Preference–Response Disentangled Policy Optimization},
  author={Zehui Feng, Tian Qiu, Tong Wu, Huayuan Xu, Ting Han},
  journal={arXiv preprint arXiv:},
  year={2025}
}