QwenLM Drops Qwen-Image-Bench to Grade AI Art Like a Pro

Qwen-Image-Bench is a new evaluation toolkit that scores images from any text-to-image model using a fine-tuned judge AI called Q-Judger. It checks generated images across five major quality dimensions, including realism, alignment, and creative expression, covering 56 specific checkpoints. The results appear as structured scores and detailed reports rather than a single vague number.

The project comes from the QwenLM team, who co-designed the benchmark with professional artists to reflect real-world creative workflows. They trained Q-Judger on Qwen3.6-27B and validated its ratings against 80 expert annotators, achieving strong agreement with human judgment. This release provides an open-source alternative for evaluating image generation models without relying on proprietary cloud tools.

Detailed scoring across five creative pillars

Local AI users can objectively compare image models without sending data to the cloud. Small agencies benefit from benchmarking different models for client projects that demand specific realism or creative flair. Privacy-conscious pros run the full evaluation pipeline on their own hardware, keeping all prompts and images confidential.

What developers should know

The Q-Judger model requires a GPU with ample memory due to its 27B parameter size, but included batch processing helps manage hardware constraints. Its ratings align strongly with human experts, showing high Spearman correlation values above 0.89 across all quality dimensions. No quantized versions exist yet, which currently limits practical use on many consumer GPUs.

"The benchmark uses a 3-level hierarchical scoring system with 5 L1 dimensions, 23 L2 sub-capabilities, and 56 L3 facets" — Source: GitHub