SVG-Score

Human-Aligned Evaluation of Text-to-SVG Generation

Marco Cipriano1†, Leonardo Zini2†, Alexandra Schild1, Valentin Teutschbein1, Afsana Mimi1, Marcella Cornia2, Lorenzo Baraldi2, Gerard de Melo1

1 Hasso Plattner Institute, Germany  ·  2 University of Modena and Reggio Emilia, Italy

† Equal contribution.

arXiv Code soon Models soon Data soon
8,671SVGs annotated with human preference, across 1,858 captions
3.2MSVG–caption pairs for domain adaptation
4evaluators: 3 CLIP + 1 VLM
16SVG generators evaluated on our new 1,616-caption benchmark

Abstract

SVG generation is attracting increasing attention as generative models improve in expressiveness and controllability. Progress, however, is held back by the lack of domain-specific evaluation protocols: current practice relies on metrics designed for natural images, most notably CLIPScore, which was never trained on vector graphics and aligns only partially with human judgment.

We introduce SVG-Score, a human-aligned evaluation framework for text-to-SVG generation. Through controlled caption and image perturbations, we first show that CLIP-based scores barely react to the errors SVG generators actually make — wrong colors, counts and spatial relations — and that off-the-shelf VLM judges, while more sensitive, respond unevenly across error types and SVG styles. We then introduce a human-annotated dataset for Semantic Alignment, measuring how faithfully a generated SVG reflects its caption, and build two complementary evaluators on it: CLIP scorers adapted to vector graphics and aligned to human preferences, for fast large-scale evaluation, and a VLM judge trained with supervised fine-tuning and reward-shaped reinforcement learning, for more expressive and interpretable assessment. Using both, we benchmark major open-source, commercial and optimization-based SVG generators on an independent caption set.

The problem

CLIP misses the errors SVG generators make

We take one SVG and the caption that belongs to it. We change the colour, the position or the count in the caption, and CLIPScore barely moves. We throw the drawing away instead, and it falls by 14 points.

OmniSVG sample · original SVG

CLIPScore  ViT-B/32
36.47
reference pair

Change in CLIPScore relative to the correct pair

Moving the splashes below the whale costs 0.32 points. Turning four splashes into seven costs 0.19. Only colour registers, at 3.32. A plain blue square costs 13.75. CLIP sees that the picture changed, not that the caption became false.

Our answer

What we built

We label caption–SVG pairs by hand and train four evaluators on them. Below is how well each one agrees with our annotators on the held-out test split (Spearman ρ ×100).

Dataset

Semantic Alignment ratings

12,957ratings · 5 annotators

8,671 SVGs and 1,858 captions, each pair rated by one of five annotators. A 1 means the SVG is unrelated to the caption, a 5 means it is a faithful rendering of it.

Evaluator 1 · embedding

SVG-adapted CLIP, preference-aligned

42.90→63.18ρ

We fine-tune CLIP on 3.2M SVG–caption pairs, then train LoRA adapters on the human preferences. Scoring is still one forward pass, so it runs over a whole benchmark cheaply.

Evaluator 2 · generative

Human-aligned Qwen3-VL-8B judge

67.76→74.85ρ

Qwen3-VL-8B, taught to write a short rationale and a score, then trained with GRPO on two rewards. It agrees with our annotators more closely than any judge we tried, open or commercial.

Analysis

Analysis of CLIP limitations

We built five perturbations and ran them on 200 samples. Three edit the caption and keep the SVG; two keep the caption and replace the SVG. An evaluator that reads meaning should score lower in all five cases.

Caption

Colour swap

Each mentioned colour is replaced by an alternative; averaged over five variants.

a light blue green whale…
Caption

Spatial swap

Spatial and relational terms are replaced by their semantic opposite.

Above Below the whale's head…
Caption

Count swap

Each quantity q becomes q−1, q+1 and one larger perturbation.

there are four seven splashes…
Image

Coloured circles

The SVG is replaced by one coloured circle per colour named in the caption.

Image

Solid background

The SVG is replaced by a uniform canvas in one colour named in the caption.

Off-the-shelf evaluators

mean Δ · 200 SVGs

Extra sensitivity after our training

% of score range

Left: the mean score change when a pair is corrupted. CLIP is in CLIPScore points, the VLM on the 1–5 scale, so the two are not comparable. Right: how much further each of our evaluators drops than its own baseline. Our CLIP mainly gets better at noticing the picture is gone; our VLM judge gains on all three caption edits.

Method

Two evaluators, one training set

We trained three CLIP scorers and one VLM judge on the same human ratings. Use CLIP as a fast in-training signal, cheap enough to score every checkpoint. Use the VLM for the exhaustive final evaluation, where it also explains each score.

SVG-adapted CLIP scorers

for in-training evaluation
Stage 0 · backbone
CLIP ViT-B/32, ViT-L/14, ViT-H/14

Off-the-shelf checkpoints, trained on natural images.

Stage 1 · domain adaptation
3.2M SVG–caption pairs

Drawn from StarVector and OmniSVG with the overlap removed. The original captions are noisy, so we recaption every image with Qwen3-VL-8B, then fine-tune both encoders contrastively.

Stage 2 · preference alignment
Pairwise human preference, LoRA

For each caption we pair two SVGs with different human scores and treat the higher-rated one as preferred. The base weights stay frozen; only the adapters learn.

Output
Cosine similarity

Human-aligned VLM judge

for final evaluation
Stage 0 · backbone
Qwen3-VL-8B, prompted zero-shot

Already a better starting point than any CLIP scorer, but its scores are not calibrated.

Stage 1 · SFT
Structured rationale + score

One epoch of supervised fine-tuning to produce <think>…</think><score>1–5</score>. On its own this improves calibration but lowers rank correlation.

Stage 2 · GRPO
Two reward terms

Two completions per input, with SVGs that share a caption kept in the same batch.

Ordinalhighest at the human score, falling off the further away the guess is.
Intra-caption rankingpays off when two SVGs of one caption come out in the human order.
Output
A score from 1 to 5, with its reasoning

Dataset 1

An SVG dataset with human preferences

We pair every caption with seven SVGs of differing quality, found by retrieval from an OmniSVG sample. Five annotators rate each pair from 1 to 5. That gives, for every caption, an ordering the evaluators learn to reproduce.

Score distribution

Caption- and SVG-disjoint split

SplitRatingsSVGsCaptions
Train10,5836,3121,519
Test2,3742,359339
Human score 5 · faithful1 · unrelated

Dataset 2

An independent generator benchmark

We wrote 1,616 captions at three levels of difficulty and checked them by hand. We generated one SVG per caption with 16 models. None of our evaluators was trained on this set.

1,616benchmark captions
3complexity levels: easy / medium / hard
16generators: open, commercial, painterly
0captions shared with the rated data
Prompt

These are the generators' own SVG files, drawn by your browser rather than screenshotted. A cell marked no output returned nothing we could parse.

Results

Agreement with human judgment

All numbers are on the held-out test split. PA is pairwise accuracy: how often an evaluator orders two SVGs of the same caption the way the humans did. Correlations ×100, higher is better except MAE.

JudgeParamsρ ↑r ↑τ ↑MAE ↓PA ↑
Natural-image preference scorers
Aesthetic304.9M17.3323.6412.681.6657.66
ImageReward446.6M56.9555.8343.271.0875.66
PickScore986.1M45.6336.9234.611.4275.72
HPSv2986.1M55.2353.9142.191.2675.66
CLIP ViT-B/32
Vanilla151.3M42.9041.1032.361.4369.35
+ SVG fine-tuned151.3M54.2654.4441.411.2775.82
SVG-Score (ours)151.3M59.3158.4345.461.1877.87
CLIP ViT-L/14
Vanilla427.6M49.8049.2337.811.3372.75
+ SVG fine-tuned427.6M54.4054.4341.421.2776.39
SVG-Score (ours)427.6M60.8759.6946.881.1878.37
CLIP ViT-H/14
Vanilla986.1M56.6456.7943.731.2176.87
+ SVG fine-tuned986.1M59.7258.5945.711.2277.52
SVG-Score (ours)986.1M63.1858.9548.621.1580.64

Adapting to the SVG domain helps on every backbone, and preference alignment adds more on top. The gain is largest on the smallest backbone, +16.41 ρ on ViT-B/32 against +6.54 on ViT-H/14: a stronger visual prior already absorbs part of the domain shift. Our ViT-H/14 beats HPSv2 by 7.95 ρ at the same parameter count.

Citation

BibTeX

We will update this once the preprint is online.

@article{cipriano2026svgscore,
  title   = {SVG-Score: Human-Aligned Evaluation of Text-to-SVG Generation},
  author  = {Cipriano, Marco and Zini, Leonardo and Schild, Alexandra and
             Teutschbein, Valentin and Mimi, Afsana and Cornia, Marcella and
             Baraldi, Lorenzo and de Melo, Gerard},
  journal = {arXiv preprint},
  year    = {2026}
}