Vargha–Delaney A12 — Worked example

Example data

Group 1: 0.91 0.88 0.90 0.93
Group 2: 0.86 0.87 0.85 0.88

Step-by-step

1. Compare all pairs (x from G1, y from G2).
2. Count gt = #(x>y), lt = #(x<y), eq = #(x=y).
3. Compute A12 = (gt + 0.5·eq) / (n1·n2).
4. Interpretation: A12 > 0.5 favors group 1; A12 < 0.5 favors group 2.

Intermediate results (illustration)

n1=4, n2=4, total pairs=16.

If most G1 values exceed G2 values, A12 will be noticeably above 0.5.

Try it in the tool

Open with these values

FAQ

What is A12 used for?

Often used in benchmarking and ML comparisons to express “probability of superiority” in an intuitive way.

How does A12 handle ties?

Ties count as 0.5 (half win) because neither group dominates in that pair.

Is A12 the same as accuracy difference?

No. It compares distributions (via pairwise wins), not just means or a single metric.