Benchmark · v1.0 · 24 October 2026

Chaordic Bench

Which decision model makes the best live choices for music? NotaGen-X drafts every bar of a piano piece, code checks the drafts, and a decision model picks the one that plays.

By the sans fin team

How it works Browse every trace

Leaderboard

panel Elo against speed

Higher is better. Left of the dashed line is fast enough to play live.

Findings

What the results show

Choosing is what moves the pieces

the same writer, one layer at a time

NotaGen-X runs hot

Sampled hot, it rarely writes the same bar twice and often writes a broken one.

What each decision model is good at

Elo says who wins. The judges' reasons say why.

Method

Decode wide, decide narrow

The approach

what this work adds

How each bar is written

how each bar is written

Pieces: Each condition writes the same 204 pieces, six styles with 34 seeds each.
Conditions: Seven decision models and three baselines.
Judges: Four AI judges compare two versions of a piece blind, as notation.
Speed: Each bar's choices must land within 400 ms at p95.

What a decision looks like

One bar from drafts to choice. Press a candidate to hear it.

one decision

Loading the decision…

1 · drafts

the bar before

2 · cut by the checks

3 · the menuFour candidates, each asked in all four orders.

4 · the choicePick a model, or plant a draft the checks cut.

Listen

One piece, five choosers

From NotaGen-X alone to d1-ultra choosing every bar. Full side-by-side page →

Listen side by side

Press play, or click any bar to start there.

piece

What the judges said

Changelog

Revisions

Chaordic Bench 1.0

CurrentOct 22, 2026

Read the method →

Changes

Oct 24, 2026

Reporting

Added Traces: every piece each condition wrote, every bar's drafts, checks, menu and choice, and every verdict with its reason.
Each bar's candidates are now recorded as the model read them.

Oct 23, 2026

Models

Added NotaGen-X alone, with no checks, menu or decision model, as a baseline.

Reporting

NotaGen-X alone is compared with Jev 1.14, no chooser and d1-ultra in every piece, by Opus 5.5, Sonnet 5.5 and GPT-6.1 Sol.
Added the share of NotaGen-X's drafts that parse.
A condition that loses almost every verdict counts as apart from another when the check-pair weighting separates them.

Oct 22, 2026

Models

Added Jev 1.14, d1-ultra, Decider v1.2, Decider Flash, GPT-6 Luna, elephant-1 and mammoth-1.
Added no chooser and random pick as baselines.
Moved elephant-1 and mammoth-1 to our own GPUs from their published weights, and wrote their pieces again.

Reporting

Every choice is the model's own: a request that fails is asked again, never answered at even odds.
Place ranges count only the pairs told apart under both judge weightings.
Added each style's win shares.