Decision models choosing for a generative model that runs hot: NotaGen-X proposes, code verifies, and a structured decision model picks every bar of a piano piece, live.
Elo says who wins. The judges' reasons say why: we coded every one of the panel's reasons into the skills it credits or faults in each piece, and compared each model's pieces with the decision model's pieces they were judged against.
Chaordic Bench measures what a decision model adds to a generative model running hot: given the same drafts, the same checks and the same menu, whose choices make the better piece, in time to play it?
Every piece is written live by sans fin's writer, one bar at a time, from a plan: a style, a form, two keys, and a short public-domain primer in the style that NotaGen-X reads as hidden context. For each bar NotaGen-X writes 32 drafts with temperatures spread from 1.0 to 1.8, and 32 more, hotter, when too few survive. Code drops the drafts that don't parse and cuts the ones the plan rules out: misspelled notes, chords no hand can span, hands copied from the primers, figuration that doesn't fit the phrase. Up to four of the rest form the menu, picked for variety. The decision model reads the piece so far and the menu once in each of the four candidate orders, and the writer plays the candidate with the highest average probability. Each condition writes the same 204 pieces, six styles with 34 seeds each, so every comparison is between two versions of the same piece.
Seven decision models drive the writer: TypeSafe's Jev 1.14, Liquid's d1-ultra, Perplexity's Decider v1.2 and Decider Flash, OpenAI's GPT-6 Luna, and Base Labs' elephant-1 and mammoth-1, the last two run from their published weights on our own GPUs. Three baselines bracket them: a random pick from the same menu; no chooser, which draws a bar from the drafts that pass the checks in proportion to how often NotaGen-X wrote it; and NotaGen-X alone, which plays one of its drafts that parse, drawn at random, with no checks at all. Every model sees the production prompt word for word.
A panel of four AI judges (Opus 5.5, GPT-6 Astra, GPT-6.1 Sol and Sonnet 5.5) compares two versions of a piece blind, as notation, and names the better music for the brief with a reason. Each pair goes to one judge in both orders, since every judge favours the piece it reads second, and check pairs with planted wrong notes measure each judge's reliability. A Crowd-BT fit turns the verdicts into Elo anchored at Jev 1.14 = 1000, with 95% intervals clustered by piece; a place range counts only the pairs the panel tells apart however its judges are weighted.
Speed is measured live: the time each bar's choices took, the deadlines missed at one shared writer speed, and the waits a pianist with a two-bar head start would hear. A bar lasts 1.4 to 3.8 seconds in these styles, and the writer's own work takes much of it, so we hold the choices to 400 ms at p95. The bench doesn't yet measure what listeners hear: the judges read notation, and the pieces run 24 to 68 bars.
Each revision freezes its plans, seeds, NotaGen-X weights, prompt and every endpoint's name and date.
One bar from start to finish: what NotaGen-X offered, what the checks cut and why, the four candidates that reached the menu, and what each model chose. Each style's example is the recorded menu, in that style, where the checks cut the most drafts. Press a candidate to hear it after the bar before.
Five versions of one piece in each style, from NotaGen-X alone to d1-ultra choosing every bar. They scroll together, and the tinted band marks the same bar in each. Press play to hear one; switch versions at any bar with Hear, or click a bar to start there. For each style, the piece is the seed the panel compared most among these five.
Every piece written live for Chaordic Bench, every bar's choice, and every verdict the judging panel gave.