Chaordic Bench

Results

Everything ran on 22 Oct from Modal's us-east region, where the writer runs: 14,602 labelled requests in track A and 40,544 menu requests in track B (elephant-1 and mammoth-1 on our own H100s), with 6,706 more in a second latency window. In track C, 2,040 pieces were written live, with 8,664 panel verdicts. Track A had no failed requests. In track B, 21 of the Baseten models' requests failed even after retries, and none on our own GPUs. No person has listened yet: the panel is four AI models, and its ratings are reported on their own.

• Speed under a real menu. Jev 1.14 and Decider Flash decide a menu inside the 400 ms gate at p95 in both windows: 190 and 254 ms in the full run, 166 and 233 ms in the second. So does elephant-1 on our own H100s (288 ms). d1-ultra, Luna and Decider v1.2 answer one request quickly, but with four sent at once they take 480–810 ms at p95 and up to 1.5 s at p99. mammoth-1 on our H100s takes 665 ms.
• Base Labs' models on our own GPUs. On Baseten's free tier, elephant-1 and mammoth-1 had 29 and 119 requests time out, so their p95 reached 3.7 and 9.3 s in the full run (287 and 849 ms in a second window). From their published weights on our own H100s, four per model, none timed out: p95 288 and 665 ms, p99 386 and 723 ms. The published checkpoints match Baseten's endpoint on track A's accuracy to within a point and give the same top pick on 87% and 93% of its requests.
• Labelled accuracy. On the five music tasks, mammoth-1, d1-ultra and Decider v1.2 lead (79%, 76%, 75%), level with production's Jev 1.13 (75%) and well short of Opus 5.5 (88%). Jev 1.14 is no better than 1.13 here (73%). Decider v1.2, Decider Flash and mammoth-1 have the lowest calibration error (0.029–0.034).
• General rankings don't carry over. elephant-1, at 65 on the chart you shared, is last on the music tasks (58%). Luna, the low anchor on both general benchmarks, matches Jev 1.14 (73% each).
• No model catches a rejected draft. With one candidate swapped for a draft our checks cut (misspelled, unplayable, copied or wrong texture), every model picks it 21–27% of the time, against 25% by chance. The writer's code checks remain the only defence.
• Models disagree. Any two pick the same bar on only 46–62% of menus, and most change their pick with the candidate order: 24–54% keep it in all four orders. Asking in every order and averaging is necessary.

Track A: labelled decisions

Accuracy with candidate orders averaged, 2,086 requests per model. Chance is 25% for the next bar of 4 and 50% for the other tasks. Brier and calibration error are lower-better. Jev 1.13 and Opus 5.5 are earlier runs on the same items, from a different machine, so their speeds aren't shown. Opus answered one order per item.

Track B: frozen menus

893 menus from 33 pieces. 636 asked again with a rejected draft in one of four places, so picking it by chance is 25%. Steering is how often the pick differs from the bar the music generation model drafted most. Production is Jev 1.13, whose picks wrote these pieces. Deadline misses are estimated from the recorded pieces' timings, among bars where the writer left time to decide. Track C measures them live. Baseten's models are free until 27 Oct. Their prices after that aren't public. The second window asked 150 of the menus again, from 07:53 to 08:01 UTC, after every other run had finished.

Track C: live pieces and the judging panel

Each condition wrote the same 204 pieces live on Modal, 2040 pieces and 65,060 bars in all: six themed plans on public-domain primers, 34 seeds each, so every plan and seed starts from the same opening. The model made every choice JEV makes in production. Then the panel (Opus 5.5 at medium effort, GPT-6 Astra at xhigh, GPT-6.1 Sol and Sonnet 5.5) compared pieces from the same plan and seed, blind, one judge per pair, each pair in both orders.

• With 204 pieces per condition, the panel separates the field into clear tiers. However the judges are weighted, it tells apart 36 of the 45 pairs among the ten conditions (16 of 36 with 24 pieces each). Decider v1.2 places first or second, d1-ultra first to third, then a middle tier of mammoth-1, Decider Flash, Jev 1.14 and GPT-6 Luna (third to sixth), elephant-1 seventh, the two baselines with checks eighth and ninth, and the music generation model alone last.
• Decider v1.2's and d1-ultra's choices make the best pieces. The panel rates them 1085 (1049–1122) and 1081 (1044–1119), above Jev 1.14 with 95% confidence. Against Jev's pieces directly, theirs won 60% and 61% of 408 verdicts each. Leave out any one judge and they stay the top two. mammoth-1 rates 1037 (997–1076): third on the point estimate, but not separable from Jev.
• Choosing beats not choosing, and a menu alone does nothing. The writer alone, playing a bar the music generation model drafted from those that pass its checks, rates 888, and random picks from the same menu 863: level with each other (random won 63 of 150 against no judge). Jev 1.14's pieces won 262 of 408 verdicts against the writer alone and 273 of 408 against random picks. elephant-1's own choices rate 941, the lowest of the decision models but above both baselines.
• The checks do their share. The music generation model on its own, with no checks and no chooser (each bar one of its drafts that parse, drawn at random), rates 698, below every other condition. Against it, the writer alone, which adds only the checks, won 287 of 406 verdicts. Jev 1.14 won 333 and d1-ultra 358. At the bench's temperatures 9.4% of the music generation model's own drafts didn't parse, and 87% of the bars it played would have failed at least one of the writer's checks.
• Stratified by plan. The tiers hold in every plan: Decider v1.2 and d1-ultra win at least 54% of their verdicts in each of the six, the music generation model alone at most 31%. The middle reorders by style: Decider Flash wins 44% of its gymnopédie and nocturne verdicts and 59% of its étude ones. GPT-6 Luna 40% in the gymnopédie and 59% in the nocturne, and mammoth-1 45% in the rag and 64% in the gymnopédie.
• The best two are slower. While writing live, Decider v1.2's and d1-ultra's choices took 636 and 822 ms per bar at p95, past the 400 ms gate. At one shared writer speed they missed 11.7% and 13.4% of deadlines, and a listener with a two-bar head start would wait 5.8 and 6.5 times per 100 bars, against 4.1% and 1.1 for Jev 1.14. mammoth-1 (770 ms, 20.9% missed) and GPT-6 Luna (1106 ms, 18.5%) are slower still. Decider Flash is nearly as quick as Jev (290 ms, 1.3 waits) and rated level with it.
• Nothing passes the real-time rule as written. The headline ranks only models that keep p95 under 400 ms, deadline misses under 1% and errors under 0.5%. At the shared writer speed even Jev 1.14 misses 4.1% of deadlines. Jev 1.14 (214 ms), Decider Flash (290 ms) and elephant-1 on our GPUs (378 ms) pass the other two rules.
• The judges see wrong notes, and lean on position. All four got all 16 of their check pairs right, yet each picked the second piece 62–72% of the time and named the same winner in both orders on 55% (Sonnet 5.5) to 74% (GPT-6.1 Sol) of pairs. GPT-6.1 Sol was the cheapest judge, at $0.014 a verdict. GPT-6 Astra at xhigh cost $0.38 a verdict, $695 of the panel's $886.
• Four pieces are missing, by design. GPT-6 Luna's provider refused to answer a request in three pieces, and one of the music generation model alone's drafts couldn't be parsed by the writer, so those pieces stopped. Nothing answers in a model's place. Every other condition wrote all 204.

Place: the places a condition can hold, from 1 + the conditions that beat it to 10 − those it beats, counting only pairs told apart however the judges are weighted (the dark cells below). No judge: the writer plays a bar drawn from those that pass its checks, in proportion to how often the music generation model wrote it. The music generation model alone: no checks either, and each bar is one of the music generation model's drafts that parse, drawn at random and played as written (9.4% of its 208,544 drafts didn't parse). It was judged against Jev 1.14, no judge and d1-ultra only, by Opus 5.5, Sonnet 5.5 and GPT-6.1 Sol, and its own timings don't count toward the shared writer speed. Random answers: every request answered at random, then averaged over the orders as usual. Deadline misses: bars where the writer's other work left time to choose, but not enough, at one shared writer speed (each bar's median writing time across conditions). Waits: stalls in a playback simulated with a two-bar head start. The baselines' waits are the writer's own. Asked again: requests that got no answer at first and were asked again until the model answered, so every choice is the model's own. The time it took stays in the bar's. Brief misses and repeats count chosen bars that failed the writer's brief checks (texture, markings, cadences) or repeat checks. elephant-1 and mammoth-1 wrote their pieces on our own H100s, four per model.

Which conditions the panel tells apart

Each cell is the row's Elo minus the column's. Dark: told apart (95% interval clear of zero) whichever way the judges are weighted, by their check pairs or by how often each names the same winner in both orders: 36 of 45 pairs. Light: told apart only with the check-pair weights: 1 more. Plain: the panel can't tell them apart. Hover a cell for its interval.

By plan

The share of its verdicts each condition won within each plan, ties as half. Darker is more. Each cell rests on 198–612 verdicts, so it is good to ±4–7 points at best (more, as a pair's two orders aren't independent): read the rows, not single cells.

The panel

Verdicts on real pairs, each pair asked in both orders by the same judge. Check pairs put a piece against itself with a semitone wrong in a quarter of its right-hand bars, also in both orders, and count in the cost. Prices are list prices per token, reasoning included.

What the chart doesn't tell us

The chart you shared is the Artificial Analysis Decision Index, a general score made of knowledge, refusal, tool-call and safety questions. Its 65 for elephant-1 is Base Labs' own replica run. The official index's 19 October snapshot doesn't list elephant-1 yet. An independent benchmark from the same week (1,342 five-choice logic questions) ranks the field differently: elephant-1 below Jev 1.14, and its 27B sibling mammoth-1, which the chart leaves out, at the top.

Neither index asks the question sans fin depends on: given four candidate bars written in ABC, a section brief and the piece so far, which one should play next, decided before the current bar ends. Our earlier tests showed why this needs its own eval. Frontier LLMs judged better than JEV (79–80% against 54% on the next bar of four), but took 4–28 s a choice, and on the music generation model's own candidates no judge could find the composer's bar at all. So Chaordic Bench measures what can be measured: rule-following against code, consistency, speed under a real deadline, and what listeners prefer.

Which models to test

Three things decide the shortlist. A model must take the /v1/systemone request we already send JEV (a state plus typed questions, answered with a probability for every option), which Jev, Decider, d1-ultra and the Base Labs models all do. It must be fast at the tail, because each bar sends four requests at once, one per candidate order, and waits for the slowest. And together the picks should span the field: the chart's leaders, the independent leaders, our incumbent, a small model, and one low anchor to test whether general rankings predict music at all.

Chart: the Decision Index image you shared. * Base Labs' replica run, not the official index. Baseten's free launch week ends 27 Oct. Independent: System 1 Decision Benchmark, accuracy, median and p95 per request, price per 1,000 decisions. † Official index's time and cost per task (no independent run).

Baselines, to show what a decision model adds at all: no judge plays a bar drawn from those that pass the writer's checks, in proportion to how often the music generation model drafted it. Music generation model alone skips the checks too and plays one of the music generation model's drafts that parse, drawn at random. Random picks uniformly from the same four candidates, isolating the menu from the judgment. A slow LLM (Opus 5.5 at low effort, about 2 s) judges the labelled items and frozen menus as a quality ceiling, though it is too slow to drive the loop.

How we test

Every model sees the production prompt word for word and is asked in all four candidate orders, exactly as JEV is today. Only the model changes. One harness covers all of them: our JEV client with its URL, key and model name made settings.

What we measure

Each metric is computed by code from logged events, or counted from blind votes. Nothing is scored by eye.

The headline. Models are ranked by panel Elo, among those that keep up in real time: p95 decision latency under 400 ms, deadline misses under 1% and errors under 0.5%. Speed, cost and adherence sit beside it in their own columns. They aren't folded into one score.

Side by side

Two views, both built from logged events, so every piece can be replayed exactly. The sketch below shows the layout only. There are no results yet.

On the site

Modelled on Cognition's FrontierCode page: versioned, with its method written out and every run replayable.

Pieces published on the site must start from public-domain scores. The current library holds private arrangements, so the public runs use a public-domain primer set (Mutopia and IMSLP piano scores, Joplin's rags) and descriptions only. Track B's menus already follow this rule.

Where it lives. The stack is the code and hosting the page is built on. Three ways, to settle later: a page in the models.seancai.com app (Next-style pages on Cloudflare Workers, which already publishes benchmark data from JSON snapshots), a page served by sans fin on Modal, which already engraves and plays pieces, or both, with the leaderboard and method on models.seancai.com and the listening player and voting on sans fin, linked or embedded. The third keeps the leaderboard fast and static and puts audio where it already works.

Schedule and cost

1. Access and harness

   Done 22 Oct. One client for TypeSafe, OpenRouter's Decisions API and Baseten. Every model answers from Modal's us-east region. OpenRouter models pinned to dated builds.

   done

2. Tracks A and B

   Done 22 Oct: 2,086 labelled requests and 5,792 menu requests per model, then 958 more per model in a second latency window, about 62,000 in all.

   done

3. Track C and the panel

   Done 22 Oct: 216 pieces written live on Modal, then the panel's verdicts, each pair in both orders. On 23 Oct, 24 more pieces from the music generation model on its own, with no checks or chooser, judged against Jev 1.14, no judge and d1-ultra: 8,664 verdicts in all.

   done

4. Our own GPUs for Base Labs' models

   Done 22 Oct: elephant-1 and mammoth-1 from their published weights on our H100s, with the runtime their cards pin. Tracks A and B run again on them and their track C pieces written again, every choice answered.

   done

5. A sharper bench

   More pieces per plan and a full round robin of comparisons, so the panel can separate more of the middle. Harder check pairs, so they tell the judges apart. Three latency windows for every model, at different hours.

   next

6. Long pieces and listening

   Pieces of 100–200 bars for the closing decision. If a paid human panel votes too, it is rated the same way and reported beside the AI panel, never mixed with it. In simulation, about 1,000 votes give each model a 95% interval of ±64 Elo. About 3,000 give ±39.

   next

7. The page

   Leaderboard, method, decision explorer, side-by-side player, changelog, and the data as JSON.

   week 3

Costs are summed from the logged rows: OpenRouter reports each request's cost, Jev is priced at TypeSafe's list price per input token, and the judges at their list prices per token, reasoning included. Baseten's models are free until 27 Oct. Their later prices aren't public. GPU time is estimated from the runs' timelines at Modal's H100 price ($0.001097 a second) with its 1.15–1.75× charge for choosing a region. The music generation model's GPU time is an estimate too. GPT-6 Astra at xhigh was 78% of the kept verdicts' cost. Its requests that timed out reported no usage, so they aren't counted. The re-judging replaced every verdict on elephant-1's and mammoth-1's earlier pieces.

Keeping it fair

• JEV's home advantage. Our prompt wording was tuned on JEV over many weeks. The leaderboard uses it unchanged, because it is what the app sends, and track A adds a neutral rewording to show how much each model depends on it.
• Routes and load. Latency depends on the provider's route and the hour. We call every model from the same region as the writer, interleave models, repeat the latency runs in a second window, and publish the route used. Baseten's free tier was slow in the first window and not in the second. Both are shown beside the numbers from our own GPUs.
• No answer key for generated music. No judge, frontier models included, could find the composer's bar among the music generation model's candidates, so tracks B and C rely on code checks, planted decoys and blind votes rather than a key.
• AI judges kept apart. The panel's verdicts are reported on their own and would never be mixed with human votes.
• Judges' position bias. In a pilot every judge preferred the second piece in all eight comparisons, and over the whole panel each judge picked it 62–72% of the time. Each pair is therefore asked in both orders of the same judge and both verdicts count, so a judge that only follows position splits its pair evenly. Check pairs (a piece against itself with wrong notes planted) measure each judge's reliability.
• Every choice is the model's own. The panel's Elo rates the music, so nothing about a route may decide a bar. A request with no answer is asked again until the model answers. It never falls back to even odds. On Baseten's free tier elephant-1 had lost 7% of its requests to that fallback, so its pieces and mammoth-1's were written again on our own GPUs and every comparison with them judged again.
• One writer speed. The GPUs are busier at some hours than others. Deadline misses and waits are computed at one writer speed for everyone: each bar's median writing time across conditions, plus the model's own decision time.
• Vendor numbers aren't ours. The chart's 65 for elephant-1 and the 66 for mammoth-1 come from Base Labs' own runs. Chaordic Bench reports only what we measure.
• Limits. elephant-1 refuses prompts over 16,384 tokens. Our longest prompt, the last 48 bars, is about 5,000, so it fits.
• Routes. elephant-1 and mammoth-1 run on our own H100s, four per model so a menu's four orders run side by side, from their published weights. The other models run on their providers' APIs. The published checkpoints matched Baseten's endpoint on track A's accuracy and agreed on 87–93% of top picks. Baseten's free tier allowed 120 requests a minute per model, so the earlier runs there were paced at 110, outside the clock.
• Position. Every model leans toward some places in the list: d1-ultra picks the first-shown candidate 36% of the time and Decider Flash the second 39%, where chance is 25%. Asking in every order and averaging cancels this, as the writer already does for JEV.

Decisions