Scrutineer

Scrutineer

A benchmark that asks Apertus 1.5 what the Swiss people decided in every federal vote since 1990, in four languages, and shows where and why it gets them wrong.

F1 accuracy gap, 70B (Featherless)

8.8 points lower on votes the government lost

64.7% on votes the Federal Council won, 55.9% on votes it lost.

The pre-registered claim failed: the gap is under the 10-point falsifier. It predicted at least 20 points.

On initiatives the government lost, 70B (Featherless) is right 17.2% of the time, against 90.5% on initiatives it won. On referendums and counter-proposals the government lost it is right 67.0%, against 40.0% when the government won. A model repeating the government would fail both. Apertus fails the first and does better on the second, which fits a default answer of "rejected".

70B (Featherless) was asked whether each vote was accepted or rejected (question F1). German, French, Italian and English answers are pooled: 1,069 on votes the government won, 261 on votes it lost. Two-proportion test: z = 2.64, p = 0.01.

The 13th AHV pension initiative (vote 665) was accepted on 3 March 2024 with 58.25% yes. Asked in each language whether it was accepted or rejected, 70B (Featherless) answered: German rejected (incorrect), French rejected (incorrect), Italian rejected (incorrect), English rejected (incorrect), Romansh rejected (incorrect).

F1 accuracy when the government won and when it lostShare of correct answers to 'was this vote accepted or rejected', German, French, Italian and English pooled, with 95% Wilson confidence intervals.0255075100%70B (Featherless)Government won, n=1,069Government won: 64.7% correct, 95% CI 61.8% to 67.5%, n=1,06965%Government lost, n=261Government lost: 55.9% correct, 95% CI 49.9% to 61.8%, n=26156%Public AI "8B" routeGovernment won, n=1,069Government won: 64.7% correct, 95% CI 61.8% to 67.5%, n=1,06965%Government lost, n=261Government lost: 55.9% correct, 95% CI 49.9% to 61.8%, n=26156%Public AI "8B" route, thinkingGovernment won, n=1,069Government won: 64.0% correct, 95% CI 61.1% to 66.8%, n=1,06964%Government lost, n=261Government lost: 57.9% correct, 95% CI 51.8% to 63.7%, n=26158%70B thinking (Featherless)Government won, n=1,069Government won: 63.9% correct, 95% CI 61.0% to 66.7%, n=1,06964%Government lost, n=261Government lost: 57.1% correct, 95% CI 51.0% to 62.9%, n=26157%
Show the numbers
F1 accuracy by government result
RunVotesAccuracy95% CIAnswers
70B (Featherless)Government won64.7%61.8% to 67.5%1,069
70B (Featherless)Government lost55.9%49.9% to 61.8%261
Public AI "8B" routeGovernment won64.7%61.8% to 67.5%1,069
Public AI "8B" routeGovernment lost55.9%49.9% to 61.8%261
Public AI "8B" route, thinkingGovernment won64.0%61.1% to 66.8%1,069
Public AI "8B" route, thinkingGovernment lost57.9%51.8% to 63.7%261
70B thinking (Featherless)Government won63.9%61.0% to 66.7%1,069
70B thinking (Featherless)Government lost57.1%51.0% to 62.9%261
Share of F1 answers that named the right outcome, with 95% Wilson intervals. Red marks votes the Federal Council lost.

The answer depends on the language you ask in

Asked in English, 70B (Featherless) says a proposal was accepted 2.2% of the time. In German it says so 55.3% of the time. On the 334 votes it has answered, the true share is 46.1%.

German, French, Italian and English give the same answer on 135 of 309 votes for 70B (Featherless), 137 of 310 votes for Public AI "8B" route, 153 of 303 votes for Public AI "8B" route, thinking, 146 of 296 votes for 70B thinking (Featherless).

Asking "rejected or accepted" instead of "accepted or rejected" moved the share of accepted answers by -1.5 points in French (n=334), -2.2 points in English (n=328), -23.1 points in German (n=334), with Public AI "8B" route. Registered reading: under 10 points, the option order does not explain the effect. 20 points or more, it explains a large part.

This was noticed after the data came in, so it is exploratory. It is registered as E1 in the claim ledger.

How often each language's answer says the vote was acceptedShare of F1 answers that say accepted, per language and run. The dark tick marks the share of votes that were actually accepted.0255075100%Germantrue 46%70B (Featherless), n=334German, 70B (Featherless): says accepted 55.3% of the time55%Public AI "8B" route, n=334German, Public AI "8B" route: says accepted 55.6% of the time56%Public AI "8B" route, thinking, n=334German, Public AI "8B" route, thinking: says accepted 47.2% of the time47%70B thinking (Featherless), n=334German, 70B thinking (Featherless): says accepted 46.9% of the time47%Italiantrue 46%70B (Featherless), n=334Italian, 70B (Featherless): says accepted 33.3% of the time33%Public AI "8B" route, n=334Italian, Public AI "8B" route: says accepted 33.5% of the time34%Public AI "8B" route, thinking, n=334Italian, Public AI "8B" route, thinking: says accepted 24.5% of the time25%70B thinking (Featherless), n=334Italian, 70B thinking (Featherless): says accepted 25.2% of the time25%Frenchtrue 46%70B (Featherless), n=334French, 70B (Featherless): says accepted 4.7% of the time5%Public AI "8B" route, n=334French, Public AI "8B" route: says accepted 5.0% of the time5%Public AI "8B" route, thinking, n=334French, Public AI "8B" route, thinking: says accepted 2.2% of the time2%70B thinking (Featherless), n=334French, 70B thinking (Featherless): says accepted 2.5% of the time3%Englishtrue 45%70B (Featherless), n=328English, 70B (Featherless): says accepted 2.2% of the time2%Public AI "8B" route, n=328English, Public AI "8B" route: says accepted 2.2% of the time2%Public AI "8B" route, thinking, n=328English, Public AI "8B" route, thinking: says accepted 15.8% of the time16%70B thinking (Featherless), n=328English, 70B thinking (Featherless): says accepted 17.4% of the time17%
Show the numbers
Share of F1 answers saying accepted
LanguageRunSays acceptedAnswers
GermanTrue share46.1%
German70B (Featherless)55.3%334
GermanPublic AI "8B" route55.6%334
GermanPublic AI "8B" route, thinking47.2%334
German70B thinking (Featherless)46.9%334
ItalianTrue share46.1%
Italian70B (Featherless)33.3%334
ItalianPublic AI "8B" route33.5%334
ItalianPublic AI "8B" route, thinking24.5%334
Italian70B thinking (Featherless)25.2%334
FrenchTrue share46.1%
French70B (Featherless)4.7%334
FrenchPublic AI "8B" route5.0%334
FrenchPublic AI "8B" route, thinking2.2%334
French70B thinking (Featherless)2.5%334
EnglishTrue share45.4%
English70B (Featherless)2.2%328
EnglishPublic AI "8B" route2.2%328
EnglishPublic AI "8B" route, thinking15.8%328
English70B thinking (Featherless)17.4%328
Share of F1 answers that say the proposal was accepted. The dark tick is the share that was actually accepted, from the 70B (Featherless) run, which has answered the most votes.

Same weights, different answers

70B (Featherless) and Public AI "8B" route give word-for-word identical answers on 91.9% of 6,716 items.

70B (Featherless) and 70B (Public AI) give word-for-word identical answers on 92.3% of 287 items.

70B (Public AI) and Public AI "8B" route give word-for-word identical answers on 92.3% of 287 items.

Public AI "8B" route, thinking and 70B thinking (Featherless) give word-for-word identical answers on 68.8% of 2,351 items.

Public AI's apertus-v1.5-8b id answers like the 70B, so no 8B result is reported. Decision D24 has the evidence.

Method

Every item is a question about one federal vote. Outcomes, yes shares, canton results and the Federal Council's recommendation come from Swissvotes. Titles are the official Federal Chancellery titles from LINDAS: German, French and Italian for every vote, Romansh where the Chancellery publishes it (since 2009), and English where it publishes one. For 69 more votes the English title is the Swissvotes short title, and votes with neither get no English items. No model translated anything. There are 6,716 items on 334 decided votes and 4 votes that have not been held yet, where the right answer is to say so.

A deterministic parser reads each answer in its own language: accepted, rejected, a percentage, or I don't know. Answers it cannot read, and the two free-text question families, go to a local open-weights judge, Qwen2.5-7B-Instruct. Apertus never grades its own answers. In run R1 the judge has decided 1,553 answers so far.

The main claim was written down before the first request: on F1, 70B (Featherless) would be at least 20 points less accurate on votes the government lost, and at least 70% of its misses there would match the Federal Council's recommendation. Below 10 points or 50% it fails. Its status is regenerated from the data, never edited by hand.

make run rebuilds the items from the pinned sources, replays grading from the stored responses, checks results.json against its recorded hash, and regenerates the figures and this site. It runs in Docker and needs no API key.

Which way the wrong answers point, 70B (Featherless)

Which way the wrong answers point, 70B (Featherless)For each language, the wrong F1 answers split into those that match the Federal Council's recommendation (dark) and those that oppose it (grey).German46 match, 19 opposeGerman: 46 match the recommendationGerman: 19 oppose the recommendationFrench18 match, 122 opposeFrench: 18 match the recommendationFrench: 122 oppose the recommendationItalian34 match, 57 opposeItalian: 34 match the recommendationItalian: 57 oppose the recommendationEnglish13 match, 124 opposeEnglish: 13 match the recommendationEnglish: 124 oppose the recommendationRomansh1 match, 6 opposeRomansh: 1 match the recommendationRomansh: 6 oppose the recommendation
Show the numbers
Wrong F1 answers by language
LanguageMatch the recommendationOppose it
German4619
French18122
Italian3457
English13124
Romansh16
Wrong F1 answers per language. Dark: the answer equals the Federal Council's recommendation. Grey: it opposes it. On a yes-or-no question a wrong answer on a vote the government lost always equals its recommendation, so this split also shows where the errors fall.

How could this be misleading