A Ground-Truth Benchmark for AI Meeting-Note Transcription: synthetic two-speaker test meeting, planted ground truth, and scored results for four commercial note-takers

Commercial AI meeting note-takers are hard to evaluate because the obvious evaluation set — real meetings — has no ground truth. Nobody holds a word-perfect record of what was said on a call, so accuracy claims stay impressionistic. This dataset inverts the problem: the meeting is generated from a known script. Two distinct synthetic voices read a scripted 80-second product meeting deliberately loaded with the failure cases that matter in business audio — quarter labels (Q2/Q3), decimal percentages, currency amounts, an uncommon person name, a latency percentile (p95), a ticket priority (P1) and an initialism (SSO). Because the script is the ground truth, each planted item can be scored exactly. The deposit contains the generation harness (synthesize_meeting.sh), the ground-truth script with its scoring checklist (ground-truth.txt), and 32 scored observations across four commercial tools — Otter.ai, Granola, Fathom and Fireflies.ai — run on the same clip (results.csv, results.md). Headline finding: on clean synthetic audio, overall word accuracy is uniformly high and therefore uninformative. What separates these systems is which class of token each one degrades: quarter labels and ticket IDs (Otter reduced both "Q3" and "Q2" to a bare "Q", and "tag it P1" to "tag at p1"), single-line garbling (Granola), punctuation (Fireflies). Those classes carry disproportionate downstream cost because they are the tokens that get pasted into a status update. Speaker diarisation was the sharpest split: Otter separated the two speakers with correct attribution throughout, while Granola's ad-hoc bot-free capture produced a single unlabelled stream. Limitations are stated in full in the README and should be read before reuse: n = 1 clip; deliberately easy audio (synthetic, non-overlapping, two clearly distinct voices, clean turn boundaries); capture paths differ between tools, so comparisons are end-to-end product comparisons rather than isolated ASR-engine comparisons; scored surfaces differ (transcript vs enhanced summary); and results are a June 2026 snapshot of hosted products that change without notice. The rendered MP3 is not included: it is third-party text-to-speech output whose redistribution licence is not the depositor's to grant. Regenerate it with your own API key using the included harness, or listen to the exact clip used alongside the source evaluation at https://aialleyway.com/otter-ai-review/. The intended reuse is as a method rather than a leaderboard: the harness plus checklist is a cheap, repeatable way to obtain real ground truth for any speech-to-text product, and extending it (more turns, overlapping speech, accented voices, a third speaker, WER over the full transcript) is a matter of editing the turn list. Per-tool observations are reproduced from the published hands-on evaluations, which carry the screenshots and full context: Otter.ai, Granola, Fathom and Fireflies.ai. Those pages also carry overall editorial ratings that weigh pricing and workflow alongside accuracy; they are not derived from this benchmark and should not be read as benchmark scores.