WorldCup Arena Avoids Leakage, but Its Leaderboard Is Not a Verdict
WorldCup Arena tests LLM forecasts before results exist, but scoring changes the ranking and the public archive does not include every original model call.
WorldCup Arena asked six large language models to forecast every match of the 2026 FIFA World Cup before kickoff, then scored 4,494 locked predictions against results that did not yet exist when the models answered. That calendar gives the experiment a rare strength: a model could not recall the outcome from training data or retrieve it from the web before the match was played.
The resulting leaderboard is much less decisive. The six systems averaged 63.9% accuracy on the three-way match-outcome task, compared with 64.4% for mechanically choosing the side favored by the fixed bookmaker line. Changing only the scoring aggregation produced three different winners, while the top model’s match-score advantage over another leading system had a bootstrap interval that crossed zero.
Removing answer leakage leaves the rest of the evaluation open to scrutiny. Task choice, retrieval, scoring, uncertainty, and artifact access still limit what the leaderboard means.
Three locks keep every prediction ahead of its result
The August 4 WorldCup Arena preprint describes a 39-day run covering all 104 tournament matches. Before each kickoff, every system received the same fixture header, a fixed bookmaker handicap line, and rolling dossiers for the two teams. The models could also use their providers’ native server-side web search.
Each system committed to seven single-choice markets: handicap, half-time/full-time, match outcome, over or under 2.5 goals, both teams to score, exact score, and odd or even total goals. The authors also asked for 12 group winners and a pre-tournament pool covering the champion, finalists, semifinalists, winning confederation, and a total-goals threshold.
The protocol used three procedural locks:
- A snapshot lock served only the dossier version current before that match. Results entered later snapshots only after the relevant match date had passed.
- A kickoff lock required every model call to finish before play began, preventing a search-enabled model from retrieving the live result.
- A template lock fixed the prompt, markets, scoring weights, and handicap line before all six systems were called.
These controls address temporal answer leakage directly. A retrospective test usually has to estimate whether a model memorized a published answer, or hide details well enough that the answer cannot be reconstructed. WorldCup Arena instead chose questions whose answers had not been created yet.
“Leakage-free” should still be read at the answer boundary. The systems were allowed to search for public team news, odds, injuries, and analysis available before kickoff. That information is legitimate forecasting evidence, not leakage. The benchmark therefore measures an end-to-end configuration that includes a model family, its provider’s search stack, the prompt and dossiers, and the scoring rule. It does not isolate a base model’s unaided reasoning.
The bookmaker baseline matters more than the winner
The paper’s most informative comparison is not first place versus sixth. It is the six-system cohort versus a simple public-consensus baseline.
Across the 104 matches, the models averaged 63.9% accuracy on home win, draw, or away win. Always choosing the team favored by the benchmark’s fixed handicap line reached 64.4%. Only one system exceeded that baseline, by two matches. A majority vote across all six reached 65.4%, which the authors describe as within the noise of the best individual result.
The models also shared recognizable errors. They predicted fewer draws than the 27 that occurred, underestimated total goals, and concentrated 28% of exact-score selections on 2–1. Performance fell in closely matched knockout fixtures even though the rolling dossiers were richer by then. Those patterns support a narrow conclusion: the systems often compressed uncertain matches toward the same conventional football forecast.
They do not show that tool-using LLMs have no forecasting value. The task gave every system a bookmaker line and allowed live search, then evaluated one unusually short, high-variance tournament. A practitioner should credit the prospective design while resisting a leap from “did not beat this football anchor” to “cannot forecast any changing event.”
Five scoreboards, three winners
WorldCup Arena assigns different point weights to its seven markets, then adds separate group and tournament-pool points. That produces one legitimate view of performance, but it is not a neutral fact of nature.
The authors rescored the same locked predictions five ways. One system won the match-market view, the first half, and the complete scorecard. Another won the second half. A third won when each match day was normalized to 100 before averaging. Every system led at least one of the 34 match days, and the paper reports that the average within-day spread was about ten times the tournament-long spread after normalization.
| Reading of the archive | What changes | What the winner can support |
|---|---|---|
| Match markets only | Excludes group and tournament-pool questions | Best total under the predeclared match weights |
| First or second half | Changes the fixtures and tournament phases included | Performance during that part of this tournament |
| Complete scorecard | Adds group winners and the pre-tournament pool | Best total under the paper’s full weighting scheme |
| Day-normalized index | Gives each match day equal weight after rescaling | Best average relative position across days |
The researchers did not hide this sensitivity; it is one of their strongest results. A leaderboard that flips when a defensible aggregation changes should be reported as a set of task-conditioned comparisons, not converted into a universal ordering of the underlying models.
Uncertainty reinforces the same reading. The paper bootstraps the 104 matches 10,000 times. Its reported match-score difference between the leading system and GPT was 22 points, with a 95% interval from -49 to 94. That interval does not identify a stable winner between those two configurations.
This benchmark does not measure calibration
Calibration asks whether stated probabilities match observed frequencies. If a system assigns 70% probability to many events, a well-calibrated forecaster should be right about 70% of the time in comparable cases.
WorldCup Arena’s scored outputs are single choices rather than probability distributions. Its accuracy and weighted points can compare selections, but they cannot reveal whether a model was appropriately uncertain. A confident 51% choice and a tentative 34% choice become the same categorical pick.
A separate preprint with the similar name WorldCupArena illustrates the design difference. That project evaluates 13 systems on result probabilities, score distributions, player and event predictions, match statistics, and tournament outcomes. Its open-source implementation uses Brier and ranked-probability scores for core outcomes, alongside task-specific metrics for other fields.
The two studies are not independent replications: they use different systems, prompts, evidence conditions, output schemas, metrics, and composite weights. They are useful together because they show how the evaluation question determines the evidence. Single-choice markets make a live run easier to settle consistently. Probability forecasts make calibration measurable. Neither design should borrow claims from the other.
Public code is not a public record of every original call
The WorldCup Arena authors released substantial materials. The public repository includes the pipeline for dossier construction, prompt assembly, provider routing, settlement, scoring, and analysis. Its linked dataset contains 48 team dossiers across 46 timestamped snapshots, the 104 fixtures, team and venue metadata, and official results. The paper says deterministic scoring over two frozen JSON files regenerates its totals and figures.
However, the repository also states that the authors’ predictions, raw model responses, and archived prompts are retained for audit but deliberately not published. The paper says a public project site carries per-match pick cards and the leaderboard, but that is not equivalent to releasing the raw provider responses, request metadata, and complete prediction archive used by the analysis.
The available artifacts support two different levels of reproducibility:
| Reproducibility question | Current public evidence |
|---|---|
| Can another team inspect the benchmark inputs, pipeline, scoring, and reported aggregate analysis? | Largely yes, through the code, frozen dossiers, fixtures, results, and public results site |
| Can another team verify every original model response and rerun the exact scoring from a fully public raw prediction archive? | No; the authors retain those artifacts for audit |
| Can another team repeat the protocol with current provider models? | In principle, but model versions, server-side search, provider behavior, and live web content will differ |
| Can another team recreate the prospective condition on the same tournament? | No; the outcomes are now public, so a fresh event is required |
The last limitation is fundamental. Once the World Cup is over, the task is no longer leakage-free for a new run. The protocol can be reused, but its clean prospective condition moves to the next event.
A separate site, WorldCupBenchmark, offers a simpler public comparison: four systems locked full-tournament brackets before the opening match and receive one point for each correct pick. Its methodology says game-day scoring remains pending. That public implementation corroborates interest in locked future-event evaluation, but it does not validate WorldCup Arena’s seven-market results or reproduce its rolling-dossier protocol.
The next clean test needs a new future
WorldCup Arena’s contribution is not a definitive model table. It is a concrete demonstration that one contamination problem can disappear when an evaluation waits for reality to create the answers.
Its own results then supply the warning label. The six systems mostly tracked the same bookmaker-informed consensus, their ordering depended on the scoring view, and the public artifacts stop short of a complete raw record of the original calls. A stronger follow-up would pre-register a probability-based scoring rule, publish immutable prediction receipts and raw outputs, include simple statistical and market baselines, and repeat the protocol across several unrelated future events.
That test cannot be rerun on the 2026 World Cup. The calendar has already turned its answers into training data.