Evidence · NRR v0.57 · October 15, 2026
Test the rules.
Keep the failures visible.
A frozen diagnostic audit found limits as well as consistent behavior. These checks support specific claims about specific inputs. They do not certify NRR, select new weights or establish the most deserving ordering.
A result that matters
More wins can add evidence and still lower one component.
With one elite win and fixed opponent values, appending a distinct zero-strength, one-point win changes provisional Quality-Win Depth from 53.569 to 52.397. The supporting evidence tail is positive, but the narrow-win debit is larger than its marginal benefit.
The added win still counts elsewhere in the model. This calculation makes no claim about the final index after a full refit. It identifies the exact follow-up: test whole-index behavior on realistic inputs, then decide whether that tradeoff fits a résumé ranking.
Source → definition → test
What the frozen checks show
All 64 result patterns for a four-team round robin: neutral venues, 30–20 games, identical zero team-efficiency inputs, no external poll, no play data and the production game-performance fallback. Each altered tournament is recomputed. Score comparisons use a 0.00001 tolerance on indices reconstructed from the returned rounded components and foundation.
| Property and source | Observed result | Scope and limit |
|---|---|---|
| Label invarianceAppendix A, PDF p. 1: relabel every team and its attributes together. | 64/64 unchanged | Neutral synthetic teams only; no conference or external-poll transformation tested. Alphabetical tie display is excluded. |
| Input orderEngineering check: reverse the same game rows. | 64/64 unchanged | A deterministic result is necessary for reproducibility, not evidence that the ranking is correct. |
| Positive responseAppendix A, PDF pp. 4–5: replace an existing loss with a win against the same opponent. | 384/384 winner indices did not fall | Full synthetic refit, including opponents. Does not cover every schedule, efficiency input or title context. |
| Score homogeneityAppendix A, PDF pp. 1–2: duplicate every game and preserve scores. | 64/64 violations | Largest index change: 3.00. This disproves score invariance for this NRR setup; it does not by itself disprove the weaker claim that every ordering is preserved. |
| Disjoint mergeAppendix A, PDF pp. 3–4: adding a disconnected tournament should preserve original scores. | 64/64 unchanged for identical copies | Only disconnected copies of the same tournament were tested. Unequal leagues and field-percentile effects remain open. |
| Bounded scoresEngineering check: finite component and index values within 0–100. | 64/64 valid | Limited to these synthetic tournaments; not proof for every legal input. |
| Adding a weak winAppendix B, PDF pp. 9–10 discusses a stronger monotonicity demand with different game counts. | 12/12 QWD decreases | Fixed opponent context and the provisional QWD component only. This is distinct from replacing a loss and is not a full-index counterexample. Whether to require this property is a policy choice. |
| Self-consistencyAppendix A, PDF pp. 4–5: matched opponents, comparable results and endogenous opponent scores. | Not established | The paper’s GWP proof and its alpha ≥ 1/2 condition cannot be transferred to NRR’s component weights. |
| Symmetry, inversion and complete-league orderingAppendix A, PDF pp. 2–3 and 6–7. | Not established | NRR uses more than the win/loss matrix. A claim requires specified transformations of margins, venues, performance, polls and title information. |
For example, duplicating every game in the first tournament changes Fixture D’s index from 63.684 to 62.919. All records retain their win percentages. This is a score-homogeneity failure, not an unnoticed arithmetic repair. The report retains the original inputs and first counterexample for each violation.
12 historical checkpoints · 48 rescoring scenarios
How much does the order depend on weights?
8largest rank movement
24/25smallest top-25 overlap
16/18smallest top-18 overlap
Transfer five percentage points between Quality-Win Depth and Schedule-Adjusted Record in either direction; separately transfer 2.5 points between performance and loss impact. Each scenario preserves the total weight. The baseline and variant use the same unrounded index sorting rule.
These are fixed-component rescores of the twelve 2024–2025 archives. Opponent values, foundation and head-to-head credit stay fixed. They are not full historical refits, evidence of ideal weights or unseen-season validation. The complete per-checkpoint results are downloadable below.
What transfers from the paper?
A definition can guide a test. A theorem needs its assumptions.
Fernando Leiva Bertrán’s Ranking in Incomplete Tournaments: The Generalized Win Percentage Method, Efficiency, and NCAA Football (2025, 26(1), 3–34; first online 2024) studies a recursive win/loss method. The matrix above follows definitions in its supplied online appendix, not a claim that NRR is that method.
NRR also includes performance, game control, titles and external standings. The paper’s efficiency criterion and calibration do not establish that NRR is optimal or that committee agreement is correctness. Independence from every unrelated game is not adopted as an automatic requirement: opponent revaluation is part of NRR.
Reviewed passages: main PDF pp. 5, 9–11, 24 and Appendix A pp. 30–31; online appendix A pp. 1–7, discussion B pp. 8–10 and proofs C pp. 11–16. The supplied appendix carries the older heading “Efficient Ranking in Incomplete Tournaments.” File hashes and this version difference are retained in the private source ledger. Numerical tables were not independently reconstructed. No reviewer endorsement is implied.
Source review completed · October 15, 2026
Three papers sharpen the next tests.
The three supplied CHANCE papers have now been read in full: 19 scanned PDF pages, with OCR and visual checks of the key method and comparison pages. Their historical calculations have not been independently reproduced. Bibliographic details can also be checked in Hal Stern’s author CV.
- Stern (1995), Who’s Number 1 in College Football? … And How Might We Decide?, 8(3), 7–14. The opening and evaluation sections distinguish the strongest team from the team with the most impressive season. Batch 5 will test whether readers understand that distinction and that an NRR Index gap is not a predicted point margin. Prior seasons, recency weights and win bonuses are choices tied to a goal; this paper does not select NRR’s weights.
- Stern (1997), How Accurately Can Sports Outcomes Be Predicted?, 10(4), 19–23. The method trains on earlier games, predicts later games, then updates. A separate future winner/margin study should use a frozen weekly cutoff, a declared warm-up period and the same eligible games for every baseline. Winner accuracy and margin error answer different questions. Its historical accuracy figures are not modern ceilings or NRR acceptance thresholds. The already-frozen beta protocol is unchanged.
- Carlin and Stern (1999), Designing a College Football Playoff System, 12(3), 21–26. The critique of repeated schedule influence motivates a component-dependency audit, not an automatic deduction. The tournament comparison motivates title probabilities under explicit strength and game assumptions. Summed title probability for included seeds in a larger reference field is different from the probability of selecting the objectively best team. Batches 6 and 9 will keep that distinction visible and test modern inputs; the old field-size conclusions, economics and calendar assumptions are not adopted.
Concrete follow-ups: a reader objective/units check in Batch 5; a schedule-influence map with fixed-context and full-refit ablations in a separately specified B04c; and a strength-versus-earned-seed comparison with declared probability assumptions in Batches 6 and 9. None of these follow-ups is presented as already implemented or measured.
Read the source-to-action addendum (JSON). This dated addendum supersedes the earlier source-access limitation in the frozen result file. The original numerical experiment, manifest and results are preserved.
Check source integrity before extending the model.
The historical source audit corrects fourteen loss summaries in a derived presentation layer and verifies coherent committee protection inputs. Frozen archives and this numerical experiment remain unchanged.
Open work
Five different questions need different evidence.
- Committee agreement: a descriptive comparison, particularly limited when committee standings are themselves an input.
- Internal consistency: expand these synthetic tests to irregular schedules, unequal disconnected leagues, titles and complete performance inputs.
- Sensitivity: rerun full opponent valuations and performance fits when the raw historical tables are available.
- Reader comprehension: observe actual readers explaining worked examples in Batch 5. No reader results exist for this audit.
- Prediction: keep the existing beta forward protocol separate from résumé validity. No prediction result is claimed here.
The new forward specification admits only newly captured checkpoints and games strictly after 2026-10-15T12:53:20.770338+00:00, with source IDs, capture times and hashes. It closes December 7, 2026 and requires at least four subsequent checkpoints; otherwise report incomplete coverage. Future results remain pending. No tuning on those observations is permitted under this experiment.
The initial property audit and supplied-paper review are complete. Broader validation and the follow-up tests above remain open. Numerical revisions require a separately documented decision and new experiment.
Frozen artifacts
Inspect the inputs, run the same checks.
From the matching site source commit, run node scripts/audit-nrr-properties.mjs. The script verifies source, harness and archive hashes before execution; it clears inherited NRR settings and uses the frozen defaults. It never changes production weights or archive inputs. The first attempted run stopped on a missing required synthetic efficiency input; its manifest is retained, and the documented v2 correction supplied neutral efficiency before a successful run.