{"id":"y-matrix-confirmatory-results","title":"Y-Matrix Confirmatory Results — Rounds 1–2","subtitle":"Linear cross-property structure absent under honest nulls; defect/bulk split 15–60×; OMat retraining removes bias, not variance.","category":"validation","tags":["mlip","y-matrix","confirmatory","softening"],"source":"articles/docs/plans/y-matrix-confirmatory-results-2026-07-01.md","lang":"en","words":1323,"readMinutes":6,"toc":[{"depth":2,"text":"Scoreboard","id":"scoreboard"},{"depth":2,"text":"Deviations / analysis notes (logged with results)","id":"deviations-analysis-notes-logged-with-results"},{"depth":2,"text":"What the answers mean","id":"what-the-answers-mean"},{"depth":2,"text":"H4 result (2026-07-02, run under the registered binding addendum)","id":"h4-result-2026-07-02-run-under-the-registered-binding-addendum"},{"depth":2,"text":"Literature positioning (2026-07-02, from the adversarially-verified synthesis)","id":"literature-positioning-2026-07-02-from-the-adversarially-verified-synthesis"},{"depth":2,"text":"R2-H1 result (2026-07-02, MACE-MPA-0 through the identical matrix)","id":"r2-h1-result-2026-07-02-mace-mpa-0-through-the-identical-matrix"},{"depth":2,"text":"Registered next tests","id":"registered-next-tests"}],"html":"<h1 id=\"y-matrix-confirmatory-results-round-1-2026-07-01\">Y-Matrix Confirmatory Results — Round 1 (2026-07-01)</h1><blockquote>\n<p><strong>Process note (2026-07-02):</strong> these two rounds ran with pass/kill\nthresholds on questions that were, in truth, exploratory cartography — and\nR2-H1&#39;s &quot;KILL by 2.6%&quot; against an arbitrary threshold shows the cost: the\nverdict frame can bury the finding (bias eliminated, variance persists).\nGoing forward, registration with thresholds is reserved for claims where we\nare the interested party (uplift claims about our own correction operator)\nand for publication-bound claims. Exploratory results are reported as\ndescriptive findings with coupling-aware nulls and uncertainty — the\nstatistics stay, the verdict theater goes. The rounds below are preserved\nas recorded, per the retraction-preserving discipline.</p>\n</blockquote>\n<blockquote>\n<p>Registered protocol: <code>y-matrix-cross-property-preregistration-2026-07-01.md</code>.\nData: 57 GPU cells (19 materials × MACE-MP-small / MACE-MP-medium / CHGNet),\n345 reference-bound properties, <strong>57/57 evidence-derived Lean modules\ntype-checked</strong> (<code>data/y_matrix_runs/bound/binding_report.json</code>).\nAnalysis artifacts: <code>data/y_matrix_runs/analysis/confirmatory_*.json</code>\n(seed 20260701, 1,000 null draws, 1,000 bootstrap).\nMatrices: <strong>primary</strong> 15 metals × {a₀, B₀, E_vac, γ₁₀₀, γ₁₁₀} (Cr dropped by\nmatched-n — coarse B₀ reference; Fe included per the registered amendment);\n<strong>secondary</strong> 7 fcc metals × 7 properties (+γ₁₁₁, γ_SFE).</p>\n</blockquote>\n<h2 id=\"scoreboard\">Scoreboard</h2><div class=\"table-wrap\"><table><thead><tr>\n<th>Hypothesis</th>\n<th>Verdict</th>\n<th>Detail</th>\n</tr>\n</thead><tbody><tr>\n<td data-label=\"Hypothesis\"><strong>H1</strong> cross-property low dimensionality</td>\n<td data-label=\"Verdict\"><strong>NOT CONFIRMED</strong> (2/6 pass)</td>\n<td data-label=\"Detail\">PR mostly <em>within</em> the coupling-aware null band. Primary: MACE-small 2.76 (null p05 2.34), MACE-medium 2.36 (2.11) — fail; <strong>CHGNet 2.00 &lt; 2.17 — pass</strong>. Secondary: <strong>MACE-medium 1.47 &lt; 1.50 — pass</strong>; others fail.</td>\n</tr>\n<tr>\n<td data-label=\"Hypothesis\"><strong>H2</strong> shared leading error mode</td>\n<td data-label=\"Verdict\"><strong>KILLED</strong> (0/3 pairs, both matrices)</td>\n<td data-label=\"Detail\">Raw cosines up to <strong>0.96</strong> — but the family-permutation null&#39;s p95 reaches <strong>0.98</strong>: the apparent shared mode is carried by within-family magnitude structure, which the null preserves. No pair exceeds its null.</td>\n</tr>\n<tr>\n<td data-label=\"Hypothesis\"><strong>H3</strong> defect ≫ bulk errors</td>\n<td data-label=\"Verdict\"><strong>PASSED, all models, both matrices</strong></td>\n<td data-label=\"Detail\">Median |rel. err.|: bulk ~0.5–0.8%, defect ~11–31%. Ratios: MACE-small <strong>18.9×</strong>, MACE-medium <strong>15.3×</strong>, CHGNet <strong>57.0×</strong> (primary; secondary 21×/16×/62×). Registered threshold 2×; kill 1.5×. All bootstrap CI95 lower bounds &gt; 1.</td>\n</tr>\n</tbody></table></div><h2 id=\"deviations-analysis-notes-logged-with-results\">Deviations / analysis notes (logged with results)</h2><ol>\n<li><strong>H1 criterion clarification.</strong> The registration&#39;s sentence &quot;lower than the\n95th percentile of the coupling-aware null&quot; is vacuous as written (a typical\nnull draw satisfies it). The harness implemented the evidently intended\none-sided 5% test, <code>PR &lt; null p05</code>, which is stricter. Under the literal\n(flawed) wording H1 would &quot;pass&quot; nearly everywhere; under the intended test\nit passes 2/6. We report the intended test as primary and record the wording\ndefect here rather than silently choosing the favorable reading.</li>\n<li>Cr excluded by matched-n (only a 2-significant-figure CRC B₀ reference;\nflagged in the compilation). Si/NiAl/Ni₃Al are outside the metal matrix by\nconstruction (no defect/surface lanes); their bound properties are reported\nin the descriptive layer and the Lean modules.</li>\n</ol>\n<h2 id=\"what-the-answers-mean\">What the answers mean</h2><p><strong>The turbo&#39;s gearing is per-family.</strong> The elastic-era low-dimensionality does\nnot extend across property families as a universal ribbon: once within-family\ncoupling is granted to the null, only isolated (model, matrix) combinations\nshow compression below it, and no shared cross-model mode survives H2. A\ncorrection operator learned on one family should not be assumed to transfer\nacross families via a shared low-dimensional mode. This refutes the strongest\nform of the cross-property universality thesis for this X×Y slice — and it is\na <em>result</em>, registered in advance, with the kill conditions doing their job.</p>\n<p><strong>What IS universal is the training-distribution split.</strong> H3 is the cleanest\neffect this program has measured: every model, both matrices, defect-family\nerrors 15–60× bulk-family errors, consistent with the published PES-softening\npathology of MPtrj-trained potentials. Bulk properties (a₀ at ~0.5%, ΔH_f at\n~3%) are essentially solved by current foundation models; defect physics\n(vacancies, surfaces, stacking faults) is where they fail, predictably and\nenormously. The correction program&#39;s highest-ROI target is therefore\nfamily-targeted defect corrections, not a global mode.</p>\n<p><strong>The coupling-aware null prevented two false positives.</strong> A naive analysis\nwould have reported cosine 0.96 cross-model mode alignment and &quot;low&quot; PRs as\nconfirmation of universality. Both are inside their null bands. The\nmethodological discipline — registered before data, nulls that preserve\nwithin-family structure — is itself a result worth publishing alongside the\nphysics.</p>\n<h2 id=\"h4-result-2026-07-02-run-under-the-registered-binding-addendum\">H4 result (2026-07-02, run under the registered binding addendum)</h2><p><strong>INCONCLUSIVE — neither the registered pass nor the registered kill fired,\nfor any model.</strong> The kill&#39;s precondition failed everywhere: the LOO stiffness\nscalar (s = 1.057 / 1.081 / 1.125 for MACE-small / MACE-medium / CHGNet) does\nnot improve even the EOS family it was fitted on — per-material B₀ softening\nis too heterogeneous (Fe ≈ 2×, most metals 1.03–1.15) for any single\nper-model scalar. No family significantly degrades; two isolated\nbootstrap-significant improvements (CHGNet surfaces 0.285→0.215; MACE-medium\nplanar fault 0.682→0.665) are candidate gears requiring replication before\nrouting. Substantive finding: <strong>the correction structure is finer-grained\nthan (model) or (model, family) — at least (model, family, material-class).</strong>\nArtifact: <code>analysis/h4_transfer.json</code> (seed 20260702, deviations: none);\nverdict theorems: <code>lean/Round1_H4.lean</code> (type-checked).</p>\n<h2 id=\"literature-positioning-2026-07-02-from-the-adversarially-verified-synthesis\">Literature positioning (2026-07-02, from the adversarially-verified synthesis)</h2><p>A 104-agent deep-research pass (25 claims verified 3-0 or merged; 22 primary\nsources; full artifact in the session transcript) situates Round 1:</p>\n<ul>\n<li><strong>H3 is the published softening pathology, quantified on our instrument.</strong>\nDeng et al. (npj Comput. Mater. 2024/25; arXiv:2405.07105) document &gt;90%\nsoftening prevalence in MPtrj-trained uMLIPs across surfaces, defects,\nphonons, and barriers. Our 15–60× defect/bulk ratios are that phenomenon\nunder a pre-registered, coupling-aware protocol.</li>\n<li><strong>H1/H2&#39;s kills answer the field&#39;s open question.</strong> The synthesis found NO\npublished study of cross-property error-correlation for foundation MLIPs\n(the closest: PCA over 47 metrics × 34 <em>classical</em> Ni potentials,\narXiv:2510.18033, which itself warns of partially orthogonal components).\nRound 1 is, to the limit of this search, the first pre-registered answer:\nfor MACE-MP-small/medium and CHGNet across 15 metals, cross-property error\nstructure does not compress below a family-coupling-aware null.</li>\n<li><strong>H4 is the first transferability-of-correction study.</strong> Verbatim from the\nsynthesis&#39;s open questions: &quot;No published transferability-of-correction\nstudy surfaced.&quot; The published rescaling correction that works (single DFT\nanchor, ~15% force-MAE recovery) is <em>per-system</em> — consistent with H4&#39;s\nfinding that a per-model scalar has no traction. The correction grain the\nfield&#39;s method implies is the grain our kill measured.</li>\n<li><strong>Caveats adopted:</strong> the classical-Ni PCA must not be silently generalized\nto foundation models; leaderboard numbers are model-version-specific.\n<em>(Citation correction 2026-07-02: this list previously cited\narXiv:2507.15190 for a &quot;residual survives OMat24&quot; claim; that preprint is\nwithdrawn and its abstract never made the attributed claim. Struck.)</em></li>\n<li><strong>Protocols adopted for Round 2+:</strong> MatCalc <code>PESCalculator.load_universal()</code>\nas the X-axis expansion path (7+ architectures, one interface); CPS-style\nmulti-property scoring; the 47-metric Ni suite as an external cross-check\nfor our Ni cells; fixed-and-reported displacement magnitudes for any\nphonon lane (ORB/eqV2 degrade at small displacements, arXiv:2412.10516).</li>\n</ul>\n<h2 id=\"r2-h1-result-2026-07-02-mace-mpa-0-through-the-identical-matrix\">R2-H1 result (2026-07-02, MACE-MPA-0 through the identical matrix)</h2><p><strong>Registered verdict: KILL — by 2.6%</strong> (ratio 7.86 vs registered threshold\n7.66). The decomposition, reported alongside because both halves are decidable\nfacts (<code>lean/Round2_Verdicts.lean</code>, type-checked):</p>\n<ul>\n<li>Defect median |rel err| <strong>collapsed 2.6×</strong> (4.65% vs MACE-MP-medium&#39;s\n12.13%) — the numerator behaved exactly as the distributional story\npredicts.</li>\n<li>Bulk median also improved (0.59% vs 0.79%) — the registered <em>ratio</em> missed\nits halving because the model got better everywhere. (Lesson for future\nregistrations: ratios move with both ends; register absolute family errors\nas co-primary next time.)</li>\n<li>Softening <strong>bias eliminated</strong>: s = 0.970 (MPtrj models: 1.057–1.125); Fe\nflipped from 2× soft to 13% stiff. But per-metal spread persists\n(0.85–1.39), now centered on 1 — <strong>OMat24 retraining removed the bias, not\nthe variance</strong>. <em>(Citation correction 2026-07-02: an earlier version cited\narXiv:2507.15190 as corroboration; that preprint is withdrawn and did not\nsupport the paraphrase. The variance-persistence claim is our own\nmeasurement, standing alone.)</em></li>\n</ul>\n<p>Artifact: <code>analysis/r2_mpa0_confirmatory.json</code> (joint 4-model run, matched\n15-metal set unchanged).</p>\n<h2 id=\"registered-next-tests\">Registered next tests</h2><ol>\n<li><strong>H4 follow-up:</strong> class-resolved operators — fit scalars per\n(model, family, structure/magnetic class) with LOO inside each class; the\ntwo isolated H4 improvements (CHGNet/surfaces, MACE-medium/planar-fault)\nare the first replication targets on new materials.</li>\n<li><strong>CHGNet-primary and MACE-medium-secondary H1 passes</strong> are follow-up leads,\nnot confirmations (2/6 at the 5% level is compatible with chance): a\nreplication set (new materials — hcp metals; MgO/NaCl now have references)\nis the honest next test.</li>\n<li><strong>Fe/Cr magnetic-element lane</strong>: Fe (soft, B₀′ &lt; 0) and Cr (stiff) anchor a\ndedicated magnetic-failure study per the investigation memo.</li>\n<li>Ledger landing of the 345 bound records + 57 verified theorem modules\n(blocked on task #12, the ingest-range fix).</li>\n</ol>\n"}