{"id":"formal-methodology","title":"In the In Between","subtitle":"Why we formalize before we simulate. A methodology for theorem-driven validation.","category":"formalization","tags":["lean","methodology","epistemology"],"source":"articles/docs/formal-methodology.md","lang":"en","words":854,"readMinutes":4,"toc":[{"depth":2,"text":"The False Dichotomy","id":"the-false-dichotomy"},{"depth":2,"text":"What \"In Between\" Means","id":"what-in-between-means"},{"depth":2,"text":"The Three Layers","id":"the-three-layers"},{"depth":3,"text":"Layer 1: Synthetic Data (Proven)","id":"layer-1-synthetic-data-proven"},{"depth":3,"text":"Layer 2: Computed Properties (Proven by nativedecide)","id":"layer-2-computed-properties-proven-by-nativedecide"},{"depth":3,"text":"Layer 3: The Epistemic Gap (Documented)","id":"layer-3-the-epistemic-gap-documented"},{"depth":2,"text":"Why Not Just Run LAMMPS?","id":"why-not-just-run-lammps"},{"depth":2,"text":"The Audit as Methodology","id":"the-audit-as-methodology"},{"depth":2,"text":"Future Work: Closing the Gap","id":"future-work-closing-the-gap"},{"depth":2,"text":"Summary","id":"summary"},{"depth":2,"text":"Related","id":"related"}],"html":"<h1 id=\"in-the-in-between-a-methodology-for-formal-validation\">In the In Between: A Methodology for Formal Validation</h1><blockquote>\n<p><em>We are not running LAMMPS. We are formalizing what it means to validate without running it all.</em></p>\n</blockquote>\n<p>This page explains why we built a Lean 4 formal specification instead of installing LAMMPS and running simulations.</p>\n<hr>\n<h2 id=\"the-false-dichotomy\">The False Dichotomy</h2><p>Computational materials science operates on a false dichotomy:</p>\n<ol>\n<li><strong>Run everything</strong> — brute-force benchmark every potential on every material. Expensive. Slow. Never complete.</li>\n<li><strong>Trust the PDF</strong> — read the paper, look at the RMSE table, and hope the authors checked their work.</li>\n</ol>\n<p>There is a third option:</p>\n<ol start=\"3\">\n<li><strong>Formalize the gap</strong> — write down exactly what would need to be true for the claim to be valid, prove what you can prove, and document the rest as acknowledged gaps.</li>\n</ol>\n<hr>\n<h2 id=\"what-quot-in-between-quot-means\">What &quot;In Between&quot; Means</h2><p>An interatomic potential claim sits on a spectrum:</p>\n<div class=\"table-wrap\"><table><thead><tr>\n<th>Position</th>\n<th>Example</th>\n<th>Status</th>\n</tr>\n</thead><tbody><tr>\n<td data-label=\"Position\"><strong>Formal proof</strong></td>\n<td data-label=\"Example\">PR ≤ min(P, N) for differentiable potentials</td>\n<td data-label=\"Status\">Proven for synthetic data; real proof needs inverse function theorem</td>\n</tr>\n<tr>\n<td data-label=\"Position\"><strong>In between</strong></td>\n<td data-label=\"Example\">Ground-truth NIST errors converge to the synthetic manifold</td>\n<td data-label=\"Status\">Well-posed conjecture; no ground truth yet</td>\n</tr>\n<tr>\n<td data-label=\"Position\"><strong>Physical experiment</strong></td>\n<td data-label=\"Example\">NIST IPR elastic constants</td>\n<td data-label=\"Status\">Available but not yet computed with our pipeline</td>\n</tr>\n</tbody></table></div><p>The &quot;in between&quot; is where most validation science actually lives. Papers pretend they are at the experiment end. We formalize that they are in the middle.</p>\n<hr>\n<h2 id=\"the-three-layers\">The Three Layers</h2><h3 id=\"layer-1-synthetic-data-proven\">Layer 1: Synthetic Data (Proven)</h3><p>We embedded 72 FCC + 42 BCC synthetic entries directly into the Lean source. These are not loaded from a file at runtime — they are <strong>compile-time constants</strong>.</p>\n<p><strong>Why this matters:</strong> If someone edits the data, the theorems about the data are re-checked at build time. You cannot silently change a value and break a proof without the build failing.</p>\n<p><strong>Theorems at this layer:</strong></p>\n<ul>\n<li>Count theorems (<code>syntheticFccCount = 72</code>)</li>\n<li>Provenance theorems (<code>syntheticFccIsSynthetic = true</code>)</li>\n<li>Structural theorems (<code>nistScaffoldAlMissing = true</code>)</li>\n</ul>\n<h3 id=\"layer-2-computed-properties-proven-by-native-decide\">Layer 2: Computed Properties (Proven by <code>native_decide</code>)</h3><p>We compute statistics (correlation, participation ratio, Simpson&#39;s paradox detection) and prove properties about the computed values.</p>\n<p><strong>Why <code>native_decide</code>:</strong> Lean compiles the expression to native code, runs it, and checks the result. This is not a symbolic proof — it is a <strong>reproducible computation</strong> that happens at compile time.</p>\n<p><strong>Theorems at this layer:</strong></p>\n<ul>\n<li><code>pooledRBelowMinus08</code> — pooled correlation &lt; −0.8</li>\n<li><code>fccEamPRBounded</code> — PR ∈ (1.2, 1.3)</li>\n<li><code>fccAllSatisfiesHyperRibbon</code> — PR/n &lt; 0.5</li>\n</ul>\n<h3 id=\"layer-3-the-epistemic-gap-documented\">Layer 3: The Epistemic Gap (Documented)</h3><p>Some claims require ground-truth data we do not have. Rather than pretend they are proven, we document them as gaps.</p>\n<p><strong>Why documented gaps are honest:</strong> Every theorem is fully proven. The gaps are not <code>sorry</code> proofs — they are explicit records of what would need to happen to upgrade from synthetic to NIST-backed validation.</p>\n<p><strong>Documented gaps:</strong></p>\n<ul>\n<li>All predicted values are hand-typed, not computed from NIST IPR potentials</li>\n<li>Reference values lack DOI citations</li>\n<li>Experiment was not pre-registered</li>\n<li>No formal data provenance tracking in the original Rust code</li>\n<li>Bootstrap CIs rely on non-deterministic random sampling</li>\n</ul>\n<hr>\n<h2 id=\"why-not-just-run-lammps\">Why Not Just Run LAMMPS?</h2><p>We could install LAMMPS, download NIST potentials, and compute elastic constants for Al. Here&#39;s why we chose not to:</p>\n<ol>\n<li><p><strong>Scope control.</strong> Running one simulation creates pressure to run ten. Formalizing the validation structure is invariant to how many simulations we run.</p>\n</li>\n<li><p><strong>Reproducibility.</strong> A LAMMPS trace is a reproducible artifact — but only if you save the exact input script, potential file, and LAMMPS version. We formalized what a valid trace looks like (<code>Computation.LammpsTrace.lean</code>) before producing any traces.</p>\n</li>\n<li><p><strong>Epistemic clarity.</strong> Running a simulation gives you a number. Proving a theorem gives you a <strong>guarantee</strong> that holds for all inputs in a class. We need both, but the formal layer comes first.</p>\n</li>\n<li><p><strong>Cost.</strong> GPU time is expensive. Compile time is cheap. We can iterate on hypotheses at compile time, then run simulations only for the hypotheses that survive formal scrutiny.</p>\n</li>\n</ol>\n<hr>\n<h2 id=\"the-audit-as-methodology\">The Audit as Methodology</h2><p>Our audit report is not a blog post. It is a <strong>computed string</strong> generated from proven theorems:</p>\n<pre><code class=\"language-lean\">def fullAuditReport : String :=\n  &quot;[CLAIM 1] Simpson&#39;s Paradox in BCC Elastic Constants\\n&quot; ++\n  &quot;  &quot; ++ simpsonsParadoxVerdict ++ &quot;\\n\\n&quot; ++\n  &quot;[CLAIM 2] Hyper-Ribbon Manifold Dimensionality\\n&quot; ++\n  &quot;  &quot; ++ hyperRibbonVerdict ++ &quot;\\n\\n&quot; ++\n  ...\n</code></pre>\n<p>The verdict strings are themselves proven:</p>\n<ul>\n<li><code>simpsonVerdictContainsFabricated</code> — the string contains &quot;FABRICATED&quot;</li>\n<li><code>hyperRibbonVerdictContainsConsistent</code> — the string contains &quot;CONSISTENT&quot;</li>\n</ul>\n<p>If the computed statistics shift, the build fails. The audit cannot drift.</p>\n<hr>\n<h2 id=\"future-work-closing-the-gap\">Future Work: Closing the Gap</h2><p>The documented gaps define exactly what needs to happen to close the epistemic boundary:</p>\n<ol>\n<li>Compute NIST-backed LAMMPS traces for the 9-row Al scaffold</li>\n<li>Re-run the paradox detector on real data</li>\n<li>Re-compute participation ratios on real error vectors</li>\n<li>Prove (or refute) the hyper-ribbon claim on ground truth</li>\n<li>Seed the RNG and record the seed for deterministic CIs</li>\n</ol>\n<p>Each step is a well-posed theorem schema. The gap is a <strong>TODO list written in logic</strong>.</p>\n<hr>\n<h2 id=\"summary\">Summary</h2><div class=\"table-wrap\"><table><thead><tr>\n<th>Approach</th>\n<th>Cost</th>\n<th>Guarantee</th>\n<th>Gap visibility</th>\n</tr>\n</thead><tbody><tr>\n<td data-label=\"Approach\">Run everything</td>\n<td data-label=\"Cost\">$$$$</td>\n<td data-label=\"Guarantee\">None (never complete)</td>\n<td data-label=\"Gap visibility\">Hidden</td>\n</tr>\n<tr>\n<td data-label=\"Approach\">Trust the PDF</td>\n<td data-label=\"Cost\">$</td>\n<td data-label=\"Guarantee\">None</td>\n<td data-label=\"Gap visibility\">Hidden</td>\n</tr>\n<tr>\n<td data-label=\"Approach\"><strong>Formalize the gap</strong></td>\n<td data-label=\"Cost\">$</td>\n<td data-label=\"Guarantee\">Compile-time checked</td>\n<td data-label=\"Gap visibility\"><strong>Explicit (documented)</strong></td>\n</tr>\n</tbody></table></div><p>We chose the third path. The build passes. The gap is documented. The hypotheses are falsifiable.</p>\n<p>That is what it means to be <strong>in the in between</strong>.</p>\n<hr>\n<h2 id=\"related\">Related</h2><ul>\n<li><a href=\"/#/article/formal-vision\">The Executable Vision</a> — build-locking contract and theorem inventory</li>\n<li><a href=\"/#/article/formal-audit\">Formal Audit Report</a> — split verdict with computational evidence</li>\n<li><a href=\"/#/article/formal-hypotheses\">Six Meta-Scientific Hypotheses</a> — the new research agenda</li>\n</ul>\n"}