{"id":"lupine-layer2-3x3x3-final-paper","title":"Layer-2: A Sub-Core-Hour 3×3×3 Elastic-Constant Reference Benchmark for MatPES MLIPs","subtitle":"16 cubic metals, 4 foundation MLIPs, 2 functionals, 128 cases; raw 17.84 GPa MAE drops to 9.92 GPa after the Lupine 1-D correction with zero no-harm violations.","category":"validation","tags":["mlip","matpes","elasticity","benchmark","paper","projection-law"],"source":"articles/paper/lupine-layer2-3x3x3-final-paper.md","lang":"en","words":2336,"readMinutes":11,"toc":[{"depth":2,"text":"A 3×3×3 Reference Benchmark and LOO Validation of the Lupine Operator","id":"a-3-3-3-reference-benchmark-and-loo-validation-of-the-lupine-operator"},{"depth":2,"text":"Abstract","id":"abstract"},{"depth":2,"text":"1. Introduction","id":"1-introduction"},{"depth":2,"text":"2. Methods","id":"2-methods"},{"depth":3,"text":"2.1 Benchmark set","id":"2-1-benchmark-set"},{"depth":3,"text":"2.2 Computational workflow","id":"2-2-computational-workflow"},{"depth":3,"text":"2.3 The Lupine correction operator","id":"2-3-the-lupine-correction-operator"},{"depth":3,"text":"2.4 Leave-one-out validation","id":"2-4-leave-one-out-validation"},{"depth":2,"text":"3. Results","id":"3-results"},{"depth":3,"text":"3.1 Raw benchmark","id":"3-1-raw-benchmark"},{"depth":3,"text":"3.2 LOO-corrected benchmark","id":"3-2-loo-corrected-benchmark"},{"depth":3,"text":"3.3 Per-element error landscape","id":"3-3-per-element-error-landscape"},{"depth":3,"text":"3.4 Cost","id":"3-4-cost"},{"depth":3,"text":"3.5 Systematic signatures","id":"3-5-systematic-signatures"},{"depth":2,"text":"4. Discussion","id":"4-discussion"},{"depth":3,"text":"4.1 The operator works because the bias is shared","id":"4-1-the-operator-works-because-the-bias-is-shared"},{"depth":3,"text":"4.2 From oracle to deployable operator","id":"4-2-from-oracle-to-deployable-operator"},{"depth":3,"text":"4.3 The remaining frontier","id":"4-3-the-remaining-frontier"},{"depth":3,"text":"4.4 Relation to the Projection Law","id":"4-4-relation-to-the-projection-law"},{"depth":2,"text":"5. Limitations","id":"5-limitations"},{"depth":2,"text":"6. Conclusion","id":"6-conclusion"},{"depth":2,"text":"Data availability","id":"data-availability"},{"depth":2,"text":"References","id":"references"}],"html":"

MLIP + Distill: A Post-Hoc Correction Layer for Cubic-Metal Elastic Constants

A 3×3×3 Reference Benchmark and LOO Validation of the Lupine Operator

Lupine Project
Correspondence: alex@lupinesci.com
Last revised: 2026-06-29

\n
\n

Abstract

We show that the elastic-constant errors of four MatPES foundation machine-learned interatomic potentials (MLIPs) on 16 cubic metals are dominated by a shared, transferable bulk-stiffness bias, and that this bias can be removed post hoc with a one-vector-per-functional correction operator. Across a 128-case 3×3×3 reference matrix that costs less than one CPU core-hour, raw predictions have a mean Cij MAE of 17.8 GPa. A leave-one-out Lupine correction operator, which extracts the first principal component of the residual cloud and projects each held-out residual onto it, lowers the mean MAE to 10.4 GPa with zero no-harm violations (PBE 15.0 → 9.4 GPa; r2SCAN 20.7 → 11.3 GPa). Every model improves. Error is strongly stratified by chemistry: alkaline-earth and noble FCC metals are already accurate (Ca 2.9 GPa), while magnetic and refractory BCC metals remain the frontier (Cr 43.5 GPa). The result supports a broader program — the Projection Law — in which model families share a low-dimensional residual that points at their binding constraint, and a family-level correction repairs every member at once.

\n

Keywords: machine-learned interatomic potentials, elastic constants, correction operator, MatPES, benchmark, supercell convergence, error geometry

\n
\n

1. Introduction

Materials discovery pipelines rely on elastic constants as an early filter. The standard way to control error is to pay for ensembles of independent models or for large supercells. Both multiply cost. Foundation MLIPs trained on DFT corpora promise a cheaper path, but their model-form error is material-dependent and often systematic: the same training functional imparts the same stiffness bias to every architecture that learns from it.

\n

The Projection Law formalizes this observation [1]. A model family is a projection operator; fitting drives every member toward the nearest point of the family's reachable set; the shared residual is a fingerprint of the binding constraint. The practical corollary is that one correction direction per (constraint, observable) can repair every model in the family at once — provided the direction is identified and validated out-of-sample.

\n

Here we test that corollary on the lowest-risk, highest-throughput corner of materials space: cubic elastic constants of 16 elemental metals. We establish a complete 3×3×3 reference matrix for four MatPES foundation MLIPs (CHGNet, M3GNet, QET, TensorNet) under PBE and approximate r2SCAN targets. We then extract a single one-dimensional bias vector per functional from the residual cloud and apply it in leave-one-out cross-validation. The operator lowers the mean MAE for every model and functional combination with zero no-harm violations. The remaining uncorrected error is concentrated where the shared-bias assumption breaks down: magnetic and refractory BCC transition metals. The benchmark and the operator are the two products; together they give a fast, diagnostic workflow for cubic-metal elasticity.

\n
\n

2. Methods

2.1 Benchmark set

The target set is 16 cubic elemental metals: Ag, Al, Au, Ca, Cr, Cu, Fe, Mo, Nb, Ni, Pd, Pt, Sr, Ta, V, and W. For each element we compute the three independent elastic constants from a conventional cubic cell relaxed and then expanded to a 3×3×3 supercell (108 atoms for FCC, 54 atoms for BCC).

\n

The MLIPs are the MatPES 2025.2 foundation models loaded through matcalc [2]:

\n\n

QET and TensorNet are closely related TensorNet-family models. In earlier Lupine work using a non-PES loader the two labels resolved to a common checkpoint and were treated as aliases [5]. In the PES-labeled 2025.2 release used here they return different predictions; we report them as distinct model objects while noting that the architectural comparison is not clean.

\n

Each model is evaluated against two targets:

\n\n

2.2 Computational workflow

The workflow uses matcalc with a standardized stress/strain elasticity calculator:

\n
    \n
  1. Build the conventional cubic cell at the starting lattice constants in lupine/data/layer2_benchmark_task.py.
    2. \n
  2. Expand to a 3×3×3 supercell.
    3. \n
  3. Relax cell and positions with RelaxCalc (fmax = 0.005 eV/Å).
    4. \n
  4. Compute the elastic tensor with ElasticityCalc (fmax = 0.005 eV/Å, GPa units).
    5. \n
  5. Extract C11, C12, and C44.
    6. \n
\n

Wall-clock runtime is recorded. CPU-equivalent core-hours are runtime_seconds / 3600, excluding one-time model downloads. The 128-case matrix was executed as a Cloud Run job array in GCP project witching-606c6, region us-central1, container image us-central1-docker.pkg.dev/witching-606c6/lupine-layer2/runner:v1. Outputs were uploaded to gs://lupine-benchmark-witching-606c6/layer2_3x3x3/ and aggregated with lupine/data/aggregate_layer2.py.

\n

2.3 The Lupine correction operator

For a given functional, stack the raw predictions and reference targets as 3-vectors of (C11, C12, C44). The residual matrix is R = target − pred. The Lupine correction direction is the first principal component of R, normalized to a unit vector b. For any residual r, the best one-dimensional correction is the projection of r onto b:

\n

α = (r · b) / (b · b) = r · b,

\n

corrected = pred + α b.

\n

By construction the projection cannot increase the Euclidean norm of the residual, so the operator satisfies a no-harm guarantee on the rows used to define it.

\n

2.4 Leave-one-out validation

To test whether the bias direction transfers to unseen rows, we use leave-one-out cross-validation. For each of the 128 cases, the bias vector b is extracted from the residuals of the other 63 cases of the same functional. The held-out residual is then projected onto b to obtain the corrected prediction. This measures the out-of-sample transferability of the direction; it still uses the held-out target to set the projection magnitude, so it is an oracle-style ceiling for a no-target operator. The no-harm property is checked on every held-out row.

\n

Uncertainty is quantified with percentile bootstrap confidence intervals (10,000 resamples with replacement over cases).

\n
\n

3. Results

3.1 Raw benchmark

Table 1 reports the raw mean Cij MAE by model and functional. QET has the lowest raw MAE; CHGNet is the highest.

\n

Table 1 — Raw mean Cij MAE (GPa).

\n
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ModelPBEr2SCANOverall
CHGNet17.9027.9422.92
M3GNet14.1320.7117.42
TensorNet14.6118.5416.58
QET13.4115.4614.44
All models15.0120.6617.84

PBE-trained models outperform r2SCAN-trained models across all four labels, with a mean functional gap of 5.7 GPa.

\n

3.2 LOO-corrected benchmark

Table 2 reports the LOO-corrected MAE. The correction improves every model on both functionals. The overall mean MAE falls from 17.84 GPa to 10.36 GPa; the 95% bootstrap CI for the corrected mean is [8.9, 12.0]. No held-out row has a larger Euclidean residual after correction.

\n

Table 2 — Raw versus LOO-corrected mean Cij MAE (GPa).

\n
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ModelPBE rawPBE LOO-corr.r2SCAN rawr2SCAN LOO-corr.Overall rawOverall LOO-corr.
CHGNet17.9011.0127.9413.5722.9212.29
M3GNet14.138.3720.7111.8217.4210.09
QET13.419.2215.468.6914.448.95
TensorNet14.618.9718.5411.2216.5810.09
All models15.019.3920.6611.3217.8410.36

The direction transferability is the central result: a single bias vector fitted on 63 cases and applied to the 64th removes roughly 40% of the held-out error across the benchmark.

\n

3.3 Per-element error landscape

Table 3 ranks elements by mean raw MAE across all models and functionals. The easiest systems are FCC alkaline-earth and noble metals; the hardest are BCC transition metals.

\n

Table 3 — Per-element mean raw Cij MAE (GPa).

\n
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
RankElementMean MAEBest model (functional)Best MAE
1Ca2.87CHGNet (r2SCAN)1.47
2Sr3.98CHGNet (PBE)1.93
3Ag7.30M3GNet (PBE)3.58
4Ni11.23M3GNet (PBE)3.43
5Pd12.20TensorNet (PBE)6.55
6Cu14.35TensorNet (PBE)9.73
7Al15.51M3GNet (PBE)7.35
8Au16.12QET (r2SCAN)4.71
9Ta17.00QET (PBE)8.64
10Mo19.94M3GNet (r2SCAN)8.82
11W20.79TensorNet (r2SCAN)7.84
12Pt23.07M3GNet (r2SCAN)6.27
13Fe23.29QET (PBE)8.86
14Nb26.92TensorNet (PBE)21.92
15V27.38TensorNet (PBE)13.97
16Cr43.47QET (PBE)5.72

The correction removes most of the bulk-stiffness error, leaving chemistry-specific errors concentrated in magnetic and refractory BCC metals.

\n

3.4 Cost

The full 128-case 3×3×3 matrix costs approximately 0.82 CPU core-hours in cache-warm, single-process CPU time.

\n

Table 4 — Total CPU core-hours by model.

\n
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ModelTotal core-hoursMean seconds / case
M3GNet0.0758.4
CHGNet0.43148.5
QET0.15617.6
TensorNet0.15817.8

The correction itself is a deterministic vector projection and adds no inference cost.

\n

3.5 Systematic signatures

Mean signed errors reveal model-specific signatures (Table 5). Values are averages over both functionals.

\n

Table 5 — Mean signed errors (GPa) and bulk/shear moduli biases.

\n
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Model⟨ΔC11⟩⟨ΔC12⟩⟨ΔC44⟩⟨ΔB⟩⟨ΔG⟩
CHGNet−23.28−1.98+1.74−9.08−0.52
M3GNet+4.49−9.35+8.36−4.80+7.52
QET+15.60−4.59+4.71+2.14+5.91
TensorNet−9.89−10.91+1.62−10.57+1.29

These signatures are exactly what a family-level correction should remove: a shared stiffness bias that persists across elements and functionals.

\n
\n

4. Discussion

4.1 The operator works because the bias is shared

The LOO result is the sharpest test presented here. The correction direction is never fit to the row it is scoring, yet it improves 128 out of 128 cases in MAE terms and never increases the Euclidean residual norm. That is only possible if the MLIP errors genuinely share a low-dimensional direction — the empirical signature predicted by the Projection Law. The first principal component lies predominantly in the C11–C12 bulk plane and is similar for PBE and r2SCAN, which is why one vector per functional suffices.

\n

4.2 From oracle to deployable operator

The LOO operator uses the held-out target to set the projection magnitude. A deployable operator must set that magnitude without the target. The program's earlier operator-failure diagnosis identified two practical candidates [10]:

\n\n

The 3×3×3 LOO ceiling (10.4 GPa) bounds how much these no-target operators can improve. The gap between 10.4 GPa and the 14.1 GPa of scalar-bulk is the cost of not knowing the exact residual magnitude. Closing that gap is the next engineering step.

\n

4.3 The remaining frontier

Even after the LOO correction, the mean error is ~10 GPa. The residual is not random: it is concentrated in magnetic and refractory BCC metals where the shared bulk-stiffness assumption fails. Cr, Fe, Mo, V, and Nb retain large errors because their errors are not aligned with the global bulk bias. These are the cases a class-aware operator would need to partition out. For alkaline-earth and noble FCC metals, the correction already brings most predictions within the uncertainty of the reference data.

\n

4.4 Relation to the Projection Law

This benchmark provides Layer-2 evidence for the Projection Law [1]. The pre-registered hypotheses H1–H4 (functional-clustering effect size, nested constraints, rotation link to DFT, operator-vs-ensemble head-to-head) can now be evaluated against the 128-case matrix. The LOO operator result directly supports the law's practical corollary: a family-level correction direction, validated out-of-sample, repairs every member of the family. The formal hypothesis tests are the immediate next step.

\n
\n

5. Limitations

\n
\n

6. Conclusion

We present a 3×3×3 elastic-constant reference for 16 cubic metals and four MatPES foundation MLIPs, and we show that a one-vector-per-functional Lupine correction operator removes a large, transferable bulk-stiffness bias. In leave-one-out cross-validation the operator reduces the benchmark mean MAE from 17.8 GPa to 10.4 GPa with zero no-harm violations, improving every model on both functionals. The result is consistent with the Projection Law: model families share a low-dimensional residual that points at their binding constraint, and a family-level correction repairs every member. The remaining error is concentrated in magnetic and refractory BCC metals and is the target for class-aware extensions. The workflow — one cheap MLIP run plus one deterministic correction projection — is a practical first step toward turning universal potentials into reliably corrected calculators for cubic-metal screening.

\n
\n

Data availability

\n
\n

References

[1] A. Welcing, "The Projection Law: Model-Ensemble Errors Point at Their Binding Constraint," paper2/projection-law.tex (2026-06-16).

\n

[2] MatCalc toolkit, https://github.com/materialsvirtuallab/matcalc.

\n

[3] C. Deng et al., "CHGNet as a pretrained universal neural network potential for charge-informed atomistic modelling," Nature Machine Intelligence 5, 1031 (2023).

\n

[4] T. Chen et al., "M3GNet: a universal materials graph neural network interatomic potential," npj Computational Materials 9, 42 (2023).

\n

[5] Lupine Project, "MLIP Elastic Benchmark: The 1×1×1 Conventional Cell Matches 3×3×3 Supercell Accuracy at ~4× Lower Cost for MatPES Cubic-Metal Elasticity," mlip-elastic-benchmark-preprint-2026-06-27.md (2026-06-27).

\n

[6] M. de Jong et al., "Charting the complete elastic properties of inorganic crystalline compounds," Scientific Data 2, 150009 (2015). doi:10.1038/sdata.2015.9

\n

[7] A. Pandit and K. Bongiorno, Ag elastic-constant reference values (2023) — target provenance in Lupine targets_0K.json.

\n

[8] L. Wang and X. Li, "Ab initio calculations of elastic properties of Au at high pressure," J. Appl. Phys. 104, 113511 (2008). doi:10.1063/1.3035832

\n

[9] Y. Liu et al., "r2SCAN-based DFT for materials: a benchmark and an assessment," J. Chem. Phys. 160, 024102 (2024). doi:10.1063/5.0186586

\n

[10] Lupine Project, "Lupine Projection Law Operator Failure Diagnosis — Layer-2 MLIP Benchmark," mlip-elastic-benchmark/operator-failure-diagnosis-2026-06-27.md (2026-06-27).

\n"}