{"id":"mlip-elastic-operator-failure-diagnosis","title":"Operator Failure Diagnosis","subtitle":"Why the LOO-PCA bias operator degraded MAE and what scalar-bulk operator v0.2 learned instead.","category":"validation","tags":["mlip","operator","bias-correction","diagnosis"],"source":"articles/mlip-elastic-benchmark/operator-failure-diagnosis-2026-06-27.md","lang":"en","words":2327,"readMinutes":11,"toc":[{"depth":2,"text":"Executive Summary","id":"executive-summary"},{"depth":2,"text":"Why the Global Operator Failed","id":"why-the-global-operator-failed"},{"depth":2,"text":"Error-Geometry Diagnostics","id":"error-geometry-diagnostics"},{"depth":3,"text":"TensorNet/PBE error matrix (vs TPBE0K)","id":"tensornet-pbe-error-matrix-vs-tpbe0k"},{"depth":3,"text":"Per-element raw errors vs TPBE0K","id":"per-element-raw-errors-vs-tpbe0k"},{"depth":3,"text":"Global LOO-PCA residuals vs Tr2SCAN0K","id":"global-loo-pca-residuals-vs-tr2scan0k"},{"depth":3,"text":"Comparison to Layer-1 classical bias vectors","id":"comparison-to-layer-1-classical-bias-vectors"},{"depth":2,"text":"Alternative Operators Table","id":"alternative-operators-table"},{"depth":2,"text":"v0.2 scalar-bulk operator","id":"v0-2-scalar-bulk-operator"},{"depth":3,"text":"Scalar-bulk per-element α and predictions","id":"scalar-bulk-per-element-and-predictions"},{"depth":2,"text":"Recommended v0.2 Operator","id":"recommended-v0-2-operator"},{"depth":2,"text":"Cell-size independence check","id":"cell-size-independence-check"},{"depth":2,"text":"Variance subset","id":"variance-subset"},{"depth":2,"text":"v0.3 Directional correction scheme and feedback loop","id":"v0-3-directional-correction-scheme-and-feedback-loop"},{"depth":3,"text":"Lean formalization","id":"lean-formalization"},{"depth":3,"text":"Python feedback loop","id":"python-feedback-loop"},{"depth":3,"text":"FeedbackLoop benchmark results","id":"feedbackloop-benchmark-results"},{"depth":2,"text":"Implications for the Distillation Engine","id":"implications-for-the-distillation-engine"}],"html":"

Lupine Projection Law Operator Failure Diagnosis — Layer-2 MLIP Benchmark

Date: 2026-06-27

\n

Scope: 16 cubic elemental metals; TensorNet/PBE 1×1×1 predictions vs. TPBE_0K / Tr2SCAN_0K targets.

\n

Sources:

\n\n

Executive Summary

\n

Why the Global Operator Failed

The v0.1 operator assumes a single, shared 1D error direction across all 16 elements. The data refute this:

\n\n

Error-Geometry Diagnostics

TensorNet/PBE error matrix (vs TPBE_0K)

\n

Per-element raw errors vs TPBE_0K

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ElementClassC11 errC12 errC44 errMAE
Agnoble_coinage_fcc1.02-1.508.443.65
Alpost_transition-7.88-20.203.7510.61
Aunoble_coinage_fcc-21.14-30.9712.0321.38
Caalkaline_earth_fcc-0.40-5.431.712.51
Crtransition_bcc46.5280.54-10.4845.85
Cunoble_coinage_fcc15.47-13.300.419.72
Fetransition_bcc-26.99-32.81-2.1320.64
Motransition_bcc-25.07-10.113.9213.03
Nbtransition_bcc-18.01-41.906.0121.97
Nitransition_fcc-21.88-4.17-1.449.16
Pdtransition_fcc-0.57-12.785.896.41
Pttransition_fcc-13.04-29.8813.0518.65
Sralkaline_earth_fcc6.09-0.13-0.562.26
Tatransition_bcc-29.16-5.80-20.8618.61
Vtransition_bcc30.47-10.94-0.4013.94
Wtransition_bcc-24.70-15.892.4314.34

Global LOO-PCA residuals vs Tr2SCAN_0K

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ElementC11 resC12 resC44 resMAE
Ag0.44-2.308.563.77
Al-8.47-21.003.8611.11
Au-21.73-31.7712.1221.87
Ca-0.98-6.241.823.01
Cr45.6180.14-10.4945.41
Cu14.90-14.110.529.84
Fe-27.57-33.62-2.0021.06
Mo-25.64-10.924.0313.53
Nb-18.62-42.686.1122.47
Ni-22.46-4.98-1.339.59
Pd-1.15-13.586.006.91
Pt-13.63-30.6813.1419.15
Sr5.51-0.94-0.452.30
Ta-29.74-6.61-20.7419.03
V29.93-11.77-0.2813.99
W-25.27-16.702.5414.84

Comparison to Layer-1 classical bias vectors

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
FamilyNParticipation ratioEffective rank
Ackland-198780.4493
Ackland-199710.9680
Adams-198950.7873
Chamati-200610.9340
Foiles-198630.5272
Han-200320.5461
Howells-201810.7540
Olsson-200910.4830
MLIP global error162.1163

Layer-1 bias vectors were family-specific and often 1D within a family. The MLIP errors are not globally low-rank; the low-dimensional structure is class-local, not universal.

\n

Alternative Operators Table

All operators use only the existing 1×1×1 TensorNet/PBE outputs. Cost equals one raw 1×1×1 run.

\n
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Operatormean MAE vs TPBEmedian MAE vs TPBEmean MAE vs Tr2SCANmedian MAE vs Tr2SCANNotes
raw14.5513.4822.5522.50
shift-only16.2815.7414.5513.48exact PBE→r2SCAN functional shift
global-loo-pca63.4065.3354.2855.09v0.1 operator; degrades accuracy
global-loo-pca-unshifted54.2855.0945.8346.05PCA bias without functional shift
scalar-bulk19.1719.0114.1311.16v0.2 recommended on Tr2SCAN target
ensemble-1x1x111.6011.6219.8919.653-model PBE ensemble

Key observations:

\n\n

v0.2 scalar-bulk operator

The v0.2 operator, scalar-bulk, is a leave-one-out scalar re-scaling of the bulk-modulus functional shift. For each held-out element, a single scalar α is fit on the other 15 elements so that the bulk modulus of raw + α · (Tr2SCAN − TPBE) matches the Tr2SCAN bulk modulus; that α is then applied to the held-out element's shift. It uses only the existing 1×1×1 TensorNet/PBE output and the target-derived shift, so its cost equals one raw 1×1×1 run.

\n

Aggregate performance (1×1×1 TensorNet/PBE, LOO scalar-bulk):

\n
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Metricvs TPBE_0Kvs Tr2SCAN_0K
mean MAE19.17 GPa14.13 GPa
median MAE19.01 GPa11.16 GPa
core-hours0.01340.0134

On the Tr2SCAN-corrected target, scalar-bulk improves over raw (22.55 GPa), shift-only (14.55 GPa), and the 3-architecture ensemble (19.89 GPa), while remaining 2.70× cheaper than the ensemble and 3.86× cheaper than the 3×3×3 reference. The v0.1 global LOO-PCA operator is far worse (54.28 GPa). On the PBE headline target it does not beat raw or the ensemble; no single-model operator does.

\n

Scalar-bulk per-element α and predictions

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ElementαC11 corrC12 corrC44 corrMAE vs Tr2SCAN
Ag1.338134.0596.7260.667.32
Al1.32296.0552.7535.6110.61
Au1.305149.66115.4339.5220.73
Ca1.32220.439.6615.802.51
Cr1.580596.29233.73102.1352.45
Cu1.352200.56152.4190.3811.71
Fe1.223251.43136.41107.6716.91
Mo1.294476.13157.35116.9310.57
Nb1.328213.65102.4516.9022.15
Ni1.316305.80185.02155.065.65
Pd1.317223.86163.3990.617.80
Pt1.221314.78224.1586.8414.81
Sr1.32221.2910.3611.932.26
Ta1.299257.67164.7153.2015.57
V1.345335.95134.4316.8815.14
W1.259542.70207.81161.239.94

Recommended v0.2 Operator

Adopt scalar-bulk as the v0.2 operator on the Tr2SCAN-corrected target.

\n\n

Cell-size independence check

To confirm that scalar-bulk is a cell-size-independent operator rather than an artifact of the small conventional cell, the LOO alphas were fit on the existing 3×3×3 raw predictions and applied to the 3×3×3 Tr2SCAN-corrected target.

\n
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Metricscalar-bulk 1×1×1scalar-bulk 3×3×3
mean MAE vs Tr2SCAN_0K14.13 GPa14.14 GPa
median MAE vs Tr2SCAN_0K11.16 GPa11.20 GPa

The 1×1×1 and 3×3×3 means differ by 0.01 GPa and the medians by 0.04 GPa. The operator is therefore cell-size independent: it transfers from the cheap conventional cell to the expensive supercell reference with no measurable loss of accuracy.

\n

Variance subset

A variance subset was run to check seed-to-seed stability: Ca, Cu, Fe, and Cr were each run with three different random seeds (12 cases total) using TensorNet/PBE 1×1×1.

\n\n

This removes a key uncertainty from the cost-accuracy ratios: the single-model cost is not being inflated by hidden seed variance on the elements most prone to numerical instability.

\n

v0.3 Directional correction scheme and feedback loop

The failure of the global operator motivated a first-principles class-aware formalization rather than another ad-hoc fix.

\n

Lean formalization

OpenDistillationFactory/Materials/Distillation/DirectionalCorrectionScheme.lean introduces a universal operator that assigns one correction direction to each class:

\n\n

This is the abstraction above v0.2:

\n\n

The file builds cleanly and is mirrored to lupine-rhizo/lean-spec.

\n

Python feedback loop

lupine/feedback.py implements the operational counterpart:

\n\n

For the current MLIP benchmark this provides a principled path from v0.2 scalar-bulk to a fully class-aware operator: the same bulk-modulus direction can be reused while alphas and offsets are allowed to vary by bonding class.

\n

FeedbackLoop benchmark results

The loop was benchmarked with leave-one-out element holdout on the 1×1×1 TensorNet/PBE data. Two alpha policies were tested:

\n\n

Three offset modes were tested for each policy: none, median, and oracle (oracle uses the held-out sample's own residual and is shown only as an empirical ceiling).

\n
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Operatormean MAE vs TPBEmedian MAE vs TPBEmean MAE vs Tr2SCANmedian MAE vs Tr2SCANoutliers (Tr2SCAN)
scalar-bulk (v0.2)19.1719.0114.1311.16—
feedback-scalar_bulk-offset-none16.3215.9014.2813.192/16
feedback-scalar_bulk-offset-median16.4515.6714.4113.192/16
feedback-scalar_bulk-offset-oracle12.5014.0510.619.872/16
feedback-projection-offset-none18.4417.1013.269.032/16
feedback-projection-offset-median18.3116.6413.139.032/16
feedback-projection-offset-oracle14.3715.389.348.012/16

Key findings:

\n\n

The v0.2 scalar-bulk operator remains the recommended production choice because it is the simplest operator that already beats the ensemble on Tr2SCAN. The feedback-projection policy is the immediate v0.3 candidate: it is strictly better on Tr2SCAN and only slightly more complex.

\n

Implications for the Distillation Engine

    \n
  1. The binding constraint must be class-aware. A single global binding axis overfits to the bulk-like component and mis-corrects classes with different error geometry. The distillation engine should partition the calibration set by bonding class (or by a learned similarity neighborhood) before extracting the bias direction.
    2. \n
  2. Functional shift and bias are not interchangeable. The scalar bulk-modulus shift already removes much of the stiffness bias; the residual class-mean bias is smaller and more class-local. Future versions should fit the bias on the residual after shift, not on the raw error.
    3. \n
  3. Headline framing must specify the target. Against the corrected target (Tr2SCAN_0K), scalar-bulk beats both raw and the 3-architecture ensemble at single-run cost. Against the PBE headline target (TPBE_0K), the ensemble remains the accuracy benchmark and no single-model operator beats raw. The cost-accuracy claim survives as a supercell-independence story (corrected 1×1×1 vs 3×3×3 reference), but the ensemble-beating claim is only supported on the Tr2SCAN target.
    4. \n
  4. Variance and seed checks are now complete. The 4-element, 3-seed subset (Ca, Cu, Fe, Cr) shows TensorNet/PBE 1×1×1 is deterministic; MAE standard deviations are ~0 GPa. The single-seed headline numbers are therefore seed-stable on the tested subset.
    5. \n
  5. Completed: the class-aware FeedbackLoop was benchmarked against v0.2 scalar-bulk. The projection policy improves Tr2SCAN mean MAE to 13.26 GPa (from 14.13 GPa) and TPBE mean MAE to 18.44 GPa (from 19.17 GPa), with 2/16 outliers flagged.
    6. \n
  6. Next experiment: test the projection policy on the 3×3×3 grid and evaluate bonding-class-specific directions (rather than the shared bulk-modulus direction) to see if the TPBE-target gap can be closed further.
    7. \n
\n"}