{"id":"mlip-elastic-benchmark-protocol","title":"MLIP Elastic Benchmark Protocol","subtitle":"Reusable protocol for the MatPES cubic-metal 16-element elastic-constant benchmark.","category":"validation","tags":["mlip","elasticity","benchmark","protocol","reproducibility"],"source":"articles/mlip-elastic-benchmark/mlip-elastic-benchmark-protocol-2026-06-27.md","lang":"en","words":382,"readMinutes":2,"toc":[{"depth":2,"text":"Claim under test","id":"claim-under-test"},{"depth":2,"text":"Element roster (16)","id":"element-roster-16"},{"depth":2,"text":"Four benchmark arms","id":"four-benchmark-arms"},{"depth":2,"text":"Bias operator (arm B)","id":"bias-operator-arm-b"},{"depth":2,"text":"Cost model","id":"cost-model"},{"depth":2,"text":"Execution steps","id":"execution-steps"},{"depth":2,"text":"Config","id":"config"},{"depth":2,"text":"Kill conditions / caveats","id":"kill-conditions-caveats"}],"html":"

MLIP Elastic Benchmark Protocol — 10× Cost Reduction for MLIP Elastic-Constant Validation

\n

Date: 2026-06-27
Parent task: t_0266945a (synthesizer)
Design source: lupine-rhizo/docs/plans/mlip-elastic-benchmark-master-2026-06-27.md (binding; this protocol is the runnable translation)

\n
\n

Claim under test

On a 16-element cubic-metal benchmark, a Lupine-corrected single-model 1×1×1 elastic-constant calculation matches the accuracy of a 3×3×3 reference run while costing ~10× fewer CPU-seconds, and outperforms a 3-architecture ensemble while costing ~5× fewer CPU-seconds.

\n

Element roster (16)

Ag Al Au Ca Cr Cu Fe Mo Nb Ni Pd Pt Sr Ta V W

\n

Four benchmark arms

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ArmLabelModelCellCorrection
Araw-1x1x1TensorNet/PBE1×1×1None
Bcorrected-1x1x1TensorNet/PBE1×1×1LOO-PCA bias + functional shift
Cref-3x3x3TensorNet/PBE3×3×3None
Densemble-1x1x1M3GNet/PBE + CHGNet/PBE + TensorNet/PBE1×1×1None (mean)
\n

Bias operator (arm B)

    \n
  1. For each element, compute the TensorNet/PBE 1×1×1 error vector: raw − TPBE_0K.
    2. \n
  2. Leave-one-out: fit the first principal component of the centered error matrix on the other 15 elements; apply that bias to the held-out element.
    3. \n
  3. Add functional shift Tr2SCAN_0K − TPBE_0K.
    4. \n
  4. Report mean and median MAE over the 16 LOO-corrected predictions.
    5. \n
\n

Implementation: lupine/python/lupine/operator.py:correct() and leave_one_out_calibration().

\n

Cost model

core_hours = runtime_seconds × n_cores / 3600, with n_cores = 1 for per-case CPU-equivalent core-hours. All headline costs are cache-warm. The 1×1×1 matrix is re-run after model download so HuggingFace cache misses do not inflate costs.

\n

Execution steps

    \n
  1. Warm model cache: run a 2-element smoke test (Ca, Cu) for all three architectures.
    2. \n
  2. Run the 16-element 1×1×1 matrix for M3GNet, CHGNet, TensorNet, PBE + r2SCAN (96 cases) using lupine/data/run_mlip_elastic_benchmark_1x1x1_matrix.py.
    3. \n
  3. Aggregate with lupine-mlip-benchmark/scripts/aggregate.py, which combines the new 1×1×1 results, the existing 3×3×3 16-element grid, and targets_0K.json.
    4. \n
  4. Verify schema and plausibility with lupine-mlip-benchmark/scripts/verify.py.
    5. \n
  5. Fill placeholders in the preprint, dashboard, and funder brief using the resulting mlip_elastic_benchmark_results.json.
    6. \n
\n

Config

The machine-readable case matrix lives in lupine-mlip-benchmark/config/mlip_elastic_benchmark.yaml (192 cases: 16 elements × 3 architectures × 2 functionals × 2 supercells).

\n

Kill conditions / caveats

\n"}