{"id":"layer2-supercell-evaluation","title":"Layer 2 Supercell Scaling Evaluation","subtitle":"Technical memo: elastic constants are converged at the 1×1×1 conventional cell; 3×3×3 adds runtime without accuracy gain.","category":"validation","tags":["mlip","matpes","supercell","convergence","projection-law"],"source":"articles/docs/layer2_supercell_evaluation.md","lang":"en","words":2165,"readMinutes":10,"toc":[{"depth":2,"text":"TL;DR","id":"tl-dr"},{"depth":2,"text":"1. Convergence of elastic constants with supercell size","id":"1-convergence-of-elastic-constants-with-supercell-size"},{"depth":3,"text":"1.1 Aggregate picture","id":"1-1-aggregate-picture"},{"depth":3,"text":"1.2 Case-by-case |ΔCᵢⱼ| between sizes (matched cases, all three constants)","id":"1-2-case-by-case-c-between-sizes-matched-cases-all-three-constants"},{"depth":3,"text":"1.3 Lattice parameter is invariant to size","id":"1-3-lattice-parameter-is-invariant-to-size"},{"depth":3,"text":"1.4 Conclusion of §1","id":"1-4-conclusion-of-1"},{"depth":2,"text":"2. Runtime / cost trade-offs","id":"2-runtime-cost-trade-offs"},{"depth":3,"text":"2.1 Per-model mean runtime (seconds/case)","id":"2-1-per-model-mean-runtime-seconds-case"},{"depth":3,"text":"2.2 The 1×1×1 total-runtime anomaly","id":"2-2-the-1-1-1-total-runtime-anomaly"},{"depth":3,"text":"2.3 CHGNet is the runtime outlier at 4×4×4","id":"2-3-chgnet-is-the-runtime-outlier-at-4-4-4"},{"depth":3,"text":"2.4 Cost recommendation","id":"2-4-cost-recommendation"},{"depth":2,"text":"3. Best-performing model / functional combinations","id":"3-best-performing-model-functional-combinations"},{"depth":3,"text":"3.1 MAE by model × functional (averaged over Cu+Ni, all available supercells)","id":"3-1-mae-by-model-functional-averaged-over-cu-ni-all-available-supercells"},{"depth":3,"text":"3.2 Findings","id":"3-2-findings"},{"depth":3,"text":"3.3 The QET ≡ TensorNet alias (data-integrity note)","id":"3-3-the-qet-tensornet-alias-data-integrity-note"},{"depth":2,"text":"4. Caveats and data-quality issues","id":"4-caveats-and-data-quality-issues"},{"depth":2,"text":"5. Recommendations","id":"5-recommendations"},{"depth":2,"text":"Appendix — per-case MAE (GPa) / runtime (s) across supercells","id":"appendix-per-case-mae-gpa-runtime-s-across-supercells"}],"html":"

Layer 2 Supercell Scaling — Technical Evaluation

\n

Task: t_22ae2ad9 · Author: synthesizer · Date: 2026-06-26

\n

Sources: data/benchmark_layer2_results.json (schema lupine.benchmark.layer2.v1, 16 cases, 1×1×1),\ndata/supercell_scaling_comparison.json (1×1×1 + 3×3×3, 32 rows),\ndata/layer2_outputs_3x3x3/ (16 per-case JSON + _summary.json),\ndata/layer2_outputs_4x4x4/ (15 per-case JSON + _summary.json + run.log),\ndata/targets_0K.json (schema lupine.targets_0K.v3, reference tensors).\nMethod driver: data/layer2_benchmark_task.py; grid runner: data/run_layer2_supercell_grid.py;\ncomparator: data/compare_supercell_scaling.py.

\n
\n
\n

TL;DR

Supercell size is not the limiting factor for MatPES-potential elastic constants of bulk FCC metals.\nThe mean absolute error (MAE) on the three independent cubic constants (C₁₁, C₁₂, C₄₄) is flat across cell\nsizes — 13.26 → 13.29 → 13.41 GPa for 1×1×1 (4 atoms) → 3×3×3 (108) → 4×4×4 (256) — while the per-atom\nruntime rises steeply. The residual ~13 GPa gap to DFT is model-form error (weights + training pool),\nnot finite-size error, and is dominated by the functional-mismatch and a few stiff/target-shift cases.\nThe cost-optimal cell is the 3×3×3 supercell: it is faster than the 1×1×1 run in aggregate (cache\nwarm-up artifact), matches it in accuracy to <0.05 GPa, and avoids the 4×4×4 blow-up. The 1×1×1\nconventional cell remains valid for screening. Going to 4×4×4 buys nothing in accuracy and ~4× the runtime.

\n
\n

1. Convergence of elastic constants with supercell size

1.1 Aggregate picture

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
SupercellAtoms/cellCasesMean MAE on Cᵢⱼ (GPa)Total runtime (s)Mean runtime/case (s)
1×1×141613.261217.7676.11
3×3×31081613.29519.5032.47
4×4×425615 †13.412161.91144.13

† The 4×4×4 grid is missing Cu_CHGNet_PBE (15/16); the runner logged the case as started but no output\nfile and no error were produced — a silent failure (see §4). All three MAE figures use only the cases\npresent at each size; the 4×4×4 figure is therefore computed over 15 cases but is within rounding of the\n16-case number (the missing case contributes ~12.8 GPa at SC1/SC3, near the mean, so its absence is benign).

\n

A 64× increase in atom count (4 → 256) moves the ensemble MAE by +0.15 GPa — less than the rounding on\nany single constant. This is the signature of an accuracy floor set by the model, not the box.

\n

1.2 Case-by-case |ΔCᵢⱼ| between sizes (matched cases, all three constants)

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
TransitionMatched casesmax |ΔCᵢⱼ| (GPa)mean |ΔCᵢⱼ| (GPa)RMS |ΔCᵢⱼ| (GPa)
1 → 3160.680.2490.304
3 → 4153.310.2640.605

Interpretation:

\n\n

1.3 Lattice parameter is invariant to size

Predicted equilibrium lattice parameters a are stable to ±0.0008 Å across all three supercell sizes\nfor every case. The relaxer (RelaxCalc, fmax=0.005) converges to the same bulk geometry regardless of\ncell padding, which is why the elastic constants are so nearly size-independent: the elastic fit is taken\nat the same minimum in every case.

\n

1.4 Conclusion of §1

Elastic constants are converged at the conventional cell for this system class. The remaining ~13 GPa\nMAE cannot be closed by supercell expansion. It is model-form error and concentrates in specific\nmodel/functional combinations (§3).

\n
\n

2. Runtime / cost trade-offs

2.1 Per-model mean runtime (seconds/case)

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Model1×1×13×3×34×4×4Scaling 1→4
M3GNet39.29.722.90.6×
CHGNet252.871.1489.81.9×
TensorNet9.724.074.77.7×
QET ‡2.725.175.528×

‡ QET and TensorNet resolve to the same TensorNet-MatPES checkpoint (see §3.3); the runtime\ndifferences between them at a given size reflect machine load / one-off cache states, not architecture.

\n

2.2 The 1×1×1 total-runtime anomaly

The 1×1×1 grid (1217.76 s) is slower in aggregate than the 3×3×3 grid (519.50 s), which is physically\nimpossible for a pure size-scaling test. This is a cache-warm / model-download confound: the 1×1×1 runs\nwere the first invocation and paid the one-time HuggingFace download + TorchScript-trace cost for each\ncheckpoint; the 3×3×3 grid reused the cached models. The per-case steady-state cost is therefore better\nread from the 3×3×3 and 4×4×4 columns. The CHGNet column still shows the genuine size trend (71 → 490 s,\nroughly linear-ish in atoms), because CHGNet is the heaviest per-atom architecture.

\n

2.3 CHGNet is the runtime outlier at 4×4×4

Two CHGNet cases at 4×4×4 dominate the wall-clock:

\n\n

This is super-linear scaling in the relaxation loop (CHGNet's iterative ionic relaxation at 256 atoms with\nfmax=0.005 does many more force evaluations than at 108). For production grids, CHGNet at 4×4×4 should be\ntreated as ~10–20× the 3×3×3 cost, and budgeted accordingly.

\n

2.4 Cost recommendation

\n
\n

3. Best-performing model / functional combinations

3.1 MAE by model × functional (averaged over Cu+Ni, all available supercells)

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ModelFunctionalSC1 MAESC3 MAESC4 MAEMean (all)
M3GNetPBE8.508.369.168.67
M3GNetr2SCAN11.1411.0711.1411.12
TensorNetPBE9.449.709.759.63
QET ‡PBE9.449.709.759.63
TensorNetr2SCAN15.5915.5615.5215.56
QET ‡r2SCAN15.5915.5615.5215.56
CHGNetPBE14.8414.7216.6515.40
CHGNetr2SCAN21.5821.6421.4021.54 (worst)

‡ QET and TensorNet share the same checkpoint and return byte-identical Cᵢⱼ in all 12 matched\n(case × supercell) pairs — verified directly. They are not independent models in this release (§3.3).

\n

3.2 Findings

    \n
  1. M3GNet/PBE is the clear winner at 8.67 GPa mean, and is the only combination that stays below\n10 GPa at every supercell size. Its best single case is Ni/PBE at 3.43 GPa (SC3).
    2. \n
  2. PBE-trained models beat r2SCAN-trained models for elasticity, across all architectures (8.67 vs\n11.12 for M3GNet; 9.63 vs 15.56 for TensorNet/QET; 15.40 vs 21.54 for CHGNet). This is\ncounterintuitive — r2SCAN is the more accurate functional — and most plausibly reflects the relative\nmaturity / coverage of the two MatPES training pools rather than a ceiling of the architectures.
    3. \n
  3. CHGNet/r2SCAN is the worst combination (21.54 GPa mean) and is the dominant contributor to the\nensemble MAE. Its bulk-modulus error on Ni is +11.2 % (B_pred 251.76 vs B_tgt 226.40 GPa) — the\nlargest single relative error in the grid.
    4. \n
  4. Errors scale with the magnitude of the target. Nickel (C₁₁ ≈ 276 GPa PBE / 315 GPa r2SCAN-shifted)\nand the r2SCAN bulk-shifted targets (which are uniformly stiffer) concentrate the largest absolute\nerrors. A 21 GPa miss on a ~315 GPa C₁₁ is ~7 % relative — the same architectures hit 3–5 % on copper.
    5. \n
\n

3.3 The QET ≡ TensorNet alias (data-integrity note)

In the MatPES 2025.2 release the QET and TensorNet labels in MODEL_MAP (data/layer2_benchmark_task.py:30)\nboth resolve to a TensorNet-MatPES-* checkpoint:

\n
("QET", "PBE"):       "TensorNet-MatPES-PBE-2025.2"\n("TensorNet", "PBE"): "TensorNet-PES-MatPES-PBE-2025.2"\n
\n

The resulting Cᵢⱼ are identical to floating-point precision in every matched case (e.g. Cu/PBE:\nc11=162.35, c12=118.18, c44=71.77 at 4×4×4 for both "QET" and "TensorNet"). The 16-case grid therefore\ncontains only 3 distinct architectures (M3GNet, CHGNet, TensorNet), with TensorNet counted twice.\nAny ensemble statistic that treats QET and TensorNet as independent models double-weights TensorNet.\nThe MAE tables above average per (model,label), so the alias does not inflate the headline numbers, but\nthe effective N for "model-form error" is 3, not 4.

\n
\n

4. Caveats and data-quality issues

    \n
  1. Missing 4×4×4 case. data/layer2_outputs_4x4x4/Cu_CHGNet_PBE.json does not exist. The runner\n(run.log) shows === Running Cu CHGNet PBE 4x4x4 === followed immediately by the next case with no\nerror/traceback — a silent process exit (likely OOM-kill or a torch crash inside the relaxer that the\ntry/except in layer2_benchmark_task.py did not surface because no output file was written). The\n_summary.json reports 15 cases and the runner exited 0, so the loss went unflagged. The\nsupercell_scaling_comparison.json predates the 4×4×4 run and therefore omits this size entirely;\nthe 4×4×4 numbers in §1.1–§1.2 were recomputed here from the raw per-case JSON.
    2. \n
  2. QET ≡ TensorNet alias. See §3.3. The "four model families" framing over-counts TensorNet.
    3. \n
  3. r2SCAN targets are synthesized, not measured. Per targets_0K.json (schema v3), the r2SCAN\nreference tensors are constructed by scaling the de Jong 2015 PBE tensors by a bulk-modulus ratio\nB_r2SCAN/B_PBE from Liu et al. 2024 (e.g. Cu ratio 1.1952, Ni ratio 1.1430). This is an\napproximation — it assumes the shear constants scale identically to the bulk modulus, which is not\ngenerally true. Some of the r2SCAN MAE is therefore target-construction error, not pure model error.\nFor elements with no published r2SCAN bulk modulus (Al, Ca, Sr in the full 15-metal table), the\nunshifted PBE tensor is used as the r2SCAN target (r2scan_shift_ratio = 1.0), which makes the\nr2SCAN/PBE comparison meaningless for those elements. Cu and Ni (the two tested here) both do have\nshifts, so this caveat does not affect the present grid but limits generalization.
    4. \n
  4. Two-element scope. Only Cu and Ni (both FCC) are tested. The convergence-on-conventional-cell\nconclusion is expected to hold for other clean bulk FCC/BCC metals but is not demonstrated for\nlow-symmetry, defect-laden, or anisotropic systems, where supercell effects can be genuine.
    5. \n
  5. Target provenance is mixed. Cu/Ni PBE tensors are from de Jong 2015 (VASP, stress-strain). The\nwider 15-metal table mixes sources (Ag: Pandit & Bongiorno 2023; Au: Wang & Li 2008 PW91, used because\nthe MP PBE tensor is unphysical). Cross-source comparisons at ~1–3 GPa level require care.
    6. \n
  6. Single random seed / single relax. Each case is one relax + one elastic fit; there is no estimate\nof run-to-run variance. The 3×3→4×4 Ni/M3GNet/PBE outlier (§1.2) shows this variance is non-zero at\nthe larger cell.
    7. \n
\n
\n

5. Recommendations

    \n
  1. Adopt 3×3×3 as the default bulk-elasticity cell for the Layer 2 MatPES benchmark, with 1×1×1\nretained as the fast-screening option when the model cache is warm. Retire 4×4×4 from the bulk-FCC\npipeline; it is not cost-justified.
    2. \n
  2. Re-run the missing Cu_CHGNet_PBE at 4×4×4 (or formally drop 4×4×4 per recommendation 1) so the\nlayer2_outputs_4x4x4/_summary.json is either complete or explicitly marked partial. The silent\nfailure should also be hardened: run_layer2_supercell_grid.py:44 should treat a missing output file\nas a failure even when the subprocess returns 0.
    3. \n
  3. Regenerate supercell_scaling_comparison.json with compare_supercell_scaling.py extended to all\nthree sizes so the published comparison artifact matches the raw data on disk.
    4. \n
  4. Resolve the QET/TensorNet alias in the model roster before publishing model-vs-model rankings:\neither drop the QET label or replace it with a genuinely distinct checkpoint. Until then, report\n"3 architectures" and de-duplicate when ensembling.
    5. \n
  5. Investigate the PBE-better-than-r2SCAN result. It likely indicates the r2SCAN MatPES training pool\nis thinner for elastic response than the PBE pool. A targeted stress/strain fine-tune on r2SCAN\nreference data (de Jong 2015 + the Liu 2024 bulk shifts) is the highest-leverage accuracy improvement\navailable — it would attack the dominant error source, whereas supercell scaling does not.
    6. \n
  6. Expand scope before generalizing. If the "conventional cell is sufficient" claim is to be made\nbroadly, add 2–3 BCC metals (e.g. W, Mo) and one lower-symmetry system to the convergence study. The\npresent evidence is FCC-only.
    7. \n
\n
\n

Appendix — per-case MAE (GPa) / runtime (s) across supercells

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ElementModelFunc1×1×13×3×34×4×4
CuCHGNetPBE12.84/308.412.79/75.4— (missing)
CuCHGNetr2SCAN23.15/58.423.05/58.623.06/318.3
CuM3GNetPBE13.37/19.213.30/10.613.31/20.6
CuM3GNetr2SCAN12.08/90.612.14/10.812.21/24.8
CuQETPBE9.72/2.99.73/23.69.81/62.2
CuQETr2SCAN12.56/3.112.65/24.812.60/64.5
CuTensorNetPBE9.72/2.79.73/23.49.81/62.8
CuTensorNetr2SCAN12.56/2.812.65/22.412.60/63.1
NiCHGNetPBE16.84/310.216.64/61.716.65/148.0
NiCHGNetr2SCAN20.00/334.220.22/88.719.74/1003.2
NiM3GNetPBE3.63/1.83.43/8.55.01/20.6
NiM3GNetr2SCAN10.19/45.39.99/9.110.08/25.4
NiQETPBE9.16/2.49.68/22.39.69/66.7
NiQETr2SCAN18.62/2.618.46/29.618.43/108.7
NiTensorNetPBE9.16/30.59.68/21.79.69/63.1
NiTensorNetr2SCAN18.62/2.718.46/28.318.43/109.9

MAE is mean absolute error over {C₁₁, C₁₂, C₄₄} vs the targets_0K.json reference tensor; runtime is\nwall-clock for relax + elastic fit, in seconds. Source: recomputed from per-case JSON in\ndata/layer2_outputs{,_3x3x3,_4x4x4}/ with data/targets_0K.json targets.

\n"}