spack-stack

An OS-independent HPC software stack on /shared, built the way EESSI builds theirs, with Spack in place of EasyBuild:

Layer	Path	What it is
Compatibility layer	/shared/stack/2026.09/compat/x86_64	Gentoo Prefix, bootstrapped in standalone mode with its own glibc and dynamic loader, plus GCC, binutils, Python, Perl and the rest of a base system. Or the same from Nix, in which case this path is a link into /nix/store.
Software layers	/shared/stack/2026.09/software/<target>	Spack 1.2.2 packages (MPI, CUDA, libraries), built only with the compatibility layer's GCC. One layer per CPU target, e.g. x86_64_v3 (AVX2) and x86_64_v4 (AVX-512).
Modules	/shared/stack/2026.09/modules/<target>	Lmod modules for each software layer.
User entry point	/shared/stack/current/init/bash	Picks the layer for this node's CPU and puts Lmod on your shell.

Every program in the stack runs on the compatibility layer's loader and glibc, not the host's. The stack therefore runs unchanged on Ubuntu, Rocky, Debian or any other x86_64 Linux that mounts /shared at the same path, whatever glibc it has. The only thing taken from the host is the NVIDIA driver (see GPUs). With the Nix layer, a machine also needs the layer in its own /nix/store, which one script puts there.

Design choices

• Compilers only from the compatibility layer. Spack treats the layer's GCC as external and gcc is not buildable. A GCC built by Spack would default to the host's /usr/include and /lib64/ld-linux-x86-64.so.2, which would tie everything it builds to the build host's OS. For more compiler versions, add GCC slots to COMPAT_PACKAGES (e.g. sys-devel/gcc:14), or with Nix add them to compilers in files/flake.nix.
• Build tools from the compatibility layer too. Spack registers the prefix's perl, make, autotools and so on as externals, searching prefix paths only, so no host tool leaks in.
• Builds run in a clean environment. After the bootstrap, every phase re-executes inside the prefix with only prefix directories on PATH (enter_prefix in files/common.sh).
• One software layer per CPU target, sharing one compatibility layer. TARGETS in files/stack.env lists them: x86_64_v3 (AVX2), x86_64_v4 (AVX-512), icelake and zen4. The init script picks the most specific layer the node can run, using Spack's archspec (see CPU targets).
• Versioned and immutable. Each stack lives in its own STACK_VERSION directory. The prefix path is compiled into every binary, so to change the compatibility layer you build a new version next to the old one and repoint /shared/stack/current.
• Builds on local disk. Portage and Spack build in the Job's /scratch (emptyDir). Only finished installs and download caches go to NFS.

Compatibility layer: Gentoo Prefix or Nix

COMPAT in files/stack.env picks what provides glibc, the loader, GCC and the base system. Spack, the software layers, the modules and the init script are the same with either.

	COMPAT=gentoo (default)	COMPAT=nix
Defined by	Bootstrap script and Gentoo tree commit, plus COMPAT_PACKAGES	files/flake.nix and files/flake.lock
Bootstrap	Several hours, compiled from source	About a minute, downloaded from nixpkgs' binary cache
Where it lives	/shared/stack/<version>/compat/x86_64	/nix/store on each machine (1.1 GB); compat/x86_64 is a link to it
What a machine needs	/shared	/shared, and init/install-compat.sh run as root
Loader and glibc are read from	NFS	Local disk
On /shared	The whole prefix	The flake, its lock and a 394 MB archive of the layer

The software layers are by far the larger part of the stack and are the same size with either, so Nix makes the stack quicker to start and to move, not much smaller.

Building a stack on Nix

The layer is compiled into every binary, so it is fixed for a stack version. Give the Nix stack its own version in files/stack.env:

COMPAT=nix
STACK_VERSION=2026.09-nix

Then build as usual (./run.sh all, or the phases one by one). The compat phase does nothing here, because the layer's packages are listed in flake.nix. The bootstrap installs Nix in its Job, realises the flake and leaves three things in /shared/stack/<version>/compat:

flake/	flake.nix and flake.lock as built
x86_64	Symlink to the layer, /nix/store/<hash>-stack-compat
x86_64.tar.gz	The layer's 139 store paths, for machines without Nix

Later phases start with an empty /nix, so they install the layer from the archive first, the same way a node does.

Putting the layer on a machine

Every machine that runs the stack needs the layer at the same path in /nix/store:

sudo /shared/stack/current/init/install-compat.sh

• Without Nix on the machine the script unpacks the archive into /nix/store. That needs no network and nothing installed.
• With Nix it realises the flake instead (nix build path:/shared/stack/<version>/compat/flake#compat) and registers the result as a garbage-collector root, so the store paths are known to Nix. That downloads from cache.nixos.org. The lock pins every input, so any machine and any Nix version arrives at the same store paths.
• It does nothing when the layer is already there, so it is safe to run before every job.
• Slurm workers run it from a prolog (40-stack-compat.sh in ../slurm/values.yaml). A worker pod's /nix is empty again after a restart, and the first job afterwards waits for the unpacking.
• The login pod has no prolog: run the script there once after each pod start. Until then, sourcing the init script fails with a message that names the command.

A machine without the layer can't run anything from the stack: programs fail with cannot execute: required file not found, because their loader is in /nix/store.

What is different on Nix

• The compiler wrappers add no flags of their own. nixpkgs' wrappers normally add hardening flags (-D_FORTIFY_SOURCE, -fstack-protector-strong, -fno-strict-overflow, -Werror=format-security, -z now) and frame pointers to every compile. flake.nix removes both; the wrappers still locate glibc and set the loader and RPATHs.
• Build tools are registered in Spack by name (NIX_EXTERNALS in files/phase-spack.sh). The layer is a bin directory with no lib or include beside it, so a library package such as xz or OpenSSL can't be used from it and Spack builds its own.
• Adding a tool or compiler to flake.nix is fine for an existing stack version: run the bootstrap and spack phases again, then install-compat.sh on each machine. Spack refers to the layer through the compat/x86_64 link, so nothing is rebuilt.
• Changing flake.lock moves glibc, which existing binaries point at. The bootstrap refuses it for an existing version; start a new STACK_VERSION.

Pinned sources and lockfiles

The Ubuntu image the builds start from doesn't affect the result; reproducibility comes from the compatibility layer and Spack. Both are pinned in files/stack.env, so a stack version can be rebuilt exactly:

Pin	Stack 2026.09	Source	What it fixes
BOOTSTRAP_COMMIT	da92e6ef (2026-09-07)	gentoo/prefix	Gentoo layer: the Gentoo Prefix bootstrap script
GENTOO_TREE_COMMIT	36485fbf (2026-09-27)	gentoo-mirror/gentoo	Gentoo layer: every package version in it
files/flake.lock	7c8764b7 (2026-10-08)	NixOS/nixpkgs, branch nixos-26.05	Nix layer: every package version in it (GCC 15.3.0, glibc 2.42, Python 3.13.15)
NIX_VERSION	2.35.2	releases.nixos.org	Nix layer: the Nix that realises the flake in the bootstrap
SPACK_VERSION	v1.2.2	spack/spack tag	Spack itself
SPACK_PACKAGES_COMMIT	d4f7c711 (2026-06-20)	spack/spack-packages, branch releases/v2026.06	Spack's package recipes

files/stack.env holds the full commit hashes; this table is for reference. A stack uses either the two Gentoo pins or the two Nix ones.

• Gentoo tree. Left alone, the bootstrap starts from the latest Gentoo tree and syncs it again midway. Instead, the bootstrap phase fetches the pinned commit and hands it to the script (TREE_FROM_SRC). It also writes etc/portage/repos.conf/gentoo.conf with an empty sync-type. Portage treats a repository with no sync type as sync-disabled: emerge --sync skips it and still succeeds, so the bootstrap's fallback to downloading the latest snapshot (emerge-webrsync) never runs. The tree therefore stays at the pinned commit during the bootstrap and every later compat run. The hand-off uses rsync, which the bootstrap installs in the build container along with the other host tools. The mirror used, gentoo-mirror/gentoo, is the one Gentoo's git sync uses; unlike gentoo/gentoo it includes the metadata cache.
• Spack packages. files/repos.yaml pins the builtin repository to the commit and clones it to /shared/stack/<version>/spack-packages. Spack's defaults also name the release branch; when both are set, Spack checks out the commit.
• Nix layer. flake.lock pins the nixpkgs commit and its hash. To move it for a new stack version, run nix flake update in files/.
• Record. The bootstrap writes the pins to /shared/stack/<version>/PINS.
• Lockfiles. Each target's Spack environment records its exact resolved versions and variants in spack.lock. Every successful build saves it to /shared/stack/locks/<version>/<target>.lock, and a build that has no lockfile yet starts from the saved one. That means a rebuild resolves to exactly the same versions, and only specs you've added or changed in spack.yaml are resolved afresh. Keep them in git as well, in locks/<version>/<target>.lock next to this README:
  • run-local.sh copies each lockfile there after a successful build, and passes locks/ to the build read-only.
  • run.sh build pulls them there when it finishes. ./run.sh locks pull and ./run.sh locks push copy them by hand, through a short-lived pod that mounts the shared volume. Push before rebuilding a stack from scratch.

Starting a new stack version: set a new STACK_VERSION, update the pins (the commands to look up the latest commits are in files/stack.env), and build. Never change the pins of an existing stack version, because its binaries were built from them.

Exact reproduction also depends on the upstream sources still being downloadable. The bootstrap caches Gentoo's downloads in /shared/stack/cache/distfiles and Spack caches its sources in /shared/stack/cache/spack-sources, so keep those if you may need to rebuild years later. The Nix layer needs cache.nixos.org to rebuild, but not to be used: the archive in compat/ holds the layer itself.

Prerequisites

• The Slurm deployment in ../slurm, specifically its slurm-shared NFS volume in namespace slurm. The Job mounts it at /shared, the same path the Slurm pods use.
• The NFS export has no_root_squash, so the Job can give /shared/stack to the build user (UID 1000). Nothing is built as root: Gentoo Prefix refuses to run as root.
• With COMPAT=nix: root on each machine that uses the stack, to put the layer in /nix/store. Nix itself is only needed in the bootstrap Job, which installs it.
• Outbound internet from the build pod (Gentoo mirrors, GitHub, Spack mirrors).
• A node with 16 CPUs and 32Gi free for the build pod, plus up to 200Gi of local disk for /scratch. Size it with CPUS= and MEMORY=.

Build

./run.sh bootstrap   # compatibility layer, once per stack version (Gentoo Prefix: several hours; Nix: about a minute)
./run.sh compat      # git, curl, unzip, ... and any extra GCC slots (Gentoo Prefix only)
./run.sh spack       # Spack, concretizer, compiler and build-tool registration
./run.sh build       # a software layer per target + Lmod modules (hours per target the first time)

Or ./run.sh all. ./run.sh build x86_64_v4 builds just one target. Each phase is a Kubernetes Job in namespace slurm, and run.sh follows its log. If your terminal disconnects, the Job carries on: reattach with kubectl -n slurm logs -f job/<name>, and find the name with kubectl -n slurm get jobs. A failed phase can simply be re-run: the bootstrap resumes, and Spack skips what's installed.

Building on one machine instead (run-local.sh)

run-local.sh runs the same phases in a Docker or Podman container on the current machine, with no Kubernetes, so a spare box can grind through the builds unattended:

export SHARED_DIR=/shared          # host dir that becomes /shared in the container
export CPUS=32                     # build jobs (default: all CPUs)
./run-local.sh --detach all        # bootstrap, compat, spack, then every target in TARGETS
./run-local.sh logs                # follow progress (Ctrl-C stops following, not the build)
./run-local.sh stop                # stop the background run

• all skips targets this CPU can't build. A Zen 4 or Zen 5 box builds x86_64_v3, x86_64_v4 and zen4, and skips icelake with a message rather than failing the run. Build icelake separately on an Ice Lake-class machine, e.g. an i7-11700K, with ./run-local.sh build icelake.
• The stack path is fixed. It is always written to /shared/stack inside the container, because that path is compiled into every binary. On the host it goes to $SHARED_DIR/stack, so either:

  • mount the cluster's NFS export at SHARED_DIR and build straight onto it, or

  • build into a local directory and copy it over afterwards. The paths match, since the cluster also sees it as /shared/stack. Run this from a machine that mounts the NFS export at /shared:

    rsync -aH --numeric-ids --info=progress2 <build-box>:<SHARED_DIR>/stack/ /shared/stack/
    

• Local disk: SCRATCH_DIR (default /var/tmp/spack-stack-scratch) is where Portage and Spack build, and needs up to about 200 GB. Logs go to logs/ next to the script.
• Use Docker or rootful Podman. The build runs as UID 1000 inside the container, and the stack's files are owned by UID 1000 on the host. Rootless Podman would map that to a different host UID.
• Each phase resumes or skips finished work when re-run, so after a failure fix the cause and run the same command again.
• The build phase exits with code 3 when the CPU can't build the target; that's how run-local.sh tells "wrong CPU" (skip) apart from a real build failure (stop).

Suggested build order for the current cluster

As of September 2026 every node in the cluster is x86_64_v3, so:

• Only the x86_64_v3 layer is used by the cluster for now. The other layers are used by the Zen 5 build box and the i7-11700K if they mount /shared, and by AVX-512 nodes added later.

• run.sh can't build the AVX-512 layers yet. Their Jobs would never find a matching node. Build them with run-local.sh on the Zen 5 box (x86_64_v4, zen4) and the 11700K (icelake).

• Build x86_64_v3 first, so the cluster can start using the stack sooner. On the Zen 5 box:

  export SHARED_DIR=/shared CPUS=32
  ./run-local.sh bootstrap && ./run-local.sh compat && ./run-local.sh spack
  ./run-local.sh --detach build x86_64_v3
  # once x86_64_v3 works on the cluster:
  ./run-local.sh --detach build x86_64_v4 zen4
  

  On the 11700K, against the same SHARED_DIR: ./run-local.sh --detach build icelake. Alternatively, leave TARGETS as it is and let ./run-local.sh --detach all take the extra time, if the box has nothing else to do.

Where builds run

The bootstrap, compat and spack phases run on any node. Each target's build has to run on a CPU that can execute that target's instructions, because many builds run what they just compiled (e.g. HDF5 and Python). A x86_64_v4 build on a CPU without AVX-512 would crash.

• run.sh pins each build to matching nodes with the CPU-feature labels that node-feature-discovery adds (it runs as part of the GPU Operator), e.g. feature.node.kubernetes.io/cpu-cpuid.AVX512F=true for x86_64_v4, plus the CPU vendor for CPU-specific targets such as zen4.
• The build also checks the CPU itself before starting and stops with a clear message if it can't run the target.
• If no node matches, the pod stays Pending and run.sh gives up after 15 minutes. Point a build at a specific node with NODE_SELECTOR=kubernetes.io/hostname=<node>.
• If your only AVX-512 nodes are dedicated Slurm nodes, add TOLERATE_SLURM=1 so the build may run on them. It takes a whole node's worth of CPU while it runs.

./run.sh --print-job build x86_64_v4 shows the Job without creating it.

What gets built

Defined in files/spack.yaml:

Group	Packages
Core	lmod, cmake, ninja, python@3.13, openblas, hdf5
MPI	openmpi@5 with Slurm support and UCX
MPI libraries	hdf5+mpi, fftw+mpi, osu-micro-benchmarks
GPU	cuda@12, nccl, ucx+cuda, for GPU architectures 80,86,89,90 (CUDA_ARCH in files/stack.env)

The same list is built for every target. To add software, edit files/spack.yaml and run ./run.sh build again.

CPU targets

Target	Instructions	CPUs that use it
x86_64_v3 (default)	AVX2, FMA	Haswell and newer, Zen 1 to Zen 3
x86_64_v4 (default)	AVX-512 F/BW/CD/DQ/VL	Skylake-X, Cascade Lake, Ice Lake, Sapphire Rapids, Zen 4, Zen 5
icelake (default)	+ VNNI, VBMI2, BITALG, GFNI, VAES	Ice Lake and Sapphire Rapids
sapphirerapids	+ AMX, AVX512-BF16/FP16	Sapphire Rapids and newer
zen4 (default)	+ AVX512-BF16, VNNI, VBMI2; Zen tuning	Zen 4 and Zen 5
zen5	+ VP2INTERSECT; Zen 5 tuning (needs GCC 14+)	Zen 5

Four targets are built by default. x86_64_v3 and x86_64_v4 cover every CPU; icelake and zen4 add newer instructions (VNNI and BF16 help CPU inference) and tuning for those CPUs. Each target is a full rebuild of the stack, so the four take about four times as long as one. To add or remove one, edit TARGETS in files/stack.env and run ./run.sh build <target>.

Selection picks the most specific target a CPU can run. A CPU can run a target when it is from the target's vendor (or the target is generic, like x86_64_v4) and has every feature the target requires. This is archspec's own compatibility rule, applied to the CPU's actual features. With the defaults: Haswell, Zen 3 and Alder Lake get x86_64_v3; Skylake-X and Cascade Lake get x86_64_v4; Ice Lake, Sapphire Rapids and Rocket Lake get icelake; Zen 4 and Zen 5 get zen4. Keep x86_64_v3 in the list so every node has a layer.

Which machines can build which targets

GCC can compile AVX-512 code on any x86_64 machine; the problem is running it. Many builds run what they have just compiled: configure test programs, HDF5's helper programs that generate headers, Python finishing its own build with the new interpreter, test suites. Built for AVX-512 and run on a CPU without it, those programs crash with "illegal instruction". So a target has to build on a CPU that can run it. The build checks this with the same script the init script uses (select-target.py --check), so the two always agree:

Build node	Can build
Any AVX2 machine	x86_64_v3
Skylake-X / Cascade Lake	x86_64_v3, x86_64_v4
Ice Lake / Sapphire Rapids / Rocket Lake	x86_64_v3, x86_64_v4, icelake
Zen 4 / Zen 5	x86_64_v3, x86_64_v4, zen4

One Ice Lake-class node and one Zen 4 node are enough for all four default targets. zen4 needs an AMD Zen 4 or Zen 5: archspec doesn't treat Intel CPUs as able to run zen4 code, even though Sapphire Rapids has most of the same instructions.

The bootstrap, compat and spack phases don't need AVX-512. The compatibility layer is built with plain -O2, not tuned for the build machine, so it runs on every node wherever it was built.

If you have no machine for a target, remove it from TARGETS, and its CPUs use the next-best layer. Emulating AVX-512 (Intel SDE, or QEMU with an Ice Lake CPU model) works but makes builds many times slower, which isn't worth it for a whole stack.

Desktop Intel CPUs

• 11th-gen Core (Rocket Lake, e.g. i7-11700K) uses icelake, through one documented exception. Its Cypress Cove cores are Ice Lake's Sunny Cove design backported to 14nm, but without CLWB (a cache-flush instruction for persistent memory). archspec's icelake entry requires CLWB. Spack, however, builds icelake with GCC's -march=icelake-client, and GCC's icelake-client instruction set has no CLWB: only icelake-server and tigerlake add it, and GCC defines its own rocketlake as icelake-client minus SGX. icelake code therefore never contains CLWB, so select-target.py doesn't require it for that target (NOT_EMITTED). Without the exception, archspec would call the chip cannonlake, which its family tree doesn't even connect to x86_64_v4, and the node would fall back to x86_64_v3.
• 12th- to 14th-gen Core (Alder Lake, Raptor Lake) have AVX-512 disabled and only get x86_64_v3. The 11th gen is the last consumer Intel generation where AVX-512 works.

Once the stack exists, the definitive check on any machine is the script itself. It prints nothing and exits 0 if the CPU qualifies, and lists what's missing otherwise:

EPREFIX=/shared/stack/current/compat/x86_64
$EPREFIX/usr/bin/python3 /shared/stack/current/init/select-target.py \
  /shared/stack/current/spack --check icelake

Before the stack exists, this rough check prints every icelake feature a CPU lacks (CLWB isn't listed, since icelake code doesn't use it):

for f in avx512f avx512bw avx512cd avx512dq avx512vl avx512_vnni avx512ifma avx512vbmi \
         avx512_vbmi2 avx512_bitalg avx512_vpopcntdq gfni vaes vpclmulqdq sha_ni rdpid; do
  grep -qw "$f" /proc/cpuinfo || echo "missing: $f"
done

spack arch --target isn't a useful check here: it reports archspec's own detection, which is cannonlake for Rocket Lake.

Use

source /shared/stack/current/init/bash
module avail
module load openmpi hdf5

From a Slurm job:

#!/bin/bash
#SBATCH -p cpu -N 2 --ntasks-per-node=4
source /shared/stack/current/init/bash
module load openmpi osu-micro-benchmarks
srun osu_latency

The init script picks the layer for the CPU it runs on; echo $STACK_TARGET shows which. Set STACK_TARGET=x86_64_v3 before sourcing to force one, for example to test that code runs on older nodes. If a job inherits modules from a node with a different CPU, such as the login node, the init script purges them and starts again for the new target.

Multi-node jobs across different CPUs: only the batch script's node sources the init script, and srun tasks inherit its environment. Tasks on a node with an older CPU would run AVX-512 code and crash. Keep each job to one CPU type, with a partition or --constraint per type, or have each task pick its own layer:

srun bash -c 'source /shared/stack/current/init/bash && module load openmpi && exec ./my_app'

Sourcing the init script changes nothing else about your shell. The compatibility layer isn't put on PATH, so host tools stay the host's.

GPUs

The NVIDIA driver library (libcuda.so) must match the host's kernel module, so it can't be part of the stack. The compatibility layer's loader doesn't search the host's library directories, so init/link-host-libs.sh links the host's driver libraries into /opt/stack-host/nvidia, and the init script adds that directory to LD_LIBRARY_PATH.

On the Slurm cluster this is automatic: ../slurm/values.yaml has a prolog (50-stack-host-libs.sh) that runs the script as root before every job. Anywhere else (a workstation, JupyterHub), run it once as root:

sudo /shared/stack/current/init/link-host-libs.sh

Using it outside the Slurm cluster

Mount the same NFS export at /shared (read-only is fine), then source the init script. Nothing needs installing on the host. The mount path has to be exactly /shared, because the stack's paths are compiled into its binaries. A stack on the Nix layer needs one more step, once per machine: sudo /shared/stack/current/init/install-compat.sh.

Files

File	Purpose
run-local.sh	Runs the phases in a local Docker/Podman container instead, optionally in the background
run.sh	Generates and runs a Job per phase and target (Ubuntu 26.04 pod, /shared, /scratch), and follows logs
files/stack.env	Versions, paths, CPU targets, CUDA architectures, extra compat packages
files/entrypoint.sh	Root setup in the pod, then drops to the build user
files/common.sh	Shared helpers: paths, enter_prefix, template rendering
files/phase-*.sh	The four phases; phase-bootstrap-nix.sh is the bootstrap for COMPAT=nix
files/flake.nix, files/flake.lock	The Nix compatibility layer: its packages and compilers, and the nixpkgs pin
files/install-compat.sh	Puts the Nix layer in a machine's /nix/store
files/{config,packages,modules,bootstrap,repos}.yaml	Spack site configuration (repos.yaml pins Spack's package repository)
files/spack.yaml	The stack definition (Spack environment)
files/init.bash	User init script template
locks/<version>/<target>.lock	Saved Spack lockfiles for exact rebuilds (commit these)
files/select-target.py	Picks the best built target for the current CPU
files/link-host-libs.sh	Links host NVIDIA driver libraries

What to expect

• The Gentoo Prefix bootstrap takes several hours but only happens once per stack version. Later phases reuse the compatibility layer.
• Each target is a full build of the stack. The four default targets take about four times as long as one. Build the one your nodes need first, e.g. ./run.sh build zen4.
• On Gentoo Prefix, the first ./run.sh build may need a few edits to files/spack.yaml. The stack definition concretizes on the Nix layer (see below), but there Spack registers a short list of build tools. On Gentoo Prefix it registers everything it finds, and spack.yaml requires the target (e.g. target=x86_64_v4) on every package, which can clash with one of those externals. The concretizer error will name the package. The fix is in the packages: section of files/spack.yaml, e.g. turning the all requirement into a preference (prefer: instead of require:) or exempting the externals.
• Check CUDA_ARCH in files/stack.env against your actual GPUs. Each architecture adds compile time to CUDA packages, so listing only the ones you have makes GPU builds much shorter.
• The Slurm prolog always exits 0. Slurm drains a node whose prolog fails, so a problem linking driver libraries must never take a node out of service. If GPU programs from the stack can't find libcuda.so, check /opt/stack-host/nvidia on that node.

What has been tested

• The pinned bootstrap script URL downloads, and the pinned Gentoo tree commit can be fetched shallowly from gentoo-mirror/gentoo.

• files/repos.yaml validates against Spack 1.2.2's schema. Spack's checkout code uses commit before the default branch.

• In Portage's source, an empty sync-type makes emerge --sync skip the repository and succeed.

• The lockfile round trip in run-local.sh was tested with the stand-in engine. run.sh locks hasn't run against a cluster.

• All scripts pass bash -n and shellcheck.

• All templates render with no placeholders left, and the rendered YAML parses.

• The four Spack config files and spack.yaml validate against Spack 1.2.2's own schema.

• The Spack command-line flags the scripts use were checked against Spack 1.2.2.

• The Slurm prolog renders into the Slurm chart.

• select-target.py was run against Spack 1.2.2's archspec, with simulated CPUs including an i7-11700K without CLWB (uses and can build icelake), Alder Lake without AVX-512 (x86_64_v3) and Zen 5 (zen4). It picks the right target for Haswell, Skylake-X, Cascade Lake, Ice Lake, Sapphire Rapids and Zen 3/4/5, and takes about 40ms.

• ./run.sh --print-job output parses for every phase, with the expected node labels per target.

• run-local.sh's control flow was tested with a stand-in container engine: all runs every phase, skips a target the CPU can't build, and --detach logs to logs/. It hasn't run a real container yet.

• Each target's environment overrides the site config as intended (module root per target).

The Nix layer

The phase scripts were run with COMPAT=nix on one machine (Debian 12, Skylake), as an ordinary user with proot standing in for the container and for root, and local directories for /shared, /scratch and /nix:

• Bootstrap: installs Nix 2.35.2, realises the flake and writes the archive in 45 seconds: 139 store paths, 1.1 GB unpacked, 394 MB archived. Nix 2.20 resolves the same flake to the same store path. A second run with a kept /scratch and a new /nix, as run-local.sh gives it, also passes.
• install-compat.sh: unpacks the layer into an empty /nix in 6 seconds from local disk, and does nothing on a second run. After a change to flake.nix it adds the new layer next to the old one. On a store managed by Nix it realises the flake and registers the root.
• Compilers: C, C++ and Fortran programs get the loader and RPATHs in /nix/store, the wrappers add no flags, and gcc -lgfortran and gfortran -lstdc++ link.
• Spack phase: registers one gcc@15.3.0 with C, C++ and Fortran and 13 build tools, all through the compat/x86_64 link.
• Concretization: the full files/spack.yaml concretizes for x86_64_v3. This found that cuda %gcc is rejected, so cuda is now outside the compiler matrices.
• Build phase: run with a stack definition cut down to lmod and zlib-ng (27 packages built, among them OpenSSL, curl, Tcl and Lua), then OpenBLAS with OpenMP. Lmod modules are generated, and the init script selects x86_64_v3 and lists them. On a machine without the layer it stops with the message that names install-compat.sh.

Not tested for the Nix layer: a real container and real root, /shared on NFS, the Slurm prolog, and the rest of the stack (Open MPI, HDF5, the CUDA packages).

Not run yet

Nothing has run on the cluster yet. The Gentoo bootstrap and a build of the full stack on either layer are untested. Expect to adjust spack.yaml after the first ./run.sh build: a spec that fails to build is normal on a first pass.