Troubleshooting

Run stack.ps1 doctor (or doctor -Target all) first; most problems below show up there as a [FAIL] line.

Hosts, SSH, and stack.ps1

Symptom	Cause and fix
pwsh: command not found on a target, or a remote command hangs	PowerShell 7 is missing. Ubuntu: re-run bootstrap-ubuntu.sh. Windows: install PowerShell 7 and set OpenSSH DefaultShell to pwsh (Windows OpenSSH (iis)).
permission denied while trying to connect to the Docker daemon socket	The deploy user was added to the docker group but has not logged in again. Log out and back in (ssh sessions too), or newgrp docker.
SSH asks for a password during -Target	The key is not in authorized_keys (Windows admins: C:\ProgramData\ssh\administrators_authorized_keys). stack.ps1 cannot answer prompts.
-Sync: “branch is not pushed (HEAD != upstream)”	Push the branch (and submodule commits) first; -Sync deploys the remote, not your tree.
Unknown stack 'viking-...'	VIKING_REPO_ROOT_HOST is not set in that host’s host.env, so the Viking registry is not loaded.
up fails with port is already allocated / address already in use on the dev box	The old viking compose project is still running. docker compose -p viking down (docker ps shows what holds the port), then stack.ps1 <stack> up again.
NORNIR_DOCKER_HOST_STACKS is empty	Set the host’s stacks in Run/_host/host.env (or re-run Initialize-DockerHost.ps1 -Role ...).
“Stack … is not configured” with a list of keys	A required key or secret file is missing; the message names the file to edit.
NORNIR_MONOREPO_ROOT is deprecated warning	Rename it to NORNIR_REPO_ROOT_HOST in host.env or your user environment.
Builds/nornir-build/start-nornir-build.ps1 fails with missing NornirDockerRun.ps1 / NornirDevRunMounts.ps1	It is an old copy of the launcher; copies break because the script dot-sources sibling helpers. Re-run Initialize-DockerHost.ps1: it replaces it with a forwarding launcher (the copy is kept as .bak).

Network and TLS

Symptom	Cause and fix
A port is reachable from outside although ufw denies it	Docker-published ports bypass ufw. Bind internal ports to 127.0.0.1 and filter cross-host ports in DOCKER-USER (Ports and remote access).
certs status: “certificate does not cover: …”	The certificate lacks a SAN for some VIKING_HOSTNAME_*. Get a wildcard *.<domain> (or SAN) certificate; a wildcard only covers single-label names, so keep hostname overrides one level under the domain. Or narrow TLS_HOSTNAMES.
certs status: no intermediate certificates	Pass the issuer chain with -Chain; gRPC clients and the Viking desktop client reject a leaf-only fullchain.pem.
certs letsencrypt times out or fails HTTP-01	Port 80 on web is not reachable from the internet, DNS does not point at web yet, or viking-proxy is not up. If Pitt will not open port 80, use TLS_MODE=acme-eab (Pitt IT’s ACME server with EAB) or certs manual.
Let’s Encrypt rate-limit errors while testing	Set TLS_STAGING=1 until the flow works, then back to 0 and re-issue.
certs manual: key is encrypted / key does not match / expires sooner	Decrypt (openssl pkey -in key.pem -out key.dec.pem); check you have the key for this certificate; use -Force to replace a later-expiring certificate on purpose.
Segmentation names fail through the proxy	compute does not accept 40443-40447 from web (Pitt firewall or DOCKER-USER), or stack.ps1 certs push -From web has not copied the certificate to compute.
doctor: DNS <name> does not resolve yet	The Pitt DNS ticket is not done, or the record is not a wildcard.

Build appliance and dashboard

Symptom	Cause and fix
network nornir-dashboard_default not found, or builds never appear on the dashboard	Start nornir-dashboard before nornir-build, then stack.ps1 nornir-build rebuild. If Run/nornir-net-mounts/.run.nornir-net-mounts.env was copied from the old template, it sets NORNIR_MQTT_HOST=host.docker.internal; comment that line out. While it is set the launcher does not join the dashboard network, and on Linux the loopback-bound broker is unreachable.
Bus error, No space left on device under /dev/shm, or pool workers dying	Shared memory too small. The launcher passes --shm-size=64g (NORNIR_DOCKER_SHM_SIZE); recreate containers started by older scripts (stack.ps1 nornir-build rebuild).
Too many open files (EMFILE)	ulimit -n should be 65536; check NORNIR_DOCKER_NOFILE_* and recreate the container.
Missing nas-mounts.tsv	Run Initialize-DockerHost.ps1 -Role compute and edit the site files.
mount: ... Operation not permitted	Path B not applied: the container lacks SYS_ADMIN (started without the launcher or the net-mounts override).
CIFS mount error(2)	The cifs driver is not loaded in the host kernel (Ubuntu: modprobe cifs; Windows: the WSL2 kernel). Check grep cifs /proc/filesystems.
CIFS mount error(13): Permission denied	Wrong user/password/domain in the .cred, a share ACL, or a credentials= path in nas-mounts.tsv that does not match a file in net-creds/. Images older than the 0600 copy fix need a pull.
/storage4 (or another target) empty	NORNIR_NET_MOUNTS is not 1 in the container; check how it was started.
Remount fails after the shell starts	Expected: CAP_SYS_ADMIN is dropped after the mount step. Restart the container.
Wrong image or no GPU	Run pwsh nornir-docker/Test-NornirGpu.ps1; on Ubuntu install the NVIDIA container toolkit (bootstrap-ubuntu.sh --gpu). Force with NORNIR_DOCKER_GPU=1.
TEM scripts fail importing CuPy on a CPU-only host	Fixed: the TEM wrappers select NumPy when no GPU is present. Pull current images.
Dashboard unreachable	stack.ps1 nornir-dashboard status; it binds 127.0.0.1 only, so use the tunnel.
Dashboard: “attempt to write a readonly database”	Run stack.ps1 nornir-dashboard up again; the launcher re-owns the data volume to uid 10001.

Programmer shell

Symptom	Cause and fix
NORNIR_TESTDATA_HOST missing, or settings in Run/nornir-dev/.env ignored	nornir-docker/.env is missing or is a stale copy; recreate the hardlink (Programmer shell and Cursor worker).
DynamicLibNotFoundError: libnvrtc / cudaErrorInsufficientDriver	Host NVIDIA driver too old for the CuPy wheel, or no GPU passthrough.
distro-services/*.sock errors with \\wsl.localhost paths	Wrong distro name or casing; use the exact name from wsl -l -v (or \\wsl$\...).