# nanoHPC documentation

> Plain Markdown copy of the How to page.

[Open the website view](./#docs)

## 0. Setup

### SSH access to nanoHPC

On your laptop, use an existing SSH key or create one at an unused path:

```bash
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_nanoHPC
```

Send the public key (.pub) to the cluster admin and ask for an account on the nanoHPC front node. Keep the private key on your laptop. Add your assigned username and key path to ~/.ssh/config:

```bash
Host nanoHPC
    HostName login.demo.example.org
    User USER
    IdentityFile ~/.ssh/id_ed25519_nanoHPC
```

## 1. SSH into nanoHPC

```bash
ssh -A nanoHPC
```

### Get your code onto nanoHPC

Forward your laptop’s SSH agent, then clone or pull from GitHub. This uses your loaded laptop key without copying its private key to nanoHPC. Only forward your agent to a machine you trust. (Other options: a read-only deploy key for one repository, or gh auth login on nanoHPC.)

```bash
git clone git@github.com:OWNER/REPOSITORY.git
cd REPOSITORY
```

## 2. Submit a job

A job script asks Slurm for resources, then runs your commands. A job goes to the default partition unless you name another one with --partition. Each partition has a maximum run time; if you leave out --time, a job gets that maximum. Default memory per CPU and CPUs per GPU are also set per partition, and memory is enforced.

The partitions, their time limits, and their machines are live values: see them on the [Cluster policy page](./#policy).

undefined

### job.sh: save this file in your project

```bash
#!/bin/bash
#SBATCH --gpus=1
#SBATCH --time=00:05:00

set -euo pipefail
uv run --locked python script.py
```

set -euo pipefail stops the script when a command fails, an unset variable is used, or a command in a pipeline fails. Replace uv run --locked python script.py with the command that runs your code. uv is installed on the front node and the compute machines. If jobs have outbound internet access, uv can download locked packages; downloads use job time. Use nvidia-smi when you only want to check the GPU assigned to the job.

### Terminal: type these commands after saving job.sh

```bash
cd ~/YOUR_PROJECT
sbatch job.sh
```

Shared vs scratch below explains where to submit from and the trade-offs of each way.

## Example job scripts

- [Download gpu-check-job.sh](job-examples/gpu-check-job.sh): one GPU, checks the assigned GPU.
- [Download one-gpu-uv-job.sh](job-examples/one-gpu-uv-job.sh): one GPU, runs a uv Python script.
- [Download four-gpu-torchrun-job.sh](job-examples/four-gpu-torchrun-job.sh): four GPUs, runs PyTorch distributed training.
- [Download cpu-memory-job.sh](job-examples/cpu-memory-job.sh): one GPU with explicit CPU and RAM requests.
- [Download interactive-notebook.sh](job-examples/interactive-notebook.sh): Jupyter or browser-based VS Code inside an interactive job.

## Do's and don'ts

Do: run installs, large downloads, Jupyter, and browser-based VS Code inside a batch job or an interactive shell on a compute machine. Keep notebook sessions protected by their token. Desktop VS Code can connect to an Existing Jupyter Server in a job.

Don't: run heavy or long-lived work on nanoHPC; the front node is shared by everyone. Do not leave personal services, crontabs, or open ports there. Desktop VS Code Remote-SSH cannot connect to the front node because SSH forwarding is limited to compute machines.

## Jobs examples

### One GPU

Submit with sbatch from a git worktree, as shown in Shared vs scratch. The job goes to the default partition. The program must accept --output, or change that argument to match your program. Results go to an absolute path in your home, so they are saved as the job writes them.

```bash
#!/bin/bash
#SBATCH --gpus=1
#SBATCH --time=02:00:00
set -euo pipefail
uv run --locked python train.py --output "$HOME/YOUR_PROJECT/results/$SLURM_JOB_ID"
```

### Four GPUs

For training code that already supports four GPU workers. Requesting four GPUs does not make single-GPU code use them. Submit this script with sbatch job.sh from the shared project.

```bash
#!/bin/bash
#SBATCH --gpus=4
#SBATCH --cpus-per-task=16
#SBATCH --time=02:00:00
#SBATCH --output=training-%j.log
set -euo pipefail
uv run --locked torchrun --nproc-per-node=4 train.py
```

### More CPUs and RAM

Request 8 CPUs and 64 GiB of RAM for one GPU. The request must fit on one compute machine; the Machines page lists each machine’s CPU cores and RAM. Submit with sbatch job.sh from the shared project.

```bash
#!/bin/bash
#SBATCH --gpus=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=64G
#SBATCH --time=02:00:00
#SBATCH --output=training-%j.log
set -euo pipefail
uv run --locked python train.py
```

## Batch vs interactive

### Batch job

Save your commands in a job script and submit it with sbatch, preferably from a git worktree (see Shared vs scratch). Slurm starts it when the requested resources are free, in the default partition unless you name another one, and stops it at its time limit. You can log out while it waits or runs; its output goes to the file named by --output.

When to use: training runs, sweeps, and anything that runs without you watching it.

```bash
cd ~/YOUR_PROJECT/.worktrees/RUN
sbatch job.sh
squeue --me
```

### Interactive shell

Partitions set up for interactive shells accept only this command form. Replace PARTITION with such a partition; the Cluster policy page lists the partitions. You can change the resource amounts and time. Slurm waits for those resources, selects the compute node, and opens Bash there. Exit the shell to release the resources. Direct SSH to compute nodes is for administrators only.

When to use: debugging, short tests, and checking that your code and environment work on a GPU before you submit a long batch job.

```bash
srun --partition=PARTITION --gpus=1 --cpus-per-task=4 --time=01:00:00 --pty bash -l
```

## Shared vs scratch

Shared mode works on project files in your home; scratch mode works on a private copy on the compute machine. The tabs explain each way and its trade-offs.

### Shared mode: With a git worktree

Make a separate git worktree for each run and submit from it with sbatch. The job runs that fixed copy of the code, so you can keep editing and pulling in your main checkout. The worktree holds the last commit only, so commit your changes first, and add .worktrees/ and logs/ to .gitignore. Write results to an absolute path in your home outside the worktree, such as $HOME/YOUR_PROJECT/results: they are saved as the job writes them, and a job stopped at its time limit loses only the work in progress. Cost: each worktree is a full copy of the code in your home, and uv sync builds a separate .venv in it. The job reads both over the network, and building the .venv copies the whole environment into your home, which takes job time for large packages such as PyTorch. Remove the worktree when the job has ended; git worktree remove refuses if it holds files you have not committed.

When to use: most jobs: training runs and sweeps, especially long ones or many at once while you keep changing the code.

```bash
cd ~/YOUR_PROJECT
mkdir -p logs
RUN=.worktrees/$(date +%Y%m%d-%H%M%S)
git worktree add --detach "$RUN"
(cd "$RUN" && sbatch --output="$HOME/YOUR_PROJECT/logs/job-%j.log" job.sh)

# After the job has ended:
git worktree remove "$RUN"
```

### Shared mode: Plain

Use this when you want the job to read and write the shared project directly. There is no project copy or output copy-back. A git pull, branch switch, or file edit can change code used by processes that start later in a pending or running job.

When to use: jobs that spend their time on CPU or GPU work and read or write few files, when you will not edit the project while the job waits or runs.

```bash
cd ~/YOUR_PROJECT
sbatch job.sh
```

### Shared mode: With local scratch

Keep code and results in your home, and put large datasets and temporary files on the compute machine’s local disk. Add these lines to job.sh. stage-dataset --private copies a dataset folder from your home (path relative to your home) to local scratch once, and later jobs on the same machine reuse it. If the administrator set up shared datasets, use stage-dataset --shared NAME to copy one from the read-only /shared/datasets area into your private local scratch. A staged copy that no job has used for 21 days is deleted automatically. Files in TMPDIR are not copied back; the last line deletes them.

When to use: large datasets that are read many times, or jobs that write many temporary files, where reading and writing over the shared home would slow the job.

```bash
export TMPDIR=/scratch/$USER/tmp/$SLURM_JOB_ID
mkdir -p "$TMPDIR"
DATA=$(stage-dataset --private datasets/MY_DATASET)
uv run --locked python train.py --data "$DATA" --output "$HOME/YOUR_PROJECT/results/$SLURM_JOB_ID"
rm -rf "$TMPDIR"
```

### Scratch mode: Basic

Use this when you want a private project copy while the job runs. cluster-submit copies Git-tracked files when the job starts, including uncommitted edits. Changes made while the job waits may be copied; later changes do not affect the running copy. Add #CLUSTER include= for other inputs and #CLUSTER copy-back= for results to return, including after an application failure. Git commands such as git rev-parse do not work in the scratch copy, which has no .git folder. --mode is required (--mode=shared is the same as plain sbatch). sbatch options go before the script in the --name=value form, for example cluster-submit --mode=scratch --output=$HOME/YOUR_PROJECT/logs/job-%j.log job.sh.

When to use: projects with many small files or heavy reading and writing inside the project folder, or when the job should use a fixed copy of the project including uncommitted edits.

```bash
cd ~/YOUR_PROJECT
cluster-submit --mode=scratch job.sh
```

### Scratch mode: Results in home

Run from a private scratch copy, but write results and checkpoints to an absolute path in your home. They are saved as the job writes them, so they do not depend on copy-back at the end of the job, and a time limit or crash loses only the work in progress. These paths need no #CLUSTER copy-back= line.

When to use: long scratch-mode runs whose checkpoints you cannot afford to lose.

```bash
#!/bin/bash
#SBATCH --gpus=1
#SBATCH --time=12:00:00
#SBATCH --output=training-%j.log
set -euo pipefail
uv run --locked python train.py --output "$HOME/YOUR_PROJECT/results/$SLURM_JOB_ID"

# Submit with:
# cd ~/YOUR_PROJECT
# cluster-submit --mode=scratch job.sh
```

### Scratch mode: Inputs and outputs

For scratch mode. Add these lines to the job script. Outputs return to the same relative path inside the original project. Choose a different output directory for each run. Non-Git projects need explicit inputs.

```bash
#CLUSTER include=datasets/small-test
#CLUSTER copy-back=results/
```

### Scratch mode: Recover partial results

For scratch mode. Failed jobs attempt to return declared outputs and keep their scratch directory. Read the job log for the recovery path. Node loss or forced termination can prevent copy-back. Write important checkpoints to your experiment’s persistent output path during execution.

```bash
less training-JOB_ID.log
```

## Other settings

### Inspect your jobs

squeue shows queued and running jobs. sacct includes completed jobs. sstat shows CPU and memory use so far. An overlapping job step can inspect the assigned GPUs. seff is not installed; use sacct and sstat. Terminal output files and application checkpoints are separate outputs.

```bash
squeue --me
sacct --starttime today --format=JobID,JobName,State,Elapsed,AllocTRES
sstat -j JOB_ID.batch --format=JobID,AveCPU,MaxRSS
srun --jobid=JOB_ID --overlap nvidia-smi
less training-JOB_ID.log
```

### Home space

Your home is shared between the front and compute machines. Your usage appears privately when you log in. At 250 GB, the login box shows a notice. At 300 GB, new jobs wait until you clear space. Running jobs continue, and writing to your home stays possible. Keep large temporary data on local scratch.

```bash
du -sh "$HOME"
du -h -d 1 "$HOME" | sort -h
```

### GPU software

The NVIDIA driver is installed on GPU machines. Check the assigned GPU and available driver from inside a job. Use the Machines page for the GPU model and memory on each machine.

```bash
nvidia-smi
```

### Containers

Apptainer runs containers inside Slurm jobs as your user, with no Docker daemon or root account in the container. It can use Docker registry images and existing Singularity .sif files. Add --nv for a GPU job; Slurm still controls which GPUs that job can use. Downloaded images and package data use your normal cache and storage settings.

```bash
# Inside a GPU job or interactive shell:
apptainer exec --nv docker://nvidia/cuda:13.0.0-base-ubuntu22.04 nvidia-smi -L

# Reuse a local image:
apptainer pull pytorch.sif docker://pytorch/pytorch
apptainer exec --nv pytorch.sif python train.py
```

### Interactive notebooks

Download interactive-notebook.sh to your laptop, copy it into your shared home on the cluster, then open an interactive Slurm shell on a compute machine. The helper starts Jupyter or browser-based VS Code and prints an SSH tunnel command to run on your laptop. Desktop VS Code Remote-SSH cannot connect to the front node because SSH forwarding is limited to compute machines. Use browser-based VS Code in a job, or choose Existing Jupyter Server in desktop VS Code and paste the Jupyter link. The session uses a token and ends when the interactive job ends. Replace PARTITION with a partition that allows interactive shells.

```bash
# On your laptop, after downloading interactive-notebook.sh:
scp interactive-notebook.sh nanoHPC:~/interactive-notebook.sh
ssh nanoHPC

# On the front node, then in the interactive shell:
srun --partition=PARTITION --gpus=1 --time=01:00:00 --pty bash -l
bash ~/interactive-notebook.sh jupyter login.demo.example.org
# Or: bash ~/interactive-notebook.sh vscode login.demo.example.org
```

### Caches

uv, Hugging Face, and PyTorch caches are placed on local scratch on compute machines so package downloads do not fill the shared home.

```bash
printf "uv: %s\nHugging Face: %s\nPyTorch: %s\n" "$UV_CACHE_DIR" "$HF_HOME" "$TORCH_HOME"
```