Kuro
The kuro cluster of SciClone contains 47 nodes each containing 64 AMD Zen 4 cores. The front-end is kuro.sciclone.wm.edu and the startup module file is named.cshrc.kuro. This cluster should be used exclusively for large MPI/parallel jobs which can effectively utilize at least one node.
Hardware
|
Front-end (kuro/ku00) |
Parallel nodes (ku01-ku47) |
||
|---|---|---|---|
| Model | Dell PowerEdge R6625 | ||
| Processor(s) |
2×16-core AMD EPYC 9124 |
2x32 core AMD EPYC 9334 |
|
| Clock speed |
3.0 GHz |
2.8 GHz |
|
| Memory | 384 GB | ||
|
Network interfaces |
Application |
HDR IB (ku00-ib) |
HDR IB (ku??-ib) |
| System |
10 GbE (ku00) |
1 GbE (ku??) |
|
| OS | Rocky Linux 9.2 | ||
Slurm
The SLURM batch system is used on kuro to run jobs. Each node has 64 cores and 384 GB of memory. The maximum walltime for all jobs is 48hrs.
To run an interactive job on kuro:
salloc -N 2 --ntasks-per-node=64 -t 1:00:00 # will get you 128 CPU cores on 2 nodes
Debug jobs:
There is a debug reservation which is available from 9am on Monday through 3pm on Friday. This reservation allows up to 2 nodes and 30 mins walltime per job. To use this reservation, you must specify it in your slurm command-line or batch script, i.e.:
salloc -N 1 --ntasks-per-node=64 -t 0:30:00 --reservation=debug # gets you 1 node for 30min
Weekend dedicated time:
From 3pm on Friday through 8pm on Sunday the entire Kuro cluster will be reserved for dedicated use by the select members of the VIMS CCRM.
User Environment
To login, use SSH from any host on the William & Mary or VIMS networks and connect to kuro.sciclone.wm.edu with your HPC username (usually the same as your WMuserid) and W&M password.
Your home directory on Kuro is the same as everywhere else on SciClone, and all of the usual filesystems (/sciclone/homeXX, /sciclone/dataXX, /sciclone/scrXX, /local/scr, etc.) are available.
SciClone uses Environment Modules (a.k.a Modules) to automatically configure the user's shell environment across multiple computing platforms, as well as to organize the dozens of different software packages which are available on the system. We support tcsh as the primary shell environment for user accounts and applications.
The file which controls startup modules for kuro is .cshrc.kuro. A copy of this file can be found in /usr/local/etc/templates on kuro.sciclone.wm.edu.
Compilers and preferred flags:
The preferred module for compilers on kuro is: intel/compiler-2024.0 which is the Intel OneAPI compiler which contains Fortan, C and C++ compilers as well as MKL libraries. The preferred MPI pacakge is: openmpi-ib/intel-2024.0/4.1.6 which contains openmpi-4.1.6 compiled for the Intel 2024.0 compiler stack. Also, the intel/mpi-2021.11 module is available which contains the latest Intel MPI software, which also works on Kuro. All W&M HPC clusters also GNU compilers available.
One important thing to point out is that the Intel OneAPI compiler has changed their compiler names from ifort,icc, and icpc to ifx,icx and icpx for Fortran, C, and C++ compilers, respectively. Most of the compiler flags are the same as before..
Here are suggested flags for AMD Zen 4 cpus for the Intel OneAPI 2024.0 compiler as well as GCC:
| Intel | C | icx -O3 -axCORE-AVX2 |
|---|---|---|
| C++ | icpx -std=c11 -O3 -axCORE-AVX2 | |
| Fortran | ifx -O3 -axCORE-AVX2 | |
| GNU | C | gcc -march=znver3 -O3 |
| C++ | g++ -std=c11 -march=znver3 -O3 | |
| Fortran | gfortran -march=znver3 -O3 |
Preferred filesystems
The Kuro cluster has a new Lustre scratch filesystem mounted as /sciclone/scr-lst/$USER and should be the preferred filesystem for I/O on this cluster. Each kuro node also has a 960 GB SSD /local/scr partition which can also be used by jobs.
MPI
All W&M HPC clusters have both Intel MPI and OpenMPI installed. Some older clusters also have Mvapich2. All parallel jobs (shared memory, MPI or hybrid) need to be run through the batch system. The standard way to do this is to use the 'srun' command:
#!/bin/tcsh
#SBATCH --job-name=test
#SBATCH --nodes=2 --ntasks-per-node=64
#SBATCH -t 30:00
srun ./a.out >& LOG
Please see our Slurm page for more help.
