• Departments & Offices
• IT
• Services
• Research Computing
• old
• Architecture & Hardware
• Nodes
• Meltemi

Meltemi

Meltemi is composed of 100 64-core/256-thread Xeon Phi "Knights Landing" compute nodes (mlt001-mlt100) connected with Omni-Path fabric. Its front-end is meltemi.sciclone.wm.edu. This cluster uses scratch file-system on the Mistral file server mounted on the Omni-Path fabric at /sciclone/scr-mlt/$USER . All other SciClone file-systems are mounted through the 1GbE interface and are, hence, not recommended for I/O intensive computation.

Access to Meltemi sub-cluster is limited to large-scale parallel codes, and users interested in gaining access should email hpc-help@wm.edu .

• Hardware
• Torque Node Specfiers
• User Environment
• Preferred File Systems
• Compiler Flags
• MPI

 

Hardware

Front-end server:

Mistral File server:

Meltemi compute nodes (mlt001-mlt100)

Torque Node Specifiers:

All access to the compute nodes (for either interactive or batch work) is via the TORQUE resource manager, as described elsewhere.   All meltemi compute nodes are defined to have np=256 in Torque. This means that all 256 threads can be used on all nodes.  The first 98 nodes (mlt001-mlt098) are configured to run at most one job.  Therefore, users should take all 64 physical cores or all 256 threads per node for all jobs.  These 98 nodes have the following node properties:

meltemi, knl, c20, opa

Since only one job can occupy meltemi at one time, the following node specs are sufficient for a pure MPI job:

#PBS -l nodes=1:meltemi:ppn=64 (64 cores on a single node)

#PBS -l nodes=4:meltemi:ppn=64 (256 cores across four nodes)

This also works for correct processor placement since cores # 0-63 are the physical cores.

For running a hybrid job which requires more than 64 threads total, the following will work:

#PBS -l nodes=1:meltemi:ppn=256 (256 threads on a single node)

#PBS -l nodes=4:meltemi:ppn=1024 (1024 threads across four nodes)

The remaining 2 meltemi nodes (mlt098 & mlt100) are temporarily set to allow for multiple jobs on a node. To distinguish these two from the other 98 meltemi nodes, we have set the following node properties:

meltemishared, knl, c20, opa

Therefore, a user can get a single core on a meltemi node by using:

#PBS -l nodes=1:meltemishared:ppn=1 (1 core on a single node)

This is a temporary situation that will be eliminated once the testing of the cluster is completed.

Torque time limit

The maximum walltime for jobs on meltemi is 72hrs.  

---

User Environment

To login, use SSH from any host on the William & Mary or VIMS networks and connect to meltemi.sciclone.wm.edu with your username and W&M password.

Your home directory on meltemi is the same as everywhere else on SciClone, and all of the usual filesystems (/sciclone/homeXX, /sciclone/dataXX, /sciclone/scrXX, /local/scr, etc.) are available throughout the meltemi subcluster.  Additionally, the scratch file-system (scr-mlt) is available throughout the meltemi subcluster.

SciClone uses Environment Modules (a.k.a Modules) to automatically configure the user's shell environment across multiple computing platforms, as well as to organize the dozens of different software packages which are available on the system. We support tcsh as the primary shell environment for user accounts and applications.  

The file which controls startup modules is .cshrc.el7-phi

The most recent versions of theses files can be found in /usr/local/etc/templates on any of the front-end servers including meltemi.

---

Preferred filesystems

The preferred file system for all work on meltemi is the scr-mlt scratch file system.  This is mounted as /sciclone/scr-mlt/$USER on the meltemi front-end and compute nodes.  This is the only recommended filesystem on Meltmei at present. 

---

Compiler flags

The Meltemi subcluster has the Intel Parallel Studio XE 2017 compiler suite as well as version 6.3.0 of the GNU compiler suite.  Here are suggested compiler flags which should result in code that can use the AVX-512 instruction set on the KNL architecture. However, in most cases, the source code would require further optimization to fully utilize the "Many Integrated Core" architecture in KNL.   (These flags are taken from: http://www.prace-ri.eu/best-practice-guide-knights-landing-january-2017/ ):

Suggested compiler options for the Meltemi subcluster
Intel C compiler	icc -O3 -xMIC-AVX512 -fma -align -finline-functions
Intel C++ compiler	icpc -std=c11 -O3 -xMIC-AVX512 -fma -align -finline-functions
Intel Fortran compiler	ifort -O3 -xMIC-AVX512 -fma -align array64byte -finline-functions
GNU C compiler	gcc -march=knl  -O3 -mavx512f -mavx512pf -mavx512er -mavx512cd -malign-data=cacheline -finline-functions
GNU C++ compiler	g++ -std=c11 -march=knl -O3  -mavx512f -mavx512pf -mavx512er -mavx512cd  -malign-data=cacheline -finline-functions
GNU Fortran compiler	gfortran -march=knl -O3  -mavx512f -mavx512pf -mavx512er -mavx512cd -malign-data=cacheline -finline-functions

---

MPI

Currently there are two versions of (intel only) MPI available on the Meltemi cluster:  openmpi (v1.10.6) and mvapich2 (v2.2).   Both of these should be used through the mvp2run wrapper script.  For pure MPI jobs, the syntax for both versions of MPI are the same:

---

#!/bin/tcsh 
#PBS -N MPI 
#PBS -l nodes=5:knl:ppn=256 # NOTE ppn=256 for pure MPI jobs 
#PBS -l walltime=12:00:00 
#PBS -j oe 

cd $PBS_O_WORKDIR 

mvp2run ./a.out >& LOG

---

Launching hybrid MPI/OpenMP jobs is being tested for OpenMPI on Meltemi, while mvapich2 can only launch pure MPI jobs.  This section will be updated soon.

This cluster is not accessible for general usage and is limited to codes that require large computing capabilities and with codes optimized for KNL platform. Users interested in gaining access should email hpc-help@wm.edu .