Lustre Best Practices
The Lustre parallel file-system mounted at /sciclone/pscr is meant to give superior performance to any of the other file-systems on SciClone. Our current configuration has six Object Storage Targets (OSTs) and one Metadata Server (MDS). In order to use this file-system efficiently, some default preferences may need to be changed. The most important of these are the stripe count of a particular file and/or directory. Additionally, the stripe size can also be adjusted. See https://www.nics.tennessee.edu/computing-resources/file-systems/lustre-striping-guide for a more detailed discussion about striping.
The default stripe size on /sciclone/pscr is 1 MB and the default stripe count is 1. For small to medium (less than ~10 GB) file-sizes, the default choices of stripe size and stripe count should be sufficient for most workloads. However, for files larger than this, it is much more efficient to increase the stripe count so that multiple disk arrays are used for I/O. We recommend that users use the lfs setstripe command to increase the stripe count. For files greater than 100 GB, you may want to also increase the stripe size to see the effect on performance.
Increasing stripe count and stripe size
The easiest way to set a stripe count and size is per directory. Pre-existing files can't be restriped on demand and must be rewritten after striping changes are made. Say we have the following directory:
[28 ewalter@bora /sciclone/pscr/ewalter ]$mkdir BIGFILES
[29 ewalter@bora /sciclone/pscr/ewalter ]$lfs getstripe -d BIGFILES/
stripe_count: 1 stripe_size: 1048576 stripe_offset: -1
[30 ewalter@bora /sciclone/pscr/ewalter ]$
This shows that the stripe_count is 1 and the stripe size is 1MB. To increase the stripe_count:
[30 ewalter@bora /sciclone/pscr/ewalter ]$lfs setstripe -c -1 BIGFILES/
[31 ewalter@bora /sciclone/pscr/ewalter ]$lfs getstripe -d BIGFILES/
stripe_count: -1 stripe_size: 1048576 stripe_offset: -1
[32 ewalter@bora /sciclone/pscr/ewalter ]$
Here we use '-1' as the stripe count which means "use all OSTs". This is the recommended setting for large files. We could also increase the stripe size:
[34 ewalter@bora /sciclone/pscr/ewalter ]$lfs setstripe -S 16m BIGFILES/ [35 ewalter@bora /sciclone/pscr/ewalter ]$lfs getstripe -d BIGFILES/ stripe_count: 1 stripe_size: 16777216 stripe_offset: -1
[36 ewalter@bora /sciclone/pscr/ewalter ]$
But notice that the count reverts to 1. Both must be set at the same time since the other settings are reverted to the default upon changes:
[36 ewalter@bora /sciclone/pscr/ewalter ]$lfs setstripe -S 16m -c -1 BIGFILES/
[37 ewalter@bora /sciclone/pscr/ewalter ]$lfs getstripe -d BIGFILES/
stripe_count: -1 stripe_size: 16777216 stripe_offset: -1
[38 ewalter@bora /sciclone/pscr/ewalter ]$
You can also probe the current striping pattern of files by querying the directory containing them:
[80 ewalter@bora /sciclone/pscr/ewalter ]$ls -lh BIGFILES/
total 12G
-rw-r-----. 1 ewalter hpcf 13G Oct 30 15:32 bigfile
[81 ewalter@bora /sciclone/pscr/ewalter ]$lfs getstripe BIGFILES/
BIGFILES/
stripe_count: -1 stripe_size: 16777216 stripe_offset: -1
BIGFILES//bigfile
lmm_stripe_count: 6
lmm_stripe_size: 16777216
lmm_pattern: 1
lmm_layout_gen: 0
lmm_stripe_offset: 42
obdidx objid objid group
42 11856640 0xb4eb00 0
31 12399579 0xbd33db 0
32 12435729 0xbdc111 0
41 12387123 0xbd0333 0
52 12036743 0xb7aa87 0
51 12376965 0xbcdb85 0
Or just one file:
[82 ewalter@bora /sciclone/pscr/ewalter ]$lfs getstripe BIGFILES/bigfile
BIGFILES/bigfile
lmm_stripe_count: 6
lmm_stripe_size: 16777216
lmm_pattern: 1
lmm_layout_gen: 0
lmm_stripe_offset: 42
obdidx objid objid group
42 11856640 0xb4eb00 0
31 12399579 0xbd33db 0
32 12435729 0xbdc111 0
41 12387123 0xbd0333 0
52 12036743 0xb7aa87 0
51 12376965 0xbcdb85 0
