Transferring data between W&M/VIMS and external clusters
Transferring files between two Globus enabled sites
Globus can be used to transfer files/folders between two HPC clusters. Typically this is done between W&M main-campus/VIMS clusters and other sites one has Globus access to, or even between W&M main-campus and VIMS HPC clusters. Here we will show how to do this between main-campus and VIMS.
To get started, visit https://www.globus.org/data-transfer:
Click the "LOGIN" button at the top right. Once there, log in using your W&M username/password:
If necessary, enter your W&M info at the CAS login page. Once logged in you will see:
At this point, you can click the "Collections" field at the top of the page to select a location to connect to, please refer to offices/it/services/researchcomputing/using/xfers/index for the collection names of various W&M/VIMS HPC filesystems. For demonstration purposes, we will transfer files from 'ewalter's main-campus cluster home directory to his VIMS data10 folder.
First we enter the home directory collection name (see: offices/it/services/researchcomputing/using/xfers/index for these names):
Once selected, if it is a "v4" collection, you will need to enter your cluster login and password. If it were a "v5" collection, you would already be authenticated since v5 collections use the credentials you used to log into Globus in the first step. Since we currently are using globus v4 for main-campus home directories, we need to log in:
Once logged into the home-directory collection, click "Transfer or Sync to ..." to select a second collection. Either collection can be transfer to the other so the order in which collections are selected is not important.
The "Transfer or Sync to ..." button will allow you to select a second collection. Here we will choose the VIMS data10 collection:
Once both collections are accessed and are in the two file transfer panels, you can then select the file paths to reveal the source and target folders for the transfer.nHere we will transfer the whole "OUTPUT" folder from my home directory (/~/) to my folder on /ches/data10. Note that no folder need be selected on the target pane since I am transferring to the currently displayed folder (/ches/data10/ewalter).
Finally, click one of the "Start" buttons above the file-transfer panes. We will select the left side button since we want to push the OUTPUT folder to /ches/data10. If the buttons are not clickable, please click "refresh list" since this means the file-listing may be out of sync.
Once the transfer is started, you will see a pop-up window acknowledging this. There is also a link to go to the transfer activity page (which you can select the "ACTIVITY" window from the left side of the window).
The activity page will show the status of the transfer and the effective speed that you are achieving for the transfer. This is a VERY small transfer so the speed is not realistic. For larger transfers between clusters you should be achieving at least 100-200 MB/s and usually much higher. However, this speed is dependent on activity on the involved fileservers and any traffic congestion in between.
