3DEM Data Transfer and Management Guide
Outline
- Data Transfer Methods
- What is Corral?
- Prerequisites for 3DEM Project User
- Using Command Line Tools to Transfer and Organize Data
- scp, sftp, & rsync
- Prerequisites for Data Transfer with Command Line Tools
- Determining the Path to Your Data File(s) on Your Local System
- Figure 1. Use Get Info to determine “Where” the path of your data file(s) is
- Determining the Path to Your Transfer Directory on Corral
- Transfer Data with scp
- Transfer Data with sftp
- Transfer Data with rsync
- Transfer Data with Cyberduck
- References
Data Transfer and Management Guide
Transferring 3DEM project data from your local machine to TACC’s Corral system can be accomplished using two methods: command line tools (scp, sftp, rsync) and graphical tools (Cyberduck).
|
|
||
Command Line Tool | scp | ||
Command Line Tool | sftp | ||
Command Line Tool | rsync | ||
|
Cyberduck |
Corral is one of TACC’s systems dedicated to storage and data management. Corral can be used for storing project data. As a 3DEM project user, you have access to the Corral server “data.tacc.utexas.edu
”, as well as the Corral folder /corral-repl/projects/NeuroNex-3DEM/projects/
. The Corral system is to be used exclusively for transferring and accessing 3DEM project data.
Prerequisites for 3DEM Project User
There are two requirements needed prior to transferring data:
- A TACC Account
- Multi-Factor Authentication (MFA) pairing with the TACC Token app
As a 3DEM project user, you should already have access to a TACC account through the TACC Portal. If you are unsure of your TACC account username, please consult the “TACC Account Request Confirmation” email you have received from TACC Accounting Staff, or contact acantu@tacc.utexas.edu.
TACC now requires MFA pairing. To set up MFA, you can follow this tutorial.
Using Command Line Tools to Transfer and Organize Data
A common method of transferring files between TACC resources and/or your local machine is through the command line.
scp, sftp, & rsync
These three command line tools are secure and can be used to accomplish data transfer. You can run these commands directly from the terminal if your local system runs Linux or macOS.
Note: It is possible to use these command line tools if your local machine runs Windows, but you will need to use a ssh client (ex. PuTTY). However, for the scope of this project and ease in data transfer, it is recommended that Windows users follow our Transfer Data with Cyberduck guide below. |
For users that are new to the command line, using either scp
or sftp
to transfer data is advised.
Prerequisites for Data Transfer with Command Line Tools
Before we begin, you will need to know:
- the path to your data file(s) on your local system
- the path to your transfer directory on Corral
Determining the Path to Your Data File(s) on Your Local System
In order to transfer your project data, you will need to know where your file(s) is located on your local system. To do so, navigate through the file system on your local computer.
For example, you can do so on Mac by going to Finder. Common data file locations such as Desktop and Documents should be accessible through the Favorites pane on the left-hand side of the window. You can also do Finder > Go to find other common locations on your computer.
Once you have navigated to where your file(s) is located, take note of its path by right-clicking on the file and selecting Get Info.
Figure 1. Use Get Info to determine “Where” the path of your data file(s) is
For example, a file located in a folder named project-data under the directory Documents would have the following path on a mac:
/Users/$Username/Documents/project-data/my_file.txt
Determining the Path to Your Transfer Directory on Corral
A transfer directory on Corral has already been set up for you and is available for data transfer. The transfer directory path is individualized to your portal projects.
For example, for a user that has created a project under the 3DEM directory, the path to their transfer directory on Corral would be:
/corral-repl/projects/NeuroNex-3DEM/projects/$PROJECT_ID
Note: The transfer directory name will always be named the same as the `Project Id` value displayed next to the project folder listed under the 3DEM.org > Workbench > Data Files > My Projects UI. |
If you are unsure of your transfer directory path, please consult the email you have received from jcarson@tacc.utexas.edu, or contact jcarson@tacc.utexas.edu directly.
scp
copies files between hosts on a network. To transfer a file (ex. my_file.txt
) to Corral via scp
, open a terminal on your local computer and navigate to the path where your data file is located.
localhost$ cd Documents/project-data/ |
Assuming your TACC username is jdoe
and you are affiliated with UT Austin, a scp
transfer that pushes my_file.txt
from your local computer to Corral’s secure system would look like this:
|
Note: This command will copy your data file directly to your individualized project directory on Corral’s system. |
If you have not done so already, enter this command in your terminal, replacing the file name, TACC username, and your individualized project directory ID appropriately.
After entering the command, you will be prompted to login to Corral’s secure system by entering the password associated with your TACC account as well as the token value generated from your TACC token app.
The data transfer was successful if your terminal output looks like this:
my_file.txt 100% ## #.# KB/s ##:## |
If you wish to learn more about scp
, you can do so here.
sftp
is a file transfer program that allows you to interactively navigate between your local file system and the remote Corral system. To transfer a file (ex. my_file.txt
) to Corral via sftp
, open a terminal on your local computer and navigate to the path where your data file is located.
localhost$ cd Documents/project-data/ |
Assuming your TACC username is jdoe
and you are affiliated with UT Austin, a sftp
transfer that pushes my_file.txt
from your local computer to Corral would look like this:
|
If you have not done so already, enter this command in your terminal, replacing the TACC username and your individualized project directory name appropriately.
At this point, you have logged into Corral and been redirected to your transfer directory. To confirm your location on Corral, enter the following command:
sftp> pwd |
To list the files currently in your transfer directory:
|
To list the files currently in your local directory (ex. /Users/myname/Documents/3dem-data/
):
|
Note: The leading l in the lls command denotes that you are listing the contents of your local working directory. |
To transfer my_file.txt
from your local computer to /corral-repl/projects/NeuroNex-3DEM/projects/$PROJECT_ID
:
|
To check if my_file.txt
is in the utaustin
subfolder:
|
To exit out of sftp
on the terminal:
|
If you wish to learn more about sftp
, you can do so here.
Transfer Data with rsync
rsync
is a file copying tool that can reduce the amount of data transferred by sending only the differences between the source files on your local system and the existing files in your transfer directory. To transfer a file (ex. my_file.txt
) to Corral via rsync
, open a terminal on your local computer and navigate to the path where your data file is located.
localhost$ cd Documents/project-data/ |
Assuming your TACC username is jdoe
and you are affiliated with UT Austin, a rsync
transfer that pushes my_file.txt
from your local computer to Corral would look like this:
localhost$ rsync ./my_file.txt jdoe@data.tacc.utexas.edu:/corral-repl/projects/NeuroNex-3DEM/projects/$PROJECT_ID/ |
If you have not done so already, enter this command in your terminal, replacing the TACC username and your individualized project directory name appropriately.
If the command returns 0 in your terminal, the data transfer was successful.
If you wish to learn more about rsync
and how to synchronize your file transfer, you can do so here or follow the “Transferring files using scp
and rsync
” section of our Stampede2 guide.
Cyberduck is a free graphical tool for data transfer and is an alternative to using the command line. With a drag-and-drop interface, it is easy to transfer a file from your local system to Corral’s secure system. You can use Cyberduck for Windows or macOS.
Download and install Cyberduck for Windows on your local machine.
Once installed, click “Open Connection” in the top left corner of your Cyberduck window.
Figure 2. Windows Cyberduck and “Open Connection” setup screen
To setup a connection, type in the server name data.tacc.utexas.edu
. Add your TACC username and password in the spaces provided. If the "More Options" area is not shown, click the small triangle button to expand the window; this will allow you to enter the path to your transfer directory (ex. /corral-repl/projects/NeuroNex-3DEM/projects/$PROJECT_ID
) so that when Cyberduck opens the connection you will immediately be in your individualized project directory on Corral. Click the "Connect" button to open your connection.
Consult Figure 3. below to ensure the information you have provided is correct. If you have not done so already, replace the “Path” with the path to your individualized transfer directory.
Figure 3. Windows “Open Connection” setup screen
Note: You will be prompted to “allow unknown fingerprint…” upon connection. Select “allow” and enter your TACC token value. |
Once connected, you can navigate through your remote file hierarchy using familiar graphical navigation techniques. You may also drag-and-drop files from your local computer into the Cyberduck window to transfer files to Corral.
Download and install Cyberduck for macOS on your local machine.
Once installed, go to Bookmark > New Bookmark to setup a connection.
Note: You cannot select “Open Connection” in the top left corner of your Cyberduck window as macOS’ setup screen is missing the “More Options” button. |
To setup a connection using “New Bookmark", type in the server name data.tacc.utexas.edu
. Add your TACC username and password in the spaces provided. If the "More Options" area is not shown, click the small triangle or button to expand the window; this will allow you to enter the path to your transfer directory (ex. /corral-repl/projects/NeuroNex-3DEM/projects/$PROJECT_ID
) so that when Cyberduck opens the connection you will immediately be in your individualized transfer directory on Corral. As you fill out the information, Cyberduck will be creating the bookmark for you. Exit out of the setup screen and click on your newly created bookmark to launch the connection.
Figure 4. macOS “New Bookmark” setup screen
Consult Figure 4. above to ensure the information you have provided is correct. If you have not done so already, replace the “Path” with the path to your individualized transfer directory.
Note: You will be prompted to “allow unknown fingerprint…” upon connection. Select “allow” and enter your TACC token value. |
Once connected, you can navigate through your remote file hierarchy using familiar graphical navigation techniques. You may also drag-and-drop files from your local computer into the Cyberduck window to transfer files to Corral.
- https://man7.org/linux/man-pages/man1/scp.1.html
- https://man7.org/linux/man-pages/man1/sftp.1.html
- https://man7.org/linux/man-pages/man1/rsync.1.html
- https://portal.tacc.utexas.edu/tutorials/multifactor-authentication
- https://portal.tacc.utexas.edu/user-guides/corral
- https://portal.tacc.utexas.edu/user-guides/stampede2
- https://portal.xsede.org/data-management