Data Transfer and Management Guide
Outline
Data Transfer and Management Guide
Transferring Portal data from your local machine to
one of TACC's remote storage systems can be accomplished using two methods: command line tools (scp
, sftp
,
rsync
) and graphical user interface (Cyberduck).
Data Transfer Methods
Usage Mode |
Transfer Method |
Command Line Tool |
scp
|
Command Line Tool |
sftp
|
Command Line Tool |
rsync
|
Graphical Tool |
Cyberduck
|
What is TACC's Storage Server?
A TACC storage system is a logically defined resource designed to provide data storage and management capabilities to TACC portal users through the portal interface. Each portal is accessible over a URL path (e.g.,
sub.domain.tacc.utexas.edu
) which we will subsequently refer to as “host”.
Storage systems can be configured for both normal and protected data (e.g., HIPAA) in a secured location, depending on the project requirements established for the portal. This location is exposed as a path to a directory on the secure system (e.g.,
/secure-server-root/projects/directory_name
) which this document will subsequently refer to as
/transfer/directory/path
.
Storage systems are to be used exclusively for transferring and accessing data through the portal.
Prerequisites for Portal User
There are two prerequisites for accessing a portal and transferring data:
- A TACC Account
-
Multi-Factor Authentication (MFA) pairing with the
TACC Token app
All portal users will need to create a TACC account in the TACC User Portal (which can be accessed at
TACC Portal.
If you have forgotten your TACC account credentials, please refer to your email for a message titled “TACC Account Request Confirmation” or use the TACC Portal's
password reset form
or
username recovery form.
Access to all TACC resources requires a completed Multi-Factor Authentication pairing with your TACC credentials. To set up MFA, please reference TACC Portal's
Multi-Factor Authentication tutorial.
A common method of transferring files between TACC resources and/or your local machine is through the command line.
scp
, sftp
, & rsync
These three command line tools are secure and can be used to accomplish data transfer. You can run these commands directly from the terminal if your local system runs Linux or macOS.
Note: It is possible to use these command line tools if your local machine runs Windows, but you will need to use a ssh client (ex.
PuTTY
).
To simplify the data transfer process, it is recommended that Windows users follow the
How to Transfer Data with Cyberduck
guide as detailed below.
For users that are new to the command line, using either scp
or sftp
to transfer data is advised.
Prerequisites for Data Transfer with Command Line Tools
Before we begin, you will need to know:
- the path to your data file(s) on your local system
- the path to your transfer directory on the remote storage server
Determining the Path to Your Data File(s) on Your Local System
In order to transfer your project data, you will first need to know where the files are located on your local system.
To do so, navigate to the location of the files on your computer. This can be accomplished on a Mac by using the Finder application or on Windows with File Explorer application. Common locations for user data at the user's home directory, the Desktop and My Documents.
Once you have identified the location of the files, you can right-click on them and select either Get Info (on Mac) or Properties (on Windows) to view the path location on your local system.
Figure 1. Use Get Info to determine “Where” the path of your data file(s) is
For example, a file located in a folder named portal-data under Documents
would have the following path:
- On Mac
/Users/username/Documents/portal-data/my_file.txt
- On Windows
\Users\username\My Documents\portal-data\my_file.txt
Determining the Path to Your Transfer Directory
A transfer directory on the remote storage server associated with the portal you are accessing it through will be established when your account is given access to the portal and completes the on-boarding procedure. The transfer directory path will be unique for every institution and project.
Examples:
/corral-secure/projects/A2CPS/submissions/utaustin/
If you are unsure of your transfer directory path, please consult your project PI directly.
How to Transfer Data with scp
scp
copies files between hosts on a network. To transfer a file (ex. my_file.txt
) to the remote secure system via scp
, open a terminal on your local computer and navigate to the path where your data file is located.
- On Mac
localhost$ cd ~/Documents/portal-data/
- On Windows
localhost$ cd %HOMEPATH%\Documents\portal-data\
Assuming your TACC username is jdoe
and you are affiliated with UT Austin, a scp
transfer that pushes my_file.txt
from the current directory of your local computer to the remote secure system would look like this:
localhost$ scp ./my_file.txt jdoe@host:/transfer/directory/path
Note: This command will copy your data file directly to your individualized transfer directory on the remote storage system.
If you have not done so already, enter this command in your terminal,
replacing the file name, TACC username, and your individualized transfer directory path appropriately.
After entering the command, you will be prompted to login to the remote secure system by entering the password associated with your TACC account as well as the token value generated from your TACC token app.
A successful data transfer will generate terminal output similar to this:
my_file.txt 100% ## #.# KB/s ##:##
If you wish to learn more about scp
and how to synchronize your file transfer, you can do so
the online man
page for scp
or follow the file transfer section of the user guide for the appropriate TACC system:
How to Transfer Data with sftp
sftp
is a file transfer program that allows you to interactively navigate between your local file system and the remote secure system. To transfer a file (ex. my_file.txt
) to the remote secure system via sftp
, open a terminal on your local computer and navigate to the path where your data file is located.
- On Mac
localhost$ cd ~/Documents/portal-data/
- On Windows
localhost$ cd %HOMEPATH%\Documents\portal-data\
Assuming your TACC username is jdoe
and you are affiliated with UT Austin, an sftp
transfer that pushes
my_file.txt
from the current directory of your local computer to the remote secure system would look like this:
localhost$ sftp jdoe@host:/transfer/directory/path
Password:
TACC Token Code:
Connected to host.
Changing to:
/transfer/directory/path
sftp>
If you have not done so already, enter this command in your terminal,
replacing the TACC username and your individualized transfer directory path appropriately.
You are now logged into the remote secure system and have been redirected to your transfer directory. To confirm your location on the server, enter the following command:
sftp> pwd
Remote working directory:
/transfer/directory/path
To list the files currently in your transfer directory:
sftp> ls
utaustin_dir.txt
To list the files currently in your local directory:
sftp> lls
my_file.txt
Note: The leading l
in the lls
command denotes that you are listing the contents of your local working directory.
To transfer my_file.txt
from your local computer to your transfer directory:
sftp> put my_file.txt
Uploading my_file.txt to /transfer/directory/path
my_file.txt 100% ## #.# KB/s ##:#
To check if my_file.txt
is in the utaustin
subfolder:
sftp> ls
my_file.txt
utaustin_dir.txt
To exit out of sftp
on the terminal:
sftp> bye
localhost1$
If you wish to learn more about sftp
, you can do so
at the online man
page for scp
.
How to Transfer Data with rsync
rsync
is a file copying tool that can reduce the amount of data transferred by sending only the differences between the source files on your local system and the existing files in your transfer directory. To transfer a file (ex. my_file.txt
) to the remote secure system via rsync
, open a terminal on your local computer and navigate to the path where your data file is located.
- On Mac
localhost$ cd ~/Documents/portal-data/
- On Windows
localhost$ cd %HOMEPATH%\Documents\portal-data\
Assuming your TACC username is jdoe
and you are affiliated with UT Austin, an rsync
transfer that pushes
my_file.txt
from the current directory of your local computer to the remote secure system would look like this:
localhost$ rsync ./my_file.txt jdoe@host:/transfer/directory/path
If you have not done so already, enter this command in your terminal,
replacing the TACC username and your individualized transfer directory path appropriately.
If the command returns 0 in your terminal, the data transfer was successful.
If you wish to learn more about rsync
and how to synchronize your file transfer, you can do so
the online man
page for rsync
or follow the file transfer section of the user guide for the appropriate TACC system:
How to Transfer Data with Cyberduck
Cyberduck is a free graphical user interface for data transfer and is an alternative to using the command line. With a drag-and-drop interface, it is easy to transfer a file from your local system to the remote secure system. You can use
Cyberduck
for Windows or macOS.
For Windows
Download and install Cyberduck for Windows
on your local machine.
Once installed, click “Open Connection” in the top left corner of your Cyberduck window.
Figure 2. Windows Cyberduck and “Open Connection” setup screen
To setup a connection, type in the server name,
host. Add your TACC username and password in the spaces provided. If the “More Options” area is not shown, click the small triangle button to expand the window; this will allow you to enter the path to your transfer directory, /transfer/directory/path, so that when Cyberduck opens the connection you will immediately be in your individualized transfer directory on the system. Click the “Connect” button to open your connection.
Consult Figure 3. below to ensure the information you have provided is correct.
If you have not done so already, replace the “Path” with the path to your individualized transfer directory.
Figure 3. Windows “Open Connection” setup screen
Note: You will be prompted to “allow unknown fingerprint…” upon connection. Select “allow” and enter your TACC token value.
Once connected, you can navigate through your remote file hierarchy using the graphical user interface. You may also drag-and-drop files from your local computer into the Cyberduck window to transfer files to the system.
For Mac
Download and install Cyberduck for macOS
on your local machine.
Once installed, go to “Bookmark > New Bookmark” to setup a connection.
Note: You cannot select “Open Connection” in the top left corner of your Cyberduck window as macOS’ setup screen is missing the “More Options” button.
To setup a connection using “New Bookmark", type in the server name,
host. Add your TACC username and password in the spaces provided. If the “More Options” area is not shown, click the small triangle or button to expand the window; this will allow you to enter the path to your transfer directory, /transfer/directory/path, so that when Cyberduck opens the connection you will immediately be in your individualized transfer directory on the system. As you fill out the information, Cyberduck will create the bookmark for you. Exit out of the setup screen and click on your newly created bookmark to launch the connection.
Figure 4. macOS “New Bookmark” setup screen
Consult Figure 4. above to ensure the information you have provided is correct.
If you have not done so already, replace the “Path” with the path to your individualized transfer directory.
Note: You will be prompted to “allow unknown fingerprint…” upon connection. Select “allow” and enter your TACC token value.
Once connected, you can navigate through your remote file hierarchy using the graphical user interface. You may also drag-and-drop files from your local computer into the Cyberduck window to transfer files to the storage system.
References