3DEM Data Transfer and Management Guide

 

Outline

Data Transfer and Management Guide

Transferring 3DEM project data from your local machine to TACC’s Corral system can be accomplished using two methods: command line tools (scp, sftp, rsync) and graphical tools (Cyberduck).

Data Transfer Methods

 

Usage Mode

Transfer Method
Command Line Tool scp
Command Line Tool sftp
Command Line Tool rsync

Graphical Tool
Cyberduck

 

What is Corral?

Corral is one of TACC’s systems dedicated to storage and data management. Corral can be used for storing project data. As a 3DEM project user, you have access to the Corral server “data.tacc.utexas.edu”, as well as the  Corral folder /corral-repl/projects/NeuroNex-3DEM/projects/. The Corral system is to be used exclusively for transferring and accessing 3DEM project data.

 

Prerequisites for 3DEM Project User

There are two requirements needed prior to transferring data:

  • A TACC Account
  • Multi-Factor Authentication (MFA) pairing with the TACC Token app

As a 3DEM project user, you should already have access to a TACC account through the TACC Portal. If you are unsure of your TACC account username, please consult the “TACC Account Request Confirmation” email you have received from TACC Accounting Staff, or contact acantu@tacc.utexas.edu.

 

TACC now requires MFA pairing. To set up MFA, you can follow this tutorial.

 

Using Command Line Tools to Transfer and Organize Data

A common method of transferring files between TACC resources and/or your local machine is through the command line.

 

scp, sftp, & rsync

These three command line tools are secure and can be used to accomplish data transfer. You can run these commands directly from the terminal if your local system runs Linux or macOS.

 

Note: It is possible to use these command line tools if your local machine runs Windows, but you will need to use a ssh client (ex. PuTTY). However, for the scope of this project and ease in data transfer, it is recommended that Windows users follow our Transfer Data with Cyberduck guide below.

 

 

For users that are new to the command line, using either scp or sftp to transfer data is advised.

 

Prerequisites for Data Transfer with Command Line Tools

Before we begin, you will need to know:

  • the path to your data file(s) on your local system
  • the path to your transfer directory on Corral
     

Determining the Path to Your Data File(s) on Your Local System

In order to transfer your project data, you will need to know where your file(s) is located on your local system. To do so, navigate through the file system on your local computer.

 

For example, you can do so on Mac by going to Finder. Common data file locations such as Desktop and Documents should be accessible through the Favorites pane on the left-hand side of the window. You can also do Finder > Go to find other common locations on your computer.

 

Once you have navigated to where your file(s) is located, take note of its path by right-clicking on the file and selecting Get Info.

 

Figure 1. Use Get Info to determine “Where” the path of your data file(s) is
 

For example, a file located in a folder named project-data under the directory Documents would have the following path on a mac:

 

/Users/$Username/Documents/project-data/my_file.txt

 

Determining the Path to Your Transfer Directory on Corral

A transfer directory on Corral has already been set up for you and is available for data transfer. The transfer directory path is individualized to your portal projects.

 

For example, for a user that has created a project under the 3DEM directory, the path to their transfer directory on Corral would be:

 

/corral-repl/projects/NeuroNex-3DEM/projects/$PROJECT_ID

 

Note: The transfer directory name will always be named the same as the `Project Id` value displayed next to the project folder listed under the 3DEM.org > Workbench > Data Files > My Projects UI.

 

If you are unsure of your transfer directory path, please consult the email you have received from jcarson@tacc.utexas.edu, or contact jcarson@tacc.utexas.edu directly.

 

Transfer Data with scp

scp copies files between hosts on a network. To transfer a file (ex. my_file.txt) to Corral via scp, open a terminal on your local computer and navigate to the path where your data file is located.

 

localhost$ cd Documents/project-data/

 

 

Assuming your TACC username is jdoe and you are affiliated with UT Austin, a scp transfer that pushes my_file.txt from your local computer to Corral’s secure system would look like this:

 

localhost$ scp ./my_file.txt jdoe@data.tacc.utexas.edu:/corral-repl/projects/NeuroNex-3DEM/projects/$PROJECT_ID/

 

Note: This command will copy your data file directly to your individualized project directory on Corral’s system.

 

If you have not done so already, enter this command in your terminal, replacing the file name, TACC username, and your individualized project directory ID appropriately.

 

After entering the command, you will be prompted to login to Corral’s secure system by entering the password associated with your TACC account as well as the token value generated from your TACC token app.

 

The data transfer was successful if your terminal output looks like this:

 

my_file.txt     100% ##  #.# KB/s  ##:##

 

 

If you wish to learn more about scp, you can do so here.

 

Transfer Data with sftp

sftp is a file transfer program that allows you to interactively navigate between your local file system and the remote Corral system. To transfer a file (ex. my_file.txt) to Corral via sftp, open a terminal on your local computer and navigate to the path where your data file is located. 

 

localhost$ cd Documents/project-data/

 

 

Assuming your TACC username is jdoe and you are affiliated with UT Austin, a sftp transfer that pushes my_file.txt from your local computer to Corral would look like this:

 

localhost$ sftp jdoe@data.tacc.utexas.edu:/corral-repl/projects/NeuroNex-3DEM/projects/$PROJECT_ID

Password:

TACC Token Code:

Connected to data.tacc.utexas.edu.

Changing to: /corral-repl/projects/NeuroNex-3DEM/projects/$PROJECT_ID

sftp>


 

If you have not done so already, enter this command in your terminal, replacing the TACC username and your individualized project directory name appropriately.

At this point, you have logged into Corral and been redirected to your transfer directory. To confirm your location on Corral, enter the following command:

 

sftp> pwd
Remote working directory: /corral-repl/projects/NeuroNex-3DEM/projects/$PROJECT_ID

 

 

To list the files currently in your transfer directory:

 

ftp> ls

utaustin_dir.txt

 

 

To list the files currently in your local directory (ex. /Users/myname/Documents/3dem-data/):

 

sftp> lls

my_file.txt

 

Note: The leading l in the lls command denotes that you are listing the contents of your local working directory.

 

To transfer my_file.txt from your local computer to /corral-repl/projects/NeuroNex-3DEM/projects/$PROJECT_ID:

 

sftp> put my_file.txt

Uploading my_file.txt to /corral-repl/projects/NeuroNex-3DEM/projects/$PROJECT_ID/my_file.txt

my_file.txt     100% ##  #.#KB/s   ##:##

 

To check if my_file.txt is in the utaustin subfolder:

 

sftp> ls

my_file.txt

utaustin_dir.txt

 

To exit out of sftp on the terminal:

 

ftp> bye

localhost1$

 

If you wish to learn more about sftp, you can do so here.

 

Transfer Data with rsync

rsync is a file copying tool that can reduce the amount of data transferred by sending only the differences between the source files on your local system and the existing files in your transfer directory. To transfer a file (ex. my_file.txt) to Corral via rsync, open a terminal on your local computer and navigate to the path where your data file is located.  

 

localhost$ cd Documents/project-data/

 

Assuming your TACC username is jdoe and you are affiliated with UT Austin, a rsync transfer that pushes my_file.txt from your local computer to Corral would look like this:

 

localhost$ rsync ./my_file.txt jdoe@data.tacc.utexas.edu:/corral-repl/projects/NeuroNex-3DEM/projects/$PROJECT_ID/

 

If you have not done so already, enter this command in your terminal, replacing the TACC username and your individualized project directory name appropriately.

 

If the command returns 0 in your terminal, the data transfer was successful.

 

If you wish to learn more about rsync and how to synchronize your file transfer, you can do so here or follow the “Transferring files using scp and rsync” section of our Stampede2 guide.

 

Transfer Data with Cyberduck

Cyberduck is a free graphical tool for data transfer and is an alternative to using the command line. With a drag-and-drop interface, it is easy to transfer a file from your local system to Corral’s secure system. You can use Cyberduck for Windows or macOS.

 

For Windows

Download and install Cyberduck for Windows on your local machine.

Once installed, click “Open Connection” in the top left corner of your Cyberduck window.

Figure 2. Windows Cyberduck and “Open Connection” setup screen

 

To setup a connection, type in the server name data.tacc.utexas.edu. Add your TACC username and password in the spaces provided. If the "More Options" area is not shown, click the small triangle button to expand the window; this will allow you to enter the path to your transfer directory (ex. /corral-repl/projects/NeuroNex-3DEM/projects/$PROJECT_ID) so that when Cyberduck opens the connection you will immediately be in your individualized project directory on Corral. Click the "Connect" button to open your connection.

Consult Figure 3. below to ensure the information you have provided is correct. If you have not done so already, replace the “Path” with the path to your individualized transfer directory.

 

Figure 3. Windows “Open Connection” setup screen

 

Note: You will be prompted to “allow unknown fingerprint…” upon connection. Select “allow” and enter your TACC token value.

 

Once connected, you can navigate through your remote file hierarchy using familiar graphical navigation techniques. You may also drag-and-drop files from your local computer into the Cyberduck window to transfer files to Corral.

 

For Mac

Download and install Cyberduck for macOS on your local machine.

Once installed, go to Bookmark > New Bookmark to setup a connection.

 

Note: You cannot select “Open Connection” in the top left corner of your Cyberduck window as macOS’ setup screen is missing the “More Options” button.

 

To setup a connection using “New Bookmark", type in the server name data.tacc.utexas.edu. Add your TACC username and password in the spaces provided. If the "More Options" area is not shown, click the small triangle or button to expand the window; this will allow you to enter the path to your transfer directory (ex. /corral-repl/projects/NeuroNex-3DEM/projects/$PROJECT_ID) so that when Cyberduck opens the connection you will immediately be in your individualized transfer directory on Corral. As you fill out the information, Cyberduck will be creating the bookmark for you. Exit out of the setup screen and click on your newly created bookmark to launch the connection.

 

Figure 4. macOS “New Bookmark” setup screen

Consult Figure 4. above to ensure the information you have provided is correct. If you have not done so already, replace the “Path” with the path to your individualized transfer directory.

 

Note: You will be prompted to “allow unknown fingerprint…” upon connection. Select “allow” and enter your TACC token value.

 

Once connected, you can navigate through your remote file hierarchy using familiar graphical navigation techniques. You may also drag-and-drop files from your local computer into the Cyberduck window to transfer files to Corral.

References