Storage and Data Transfer

Different Storage Types in DiCOS

DiCOS Storages

Usage

File system

User Access

Note

Data Ensurance

DiCOS UI Home

NFS

Direct Access

Space for the home directories of DiCOS users

No

Group working directory

Ceph

Direct Access

Space for the working directories of DiCOS groups

Yes

Cloud Storage (Ceph)

Ceph provides capability with more intensive I/O operations and loading sharing with multiple machines. We provide Ceph storage for data preservation and jobs’ working directories.

Important

To avoid the burden of the MDS server of ceph, please avoid putting too many files in the same directory. Please make the count of your files in the same directory below 106. Thank you for your cooperation.

See also

User Spaces in DiCOS System

DiCOS UI Home (NFS)

It’s the home directory in slurm_ui and dicos-sftp. Your home path will be speicfied as your account name, like /dicos_ui_home/{your_dicos_account}. It’s your temporary data store of limited performance with NFS mount. We will clear dicos_ui_home periodically. Please use ceph as your job working path & backup your long-term data preservation (Please see detail in ceph section).

Important

  • /dicos_ui_home/ is a temporarily space. Please backup your data in the ceph directory or transfer to your own data storage.

Group Spaces

Group Directory (Ceph)

/ceph/work/{group_name} is the group space shared among group members for the working directory of jobs and data preservation for collaborations. For customized group storage pool, please contact DiCOS-Support@twgrid.org for details.

Data Transfer Data Endpoints

You can transfer data either through terminal commands or SFTP sofware through the following endpoints. 1. slurm-ui(asiop).twgrid.org 2. dicos-sftp.twgrid.org

scp (command line)

You can transfer data from the above endpoint by scp command in terminal.

For example:

# Copy my_result.txt from the DiCOS UI home to local directory
scp jack@dicos-sftp.grid.sinica.edu.tw:my_result.txt /home/jack

Note

For the help of scp, you could type the following command for the manual page:

man scp

FileZilla™ (Windows™/MacOS™/Linux)

FileZilla™ have file manager like UI. Connect to our UI with your DiCOS account, and the left panel will show the files in the UI, then you could go ahead with the download/upload operations.

See also

MobaXterm™ (Windows™)

MobaXterm™ has a built-in function for SFTP. Connect to our UI with your DiCOS account, and the left panel will show the files in the UI, then you could go ahead with the download/upload operations.

See also

WinSCP™ (Windows™)

WinSCP™ have file manager like UI. You could install one of them and login with SFTP/SCP protocol to our user interface, then you could drag to copy your files/directories between the UI and your computer.

See also