Difference between revisions of "Swestore-iRODS-at-NSC"

From SNIC Documentation
Jump to: navigation, search
(Accessing iRODS from the compute cluster)
 
Line 20: Line 20:
  
 
You should have the client configuration files in place under
 
You should have the client configuration files in place under
the <code>.irods</code> directory as it has been described under
+
the <code>~/.irods</code> directory as it has been described under
 
the introductory pages.
 
the introductory pages.
  
 
Under your home directory you should create a directory called
 
Under your home directory you should create a directory called
<code>.irods</code> which contains the iRODS client configuration
+
<code>.irods</code> which will contain the iRODS client configuration
 
file. That file is <code>irods_environment.json</code>. It should
 
file. That file is <code>irods_environment.json</code>. It should
 
look like:
 
look like:
Line 68: Line 68:
 
To transfer data from the cluster directory it is a good idea to use
 
To transfer data from the cluster directory it is a good idea to use
 
<code>irsync</code> like:
 
<code>irsync</code> like:
   irsync -r -K local_directory i:irods_directory
+
   irsync -v -r -K local_directory i:irods_directory
 
where
 
where
   -r for recursive copy of the whole directory tree
+
  -v for verbose.
   -K to compute and check checksums
+
   -r for recursive copy of the whole directory tree.
 +
   -K to compute and check checksums. This involves higher CPU usage
 +
    but gives confidence for the integrity of the data stored.
 
i.e.:
 
i.e.:
 
   irsync -r -K mydir i:/snic.se/projects/projectdir
 
   irsync -r -K mydir i:/snic.se/projects/projectdir

Latest revision as of 09:41, 9 December 2020

Swestore iRODS at NSC

iRODS is a distributed storage management system which enables the user to control the access to his/her data via policies which are expressed as rules. Swestore iRODS is hosted on servers at PDC and NSC. We are running version 4.2.6. Basic information about how to store and access data had been described above. Additional information about iRODS can be found at the iRODS web site.

iRODS system at NSC

At NSC we host a replica metadata server and two storage server nodes at the moment. All data stored is replicated between PDC and NSC.

Accessing iRODS from the compute cluster

Data stored in iRODS is accessible from tetralith. We have got the command line client installed on the cluster, iCommands should work from the login node and also from the batch nodes.

You should have the client configuration files in place under the ~/.irods directory as it has been described under the introductory pages.

Under your home directory you should create a directory called .irods which will contain the iRODS client configuration file. That file is irods_environment.json. It should look like:

 {
     "irods_host": "irods-login.swestore.se",
     "irods_port": 2432,
     "irods_default_resource": "pdc-gpfs",
     "irods_home": "/snic.se/home/<YOUR_SNIC_USER_NAME>",
     "irods_cwd": "/snic.se/home/<YOUR_SNIC_USER_NAME>",
     "irods_user_name": "<YOUR_SNIC_USER_NAME>",
     "irods_zone_name": "snic.se",
     "irods_client_server_negotiation": "request_server_negotiation",
     "irods_client_server_policy": "CS_NEG_REFUSE",
     "irods_encryption_key_size": 32,
     "irods_encryption_salt_size": 8,
     "irods_encryption_num_hash_rounds": 16,
     "irods_encryption_algorithm": "AES-256-CBC",
     "irods_default_hash_scheme": "SHA256",
     "irods_match_hash_policy": "compatible",
     "irods_authentication_scheme": "PAM" 
}

where the token <YOUR_SNIC_USER_NAME> should be replaced with your SNIC user name.

Issue:

 module load irods-client/4.2.6

after which you should authenticate using iinit and then the iCommands like iget, iput, ils should work as expected. The ihelp command will print a short description of the irods commands.

First issue

 iinit

which will ask for your iRODS password. You can specify a longer time limit for the session using the --ttl option, iinit -h --ttl will print help using this. The default generally will suffice.

Use

 ils

just to check if you can access your home directory. Please use the project directory to store files, which will be a subdirectory under //snic.se/projects/<SNIC_PROJECT_NAME>.

To transfer data from the cluster directory it is a good idea to use irsync like:

 irsync -v -r -K local_directory i:irods_directory

where

 -v for verbose.
 -r for recursive copy of the whole directory tree.
 -K to compute and check checksums. This involves higher CPU usage
    but gives confidence for the integrity of the data stored.

i.e.:

 irsync -r -K mydir i:/snic.se/projects/projectdir

If, for whatever reason the transfer had been interrupted the second invocation of the command with the same arguments should pick up and continue from that point.

Detailed description of the command line client can be found at documentation at the iRODS site.