Difference between revisions of "Swestore-iRODS-at-NSC"

From SNIC Documentation
Jump to: navigation, search
(Accessing iRODS from the compute cluster)
(Accessing iRODS from the compute cluster)
 
(5 intermediate revisions by the same user not shown)
Line 3: Line 3:
 
iRODS is a distributed storage management system which enables the user to
 
iRODS is a distributed storage management system which enables the user to
 
control the access to his/her data via policies which are expressed as rules.
 
control the access to his/her data via policies which are expressed as rules.
 +
Swestore iRODS is hosted on servers at PDC and NSC. We are running version 4.2.6.
 
Basic information about how to store and access data had been described above.
 
Basic information about how to store and access data had been described above.
 
Additional information about iRODS can be found at [http://www.irods.org the iRODS web site].
 
Additional information about iRODS can be found at [http://www.irods.org the iRODS web site].
Line 13: Line 14:
 
== Accessing iRODS from the compute cluster ==
 
== Accessing iRODS from the compute cluster ==
  
Data stored in iRODS is accessible from tetralith.
+
Data stored in iRODS is accessible from <code>tetralith</code>.
We have got the command line client installed on the cluster.
+
We have got the command line client installed on the cluster,
 +
iCommands should work from the login node and also from the batch
 +
nodes.
 +
 
 
You should have the client configuration files in place under
 
You should have the client configuration files in place under
the <code>.irods</code> directory as it has been described under
+
the <code>~/.irods</code> directory as it has been described under
 
the introductory pages.
 
the introductory pages.
 +
 +
Under your home directory you should create a directory called
 +
<code>.irods</code> which will contain the iRODS client configuration
 +
file. That file is <code>irods_environment.json</code>. It should
 +
look like:
 +
  {
 +
      "irods_host": "irods-login.swestore.se",
 +
      "irods_port": 2432,
 +
      "irods_default_resource": "pdc-gpfs",
 +
      "irods_home": "/snic.se/home/<YOUR_SNIC_USER_NAME>",
 +
      "irods_cwd": "/snic.se/home/<YOUR_SNIC_USER_NAME>",
 +
      "irods_user_name": "<YOUR_SNIC_USER_NAME>",
 +
      "irods_zone_name": "snic.se",
 +
      "irods_client_server_negotiation": "request_server_negotiation",
 +
      "irods_client_server_policy": "CS_NEG_REFUSE",
 +
      "irods_encryption_key_size": 32,
 +
      "irods_encryption_salt_size": 8,
 +
      "irods_encryption_num_hash_rounds": 16,
 +
      "irods_encryption_algorithm": "AES-256-CBC",
 +
      "irods_default_hash_scheme": "SHA256",
 +
      "irods_match_hash_policy": "compatible",
 +
      "irods_authentication_scheme": "PAM"
 +
}
 +
where the token <code><YOUR_SNIC_USER_NAME></code> should be replaced
 +
with your SNIC user name.
 +
 
Issue:
 
Issue:
 
   module load irods-client/4.2.6
 
   module load irods-client/4.2.6
after which you authenticate using <code>iinit</code> and then the iCommands like
+
after which you should authenticate using <code>iinit</code> and then the iCommands like
 
<code>iget</code>, <code>iput</code>, <code>ils</code> should work as expected.
 
<code>iget</code>, <code>iput</code>, <code>ils</code> should work as expected.
 +
The <code>ihelp</code> command will print a short description of the irods commands.
 +
 +
First issue
 +
  iinit
 +
which will ask for your iRODS password. You can specify a longer time limit
 +
for the session using the <code>--ttl</code> option, <code>iinit -h --ttl</code>
 +
will print help using this. The default generally will suffice.
 +
 +
Use
 +
  ils
 +
just to check if you can access your home directory. Please use the project
 +
directory to store files, which will be a subdirectory under
 +
<code>//snic.se/projects/<SNIC_PROJECT_NAME></code>.
 +
 +
To transfer data from the cluster directory it is a good idea to use
 +
<code>irsync</code> like:
 +
  irsync -v -r -K local_directory i:irods_directory
 +
where
 +
  -v for verbose.
 +
  -r for recursive copy of the whole directory tree.
 +
  -K to compute and check checksums. This involves higher CPU usage
 +
    but gives confidence for the integrity of the data stored.
 +
i.e.:
 +
  irsync -r -K mydir i:/snic.se/projects/projectdir
 +
If, for whatever reason the transfer had been interrupted the second
 +
invocation of the command with the same arguments should pick up and
 +
continue from that point.
 +
 +
Detailed description of the command line client can be found at
 +
[https://docs.irods.org/master/icommands/user/ documentation at the iRODS site].

Latest revision as of 09:41, 9 December 2020

Swestore iRODS at NSC

iRODS is a distributed storage management system which enables the user to control the access to his/her data via policies which are expressed as rules. Swestore iRODS is hosted on servers at PDC and NSC. We are running version 4.2.6. Basic information about how to store and access data had been described above. Additional information about iRODS can be found at the iRODS web site.

iRODS system at NSC

At NSC we host a replica metadata server and two storage server nodes at the moment. All data stored is replicated between PDC and NSC.

Accessing iRODS from the compute cluster

Data stored in iRODS is accessible from tetralith. We have got the command line client installed on the cluster, iCommands should work from the login node and also from the batch nodes.

You should have the client configuration files in place under the ~/.irods directory as it has been described under the introductory pages.

Under your home directory you should create a directory called .irods which will contain the iRODS client configuration file. That file is irods_environment.json. It should look like:

 {
     "irods_host": "irods-login.swestore.se",
     "irods_port": 2432,
     "irods_default_resource": "pdc-gpfs",
     "irods_home": "/snic.se/home/<YOUR_SNIC_USER_NAME>",
     "irods_cwd": "/snic.se/home/<YOUR_SNIC_USER_NAME>",
     "irods_user_name": "<YOUR_SNIC_USER_NAME>",
     "irods_zone_name": "snic.se",
     "irods_client_server_negotiation": "request_server_negotiation",
     "irods_client_server_policy": "CS_NEG_REFUSE",
     "irods_encryption_key_size": 32,
     "irods_encryption_salt_size": 8,
     "irods_encryption_num_hash_rounds": 16,
     "irods_encryption_algorithm": "AES-256-CBC",
     "irods_default_hash_scheme": "SHA256",
     "irods_match_hash_policy": "compatible",
     "irods_authentication_scheme": "PAM" 
}

where the token <YOUR_SNIC_USER_NAME> should be replaced with your SNIC user name.

Issue:

 module load irods-client/4.2.6

after which you should authenticate using iinit and then the iCommands like iget, iput, ils should work as expected. The ihelp command will print a short description of the irods commands.

First issue

 iinit

which will ask for your iRODS password. You can specify a longer time limit for the session using the --ttl option, iinit -h --ttl will print help using this. The default generally will suffice.

Use

 ils

just to check if you can access your home directory. Please use the project directory to store files, which will be a subdirectory under //snic.se/projects/<SNIC_PROJECT_NAME>.

To transfer data from the cluster directory it is a good idea to use irsync like:

 irsync -v -r -K local_directory i:irods_directory

where

 -v for verbose.
 -r for recursive copy of the whole directory tree.
 -K to compute and check checksums. This involves higher CPU usage
    but gives confidence for the integrity of the data stored.

i.e.:

 irsync -r -K mydir i:/snic.se/projects/projectdir

If, for whatever reason the transfer had been interrupted the second invocation of the command with the same arguments should pick up and continue from that point.

Detailed description of the command line client can be found at documentation at the iRODS site.