Difference between revisions of "Accessing Swestore with the ARC client"

From SNIC Documentation
Jump to: navigation, search
Line 5: Line 5:
 
[[Getting started with SweGrid|< Getting started with SweGrid]]<br>
 
[[Getting started with SweGrid|< Getting started with SweGrid]]<br>
 
[[Swestore National Storage|< Swestore National Storage]]
 
[[Swestore National Storage|< Swestore National Storage]]
= Introduction =
 
  
(UNDER DEVELOPMENT)
+
(PAGE UNDER DEVELOPMENT)
  
The SNIC national storage infrastructure is implemented using dCache, which is a distributed storage system presenting the user with a single filesystem view. The front-end host of the system is srm://srm.swegrid.se which is the machine presenting the filesystem for the user.
+
= The ARC client =
  
The system uses VOMS proxy certificates for authentication. Please see, [[Grid certificates#VOMS certificates|VOMS Certificates]] , for information on how to generate a voms proxy certificate.
+
This guide describes how to use the [http://www.nordugrid.org Nordugrid]
 +
''ARC'' client for storing and retrieving files from SweStore National
 +
Storage. The ARC client is usually used for sending grid jobs to grid
 +
clusters, but it also contain commands for data management.
 +
 
 +
If your system doesn't have the ARC client installed you can look at
 +
the [[ARC_client_installation|ARC client installation]] page in this wiki or the official
 +
Nordugrid [http://www.nordugrid.org/documents/arc-client-install.html ARC installation] page for more information. Please note that
 +
the windows and MacOS clients are a bit unstable.
 +
 
 +
A complete user guide for the ARC client can be found in
 +
http://www.nordugrid.org/documents/arc-ui.pdf.
 +
 
 +
== ARC 0.8 versus 1.0 ==
 +
 
 +
In late spring 2011 Nordugrid release the 1.0 version of ARC
 +
(sometimes called 11.05). One of the new features of 1.0 compared to
 +
the previous 0.8 release was a new command set. Basically most of the
 +
ng* commands was replaced with the new arc* commands. Some
 +
functionality moved between commands (ngstat became arcinfo and
 +
arcstat) and some new commands was introduced (arcproxy as an
 +
replacement for grid-proxy-init, which wasn't an arc command at all
 +
but a part of the Globus Toolkit). There are still legacy compat
 +
binaries in place for the old ng* commands, but I strongly suggest
 +
that you use arc* when available.
  
 
= Copying files =  
 
= Copying files =  
  
Copying files to and from resources is accomplished using the '''arccp''' command. The command can handle many of existing protocols, such as: ''http, https, httpg, ftp, gsiftp, lfc, rls, srm''.  
+
Copying files to and from resources is accomplished using the
 +
'''arccp''' command. The command can handle many  
 +
protocols, such as: ''http, https, httpg, ftp, gsiftp, lfc, rls,
 +
srm''. Only the srm protocol is supported when accessing SweStore using the ARC client.
  
 
== Copying single files ==
 
== Copying single files ==
Line 21: Line 47:
 
Copying single files is accomplished in the same way as using the normal '''cp''' command as shown in the following example:
 
Copying single files is accomplished in the same way as using the normal '''cp''' command as shown in the following example:
  
<pre>[jonas@localhost jonas2]$ arccp archive.tar.gz srm://srm.swegrid.se/ops/</pre>
+
[jonas@localhost]$ arccp archive.tar.gz srm://srm.swegrid.se/ops/
  
 
== Recursive copying ==
 
== Recursive copying ==
Line 27: Line 53:
 
Recursive copying is accomplished using the '''--recursive''' option. The argument to the option determines the depth of the recursive copy.
 
Recursive copying is accomplished using the '''--recursive''' option. The argument to the option determines the depth of the recursive copy.
  
<pre>
+
[jonas@localhost$ arccp --recursive=3 jonas2/ srm://srm.swegrid.se/ops/jonas/jonas2/
[jonas@localhost swestore]$ arccp --recursive=3 jonas2/ srm://srm.swegrid.se/ops/jonas/jonas2/
 
</pre>
 
  
 
= Listing files =  
 
= Listing files =  
Line 35: Line 59:
 
Listing files on a resources is done using the '''arcls''' command. In the simplest form the command just takes a URL as input and displays names and directories without any extra information as shown in the following example:
 
Listing files on a resources is done using the '''arcls''' command. In the simplest form the command just takes a URL as input and displays names and directories without any extra information as shown in the following example:
  
<pre>
+
[jonas@localhost]$ arcls srm://srm.swegrid.se/ops/
[jonas@localhost jonas2]$ arcls srm://srm.swegrid.se/ops/
+
motd.1
motd.1
+
bla
bla
+
generated
generated
+
ops
ops
+
motd.f343
motd.f343
+
GridFTP-1.0.20-linux.gtk.x86.zip
GridFTP-1.0.20-linux.gtk.x86.zip
 
test23
 
test24
 
ronstestfile
 
ronstestfile3
 
testfile-put-1296854868-6468a2d5df07.txt
 
</pre>
 
  
 
Additional information can be listed by adding the '''--long''' option.
 
Additional information can be listed by adding the '''--long''' option.
  
<pre>
+
[jonas@localhost]$ arcls --long srm://srm.swegrid.se/ops
[jonas@localhost swestore]$ arcls --long srm://srm.swegrid.se/ops
+
<Name> <Type> <Size> <Creation> <Validity> <CheckSum> <Latency>
<Name> <Type> <Size> <Creation> <Validity> <CheckSum> <Latency>
+
motd.1 file 354 2008-06-05 12:28:23 (n/a) adler32:582d7718 NEARLINE
motd.1 file 354 2008-06-05 12:28:23 (n/a) adler32:582d7718 NEARLINE
+
bla dir 512 2008-08-22 12:23:49 (n/a) (n/a) NEARLINE
bla dir 512 2008-08-22 12:23:49 (n/a) (n/a) NEARLINE
+
generated dir 512 2008-11-17 15:07:39 (n/a) (n/a) NEARLINE
generated dir 512 2008-11-17 15:07:39 (n/a) (n/a) NEARLINE
+
ops dir 512 2010-01-21 11:26:00 (n/a) (n/a) NEARLINE
ops dir 512 2010-01-21 11:26:00 (n/a) (n/a) NEARLINE
+
motd.f343 file 436 2010-01-08 14:35:40 (n/a) adler32:0fed94f2 ONLINE
motd.f343 file 436 2010-01-08 14:35:40 (n/a) adler32:0fed94f2 ONLINE
+
GridFTP-1.0.20-linux.gtk.x86.zip file 30479220 2011-03-09 09:49:20 (n/a) adler32:5c67686b ONLINE
GridFTP-1.0.20-linux.gtk.x86.zip file 30479220 2011-03-09 09:49:20 (n/a) adler32:5c67686b ONLINE
 
test23 file 0 2011-03-04 09:44:44 (n/a) adler32:00000001 ONLINE
 
test24 file 0 2011-03-04 09:45:07 (n/a) adler32:00000001 ONLINE
 
ronstestfile file 4 2010-11-23 17:46:13 (n/a) adler32:03cd014b ONLINE
 
ronstestfile3 file 4 2010-11-23 17:49:07 (n/a) adler32:03cd014b ONLINE
 
testfile-put-1296854868-6468a2d5df07.txt file 20 2011-02-04 22:28:47 (n/a) adler32:1a400272 ONLINE
 
</pre>
 
  
Metadatainformation on a specific file can be listed by specifying the '''-m''' or '''--metadata''' option. In the following example the metadata information of the '''test23''' file is shown:
+
Metadatainformation on a specific file can be listed by specifying the '''-m''' or '''--metadata''' option. In the following example the metadata information of the '''motd.1''' file is shown:
  
<pre>
+
[jonas@localhost]$ arcls --metadata srm://srm.swegrid.se/ops/motd.1
[jonas@localhost swestore]$ arcls --metadata srm://srm.swegrid.se/ops/test23
+
/ops/motd.1
/ops/test23
+
accessperm:rw-r--r--
accessperm:rw-r-----
+
checksum:adler32:582d7718
checksum:adler32:00000001
+
ctime:2008-06-05 12:28:23
ctime:2011-03-04 09:44:44
+
filestoragetype:PERMANENT
filestoragetype:PERMANENT
+
group:25001
group:25001
+
latency:NEARLINE
latency:ONLINE
+
lifetimeassigned:PT1S
lifetimeassigned:PT1S
+
lifetimeleft:PT1S
lifetimeleft:PT1S
+
mtime:2011-03-24 12:57:42
mtime:2011-03-04 09:44:45
+
owner:25001
owner:25001
+
path:/ops/motd.1
path:/ops/test23
+
size:354
size:0
+
spacetokens:
spacetokens:
+
type:file
type:file
 
</pre>
 
  
 
= Creating directories =  
 
= Creating directories =  
  
There are no command line tool yet for creating directories. This command will probably be added in a upcoming release. In the meantime the following procedure can be used:  
+
There is no command line tool for creating directories. This command will probably be added in a upcoming release. All required directories are created when needed. In the meantime the following procedure can be used:  
 
 
<pre>
 
[jonas@localhost swestore]$ mkdir mydir
 
[jonas@localhost swestore]$ touch mydir/dummyfile
 
[jonas@localhost swestore]$ arccp ./mydir/ srm://srm.swegrid.se/ops/mydir/
 
[jonas@localhost swestore]$ arcls srm://srm.swegrid.se/ops/mydir
 
dummyfile
 
</pre>
 
 
 
The dummmy file can be removed using the arcrm command described in the following sections.
 
  
= Removing files =
+
[jonas@localhost]$ mkdir mydir
 +
[jonas@localhost]$ touch mydir/dummyfile
 +
[jonas@localhost]$ arccp ./mydir/ srm://srm.swegrid.se/ops/mydir/
 +
[jonas@localhost]$ arcls srm://srm.swegrid.se/ops/mydir
 +
dummyfile
  
<pre>[jonas@localhost swestore]$ arcrm srm://srm.swegrid.se/ops/mydir/dummyfile</pre>
+
The dummmy file can be removed using the arcrm command described in the following section.
  
= Removing directories =  
+
= Removing files or directories =
  
To remove directories they have to be empty. To indicate a directory the URL should end with a '''/'''.
+
[jonas@localhost]$ arcrm srm://srm.swegrid.se/ops/mydir/dummyfile
 +
[jonas@localhost]$ arcrm srm://srm.swegrid.se/ops/mydir
  
<pre>[jonas@localhost swestore]$ arcrm srm://srm.swegrid.se/ops/mydir/</pre>
+
To remove directories they have to be empty.

Revision as of 13:51, 10 November 2011

< Getting started with SweGrid
< Swestore National Storage

(PAGE UNDER DEVELOPMENT)

The ARC client

This guide describes how to use the Nordugrid ARC client for storing and retrieving files from SweStore National Storage. The ARC client is usually used for sending grid jobs to grid clusters, but it also contain commands for data management.

If your system doesn't have the ARC client installed you can look at the ARC client installation page in this wiki or the official Nordugrid ARC installation page for more information. Please note that the windows and MacOS clients are a bit unstable.

A complete user guide for the ARC client can be found in http://www.nordugrid.org/documents/arc-ui.pdf.

ARC 0.8 versus 1.0

In late spring 2011 Nordugrid release the 1.0 version of ARC (sometimes called 11.05). One of the new features of 1.0 compared to the previous 0.8 release was a new command set. Basically most of the ng* commands was replaced with the new arc* commands. Some functionality moved between commands (ngstat became arcinfo and arcstat) and some new commands was introduced (arcproxy as an replacement for grid-proxy-init, which wasn't an arc command at all but a part of the Globus Toolkit). There are still legacy compat binaries in place for the old ng* commands, but I strongly suggest that you use arc* when available.

Copying files

Copying files to and from resources is accomplished using the arccp command. The command can handle many protocols, such as: http, https, httpg, ftp, gsiftp, lfc, rls, srm. Only the srm protocol is supported when accessing SweStore using the ARC client.

Copying single files

Copying single files is accomplished in the same way as using the normal cp command as shown in the following example:

[jonas@localhost]$ arccp archive.tar.gz srm://srm.swegrid.se/ops/

Recursive copying

Recursive copying is accomplished using the --recursive option. The argument to the option determines the depth of the recursive copy.

[jonas@localhost$ arccp --recursive=3 jonas2/ srm://srm.swegrid.se/ops/jonas/jonas2/

Listing files

Listing files on a resources is done using the arcls command. In the simplest form the command just takes a URL as input and displays names and directories without any extra information as shown in the following example:

[jonas@localhost]$ arcls srm://srm.swegrid.se/ops/
motd.1
bla
generated
ops
motd.f343
GridFTP-1.0.20-linux.gtk.x86.zip

Additional information can be listed by adding the --long option.

[jonas@localhost]$ arcls --long srm://srm.swegrid.se/ops
<Name> <Type> <Size> <Creation> <Validity> <CheckSum> <Latency>
motd.1 file 354 2008-06-05 12:28:23 (n/a) adler32:582d7718 NEARLINE
bla dir 512 2008-08-22 12:23:49 (n/a) (n/a) NEARLINE
generated dir 512 2008-11-17 15:07:39 (n/a) (n/a) NEARLINE
ops dir 512 2010-01-21 11:26:00 (n/a) (n/a) NEARLINE
motd.f343 file 436 2010-01-08 14:35:40 (n/a) adler32:0fed94f2 ONLINE
GridFTP-1.0.20-linux.gtk.x86.zip file 30479220 2011-03-09 09:49:20 (n/a) adler32:5c67686b ONLINE

Metadatainformation on a specific file can be listed by specifying the -m or --metadata option. In the following example the metadata information of the motd.1 file is shown:

[jonas@localhost]$ arcls --metadata srm://srm.swegrid.se/ops/motd.1
/ops/motd.1
accessperm:rw-r--r--
checksum:adler32:582d7718
ctime:2008-06-05 12:28:23
filestoragetype:PERMANENT
group:25001
latency:NEARLINE
lifetimeassigned:PT1S
lifetimeleft:PT1S
mtime:2011-03-24 12:57:42
owner:25001
path:/ops/motd.1
size:354
spacetokens:
type:file

Creating directories

There is no command line tool for creating directories. This command will probably be added in a upcoming release. All required directories are created when needed. In the meantime the following procedure can be used:

[jonas@localhost]$ mkdir mydir
[jonas@localhost]$ touch mydir/dummyfile
[jonas@localhost]$ arccp ./mydir/ srm://srm.swegrid.se/ops/mydir/
[jonas@localhost]$ arcls srm://srm.swegrid.se/ops/mydir
dummyfile

The dummmy file can be removed using the arcrm command described in the following section.

Removing files or directories

[jonas@localhost]$ arcrm srm://srm.swegrid.se/ops/mydir/dummyfile
[jonas@localhost]$ arcrm srm://srm.swegrid.se/ops/mydir

To remove directories they have to be empty.