Difference between revisions of "Accessing Swestore with rclone"

From SNIC Documentation
Jump to: navigation, search
(Requirements)
(FAQ - cert and login failed)
(7 intermediate revisions by the same user not shown)
Line 9: Line 9:
  
 
= Requirements =
 
= Requirements =
 +
 +
SNIC resources should already have rclone installed. To install rclone on your own computer, please follow your systems instructions at the official rclone documentation pages [https://rclone.org/install/ here].
 +
 
To access Swestore using rclone you need to be a member of a Swestore storage project, see [[Swestore#Getting access to Swestore]].
 
To access Swestore using rclone you need to be a member of a Swestore storage project, see [[Swestore#Getting access to Swestore]].
  
You also need to know your Username and password, see [[Setting your Swestore password]].
+
Depending on your preference and security needs, you can use rclone with either ''certificate'' or ''username/password'' authentication.
  
To install rclone on your own computer, please follow your systems instructions at the official rclone documentation pages [https://rclone.org/install/ here].
+
== Certificate authentication ==
 +
 
 +
This method requires you to obtain a security certificate and install it. While more cumbersome to set up, it provides a higher level of security.
 +
 
 +
We recommend this method when logged in to SNIC resources.
 +
 
 +
To use this authentication method you need to do the Certificate setup described in [[Accessing Swestore with the ARC client#Requirements]]
 +
 
 +
== Username/Password ==
 +
 
 +
This method will require you to enter your username/password in the rclone configuration file. While this is convenient, it is also a security risk since a malicious user gaining access to the rclone configuration file or your login on the affected resource will also gain access to your Swestore dCache storage.
 +
 
 +
To use this authentication method you need to know your Username and password, see [[Setting your Swestore password]].
  
 
= Quickstart =
 
= Quickstart =
Line 34: Line 49:
 
::: <code>rclone rmdir</code> - removing path if empty. Works similarly to <code>rm -d</code>. See [https://rclone.org/commands/rclone_rmdir/ Rclone rmdir].
 
::: <code>rclone rmdir</code> - removing path if empty. Works similarly to <code>rm -d</code>. See [https://rclone.org/commands/rclone_rmdir/ Rclone rmdir].
 
::: <code>rclone rmdirs</code> - removing all empty directories under path. Works similarly to <code>find path -type d | awk '{ print length, $0 }' | sort -nsr | cut -d" " -f2- | xargs rm -d</code> (example modified from [https://stackoverflow.com/a/5917762 stackoverflow]). See [https://rclone.org/commands/rclone_rmdirs/ Rclone rmdirs].
 
::: <code>rclone rmdirs</code> - removing all empty directories under path. Works similarly to <code>find path -type d | awk '{ print length, $0 }' | sort -nsr | cut -d" " -f2- | xargs rm -d</code> (example modified from [https://stackoverflow.com/a/5917762 stackoverflow]). See [https://rclone.org/commands/rclone_rmdirs/ Rclone rmdirs].
::: <code>rclone purge</code> - removing all data under path, including path - i.e. '''you loose your data'''. Works similarly to <code>rm -rf</code>. See [https://rclone.org/commands/rclone_purge/ Rclone purge].
+
::: <code>rclone purge</code> - '''be very careful using this''' since it will remove all data under path, including path - i.e. '''you loose your data'''. Works similarly to <code>rm -rf</code>. See [https://rclone.org/commands/rclone_purge/ Rclone purge].
  
 
Use <code>man</code> and <code>--help</code> to get more info on rclone and its commands. Examples: <code>man rclone</code>, <code>rclone --help</code> or <code>rclone copy --help</code> .
 
Use <code>man</code> and <code>--help</code> to get more info on rclone and its commands. Examples: <code>man rclone</code>, <code>rclone --help</code> or <code>rclone copy --help</code> .
Line 43: Line 58:
 
= Configuration =
 
= Configuration =
  
You have to configure rclone with your access protocol, URL and user login credentials for it to work. Simply issue the following configuration command to interactively configure rclone:
+
You have to configure rclone with your access protocol, URL and possibly user login credentials for it to work with Swestore dCache.
 +
 
 +
Issue the following configuration command to interactively configure rclone:
  
 
  rclone config
 
  rclone config
  
For configuring Swestore dCache access using WebDAV:
+
Then choose between the two configuration alternatives below, depending on you authentication preference.
 +
 
 +
For configuring Swestore dCache access using WebDAV and ''Certificate'' authentication:
 +
 
 +
: <code>n</code> for '''New remote'''
 +
: <code>swestore</code> for '''name'''
 +
: <code>webdav</code> for '''Storage''' (or the corresponding number in the list)
 +
: <code><nowiki>https://webdav.swestore.se</nowiki></code> for '''url'''
 +
: <code>other</code> for '''vendor''' (or the corresponding number in the list)
 +
: Just press <code><nowiki><Enter></nowiki></code> for '''user''' (ie. leave blank)
 +
: <code>n</code> for '''No leave this optional password blank'''
 +
: Just press <code><nowiki><Enter></nowiki></code> for '''bearer_token'''
 +
: <code>n</code> for '''Edit advanced config?'''
 +
: <code>y</code> if you think the resulting config is correct, otherwise <code>e</code> to edit again.
 +
: <code>q</code> to '''Quit config'''
 +
 
 +
Alternatively, for configuring Swestore dCache access using WebDAV and ''Username/Password'' authentication:
  
 
: <code>n</code> for '''New remote'''
 
: <code>n</code> for '''New remote'''
Line 71: Line 104:
  
 
In the following sections, we are assuming your swestore remote is named <code>swestore</code>
 
In the following sections, we are assuming your swestore remote is named <code>swestore</code>
 +
 +
== Additional steps for Certificate authentication ==
 +
 +
Unfortunately rclone doesn't (yet) support adding the needed client certificate paths to the configuration, requiring the following invocation for all rclone commands:
 +
rclone --client-cert=/tmp/x509up_u${UID} --client-key=/tmp/x509up_u${UID}
 +
 +
Since this is obviously very cumbersome, we strongly suggest adding an alias for it. In the example below, an alias is added for <code>rclone-cert</code> to do the above invocation. The example assumes that the bash shell is used:
 +
 +
echo 'alias rclone-cert="rclone --client-cert=/tmp/x509up_u${UID} --client-key=/tmp/x509up_u${UID}"' >> ~/.bashrc
 +
 +
The alias will be activated in a new shell or login session.
 +
 +
For all invocations using certificate authentication, use <code>rclone-cert</code> instead of <code>rclone</code> to pass the needed arguments to rclone.
 +
 +
= Logging in =
 +
 +
== Certificate authentication ==
 +
 +
Your certificate needs to be unlocked, similar to a login process, in order to enable access. This is done using the <code>arcproxy</code> command.
 +
 +
See [[Accessing Swestore with the ARC client#Unlock your certificate]] for details.
 +
 +
Remember to use <code>rclone-cert</code> instead of <code>rclone</code> as mentioned in [[#Additional steps for Certificate authentication]]
 +
 +
== Username/Password authentication ==
 +
 +
When using this method, an explicite login isn't needed since the username/password is entered into the rclone configuration file. It's convenient, but less secure.
  
 
= Copying files =  
 
= Copying files =  
Line 81: Line 141:
 
normal '''cp''' command as shown in the following example:
 
normal '''cp''' command as shown in the following example:
  
  $ rclone copyto archive.tar.gz swestore:/snic/YOUR_PROJECT_DIR/archive.tar.gz
+
  rclone copyto archive.tar.gz swestore:/snic/YOUR_PROJECT_DIR/archive.tar.gz
  
 
You can also use '''copyto''' in order to rename the file in the process of copying it to the destination, by specifying a different filename on the remote.
 
You can also use '''copyto''' in order to rename the file in the process of copying it to the destination, by specifying a different filename on the remote.
Line 91: Line 151:
 
Example:
 
Example:
  
  $ rclone copy /path/to/src swestore:/snic/YOUR_PROJECT_DIR/DESTINATION_DIRECTORY
+
  rclone copy /path/to/src swestore:/snic/YOUR_PROJECT_DIR/DESTINATION_DIRECTORY
  
 
The option '''--no-traverse''' can be used to not list files on the destination (good for huge directories). '''--max-age''' can be used to select the most recently modified files for transfer, and '''-P''' gives you status on progress.
 
The option '''--no-traverse''' can be used to not list files on the destination (good for huge directories). '''--max-age''' can be used to select the most recently modified files for transfer, and '''-P''' gives you status on progress.
Line 97: Line 157:
 
Example, copying the last days modifications, with progress:
 
Example, copying the last days modifications, with progress:
  
  $ rclone copy --max-age 24h --no-traverse -P /path/to/src swestore:/snic/YOUR_PROJECT_DIR/DESTINATION_DIRECTORY
+
  rclone copy --max-age 24h --no-traverse -P /path/to/src swestore:/snic/YOUR_PROJECT_DIR/DESTINATION_DIRECTORY
  
 
'''NOTE:''' The above example will copy all files in the directory <code>src</code> into  
 
'''NOTE:''' The above example will copy all files in the directory <code>src</code> into  
Line 103: Line 163:
 
to be part of the destination path you have to explicitly supply it as shown in the example below:
 
to be part of the destination path you have to explicitly supply it as shown in the example below:
  
  $ rclone copy /path/to/src swestore:/snic/YOUR_PROJECT_DIR/DESTINATION_DIRECTORY/src
+
  rclone copy /path/to/src swestore:/snic/YOUR_PROJECT_DIR/DESTINATION_DIRECTORY/src
  
 
= Listing =
 
= Listing =
Line 113: Line 173:
 
only lists objects and their sizes in the specified directory ''and all subdirectories due to recursion being enabled by default''.
 
only lists objects and their sizes in the specified directory ''and all subdirectories due to recursion being enabled by default''.
  
  $ rclone ls swestore:/snic/YOUR_PROJECT_DIR
+
  rclone ls swestore:/snic/YOUR_PROJECT_DIR
  
 
Further functionality can be be achieved from using any of the the following '''ls*''' commands;
 
Further functionality can be be achieved from using any of the the following '''ls*''' commands;
Line 126: Line 186:
 
Examples:
 
Examples:
  
  $ rclone lsl --max-depth 1 swestore:/snic/YOUR_PROJECT_DIR
+
  rclone lsl --max-depth 1 swestore:/snic/YOUR_PROJECT_DIR
  
  $ rclone lsd swestore:/snic/YOUR_PROJECT_DIR
+
  rclone lsd swestore:/snic/YOUR_PROJECT_DIR
  
 
= Creating directories =  
 
= Creating directories =  
Line 134: Line 194:
 
Directories are generally created on demand. If you copy a file with the destination /snic/YOUR_PROJECT_DIR/newdir/dummyfile the newdir directory will be created if missing. But you can explicitly create directories using the '''mkdir''' command.
 
Directories are generally created on demand. If you copy a file with the destination /snic/YOUR_PROJECT_DIR/newdir/dummyfile the newdir directory will be created if missing. But you can explicitly create directories using the '''mkdir''' command.
  
  $ rclone mkdir swestore:/snic/YOUR_PROJECT_DIR/newdir
+
  rclone mkdir swestore:/snic/YOUR_PROJECT_DIR/newdir
  
 
= Removing files or directories =
 
= Removing files or directories =
Line 142: Line 202:
 
To remove the file <code>dummyfile</code> under <code>/snic/YOUR_PROJECT_DIR/newdir</code>,
 
To remove the file <code>dummyfile</code> under <code>/snic/YOUR_PROJECT_DIR/newdir</code>,
  
  $ rclone deletefile swestore:/snic/YOUR_PROJECT_DIR/newdir/dummyfile
+
  rclone deletefile swestore:/snic/YOUR_PROJECT_DIR/newdir/dummyfile
  
 
To remove a directory, they have to be empty, and you use the command
 
To remove a directory, they have to be empty, and you use the command
  
  $ rclone rmdir swestore:/snic/YOUR_PROJECT_DIR/newdir/
+
  rclone rmdir swestore:/snic/YOUR_PROJECT_DIR/newdir/
  
 
To remove '''all''' empty directories under a path, use
 
To remove '''all''' empty directories under a path, use
  
  $ rclone rmdirs swestore:/snic/YOUR_PROJECT_DIR/newdir/
+
  rclone rmdirs swestore:/snic/YOUR_PROJECT_DIR/newdir/
  
 
To recursively remove '''all''' files under <code>/snic/YOUR_PROJECT_DIR/newdir</code>, leaving the empty directory structure in place, '''be careful''';
 
To recursively remove '''all''' files under <code>/snic/YOUR_PROJECT_DIR/newdir</code>, leaving the empty directory structure in place, '''be careful''';
  
  $ rclone delete swestore:/snic/YOUR_PROJECT_DIR/newdir/
+
  rclone delete swestore:/snic/YOUR_PROJECT_DIR/newdir/
  
 
'''Do not use''' the command purge, as that will '''delete everything''' under the path specified.
 
'''Do not use''' the command purge, as that will '''delete everything''' under the path specified.
Line 162: Line 222:
 
'''Q''': I used <code>rclone purge</code> or <code>rclone delete</code> and have now deleted all my files; can I get them back somehow?
 
'''Q''': I used <code>rclone purge</code> or <code>rclone delete</code> and have now deleted all my files; can I get them back somehow?
 
: '''A''': '''No'''. Swestore does not currently support recovery of data for which the user has explicably requested to be deleted from the system, be it intentionally or by mistake. Therefore '''caution''' is advised while using powerful tools such as <code>rclone</code>.
 
: '''A''': '''No'''. Swestore does not currently support recovery of data for which the user has explicably requested to be deleted from the system, be it intentionally or by mistake. Therefore '''caution''' is advised while using powerful tools such as <code>rclone</code>.
 +
 +
'''Q''': When trying to use rclone with certificate authentication, I get an error message ending with <code>401 login failed</code>
 +
: '''A''': Check the following: 1) <code>arcproxy -I</code> shows a valid credential. 2) You're using <code>rclone-cert</code> to get the proper rclone invocation.

Revision as of 12:31, 9 March 2022


< Swestore-dCache

This guide describes how to use the Rclone WebDAV client for storing and retrieving files from Swestore. Rclone is versatile and supports many protocols through a simple command line interface (CLI).

Requirements

SNIC resources should already have rclone installed. To install rclone on your own computer, please follow your systems instructions at the official rclone documentation pages here.

To access Swestore using rclone you need to be a member of a Swestore storage project, see Swestore#Getting access to Swestore.

Depending on your preference and security needs, you can use rclone with either certificate or username/password authentication.

Certificate authentication

This method requires you to obtain a security certificate and install it. While more cumbersome to set up, it provides a higher level of security.

We recommend this method when logged in to SNIC resources.

To use this authentication method you need to do the Certificate setup described in Accessing Swestore with the ARC client#Requirements

Username/Password

This method will require you to enter your username/password in the rclone configuration file. While this is convenient, it is also a security risk since a malicious user gaining access to the rclone configuration file or your login on the affected resource will also gain access to your Swestore dCache storage.

To use this authentication method you need to know your Username and password, see Setting your Swestore password.

Quickstart

Swestore access URL

The WebDAV access URL for Swestore should be specified for rclone as;

https://webdav.swestore.se

Basic commands

rclone config - configure rclone. See Rclone configuration for details.
rclone ls - for listing files. Works similarly to ls. See Rclone ls.
rclone copyto - for copying directories or separate files. Works similarly to cp with wildcards. See Rclone copyto.
rclone copy - for copying contents of directories. Works similarly to cp -R path/*. See Rclone copy.
rclone mkdir - for creating directories. Works similarly to mkdir -p. See Rclone mkdir.
rclone deletefile - for removing specific files. Works similarly to rm. See Rclone deletefile.
More powerful removal functions are available, but be careful with these;
rclone delete - for removing all files under path - i.e. you loose your data. Works similarly to find path -type f | xargs rm. See Rclone delete.
rclone rmdir - removing path if empty. Works similarly to rm -d. See Rclone rmdir.
rclone rmdirs - removing all empty directories under path. Works similarly to find path -type d | awk '{ print length, $0 }' | sort -nsr | cut -d" " -f2- | xargs rm -d (example modified from stackoverflow). See Rclone rmdirs.
rclone purge - be very careful using this since it will remove all data under path, including path - i.e. you loose your data. Works similarly to rm -rf. See Rclone purge.

Use man and --help to get more info on rclone and its commands. Examples: man rclone, rclone --help or rclone copy --help .

Paths

The rclone commands supports multiple storage protocols. Given that you have configured https://webdav.swestore.se as the swestore remote, we recommend using WebDAV with paths on the form swestore:/snic/YOUR_PROJECT_DIR/... .

Configuration

You have to configure rclone with your access protocol, URL and possibly user login credentials for it to work with Swestore dCache.

Issue the following configuration command to interactively configure rclone:

rclone config

Then choose between the two configuration alternatives below, depending on you authentication preference.

For configuring Swestore dCache access using WebDAV and Certificate authentication:

n for New remote
swestore for name
webdav for Storage (or the corresponding number in the list)
https://webdav.swestore.se for url
other for vendor (or the corresponding number in the list)
Just press <Enter> for user (ie. leave blank)
n for No leave this optional password blank
Just press <Enter> for bearer_token
n for Edit advanced config?
y if you think the resulting config is correct, otherwise e to edit again.
q to Quit config

Alternatively, for configuring Swestore dCache access using WebDAV and Username/Password authentication:

n for New remote
swestore for name
webdav for Storage (or the corresponding number in the list)
https://webdav.swestore.se for url
other for vendor (or the corresponding number in the list)
yourusername for user, should be on the format s_user
y for Yes type in my own password
then enter your swestore password twice
Just press <Enter> for bearer_token
n for Edit advanced config?
y if you think the resulting config is correct, otherwise e to edit again.
q to Quit config

To see your configuration afterwards, run

cat ~/.config/rclone/rclone.conf

You can also list your configured remotes by issuing

rclone listremotes

In the following sections, we are assuming your swestore remote is named swestore

Additional steps for Certificate authentication

Unfortunately rclone doesn't (yet) support adding the needed client certificate paths to the configuration, requiring the following invocation for all rclone commands:

rclone --client-cert=/tmp/x509up_u${UID} --client-key=/tmp/x509up_u${UID}

Since this is obviously very cumbersome, we strongly suggest adding an alias for it. In the example below, an alias is added for rclone-cert to do the above invocation. The example assumes that the bash shell is used:

echo 'alias rclone-cert="rclone --client-cert=/tmp/x509up_u${UID} --client-key=/tmp/x509up_u${UID}"' >> ~/.bashrc

The alias will be activated in a new shell or login session.

For all invocations using certificate authentication, use rclone-cert instead of rclone to pass the needed arguments to rclone.

Logging in

Certificate authentication

Your certificate needs to be unlocked, similar to a login process, in order to enable access. This is done using the arcproxy command.

See Accessing Swestore with the ARC client#Unlock your certificate for details.

Remember to use rclone-cert instead of rclone as mentioned in #Additional steps for Certificate authentication

Username/Password authentication

When using this method, an explicite login isn't needed since the username/password is entered into the rclone configuration file. It's convenient, but less secure.

Copying files

Copying files to and from resources is accomplished using the rclone copy and copyto command.

Copying single files

Copying single files is accomplished in the same way as using the normal cp command as shown in the following example:

rclone copyto archive.tar.gz swestore:/snic/YOUR_PROJECT_DIR/archive.tar.gz

You can also use copyto in order to rename the file in the process of copying it to the destination, by specifying a different filename on the remote.

Recursive copying

Recursive copying of a directory is accomplished using the copy command. The command will only copy files that have changed on the source compared to the destination, which is determined by checksums and timestamps. Observe that the source directory is not copied over, only its contents. Also, empty directories are omitted.

Example:

rclone copy /path/to/src swestore:/snic/YOUR_PROJECT_DIR/DESTINATION_DIRECTORY

The option --no-traverse can be used to not list files on the destination (good for huge directories). --max-age can be used to select the most recently modified files for transfer, and -P gives you status on progress.

Example, copying the last days modifications, with progress:

rclone copy --max-age 24h --no-traverse -P /path/to/src swestore:/snic/YOUR_PROJECT_DIR/DESTINATION_DIRECTORY

NOTE: The above example will copy all files in the directory src into the destination directory DESTINATION_DIRECTORY. If you want the directory src to be part of the destination path you have to explicitly supply it as shown in the example below:

rclone copy /path/to/src swestore:/snic/YOUR_PROJECT_DIR/DESTINATION_DIRECTORY/src

Listing

Rclone supports listing all files under a path.

Listing files and directories on a resources is done using the ls or one of the ls* commands (see below). The simple ls command only lists objects and their sizes in the specified directory and all subdirectories due to recursion being enabled by default.

rclone ls swestore:/snic/YOUR_PROJECT_DIR

Further functionality can be be achieved from using any of the the following ls* commands;

lsl long listing with additional info
lsd list only directories
lsf list objects and directories, in a fashion good for scripting
lsjson gives advanced output in JSON format

NOTE: The ls and lsl recurses to show all subdirectories by default - use --max-depth 1 to disable it. For other commands you need to specify the -R flag to enable recursion.

Examples:

rclone lsl --max-depth 1 swestore:/snic/YOUR_PROJECT_DIR
rclone lsd swestore:/snic/YOUR_PROJECT_DIR

Creating directories

Directories are generally created on demand. If you copy a file with the destination /snic/YOUR_PROJECT_DIR/newdir/dummyfile the newdir directory will be created if missing. But you can explicitly create directories using the mkdir command.

rclone mkdir swestore:/snic/YOUR_PROJECT_DIR/newdir

Removing files or directories

Beware that the command delete will recursively delete all file objects under specified path!

To remove the file dummyfile under /snic/YOUR_PROJECT_DIR/newdir,

rclone deletefile swestore:/snic/YOUR_PROJECT_DIR/newdir/dummyfile

To remove a directory, they have to be empty, and you use the command

rclone rmdir swestore:/snic/YOUR_PROJECT_DIR/newdir/

To remove all empty directories under a path, use

rclone rmdirs swestore:/snic/YOUR_PROJECT_DIR/newdir/

To recursively remove all files under /snic/YOUR_PROJECT_DIR/newdir, leaving the empty directory structure in place, be careful;

rclone delete swestore:/snic/YOUR_PROJECT_DIR/newdir/

Do not use the command purge, as that will delete everything under the path specified.

FAQ

Q: I used rclone purge or rclone delete and have now deleted all my files; can I get them back somehow?

A: No. Swestore does not currently support recovery of data for which the user has explicably requested to be deleted from the system, be it intentionally or by mistake. Therefore caution is advised while using powerful tools such as rclone.

Q: When trying to use rclone with certificate authentication, I get an error message ending with 401 login failed

A: Check the following: 1) arcproxy -I shows a valid credential. 2) You're using rclone-cert to get the proper rclone invocation.