|
|
| (4 intermediate revisions by the same user not shown) |
| Line 1: |
Line 1: |
| − | [[Category:Grid computing]]
| + | #REDIRECT[[Swestore Documentation Moved]] |
| − | [[Category:SweGrid user guide]]
| |
| − | [[Category:Swestore]]
| |
| − | [[Category:Swestore user guide]]
| |
| − | | |
| − | [[Swestore-dCache|< Swestore-dCache]]
| |
| − | | |
| − | This guide describes how to use the [https://rclone.org/webdav/ Rclone] ''WebDAV'' client for storing and retrieving files from Swestore. Rclone is versatile and supports many protocols through a simple command line interface (CLI).
| |
| − | | |
| − | = Requirements =
| |
| − | | |
| − | SNIC resources should already have rclone installed. To install rclone on your own computer, please follow your systems instructions at the official rclone documentation pages [https://rclone.org/install/ here].
| |
| − | | |
| − | To access Swestore using rclone you need to be a member of a Swestore storage project, see [[Swestore#Getting access to Swestore]].
| |
| − | | |
| − | Depending on your preference and security needs, you can use rclone with either ''certificate'' or ''username/password'' authentication.
| |
| − | | |
| − | == Username/Password ==
| |
| − | | |
| − | This method will require you to enter your username/password in the rclone configuration file. While this is convenient, it is also a security risk since a malicious user gaining access to the rclone configuration file or your login on the affected resource will also gain access to your Swestore dCache storage.
| |
| − | | |
| − | To use this authentication method you need to know your Username and password, see [[Setting your Swestore password]].
| |
| − | | |
| − | == Certificate authentication ==
| |
| − | | |
| − | This method requires you to obtain a security certificate and install it. While more cumbersome to set up, it provides a higher level of security.
| |
| − | | |
| − | To use this authentication method you need to do the Certificate setup described in [[Accessing Swestore with the ARC client#Requirements]]
| |
| − | | |
| − | = Quickstart =
| |
| − | | |
| − | == Swestore access URL ==
| |
| − | | |
| − | The WebDAV access URL for Swestore should be specified for <code>rclone</code> as;
| |
| − | | |
| − | :: https://webdav.swestore.se
| |
| − | | |
| − | == Basic commands ==
| |
| − | : <code>rclone config</code> - configure rclone. See [https://rclone.org/docs/#configure Rclone configuration] for details.
| |
| − | : <code>rclone ls</code> - for listing files. Works similarly to <code>ls</code>. See [https://rclone.org/commands/rclone_ls/ Rclone ls].
| |
| − | : <code>rclone copyto</code> - for copying directories or separate files. Works similarly to <code>cp</code> with wildcards. See [https://rclone.org/commands/rclone_copyto/ Rclone copyto].
| |
| − | : <code>rclone copy</code> - for copying contents of directories. Works similarly to <code>cp -R path/*</code>. See [https://rclone.org/commands/rclone_copy/ Rclone copy].
| |
| − | : <code>rclone mkdir</code> - for creating directories. Works similarly to <code>mkdir -p</code>. See [https://rclone.org/commands/rclone_mkdir/ Rclone mkdir].
| |
| − | : <code>rclone deletefile</code> - for removing specific files. Works similarly to <code>rm</code>. See [https://rclone.org/commands/rclone_deletefile/ Rclone deletefile].
| |
| − | :: More powerful removal functions are available, but '''be careful''' with these;
| |
| − | ::: <code>rclone delete</code> - for removing all files under path - i.e. '''you loose your data'''. Works similarly to <code>find path -type f | xargs rm</code>. See [https://rclone.org/commands/rclone_delete/ Rclone delete].
| |
| − | ::: <code>rclone rmdir</code> - removing path if empty. Works similarly to <code>rm -d</code>. See [https://rclone.org/commands/rclone_rmdir/ Rclone rmdir].
| |
| − | ::: <code>rclone rmdirs</code> - removing all empty directories under path. Works similarly to <code>find path -type d | awk '{ print length, $0 }' | sort -nsr | cut -d" " -f2- | xargs rm -d</code> (example modified from [https://stackoverflow.com/a/5917762 stackoverflow]). See [https://rclone.org/commands/rclone_rmdirs/ Rclone rmdirs].
| |
| − | ::: <code>rclone purge</code> - '''be very careful using this''' since it will remove all data under path, including path - i.e. '''you loose your data'''. Works similarly to <code>rm -rf</code>. See [https://rclone.org/commands/rclone_purge/ Rclone purge].
| |
| − | | |
| − | Use <code>man</code> and <code>--help</code> to get more info on rclone and its commands. Examples: <code>man rclone</code>, <code>rclone --help</code> or <code>rclone copy --help</code> .
| |
| − | | |
| − | == Paths ==
| |
| − | The rclone commands supports multiple storage protocols. Given that you have configured <code><nowiki>https://webdav.swestore.se</nowiki></code> as the <code>swestore</code> remote, we recommend using WebDAV with paths on the form <code><nowiki>swestore:/snic/YOUR_PROJECT_DIR/...</nowiki></code> .
| |
| − | | |
| − | = Configuration =
| |
| − | | |
| − | You have to configure rclone with your access protocol, URL and possibly user login credentials for it to work with Swestore dCache.
| |
| − | | |
| − | Issue the following configuration command to interactively configure rclone:
| |
| − | | |
| − | rclone config
| |
| − | | |
| − | Then choose between the two configuration alternatives below, depending on you authentication preference.
| |
| − | | |
| − | For configuring Swestore dCache access using WebDAV and ''Certificate'' authentication:
| |
| − | | |
| − | : <code>n</code> for '''New remote'''
| |
| − | : <code>swestore</code> for '''name'''
| |
| − | : <code>webdav</code> for '''Storage''' (or the corresponding number in the list)
| |
| − | : <code><nowiki>https://webdav.swestore.se</nowiki></code> for '''url'''
| |
| − | : <code>other</code> for '''vendor''' (or the corresponding number in the list)
| |
| − | : Just press <code><nowiki><Enter></nowiki></code> for '''user''' (ie. leave blank)
| |
| − | : <code>n</code> for '''No leave this optional password blank'''
| |
| − | : Just press <code><nowiki><Enter></nowiki></code> for '''bearer_token'''
| |
| − | : <code>n</code> for '''Edit advanced config?'''
| |
| − | : <code>y</code> if you think the resulting config is correct, otherwise <code>e</code> to edit again.
| |
| − | : <code>q</code> to '''Quit config'''
| |
| − | | |
| − | Alternatively, for configuring Swestore dCache access using WebDAV and ''Username/Password'' authentication:
| |
| − | | |
| − | : <code>n</code> for '''New remote'''
| |
| − | : <code>swestore</code> for '''name'''
| |
| − | : <code>webdav</code> for '''Storage''' (or the corresponding number in the list)
| |
| − | : <code><nowiki>https://webdav.swestore.se</nowiki></code> for '''url'''
| |
| − | : <code>other</code> for '''vendor''' (or the corresponding number in the list)
| |
| − | : <code>yourusername</code> for '''user''', should be on the format <code>s_user</code>
| |
| − | : <code>y</code> for '''Yes type in my own password'''
| |
| − | :: then enter your swestore password twice
| |
| − | : Just press <code><nowiki><Enter></nowiki></code> for '''bearer_token'''
| |
| − | : <code>n</code> for '''Edit advanced config?'''
| |
| − | : <code>y</code> if you think the resulting config is correct, otherwise <code>e</code> to edit again.
| |
| − | : <code>q</code> to '''Quit config'''
| |
| − | | |
| − | To see your configuration afterwards, run
| |
| − | | |
| − | cat ~/.config/rclone/rclone.conf
| |
| − | | |
| − | You can also list your configured remotes by issuing
| |
| − | | |
| − | rclone listremotes
| |
| − | | |
| − | In the following sections, we are assuming your swestore remote is named <code>swestore</code>
| |
| − | | |
| − | = Copying files =
| |
| − | | |
| − | Copying files to and from resources is accomplished using the rclone '''copy''' and '''copyto''' command.
| |
| − | | |
| − | == Copying single files ==
| |
| − | | |
| − | Copying single files is accomplished in the same way as using the
| |
| − | normal '''cp''' command as shown in the following example:
| |
| − | | |
| − | rclone copyto archive.tar.gz swestore:/snic/YOUR_PROJECT_DIR/archive.tar.gz
| |
| − | | |
| − | You can also use '''copyto''' in order to rename the file in the process of copying it to the destination, by specifying a different filename on the remote.
| |
| − | | |
| − | == Recursive copying ==
| |
| − | | |
| − | Recursive copying of a directory is accomplished using the '''copy''' command. The command will only copy files that have changed on the source compared to the destination, which is determined by checksums and timestamps. Observe that the source directory is not copied over, only its contents. Also, empty directories are omitted.
| |
| − | | |
| − | Example:
| |
| − | | |
| − | rclone copy /path/to/src swestore:/snic/YOUR_PROJECT_DIR/DESTINATION_DIRECTORY
| |
| − | | |
| − | The option '''--no-traverse''' can be used to not list files on the destination (good for huge directories). '''--max-age''' can be used to select the most recently modified files for transfer, and '''-P''' gives you status on progress.
| |
| − | | |
| − | Example, copying the last days modifications, with progress:
| |
| − | | |
| − | rclone copy --max-age 24h --no-traverse -P /path/to/src swestore:/snic/YOUR_PROJECT_DIR/DESTINATION_DIRECTORY
| |
| − | | |
| − | '''NOTE:''' The above example will copy all files in the directory <code>src</code> into
| |
| − | the destination directory <code>DESTINATION_DIRECTORY</code>. If you want the directory <code>src</code>
| |
| − | to be part of the destination path you have to explicitly supply it as shown in the example below:
| |
| − | | |
| − | rclone copy /path/to/src swestore:/snic/YOUR_PROJECT_DIR/DESTINATION_DIRECTORY/src
| |
| − | | |
| − | = Listing =
| |
| − | | |
| − | Rclone supports listing all files under a path.
| |
| − | | |
| − | Listing files and directories on a resources is done using the '''ls'''
| |
| − | or one of the '''ls*''' commands (see below). The simple '''ls''' command
| |
| − | only lists objects and their sizes in the specified directory ''and all subdirectories due to recursion being enabled by default''.
| |
| − | | |
| − | rclone ls swestore:/snic/YOUR_PROJECT_DIR
| |
| − | | |
| − | Further functionality can be be achieved from using any of the the following '''ls*''' commands;
| |
| − | | |
| − | : <code>lsl</code> long listing with additional info
| |
| − | : <code>lsd</code> list only directories
| |
| − | : <code>lsf</code> list objects and directories, in a fashion good for scripting
| |
| − | : <code>lsjson</code> gives advanced output in JSON format
| |
| − | | |
| − | '''NOTE:''' The <code>ls</code> and <code>lsl</code> recurses to show all subdirectories by default - use <code>--max-depth 1</code> to disable it. For other commands you need to specify the <code>-R</code> flag to enable recursion.
| |
| − | | |
| − | Examples:
| |
| − | | |
| − | rclone lsl --max-depth 1 swestore:/snic/YOUR_PROJECT_DIR
| |
| − | | |
| − | rclone lsd swestore:/snic/YOUR_PROJECT_DIR
| |
| − | | |
| − | = Creating directories =
| |
| − | | |
| − | Directories are generally created on demand. If you copy a file with the destination /snic/YOUR_PROJECT_DIR/newdir/dummyfile the newdir directory will be created if missing. But you can explicitly create directories using the '''mkdir''' command.
| |
| − | | |
| − | rclone mkdir swestore:/snic/YOUR_PROJECT_DIR/newdir
| |
| − | | |
| − | = Removing files or directories =
| |
| − | | |
| − | '''Beware''' that the command '''delete''' will recursively delete all file objects under specified path!
| |
| − | | |
| − | To remove the file <code>dummyfile</code> under <code>/snic/YOUR_PROJECT_DIR/newdir</code>,
| |
| − | | |
| − | rclone deletefile swestore:/snic/YOUR_PROJECT_DIR/newdir/dummyfile
| |
| − | | |
| − | To remove a directory, they have to be empty, and you use the command
| |
| − | | |
| − | rclone rmdir swestore:/snic/YOUR_PROJECT_DIR/newdir/
| |
| − | | |
| − | To remove '''all''' empty directories under a path, use
| |
| − | | |
| − | rclone rmdirs swestore:/snic/YOUR_PROJECT_DIR/newdir/
| |
| − | | |
| − | To recursively remove '''all''' files under <code>/snic/YOUR_PROJECT_DIR/newdir</code>, leaving the empty directory structure in place, '''be careful''';
| |
| − | | |
| − | rclone delete swestore:/snic/YOUR_PROJECT_DIR/newdir/
| |
| − | | |
| − | '''Do not use''' the command purge, as that will '''delete everything''' under the path specified.
| |
| − | | |
| − | = FAQ =
| |
| − | | |
| − | '''Q''': I used <code>rclone purge</code> or <code>rclone delete</code> and have now deleted all my files; can I get them back somehow?
| |
| − | : '''A''': '''No'''. Swestore does not currently support recovery of data for which the user has explicably requested to be deleted from the system, be it intentionally or by mistake. Therefore '''caution''' is advised while using powerful tools such as <code>rclone</code>.
| |
