Difference between revisions of "Accessing Swestore with lftp"

From SNIC Documentation
Jump to: navigation, search
(lftp settings)
(Status)
 
(14 intermediate revisions by 3 users not shown)
Line 1: Line 1:
== Introduction ==
+
[[Category:Swestore]]
 +
[[Category:Swestore user guide]]
 +
[[Swestore|< Swestore]]
  
lftp is a file transfer tool that understands a range of protocols from plain old FTP, HTTP, SCP to more esoteric ones like WebDAV and BitTorrent.
+
= Status =
  
The benefit over tools like cURL is that it has interactive traversal of the directory hierarchy, as well as powerful mass-transfer functionality like the mirror command.
+
Current lftp version as shipped in most Linux distributions are having issues in presenting a WebDAV directory listing that makes sense to users, among the issues is a tendency to mix files from the root directory and the current directory.
  
Out of the protocols mentioned above, the ones that aligns most with the access doors that Swestore offers are the WebDAV over HTTP/HTTPS protocols.
+
Due to these and other issues '''we recommend that lftp is not used for file transfers from/to Swestore'''.
  
Authentication against Swestore over the WebDAV door is done with client certificates over HTTPS, where there is a choice of either using the real client certificate or by using a RFC or VOMS proxy certificate generated from the real certificate.
+
= Introduction =
  
Proxy certificates are preferred as they are valid for a limited period of time, especially as lftp does not seem to offer any way to enter any passphrase to unlock keys, so in order to use a protected real certificate, it will have to be unlocked in advance.
+
lftp is an interactive text-mode file transfer tool that understands multiple access protocols. It is commonly available in Linux/Unix environments.
  
Notable is that while file uploads are securely tunneled inside a SSL connection, downloads are in the plain from storage nodes. As such, the usual guidelines about sensitive data holds, where you should aim to have your data encrypted before transmitting it.
+
The benefit over tools like cURL is that it has interactive traversal of the directory hierarchy, as well as powerful mass-transfer functionality like the <code>mirror</code> command, <code>mput/mget</code> and more. Use <code>help</code> in lftp for details. This is not intended to be a complete lftp usage howto, refer to tutorials online for this.
  
== Required software versions ==
+
This guide explains how to use lftp with username/password authentication.
  
The versions of lftp and its major dependency GNUTLS that have been verified to work with proxy certificates are:
+
= Authenticate using username/password =
* lftp 4.3.3
 
* gnutls 2.12.0
 
* libnettle 2.4 (for building gnutls)
 
  
This gnutls version is the absolute minimum version that will work, any version prior to that (2.10.5 and below) will not be able to connect to the door.
+
* Prerequisite: Username and password, see [[Setting your Swestore password]]
 +
* Open a terminal window
 +
* Start lftp with: '''<code>lftp -u username https://webdav.swestore.se/</code>'''
 +
* Enter your password at the prompt
 +
** '''NOTE:''' Password will be verified on first command/connection, so typing the wrong password will error out later!
  
This means that the gnutls version in Ubuntu oneric and older, Scientific Linux/RHEL/CentOS 6.1 and older will not work.
+
= Usage =
  
Ubuntu precise has a sufficiently new gnutls and will work out of the box.
+
There are some quirks that lftp has together with the Swestore WebDAV door.
  
== lftp settings ==
+
When giving a directory path to a command, it should end with a trailing slash to indicate that it is a directory. If this is omitted, the client will get a redirection response that the tool doesn't handle properly.
 
 
While running the <code>lftp</code> program, there are several settings that need to be configured in order to successfully connect and interact with Swestore:
 
<nowiki>
 
set ssl:ca-file /etc/grid-security/certificates/NorduGrid.pem
 
set ssl:check-hostname true
 
set ssl:verify-certificate true</nowiki>
 
 
 
and
 
<nowiki>
 
set ssl:key-file /tmp/x509up_u1234
 
set ssl:cert-file /tmp/x509up_u1234</nowiki>
 
 
 
where the last two are to indicate the filename of the proxy certificate generated by the <code>arcproxy</code> or <code>grid-proxy-init<code> tool. The trailing digits in the filename will vary based on the user ID (uid) of the local user, which you can find out in a terminal shell by running the <code>id</code> program.
 
 
 
The CA file is for verifying the identity of the server and is strongly recommended that both certificate and hostname verification is in effect to ensure that the server communicated with is the intended target machine.
 
 
 
They can be stored in the configuration file named ~/.lftp/rc together with any other commands you wish to perform during lftp startup.
 
 
 
== Usage ==
 
  
Assuming that you've managed to obtain a working lftp binary, there are some quirks that lftp has together with the Swestore WebDAV door.
+
All the commands mentioned in this section and the previous configuration section are commands inside of a running <code>lftp</code> program.
 
 
When giving a directory path to a command, it should end with a trailing slash to indicate that it is a directory. If this is omitted, the client will get a redirection response that the tool doesn't handle properly.
 
  
 
Some sample tasks that can be achieved with lftp is retrieving or uploading single files or whole directory trees.
 
Some sample tasks that can be achieved with lftp is retrieving or uploading single files or whole directory trees.
  
The command to connect to the door is:
+
Navigate around with the use of the <code>cd</code> command:
<nowiki>
+
  cd snic/project_name_here/
open https://webdav.swestore.se/</nowiki>
+
Note the trailing / !
 
 
after which you can navigate around with the use of the 'cd' command:
 
<nowiki>
 
  cd snic/project_name_here/foo/</nowiki>
 
  
Individual files can be manipulated using the ''get'' and ''put'' commands and the ''mget'' and ''mirror'' commands can transfer multiple files and whole trees, respectively.
+
Individual files can be manipulated using the <code>get</code> and <code>put</code> commands, while <code>mget</code> and <code>mput</code> transfers multiple files at once. The <code>mirror</code> command can transfer whole trees.
  
The program has interactive help for any command through the ''help'' command.
+
The program has interactive help for any command through the <code>help</code> command.
  
For mirroring the flags ''-R'' and ''-c'' are particularly relevant as ''-R'' controls the direction of the operation - if it is present the transfer is mirroring '''to''' the server, otherwise it's mirroring '''from''' the server.
+
For mirroring the flag ''-R'' is particularly relevant as ''-R'' controls the direction of the operation - if it is present the transfer is mirroring '''to''' the server, otherwise it's mirroring '''from''' the server.
 
 
''-c'' indicates that the operation should resume whenever possible which may improve synchronization time if you know that any partial files present on the other side are identical to the local files.
 
  
 
For example, the command
 
For example, the command
  <nowiki>mirror -c A B</nowiki>
+
  mirror A B
  
 
will download all of the remote directory ''A'' into the local directory named ''B''.
 
will download all of the remote directory ''A'' into the local directory named ''B''.
  
 
The command
 
The command
  <nowiki>mirror -cR C D</nowiki>
+
  mirror -R C D
  
 
will upload all of the local directory ''C'' into the remote directory named ''D''. Note that the role of the directories is reversed compared to the previous example.
 
will upload all of the local directory ''C'' into the remote directory named ''D''. Note that the role of the directories is reversed compared to the previous example.
 
== Build instructions ==
 
 
Building lftp and its dependencies from scratch does not require any particular build flags but you might want to install it into a private destination (prefix) to avoid it interfering with system-provided libraries.
 
 
libnettle depends on GMP and lftp depends on readline and gperf, the distribution packages for those are sufficient on Ubuntu and CentOS 5/7 and 6.2.
 
 
These instructions assume that the ''bash'' shell is used when building, ''tcsh'' and other shells will have slightly different syntax for environment variables.
 
<nowiki>
 
export LFTP_PREFIX="${HOME}/local"
 
export PKG_CONFIG_PATH="${LFTP_PREFIX}/lib/pkgconfig:${PKG_CONFIG_PATH}"
 
export CPPFLAGS="-I${LFTP_PREFIX}/include"
 
export LDFLAGS="-L${LFTP_PREFIX}/lib -Wl,-R${LFTP_PREFIX}/lib"</nowiki>
 
 
Start by extracting the source distributions:
 
<nowiki>
 
tar xzf ~/Downloads/nettle-2.4.tar.gz
 
tar xJf ~/Downloads/gnutls-3.0.12.tar.xz
 
tar xJf ~/Downloads/lftp-4.3.5.tar.xz</nowiki>
 
 
You might want to avoid building a shared gnutls, so passing
 
<nowiki>
 
--enable-static --disable-shared</nowiki>
 
 
on the gnutls configure command line might be a good idea.
 
 
libnettle is a dependency for gnutls, and gnutls is a dependency for lftp, so we build them in that order:
 
<nowiki>
 
mkdir ${LFTP_PREFIX}
 
 
pushd nettle-2.4
 
./configure --prefix=${LFTP_PREFIX} &&
 
make && make install
 
popd
 
 
pushd gnutls-3.0.12
 
./configure --prefix=${LFTP_PREFIX} --without-p11-kit &&
 
make && make install
 
popd
 
 
pushd lftp-4.3.5
 
./configure --prefix=${LFTP_PREFIX} &&
 
make && make install
 
popd</nowiki>
 
 
If the platform already has development files for p11-kit there is no harm in letting it use them (it allows gnutls to understand PKCS-style certificates), but it's nothing that lftp can utilize so it's not considered a dependency in this document.
 
 
After the build process completes, a lftp binary will exist in ''${LFTP_PREFIX}/bin'' and depend on the gnutls shared library in ''${LFTP_PREFIX}/lib'' if you did not build it statically.
 
  
 
= Credits =
 
= Credits =
  
This guide was written by Lars Viklund
+
This guide was initially written by Lars Viklund and subsequently revised by Niklas Edmundsson.

Latest revision as of 08:22, 14 October 2019

< Swestore

Status

Current lftp version as shipped in most Linux distributions are having issues in presenting a WebDAV directory listing that makes sense to users, among the issues is a tendency to mix files from the root directory and the current directory.

Due to these and other issues we recommend that lftp is not used for file transfers from/to Swestore.

Introduction

lftp is an interactive text-mode file transfer tool that understands multiple access protocols. It is commonly available in Linux/Unix environments.

The benefit over tools like cURL is that it has interactive traversal of the directory hierarchy, as well as powerful mass-transfer functionality like the mirror command, mput/mget and more. Use help in lftp for details. This is not intended to be a complete lftp usage howto, refer to tutorials online for this.

This guide explains how to use lftp with username/password authentication.

Authenticate using username/password

  • Prerequisite: Username and password, see Setting your Swestore password
  • Open a terminal window
  • Start lftp with: lftp -u username https://webdav.swestore.se/
  • Enter your password at the prompt
    • NOTE: Password will be verified on first command/connection, so typing the wrong password will error out later!

Usage

There are some quirks that lftp has together with the Swestore WebDAV door.

When giving a directory path to a command, it should end with a trailing slash to indicate that it is a directory. If this is omitted, the client will get a redirection response that the tool doesn't handle properly.

All the commands mentioned in this section and the previous configuration section are commands inside of a running lftp program.

Some sample tasks that can be achieved with lftp is retrieving or uploading single files or whole directory trees.

Navigate around with the use of the cd command:

cd snic/project_name_here/

Note the trailing / !

Individual files can be manipulated using the get and put commands, while mget and mput transfers multiple files at once. The mirror command can transfer whole trees.

The program has interactive help for any command through the help command.

For mirroring the flag -R is particularly relevant as -R controls the direction of the operation - if it is present the transfer is mirroring to the server, otherwise it's mirroring from the server.

For example, the command

mirror A B

will download all of the remote directory A into the local directory named B.

The command

mirror -R C D

will upload all of the local directory C into the remote directory named D. Note that the role of the directories is reversed compared to the previous example.

Credits

This guide was initially written by Lars Viklund and subsequently revised by Niklas Edmundsson.