Difference between revisions of "Accessing Swestore with lftp"

From SNIC Documentation
Jump to: navigation, search
(lftp settings)
(Status)
(5 intermediate revisions by the same user not shown)
Line 3: Line 3:
 
[[Swestore|< Swestore]]
 
[[Swestore|< Swestore]]
  
 +
= Status =
  
== Introduction ==
+
Current lftp version as shipped in most Linux distributions are having issues in presenting a WebDAV directory listing that makes sense to users, among the issues is a tendency to mix files from the root directory and the current directory.
  
lftp is a file transfer tool that understands a range of protocols from plain old FTP, HTTP, SCP, WebDAV to more esoteric ones like BitTorrent.
+
Due to these and other issues '''we recommend that lftp is not used for file transfers from/to Swestore'''.
  
The benefit over tools like cURL is that it has interactive traversal of the directory hierarchy, as well as powerful mass-transfer functionality like the mirror command.
+
= Introduction =
  
Out of the protocols mentioned above, the ones that aligns most with the access doors that Swestore offers are the WebDAV over HTTP/HTTPS protocols.
+
lftp is an interactive text-mode file transfer tool that understands multiple access protocols. It is commonly available in Linux/Unix environments.
  
Authentication against Swestore over the WebDAV door is done with client certificates over HTTPS, where there is a choice of either using the real client certificate or by using a proxy certificate generated from the real certificate.
+
The benefit over tools like cURL is that it has interactive traversal of the directory hierarchy, as well as powerful mass-transfer functionality like the <code>mirror</code> command, <code>mput/mget</code> and more. Use <code>help</code> in lftp for details. This is not intended to be a complete lftp usage howto, refer to tutorials online for this.
  
Proxy certificates are preferred as they are valid for a limited period of time, especially as lftp does not seem to offer any way to enter any passphrase to unlock keys, so in order to use a protected real certificate, it will have to be unlocked in advance.
+
This guide explains how to use lftp with username/password authentication.
  
== Required software versions ==
+
= Authenticate using username/password =
  
The versions of lftp and its major dependency GNUTLS that have been verified to work with proxy certificates are:
+
* Prerequisite: Username and password, see [[Setting your Swestore password]]
 +
* Open a terminal window
 +
* Start lftp with: '''<code>lftp -u username https://webdav.swestore.se/</code>'''
 +
* Enter your password at the prompt
 +
** '''NOTE:''' Password will be verified on first command/connection, so typing the wrong password will error out later!
  
{| class="wikitable"
+
= Usage =
|+ Tested versions
 
|-
 
| '''Tested on''' ||        || Ubuntu 13.10
 
|-
 
| '''lftp'''      || 4.3.3  || 4.4.9
 
|-
 
| '''gnutls'''    || 2.12.0 || 2.12.23
 
|-
 
| '''libnettle (for building gnutls)''' || 2.4    || 2.7.1
 
|}
 
  
 
+
There are some quirks that lftp has together with the Swestore WebDAV door.
This gnutls version is the absolute minimum version that will work, any version prior to that (2.10.5 and below) will not be able to connect to the door.
 
 
 
This means that the gnutls version in Ubuntu oneric and older, Scientific Linux/RHEL/CentOS 6.1 and older will not work.
 
 
 
Ubuntu precise has a sufficiently new gnutls and will work out of the box.
 
 
 
== lftp settings ==
 
 
 
While running the <code>lftp</code> program, there are several settings that need to be configured in order to successfully connect and interact with Swestore:
 
<nowiki>
 
set ssl:ca-file /etc/grid-security/certificates/TERENA-eScience-SSL-CA-3.pem
 
set ssl:check-hostname true
 
set ssl:verify-certificate true</nowiki>
 
 
 
and
 
<nowiki>
 
set ssl:key-file /tmp/x509up_u1234
 
set ssl:cert-file /tmp/x509up_u1234</nowiki>
 
 
 
where the last two are to indicate the filename of the proxy certificate generated by the <code>arcproxy</code> or <code>grid-proxy-init</code> tool. The trailing digits in the filename will vary based on the user ID (uid) of the local user, which you can find out in a terminal shell by running the <code>id</code> program.
 
 
 
The CA file is for verifying the identity of the server and is strongly recommended that both certificate and hostname verification is in effect to ensure that the server communicated with is the intended target machine.
 
 
 
They can be stored in the configuration file named ~/.lftp/rc together with any other commands you wish to perform during lftp startup.
 
 
 
 
 
'''Note!'''
 
 
 
Due to a bug in GnuTLS the verification of host certificates doesn't seem to work. Lftp will return "cd: Fatal error: Certificate verification: Not trusted" when the connection is opened. Please use:
 
set ssl:verify-certificate false
 
until this is solved.
 
 
 
== Usage ==
 
 
 
Assuming that you've managed to obtain a working lftp binary, there are some quirks that lftp has together with the Swestore WebDAV door.
 
  
 
When giving a directory path to a command, it should end with a trailing slash to indicate that it is a directory. If this is omitted, the client will get a redirection response that the tool doesn't handle properly.
 
When giving a directory path to a command, it should end with a trailing slash to indicate that it is a directory. If this is omitted, the client will get a redirection response that the tool doesn't handle properly.
Line 75: Line 35:
 
Some sample tasks that can be achieved with lftp is retrieving or uploading single files or whole directory trees.
 
Some sample tasks that can be achieved with lftp is retrieving or uploading single files or whole directory trees.
  
The command to connect to the door is:
+
Navigate around with the use of the <code>cd</code> command:
<nowiki>
+
  cd snic/project_name_here/
  open https://webdav.swestore.se/</nowiki>
+
Note the trailing / !
  
after which you can navigate around with the use of the 'cd' command:
+
Individual files can be manipulated using the <code>get</code> and <code>put</code> commands, while <code>mget</code> and <code>mput</code> transfers multiple files at once. The <code>mirror</code> command can transfer whole trees.
<nowiki>
 
cd snic/project_name_here/foo/</nowiki>
 
  
Individual files can be manipulated using the ''get'' and ''put'' commands and the ''mget'' and ''mirror'' commands can transfer multiple files and whole trees, respectively.
+
The program has interactive help for any command through the <code>help</code> command.
  
The program has interactive help for any command through the ''help'' command.
+
For mirroring the flag ''-R'' is particularly relevant as ''-R'' controls the direction of the operation - if it is present the transfer is mirroring '''to''' the server, otherwise it's mirroring '''from''' the server.
 
 
For mirroring the flags ''-R'' and ''-c'' are particularly relevant as ''-R'' controls the direction of the operation - if it is present the transfer is mirroring '''to''' the server, otherwise it's mirroring '''from''' the server.
 
 
 
''-c'' indicates that the operation should resume whenever possible which may improve synchronization time if you know that any partial files present on the other side are identical to the local files.
 
  
 
For example, the command
 
For example, the command
  <nowiki>mirror -c A B</nowiki>
+
  mirror A B
  
 
will download all of the remote directory ''A'' into the local directory named ''B''.
 
will download all of the remote directory ''A'' into the local directory named ''B''.
  
 
The command
 
The command
  <nowiki>mirror -cR C D</nowiki>
+
  mirror -R C D
  
 
will upload all of the local directory ''C'' into the remote directory named ''D''. Note that the role of the directories is reversed compared to the previous example.
 
will upload all of the local directory ''C'' into the remote directory named ''D''. Note that the role of the directories is reversed compared to the previous example.
 
== Build instructions ==
 
 
Building lftp and its dependencies from scratch does not require any particular build flags but you might want to install it into a private destination (prefix) to avoid it interfering with system-provided libraries.
 
 
libnettle depends on GMP and lftp depends on readline and gperf, the distribution packages for those are sufficient on Ubuntu and CentOS 5/7 and 6.2.
 
 
These instructions assume that the ''bash'' shell is used when building, ''tcsh'' and other shells will have slightly different syntax for environment variables.
 
<nowiki>
 
export LFTP_PREFIX="${HOME}/local"
 
export PKG_CONFIG_PATH="${LFTP_PREFIX}/lib/pkgconfig:${PKG_CONFIG_PATH}"
 
export CPPFLAGS="-I${LFTP_PREFIX}/include"
 
export LDFLAGS="-L${LFTP_PREFIX}/lib -Wl,-R${LFTP_PREFIX}/lib"</nowiki>
 
 
Start by extracting the source distributions:
 
<nowiki>
 
tar xzf ~/Downloads/nettle-2.4.tar.gz
 
tar xJf ~/Downloads/gnutls-3.0.12.tar.xz
 
tar xJf ~/Downloads/lftp-4.3.5.tar.xz</nowiki>
 
 
You might want to avoid building a shared gnutls, so passing
 
<nowiki>
 
--enable-static --disable-shared</nowiki>
 
 
on the gnutls configure command line might be a good idea.
 
 
libnettle is a dependency for gnutls, and gnutls is a dependency for lftp, so we build them in that order:
 
<nowiki>
 
mkdir ${LFTP_PREFIX}
 
 
pushd nettle-2.4
 
./configure --prefix=${LFTP_PREFIX} &&
 
make && make install
 
popd
 
 
pushd gnutls-3.0.12
 
./configure --prefix=${LFTP_PREFIX} --without-p11-kit &&
 
make && make install
 
popd
 
 
pushd lftp-4.3.5
 
./configure --prefix=${LFTP_PREFIX} &&
 
make && make install
 
popd</nowiki>
 
 
If the platform already has development files for p11-kit there is no harm in letting it use them (it allows gnutls to understand PKCS-style certificates), but it's nothing that lftp can utilize so it's not considered a dependency in this document.
 
 
After the build process completes, a lftp binary will exist in ''${LFTP_PREFIX}/bin'' and depend on the gnutls shared library in ''${LFTP_PREFIX}/lib'' if you did not build it statically.
 
  
 
= Credits =
 
= Credits =
  
This guide was written by Lars Viklund
+
This guide was initially written by Lars Viklund and subsequently revised by Niklas Edmundsson.

Revision as of 08:22, 14 October 2019

< Swestore

Status

Current lftp version as shipped in most Linux distributions are having issues in presenting a WebDAV directory listing that makes sense to users, among the issues is a tendency to mix files from the root directory and the current directory.

Due to these and other issues we recommend that lftp is not used for file transfers from/to Swestore.

Introduction

lftp is an interactive text-mode file transfer tool that understands multiple access protocols. It is commonly available in Linux/Unix environments.

The benefit over tools like cURL is that it has interactive traversal of the directory hierarchy, as well as powerful mass-transfer functionality like the mirror command, mput/mget and more. Use help in lftp for details. This is not intended to be a complete lftp usage howto, refer to tutorials online for this.

This guide explains how to use lftp with username/password authentication.

Authenticate using username/password

  • Prerequisite: Username and password, see Setting your Swestore password
  • Open a terminal window
  • Start lftp with: lftp -u username https://webdav.swestore.se/
  • Enter your password at the prompt
    • NOTE: Password will be verified on first command/connection, so typing the wrong password will error out later!

Usage

There are some quirks that lftp has together with the Swestore WebDAV door.

When giving a directory path to a command, it should end with a trailing slash to indicate that it is a directory. If this is omitted, the client will get a redirection response that the tool doesn't handle properly.

All the commands mentioned in this section and the previous configuration section are commands inside of a running lftp program.

Some sample tasks that can be achieved with lftp is retrieving or uploading single files or whole directory trees.

Navigate around with the use of the cd command:

cd snic/project_name_here/

Note the trailing / !

Individual files can be manipulated using the get and put commands, while mget and mput transfers multiple files at once. The mirror command can transfer whole trees.

The program has interactive help for any command through the help command.

For mirroring the flag -R is particularly relevant as -R controls the direction of the operation - if it is present the transfer is mirroring to the server, otherwise it's mirroring from the server.

For example, the command

mirror A B

will download all of the remote directory A into the local directory named B.

The command

mirror -R C D

will upload all of the local directory C into the remote directory named D. Note that the role of the directories is reversed compared to the previous example.

Credits

This guide was initially written by Lars Viklund and subsequently revised by Niklas Edmundsson.