Accessing Swestore with lftp
Contents
Introduction
lftp is a file transfer tool that understands a range of protocols from plain old FTP, HTTP, SCP to more esoteric ones like WebDAV and BitTorrent.
The benefit over tools like cURL is that it has interactive traversal of the directory hierarchy, as well as powerful mass-transfer functionality like the mirror command.
Out of the protocols mentioned above, the ones that aligns most with the access doors that Swestore offers are the WebDAV over HTTP/HTTPS protocols.
Authentication against Swestore over the WebDAV door is done with client certificates over HTTPS, where there is a choice of either using the real client certificate or by using a RFC or VOMS proxy certificate generated from the real certificate.
Proxy certificates are preferred as they are valid for a limited period of time, especially as lftp does not seem to offer any way to enter any passphrase to unlock keys, so in order to use a protected real certificate, it will have to be unlocked in advance.
Notable is that while file uploads are securely tunneled inside a SSL connection, downloads are in the plain from storage nodes. As such, the usual guidelines about sensitive data holds, where you should aim to have your data encrypted before transmitting it.
Required software versions
The versions of lftp and its major dependency GNUTLS that have been verified to work with proxy certificates are:
- lftp 4.3.3
- gnutls 2.12.0
- libnettle 2.4 (for building gnutls)
This gnutls version is the absolute minimum version that will work, any version prior to that (2.10.5 and below) will not be able to connect to the door.
This means that the gnutls version in Ubuntu oneric and older, Scientific Linux/RHEL/CentOS 6.1 and older will not work.
Ubuntu precise has a sufficiently new gnutls and will work out of the box.
lftp settings
While running the lftp program, there are several settings that need to be configured in order to successfully connect and interact with Swestore:
set ssl:ca-file /etc/grid-security/certificates/TERENA-eScience-SSL-CA.pem set ssl:check-hostname true set ssl:verify-certificate true
and
set ssl:key-file /tmp/x509up_u1234 set ssl:cert-file /tmp/x509up_u1234
where the last two are to indicate the filename of the proxy certificate generated by the arcproxy or grid-proxy-init tool. The trailing digits in the filename will vary based on the user ID (uid) of the local user, which you can find out in a terminal shell by running the id program.
The CA file is for verifying the identity of the server and is strongly recommended that both certificate and hostname verification is in effect to ensure that the server communicated with is the intended target machine.
They can be stored in the configuration file named ~/.lftp/rc together with any other commands you wish to perform during lftp startup.
Note!
Due to a bug in GnuTLS the verification of host certificates doesn't seem to work. Lftp will return "cd: Fatal error: Certificate verification: Not trusted" when the connection is opened. Please use:
set ssl:verify-certificate false
until this is solved.
Usage
Assuming that you've managed to obtain a working lftp binary, there are some quirks that lftp has together with the Swestore WebDAV door.
When giving a directory path to a command, it should end with a trailing slash to indicate that it is a directory. If this is omitted, the client will get a redirection response that the tool doesn't handle properly.
All the commands mentioned in this section and the previous configuration section are commands inside of a running lftp program.
Some sample tasks that can be achieved with lftp is retrieving or uploading single files or whole directory trees.
The command to connect to the door is:
open https://webdav.swestore.se/
after which you can navigate around with the use of the 'cd' command:
cd snic/project_name_here/foo/
Individual files can be manipulated using the get and put commands and the mget and mirror commands can transfer multiple files and whole trees, respectively.
The program has interactive help for any command through the help command.
For mirroring the flags -R and -c are particularly relevant as -R controls the direction of the operation - if it is present the transfer is mirroring to the server, otherwise it's mirroring from the server.
-c indicates that the operation should resume whenever possible which may improve synchronization time if you know that any partial files present on the other side are identical to the local files.
For example, the command
mirror -c A B
will download all of the remote directory A into the local directory named B.
The command
mirror -cR C D
will upload all of the local directory C into the remote directory named D. Note that the role of the directories is reversed compared to the previous example.
Build instructions
Building lftp and its dependencies from scratch does not require any particular build flags but you might want to install it into a private destination (prefix) to avoid it interfering with system-provided libraries.
libnettle depends on GMP and lftp depends on readline and gperf, the distribution packages for those are sufficient on Ubuntu and CentOS 5/7 and 6.2.
These instructions assume that the bash shell is used when building, tcsh and other shells will have slightly different syntax for environment variables.
export LFTP_PREFIX="${HOME}/local"
export PKG_CONFIG_PATH="${LFTP_PREFIX}/lib/pkgconfig:${PKG_CONFIG_PATH}"
export CPPFLAGS="-I${LFTP_PREFIX}/include"
export LDFLAGS="-L${LFTP_PREFIX}/lib -Wl,-R${LFTP_PREFIX}/lib"
Start by extracting the source distributions:
tar xzf ~/Downloads/nettle-2.4.tar.gz tar xJf ~/Downloads/gnutls-3.0.12.tar.xz tar xJf ~/Downloads/lftp-4.3.5.tar.xz
You might want to avoid building a shared gnutls, so passing
--enable-static --disable-shared
on the gnutls configure command line might be a good idea.
libnettle is a dependency for gnutls, and gnutls is a dependency for lftp, so we build them in that order:
mkdir ${LFTP_PREFIX}
pushd nettle-2.4
./configure --prefix=${LFTP_PREFIX} &&
make && make install
popd
pushd gnutls-3.0.12
./configure --prefix=${LFTP_PREFIX} --without-p11-kit &&
make && make install
popd
pushd lftp-4.3.5
./configure --prefix=${LFTP_PREFIX} &&
make && make install
popd
If the platform already has development files for p11-kit there is no harm in letting it use them (it allows gnutls to understand PKCS-style certificates), but it's nothing that lftp can utilize so it's not considered a dependency in this document.
After the build process completes, a lftp binary will exist in ${LFTP_PREFIX}/bin and depend on the gnutls shared library in ${LFTP_PREFIX}/lib if you did not build it statically.
Credits
This guide was written by Lars Viklund
