https://snicdocs.nsc.liu.se/w/api.php?action=feedcontributions&user=Lars+Viklund+%28HPC2N%29&feedformat=atomSNIC Documentation - User contributions [en]2024-03-28T18:33:20ZUser contributionsMediaWiki 1.31.10https://snicdocs.nsc.liu.se/w/index.php?title=Apply_for_storage_on_Swestore&diff=5840Apply for storage on Swestore2014-05-15T11:19:03Z<p>Lars Viklund (HPC2N): /* Research Communities */</p>
<hr />
<div>[[Category:SweStore]]<br />
[[Category:SweStore user guide]]<br />
[[SweStore|< SweStore]]<br />
<br />
The SweStore nationally accessible storage is available for researchers financed by VR (which includes all researchers using SNIC compute resources) and FORMA.<br />
<br />
= Research Communities =<br />
<br />
SweStore is also in collaboration with [http://www.ecds.se/ ECDS], [http://snd.gu.se/ SND], [http://www.bioimaging.se/swedish_bioimaging_network/Welcome.html Bioimage], [http://www.bils.se/ BILS], [http://www.uppnex.uu.se/ UPPNEX],[http://wlcg.web.cern.ch/ WLCG], [http://www.nrm.se/ Naturhistoriska Riksmuseet]. If any of these cover your research area, first read their information on applying for SweStore storage.<br />
<br />
== UPPNEX ==<br />
If you are a member of an UPPNEX project on UPPMAX, an iRODS service is already available and configured for you, please see http://www.uppmax.uu.se/faq/how-to-move-files-to-swestore-using-irods for more information on how to use it.<br />
<br />
== Other communities ==<br />
Unless you are instructed otherwise, submit an application to SweStore as outlined below.<br />
<br />
= Application instructions =<br />
<br />
Send an email to [mailto:support@swestore.se support@swestore.se] <br />
<br />
Please include the following information in the application:<br><br />
* Whether you want regular SweStore storage (based on dCache) or the new iRODS based storage.<br />
** If you apply for iRODS-storage: please provide a shipping address to where your yubikey should be sent.<br />
* Name of the principal investigator (PI), including email address.<br />
* Purpose for the storage: A short description of the project and type of data.<br />
* Required storage capacity: Preferably a maximum size, but if this is not currently determinable, please calculate a starting size and expansion by time period. '''NOTE''' that applications larger than 10TB takes longer to process.<br />
* Suggested project name: This will be used as root directory name for your storage.<br />
# '''NOTE''' that this name is long-lived and will persist. It is not coupled to the lifetime of SNIC compute time allocations.<br />
# We recommend a project name not tied to a person.<br />
# Additionally, we recommend that the name is not a common word or term easily confusable with other current or future research efforts.<br />
# It is a good idea to select a name that's short and easy to type.<br />
# The name is limited to lower-case letters a-z, digits 0-9, hyphens - and underscores _.</div>Lars Viklund (HPC2N)https://snicdocs.nsc.liu.se/w/index.php?title=Swestore-dCache&diff=5839Swestore-dCache2014-05-15T11:16:57Z<p>Lars Viklund (HPC2N): Link Bioimage Sweden to site provided by Otto Manneberg (RT#100177)</p>
<hr />
<div>[[Category:Storage]]<br />
[[Category:SweStore]]<br />
SNIC is building a storage infrastructure to complement the computational resources.<br />
<br />
Many forms of automated measurements can produce large amounts of data. In scientific areas such as high energy physics (the Large Hadron Collider at CERN), climate modeling, bioinformatics, bioimaging etc., the demands for storage are increasing dramatically. To serve these and other user communities, SNIC has appointed a working group to design a storage strategy, taking into account the needs on many levels and creating a unified storage infrastructure, which is now being implemented.<br />
<br />
Swestore is in collaboration with [http://www.ecds.se/ ECDS], [http://snd.gu.se/ SND], [http://www.bioimaging.se/swedish_bioimaging_network/Welcome.html Bioimage Sweden], [http://www.bils.se/ BILS], [http://www.uppnex.uu.se/ UPPNEX],[http://wlcg.web.cern.ch/ WLCG], [http://www.nrm.se/ NaturHistoriska RiksMuseet].<br />
<br />
= National storage "SweStore"=<br />
The Swestore Nationally Accessible Storage, commonly called just Swestore, is a robust, flexible and expandable long term storage system aimed at storing large amounts of data produced by various Swedish research projects. It is based on the [http://www.dcache.org dCache] and [http://www.irods.org iRODS]<br />
storage systems.<br />
<br />
Swestore is distributed across the SNIC centres [http://www.c3se.chalmers.se/ C3SE], [http://www.hpc2n.umu.se/ HPC2N], [http://www.lunarc.lu.se/ Lunarc], [http://www.nsc.liu.se/ NSC], [http://www.pdc.kth.se PDC] and [http://www.uppmax.uu.se Uppmax]. Data is stored in two copies with each copy at a different SNIC centre. This enables the system to cope with a multitude of issues ranging from a simple crash of a storage element to losing an entire site while still providing access to the stored data. <br />
<br />
One of the major advantages to the distributed nature of dCache and iRODS ([[Swestore-irods]]) is the excellent aggregated transfer rates possible. This is achieved by bypassing a central node and having transfers going directly to/from the storage elements if the protocol allows it. The Swestore Nationally Accessible Storage system can achieve aggregated transfer rates in excess of 100 Gigabit per second, but in practice this is limited by connectivity to each University (usually 10 Gbit/s) or a limited number of files (typically<br />
max 1 Gbit/s per file/connection).<br />
<br />
To protect against silent data corruption the dCache storage system checksums all stored data and periodically verifies the data using this checksum.<br />
<br />
The dCache system does NOT yet provide protection against user errors like inadvertent file deletions and so on. The [[Swestore-irods]] system provides this protection. Deleted files are moved to a trashcan.<br />
<br />
== Getting access ==<br />
; Apply for storage<br />
: Please follow the instructions on the [[Apply for storage on SweStore]] page.<br />
<br />
;Difference between dCache and iRODS user authentication<br />
:SweStore's dCache system uses eScience client certificates.<br />
:SweStore's iRODS system uses [http://www.yubico.com/products/yubikey-hardware/yubikey/ Yubikey] one-time passwords (OTP). With a simple touch of a button, a 44 character one-time password is generated and sent to the system. The user will be provided with a SweStore yubikey.<br />
:Yubikey has a status as pilot now. It can be changed in the future.<br />
<br />
; dCache usage - How to acquire an eScience client certificate<br />
: Follow the instructions on [[Grid_certificates#Requesting_a_certificate|Requesting a certificate]] to get your client certificate. This step can be performed while waiting for the storage application to be approved and processed. Of course, if you already have a valid eScience certificate you don't need to acquire another one.<br />
:; For Terena certificates<br />
:: If intending to access SweStore from a SNIC resource, please make sure you also [[Exporting_a_client_certificate|export the certificate]], transfer it to the intended SNIC resource and [[Preparing_a_client_certificate|prepare it for use with grid tools]] (not necessarily needed with ARC 3.x, see [[Grid_certificates#Creating_a_proxy_certificate_using_the_Firefox.2FThunderbird_credential_store|proxy certificates using Firefox credential store]]).<br />
:; For Nordugrid certificates<br />
:: Please make sure to also [[Requesting_a_grid_certificate_from_the_Nordugrid_CA#Installing_the_certificate_in_your_browser|install your client certificate in your browser]].<br />
:; Request membership in the SweGrid VO<br />
:: Follow the instructions on [[Grid_certificates#Requesting_membership_in_the_SweGrid_VO|Requesting membership in the SweGrid VO]] to get added to the SweGrid Virtual Organisation (VO) and request membership to your allocated storage project.<br />
<br />
; iRODS usage - How to acquire a SweStore yubikey<br />
:Please send an email to [mailto:support@swestore.se?subject=Yubikey support@swestore.se] and provide the shipping address to where the yubikey should be sent.<br><br />
:Yubikey has a status as pilot now. It can be changed in the future.<br />
<br />
== Support == <br />
<br />
If you have any issues using SweStore please do not hesitate to contact [mailto:support@swestore.se support@swestore.se].<br />
<br />
== dCache ==<br />
<br />
=== Access protocols ===<br />
; Currently supported protocols<br />
: GridFTP - gsiftp://gsiftp.swestore.se/<br />
: Storage Resource Manager - srm://srm.swegrid.se/<br />
: Hypertext Transfer Protocol (read-only), Web Distributed Authoring and Versioning - http://webdav.swestore.se/ (unauthenticated), https://webdav.swestore.se/<br />
: NFS4.1<br />
<br />
For authentication eScience certificates are used, which provides a higher level of security than legacy username/password schemes.<br />
<br />
=== Download and upload data ===<br />
; Interactive browsing and manipulation of single files<br />
: SweStore is accessible in your web browser in two ways, as a simple and reliable directory index interface at https://webdav.swestore.se/ and with a richer interactive file manager at https://webdav.swestore.se/browser/. '''Note''' that the interactive file manager has a lot of features and functions not supported in SweStore, only the basic file transfer features are supported.<br />
: To browse private data you need to have your certificate installed in your browser (default with Terena certificates, see above). Projects are organized under the <code>/snic</code> directory as <code><nowiki>https://webdav.swestore.se/snic/YOUR_PROJECT_NAME/</nowiki></code>.<br />
; Upload and delete data interactively or with automation<br />
There are several tools that are capable of using the protocols provided by SweStore national storage.<br />
For interactive usage on SNIC clusters we recommend using the ARC tools which should be installed on all SNIC resources.<br />
As an integration point for building scripts and automated systems we suggest using the curl program and library.<br />
: Use the ARC client. Please see the instructions for [[Accessing SweStore national storage with the ARC client]]. '''Recommended''' method when logged in on SNIC resources.<br />
: Use lftp. Please see the instructions for [[Accessing SweStore national storage with lftp]].<br />
: Use cURL. Please see the instructions for [[Accessing SweStore national storage with cURL]].<br />
: Use globus-url-copy. Please see the instructions for [[Accessing SweStore national storage with globus-url-copy]].<br />
<br />
=== Tools and scripts ===<br />
<br />
There exists a number of tools and utilities developed externally that can be useful. Here are some links:<br />
<br />
* [https://github.com/samuell/arc_tools ARC_Tools] - Convenience scripts for the arc client (Only a recursive rmdir so far).<br />
* [http://sourceforge.net/projects/arc-gui-clients ARC Graphical Clients] - Contains the ARC Storage Explorer (SweStore supported development).<br />
* Transfer script, [[SweStore/swetrans_arc|swetrans_arc]], provided by Adam Peplinski / Philipp Schlatter<br />
* [http://www.nordugrid.org/documents/SWIG-wrapped-ARC-Python-API.pdf Documentation of the ARC Python API (PDF)]<br />
<br />
=== Slides and more ===<br />
<br />
[http://docs.snic.se/wiki/Swestore/Lund_Seminar_Apr18 Slides and material from seminar for Lund users on April 18th]<br />
<br />
=== Usage monitoring ===<br />
* [http://status.swestore.se/munin/monitor/monitor/ Per Project Monitoring of Swestore usage]<br />
<br />
== iRODS ==<br />
<br />
Documentation of the SNIC iRODS system: [[Swestore-irods]].</div>Lars Viklund (HPC2N)https://snicdocs.nsc.liu.se/w/index.php?title=Swestore-dCache&diff=5623Swestore-dCache2013-12-04T13:11:46Z<p>Lars Viklund (HPC2N): /* Download and upload data */</p>
<hr />
<div>[[Category:Storage]]<br />
[[Category:SweStore]]<br />
SNIC is building a storage infrastructure to complement the computational resources.<br />
<br />
Many forms of automated measurements can produce large amounts of data. In scientific areas such as high energy physics (the Large Hadron Collider at CERN), climate modeling, bioinformatics, bioimaging etc., the demands for storage are increasing dramatically. To serve these and other user communities, SNIC has appointed a working group to design a storage strategy, taking into account the needs on many levels and creating a unified storage infrastructure, which is now being implemented.<br />
<br />
Swestore is in collaboration with [http://www.ecds.se/ ECDS], [http://snd.gu.se/ SND], Bioimage Sweden, [http://www.bils.se/ BILS], [http://www.uppnex.uu.se/ UPPNEX],[http://wlcg.web.cern.ch/ WLCG], [http://www.nrm.se/ NaturHistoriska RiksMuseet].<br />
<br />
= National storage =<br />
The Swestore Nationally Accessible Storage, commonly called just Swestore, is a robust, flexible and expandable long<br />
term storage system aimed at storing large amounts of data produced by various Swedish research projects. It is based on the [http://www.dcache.org dCache]<br />
storage system and is distributed across the SNIC centres [http://www.c3se.chalmers.se/ C3SE], [http://www.hpc2n.umu.se/ HPC2N], [http://www.lunarc.lu.se/ Lunarc],<br />
[http://www.nsc.liu.se/ NSC], [http://www.pdc.kth.se PDC] and [http://www.uppmax.uu.se Uppmax].<br />
<br />
Data is stored in two copies with each copy at a different SNIC centre. This enables the system to cope with a multitude of issues ranging from a simple<br />
crash of a storage element to losing an entire site while stil providing access to the stored data. To protect against silent data corruption the<br />
dCache storage system checksums all stored data and periodically verifies the data using this checksum.<br />
<br />
The system does NOT yet provide protection against user errors like inadvertent file deletions and so on.<br />
<br />
One of the major advantages to the distributed nature of dCache is the excellent aggregated transfer rates possible. This is achieved by bypassing a central node<br />
and having transfers going directly to/from the storage elements if the protocol allows it.<br />
The Swestore Nationally Accessible Storage system can achieve aggregated transfer rates<br />
in excess of 100 Gigabit per second, but in practice this is limited by connectivity to each University (usually 10 Gbit/s) or a limited number of files (typically<br />
max 1 Gbit/s per file/connection).<br />
<br />
==Access protocols==<br />
; Currently supported protocols<br />
: GridFTP - gsiftp://gsiftp.swestore.se/<br />
: Storage Resource Manager - srm://srm.swegrid.se/<br />
: Hypertext Transfer Protocol (read-only), Web Distributed Authoring and Versioning - http://webdav.swestore.se/ (unauthenticated), https://webdav.swestore.se/<br />
<br />
; Protocols in evaluation/development<br />
: NFS4.1, iRODS<br />
<br />
For authentication eScience certificates are used, which provides a higher level of security than legacy username/password schemes.<br />
<br />
== Getting access ==<br />
; Apply for storage<br />
: Please follow the instructions on the [[Apply for storage on SweStore]] page.<br />
; Acquire an eScience client certificate<br />
: Follow the instructions on [[Grid_certificates#Requesting_a_certificate|Requesting a certificate]] to get your client certificate. This step can be performed while waiting for the storage application to be approved and processed. Of course, if you already have a valid eScience certificate you don't need to acquire another one.<br />
:; For Terena certificates<br />
:: If intending to access SweStore from a SNIC resource, please make sure you also [[Exporting_a_client_certificate|export the certificate]], transfer it to the intended SNIC resource and [[Preparing_a_client_certificate|prepare it for use with grid tools]] (not necessarily needed with ARC 3.x, see [[Grid_certificates#Creating_a_proxy_certificate_using_the_Firefox.2FThunderbird_credential_store|proxy certificates using Firefox credential store]]).<br />
:; For Nordugrid certificates<br />
:: Please make sure to also [[Requesting_a_grid_certificate_from_the_Nordugrid_CA#Installing_the_certificate_in_your_browser|install your client certificate in your browser]].<br />
; Request membership in the SweGrid VO<br />
: Follow the instructions on [[Grid_certificates#Requesting_membership_in_the_SweGrid_VO|Requesting membership in the SweGrid VO]] to get added to the SweGrid Virtual Organisation (VO) and request membership to your allocated storage project.<br />
<br />
== Download and upload data ==<br />
; Interactive browsing and manipulation of single files<br />
: SweStore is accessible in your web browser in two ways, as a simple and reliable directory index interface at https://webdav.swestore.se/ and with a richer interactive file manager at https://webdav.swestore.se/browser/. '''Note''' that the interactive file manager has a lot of features and functions not supported in SweStore, only the basic file transfer features are supported.<br />
: To browse private data you need to have your certificate installed in your browser (default with Terena certificates, see above). Projects are organized under the <code>/snic</code> directory as <code><nowiki>https://webdav.swestore.se/snic/YOUR_PROJECT_NAME/</nowiki></code>.<br />
; Upload and delete data interactively or with automation<br />
There are several tools that are capable of using the protocols provided by SweStore national storage.<br />
For interactive usage on SNIC clusters we recommend using the ARC tools which should be installed on all SNIC resources.<br />
As an integration point for building scripts and automated systems we suggest using the curl program and library.<br />
: Use the ARC client. Please see the instructions for [[Accessing SweStore national storage with the ARC client]]. '''Recommended''' method when logged in on SNIC resources.<br />
: Use lftp. Please see the instructions for [[Accessing SweStore national storage with lftp]].<br />
: Use cURL. Please see the instructions for [[Accessing SweStore national storage with cURL]].<br />
: Use globus-url-copy. Please see the instructions for [[Accessing SweStore national storage with globus-url-copy]].<br />
<br />
== More information ==<br />
* [http://status.swestore.se/munin/monitor/monitor/ Per Project Monitoring of Swestore usage]<br />
<br />
If you have any issues using SweStore please do not hesitate to contact [mailto:support@swestore.se support@swestore.se].<br />
<br />
== Tools and scripts ==<br />
<br />
There exists a number of tools and utilities developed externally that can be useful. Here are some links:<br />
<br />
* [https://github.com/samuell/arc_tools ARC_Tools] - Convenience scripts for the arc client (Only a recursive rmdir so far).<br />
* [http://sourceforge.net/projects/arc-gui-clients ARC Graphical Clients] - Contains the ARC Storage Explorer (SweStore supported development).<br />
* Transfer script, [[SweStore/swetrans_arc|swetrans_arc]], provided by Adam Peplinski / Philipp Schlatter<br />
* [http://www.nordugrid.org/documents/SWIG-wrapped-ARC-Python-API.pdf Documentation of the ARC Python API (PDF)]<br />
<br />
== Slides and more ==<br />
<br />
[http://docs.snic.se/wiki/Swestore/Lund_Seminar_Apr18 Slides and material from seminar for Lund users on April 18th]<br />
<br />
= Center storage =<br />
Centre storage, as defined by the SNIC storage group, is a storage solution that lives independently of the computational resources and can be accessed from all such resources at a centre. Key features include the ability to access the same filesystem the same way on all computational resources at a centre, and a unified structure and nomenclature for all centra. Unlike cluster storage which is tightly associated with a single cluster, and thus has a limited life-time, centre storage does not require the users to migrate their own data when clusters are decommissioned, not even when the storage hardware itself is being replaced.<br />
<br />
== Unified environment ==<br />
To make the usage more transparent for SNIC users, a set of environment variables are available on all SNIC resources:<br />
<br />
* <code>SNIC_BACKUP</code> – the user's primary directory at the centre<br>(the part of the centre storage that is backed up)<br />
* <code>SNIC_NOBACKUP</code> – recommended directory for project storage without backup<br>(also on the centre storage)<br />
* <code>SNIC_TMP</code> – recommended directory for best performance during a job<br>(local disk on nodes if applicable)</div>Lars Viklund (HPC2N)https://snicdocs.nsc.liu.se/w/index.php?title=Accessing_Swestore_with_cURL&diff=5608Accessing Swestore with cURL2013-11-21T16:00:15Z<p>Lars Viklund (HPC2N): /* Optional parameters */</p>
<hr />
<div>[[Category:SweStore]]<br />
[[Category:SweStore user guide]]<br />
[[SweStore|< SweStore]]<br />
<br />
<br />
This guide outlines the procedure for using cURL to access files through the WebDav door of dCache.<br />
<br />
== Essential parameters ==<br />
<br />
--capath /etc/grid-security/certificates<br />
The certificate bundle provided through --capath is required in order for cURL to accept the server certificates the door presents. If the certificate bundle is not available, the -k flag may be passed to allow untrusted server certificates.<br />
<br />
--cert /tmp/x509up_u1234<br />
--cert (or -E) names the proxy certificate generated by arcproxy or similar tools, which is a single PEM file consisting of the client certificate, the proxy key and the proxy certificate. The name will vary based on the user issuing it.<br />
grid-proxy-init (and thus arcproxy) will put the certificate in /tmp by default and name it according to the pattern x509up_u<NumericUID>. The -out parameter to grid-proxy-init takes a location to store the certificate in if the default is not sufficient.<br />
<br />
--location<br />
--location (or -L) instructs cURL to follow HTTP redirects, in this case the 302 redirects that the dCache door uses to direct clients to different storage nodes.<br />
<br />
== Optional parameters ==<br />
<br />
--sslv3<br />
There exists older cURL versions out there which still prefer SSLv2 when making a connection. They will fail to connect to SweStore national storage with an error along the lines of <tt>curl: (35) Unknown SSL protocol error in connection to ...</tt>. If you use such a client and cannot upgrade or otherwise circumvent the problem, --sslv3 (or -3) instructs those versions of cURL to force SSLv3. This parameter is only recommended and needed for older versions, if you use it you should re-evaluate your need for it whenever you end up upgrading cURL to see if it's still required. Using it unnecessarily for newer versions of cURL that do not exhibit the problem will reduce their choice of SSL/TLS versions and ultimately reduce security strength.<br />
<br />
== Sample invocations ==<br />
<br />
Downloads the file 'file-to-download.ext':<br />
curl --location --capath /etc/grid-security/certificates --cert /tmp/x509up_u1234 -O https://webdav.swegrid.se/target/path/file-to-download.ext<br />
<br />
Upload the file 'source.file' as 'uploaded.ext':<br />
curl --location --capath /etc/grid-security/certificates --cert /tmp/x509up_u1234 -T ~/source.file https://webdav.swegrid.se/target/path/uploaded.ext<br />
<br />
= Credits =<br />
<br />
This guide was written by Lars Viklund</div>Lars Viklund (HPC2N)https://snicdocs.nsc.liu.se/w/index.php?title=Accessing_Swestore_with_cURL&diff=5607Accessing Swestore with cURL2013-11-21T15:58:36Z<p>Lars Viklund (HPC2N): Add section on SSLv3 as the problem has been mentioned on dCache user-forum mailing list</p>
<hr />
<div>[[Category:SweStore]]<br />
[[Category:SweStore user guide]]<br />
[[SweStore|< SweStore]]<br />
<br />
<br />
This guide outlines the procedure for using cURL to access files through the WebDav door of dCache.<br />
<br />
== Essential parameters ==<br />
<br />
--capath /etc/grid-security/certificates<br />
The certificate bundle provided through --capath is required in order for cURL to accept the server certificates the door presents. If the certificate bundle is not available, the -k flag may be passed to allow untrusted server certificates.<br />
<br />
--cert /tmp/x509up_u1234<br />
--cert (or -E) names the proxy certificate generated by arcproxy or similar tools, which is a single PEM file consisting of the client certificate, the proxy key and the proxy certificate. The name will vary based on the user issuing it.<br />
grid-proxy-init (and thus arcproxy) will put the certificate in /tmp by default and name it according to the pattern x509up_u<NumericUID>. The -out parameter to grid-proxy-init takes a location to store the certificate in if the default is not sufficient.<br />
<br />
--location<br />
--location (or -L) instructs cURL to follow HTTP redirects, in this case the 302 redirects that the dCache door uses to direct clients to different storage nodes.<br />
<br />
== Optional parameters ==<br />
<br />
--sslv3<br />
There exists older cURL versions out there which still prefer SSLv2 when making a connection. They will fail to connect to SweStore national storage with an error along the lines of "curl: (35) Unknown SSL protocol error in connection to ...". If you use such a client and cannot upgrade or otherwise circumvent the problem, --sslv3 (or -3) instructs those versions of cURL to force SSLv3. This parameter is only recommended and needed for older versions, if you use it you should re-evaluate your need for it whenever you end up upgrading cURL to see if it's still required. Using it unnecessarily for newer versions of cURL that do not exhibit the problem will reduce their choice of SSL/TLS versions and ultimately reduce security strength.<br />
<br />
== Sample invocations ==<br />
<br />
Downloads the file 'file-to-download.ext':<br />
curl --location --capath /etc/grid-security/certificates --cert /tmp/x509up_u1234 -O https://webdav.swegrid.se/target/path/file-to-download.ext<br />
<br />
Upload the file 'source.file' as 'uploaded.ext':<br />
curl --location --capath /etc/grid-security/certificates --cert /tmp/x509up_u1234 -T ~/source.file https://webdav.swegrid.se/target/path/uploaded.ext<br />
<br />
= Credits =<br />
<br />
This guide was written by Lars Viklund</div>Lars Viklund (HPC2N)https://snicdocs.nsc.liu.se/w/index.php?title=Grid_certificates&diff=5588Grid certificates2013-11-05T12:10:50Z<p>Lars Viklund (HPC2N): /* Requesting a certificate */</p>
<hr />
<div>[[Category:Grid computing]]<br />
[[Category:SweGrid user guide]]<br />
[[Category:SweStore]]<br />
[[Category:SweStore user guide]]<br />
[[Getting started with SweGrid|< Getting started with SweGrid]]<br><br />
[[SweStore|< SweStore]]<br />
<br />
=Introduction to certificates=<br />
<br />
In order to get access to computer and storage resources on the grid or [[SweStore]] you must have a valid (grid) certificate. This certificate is used instead of a username and password when accessing the resource. The resource have a certificate that tells you that you have contacted the right resource. This is exactly the same mechanism used when you use a web browser to contact your bank.<br />
<br />
A certificate is the similar to a passport in real-life. In the same way you have prove your credentials when you acquire a passport the same is true for a certificate. A third party, the Certificate Authority or CA, that both you and the resource trust has to vouch for your identity and sign your certificate.<br />
<br />
A certificate consist of a public key, some user information and a signature of the CA. In addition to the certificate you have a private key. The private key is secret and should be kept as secure as possible.<br />
<br />
For more information regarding certificates and public key cryptography:<br />
<br />
[http://en.wikipedia.org/wiki/Public-key_cryptography http://en.wikipedia.org/wiki/Public-key_cryptography]<br />
<br />
[http://en.wikipedia.org/wiki/Public_key_certificate http://en.wikipedia.org/wiki/Public_key_certificate]<br />
<br />
[http://www.nordugrid.org/documents/certificate_howto.html http://www.nordugrid.org/documents/certificate_howto.html]<br />
<br />
* The grid certificate and the private key are stored in your web browser and/or located in ~/.globus at the host(s) from where you will be accessing the resource:<br />
usercert.pem<br />
userkey.pem<br />
* The certificate contains your public key, your name and organization and a signature by the CA. It is does not contain any username.<br />
* The certificate is valid for 13 month and should be renewed yearly.<br />
* The private key should be handled with great care. It should only be readable by you and not by the group or others (i.e. ``chmod 400 userkey.pem''). Store the key on trusted computers and transfer the key between computers using encryption (using for example scp).<br />
* On shared file systems make sure that ~/.globus is not readible by everybody:<br />
chmod 700 ~/.globus<br />
and on AFS:<br />
fs sa ~/.globus system:anyuser none<br />
* The private key is encrypted using a passphrase. Anyone that can decrypt the private key will be able to authenticate as you to grid resources. This is similar to the private key in SSH. You must choose a strong passphrase for the private key. This passphrase must not be used anywhere else. You must never ever give away the passphrase to somebody else.<br />
* You should not share the certificate with someone. It's personal. <br />
<br />
For more information regarding certificates and public key cryptography:<br />
<br />
[http://en.wikipedia.org/wiki/Public-key_cryptography http://en.wikipedia.org/wiki/Public-key_cryptography]<br />
[http://en.wikipedia.org/wiki/Public_key_certificate http://en.wikipedia.org/wiki/Public_key_certificate]<br />
<br />
= Requesting a certificate =<br />
<br />
Certificates are issued by a Certificate Authority or CA. The certificate needed for this purpose is an ''eScience Personal'' certificate, not all CA:s are certified by [http://www.igtf.net/ The International Grid Trust Federation] to issue these.<br />
<br />
For users residing in Sweden there are two relevant CA:s that can issue grid/eScience/e-Science certificates: ''Terena'' and ''Nordugrid''. The Terena CA is preferred if it is available for your university or research group, but many sites has not enabled this service yet. The Nordugrid CA can also be used but requires more manual work by all parties.<br />
<br />
For researchers residing outside of Sweden we refer to your closest CA, but remember that you need an ''eScience Personal'' certificate! http://www.terena.org/activities/tcs/participants.html lists the Terena members in the EU. Additionally, http://www.eugridpma.org/members/worldmap/ is a map of countries with additional CA:s. If you are located outside the EU, see http://www.igtf.net/ for your closest registry.<br />
<br />
Recommended procedure for each university:<br />
<br />
{| class="wikitable"<br />
| University<br />
| CA<br />
| Specific instructions<br />
|-<br />
| Chalmers University of Technology (CTH)<br />
| NorduGrid CA<br />
| [[Chalmers_Certificate_Instructions|more...]]<br />
|-<br />
| University of Gothenburg (GU)<br />
| NorduGrid CA<br />
| [[GU_Certificate_Instructions|more...]]<br />
|-<br />
| Karolinska Institutet (KI)<br />
| Terena CA<br />
| [[KI_Certificate_Information|more...]]<br />
|-<br />
| KTH Royal Institute of Technology (KTH)<br />
| Terena CA<br />
| [[KTH_Certificate_Information|more...]]<br />
|-<br />
| Linköping University (LiU)<br />
| Terena CA<br />
| [[LiU_Certificate_Instructions|more...]]<br />
|-<br />
| Luleå University of Technology (LTU)<br />
| NorduGrid CA<br />
| N/A<br />
|-<br />
| Lund University (LU)<br />
| Terena CA<br />
| [[LU_Certificate_Information|more...]]<br />
|-<br />
| Stockholm University (SU)<br />
| NorduGrid CA<br />
| [[SU_Certificate_Information|more...]]<br />
|-<br />
| Umeå University (UmU)<br />
| Terena CA<br />
| [[UmU_Certificate_Information|more...]]<br />
|-<br />
| University of Borås (UB)<br />
| Terena CA<br />
| N/A<br />
|-<br />
| Uppsala University (UU)<br />
| Terena CA<br />
| [[UU_Certificate_Instructions|more...]]<br />
|-<br />
|}<br />
<br />
[[Requesting a grid certificate using the Terena eScience Portal|Instructions for the Terena CA]]<br />
<br />
[[Requesting a grid certificate from the Nordugrid CA|Instructions for the NorduGrid CA (use only if Terena eScience isn't available at your site)]]<br />
<br />
= Requesting membership in the SweGrid VO =<br />
<br />
== Introduction ==<br />
<br />
SweGrid and SweStore resources are currently being allocated for VO:s, virtual organizations, rather than individual users. A VO is basically just a list of users. To be able to use a SweGrid or SweStore resource a membership in the SweGrid VO (virtual organization) and a corresponding subgroup is required.<br />
<br />
== Preparations ==<br />
<br />
To apply for membership, make sure that the NorduGrid root CA certificate and your personal certificate is installed in the browser. <br />
<br />
The NorduGrid CA certificate can be installed by clicking on the following link:<br />
<br />
[http://ca.nordugrid.org/cacrt.crt http://ca.nordugrid.org/cacrt.crt]<br />
<br />
Make sure you check the "Trust this CA to identify web sites." boxes in the dialog shown.<br />
<br />
<br />
[[File:certinstall.png]]<br />
<br />
== Step 1 - Apply for VO membership ==<br />
<br />
When the NorduGrid CA certificate have been installed in the browser go to the following URL:<br />
<br />
[https://voms.ndgf.org:8443/voms/swegrid.se https://voms.ndgf.org:8443/voms/swegrid.se]<br />
<br />
and follow the instructions. After a manual review, normally within a couple of hours, you will be added to the SweGrid VO.<br />
<br />
== Step 2 - Request group membership ==<br />
<br />
After being added to the Swegrid VO you need to be added to the correct project/allocation group to use that allocation. Use the '''Request membership''' function in the '''Your groups and roles section''' of your VOMS homepage at https://voms.ndgf.org:8443/voms/swegrid.se/user/home.action as shown in the following screenshot, selecting the project in the dropdown box and clicking the '''Request membership''' button. No further actions need to be taken on that page after requesting the membership.<br />
<br />
[[File:request-vo-membership.png]]<br />
<br />
The request is handled manually, usually within a few hours. Allow for a couple more hours for the membership to propagate to SweStore.<br />
<br />
== If it doesn't work ==<br />
<br />
If things doesn't work for some reason, contact SweGrid support at [mailto:support@swegrid.se support@swegrid.se] or SweStore support at [mailto:support@swestore.se support@swestore.se] as appropriate.<br />
<br />
= Proxy certificates =<br />
<br />
Authentication on the grid is done using special short lived ''proxy'' certificates. There are several tools available for creating, checking and destroying these proxy certificates.<br />
The examples below demonstrates the '''arcproxy''' command from the ARC software suite.<br />
<br />
== Creating a proxy certificate ==<br />
<br />
This example requires that the certificate is available for use with grid tools. This is the default with '''Nordugrid certificates''', although you might need<br />
to transfer the certificate to the resource where you are using the grid tools.<br />
<br />
For '''Terena certificates''' you must first [[Exporting_a_client_certificate|export the certificate]], transfer it to the resource where you are using the grid tools if needed<br />
and [[Preparing_a_client_certificate|prepare it for use with grid tools]].<br />
If you have ARC 3.x or newer installed it can use the Firefox certificate store directly, as described in the next section.<br />
<br />
To create a short lived proxy that can be used for authentication with grid services, the '''arcproxy''' command can be used. A 12 hour (default) proxy is created in the following example:<br />
<br />
$ arcproxy<br />
Your identity: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=Kalle Kula<br />
Enter pass phrase for /home/kalle/.globus/userkey.pem:<br />
.++++++<br />
.....++++++<br />
Proxy generation succeeded<br />
Your proxy is valid until: 2011-03-11 03:00:14<br />
<br />
The proxy file itself will be created in the '''/tmp''' directory with the format '''x509up_uid''', where uid is the user id number for your account.<br />
<br />
In some cases a longer lived proxy will be needed. This is achieved using the '''--constraint''' switch. A 24-hour can be created by issuing the following command:<br />
<br />
$ arcproxy --constraint="validityPeriod=24H"<br />
Your identity: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=Kalle Kula<br />
Enter pass phrase for /home/kalle/.globus/userkey.pem:<br />
....++++++<br />
.....++++++<br />
Proxy generation succeeded<br />
Your proxy is valid until: 2011-03-11 15:03:19<br />
<br />
== Creating a proxy certificate using the Firefox/Thunderbird credential store ==<br />
<br />
Using the ARC 3.x client tools it is now possible to generate a proxy certificate directly from the Firefox or Thunderbird credential stores. To do this the '''-F''' flag is used as shown in the following example:<br />
<br />
$ arcproxy -F<br />
There are 2 NSS base directories where the certificate, key, and module datbases live<br />
Number 1 is: /Users/lindemann/Library/Application Support/Firefox/Profiles/t22f3aj2.default<br />
Number 2 is: /Users/lindemann/Library/Thunderbird/Profiles/7abb733v.default<br />
Please choose the NSS database you would use (1-2): 1<br />
<br />
Here ARC finds the available Firefox and Thunderbird profile in which the credential stores are stored. Next the passphrase for the credential store is used to unlock the stored credentials:<br />
<br />
NSS database to be accessed: /Users/lindemann/Library/Application Support/Firefox/Profiles/t22f3aj2.default<br />
Enter Password or Pin for "internal (software)":<br />
<br />
If the passphrase was correct, ARC will list the available certificates in the credential store and ask you for which you would like to use.<br />
<br />
There are 2 user certificates existing in the NSS database<br />
Number 1 is with nickname: Jonas Lindemann xxxxx@lu.se's TERENA ID (Jonas Lindemann xxxxx@lu.se)<br />
expiration time: 2013-06-04 01:59:59<br />
Number 2 is with nickname: Imported Certificate (Jonas Lindemann)<br />
expiration time: 2014-01-18 16:55:52<br />
Please choose the one you would use (1-2): 1<br />
Certificate to use is: Jonas Lindemann xxxxxx@lu.se's TERENA ID<br />
Proxy generation succeeded<br />
Your proxy is valid until: 2013-05-01 04:11:37<br />
<br />
== Checking proxy lifetime ==<br />
<br />
The remaining lifetime of a proxy certificate can be checked using the '''arcproxy''' command with the '''--info''' switch.<br />
<br />
$ arcproxy --info<br />
Subject: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=Kalle Kula/CN=1567862803<br />
Identity: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=Kalle Kula<br />
Time left for proxy: 11 hours 55 minutes<br />
Proxy path: /tmp/x509up_u500<br />
Proxy type: X.509 Proxy Certificate Profile RFC compliant restricted proxy<br />
<br />
In this example the proxy certificate is valid for 11 hours 55 minutes more.<br />
<br />
== Destroying a proxy certificate ==<br />
<br />
A proxy can be destroyed with the '''-r''' or '''--remove''' switch.<br />
<br />
$ arcproxy -r<br />
<br />
or<br />
<br />
$ arcproxy --remove<br />
<br />
= VOMS certificates =<br />
<br />
As long as you are a member of only one VO or VO group, you can<br />
authenticate to a grid service with the regular grid proxy certificate<br />
as defined in the previous section. If you are a member of more than<br />
one VO or VO group you may want to select which membership you want to<br />
be authenticated as. For example, if you are a member of<br />
''swegrid.se:/swegrid.se/ops'' (operations staff) and<br />
''swegrid.se:/swegrid.se/bils'' and want to write a file, who should<br />
be the owner? Ops or bils? You need to provide some additional<br />
information. In the grid world this is done with a voms proxy<br />
certificate which basically is a regular proxy certificate but with a<br />
so called voms extension that contains a list of your VO group<br />
memberships (and roles and attributes, which we don't use in<br />
Swegrid/Swestore at the moment).<br />
<br />
'''Please note, if you only have one membership you can skip this section!'''<br />
<br />
The voms extension of the certificate is signed by the virtual<br />
organization management server, or VOMS server. The same VOMS server<br />
you used when applying for the swegrid.se VO membership in the first<br />
place. To enable this signing process you need to add a few<br />
configuration files to your system. First add this to the file<br />
'''/etc/vomses''':<br />
<br />
"swegrid.se" "voms.ndgf.org" "15009" "/O=Grid/O=NorduGrid/CN=host/voms.ndgf.org" "swegrid.se"<br />
<br />
Next create the necessary directories and the file<br />
'''/etc/grid-security/vomsdir/swegrid.se/voms.ndgf.org.lsc''' with the<br />
following contents:<br />
<br />
/O=Grid/O=NorduGrid/CN=host/voms.ndgf.org<br />
/O=Grid/O=NorduGrid/CN=NorduGrid Certification Authority<br />
<br />
== Creating a VOMS proxy ==<br />
<br />
VOMS proxies in ARC1 can be created using the '''arcproxy''' command<br />
and the '''-S''' or '''--voms''' switches as shown in the following<br />
example (if you are a member of the /swegrid.se/ops group. Adjust as<br />
necessary):<br />
<br />
$ arcproxy -S swegrid.se:/swegrid.se/ops<br />
Your identity: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=Kalle Kula<br />
Enter pass phrase for /home/kalle/.globus/userkey.pem:<br />
.....++++++<br />
............++++++<br />
Contacting VOMS server (named swegrid.se): voms.ndgf.org on port: 15009<br />
Proxy generation succeeded<br />
Your proxy is valid until: 2011-03-10 23:33:06<br />
<br />
<br />
= Signing your e-mail with your certificate =<br />
<br />
First, you will need your grid certificate in PKCS12 format:<br />
== How to transform your certificate from PEM format into PKCS#12 format ==<br />
<br />
This is how you transform your cert into PKCS12 format that can be used within your webbrowser or email send program:<br />
You first will have to change directory into where you created and keep the certificate, historically this is often in ~/.globus<br />
<br />
openssl pkcs12 -export -in usercert.pem -inkey userkey.pem -out cert+key.p12 <br />
<br />
First you will have to enter the password you used for your private key, then you will be asked for a new password to protect the new file. '''cert+key.p12 contains your private key, and is therefore 'lika känslig' as userkey.pem'''. See also [[#Introduction to certificates]]. Security wise the safest way is to delete the PKCS12 file after having imported it into your mail client or browser. Don't forget this.<br />
<br />
Remarks: openssl will either need the variable RANDFILE to be set or that ~/.rnd is writable. So you have to make sure that the current $HOME is yours if you have pagshed away, otherwise the command will fail with ''unable to write 'random state''.<br />
<br />
<br />
=== Signing in mew ===<br />
<br />
Mew uses gpgsm. <br />
<br />
<pre><br />
1. Import the nordugrid root cert<br />
<br />
1.1. get 1f0e8352.0 from nordugrid web<br />
<br />
1.2. gpgsm --import 1f0e8352.0<br />
<br />
1.2. Make it trusted:<br />
gpgsm --list-keys 2>/dev/null | grep fingerprint | awk '{print $2 " S"}' | grep THE-FINGERPRIT-YOU-WANT >> .gnupg/trustlist.txt<br />
<br />
2. Add your own key from the cert+key.p12 file in this case<br />
<br />
2.1 openssl pkcs12 -in cert+key.p12 -out tmp.pem -nokeys<br />
<br />
2.2. gpgsm --import tmp.pem ; rm tmp.pem<br />
<br />
2.3. Tell gpgsm not to use revocation lists (bad bad security)<br />
echo disable-crl-checks >> .gnupg/gpgsm.conf<br />
<br />
3. Test<br />
gpgsm --detach-sign file > sign # should ask for passphrase and give some kind of sign file<br />
<br />
4. Use:<br />
C-uC-cC-s then enter your email address (must match email in cert) and passphrase<br />
<br />
</pre><br />
<br />
=== Signing in thunderbird ===<br />
In thunderbird: options/security/digitally sign this message.<br />
<br />
If you do this for the first time and haven't defined yet the certificate to sign with, thunderbird will pop up the according preferences [Account settings/Security], where you can choose between your imported certificates in PKCS12 format.<br />
<br />
In the beginning, of course, you haven't imported any: Click there on the same preferences tab that popped up on [View Certificates]. In the new window that opens you can import the certificate.<br />
<br />
Afterwards you can then choose this certificate to be used for signing and for encryption for this email account.<br />
<br />
Don't forget to actually check that you then really sign the corresponding mail.</div>Lars Viklund (HPC2N)https://snicdocs.nsc.liu.se/w/index.php?title=Grid_certificates&diff=5477Grid certificates2013-10-30T09:57:11Z<p>Lars Viklund (HPC2N): /* Requesting a certificate */</p>
<hr />
<div>[[Category:Grid computing]]<br />
[[Category:SweGrid user guide]]<br />
[[Category:SweStore]]<br />
[[Category:SweStore user guide]]<br />
[[Getting started with SweGrid|< Getting started with SweGrid]]<br><br />
[[SweStore|< SweStore]]<br />
<br />
=Introduction to certificates=<br />
<br />
In order to get access to computer and storage resources on the grid or [[SweStore]] you must have a valid (grid) certificate. This certificate is used instead of a username and password when accessing the resource. The resource have a certificate that tells you that you have contacted the right resource. This is exactly the same mechanism used when you use a web browser to contact your bank.<br />
<br />
A certificate is the similar to a passport in real-life. In the same way you have prove your credentials when you acquire a passport the same is true for a certificate. A third party, the Certificate Authority or CA, that both you and the resource trust has to vouch for your identity and sign your certificate.<br />
<br />
A certificate consist of a public key, some user information and a signature of the CA. In addition to the certificate you have a private key. The private key is secret and should be kept as secure as possible.<br />
<br />
For more information regarding certificates and public key cryptography:<br />
<br />
[http://en.wikipedia.org/wiki/Public-key_cryptography http://en.wikipedia.org/wiki/Public-key_cryptography]<br />
<br />
[http://en.wikipedia.org/wiki/Public_key_certificate http://en.wikipedia.org/wiki/Public_key_certificate]<br />
<br />
[http://www.nordugrid.org/documents/certificate_howto.html http://www.nordugrid.org/documents/certificate_howto.html]<br />
<br />
* The grid certificate and the private key are stored in your web browser and/or located in ~/.globus at the host(s) from where you will be accessing the resource:<br />
usercert.pem<br />
userkey.pem<br />
* The certificate contains your public key, your name and organization and a signature by the CA. It is does not contain any username.<br />
* The certificate is valid for 13 month and should be renewed yearly.<br />
* The private key should be handled with great care. It should only be readable by you and not by the group or others (i.e. ``chmod 400 userkey.pem''). Store the key on trusted computers and transfer the key between computers using encryption (using for example scp).<br />
* On shared file systems make sure that ~/.globus is not readible by everybody:<br />
chmod 700 ~/.globus<br />
and on AFS:<br />
fs sa ~/.globus system:anyuser none<br />
* The private key is encrypted using a passphrase. Anyone that can decrypt the private key will be able to authenticate as you to grid resources. This is similar to the private key in SSH. You must choose a strong passphrase for the private key. This passphrase must not be used anywhere else. You must never ever give away the passphrase to somebody else.<br />
* You should not share the certificate with someone. It's personal. <br />
<br />
For more information regarding certificates and public key cryptography:<br />
<br />
[http://en.wikipedia.org/wiki/Public-key_cryptography http://en.wikipedia.org/wiki/Public-key_cryptography]<br />
[http://en.wikipedia.org/wiki/Public_key_certificate http://en.wikipedia.org/wiki/Public_key_certificate]<br />
<br />
= Requesting a certificate =<br />
<br />
Certificates are issued by a Certificate Authority or CA. The certificate needed for this purpose is an ''eScience Personal'' certificate, not all CA:s are certified by [http://www.igtf.net/ The International Grid Trust Federation] to issue these.<br />
<br />
For users residing in Sweden there are two relevant CA:s that can issue grid/eScience/e-Science certificates: ''Terena'' and ''Nordugrid''. The Terena CA is preferred if it is available for your university or research group, but many sites has not enabled this service yet. The Nordugrid CA can also be used but requires more manual work by all parties.<br />
<br />
For researchers residing outside of Sweden we refer to your closest CA, but remember that you need an ''eScience Personal'' certificate! http://www.terena.org/activities/tcs/participants.html lists the Terena members in the EU. Additionally, http://www.eugridpma.org/members/worldmap/ is a map of countries with additional CA:s. If you are located outside the EU, see http://www.igtf.net/ for your closest registry.<br />
<br />
Recommended procedure for each university:<br />
<br />
{| class="wikitable"<br />
| University<br />
| CA<br />
| Specific instructions<br />
|-<br />
| Chalmers University of Technology (CTH)<br />
| NorduGrid CA<br />
| [[Chalmers_Certificate_Instructions|more...]]<br />
|-<br />
| University of Gothenburg (GU)<br />
| NorduGrid CA<br />
| [[GU_Certificate_Instructions|more...]]<br />
|-<br />
| Karolinska Institutet (KI)<br />
| Terena CA<br />
| [[KI_Certificate_Information|more...]]<br />
|-<br />
| KTH Royal Institute of Technology (KTH)<br />
| Terena CA<br />
| [[KTH_Certificate_Information|more...]]<br />
|-<br />
| Linköping University (LiU)<br />
| Terena CA<br />
| [[LiU_Certificate_Instructions|more...]]<br />
|-<br />
| Luleå University of Technology (LTU)<br />
| NorduGrid CA<br />
| [[LTH_Certificate_Instructions|more...]]<br />
|-<br />
| Lund University (LU)<br />
| Terena CA<br />
| [[LU_Certificate_Information|more...]]<br />
|-<br />
| Stockholm University (SU)<br />
| NorduGrid CA<br />
| [[SU_Certificate_Information|more...]]<br />
|-<br />
| Umeå University (UmU)<br />
| Terena CA<br />
| [[UmU_Certificate_Information|more...]]<br />
|-<br />
| Uppsala University (UU)<br />
| Terena CA<br />
| [[UU_Certificate_Instructions|more...]]<br />
|-<br />
|}<br />
<br />
[[Requesting a grid certificate using the Terena eScience Portal|Instructions for the Terena CA]]<br />
<br />
[[Requesting a grid certificate from the Nordugrid CA|Instructions for the NorduGrid CA (use only if Terena eScience isn't available at your site)]]<br />
<br />
= Requesting membership in the SweGrid VO =<br />
<br />
== Introduction ==<br />
<br />
SweGrid and SweStore resources are currently being allocated for VO:s, virtual organizations, rather than individual users. A VO is basically just a list of users. To be able to use a SweGrid or SweStore resource a membership in the SweGrid VO (virtual organization) and a corresponding subgroup is required.<br />
<br />
== Preparations ==<br />
<br />
To apply for membership, make sure that the NorduGrid root CA certificate and your personal certificate is installed in the browser. <br />
<br />
The NorduGrid CA certificate can be installed by clicking on the following link:<br />
<br />
[http://ca.nordugrid.org/cacrt.crt http://ca.nordugrid.org/cacrt.crt]<br />
<br />
Make sure you check the "Trust this CA to identify web sites." boxes in the dialog shown.<br />
<br />
<br />
[[File:certinstall.png]]<br />
<br />
== Step 1 - Apply for VO membership ==<br />
<br />
When the NorduGrid CA certificate have been installed in the browser go to the following URL:<br />
<br />
[https://voms.ndgf.org:8443/voms/swegrid.se https://voms.ndgf.org:8443/voms/swegrid.se]<br />
<br />
and follow the instructions. After a manual review, normally within a couple of hours, you will be added to the SweGrid VO.<br />
<br />
== Step 2 - Request group membership ==<br />
<br />
After being added to the Swegrid VO you need to be added to the correct project/allocation group to use that allocation. Use the '''Request membership''' function in the '''Your groups and roles section''' of your VOMS homepage at https://voms.ndgf.org:8443/voms/swegrid.se/user/home.action as shown in the following screenshot, selecting the project in the dropdown box and clicking the '''Request membership''' button. No further actions need to be taken on that page after requesting the membership.<br />
<br />
[[File:request-vo-membership.png]]<br />
<br />
The request is handled manually, usually within a few hours. Allow for a couple more hours for the membership to propagate to SweStore.<br />
<br />
== If it doesn't work ==<br />
<br />
If things doesn't work for some reason, contact SweGrid support at [mailto:support@swegrid.se support@swegrid.se] or SweStore support at [mailto:support@swestore.se support@swestore.se] as appropriate.<br />
<br />
= Proxy certificates =<br />
<br />
Authentication on the grid is done using special short lived ''proxy'' certificates. There are several tools available for creating, checking and destroying these proxy certificates.<br />
The examples below demonstrates the '''arcproxy''' command from the ARC software suite.<br />
<br />
== Creating a proxy certificate ==<br />
<br />
This example requires that the certificate is available for use with grid tools. This is the default with '''Nordugrid certificates''', although you might need<br />
to transfer the certificate to the resource where you are using the grid tools.<br />
<br />
For '''Terena certificates''' you must first [[Exporting_a_client_certificate|export the certificate]], transfer it to the resource where you are using the grid tools if needed<br />
and [[Preparing_a_client_certificate|prepare it for use with grid tools]].<br />
If you have ARC 3.x or newer installed it can use the Firefox certificate store directly, as described in the next section.<br />
<br />
To create a short lived proxy that can be used for authentication with grid services, the '''arcproxy''' command can be used. A 12 hour (default) proxy is created in the following example:<br />
<br />
$ arcproxy<br />
Your identity: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=Kalle Kula<br />
Enter pass phrase for /home/kalle/.globus/userkey.pem:<br />
.++++++<br />
.....++++++<br />
Proxy generation succeeded<br />
Your proxy is valid until: 2011-03-11 03:00:14<br />
<br />
The proxy file itself will be created in the '''/tmp''' directory with the format '''x509up_uid''', where uid is the user id number for your account.<br />
<br />
In some cases a longer lived proxy will be needed. This is achieved using the '''--constraint''' switch. A 24-hour can be created by issuing the following command:<br />
<br />
$ arcproxy --constraint="validityPeriod=24H"<br />
Your identity: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=Kalle Kula<br />
Enter pass phrase for /home/kalle/.globus/userkey.pem:<br />
....++++++<br />
.....++++++<br />
Proxy generation succeeded<br />
Your proxy is valid until: 2011-03-11 15:03:19<br />
<br />
== Creating a proxy certificate using the Firefox/Thunderbird credential store ==<br />
<br />
Using the ARC 3.x client tools it is now possible to generate a proxy certificate directly from the Firefox or Thunderbird credential stores. To do this the '''-F''' flag is used as shown in the following example:<br />
<br />
$ arcproxy -F<br />
There are 2 NSS base directories where the certificate, key, and module datbases live<br />
Number 1 is: /Users/lindemann/Library/Application Support/Firefox/Profiles/t22f3aj2.default<br />
Number 2 is: /Users/lindemann/Library/Thunderbird/Profiles/7abb733v.default<br />
Please choose the NSS database you would use (1-2): 1<br />
<br />
Here ARC finds the available Firefox and Thunderbird profile in which the credential stores are stored. Next the passphrase for the credential store is used to unlock the stored credentials:<br />
<br />
NSS database to be accessed: /Users/lindemann/Library/Application Support/Firefox/Profiles/t22f3aj2.default<br />
Enter Password or Pin for "internal (software)":<br />
<br />
If the passphrase was correct, ARC will list the available certificates in the credential store and ask you for which you would like to use.<br />
<br />
There are 2 user certificates existing in the NSS database<br />
Number 1 is with nickname: Jonas Lindemann xxxxx@lu.se's TERENA ID (Jonas Lindemann xxxxx@lu.se)<br />
expiration time: 2013-06-04 01:59:59<br />
Number 2 is with nickname: Imported Certificate (Jonas Lindemann)<br />
expiration time: 2014-01-18 16:55:52<br />
Please choose the one you would use (1-2): 1<br />
Certificate to use is: Jonas Lindemann xxxxxx@lu.se's TERENA ID<br />
Proxy generation succeeded<br />
Your proxy is valid until: 2013-05-01 04:11:37<br />
<br />
== Checking proxy lifetime ==<br />
<br />
The remaining lifetime of a proxy certificate can be checked using the '''arcproxy''' command with the '''--info''' switch.<br />
<br />
$ arcproxy --info<br />
Subject: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=Kalle Kula/CN=1567862803<br />
Identity: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=Kalle Kula<br />
Time left for proxy: 11 hours 55 minutes<br />
Proxy path: /tmp/x509up_u500<br />
Proxy type: X.509 Proxy Certificate Profile RFC compliant restricted proxy<br />
<br />
In this example the proxy certificate is valid for 11 hours 55 minutes more.<br />
<br />
== Destroying a proxy certificate ==<br />
<br />
A proxy can be destroyed with the '''-r''' or '''--remove''' switch.<br />
<br />
$ arcproxy -r<br />
<br />
or<br />
<br />
$ arcproxy --remove<br />
<br />
= VOMS certificates =<br />
<br />
As long as you are a member of only one VO or VO group, you can<br />
authenticate to a grid service with the regular grid proxy certificate<br />
as defined in the previous section. If you are a member of more than<br />
one VO or VO group you may want to select which membership you want to<br />
be authenticated as. For example, if you are a member of<br />
''swegrid.se:/swegrid.se/ops'' (operations staff) and<br />
''swegrid.se:/swegrid.se/bils'' and want to write a file, who should<br />
be the owner? Ops or bils? You need to provide some additional<br />
information. In the grid world this is done with a voms proxy<br />
certificate which basically is a regular proxy certificate but with a<br />
so called voms extension that contains a list of your VO group<br />
memberships (and roles and attributes, which we don't use in<br />
Swegrid/Swestore at the moment).<br />
<br />
'''Please note, if you only have one membership you can skip this section!'''<br />
<br />
The voms extension of the certificate is signed by the virtual<br />
organization management server, or VOMS server. The same VOMS server<br />
you used when applying for the swegrid.se VO membership in the first<br />
place. To enable this signing process you need to add a few<br />
configuration files to your system. First add this to the file<br />
'''/etc/vomses''':<br />
<br />
"swegrid.se" "voms.ndgf.org" "15009" "/O=Grid/O=NorduGrid/CN=host/voms.ndgf.org" "swegrid.se"<br />
<br />
Next create the necessary directories and the file<br />
'''/etc/grid-security/vomsdir/swegrid.se/voms.ndgf.org.lsc''' with the<br />
following contents:<br />
<br />
/O=Grid/O=NorduGrid/CN=host/voms.ndgf.org<br />
/O=Grid/O=NorduGrid/CN=NorduGrid Certification Authority<br />
<br />
== Creating a VOMS proxy ==<br />
<br />
VOMS proxies in ARC1 can be created using the '''arcproxy''' command<br />
and the '''-S''' or '''--voms''' switches as shown in the following<br />
example (if you are a member of the /swegrid.se/ops group. Adjust as<br />
necessary):<br />
<br />
$ arcproxy -S swegrid.se:/swegrid.se/ops<br />
Your identity: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=Kalle Kula<br />
Enter pass phrase for /home/kalle/.globus/userkey.pem:<br />
.....++++++<br />
............++++++<br />
Contacting VOMS server (named swegrid.se): voms.ndgf.org on port: 15009<br />
Proxy generation succeeded<br />
Your proxy is valid until: 2011-03-10 23:33:06<br />
<br />
<br />
= Signing your e-mail with your certificate =<br />
<br />
First, you will need your grid certificate in PKCS12 format:<br />
== How to transform your certificate from PEM format into PKCS#12 format ==<br />
<br />
This is how you transform your cert into PKCS12 format that can be used within your webbrowser or email send program:<br />
You first will have to change directory into where you created and keep the certificate, historically this is often in ~/.globus<br />
<br />
openssl pkcs12 -export -in usercert.pem -inkey userkey.pem -out cert+key.p12 <br />
<br />
First you will have to enter the password you used for your private key, then you will be asked for a new password to protect the new file. '''cert+key.p12 contains your private key, and is therefore 'lika känslig' as userkey.pem'''. See also [[#Introduction to certificates]]. Security wise the safest way is to delete the PKCS12 file after having imported it into your mail client or browser. Don't forget this.<br />
<br />
Remarks: openssl will either need the variable RANDFILE to be set or that ~/.rnd is writable. So you have to make sure that the current $HOME is yours if you have pagshed away, otherwise the command will fail with ''unable to write 'random state''.<br />
<br />
<br />
=== Signing in mew ===<br />
<br />
Mew uses gpgsm. <br />
<br />
<pre><br />
1. Import the nordugrid root cert<br />
<br />
1.1. get 1f0e8352.0 from nordugrid web<br />
<br />
1.2. gpgsm --import 1f0e8352.0<br />
<br />
1.2. Make it trusted:<br />
gpgsm --list-keys 2>/dev/null | grep fingerprint | awk '{print $2 " S"}' | grep THE-FINGERPRIT-YOU-WANT >> .gnupg/trustlist.txt<br />
<br />
2. Add your own key from the cert+key.p12 file in this case<br />
<br />
2.1 openssl pkcs12 -in cert+key.p12 -out tmp.pem -nokeys<br />
<br />
2.2. gpgsm --import tmp.pem ; rm tmp.pem<br />
<br />
2.3. Tell gpgsm not to use revocation lists (bad bad security)<br />
echo disable-crl-checks >> .gnupg/gpgsm.conf<br />
<br />
3. Test<br />
gpgsm --detach-sign file > sign # should ask for passphrase and give some kind of sign file<br />
<br />
4. Use:<br />
C-uC-cC-s then enter your email address (must match email in cert) and passphrase<br />
<br />
</pre><br />
<br />
=== Signing in thunderbird ===<br />
In thunderbird: options/security/digitally sign this message.<br />
<br />
If you do this for the first time and haven't defined yet the certificate to sign with, thunderbird will pop up the according preferences [Account settings/Security], where you can choose between your imported certificates in PKCS12 format.<br />
<br />
In the beginning, of course, you haven't imported any: Click there on the same preferences tab that popped up on [View Certificates]. In the new window that opens you can import the certificate.<br />
<br />
Afterwards you can then choose this certificate to be used for signing and for encryption for this email account.<br />
<br />
Don't forget to actually check that you then really sign the corresponding mail.</div>Lars Viklund (HPC2N)https://snicdocs.nsc.liu.se/w/index.php?title=Apply_for_storage_on_Swestore&diff=5245Apply for storage on Swestore2013-06-14T09:45:04Z<p>Lars Viklund (HPC2N): </p>
<hr />
<div>[[Category:SweStore]]<br />
[[Category:SweStore user guide]]<br />
[[SweStore|< SweStore]]<br />
<br />
The SweStore nationally accessible storage is available for researchers financed by VR (which includes all researchers using SNIC compute resources) and FORMA.<br />
<br />
SweStore is also in collaboration with [http://www.ecds.se/ ECDS], [http://snd.gu.se/ SND], Bioimage, [http://www.bils.se/ BILS], [http://www.uppnex.uu.se/ UPPNEX],[http://wlcg.web.cern.ch/ WLCG], [http://www.nrm.se/ Naturhistoriska Riksmuseet]. If any of these cover your research area, first read their information on applying for SweStore storage.<br />
<br />
In the future, applications for storage will be handled by each research community, but for now an email to [mailto:support@swestore.se support@swestore.se] will suffice. <br />
<br />
Please include the following information in the application:<br />
* Name of the principal investigator (PI), including email address.<br />
* Purpose for the storage: A short description of the project and type of data.<br />
* Required storage capacity: Preferably a maximum size, but if this is not currently determinable, please calculate a starting size and expansion by time period. '''NOTE''' that applications larger than 10TB takes longer to process.<br />
* Suggested project name: This will be used as root directory name for your storage.<br />
# '''NOTE''' that this name is long-lived and will persist. It is not coupled to the lifetime of SNIC compute time allocations.<br />
# We recommend a project name not tied to a person.<br />
# Additionally, we recommend that the name is not a common word or term easily confusable with other current or future research efforts.<br />
# It is a good idea to select a name that's short and easy to type.<br />
# The name is limited to lower-case letters a-z, digits 0-9, hyphens - and underscores _.</div>Lars Viklund (HPC2N)https://snicdocs.nsc.liu.se/w/index.php?title=Preparing_a_client_certificate&diff=5164Preparing a client certificate2013-05-17T06:45:33Z<p>Lars Viklund (HPC2N): /* Uploading and conversion of the .p12 for your target machine */</p>
<hr />
<div>[[Category:Grid computing]]<br />
[[Category:SweGrid user guide]]<br />
[[Category:SweStore]]<br />
[[Category:SweStore user guide]]<br />
[[SweStore|< SweStore]]<br />
<br />
Most of the standalone third party tools installed on SNIC resources and your own machine will not be able to use a <tt>.p12</tt> certificate bundle (or <tt>.pfx</tt> if you exported from IE), as that format is intended primarily for secure transport and backup of certificates and their private keys.<br />
<br />
Instead of a single <tt>.p12</tt> file, they expect a pair of files in <tt>.pem</tt> format, one containing the certificate and the other containing the private key that matches the certificate.<br />
<br />
== Uploading and conversion of the .p12 for your target machine ==<br />
<br />
As the authentication methods for clusters differ, this section will defer to documentations for your particular site when it comes to transferring files to and from the cluster storage.<br />
<br />
The goal is to end up with a <tt>.globus</tt> directory in your home directory, containing two files named <tt>usercert.pem</tt> and <tt>userkey.pem</tt>.<br />
<br />
The instructions below assume that your exported certificate file is named <tt>export.p12</tt> directly in your home directory. If it's a <tt>.pfx</tt> or with a different name, change <tt>export.p12</tt> in the instructions to your actual filename or rename your file to <tt>export.p12</tt>.<br />
<br />
* Transfer the <tt>export.p12</tt> file to your home directory on the cluster.<br />
* Get an interactive shell on the login node, via ssh.<br />
* If an .globus directory already exists, rename it with something like<br />
<tt>mv ~/.globus ~/.globus-old</tt><br />
* Create the directory with<br />
<tt>mkdir ~/.globus</tt><br />
* Extract and protect the private key part of <tt>export.p12</tt>:<br />
openssl pkcs12 -nocerts -in ~/export.p12 -out ~/.globus/userkey.pem<br />
* When asked for import password, specify the password specified when exporting the certificate bundle from your browser. The PEM pass phrase should be a new password that you need to provide whenever using the certificate for tasks like generating a proxy certificate. The output from this command will be similar to the following:<br />
Enter Import Password: *******<br />
MAC verified OK<br />
Enter PEM pass phrase: *******<br />
Verifying - Enter PEM pass phrase: *******<br />
<br />
* Extract the public client certificate part of <tt>export.p12</tt>:<br />
openssl pkcs12 -clcerts -nokeys -in ~/export.p12 -out ~/globus/usercert.pem<br />
* The output will be similar to the following:<br />
Enter Import Password: *******<br />
MAC verified OK<br />
* Finally ensure that only your user is allowed to read the private key file. This is important, both for security and due to some tools refusing to use private keys with insufficient restrictions.<br />
chmod 0400 ~/.globus/userkey.pem</div>Lars Viklund (HPC2N)https://snicdocs.nsc.liu.se/w/index.php?title=Grid_certificates&diff=5162Grid certificates2013-05-16T13:14:34Z<p>Lars Viklund (HPC2N): /* Requesting membership in the SweGrid VO */</p>
<hr />
<div>[[Category:Grid computing]]<br />
[[Category:SweGrid user guide]]<br />
[[Category:SweStore]]<br />
[[Category:SweStore user guide]]<br />
[[Getting started with SweGrid|< Getting started with SweGrid]]<br><br />
[[SweStore|< SweStore]]<br />
<br />
=Introduction to certificates=<br />
<br />
In order to get access to computer and storage resources on the grid or [[SweStore]] you must have a valid (grid) certificate. This certificate is used instead of a username and password when accessing the resource. The resource have a certificate that tells you that you have contacted the right resource. This is exactly the same mechanism used when you use a web browser to contact your bank.<br />
<br />
A certificate is the similar to a passport in real-life. In the same way you have prove your credentials when you acquire a passport the same is true for a certificate. A third party, the Certificate Authority or CA, that both you and the resource trust has to vouch for your identity and sign your certificate.<br />
<br />
A certificate consist of a public key, some user information and a signature of the CA. In addition to the certificate you have a private key. The private key is secret and should be kept as secure as possible.<br />
<br />
For more information regarding certificates and public key cryptography:<br />
<br />
[http://en.wikipedia.org/wiki/Public-key_cryptography http://en.wikipedia.org/wiki/Public-key_cryptography]<br />
<br />
[http://en.wikipedia.org/wiki/Public_key_certificate http://en.wikipedia.org/wiki/Public_key_certificate]<br />
<br />
[http://www.nordugrid.org/documents/certificate_howto.html http://www.nordugrid.org/documents/certificate_howto.html]<br />
<br />
* The grid certificate and the private key are stored in your web browser and/or located in ~/.globus at the host(s) from where you will be accessing the resource:<br />
usercert.pem<br />
userkey.pem<br />
* The certificate contains your public key, your name and organization and a signature by the CA. It is does not contain any username.<br />
* The certificate is valid for 13 month and should be renewed yearly.<br />
* The private key should be handled with great care. It should only be readable by you and not by the group or others (i.e. ``chmod 400 userkey.pem''). Store the key on trusted computers and transfer the key between computers using encryption (using for example scp).<br />
* On shared file systems make sure that ~/.globus is not readible by everybody:<br />
chmod 700 ~/.globus<br />
and on AFS:<br />
fs sa ~/.globus system:anyuser none<br />
* The private key is encrypted using a passphrase. Anyone that can decrypt the private key will be able to authenticate as you to grid resources. This is similar to the private key in SSH. You must choose a strong passphrase for the private key. This passphrase must not be used anywhere else. You must never ever give away the passphrase to somebody else.<br />
* You should not share the certificate with someone. It's personal. <br />
<br />
For more information regarding certificates and public key cryptography:<br />
<br />
[http://en.wikipedia.org/wiki/Public-key_cryptography http://en.wikipedia.org/wiki/Public-key_cryptography]<br />
[http://en.wikipedia.org/wiki/Public_key_certificate http://en.wikipedia.org/wiki/Public_key_certificate]<br />
<br />
= Requesting a certificate =<br />
<br />
Certificates are issued by a Certificate Authority or CA. For Swedish users there are two relevant CA:s that can issue grid/eScience certificates, Terena and Nordugrid. The Terena CA is preferred if it is available for your university or research group, but many sites has not enabled this service yet. The Nordugrid CA can also be used but requires more manual work by all parties.<br />
<br />
Recommended procedure for each university:<br />
<br />
{| class="wikitable"<br />
| University<br />
| CA<br />
| Specific instructions<br />
|-<br />
| LU<br />
| Terena CA<br />
| [[LU_Certificate_Information|more...]]<br />
|-<br />
| LiU<br />
| Terena CA<br />
| [[LiU_Certificate_Instructions|more...]]<br />
|-<br />
| CTH<br />
| NorduGrid CA<br />
| [[Chalmers_Certificate_Instructions|more...]]<br />
|-<br />
| GU<br />
| NorduGrid CA<br />
| [[GU_Certificate_Instructions|more...]]<br />
|-<br />
| UU<br />
| Terena CA<br />
| [[UU_Certificate_Instructions|more...]]<br />
|-<br />
| KTH<br />
| Terena CA<br />
| [[KTH_Certificate_Information|more...]]<br />
|-<br />
| SU<br />
| NorduGrid CA<br />
| [[SU_Certificate_Information|more...]]<br />
|-<br />
| KI<br />
| NorduGrid CA<br />
| [[KI_Certificate_Information|more...]]<br />
|-<br />
| UmU<br />
| Terena CA<br />
| [[UmU_Certificate_Information|more...]]<br />
|-<br />
|}<br />
<br />
[[Requesting a grid certificate using the Terena eScience Portal|Instructions for the Terena CA]]<br />
<br />
[[Requesting a grid certificate from the Nordugrid CA|Instructions for the NorduGrid CA (use only if Terena eScience isn't available at your site)]]<br />
<br />
= Requesting membership in the SweGrid VO =<br />
<br />
SweGrid and SweStore resources are currently being allocated for VO:s, virtual organizations, rather than individual users. A VO is basically just a list of users. To be able to use a SweGrid or SweStore resource a membership in the SweGrid VO (virtual organization) and a corresponding subgroup is required. To apply for membership, make sure that the NorduGrid root CA certificate and your personal certificate is installed in the browser. <br />
<br />
The NorduGrid CA cert can be installed by clicking on the following link:<br />
<br />
[http://ca.nordugrid.org/cacrt.crt http://ca.nordugrid.org/cacrt.crt]<br />
<br />
Make sure you check the "Trust this CA to identify web sites." boxes in the dialog shown.<br />
<br />
<br />
[[File:certinstall.png]]<br />
<br />
<br />
When certificates have been installed in the browser go to the following URL:<br />
<br />
[https://voms.ndgf.org:8443/voms/swegrid.se https://voms.ndgf.org:8443/voms/swegrid.se]<br />
<br />
and follow the instructions. In a couple of hours you will be added to the SweGrid VO. <br />
<br />
In order to be added to the correct project/allocation groups use the '''Request membership''' function in the '''Your groups and roles section''' of your VOMS homepage at https://voms.ndgf.org:8443/voms/swegrid.se/user/home.action as shown in the following screenshot, selecting the project in the dropdown box and clicking the '''Request membership''' button. No further actions need to be taken on that page after requesting the membership.<br />
<br />
[[File:request-vo-membership.png]]<br />
<br />
If that doesn't work for some reason, contact SweGrid support at [mailto:support@swegrid.se support@swegrid.se] or SweStore support at [mailto:swestore-support@snic.vr.se swestore-support@snic.vr.se] as appropritate.<br />
<br />
= Proxy certificates =<br />
<br />
Authentication on the grid is done using special short lived ''proxy'' certificates. There are several tools available for creating, checking and destroying these proxy certificates.<br />
<br />
== Creating a proxy certificate ==<br />
<br />
To create a short lived proxy that can be used for authentication with grid services, the '''arcproxy''' command can be used. A 12 hour (default) proxy is created in the following example:<br />
<br />
$ arcproxy<br />
Your identity: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=Kalle Kula<br />
Enter pass phrase for /home/kalle/.globus/userkey.pem:<br />
.++++++<br />
.....++++++<br />
Proxy generation succeeded<br />
Your proxy is valid until: 2011-03-11 03:00:14<br />
<br />
The proxy file itself will be created in the '''/tmp''' directory with the format '''x509up_uid''', where uid is the user id number for your account.<br />
<br />
In some cases a longer lived proxy will be needed. This is achieved using the '''--constraint''' switch. A 24-hour can be created by issuing the following command:<br />
<br />
$ arcproxy --constraint="validityPeriod=24H"<br />
Your identity: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=Kalle Kula<br />
Enter pass phrase for /home/kalle/.globus/userkey.pem:<br />
....++++++<br />
.....++++++<br />
Proxy generation succeeded<br />
Your proxy is valid until: 2011-03-11 15:03:19<br />
<br />
== Creating a proxy certificate using the Firefox/Thunderbird credential store ==<br />
<br />
Using the ARC 3.x client tools it is now possible to generate a proxy certificate directly from the Firefox or Thunderbird credential stores. To do this the '''-F''' flag is used as shown in the following example:<br />
<br />
$ arcproxy -F<br />
There are 2 NSS base directories where the certificate, key, and module datbases live<br />
Number 1 is: /Users/lindemann/Library/Application Support/Firefox/Profiles/t22f3aj2.default<br />
Number 2 is: /Users/lindemann/Library/Thunderbird/Profiles/7abb733v.default<br />
Please choose the NSS database you would use (1-2): 1<br />
<br />
Here ARC finds the available Firefox and Thunderbird profile in which the credential stores are stored. Next the passphrase for the credential store is used to unlock the stored credentials:<br />
<br />
NSS database to be accessed: /Users/lindemann/Library/Application Support/Firefox/Profiles/t22f3aj2.default<br />
Enter Password or Pin for "internal (software)":<br />
<br />
If the passphrase was correct, ARC will list the available certificates in the credential store and ask you for which you would like to use.<br />
<br />
There are 2 user certificates existing in the NSS database<br />
Number 1 is with nickname: Jonas Lindemann xxxxx@lu.se's TERENA ID (Jonas Lindemann xxxxx@lu.se)<br />
expiration time: 2013-06-04 01:59:59<br />
Number 2 is with nickname: Imported Certificate (Jonas Lindemann)<br />
expiration time: 2014-01-18 16:55:52<br />
Please choose the one you would use (1-2): 1<br />
Certificate to use is: Jonas Lindemann xxxxxx@lu.se's TERENA ID<br />
Proxy generation succeeded<br />
Your proxy is valid until: 2013-05-01 04:11:37<br />
<br />
== Checking proxy lifetime ==<br />
<br />
The remaining lifetime of a proxy certificate can be checked using the '''arcproxy''' command with the '''--info''' switch.<br />
<br />
$ arcproxy --info<br />
Subject: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=Kalle Kula/CN=1567862803<br />
Identity: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=Kalle Kula<br />
Time left for proxy: 11 hours 55 minutes<br />
Proxy path: /tmp/x509up_u500<br />
Proxy type: X.509 Proxy Certificate Profile RFC compliant restricted proxy<br />
<br />
In this example the proxy certificate is valid for 11 hours 55 minutes more.<br />
<br />
== Destroying a proxy certificate ==<br />
<br />
A proxy can be destroyed with the '''-r''' or '''--remove''' switch.<br />
<br />
$ arcproxy -r<br />
<br />
or<br />
<br />
$ arcproxy --remove<br />
<br />
= VOMS certificates =<br />
<br />
As long as you are a member of only one VO or VO group, you can<br />
authenticate to a grid service with the regular grid proxy certificate<br />
as defined in the previous section. If you are a member of more than<br />
one VO or VO group you may want to select which membership you want to<br />
be authenticated as. For example, if you are a member of<br />
''swegrid.se:/swegrid.se/ops'' (operations staff) and<br />
''swegrid.se:/swegrid.se/bils'' and want to write a file, who should<br />
be the owner? Ops or bils? You need to provide some additional<br />
information. In the grid world this is done with a voms proxy<br />
certificate which basically is a regular proxy certificate but with a<br />
so called voms extension that contains a list of your VO group<br />
memberships (and roles and attributes, which we don't use in<br />
Swegrid/Swestore at the moment).<br />
<br />
'''Please note, if you only have one membership you can skip this section!'''<br />
<br />
The voms extension of the certificate is signed by the virtual<br />
organization management server, or VOMS server. The same VOMS server<br />
you used when applying for the swegrid.se VO membership in the first<br />
place. To enable this signing process you need to add a few<br />
configuration files to your system. First add this to the file<br />
'''/etc/vomses''':<br />
<br />
"swegrid.se" "voms.ndgf.org" "15009" "/O=Grid/O=NorduGrid/CN=host/voms.ndgf.org" "swegrid.se"<br />
<br />
Next create the necessary directories and the file<br />
'''/etc/grid-security/vomsdir/swegrid.se/voms.ndgf.org.lsc''' with the<br />
following contents:<br />
<br />
/O=Grid/O=NorduGrid/CN=host/voms.ndgf.org<br />
/O=Grid/O=NorduGrid/CN=NorduGrid Certification Authority<br />
<br />
== Creating a VOMS proxy ==<br />
<br />
VOMS proxies in ARC1 can be created using the '''arcproxy''' command<br />
and the '''-S''' or '''--voms''' switches as shown in the following<br />
example (if you are a member of the /swegrid.se/ops group. Adjust as<br />
necessary):<br />
<br />
$ arcproxy -S swegrid.se:/swegrid.se/ops<br />
Your identity: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=Kalle Kula<br />
Enter pass phrase for /home/kalle/.globus/userkey.pem:<br />
.....++++++<br />
............++++++<br />
Contacting VOMS server (named swegrid.se): voms.ndgf.org on port: 15009<br />
Proxy generation succeeded<br />
Your proxy is valid until: 2011-03-10 23:33:06<br />
<br />
<br />
= Signing your e-mail with your certificate =<br />
<br />
First, you will need your grid certificate in PKCS12 format:<br />
== How to transform your certificate from PEM format into PKCS#12 format ==<br />
<br />
This is how you transform your cert into PKCS12 format that can be used within your webbrowser or email send program:<br />
You first will have to change directory into where you created and keep the certificate, historically this is often in ~/.globus<br />
<br />
openssl pkcs12 -export -in usercert.pem -inkey userkey.pem -out cert+key.p12 <br />
<br />
First you will have to enter the password you used for your private key, then you will be asked for a new password to protect the new file. '''cert+key.p12 contains your private key, and is therefore 'lika känslig' as userkey.pem'''. See also [[#Introduction to certificates]]. Security wise the safest way is to delete the PKCS12 file after having imported it into your mail client or browser. Don't forget this.<br />
<br />
Remarks: openssl will either need the variable RANDFILE to be set or that ~/.rnd is writable. So you have to make sure that the current $HOME is yours if you have pagshed away, otherwise the command will fail with ''unable to write 'random state''.<br />
<br />
<br />
=== Signing in mew ===<br />
<br />
Mew uses gpgsm. <br />
<br />
<pre><br />
1. Import the nordugrid root cert<br />
<br />
1.1. get 1f0e8352.0 from nordugrid web<br />
<br />
1.2. gpgsm --import 1f0e8352.0<br />
<br />
1.2. Make it trusted:<br />
gpgsm --list-keys 2>/dev/null | grep fingerprint | awk '{print $2 " S"}' | grep THE-FINGERPRIT-YOU-WANT >> .gnupg/trustlist.txt<br />
<br />
2. Add your own key from the cert+key.p12 file in this case<br />
<br />
2.1 openssl pkcs12 -in cert+key.p12 -out tmp.pem -nokeys<br />
<br />
2.2. gpgsm --import tmp.pem ; rm tmp.pem<br />
<br />
2.3. Tell gpgsm not to use revocation lists (bad bad security)<br />
echo disable-crl-checks >> .gnupg/gpgsm.conf<br />
<br />
3. Test<br />
gpgsm --detach-sign file > sign # should ask for passphrase and give some kind of sign file<br />
<br />
4. Use:<br />
C-uC-cC-s then enter your email address (must match email in cert) and passphrase<br />
<br />
</pre><br />
<br />
=== Signing in thunderbird ===<br />
In thunderbird: options/security/digitally sign this message.<br />
<br />
If you do this for the first time and haven't defined yet the certificate to sign with, thunderbird will pop up the according preferences [Account settings/Security], where you can choose between your imported certificates in PKCS12 format.<br />
<br />
In the beginning, of course, you haven't imported any: Click there on the same preferences tab that popped up on [View Certificates]. In the new window that opens you can import the certificate.<br />
<br />
Afterwards you can then choose this certificate to be used for signing and for encryption for this email account.<br />
<br />
Don't forget to actually check that you then really sign the corresponding mail.</div>Lars Viklund (HPC2N)https://snicdocs.nsc.liu.se/w/index.php?title=Grid_certificates&diff=5161Grid certificates2013-05-16T13:02:55Z<p>Lars Viklund (HPC2N): /* Requesting membership in the SweGrid VO */</p>
<hr />
<div>[[Category:Grid computing]]<br />
[[Category:SweGrid user guide]]<br />
[[Category:SweStore]]<br />
[[Category:SweStore user guide]]<br />
[[Getting started with SweGrid|< Getting started with SweGrid]]<br><br />
[[SweStore|< SweStore]]<br />
<br />
=Introduction to certificates=<br />
<br />
In order to get access to computer and storage resources on the grid or [[SweStore]] you must have a valid (grid) certificate. This certificate is used instead of a username and password when accessing the resource. The resource have a certificate that tells you that you have contacted the right resource. This is exactly the same mechanism used when you use a web browser to contact your bank.<br />
<br />
A certificate is the similar to a passport in real-life. In the same way you have prove your credentials when you acquire a passport the same is true for a certificate. A third party, the Certificate Authority or CA, that both you and the resource trust has to vouch for your identity and sign your certificate.<br />
<br />
A certificate consist of a public key, some user information and a signature of the CA. In addition to the certificate you have a private key. The private key is secret and should be kept as secure as possible.<br />
<br />
For more information regarding certificates and public key cryptography:<br />
<br />
[http://en.wikipedia.org/wiki/Public-key_cryptography http://en.wikipedia.org/wiki/Public-key_cryptography]<br />
<br />
[http://en.wikipedia.org/wiki/Public_key_certificate http://en.wikipedia.org/wiki/Public_key_certificate]<br />
<br />
[http://www.nordugrid.org/documents/certificate_howto.html http://www.nordugrid.org/documents/certificate_howto.html]<br />
<br />
* The grid certificate and the private key are stored in your web browser and/or located in ~/.globus at the host(s) from where you will be accessing the resource:<br />
usercert.pem<br />
userkey.pem<br />
* The certificate contains your public key, your name and organization and a signature by the CA. It is does not contain any username.<br />
* The certificate is valid for 13 month and should be renewed yearly.<br />
* The private key should be handled with great care. It should only be readable by you and not by the group or others (i.e. ``chmod 400 userkey.pem''). Store the key on trusted computers and transfer the key between computers using encryption (using for example scp).<br />
* On shared file systems make sure that ~/.globus is not readible by everybody:<br />
chmod 700 ~/.globus<br />
and on AFS:<br />
fs sa ~/.globus system:anyuser none<br />
* The private key is encrypted using a passphrase. Anyone that can decrypt the private key will be able to authenticate as you to grid resources. This is similar to the private key in SSH. You must choose a strong passphrase for the private key. This passphrase must not be used anywhere else. You must never ever give away the passphrase to somebody else.<br />
* You should not share the certificate with someone. It's personal. <br />
<br />
For more information regarding certificates and public key cryptography:<br />
<br />
[http://en.wikipedia.org/wiki/Public-key_cryptography http://en.wikipedia.org/wiki/Public-key_cryptography]<br />
[http://en.wikipedia.org/wiki/Public_key_certificate http://en.wikipedia.org/wiki/Public_key_certificate]<br />
<br />
= Requesting a certificate =<br />
<br />
Certificates are issued by a Certificate Authority or CA. For Swedish users there are two relevant CA:s that can issue grid/eScience certificates, Terena and Nordugrid. The Terena CA is preferred if it is available for your university or research group, but many sites has not enabled this service yet. The Nordugrid CA can also be used but requires more manual work by all parties.<br />
<br />
Recommended procedure for each university:<br />
<br />
{| class="wikitable"<br />
| University<br />
| CA<br />
| Specific instructions<br />
|-<br />
| LU<br />
| Terena CA<br />
| [[LU_Certificate_Information|more...]]<br />
|-<br />
| LiU<br />
| Terena CA<br />
| [[LiU_Certificate_Instructions|more...]]<br />
|-<br />
| CTH<br />
| NorduGrid CA<br />
| [[Chalmers_Certificate_Instructions|more...]]<br />
|-<br />
| GU<br />
| NorduGrid CA<br />
| [[GU_Certificate_Instructions|more...]]<br />
|-<br />
| UU<br />
| Terena CA<br />
| [[UU_Certificate_Instructions|more...]]<br />
|-<br />
| KTH<br />
| Terena CA<br />
| [[KTH_Certificate_Information|more...]]<br />
|-<br />
| SU<br />
| NorduGrid CA<br />
| [[SU_Certificate_Information|more...]]<br />
|-<br />
| KI<br />
| NorduGrid CA<br />
| [[KI_Certificate_Information|more...]]<br />
|-<br />
| UmU<br />
| Terena CA<br />
| [[UmU_Certificate_Information|more...]]<br />
|-<br />
|}<br />
<br />
[[Requesting a grid certificate using the Terena eScience Portal|Instructions for the Terena CA]]<br />
<br />
[[Requesting a grid certificate from the Nordugrid CA|Instructions for the NorduGrid CA (use only if Terena eScience isn't available at your site)]]<br />
<br />
= Requesting membership in the SweGrid VO =<br />
<br />
SweGrid and SweStore resources are currently being allocated for VO:s, virtual organizations, rather than individual users. A VO is basically just a list of users. To be able to use a SweGrid or SweStore resource a membership in the SweGrid VO (virtual organization) and a corresponding subgroup is required. To apply for membership, make sure that the NorduGrid root CA certificate and your personal certificate is installed in the browser. <br />
<br />
The NorduGrid CA cert can be installed by clicking on the following link:<br />
<br />
[http://ca.nordugrid.org/cacrt.crt http://ca.nordugrid.org/cacrt.crt]<br />
<br />
Make sure you check the "Trust this CA to identify web sites." boxes in the dialog shown.<br />
<br />
<br />
[[File:certinstall.png]]<br />
<br />
<br />
When certificates have been installed in the browser go to the following URL:<br />
<br />
[https://voms.ndgf.org:8443/voms/swegrid.se https://voms.ndgf.org:8443/voms/swegrid.se]<br />
<br />
and follow the instructions. In a couple of hours you will be added to the SweGrid VO. <br />
<br />
In order to be added to the correct project/allocation groups use the '''Request membership''' function in the '''Your groups and roles section''' of your VOMS homepage at https://voms.ndgf.org:8443/voms/swegrid.se/user/home.action as shown in the following screenshot, selecting the project in the dropdown box and clicking the '''Request membership''' button.<br />
<br />
[[File:request-vo-membership.png]]<br />
<br />
If that doesn't work for some reason, contact SweGrid support at [mailto:support@swegrid.se support@swegrid.se] or SweStore support at [mailto:swestore-support@snic.vr.se swestore-support@snic.vr.se] as appropritate.<br />
<br />
= Proxy certificates =<br />
<br />
Authentication on the grid is done using special short lived ''proxy'' certificates. There are several tools available for creating, checking and destroying these proxy certificates.<br />
<br />
== Creating a proxy certificate ==<br />
<br />
To create a short lived proxy that can be used for authentication with grid services, the '''arcproxy''' command can be used. A 12 hour (default) proxy is created in the following example:<br />
<br />
$ arcproxy<br />
Your identity: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=Kalle Kula<br />
Enter pass phrase for /home/kalle/.globus/userkey.pem:<br />
.++++++<br />
.....++++++<br />
Proxy generation succeeded<br />
Your proxy is valid until: 2011-03-11 03:00:14<br />
<br />
The proxy file itself will be created in the '''/tmp''' directory with the format '''x509up_uid''', where uid is the user id number for your account.<br />
<br />
In some cases a longer lived proxy will be needed. This is achieved using the '''--constraint''' switch. A 24-hour can be created by issuing the following command:<br />
<br />
$ arcproxy --constraint="validityPeriod=24H"<br />
Your identity: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=Kalle Kula<br />
Enter pass phrase for /home/kalle/.globus/userkey.pem:<br />
....++++++<br />
.....++++++<br />
Proxy generation succeeded<br />
Your proxy is valid until: 2011-03-11 15:03:19<br />
<br />
== Creating a proxy certificate using the Firefox/Thunderbird credential store ==<br />
<br />
Using the ARC 3.x client tools it is now possible to generate a proxy certificate directly from the Firefox or Thunderbird credential stores. To do this the '''-F''' flag is used as shown in the following example:<br />
<br />
$ arcproxy -F<br />
There are 2 NSS base directories where the certificate, key, and module datbases live<br />
Number 1 is: /Users/lindemann/Library/Application Support/Firefox/Profiles/t22f3aj2.default<br />
Number 2 is: /Users/lindemann/Library/Thunderbird/Profiles/7abb733v.default<br />
Please choose the NSS database you would use (1-2): 1<br />
<br />
Here ARC finds the available Firefox and Thunderbird profile in which the credential stores are stored. Next the passphrase for the credential store is used to unlock the stored credentials:<br />
<br />
NSS database to be accessed: /Users/lindemann/Library/Application Support/Firefox/Profiles/t22f3aj2.default<br />
Enter Password or Pin for "internal (software)":<br />
<br />
If the passphrase was correct, ARC will list the available certificates in the credential store and ask you for which you would like to use.<br />
<br />
There are 2 user certificates existing in the NSS database<br />
Number 1 is with nickname: Jonas Lindemann xxxxx@lu.se's TERENA ID (Jonas Lindemann xxxxx@lu.se)<br />
expiration time: 2013-06-04 01:59:59<br />
Number 2 is with nickname: Imported Certificate (Jonas Lindemann)<br />
expiration time: 2014-01-18 16:55:52<br />
Please choose the one you would use (1-2): 1<br />
Certificate to use is: Jonas Lindemann xxxxxx@lu.se's TERENA ID<br />
Proxy generation succeeded<br />
Your proxy is valid until: 2013-05-01 04:11:37<br />
<br />
== Checking proxy lifetime ==<br />
<br />
The remaining lifetime of a proxy certificate can be checked using the '''arcproxy''' command with the '''--info''' switch.<br />
<br />
$ arcproxy --info<br />
Subject: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=Kalle Kula/CN=1567862803<br />
Identity: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=Kalle Kula<br />
Time left for proxy: 11 hours 55 minutes<br />
Proxy path: /tmp/x509up_u500<br />
Proxy type: X.509 Proxy Certificate Profile RFC compliant restricted proxy<br />
<br />
In this example the proxy certificate is valid for 11 hours 55 minutes more.<br />
<br />
== Destroying a proxy certificate ==<br />
<br />
A proxy can be destroyed with the '''-r''' or '''--remove''' switch.<br />
<br />
$ arcproxy -r<br />
<br />
or<br />
<br />
$ arcproxy --remove<br />
<br />
= VOMS certificates =<br />
<br />
As long as you are a member of only one VO or VO group, you can<br />
authenticate to a grid service with the regular grid proxy certificate<br />
as defined in the previous section. If you are a member of more than<br />
one VO or VO group you may want to select which membership you want to<br />
be authenticated as. For example, if you are a member of<br />
''swegrid.se:/swegrid.se/ops'' (operations staff) and<br />
''swegrid.se:/swegrid.se/bils'' and want to write a file, who should<br />
be the owner? Ops or bils? You need to provide some additional<br />
information. In the grid world this is done with a voms proxy<br />
certificate which basically is a regular proxy certificate but with a<br />
so called voms extension that contains a list of your VO group<br />
memberships (and roles and attributes, which we don't use in<br />
Swegrid/Swestore at the moment).<br />
<br />
'''Please note, if you only have one membership you can skip this section!'''<br />
<br />
The voms extension of the certificate is signed by the virtual<br />
organization management server, or VOMS server. The same VOMS server<br />
you used when applying for the swegrid.se VO membership in the first<br />
place. To enable this signing process you need to add a few<br />
configuration files to your system. First add this to the file<br />
'''/etc/vomses''':<br />
<br />
"swegrid.se" "voms.ndgf.org" "15009" "/O=Grid/O=NorduGrid/CN=host/voms.ndgf.org" "swegrid.se"<br />
<br />
Next create the necessary directories and the file<br />
'''/etc/grid-security/vomsdir/swegrid.se/voms.ndgf.org.lsc''' with the<br />
following contents:<br />
<br />
/O=Grid/O=NorduGrid/CN=host/voms.ndgf.org<br />
/O=Grid/O=NorduGrid/CN=NorduGrid Certification Authority<br />
<br />
== Creating a VOMS proxy ==<br />
<br />
VOMS proxies in ARC1 can be created using the '''arcproxy''' command<br />
and the '''-S''' or '''--voms''' switches as shown in the following<br />
example (if you are a member of the /swegrid.se/ops group. Adjust as<br />
necessary):<br />
<br />
$ arcproxy -S swegrid.se:/swegrid.se/ops<br />
Your identity: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=Kalle Kula<br />
Enter pass phrase for /home/kalle/.globus/userkey.pem:<br />
.....++++++<br />
............++++++<br />
Contacting VOMS server (named swegrid.se): voms.ndgf.org on port: 15009<br />
Proxy generation succeeded<br />
Your proxy is valid until: 2011-03-10 23:33:06<br />
<br />
<br />
= Signing your e-mail with your certificate =<br />
<br />
First, you will need your grid certificate in PKCS12 format:<br />
== How to transform your certificate from PEM format into PKCS#12 format ==<br />
<br />
This is how you transform your cert into PKCS12 format that can be used within your webbrowser or email send program:<br />
You first will have to change directory into where you created and keep the certificate, historically this is often in ~/.globus<br />
<br />
openssl pkcs12 -export -in usercert.pem -inkey userkey.pem -out cert+key.p12 <br />
<br />
First you will have to enter the password you used for your private key, then you will be asked for a new password to protect the new file. '''cert+key.p12 contains your private key, and is therefore 'lika känslig' as userkey.pem'''. See also [[#Introduction to certificates]]. Security wise the safest way is to delete the PKCS12 file after having imported it into your mail client or browser. Don't forget this.<br />
<br />
Remarks: openssl will either need the variable RANDFILE to be set or that ~/.rnd is writable. So you have to make sure that the current $HOME is yours if you have pagshed away, otherwise the command will fail with ''unable to write 'random state''.<br />
<br />
<br />
=== Signing in mew ===<br />
<br />
Mew uses gpgsm. <br />
<br />
<pre><br />
1. Import the nordugrid root cert<br />
<br />
1.1. get 1f0e8352.0 from nordugrid web<br />
<br />
1.2. gpgsm --import 1f0e8352.0<br />
<br />
1.2. Make it trusted:<br />
gpgsm --list-keys 2>/dev/null | grep fingerprint | awk '{print $2 " S"}' | grep THE-FINGERPRIT-YOU-WANT >> .gnupg/trustlist.txt<br />
<br />
2. Add your own key from the cert+key.p12 file in this case<br />
<br />
2.1 openssl pkcs12 -in cert+key.p12 -out tmp.pem -nokeys<br />
<br />
2.2. gpgsm --import tmp.pem ; rm tmp.pem<br />
<br />
2.3. Tell gpgsm not to use revocation lists (bad bad security)<br />
echo disable-crl-checks >> .gnupg/gpgsm.conf<br />
<br />
3. Test<br />
gpgsm --detach-sign file > sign # should ask for passphrase and give some kind of sign file<br />
<br />
4. Use:<br />
C-uC-cC-s then enter your email address (must match email in cert) and passphrase<br />
<br />
</pre><br />
<br />
=== Signing in thunderbird ===<br />
In thunderbird: options/security/digitally sign this message.<br />
<br />
If you do this for the first time and haven't defined yet the certificate to sign with, thunderbird will pop up the according preferences [Account settings/Security], where you can choose between your imported certificates in PKCS12 format.<br />
<br />
In the beginning, of course, you haven't imported any: Click there on the same preferences tab that popped up on [View Certificates]. In the new window that opens you can import the certificate.<br />
<br />
Afterwards you can then choose this certificate to be used for signing and for encryption for this email account.<br />
<br />
Don't forget to actually check that you then really sign the corresponding mail.</div>Lars Viklund (HPC2N)https://snicdocs.nsc.liu.se/w/index.php?title=Grid_certificates&diff=5158Grid certificates2013-05-16T11:36:16Z<p>Lars Viklund (HPC2N): /* Requesting membership in the SweGrid VO */</p>
<hr />
<div>[[Category:Grid computing]]<br />
[[Category:SweGrid user guide]]<br />
[[Category:SweStore]]<br />
[[Category:SweStore user guide]]<br />
[[Getting started with SweGrid|< Getting started with SweGrid]]<br><br />
[[SweStore|< SweStore]]<br />
<br />
=Introduction to certificates=<br />
<br />
In order to get access to computer and storage resources on the grid or [[SweStore]] you must have a valid (grid) certificate. This certificate is used instead of a username and password when accessing the resource. The resource have a certificate that tells you that you have contacted the right resource. This is exactly the same mechanism used when you use a web browser to contact your bank.<br />
<br />
A certificate is the similar to a passport in real-life. In the same way you have prove your credentials when you acquire a passport the same is true for a certificate. A third party, the Certificate Authority or CA, that both you and the resource trust has to vouch for your identity and sign your certificate.<br />
<br />
A certificate consist of a public key, some user information and a signature of the CA. In addition to the certificate you have a private key. The private key is secret and should be kept as secure as possible.<br />
<br />
For more information regarding certificates and public key cryptography:<br />
<br />
[http://en.wikipedia.org/wiki/Public-key_cryptography http://en.wikipedia.org/wiki/Public-key_cryptography]<br />
<br />
[http://en.wikipedia.org/wiki/Public_key_certificate http://en.wikipedia.org/wiki/Public_key_certificate]<br />
<br />
[http://www.nordugrid.org/documents/certificate_howto.html http://www.nordugrid.org/documents/certificate_howto.html]<br />
<br />
* The grid certificate and the private key are stored in your web browser and/or located in ~/.globus at the host(s) from where you will be accessing the resource:<br />
usercert.pem<br />
userkey.pem<br />
* The certificate contains your public key, your name and organization and a signature by the CA. It is does not contain any username.<br />
* The certificate is valid for 13 month and should be renewed yearly.<br />
* The private key should be handled with great care. It should only be readable by you and not by the group or others (i.e. ``chmod 400 userkey.pem''). Store the key on trusted computers and transfer the key between computers using encryption (using for example scp).<br />
* On shared file systems make sure that ~/.globus is not readible by everybody:<br />
chmod 700 ~/.globus<br />
and on AFS:<br />
fs sa ~/.globus system:anyuser none<br />
* The private key is encrypted using a passphrase. Anyone that can decrypt the private key will be able to authenticate as you to grid resources. This is similar to the private key in SSH. You must choose a strong passphrase for the private key. This passphrase must not be used anywhere else. You must never ever give away the passphrase to somebody else.<br />
* You should not share the certificate with someone. It's personal. <br />
<br />
For more information regarding certificates and public key cryptography:<br />
<br />
[http://en.wikipedia.org/wiki/Public-key_cryptography http://en.wikipedia.org/wiki/Public-key_cryptography]<br />
[http://en.wikipedia.org/wiki/Public_key_certificate http://en.wikipedia.org/wiki/Public_key_certificate]<br />
<br />
= Requesting a certificate =<br />
<br />
Certificates are issued by a Certificate Authority or CA. For Swedish users there are two relevant CA:s that can issue grid/eScience certificates, Terena and Nordugrid. The Terena CA is preferred if it is available for your university or research group, but many sites has not enabled this service yet. The Nordugrid CA can also be used but requires more manual work by all parties.<br />
<br />
Recommended procedure for each university:<br />
<br />
{| class="wikitable"<br />
| University<br />
| CA<br />
| Specific instructions<br />
|-<br />
| LU<br />
| Terena CA<br />
| [[LU_Certificate_Information|more...]]<br />
|-<br />
| LiU<br />
| Terena CA<br />
| [[LiU_Certificate_Instructions|more...]]<br />
|-<br />
| CTH<br />
| NorduGrid CA<br />
| [[Chalmers_Certificate_Instructions|more...]]<br />
|-<br />
| GU<br />
| NorduGrid CA<br />
| [[GU_Certificate_Instructions|more...]]<br />
|-<br />
| UU<br />
| Terena CA<br />
| [[UU_Certificate_Instructions|more...]]<br />
|-<br />
| KTH<br />
| Terena CA<br />
| [[KTH_Certificate_Information|more...]]<br />
|-<br />
| SU<br />
| NorduGrid CA<br />
| [[SU_Certificate_Information|more...]]<br />
|-<br />
| KI<br />
| NorduGrid CA<br />
| [[KI_Certificate_Information|more...]]<br />
|-<br />
| UmU<br />
| Terena CA<br />
| [[UmU_Certificate_Information|more...]]<br />
|-<br />
|}<br />
<br />
[[Requesting a grid certificate using the Terena eScience Portal|Instructions for the Terena CA]]<br />
<br />
[[Requesting a grid certificate from the Nordugrid CA|Instructions for the NorduGrid CA (use only if Terena eScience isn't available at your site)]]<br />
<br />
= Requesting membership in the SweGrid VO =<br />
<br />
SweGrid and SweStore resources are currently being allocated for VO:s, virtual organizations, rather than individual users. A VO is basically just a list of users. To be able to use a SweGrid or SweStore resource a membership in the SweGrid VO (virtual organization) and a corresponding subgroup is required. To apply for membership, make sure that the NorduGrid root CA certificate and your personal certificate is installed in the browser. <br />
<br />
The NorduGrid CA cert can be installed by clicking on the following link:<br />
<br />
[http://ca.nordugrid.org/cacrt.crt http://ca.nordugrid.org/cacrt.crt]<br />
<br />
Make sure you check the "Trust this CA to identify web sites." boxes in the dialog shown.<br />
<br />
<br />
[[File:certinstall.png]]<br />
<br />
<br />
When certificates have been installed in the browser go to the following URL:<br />
<br />
[https://voms.ndgf.org:8443/voms/swegrid.se https://voms.ndgf.org:8443/voms/swegrid.se]<br />
<br />
and follow the instructions. In a couple of hours you will be added to the SweGrid VO. <br />
<br />
In order to be added to the correct project/allocation groups use the '''Request membership''' function in the '''Your groups and roles section''' of your VOMS homepage at https://voms.ndgf.org:8443/voms/swegrid.se/user/home.action as shown in the following screenshot, selecting the project in the dropdown box.<br />
<br />
[[File:request-vo-membership.png]]<br />
<br />
If that doesn't work for some reason, contact SweGrid support at [mailto:support@swegrid.se support@swegrid.se] or SweStore support at [mailto:swestore-support@snic.vr.se swestore-support@snic.vr.se] as appropritate.<br />
<br />
= Proxy certificates =<br />
<br />
Authentication on the grid is done using special short lived ''proxy'' certificates. There are several tools available for creating, checking and destroying these proxy certificates.<br />
<br />
== Creating a proxy certificate ==<br />
<br />
To create a short lived proxy that can be used for authentication with grid services, the '''arcproxy''' command can be used. A 12 hour (default) proxy is created in the following example:<br />
<br />
$ arcproxy<br />
Your identity: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=Kalle Kula<br />
Enter pass phrase for /home/kalle/.globus/userkey.pem:<br />
.++++++<br />
.....++++++<br />
Proxy generation succeeded<br />
Your proxy is valid until: 2011-03-11 03:00:14<br />
<br />
The proxy file itself will be created in the '''/tmp''' directory with the format '''x509up_uid''', where uid is the user id number for your account.<br />
<br />
In some cases a longer lived proxy will be needed. This is achieved using the '''--constraint''' switch. A 24-hour can be created by issuing the following command:<br />
<br />
$ arcproxy --constraint="validityPeriod=24H"<br />
Your identity: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=Kalle Kula<br />
Enter pass phrase for /home/kalle/.globus/userkey.pem:<br />
....++++++<br />
.....++++++<br />
Proxy generation succeeded<br />
Your proxy is valid until: 2011-03-11 15:03:19<br />
<br />
== Creating a proxy certificate using the Firefox/Thunderbird credential store ==<br />
<br />
Using the ARC 3.x client tools it is now possible to generate a proxy certificate directly from the Firefox or Thunderbird credential stores. To do this the '''-F''' flag is used as shown in the following example:<br />
<br />
$ arcproxy -F<br />
There are 2 NSS base directories where the certificate, key, and module datbases live<br />
Number 1 is: /Users/lindemann/Library/Application Support/Firefox/Profiles/t22f3aj2.default<br />
Number 2 is: /Users/lindemann/Library/Thunderbird/Profiles/7abb733v.default<br />
Please choose the NSS database you would use (1-2): 1<br />
<br />
Here ARC finds the available Firefox and Thunderbird profile in which the credential stores are stored. Next the passphrase for the credential store is used to unlock the stored credentials:<br />
<br />
NSS database to be accessed: /Users/lindemann/Library/Application Support/Firefox/Profiles/t22f3aj2.default<br />
Enter Password or Pin for "internal (software)":<br />
<br />
If the passphrase was correct, ARC will list the available certificates in the credential store and ask you for which you would like to use.<br />
<br />
There are 2 user certificates existing in the NSS database<br />
Number 1 is with nickname: Jonas Lindemann xxxxx@lu.se's TERENA ID (Jonas Lindemann xxxxx@lu.se)<br />
expiration time: 2013-06-04 01:59:59<br />
Number 2 is with nickname: Imported Certificate (Jonas Lindemann)<br />
expiration time: 2014-01-18 16:55:52<br />
Please choose the one you would use (1-2): 1<br />
Certificate to use is: Jonas Lindemann xxxxxx@lu.se's TERENA ID<br />
Proxy generation succeeded<br />
Your proxy is valid until: 2013-05-01 04:11:37<br />
<br />
== Checking proxy lifetime ==<br />
<br />
The remaining lifetime of a proxy certificate can be checked using the '''arcproxy''' command with the '''--info''' switch.<br />
<br />
$ arcproxy --info<br />
Subject: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=Kalle Kula/CN=1567862803<br />
Identity: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=Kalle Kula<br />
Time left for proxy: 11 hours 55 minutes<br />
Proxy path: /tmp/x509up_u500<br />
Proxy type: X.509 Proxy Certificate Profile RFC compliant restricted proxy<br />
<br />
In this example the proxy certificate is valid for 11 hours 55 minutes more.<br />
<br />
== Destroying a proxy certificate ==<br />
<br />
A proxy can be destroyed with the '''-r''' or '''--remove''' switch.<br />
<br />
$ arcproxy -r<br />
<br />
or<br />
<br />
$ arcproxy --remove<br />
<br />
= VOMS certificates =<br />
<br />
As long as you are a member of only one VO or VO group, you can<br />
authenticate to a grid service with the regular grid proxy certificate<br />
as defined in the previous section. If you are a member of more than<br />
one VO or VO group you may want to select which membership you want to<br />
be authenticated as. For example, if you are a member of<br />
''swegrid.se:/swegrid.se/ops'' (operations staff) and<br />
''swegrid.se:/swegrid.se/bils'' and want to write a file, who should<br />
be the owner? Ops or bils? You need to provide some additional<br />
information. In the grid world this is done with a voms proxy<br />
certificate which basically is a regular proxy certificate but with a<br />
so called voms extension that contains a list of your VO group<br />
memberships (and roles and attributes, which we don't use in<br />
Swegrid/Swestore at the moment).<br />
<br />
'''Please note, if you only have one membership you can skip this section!'''<br />
<br />
The voms extension of the certificate is signed by the virtual<br />
organization management server, or VOMS server. The same VOMS server<br />
you used when applying for the swegrid.se VO membership in the first<br />
place. To enable this signing process you need to add a few<br />
configuration files to your system. First add this to the file<br />
'''/etc/vomses''':<br />
<br />
"swegrid.se" "voms.ndgf.org" "15009" "/O=Grid/O=NorduGrid/CN=host/voms.ndgf.org" "swegrid.se"<br />
<br />
Next create the necessary directories and the file<br />
'''/etc/grid-security/vomsdir/swegrid.se/voms.ndgf.org.lsc''' with the<br />
following contents:<br />
<br />
/O=Grid/O=NorduGrid/CN=host/voms.ndgf.org<br />
/O=Grid/O=NorduGrid/CN=NorduGrid Certification Authority<br />
<br />
== Creating a VOMS proxy ==<br />
<br />
VOMS proxies in ARC1 can be created using the '''arcproxy''' command<br />
and the '''-S''' or '''--voms''' switches as shown in the following<br />
example (if you are a member of the /swegrid.se/ops group. Adjust as<br />
necessary):<br />
<br />
$ arcproxy -S swegrid.se:/swegrid.se/ops<br />
Your identity: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=Kalle Kula<br />
Enter pass phrase for /home/kalle/.globus/userkey.pem:<br />
.....++++++<br />
............++++++<br />
Contacting VOMS server (named swegrid.se): voms.ndgf.org on port: 15009<br />
Proxy generation succeeded<br />
Your proxy is valid until: 2011-03-10 23:33:06<br />
<br />
<br />
= Signing your e-mail with your certificate =<br />
<br />
First, you will need your grid certificate in PKCS12 format:<br />
== How to transform your certificate from PEM format into PKCS#12 format ==<br />
<br />
This is how you transform your cert into PKCS12 format that can be used within your webbrowser or email send program:<br />
You first will have to change directory into where you created and keep the certificate, historically this is often in ~/.globus<br />
<br />
openssl pkcs12 -export -in usercert.pem -inkey userkey.pem -out cert+key.p12 <br />
<br />
First you will have to enter the password you used for your private key, then you will be asked for a new password to protect the new file. '''cert+key.p12 contains your private key, and is therefore 'lika känslig' as userkey.pem'''. See also [[#Introduction to certificates]]. Security wise the safest way is to delete the PKCS12 file after having imported it into your mail client or browser. Don't forget this.<br />
<br />
Remarks: openssl will either need the variable RANDFILE to be set or that ~/.rnd is writable. So you have to make sure that the current $HOME is yours if you have pagshed away, otherwise the command will fail with ''unable to write 'random state''.<br />
<br />
<br />
=== Signing in mew ===<br />
<br />
Mew uses gpgsm. <br />
<br />
<pre><br />
1. Import the nordugrid root cert<br />
<br />
1.1. get 1f0e8352.0 from nordugrid web<br />
<br />
1.2. gpgsm --import 1f0e8352.0<br />
<br />
1.2. Make it trusted:<br />
gpgsm --list-keys 2>/dev/null | grep fingerprint | awk '{print $2 " S"}' | grep THE-FINGERPRIT-YOU-WANT >> .gnupg/trustlist.txt<br />
<br />
2. Add your own key from the cert+key.p12 file in this case<br />
<br />
2.1 openssl pkcs12 -in cert+key.p12 -out tmp.pem -nokeys<br />
<br />
2.2. gpgsm --import tmp.pem ; rm tmp.pem<br />
<br />
2.3. Tell gpgsm not to use revocation lists (bad bad security)<br />
echo disable-crl-checks >> .gnupg/gpgsm.conf<br />
<br />
3. Test<br />
gpgsm --detach-sign file > sign # should ask for passphrase and give some kind of sign file<br />
<br />
4. Use:<br />
C-uC-cC-s then enter your email address (must match email in cert) and passphrase<br />
<br />
</pre><br />
<br />
=== Signing in thunderbird ===<br />
In thunderbird: options/security/digitally sign this message.<br />
<br />
If you do this for the first time and haven't defined yet the certificate to sign with, thunderbird will pop up the according preferences [Account settings/Security], where you can choose between your imported certificates in PKCS12 format.<br />
<br />
In the beginning, of course, you haven't imported any: Click there on the same preferences tab that popped up on [View Certificates]. In the new window that opens you can import the certificate.<br />
<br />
Afterwards you can then choose this certificate to be used for signing and for encryption for this email account.<br />
<br />
Don't forget to actually check that you then really sign the corresponding mail.</div>Lars Viklund (HPC2N)https://snicdocs.nsc.liu.se/w/index.php?title=Swestore-dCache&diff=4880Swestore-dCache2013-04-18T12:18:07Z<p>Lars Viklund (HPC2N): /* Getting access */</p>
<hr />
<div>[[Category:Storage]]<br />
[[Category:SweStore]]<br />
SNIC is building a storage infrastructure to complement the computational resources.<br />
<br />
Many forms of automated measurements can produce large amounts of data. In scientific areas such as high energy physics (the Large Hadron Collider at CERN), climate modeling, bioinformatics, bioimaging etc., the demands for storage are increasing dramatically. To serve these and other user communities, SNIC has appointed a working group to design a storage strategy, taking into account the needs on many levels and creating a unified storage infrastructure, which is now being implemented.<br />
<br />
Swestore is in collaboration with [http://www.ecds.se/ ECDS], [http://snd.gu.se/ SND], Bioimage Sweden, [http://www.bils.se/ BILS], [http://www.uppnex.uu.se/ UPPNEX],[http://wlcg.web.cern.ch/ WLCG], [http://www.nrm.se/ NaturHistoriska RiksMuseet].<br />
<br />
= National storage =<br />
The Swestore Nationally Accessible Storage, commonly called just Swestore, is a robust, flexible and expandable long<br />
term storage system aimed at storing large amounts of data produced by various Swedish research projects. It is based on the [http://www.dcache.org dCache]<br />
storage system and is distributed across the SNIC centres [http://www.c3se.chalmers.se/ C3SE], [http://www.hpc2n.umu.se/ HPC2N], [http://www.lunarc.lu.se/ Lunarc],<br />
[http://www.nsc.liu.se/ NSC], [http://www.pdc.kth.se PDC] and [http://www.uppmax.uu.se Uppmax].<br />
<br />
Data is stored in two copies with each copy at a different SNIC centre. This enables the system to cope with a multitude of issues ranging from a simple<br />
crash of a storage element to losing an entire site while stil providing access to the stored data. To protect against silent data corruption the<br />
dCache storage system checksums all stored data and periodically verifies the data using this checksum.<br />
<br />
The system does NOT yet provide protection against user errors like inadvertent file deletions and so on.<br />
<br />
One of the major advantages to the distributed nature of dCache is the excellent aggregated transfer rates possible. This is achieved by bypassing a central node<br />
and having transfers going directly to/from the storage elements if the protocol allows it.<br />
The Swestore Nationally Accessible Storage system can achieve aggregated transfer rates<br />
in excess of 100 Gigabit per second, but in practice this is limited by connectivity to each University (usually 10 Gbit/s) or a limited number of files (typically<br />
max 1 Gbit/s per file/connection).<br />
<br />
==Access protocols==<br />
; Currently supported protocols<br />
: GridFTP - gsiftp://gsiftp.swestore.se/<br />
: Storage Resource Manager - srm://srm.swegrid.se/<br />
: Hypertext Transfer Protocol (read-only), Web Distributed Authoring and Versioning - http://webdav.swestore.se/ (unauthenticated), https://webdav.swestore.se/<br />
<br />
; Protocols in evaluation/development<br />
: NFS4.1, iRODS<br />
<br />
For most of the access protocols the form of authentication is not username/password but X.509 client certificates, typically acquired from TCS eScience.<br />
<br />
== Getting access ==<br />
; Apply for storage<br />
: Please follow instructions [[Apply for storage on SweStore|here]]<br />
; Get a client certificate.<br />
: Follow the instructions [[Grid_certificates#Requesting_a_certificate|here]] to get your client certificate. For Terena certificates, please make sure you also [[Exporting_a_client_certificate|export the certificate for use with grid tools]]. For Nordugrid certificates, please make sure to also [[Requesting_a_grid_certificate_from_the_Nordugrid_CA#Installing_the_certificate_in_your_browser|install your client certificate in your browser]].<br />
; Request membership in the SweGrid VO.<br />
: Follow the instructions [[Grid_certificates#Requesting_membership_in_the_SweGrid_VO|here]] to get added to the SweGrid virtual organisation.<br />
; Transmit and prepare the certificate.<br />
: In order to use the client certificate on SNIC resources for generating proxy certificates and using command line tools, the certificate needs to be [[Preparing_a_client_certificate|converted into PEM files]] on the target cluster if not already in that format.<br />
<br />
== Download and upload data ==<br />
; Interactive browsing and manipulation of single files<br />
: SweStore is accessible in your web browser in two ways, as a directory index interface at https://webdav.swestore.se/ and with an interactive file manager at https://webdav.swestore.se/browser/. To browse private data you must first install your certificate in your browser (see above). Projects are organized under the <code>/snic</code> directory as <code><nowiki>https://webdav.swestore.se/snic/YOUR_PROJECT_NAME/</nowiki></code>.<br />
; Upload and delete data interactively or with automation<br />
There are several tools that are capable of using the protocols provided by SweStore national storage.<br />
For interactive usage on SNIC clusters we recommend using the ARC tools which should be installed on all SNIC resources.<br />
As an integration point for building scripts and automated systems we suggest using the curl program and library.<br />
: Use the ARC client. Please see the instructions for [[Accessing SweStore national storage with the ARC client]].<br />
: Use lftp. Please see the instructions for [[Accessing SweStore national storage with lftp]].<br />
: Use cURL. Please see the instructions for [[Accessing SweStore national storage with cURL]].<br />
: Use globus-url-copy. Please see the instructions for [[Accessing SweStore national storage with globus-url-copy]].<br />
<br />
== More information ==<br />
* [http://status.swestore.se/munin/monitor/monitor/ Per Project Monitoring of Swestore usage]<br />
* [[Accessing SweStore national storage with the ARC client]]<br />
<!-- * [[Mounting SweStore national storage via WebDAV|Mounting SweStore national storage via WebDAV (Not recomendated at the moment)]] --><br />
If you have any issues using SweStore please do not hesitate to contact [mailto:swestore-support@snic.vr.se swestore-support].<br />
<br />
== Tools and scripts ==<br />
<br />
There exists a number of tools and utilities developed externally that can be useful. Here are some links:<br />
<br />
* [https://github.com/samuell/arc_tools ARC_Tools] - Convenience scripts for the arc client (Only a recursive rmdir so far).<br />
* [http://sourceforge.net/projects/arc-gui-clients ARC Graphical Clients] - Contains the ARC Storage Explorer (SweStore supported development).<br />
* Transfer script, [http://snicdocs.nsc.liu.se/wiki/SweStore/swstrans_arc swetrans_arc], provided by Adam Peplinski / Philipp Schlatter<br />
* [http://www.nordugrid.org/documents/SWIG-wrapped-ARC-Python-API.pdf Documentation of the ARC Python API (PDF)]<br />
<br />
= Center storage =<br />
Centre storage, as defined by the SNIC storage group, is a storage solution that lives independently of the computational resources and can be accessed from all such resources at a centre. Key features include the ability to access the same filesystem the same way on all computational resources at a centre, and a unified structure and nomenclature for all centra. Unlike cluster storage which is tightly associated with a single cluster, and thus has a limited life-time, centre storage does not require the users to migrate their own data when clusters are decommissioned, not even when the storage hardware itself is being replaced.<br />
<br />
== Unified environment ==<br />
To make the usage more transparent for SNIC users, a set of environment variables are available on all SNIC resources:<br />
<br />
* <code>SNIC_BACKUP</code> – the user's primary directory at the centre<br>(the part of the centre storage that is backed up)<br />
* <code>SNIC_NOBACKUP</code> – recommended directory for project storage without backup<br>(also on the centre storage)<br />
* <code>SNIC_TMP</code> – recommended directory for best performance during a job<br>(local disk on nodes if applicable)</div>Lars Viklund (HPC2N)https://snicdocs.nsc.liu.se/w/index.php?title=Preparing_a_client_certificate&diff=4879Preparing a client certificate2013-04-18T11:56:59Z<p>Lars Viklund (HPC2N): /* Uploading and conversion of the .p12 for your target machine */</p>
<hr />
<div>Most of the standalone third party tools installed on SNIC resources and your own machine will not be able to use a <tt>.p12</tt> certificate bundle (or <tt>.pfx</tt> if you exported from IE), as that format is intended primarily for secure transport and backup of certificates and their private keys.<br />
<br />
Instead of a single <tt>.p12</tt> file, they expect a pair of files in <tt>.pem</tt> format, one containing the certificate and the other containing the private key that matches the certificate.<br />
<br />
== Uploading and conversion of the .p12 for your target machine ==<br />
<br />
As the authentication methods for clusters differ, this section will defer to documentations for your particular site when it comes to transferring files to and from the cluster storage.<br />
<br />
The goal is to end up with a <tt>.globus</tt> directory in your home directory, containing two files named <tt>usercert.pem</tt> and <tt>userkey.pem</tt>. The instructions below assume that your exported certificate file is named <tt>export.p12</tt> directly in your home directory. If it's a <tt>.pfx</tt> or with a different name, adjust accordingly.<br />
<br />
* Transfer the <tt>export.p12</tt> file to your home directory on the cluster.<br />
* Get an interactive shell on the login node, via ssh.<br />
* If an .globus directory already exists, rename it with something like<br />
<tt>mv ~/.globus ~/.globus-old</tt><br />
* Create the directory with<br />
<tt>mkdir ~/.globus</tt><br />
* Extract and protect the private key part of <tt>export.p12</tt>:<br />
openssl pkcs12 -nocerts -in ~/export.p12 -out ~/.globus/userkey.pem<br />
* When asked for import password, specify the password specified when exporting the certificate bundle from your browser. The PEM pass phrase should be a new password that you need to provide whenever using the certificate for tasks like generating a proxy certificate. The output from this command will be similar to the following:<br />
Enter Import Password: *******<br />
MAC verified OK<br />
Enter PEM pass phrase: *******<br />
Verifying - Enter PEM pass phrase: *******<br />
<br />
* Extract the public client certificate part of <tt>export.p12</tt>:<br />
openssl pkcs12 -clcerts -nokeys -in ~/export.p12 -out ~/globus/usercert.pem<br />
* The output will be similar to the following:<br />
Enter Import Password: *******<br />
MAC verified OK<br />
* Finally ensure that only your user is allowed to read the private key file. This is important, both for security and due to some tools refusing to use private keys with insufficient restrictions.<br />
chmod 0400 ~/.globus/userkey.pem</div>Lars Viklund (HPC2N)https://snicdocs.nsc.liu.se/w/index.php?title=Preparing_a_client_certificate&diff=4878Preparing a client certificate2013-04-18T11:35:38Z<p>Lars Viklund (HPC2N): /* Uploading and conversion of the .p12 for your target machine */</p>
<hr />
<div>Most of the standalone third party tools installed on SNIC resources and your own machine will not be able to use a <tt>.p12</tt> certificate bundle (or <tt>.pfx</tt> if you exported from IE), as that format is intended primarily for secure transport and backup of certificates and their private keys.<br />
<br />
Instead of a single <tt>.p12</tt> file, they expect a pair of files in <tt>.pem</tt> format, one containing the certificate and the other containing the private key that matches the certificate.<br />
<br />
== Uploading and conversion of the .p12 for your target machine ==<br />
<br />
As the authentication methods for clusters differ, this section will defer to documentations for your particular site when it comes to transferring files to and from the cluster storage.<br />
<br />
The goal is to end up with a <tt>.globus</tt> directory in your home directory, containing two files named <tt>usercert.pem</tt> and <tt>userkey.pem</tt>. The instructions below assume that your exported certificate file is named <tt>export.p12</tt> directly in your home directory. If it's a <tt>.pfx</tt> or with a different name, adjust accordingly.<br />
<br />
* Transfer the <tt>export.p12</tt> file to your home directory on the cluster.<br />
* Get an interactive shell on the login node, via ssh.<br />
* If an .globus directory already exists, rename it with something like<br />
<tt>mv ~/.globus ~/.globus-old</tt><br />
* Create the directory with<br />
<tt>mkdir ~/.globus</tt><br />
* Extract and protect the private key part of <tt>export.p12</tt>:<br />
``openssl pkcs12 -nocerts -in ~/export.p12 -out ~/.globus/userkey.pem``<br />
* When asked for import password, specify the password specified when exporting the certificate bundle from your browser. The PEM pass phrase should be a new password that you need to provide whenever using the certificate for tasks like generating a proxy certificate. The output from this command will be similar to the following:<br />
Enter Import Password: *******<br />
MAC verified OK<br />
Enter PEM pass phrase: *******<br />
Verifying - Enter PEM pass phrase: *******<br />
<br />
* Extract the public client certificate part of <tt>export.p12</tt>:<br />
openssl pkcs12 -clcerts -nokeys -in ~/export.p12 -out ~/globus/usercert.pem<br />
* The output will be similar to the following:<br />
Enter Import Password: *******<br />
MAC verified OK<br />
* Finally ensure that only your user is allowed to read the private key file. This is important, both for security and due to some tools refusing to use private keys with insufficient restrictions.<br />
chmod 0400 ~/.globus/userkey.pem</div>Lars Viklund (HPC2N)https://snicdocs.nsc.liu.se/w/index.php?title=Swestore-dCache&diff=4871Swestore-dCache2013-04-16T14:28:55Z<p>Lars Viklund (HPC2N): /* Access protocols */</p>
<hr />
<div>[[Category:Storage]]<br />
[[Category:SweStore]]<br />
SNIC is building a storage infrastructure to complement the computational resources.<br />
<br />
Many forms of automated measurements can produce large amounts of data. In scientific areas such as high energy physics (the Large Hadron Collider at CERN), climate modeling, bioinformatics, bioimaging etc., the demands for storage are increasing dramatically. To serve these and other user communities, SNIC has appointed a working group to design a storage strategy, taking into account the needs on many levels and creating a unified storage infrastructure, which is now being implemented.<br />
<br />
Swestore is in collaboration with [http://www.ecds.se/ ECDS], [http://snd.gu.se/ SND], Bioimage Sweden, [http://www.bils.se/ BILS], [http://www.uppnex.uu.se/ UPPNEX],[http://wlcg.web.cern.ch/ WLCG], [http://www.nrm.se/ NaturHistoriska RiksMuseet].<br />
<br />
= National storage =<br />
The Swestore Nationally Accessible Storage, commonly called just Swestore, is a robust, flexible and expandable long<br />
term storage system aimed at storing large amounts of data produced by various Swedish research projects. It is based on the [http://www.dcache.org dCache]<br />
storage system and is distributed across the SNIC centres [http://www.c3se.chalmers.se/ C3SE], [http://www.hpc2n.umu.se/ HPC2N], [http://www.lunarc.lu.se/ Lunarc],<br />
[http://www.nsc.liu.se/ NSC], [http://www.pdc.kth.se PDC] and [http://www.uppmax.uu.se Uppmax].<br />
<br />
Data is stored in two copies with each copy at a different SNIC centre. This enables the system to cope with a multitude of issues ranging from a simple<br />
crash of a storage element to losing an entire site while stil providing access to the stored data. To protect against silent data corruption the<br />
dCache storage system checksums all stored data and periodically verifies the data using this checksum.<br />
<br />
The system does NOT yet provide protection against user errors like inadvertent file deletions and so on.<br />
<br />
One of the major advantages to the distributed nature of dCache is the excellent aggregated transfer rates possible. This is achieved by bypassing a central node<br />
and having transfers going directly to/from the storage elements if the protocol allows it.<br />
The Swestore Nationally Accessible Storage system can achieve aggregated transfer rates<br />
in excess of 100 Gigabit per second, but in practice this is limited by connectivity to each University (usually 10 Gbit/s) or a limited number of files (typically<br />
max 1 Gbit/s per file/connection).<br />
<br />
==Access protocols==<br />
; Currently supported protocols<br />
: GridFTP - gsiftp://gsiftp.swestore.se/<br />
: Storage Resource Manager - srm://srm.swegrid.se/<br />
: Hypertext Transfer Protocol (read-only), Web Distributed Authoring and Versioning - http://webdav.swestore.se/ (unauthenticated), https://webdav.swestore.se/<br />
<br />
; Protocols in evaluation/development<br />
: NFS4.1, iRODS<br />
<br />
For most of the access protocols the form of authentication is not username/password but X.509 client certificates, typically acquired from TCS eScience.<br />
<br />
== Getting access ==<br />
; Apply for storage<br />
: Please follow instructions [[Apply for storage on SweStore|here]]<br />
; Get a client certificate.<br />
: Follow the instructions [[Grid_certificates#Requesting_a_certificate|here]] to get your client certificate. For Terena certificates, please make sure you also [[Exporting_a_client_certificate|export the certificate for use with grid tools]]. For Nordugrid certificates, please make sure to also [[Requesting_a_grid_certificate_from_the_Nordugrid_CA#Installing_the_certificate_in_your_browser|install your client certificate in your browser]].<br />
; Request membership in the SweGrid VO.<br />
: Follow the instructions [[Grid_certificates#Requesting_membership_in_the_SweGrid_VO|here]] to get added to the SweGrid virtual organisation.<br />
<br />
== Download and upload data ==<br />
; Interactive browsing and manipulation of single files<br />
: SweStore is accessible in your web browser in two ways, as a directory index interface at https://webdav.swestore.se/ and with an interactive file manager at https://webdav.swestore.se/browser/. To browse private data you must first install your certificate in your browser (see above). Projects are organized under the <code>/snic</code> directory as <code><nowiki>https://webdav.swestore.se/snic/YOUR_PROJECT_NAME/</nowiki></code>.<br />
; Upload and delete data interactively or with automation<br />
There are several tools that are capable of using the protocols provided by SweStore national storage.<br />
For interactive usage on SNIC clusters we recommend using the ARC tools which should be installed on all SNIC resources.<br />
As an integration point for building scripts and automated systems we suggest using the curl program and library.<br />
: Use the ARC client. Please see the instructions for [[Accessing SweStore national storage with the ARC client]].<br />
: Use lftp. Please see the instructions for [[Accessing SweStore national storage with lftp]].<br />
: Use cURL. Please see the instructions for [[Accessing SweStore national storage with cURL]].<br />
: Use globus-url-copy. Please see the instructions for [[Accessing SweStore national storage with globus-url-copy]].<br />
<br />
== More information ==<br />
* [http://status.swestore.se/munin/monitor/monitor/ Per Project Monitoring of Swestore usage]<br />
* [[Accessing SweStore national storage with the ARC client]]<br />
<!-- * [[Mounting SweStore national storage via WebDAV|Mounting SweStore national storage via WebDAV (Not recomendated at the moment)]] --><br />
If you have any issues using SweStore please do not hesitate to contact [mailto:swestore-support@snic.vr.se swestore-support].<br />
<br />
== Tools and scripts ==<br />
<br />
There exists a number of tools and utilities developed externally that can be useful. Here are some links:<br />
<br />
* [https://github.com/samuell/arc_tools ARC_Tools] - Convenience scripts for the arc client (Only a recursive rmdir so far).<br />
* [http://sourceforge.net/projects/arc-gui-clients ARC Graphical Clients] - Contains the ARC Storage Explorer (SweStore supported development).<br />
* Transfer script, [http://snicdocs.nsc.liu.se/wiki/SweStore/swstrans_arc swetrans_arc], provided by Adam Peplinski / Philipp Schlatter<br />
* [http://www.nordugrid.org/documents/SWIG-wrapped-ARC-Python-API.pdf Documentation of the ARC Python API (PDF)]<br />
<br />
= Center storage =<br />
Centre storage, as defined by the SNIC storage group, is a storage solution that lives independently of the computational resources and can be accessed from all such resources at a centre. Key features include the ability to access the same filesystem the same way on all computational resources at a centre, and a unified structure and nomenclature for all centra. Unlike cluster storage which is tightly associated with a single cluster, and thus has a limited life-time, centre storage does not require the users to migrate their own data when clusters are decommissioned, not even when the storage hardware itself is being replaced.<br />
<br />
== Unified environment ==<br />
To make the usage more transparent for SNIC users, a set of environment variables are available on all SNIC resources:<br />
<br />
* <code>SNIC_BACKUP</code> – the user's primary directory at the centre<br>(the part of the centre storage that is backed up)<br />
* <code>SNIC_NOBACKUP</code> – recommended directory for project storage without backup<br>(also on the centre storage)<br />
* <code>SNIC_TMP</code> – recommended directory for best performance during a job<br>(local disk on nodes if applicable)</div>Lars Viklund (HPC2N)https://snicdocs.nsc.liu.se/w/index.php?title=Swestore-dCache&diff=4869Swestore-dCache2013-04-16T14:01:22Z<p>Lars Viklund (HPC2N): /* Access protocols */</p>
<hr />
<div>[[Category:Storage]]<br />
[[Category:SweStore]]<br />
SNIC is building a storage infrastructure to complement the computational resources.<br />
<br />
Many forms of automated measurements can produce large amounts of data. In scientific areas such as high energy physics (the Large Hadron Collider at CERN), climate modeling, bioinformatics, bioimaging etc., the demands for storage are increasing dramatically. To serve these and other user communities, SNIC has appointed a working group to design a storage strategy, taking into account the needs on many levels and creating a unified storage infrastructure, which is now being implemented.<br />
<br />
Swestore is in collaboration with [http://www.ecds.se/ ECDS], [http://snd.gu.se/ SND], Bioimage Sweden, [http://www.bils.se/ BILS], [http://www.uppnex.uu.se/ UPPNEX],[http://wlcg.web.cern.ch/ WLCG], [http://www.nrm.se/ NaturHistoriska RiksMuseet].<br />
<br />
= National storage =<br />
The Swestore Nationally Accessible Storage, commonly called just Swestore, is a robust, flexible and expandable long<br />
term storage system aimed at storing large amounts of data produced by various Swedish research projects. It is based on the [http://www.dcache.org dCache]<br />
storage system and is distributed across the SNIC centres [http://www.c3se.chalmers.se/ C3SE], [http://www.hpc2n.umu.se/ HPC2N], [http://www.lunarc.lu.se/ Lunarc],<br />
[http://www.nsc.liu.se/ NSC], [http://www.pdc.kth.se PDC] and [http://www.uppmax.uu.se Uppmax].<br />
<br />
Data is stored in two copies with each copy at a different SNIC centre. This enables the system to cope with a multitude of issues ranging from a simple<br />
crash of a storage element to losing an entire site while stil providing access to the stored data. To protect against silent data corruption the<br />
dCache storage system checksums all stored data and periodically verifies the data using this checksum.<br />
<br />
The system does NOT yet provide protection against user errors like inadvertent file deletions and so on.<br />
<br />
One of the major advantages to the distributed nature of dCache is the excellent aggregated transfer rates possible. This is achieved by bypassing a central node<br />
and having transfers going directly to/from the storage elements if the protocol allows it.<br />
The Swestore Nationally Accessible Storage system can achieve aggregated transfer rates<br />
in excess of 100 Gigabit per second, but in practice this is limited by connectivity to each University (usually 10 Gbit/s) or a limited number of files (typically<br />
max 1 Gbit/s per file/connection).<br />
<br />
==Access protocols==<br />
; Currently supported protocols<br />
: GridFTP - gsiftp://gsiftp.swestore.se/<br />
: Storage Resource Manager - srm://srm.swegrid.se/<br />
: Hypertext Transfer Protocol (read-only), Web Distributed Authoring and Versioning - http://webdav.swestore.se/ (unauthenticated), https://webdav.swestore.se/<br />
<br />
; Protocols in evaluation/development<br />
: NFS4.1, iRODS<br />
<br />
For most of the access protocols the form of authentication is X.509 client certificates, typically acquired from TCS eScience.<br />
<br />
== Getting access ==<br />
; Apply for storage<br />
: Please follow instructions [[Apply for storage on SweStore|here]]<br />
; Get a client certificate.<br />
: Follow the instructions [[Grid_certificates#Requesting_a_certificate|here]] to get your client certificate. For Terena certificates, please make sure you also [[Exporting_a_client_certificate|export the certificate for use with grid tools]]. For Nordugrid certificates, please make sure to also [[Requesting_a_grid_certificate_from_the_Nordugrid_CA#Installing_the_certificate_in_your_browser|install your client certificate in your browser]].<br />
; Request membership in the SweGrid VO.<br />
: Follow the instructions [[Grid_certificates#Requesting_membership_in_the_SweGrid_VO|here]] to get added to the SweGrid virtual organisation.<br />
<br />
== Download and upload data ==<br />
; Interactive browsing and manipulation of single files<br />
: SweStore is accessible in your web browser in two ways, as a directory index interface at https://webdav.swestore.se/ and with an interactive file manager at https://webdav.swestore.se/browser/. To browse private data you must first install your certificate in your browser (see above). Projects are organized under the <code>/snic</code> directory as <code><nowiki>https://webdav.swestore.se/snic/YOUR_PROJECT_NAME/</nowiki></code>.<br />
; Upload and delete data interactively or with automation<br />
There are several tools that are capable of using the protocols provided by SweStore national storage.<br />
For interactive usage on SNIC clusters we recommend using the ARC tools which should be installed on all SNIC resources.<br />
As an integration point for building scripts and automated systems we suggest using the curl program and library.<br />
: Use the ARC client. Please see the instructions for [[Accessing SweStore national storage with the ARC client]].<br />
: Use lftp. Please see the instructions for [[Accessing SweStore national storage with lftp]].<br />
: Use cURL. Please see the instructions for [[Accessing SweStore national storage with cURL]].<br />
: Use globus-url-copy. Please see the instructions for [[Accessing SweStore national storage with globus-url-copy]].<br />
<br />
== More information ==<br />
* [http://status.swestore.se/munin/monitor/monitor/ Per Project Monitoring of Swestore usage]<br />
* [[Accessing SweStore national storage with the ARC client]]<br />
<!-- * [[Mounting SweStore national storage via WebDAV|Mounting SweStore national storage via WebDAV (Not recomendated at the moment)]] --><br />
If you have any issues using SweStore please do not hesitate to contact [mailto:swestore-support@snic.vr.se swestore-support].<br />
<br />
== Tools and scripts ==<br />
<br />
There exists a number of tools and utilities developed externally that can be useful. Here are some links:<br />
<br />
* [https://github.com/samuell/arc_tools ARC_Tools] - Convenience scripts for the arc client (Only a recursive rmdir so far).<br />
* [http://sourceforge.net/projects/arc-gui-clients ARC Graphical Clients] - Contains the ARC Storage Explorer (SweStore supported development).<br />
* Transfer script, [http://snicdocs.nsc.liu.se/wiki/SweStore/swstrans_arc swetrans_arc], provided by Adam Peplinski / Philipp Schlatter<br />
* [http://www.nordugrid.org/documents/SWIG-wrapped-ARC-Python-API.pdf Documentation of the ARC Python API (PDF)]<br />
<br />
= Center storage =<br />
Centre storage, as defined by the SNIC storage group, is a storage solution that lives independently of the computational resources and can be accessed from all such resources at a centre. Key features include the ability to access the same filesystem the same way on all computational resources at a centre, and a unified structure and nomenclature for all centra. Unlike cluster storage which is tightly associated with a single cluster, and thus has a limited life-time, centre storage does not require the users to migrate their own data when clusters are decommissioned, not even when the storage hardware itself is being replaced.<br />
<br />
== Unified environment ==<br />
To make the usage more transparent for SNIC users, a set of environment variables are available on all SNIC resources:<br />
<br />
* <code>SNIC_BACKUP</code> – the user's primary directory at the centre<br>(the part of the centre storage that is backed up)<br />
* <code>SNIC_NOBACKUP</code> – recommended directory for project storage without backup<br>(also on the centre storage)<br />
* <code>SNIC_TMP</code> – recommended directory for best performance during a job<br>(local disk on nodes if applicable)</div>Lars Viklund (HPC2N)https://snicdocs.nsc.liu.se/w/index.php?title=Preparing_a_client_certificate&diff=4858Preparing a client certificate2013-04-15T14:02:27Z<p>Lars Viklund (HPC2N): </p>
<hr />
<div>Most of the standalone third party tools installed on SNIC resources and your own machine will not be able to use a <tt>.p12</tt> certificate bundle (or <tt>.pfx</tt> if you exported from IE), as that format is intended primarily for secure transport and backup of certificates and their private keys.<br />
<br />
Instead of a single <tt>.p12</tt> file, they expect a pair of files in <tt>.pem</tt> format, one containing the certificate and the other containing the private key that matches the certificate.<br />
<br />
== Uploading and conversion of the .p12 for your target machine ==<br />
<br />
As the authentication methods for clusters differ, this section will defer to documentations for your particular site when it comes to transferring files to and from the cluster storage.<br />
<br />
The goal is to end up with a <tt>.globus</tt> directory in your home directory, containing two files named <tt>usercert.pem</tt> and <tt>userkey.pem</tt>.<br />
<br />
* Transfer the <tt>.p12</tt> file to your home directory on the cluster.<br />
* Get an interactive shell on the login node, via ssh.<br />
* If an .globus directory already exists, rename it with something like<br />
<tt>mv ~/.globus ~/.globus-old</tt><br />
* Create the directory with<br />
<tt>mkdir ~/.globus</tt><br />
* Run the following commands to extract the components from the <tt>.p12</tt> or <tt>.pfx</tt>, when asked for import password, specify the password specified when exporting the certificate bundle from your brower:<br />
openssl x509 ..<br />
openssl x509 ..<br />
<br />
chmod 0400 ~/.globus/usercert.pem<br />
chmod 0400 ~/.globus/userkey.pem</div>Lars Viklund (HPC2N)https://snicdocs.nsc.liu.se/w/index.php?title=Preparing_a_client_certificate&diff=4857Preparing a client certificate2013-04-12T11:29:18Z<p>Lars Viklund (HPC2N): Created page with "Most of the standalone third party tools installed on SNIC resources and your own machine will not be able to use a <tt>.p12</tt> certificate bundle, as that format is intended p..."</p>
<hr />
<div>Most of the standalone third party tools installed on SNIC resources and your own machine will not be able to use a <tt>.p12</tt> certificate bundle, as that format is intended primarily for secure transport and backup of certificates and their private keys.<br />
<br />
Instead of a single <tt>.p12</tt> file, they expect a pair of files in <tt>.pem</tt> format, one containing the certificate and the other containing the private key that matches the certificate.<br />
<br />
== Uploading the .p12 to your target machine ==<br />
<br />
== Prepare .globus directory in home directory ==<br />
<br />
== Make protected .pem files ==</div>Lars Viklund (HPC2N)https://snicdocs.nsc.liu.se/w/index.php?title=Swestore-dCache&diff=4841Swestore-dCache2013-03-27T15:55:31Z<p>Lars Viklund (HPC2N): /* Download and upload data */</p>
<hr />
<div>[[Category:Storage]]<br />
[[Category:SweStore]]<br />
SNIC is building a storage infrastructure to complement the computational resources.<br />
<br />
Many forms of automated measurements can produce large amounts of data. In scientific areas such as high energy physics (the Large Hadron Collider at CERN), climate modeling, bioinformatics, bioimaging etc., the demands for storage are increasing dramatically. To serve these and other user communities, SNIC has appointed a working group to design a storage strategy, taking into account the needs on many levels and creating a unified storage infrastructure, which is now being implemented.<br />
<br />
Swestore is in collaboration with [http://www.ecds.se/ ECDS], [http://snd.gu.se/ SND], Bioimage Sweden, [http://www.bils.se/ BILS], [http://www.uppnex.uu.se/ UPPNEX],[http://wlcg.web.cern.ch/ WLCG], [http://www.nrm.se/ NaturHistoriska RiksMuseet].<br />
<br />
= National storage =<br />
The aim of the nationally accessible storage is to build a robust, flexible and expandable system that can<br />
be used in most cases where access to large scale storage is needed. To the user it should appear as a single large system,<br />
while it is desirable that some parts of the system are distributed across all SNIC centra to benefit from the advantages<br />
of, among other things, locality and cache effects. The system is intended as a versatile long-term storage system.<br />
<br />
==Access protocols==<br />
; Currently supported protocols<br />
: srm://, gsiftp://, http:// (ro), https:// (ro), webdav (rw).<br />
; Protocols in evaluation/development<br />
: NFS4.1, iRODS<br />
<br />
== Getting access ==<br />
; Apply for storage<br />
: Please follow instructions [[Apply for storage on SweStore|here]]<br />
; Get a client certificate.<br />
: Follow the instructions [[Grid_certificates#Requesting_a_certificate|here]] to get your client certificate. For Terena certificates, please make sure you also [[Exporting_a_client_certificate|export the certificate for use with grid tools]]. For Nordugrid certificates, please make sure to also [[Requesting_a_grid_certificate_from_the_Nordugrid_CA#Installing_the_certificate_in_your_browser|install your client certificate in your browser]].<br />
; Request membership in the SweGrid VO.<br />
: Follow the instructions [[Grid_certificates#Requesting_membership_in_the_SweGrid_VO|here]] to get added to the SweGrid virtual organisation.<br />
<br />
== Download and upload data ==<br />
; Interactive browsing and manipulation of single files<br />
: SweStore is accessible in your web browser in two ways, as a directory index interface at https://webdav.swestore.se/ and with an interactive file manager at https://webdav.swestore.se/browser/. To browse private data you must first install your certificate in your browser (see above). Projects are organized under the <code>/snic</code> directory as <code><nowiki>https://webdav.swestore.se/snic/YOUR_PROJECT_NAME/</nowiki></code>.<br />
; Upload and delete data interactively or with automation<br />
There are several tools that are capable of using the protocols provided by SweStore national storage.<br />
For interactive usage on SNIC clusters we recommend using the ARC tools which should be installed on all SNIC resources.<br />
As an integration point for building scripts and automated systems we suggest using the curl program and library.<br />
: Use the ARC client. Please see the instructions for [[Accessing SweStore national storage with the ARC client]].<br />
: Use lftp. Please see the instructions for [[Accessing SweStore national storage with lftp]].<br />
: Use cURL. Please see the instructions for [[Accessing SweStore national storage with cURL]].<br />
: Use globus-url-copy. Please see the instructions for [[Accessing SweStore national storage with globus-url-copy]].<br />
<br />
== More information ==<br />
* [[SweStore introduction]]<br />
* [http://status.swestore.se/munin/monitor/monitor/ Per Project Monitoring of Swestore usage]<br />
* [[Accessing SweStore national storage with the ARC client]]<br />
<!-- * [[Mounting SweStore national storage via WebDAV|Mounting SweStore national storage via WebDAV (Not recomendated at the moment)]] --><br />
If you have any issues using SweStore please do not hesitate to contact [mailto:swestore-support@snic.vr.se swestore-support].<br />
<br />
== Tools and scripts ==<br />
<br />
There exists a number of tools and utilities developed externally that can be useful. Here are some links:<br />
<br />
* [https://github.com/samuell/arc_tools ARC_Tools] - Convenience scripts for the arc client (Only a recursive rmdir so far).<br />
* [http://sourceforge.net/projects/arc-gui-clients ARC Graphical Clients] - Contains the ARC Storage Explorer (SweStore supported development).<br />
* Transfer script, [http://snicdocs.nsc.liu.se/wiki/SweStore/swstrans_arc swetrans_arc], provided by Adam Peplinski / Philipp Schlatter<br />
* [http://www.nordugrid.org/documents/SWIG-wrapped-ARC-Python-API.pdf Documentation of the ARC Python API (PDF)]<br />
<br />
= Center storage =<br />
Centre storage, as defined by the SNIC storage group, is a storage solution that lives independently of the computational resources and can be accessed from all such resources at a centre. Key features include the ability to access the same filesystem the same way on all computational resources at a centre, and a unified structure and nomenclature for all centra. Unlike cluster storage which is tightly associated with a single cluster, and thus has a limited life-time, centre storage does not require the users to migrate their own data when clusters are decommissioned, not even when the storage hardware itself is being replaced.<br />
<br />
== Unified environment ==<br />
To make the usage more transparent for SNIC users, a set of environment variables are available on all SNIC resources:<br />
<br />
* <code>SNIC_BACKUP</code> – the user's primary directory at the centre<br>(the part of the centre storage that is backed up)<br />
* <code>SNIC_NOBACKUP</code> – recommended directory for project storage without backup<br>(also on the centre storage)<br />
* <code>SNIC_TMP</code> – recommended directory for best performance during a job<br>(local disk on nodes if applicable)</div>Lars Viklund (HPC2N)https://snicdocs.nsc.liu.se/w/index.php?title=Accessing_Swestore_with_lftp&diff=4840Accessing Swestore with lftp2013-03-27T15:52:55Z<p>Lars Viklund (HPC2N): /* Usage */</p>
<hr />
<div>== Introduction ==<br />
<br />
lftp is a file transfer tool that understands a range of protocols from plain old FTP, HTTP, SCP to more esoteric ones like WebDAV and BitTorrent.<br />
<br />
The benefit over tools like cURL is that it has interactive traversal of the directory hierarchy, as well as powerful mass-transfer functionality like the mirror command.<br />
<br />
Out of the protocols mentioned above, the ones that aligns most with the access doors that Swestore offers are the WebDAV over HTTP/HTTPS protocols.<br />
<br />
Authentication against Swestore over the WebDAV door is done with client certificates over HTTPS, where there is a choice of either using the real client certificate or by using a RFC or VOMS proxy certificate generated from the real certificate.<br />
<br />
Proxy certificates are preferred as they are valid for a limited period of time, especially as lftp does not seem to offer any way to enter any passphrase to unlock keys, so in order to use a protected real certificate, it will have to be unlocked in advance.<br />
<br />
Notable is that while file uploads are securely tunneled inside a SSL connection, downloads are in the plain from storage nodes. As such, the usual guidelines about sensitive data holds, where you should aim to have your data encrypted before transmitting it.<br />
<br />
== Required software versions ==<br />
<br />
The versions of lftp and its major dependency GNUTLS that have been verified to work with proxy certificates are:<br />
* lftp 4.3.3<br />
* gnutls 2.12.0<br />
* libnettle 2.4 (for building gnutls)<br />
<br />
This gnutls version is the absolute minimum version that will work, any version prior to that (2.10.5 and below) will not be able to connect to the door.<br />
<br />
This means that the gnutls version in Ubuntu oneric and older, Scientific Linux/RHEL/CentOS 6.1 and older will not work.<br />
<br />
Ubuntu precise has a sufficiently new gnutls and will work out of the box.<br />
<br />
== lftp settings ==<br />
<br />
While running the <code>lftp</code> program, there are several settings that need to be configured in order to successfully connect and interact with Swestore:<br />
<nowiki><br />
set ssl:ca-file /etc/grid-security/certificates/NorduGrid.pem<br />
set ssl:check-hostname true<br />
set ssl:verify-certificate true</nowiki><br />
<br />
and<br />
<nowiki><br />
set ssl:key-file /tmp/x509up_u1234<br />
set ssl:cert-file /tmp/x509up_u1234</nowiki><br />
<br />
where the last two are to indicate the filename of the proxy certificate generated by the <code>arcproxy</code> or <code>grid-proxy-init</code> tool. The trailing digits in the filename will vary based on the user ID (uid) of the local user, which you can find out in a terminal shell by running the <code>id</code> program.<br />
<br />
The CA file is for verifying the identity of the server and is strongly recommended that both certificate and hostname verification is in effect to ensure that the server communicated with is the intended target machine.<br />
<br />
They can be stored in the configuration file named ~/.lftp/rc together with any other commands you wish to perform during lftp startup.<br />
<br />
== Usage ==<br />
<br />
Assuming that you've managed to obtain a working lftp binary, there are some quirks that lftp has together with the Swestore WebDAV door.<br />
<br />
When giving a directory path to a command, it should end with a trailing slash to indicate that it is a directory. If this is omitted, the client will get a redirection response that the tool doesn't handle properly.<br />
<br />
All the commands mentioned in this section and the previous configuration section are commands inside of a running <code>lftp</code> program.<br />
<br />
Some sample tasks that can be achieved with lftp is retrieving or uploading single files or whole directory trees.<br />
<br />
The command to connect to the door is:<br />
<nowiki><br />
open https://webdav.swestore.se/</nowiki><br />
<br />
after which you can navigate around with the use of the 'cd' command:<br />
<nowiki><br />
cd snic/project_name_here/foo/</nowiki><br />
<br />
Individual files can be manipulated using the ''get'' and ''put'' commands and the ''mget'' and ''mirror'' commands can transfer multiple files and whole trees, respectively.<br />
<br />
The program has interactive help for any command through the ''help'' command.<br />
<br />
For mirroring the flags ''-R'' and ''-c'' are particularly relevant as ''-R'' controls the direction of the operation - if it is present the transfer is mirroring '''to''' the server, otherwise it's mirroring '''from''' the server.<br />
<br />
''-c'' indicates that the operation should resume whenever possible which may improve synchronization time if you know that any partial files present on the other side are identical to the local files.<br />
<br />
For example, the command<br />
<nowiki>mirror -c A B</nowiki><br />
<br />
will download all of the remote directory ''A'' into the local directory named ''B''.<br />
<br />
The command<br />
<nowiki>mirror -cR C D</nowiki><br />
<br />
will upload all of the local directory ''C'' into the remote directory named ''D''. Note that the role of the directories is reversed compared to the previous example.<br />
<br />
== Build instructions ==<br />
<br />
Building lftp and its dependencies from scratch does not require any particular build flags but you might want to install it into a private destination (prefix) to avoid it interfering with system-provided libraries.<br />
<br />
libnettle depends on GMP and lftp depends on readline and gperf, the distribution packages for those are sufficient on Ubuntu and CentOS 5/7 and 6.2.<br />
<br />
These instructions assume that the ''bash'' shell is used when building, ''tcsh'' and other shells will have slightly different syntax for environment variables.<br />
<nowiki><br />
export LFTP_PREFIX="${HOME}/local"<br />
export PKG_CONFIG_PATH="${LFTP_PREFIX}/lib/pkgconfig:${PKG_CONFIG_PATH}"<br />
export CPPFLAGS="-I${LFTP_PREFIX}/include"<br />
export LDFLAGS="-L${LFTP_PREFIX}/lib -Wl,-R${LFTP_PREFIX}/lib"</nowiki><br />
<br />
Start by extracting the source distributions:<br />
<nowiki><br />
tar xzf ~/Downloads/nettle-2.4.tar.gz<br />
tar xJf ~/Downloads/gnutls-3.0.12.tar.xz<br />
tar xJf ~/Downloads/lftp-4.3.5.tar.xz</nowiki><br />
<br />
You might want to avoid building a shared gnutls, so passing<br />
<nowiki><br />
--enable-static --disable-shared</nowiki><br />
<br />
on the gnutls configure command line might be a good idea.<br />
<br />
libnettle is a dependency for gnutls, and gnutls is a dependency for lftp, so we build them in that order:<br />
<nowiki><br />
mkdir ${LFTP_PREFIX}<br />
<br />
pushd nettle-2.4<br />
./configure --prefix=${LFTP_PREFIX} &&<br />
make && make install<br />
popd<br />
<br />
pushd gnutls-3.0.12<br />
./configure --prefix=${LFTP_PREFIX} --without-p11-kit &&<br />
make && make install<br />
popd<br />
<br />
pushd lftp-4.3.5<br />
./configure --prefix=${LFTP_PREFIX} &&<br />
make && make install<br />
popd</nowiki><br />
<br />
If the platform already has development files for p11-kit there is no harm in letting it use them (it allows gnutls to understand PKCS-style certificates), but it's nothing that lftp can utilize so it's not considered a dependency in this document.<br />
<br />
After the build process completes, a lftp binary will exist in ''${LFTP_PREFIX}/bin'' and depend on the gnutls shared library in ''${LFTP_PREFIX}/lib'' if you did not build it statically.<br />
<br />
= Credits =<br />
<br />
This guide was written by Lars Viklund</div>Lars Viklund (HPC2N)https://snicdocs.nsc.liu.se/w/index.php?title=Accessing_Swestore_with_lftp&diff=4839Accessing Swestore with lftp2013-03-27T15:51:33Z<p>Lars Viklund (HPC2N): /* lftp settings */</p>
<hr />
<div>== Introduction ==<br />
<br />
lftp is a file transfer tool that understands a range of protocols from plain old FTP, HTTP, SCP to more esoteric ones like WebDAV and BitTorrent.<br />
<br />
The benefit over tools like cURL is that it has interactive traversal of the directory hierarchy, as well as powerful mass-transfer functionality like the mirror command.<br />
<br />
Out of the protocols mentioned above, the ones that aligns most with the access doors that Swestore offers are the WebDAV over HTTP/HTTPS protocols.<br />
<br />
Authentication against Swestore over the WebDAV door is done with client certificates over HTTPS, where there is a choice of either using the real client certificate or by using a RFC or VOMS proxy certificate generated from the real certificate.<br />
<br />
Proxy certificates are preferred as they are valid for a limited period of time, especially as lftp does not seem to offer any way to enter any passphrase to unlock keys, so in order to use a protected real certificate, it will have to be unlocked in advance.<br />
<br />
Notable is that while file uploads are securely tunneled inside a SSL connection, downloads are in the plain from storage nodes. As such, the usual guidelines about sensitive data holds, where you should aim to have your data encrypted before transmitting it.<br />
<br />
== Required software versions ==<br />
<br />
The versions of lftp and its major dependency GNUTLS that have been verified to work with proxy certificates are:<br />
* lftp 4.3.3<br />
* gnutls 2.12.0<br />
* libnettle 2.4 (for building gnutls)<br />
<br />
This gnutls version is the absolute minimum version that will work, any version prior to that (2.10.5 and below) will not be able to connect to the door.<br />
<br />
This means that the gnutls version in Ubuntu oneric and older, Scientific Linux/RHEL/CentOS 6.1 and older will not work.<br />
<br />
Ubuntu precise has a sufficiently new gnutls and will work out of the box.<br />
<br />
== lftp settings ==<br />
<br />
While running the <code>lftp</code> program, there are several settings that need to be configured in order to successfully connect and interact with Swestore:<br />
<nowiki><br />
set ssl:ca-file /etc/grid-security/certificates/NorduGrid.pem<br />
set ssl:check-hostname true<br />
set ssl:verify-certificate true</nowiki><br />
<br />
and<br />
<nowiki><br />
set ssl:key-file /tmp/x509up_u1234<br />
set ssl:cert-file /tmp/x509up_u1234</nowiki><br />
<br />
where the last two are to indicate the filename of the proxy certificate generated by the <code>arcproxy</code> or <code>grid-proxy-init</code> tool. The trailing digits in the filename will vary based on the user ID (uid) of the local user, which you can find out in a terminal shell by running the <code>id</code> program.<br />
<br />
The CA file is for verifying the identity of the server and is strongly recommended that both certificate and hostname verification is in effect to ensure that the server communicated with is the intended target machine.<br />
<br />
They can be stored in the configuration file named ~/.lftp/rc together with any other commands you wish to perform during lftp startup.<br />
<br />
== Usage ==<br />
<br />
Assuming that you've managed to obtain a working lftp binary, there are some quirks that lftp has together with the Swestore WebDAV door.<br />
<br />
When giving a directory path to a command, it should end with a trailing slash to indicate that it is a directory. If this is omitted, the client will get a redirection response that the tool doesn't handle properly.<br />
<br />
Some sample tasks that can be achieved with lftp is retrieving or uploading single files or whole directory trees.<br />
<br />
The command to connect to the door is:<br />
<nowiki><br />
open https://webdav.swestore.se/</nowiki><br />
<br />
after which you can navigate around with the use of the 'cd' command:<br />
<nowiki><br />
cd snic/project_name_here/foo/</nowiki><br />
<br />
Individual files can be manipulated using the ''get'' and ''put'' commands and the ''mget'' and ''mirror'' commands can transfer multiple files and whole trees, respectively.<br />
<br />
The program has interactive help for any command through the ''help'' command.<br />
<br />
For mirroring the flags ''-R'' and ''-c'' are particularly relevant as ''-R'' controls the direction of the operation - if it is present the transfer is mirroring '''to''' the server, otherwise it's mirroring '''from''' the server.<br />
<br />
''-c'' indicates that the operation should resume whenever possible which may improve synchronization time if you know that any partial files present on the other side are identical to the local files.<br />
<br />
For example, the command<br />
<nowiki>mirror -c A B</nowiki><br />
<br />
will download all of the remote directory ''A'' into the local directory named ''B''.<br />
<br />
The command<br />
<nowiki>mirror -cR C D</nowiki><br />
<br />
will upload all of the local directory ''C'' into the remote directory named ''D''. Note that the role of the directories is reversed compared to the previous example.<br />
<br />
== Build instructions ==<br />
<br />
Building lftp and its dependencies from scratch does not require any particular build flags but you might want to install it into a private destination (prefix) to avoid it interfering with system-provided libraries.<br />
<br />
libnettle depends on GMP and lftp depends on readline and gperf, the distribution packages for those are sufficient on Ubuntu and CentOS 5/7 and 6.2.<br />
<br />
These instructions assume that the ''bash'' shell is used when building, ''tcsh'' and other shells will have slightly different syntax for environment variables.<br />
<nowiki><br />
export LFTP_PREFIX="${HOME}/local"<br />
export PKG_CONFIG_PATH="${LFTP_PREFIX}/lib/pkgconfig:${PKG_CONFIG_PATH}"<br />
export CPPFLAGS="-I${LFTP_PREFIX}/include"<br />
export LDFLAGS="-L${LFTP_PREFIX}/lib -Wl,-R${LFTP_PREFIX}/lib"</nowiki><br />
<br />
Start by extracting the source distributions:<br />
<nowiki><br />
tar xzf ~/Downloads/nettle-2.4.tar.gz<br />
tar xJf ~/Downloads/gnutls-3.0.12.tar.xz<br />
tar xJf ~/Downloads/lftp-4.3.5.tar.xz</nowiki><br />
<br />
You might want to avoid building a shared gnutls, so passing<br />
<nowiki><br />
--enable-static --disable-shared</nowiki><br />
<br />
on the gnutls configure command line might be a good idea.<br />
<br />
libnettle is a dependency for gnutls, and gnutls is a dependency for lftp, so we build them in that order:<br />
<nowiki><br />
mkdir ${LFTP_PREFIX}<br />
<br />
pushd nettle-2.4<br />
./configure --prefix=${LFTP_PREFIX} &&<br />
make && make install<br />
popd<br />
<br />
pushd gnutls-3.0.12<br />
./configure --prefix=${LFTP_PREFIX} --without-p11-kit &&<br />
make && make install<br />
popd<br />
<br />
pushd lftp-4.3.5<br />
./configure --prefix=${LFTP_PREFIX} &&<br />
make && make install<br />
popd</nowiki><br />
<br />
If the platform already has development files for p11-kit there is no harm in letting it use them (it allows gnutls to understand PKCS-style certificates), but it's nothing that lftp can utilize so it's not considered a dependency in this document.<br />
<br />
After the build process completes, a lftp binary will exist in ''${LFTP_PREFIX}/bin'' and depend on the gnutls shared library in ''${LFTP_PREFIX}/lib'' if you did not build it statically.<br />
<br />
= Credits =<br />
<br />
This guide was written by Lars Viklund</div>Lars Viklund (HPC2N)https://snicdocs.nsc.liu.se/w/index.php?title=Accessing_Swestore_with_lftp&diff=4838Accessing Swestore with lftp2013-03-27T15:51:14Z<p>Lars Viklund (HPC2N): /* lftp settings */</p>
<hr />
<div>== Introduction ==<br />
<br />
lftp is a file transfer tool that understands a range of protocols from plain old FTP, HTTP, SCP to more esoteric ones like WebDAV and BitTorrent.<br />
<br />
The benefit over tools like cURL is that it has interactive traversal of the directory hierarchy, as well as powerful mass-transfer functionality like the mirror command.<br />
<br />
Out of the protocols mentioned above, the ones that aligns most with the access doors that Swestore offers are the WebDAV over HTTP/HTTPS protocols.<br />
<br />
Authentication against Swestore over the WebDAV door is done with client certificates over HTTPS, where there is a choice of either using the real client certificate or by using a RFC or VOMS proxy certificate generated from the real certificate.<br />
<br />
Proxy certificates are preferred as they are valid for a limited period of time, especially as lftp does not seem to offer any way to enter any passphrase to unlock keys, so in order to use a protected real certificate, it will have to be unlocked in advance.<br />
<br />
Notable is that while file uploads are securely tunneled inside a SSL connection, downloads are in the plain from storage nodes. As such, the usual guidelines about sensitive data holds, where you should aim to have your data encrypted before transmitting it.<br />
<br />
== Required software versions ==<br />
<br />
The versions of lftp and its major dependency GNUTLS that have been verified to work with proxy certificates are:<br />
* lftp 4.3.3<br />
* gnutls 2.12.0<br />
* libnettle 2.4 (for building gnutls)<br />
<br />
This gnutls version is the absolute minimum version that will work, any version prior to that (2.10.5 and below) will not be able to connect to the door.<br />
<br />
This means that the gnutls version in Ubuntu oneric and older, Scientific Linux/RHEL/CentOS 6.1 and older will not work.<br />
<br />
Ubuntu precise has a sufficiently new gnutls and will work out of the box.<br />
<br />
== lftp settings ==<br />
<br />
While running the <code>lftp</code> program, there are several settings that need to be configured in order to successfully connect and interact with Swestore:<br />
<nowiki><br />
set ssl:ca-file /etc/grid-security/certificates/NorduGrid.pem<br />
set ssl:check-hostname true<br />
set ssl:verify-certificate true</nowiki><br />
<br />
and<br />
<nowiki><br />
set ssl:key-file /tmp/x509up_u1234<br />
set ssl:cert-file /tmp/x509up_u1234</nowiki><br />
<br />
where the last two are to indicate the filename of the proxy certificate generated by the <code>arcproxy</code> or <code>grid-proxy-init<code> tool. The trailing digits in the filename will vary based on the user ID (uid) of the local user, which you can find out in a terminal shell by running the <code>id</code> program.<br />
<br />
The CA file is for verifying the identity of the server and is strongly recommended that both certificate and hostname verification is in effect to ensure that the server communicated with is the intended target machine.<br />
<br />
They can be stored in the configuration file named ~/.lftp/rc together with any other commands you wish to perform during lftp startup.<br />
<br />
== Usage ==<br />
<br />
Assuming that you've managed to obtain a working lftp binary, there are some quirks that lftp has together with the Swestore WebDAV door.<br />
<br />
When giving a directory path to a command, it should end with a trailing slash to indicate that it is a directory. If this is omitted, the client will get a redirection response that the tool doesn't handle properly.<br />
<br />
Some sample tasks that can be achieved with lftp is retrieving or uploading single files or whole directory trees.<br />
<br />
The command to connect to the door is:<br />
<nowiki><br />
open https://webdav.swestore.se/</nowiki><br />
<br />
after which you can navigate around with the use of the 'cd' command:<br />
<nowiki><br />
cd snic/project_name_here/foo/</nowiki><br />
<br />
Individual files can be manipulated using the ''get'' and ''put'' commands and the ''mget'' and ''mirror'' commands can transfer multiple files and whole trees, respectively.<br />
<br />
The program has interactive help for any command through the ''help'' command.<br />
<br />
For mirroring the flags ''-R'' and ''-c'' are particularly relevant as ''-R'' controls the direction of the operation - if it is present the transfer is mirroring '''to''' the server, otherwise it's mirroring '''from''' the server.<br />
<br />
''-c'' indicates that the operation should resume whenever possible which may improve synchronization time if you know that any partial files present on the other side are identical to the local files.<br />
<br />
For example, the command<br />
<nowiki>mirror -c A B</nowiki><br />
<br />
will download all of the remote directory ''A'' into the local directory named ''B''.<br />
<br />
The command<br />
<nowiki>mirror -cR C D</nowiki><br />
<br />
will upload all of the local directory ''C'' into the remote directory named ''D''. Note that the role of the directories is reversed compared to the previous example.<br />
<br />
== Build instructions ==<br />
<br />
Building lftp and its dependencies from scratch does not require any particular build flags but you might want to install it into a private destination (prefix) to avoid it interfering with system-provided libraries.<br />
<br />
libnettle depends on GMP and lftp depends on readline and gperf, the distribution packages for those are sufficient on Ubuntu and CentOS 5/7 and 6.2.<br />
<br />
These instructions assume that the ''bash'' shell is used when building, ''tcsh'' and other shells will have slightly different syntax for environment variables.<br />
<nowiki><br />
export LFTP_PREFIX="${HOME}/local"<br />
export PKG_CONFIG_PATH="${LFTP_PREFIX}/lib/pkgconfig:${PKG_CONFIG_PATH}"<br />
export CPPFLAGS="-I${LFTP_PREFIX}/include"<br />
export LDFLAGS="-L${LFTP_PREFIX}/lib -Wl,-R${LFTP_PREFIX}/lib"</nowiki><br />
<br />
Start by extracting the source distributions:<br />
<nowiki><br />
tar xzf ~/Downloads/nettle-2.4.tar.gz<br />
tar xJf ~/Downloads/gnutls-3.0.12.tar.xz<br />
tar xJf ~/Downloads/lftp-4.3.5.tar.xz</nowiki><br />
<br />
You might want to avoid building a shared gnutls, so passing<br />
<nowiki><br />
--enable-static --disable-shared</nowiki><br />
<br />
on the gnutls configure command line might be a good idea.<br />
<br />
libnettle is a dependency for gnutls, and gnutls is a dependency for lftp, so we build them in that order:<br />
<nowiki><br />
mkdir ${LFTP_PREFIX}<br />
<br />
pushd nettle-2.4<br />
./configure --prefix=${LFTP_PREFIX} &&<br />
make && make install<br />
popd<br />
<br />
pushd gnutls-3.0.12<br />
./configure --prefix=${LFTP_PREFIX} --without-p11-kit &&<br />
make && make install<br />
popd<br />
<br />
pushd lftp-4.3.5<br />
./configure --prefix=${LFTP_PREFIX} &&<br />
make && make install<br />
popd</nowiki><br />
<br />
If the platform already has development files for p11-kit there is no harm in letting it use them (it allows gnutls to understand PKCS-style certificates), but it's nothing that lftp can utilize so it's not considered a dependency in this document.<br />
<br />
After the build process completes, a lftp binary will exist in ''${LFTP_PREFIX}/bin'' and depend on the gnutls shared library in ''${LFTP_PREFIX}/lib'' if you did not build it statically.<br />
<br />
= Credits =<br />
<br />
This guide was written by Lars Viklund</div>Lars Viklund (HPC2N)https://snicdocs.nsc.liu.se/w/index.php?title=Swestore-dCache&diff=4816Swestore-dCache2013-03-20T13:30:34Z<p>Lars Viklund (HPC2N): Linking landing page for certificate export</p>
<hr />
<div>[[Category:Storage]]<br />
[[Category:SweStore]]<br />
SNIC is building a storage infrastructure to complement the computational resources.<br />
<br />
Many forms of automated measurements can produce large amounts of data. In scientific areas such as high energy physics (the Large Hadron Collider at CERN), climate modeling, bioinformatics, bioimaging etc., the demands for storage are increasing dramatically. To serve these and other user communities, SNIC has appointed a working group to design a storage strategy, taking into account the needs on many levels and creating a unified storage infrastructure, which is now being implemented.<br />
<br />
Swestore is in collaboration with [http://www.ecds.se/ ECDS], [http://snd.gu.se/ SND], Bioimage Sweden, [http://www.bils.se/ BILS], [http://www.uppnex.uu.se/ UPPNEX],[http://wlcg.web.cern.ch/ WLCG], [http://www.nrm.se/ NaturHistoriska RiksMuseet].<br />
<br />
= National storage =<br />
The aim of the nationally accessible storage is to build a robust, flexible and expandable system that can<br />
be used in most cases where access to large scale storage is needed. To the user it should appear as a single large system,<br />
while it is desirable that some parts of the system are distributed across all SNIC centra to benefit from the advantages<br />
of, among other things, locality and cache effects. The system is intended as a versatile long-term storage system.<br />
<br />
==Access protocols==<br />
; Currently supported protocols<br />
: srm://, gsiftp://, http:// (ro), https:// (ro), webdav (rw).<br />
; Protocols in evaluation/development<br />
: NFS4.1, iRODS<br />
<br />
== Getting access ==<br />
; Apply for storage<br />
: Please follow instructions [[Apply for storage on SweStore|here]]<br />
; Get a client certificate.<br />
: Follow the instructions [[Grid_certificates#Requesting_a_certificate|here]] to get your client certificate. For Terena certificates, please make sure you also [[Exporting_a_client_certificate|export the certificate for use with grid tools]]. For Nordugrid certificates, please make sure to also [[Requesting_a_grid_certificate_from_the_Nordugrid_CA#Installing_the_certificate_in_your_browser|install your client certificate in your browser]].<br />
; Request membership in the SweGrid VO.<br />
: Follow the instructions [[Grid_certificates#Requesting_membership_in_the_SweGrid_VO|here]] to get added to the SweGrid virtual organisation.<br />
<br />
== Download and upload data ==<br />
; Interactive browsing and manipulation of single files<br />
: SweStore is accessible from your web browser, here https://webdav.swegrid.se/. To browse private data you must first install your certificate in your browser (see above). Your data is available at <code><nowiki>https://webdav.swegrid.se/snic/YOUR_PROJECT_NAME</nowiki></code>.<br />
; Upload and delete data interactively or with automation<br />
There are several tools that are capable of using the protocols provided by SweStore national storage.<br />
For interactive usage on SNIC clusters we recommend using the ARC tools which should be installed on all SNIC resources.<br />
As an integration point for building scripts and automated systems we suggest using the curl program and library.<br />
: Use the ARC client. Please see the instructions for [[Accessing SweStore national storage with the ARC client]].<br />
: Use cURL. Please see the instructions for [[Accessing SweStore national storage with cURL]].<br />
: Use lftp. Please see the instructions for [[Accessing SweStore national storage with lftp]].<br />
: Use globus-url-copy. Please see the instructions for [[Accessing SweStore national storage with globus-url-copy]].<br />
<br />
== More information ==<br />
* [[SweStore introduction]]<br />
* [http://status.swestore.se/munin/monitor/monitor/ Per Project Monitoring of Swestore usage]<br />
* [[Accessing SweStore national storage with the ARC client]]<br />
<!-- * [[Mounting SweStore national storage via WebDAV|Mounting SweStore national storage via WebDAV (Not recomendated at the moment)]] --><br />
If you have any issues using SweStore please do not hesitate to contact [mailto:swestore-support@snic.vr.se swestore-support].<br />
<br />
== Tools and scripts ==<br />
<br />
There exists a number of tools and utilities developed externally that can be useful. Here are some links:<br />
<br />
* [https://github.com/samuell/arc_tools ARC_Tools] - Convenience scripts for the arc client (Only a recursive rmdir so far).<br />
* [http://sourceforge.net/projects/arc-gui-clients ARC Graphical Clients] - Contains the ARC Storage Explorer (SweStore supported development).<br />
* Transfer script, [http://snicdocs.nsc.liu.se/wiki/SweStore/swstrans_arc swetrans_arc], provided by Adam Peplinski / Philipp Schlatter<br />
* [http://www.nordugrid.org/documents/SWIG-wrapped-ARC-Python-API.pdf Documentation of the ARC Python API (PDF)]<br />
<br />
= Center storage =<br />
Centre storage, as defined by the SNIC storage group, is a storage solution that lives independently of the computational resources and can be accessed from all such resources at a centre. Key features include the ability to access the same filesystem the same way on all computational resources at a centre, and a unified structure and nomenclature for all centra. Unlike cluster storage which is tightly associated with a single cluster, and thus has a limited life-time, centre storage does not require the users to migrate their own data when clusters are decommissioned, not even when the storage hardware itself is being replaced.<br />
<br />
== Unified environment ==<br />
To make the usage more transparent for SNIC users, a set of environment variables are available on all SNIC resources:<br />
<br />
* <code>SNIC_BACKUP</code> – the user's primary directory at the centre<br>(the part of the centre storage that is backed up)<br />
* <code>SNIC_NOBACKUP</code> – recommended directory for project storage without backup<br>(also on the centre storage)<br />
* <code>SNIC_TMP</code> – recommended directory for best performance during a job<br>(local disk on nodes if applicable)</div>Lars Viklund (HPC2N)https://snicdocs.nsc.liu.se/w/index.php?title=Exporting_a_client_certificate&diff=4815Exporting a client certificate2013-03-20T13:27:28Z<p>Lars Viklund (HPC2N): Created page with "In order to use your client certificate for tasks other than authenticating your browser sessions, you need to export it to a protected file which you can import into browsers on..."</p>
<hr />
<div>In order to use your client certificate for tasks other than authenticating your browser sessions, you need to export it to a protected file which you can import into browsers on other machines, import into other browsers, and upload to SNIC resources where it can be used to generate proxy certificates for use with client tools.<br />
<br />
The export process differs between operating systems and browsers, the following links outline the process for the common browsers and operating systems, resulting in a '''.p12''' or '''.pfx''' file.<br />
<br />
* [[Exporting_a_client_certificate_on_Windows|Exporting on Windows (Chrome and Internet Explorer)]]<br />
* [[Exporting_a_client_certificate_on_OS_X|Exporting on OS X (Chrome and Safari)]]<br />
* [[Requesting_a_grid_certificate_using_the_Terena_eScience_Portal#Exporting_Terena_certificate_for_use_with_Grid_tools|Exporting from Firefox]]</div>Lars Viklund (HPC2N)https://snicdocs.nsc.liu.se/w/index.php?title=Exporting_a_client_certificate_on_macOS&diff=4814Exporting a client certificate on macOS2013-03-20T12:59:33Z<p>Lars Viklund (HPC2N): </p>
<hr />
<div>''Safari'' and ''Chrome'' on uses the infrastructure present in the ''OS X'' operating system for certificate storage and retrieval, so in order to export a certificate you need to use the ''Keychain Access'' interface to manipulate and export certificates.<br />
<br />
* Launch ''Keychain Access'', select the certificate to export.<br />
[[File:cert_export-keychain-1.png|300px]]<br />
<br />
* In the context menu, select ''Export''.<br />
[[File:cert_export-keychain-2.png|300px]]<br />
<br />
* Enter a name for the certificate file and select a location for it. Ensure that the File Format is ''Personal Information Exchange (.p12)''.<br />
[[File:cert_export-keychain-3.png|300px]]<br />
<br />
* Enter and remember a password to protect the file with, you'll need this to access it later.</div>Lars Viklund (HPC2N)https://snicdocs.nsc.liu.se/w/index.php?title=Exporting_a_client_certificate_on_macOS&diff=4810Exporting a client certificate on macOS2013-03-20T12:58:41Z<p>Lars Viklund (HPC2N): Created page with "''Safari'' and ''Chrome'' on uses the infrastructure present in the ''OS X'' operating system for certificate storage and retrieval, so in order to export a certificate you need ..."</p>
<hr />
<div>''Safari'' and ''Chrome'' on uses the infrastructure present in the ''OS X'' operating system for certificate storage and retrieval, so in order to export a certificate you need to use the ''Keychain Access'' interface to manipulate and export certificates.<br />
<br />
Launch ''Keychain Access'', select the certificate to export.<br />
[[File:cert_export-keychain-1.png|300px]]<br />
<br />
In the context menu, select ''Export''.<br />
[[File:cert_export-keychain-2.png|300px]]<br />
<br />
Enter a name for the certificate file and select a location for it. Ensure that the File Format is ''Personal Information Exchange (.p12)''.<br />
[[File:cert_export-keychain-3.png|300px]]<br />
<br />
Enter and remember a password to protect the file with, you'll need this to access it later.</div>Lars Viklund (HPC2N)https://snicdocs.nsc.liu.se/w/index.php?title=Exporting_a_client_certificate_on_Windows&diff=4809Exporting a client certificate on Windows2013-03-20T12:44:42Z<p>Lars Viklund (HPC2N): </p>
<hr />
<div>Both Internet Explorer and Chrome uses the infrastructure present in the Windows operating system for certificate storage and retrieval, so the process to export is the same for both browsers.<br />
<br />
* In the ''Control Panel'', open ''Internet Options'' and click the ''Certificates'' button on the ''Content'' tab.<br />
[[File:cert_export-ie9-1.png|300px]]<br />
<br />
* On the ''Personal'' tab, select the certificate to export and click the ''Export...'' button.<br />
[[File:cert_export-ie9-2.png|300px]]<br />
<br />
* Follow the wizard process, select ''Yes, export the private key'' on the page asking whether to export the private key.<br />
<br />
* Select ''Personal Information Exchange - PKCS #12 (.PFX)'' on the ''Export File Format'' page.<br />
[[File:cert_export-ie9-3.png|300px]]<br />
<br />
* Enter and remember a password to protect the file with, you'll need this to access it later.<br />
<br />
* Select a location and filename for the exported certificate.<br />
<br />
* Finish the wizard.</div>Lars Viklund (HPC2N)https://snicdocs.nsc.liu.se/w/index.php?title=Exporting_a_client_certificate_on_Windows&diff=4805Exporting a client certificate on Windows2013-03-20T12:33:45Z<p>Lars Viklund (HPC2N): Created page with "Both Internet Explorer and Chrome uses the infrastructure present in the Windows operating system for certificate storage and retrieval, so the process to export is the same for ..."</p>
<hr />
<div>Both Internet Explorer and Chrome uses the infrastructure present in the Windows operating system for certificate storage and retrieval, so the process to export is the same for both browsers.<br />
<br />
* In the ''Control Panel'', open ''Internet Options'' and click the ''Certificates'' button on the ''Content'' tab.<br />
[[File:cert_export-ie9-1.png]]<br />
<br />
* On the ''Personal'' tab, select the certificate to export and click the ''Export...'' button.<br />
[[File:cert_export-ie9-2.png]]<br />
<br />
* Follow the wizard process, select ''Yes, export the private key'' on the page asking whether to export the private key.<br />
<br />
* Select ''Personal Information Exchange - PKCS #12 (.PFX)'' on the ''Export File Format'' page.<br />
[[File:cert_export-ie9-3.png]]<br />
<br />
* Enter and remember a password to protect the file with, you'll need this to access it later.<br />
<br />
* Select a location and filename for the exported certificate.<br />
<br />
* Finish the wizard.</div>Lars Viklund (HPC2N)https://snicdocs.nsc.liu.se/w/index.php?title=Swestore-dCache&diff=4741Swestore-dCache2013-02-22T08:27:10Z<p>Lars Viklund (HPC2N): /* Supported access protocol */</p>
<hr />
<div>[[Category:Storage]]<br />
[[Category:SweStore]]<br />
SNIC is building a storage infrastructure to complement the computational resources.<br />
<br />
Many forms of automated measurements can produce large amounts of data. In scientific areas such as high energy physics (the Large Hadron Collider at CERN), climate modeling, bioinformatics, bioimaging etc., the demands for storage are increasing dramatically. To serve these and other user communities, SNIC has appointed a working group to design a storage strategy, taking into account the needs on many levels and creating a unified storage infrastructure, which is now being implemented.<br />
<br />
Swestore is in collaboration with [http://www.ecds.se/ ECDS], [http://snd.gu.se/ SND], Bioimage Sweden, [http://www.bils.se/ BILS], [http://www.uppnex.uu.se/ UPPNEX],[http://wlcg.web.cern.ch/ WLCG], [http://www.nrm.se/ NaturHistoriska RiksMuseet].<br />
<br />
= National storage =<br />
The aim of the nationally accessible storage is to build a robust, flexible and expandable system that can<br />
be used in most cases where access to large scale storage is needed. To the user it should appear as a single large system,<br />
while it is desirable that some parts of the system are distributed across all SNIC centra to benefit from the advantages<br />
of, among other things, locality and cache effects. The system is intended as a versatile long-term storage system.<br />
<br />
==Access protocols==<br />
; Currently supported protocols<br />
: srm://, gsiftp://, http:// (ro), https:// (ro), webdav (rw).<br />
; Protocols in evaluation/development<br />
: NFS4.1, iRODS<br />
<br />
== Getting access ==<br />
; Apply for storage<br />
: Please follow instructions [[Apply for storage on SweStore|here]]<br />
; Get a client certificate.<br />
: Follow the instructions [[Grid_certificates#Requesting_a_certificate|here]] to get your client certificate. For Terena certificates, please make sure you also [[Requesting_a_grid_certificate_using_the_Terena_eScience_Portal#Exporting Terena certificate for use with Grid tools|export the certificate for use with grid tools]]. For Nordugrid certificates, please make sure to also [[Requesting_a_grid_certificate_from_the_Nordugrid_CA#Installing_the_certificate_in_your_browser|install your client certificate in your browser]].<br />
; Request membership in the SweGrid VO.<br />
: Follow the instructions [[Grid_certificates#Requesting_membership_in_the_SweGrid_VO|here]] to get added to the SweGrid virtual organisation.<br />
<br />
== Download and upload data ==<br />
; Interactive browsing and manipulation of single files<br />
: SweStore is accessible from your web browser, here https://webdav.swegrid.se/. To browse private data you must first install your certificate in your browser (see above). Your data is available at <code><nowiki>https://webdav.swegrid.se/snic/YOUR_PROJECT_NAME</nowiki></code>.<br />
; Upload and delete data interactively or with automation<br />
There are several tools that are capable of using the protocols provided by SweStore national storage.<br />
For interactive usage on SNIC clusters we recommend using the ARC tools which should be installed on all SNIC resources.<br />
As an integration point for building scripts and automated systems we suggest using the curl program and library.<br />
: Use the ARC client. Please see the instructions for [[Accessing SweStore national storage with the ARC client]].<br />
: Use cURL. Please see the instructions for [[Accessing SweStore national storage with cURL]].<br />
: Use lftp. Please see the instructions for [[Accessing SweStore national storage with lftp]].<br />
: Use globus-url-copy. Please see the instructions for [[Accessing SweStore national storage with globus-url-copy]].<br />
<br />
== More information ==<br />
* [[SweStore introduction]]<br />
* [http://status.swestore.se/munin/monitor/monitor/ Per Project Monitoring of Swestore usage]<br />
* [[Accessing SweStore national storage with the ARC client]]<br />
<!-- * [[Mounting SweStore national storage via WebDAV|Mounting SweStore national storage via WebDAV (Not recomendated at the moment)]] --><br />
If you have any issues using SweStore please do not hesitate to contact [mailto:swestore-support@snic.vr.se swestore-support].<br />
<br />
== Tools and scripts ==<br />
<br />
There exists a number of tools and utilities developed externally that can be useful. Here are some links:<br />
<br />
* [https://github.com/samuell/arc_tools ARC_Tools] - Convenience scripts for the arc client (Only a recursive rmdir so far).<br />
* [http://sourceforge.net/projects/arc-gui-clients ARC Graphical Clients] - Contains the ARC Storage Explorer (SweStore supported development).<br />
* Transfer script, [http://snicdocs.nsc.liu.se/wiki/SweStore/swstrans_arc swetrans_arc], provided by Adam Peplinski / Philipp Schlatter<br />
* [http://www.nordugrid.org/documents/SWIG-wrapped-ARC-Python-API.pdf Documentation of the ARC Python API (PDF)]<br />
<br />
= Center storage =<br />
Centre storage, as defined by the SNIC storage group, is a storage solution that lives independently of the computational resources and can be accessed from all such resources at a centre. Key features include the ability to access the same filesystem the same way on all computational resources at a centre, and a unified structure and nomenclature for all centra. Unlike cluster storage which is tightly associated with a single cluster, and thus has a limited life-time, centre storage does not require the users to migrate their own data when clusters are decommissioned, not even when the storage hardware itself is being replaced.<br />
<br />
== Unified environment ==<br />
To make the usage more transparent for SNIC users, a set of environment variables are available on all SNIC resources:<br />
<br />
* <code>SNIC_BACKUP</code> – the user's primary directory at the centre<br>(the part of the centre storage that is backed up)<br />
* <code>SNIC_NOBACKUP</code> – recommended directory for project storage without backup<br>(also on the centre storage)<br />
* <code>SNIC_TMP</code> – recommended directory for best performance during a job<br>(local disk on nodes if applicable)</div>Lars Viklund (HPC2N)https://snicdocs.nsc.liu.se/w/index.php?title=Swestore-dCache&diff=4740Swestore-dCache2013-02-20T08:08:45Z<p>Lars Viklund (HPC2N): /* Download and upload data */</p>
<hr />
<div>[[Category:Storage]]<br />
[[Category:SweStore]]<br />
SNIC is building a storage infrastructure to complement the computational resources.<br />
<br />
Many forms of automated measurements can produce large amounts of data. In scientific areas such as high energy physics (the Large Hadron Collider at CERN), climate modeling, bioinformatics, bioimaging etc., the demands for storage are increasing dramatically. To serve these and other user communities, SNIC has appointed a working group to design a storage strategy, taking into account the needs on many levels and creating a unified storage infrastructure, which is now being implemented.<br />
<br />
Swestore is in collaboration with [http://www.ecds.se/ ECDS], [http://snd.gu.se/ SND], Bioimage Sweden, [http://www.bils.se/ BILS], [http://www.uppnex.uu.se/ UPPNEX],[http://wlcg.web.cern.ch/ WLCG], [http://www.nrm.se/ NaturHistoriska RiksMuseet].<br />
<br />
= National storage =<br />
The aim of the nationally accessible storage is to build a robust, flexible and expandable system that can<br />
be used in most cases where access to large scale storage is needed. To the user it should appear as a single large system,<br />
while it is desirable that some parts of the system are distributed across all SNIC centra to benefit from the advantages<br />
of, among other things, locality and cache effects. The system is intended as a versatile long-term storage system.<br />
<br />
==Supported access protocol==<br />
; Today SweStore support this protocols<br />
: srm://, gsiftp://, http:// (ro), https:// (ro), webdav (rw).<br />
; Coming to support this protocols<br />
: NFS4.1, iRODS<br />
<br />
== Getting access ==<br />
; Apply for storage<br />
: Please follow instructions [[Apply for storage on SweStore|here]]<br />
; Get a client certificate.<br />
: Follow the instructions [[Grid_certificates#Requesting_a_certificate|here]] to get your client certificate. For Terena certificates, please make sure you also [[Requesting_a_grid_certificate_using_the_Terena_eScience_Portal#Exporting Terena certificate for use with Grid tools|export the certificate for use with grid tools]]. For Nordugrid certificates, please make sure to also [[Requesting_a_grid_certificate_from_the_Nordugrid_CA#Installing_the_certificate_in_your_browser|install your client certificate in your browser]].<br />
; Request membership in the SweGrid VO.<br />
: Follow the instructions [[Grid_certificates#Requesting_membership_in_the_SweGrid_VO|here]] to get added to the SweGrid virtual organisation.<br />
<br />
== Download and upload data ==<br />
; Interactive browsing and manipulation of single files<br />
: SweStore is accessible from your web browser, here https://webdav.swegrid.se/. To browse private data you must first install your certificate in your browser (see above). Your data is available at <code><nowiki>https://webdav.swegrid.se/snic/YOUR_PROJECT_NAME</nowiki></code>.<br />
; Upload and delete data interactively or with automation<br />
There are several tools that are capable of using the protocols provided by SweStore national storage.<br />
For interactive usage on SNIC clusters we recommend using the ARC tools which should be installed on all SNIC resources.<br />
As an integration point for building scripts and automated systems we suggest using the curl program and library.<br />
: Use the ARC client. Please see the instructions for [[Accessing SweStore national storage with the ARC client]].<br />
: Use cURL. Please see the instructions for [[Accessing SweStore national storage with cURL]].<br />
: Use lftp. Please see the instructions for [[Accessing SweStore national storage with lftp]].<br />
: Use globus-url-copy. Please see the instructions for [[Accessing SweStore national storage with globus-url-copy]].<br />
<br />
== More information ==<br />
* [[SweStore introduction]]<br />
* [http://status.swestore.se/munin/monitor/monitor/ Per Project Monitoring of Swestore usage]<br />
* [[Accessing SweStore national storage with the ARC client]]<br />
<!-- * [[Mounting SweStore national storage via WebDAV|Mounting SweStore national storage via WebDAV (Not recomendated at the moment)]] --><br />
If you have any issues using SweStore please do not hesitate to contact [mailto:swestore-support@snic.vr.se swestore-support].<br />
<br />
== Tools and scripts ==<br />
<br />
There exists a number of tools and utilities developed externally that can be useful. Here are some links:<br />
<br />
* [https://github.com/samuell/arc_tools ARC_Tools] - Convenience scripts for the arc client (Only a recursive rmdir so far).<br />
* [http://sourceforge.net/projects/arc-gui-clients ARC Graphical Clients] - Contains the ARC Storage Explorer (SweStore supported development).<br />
* Transfer script, [http://snicdocs.nsc.liu.se/wiki/SweStore/swstrans_arc swetrans_arc], provided by Adam Peplinski / Philipp Schlatter<br />
* [http://www.nordugrid.org/documents/SWIG-wrapped-ARC-Python-API.pdf Documentation of the ARC Python API (PDF)]<br />
<br />
= Center storage =<br />
Centre storage, as defined by the SNIC storage group, is a storage solution that lives independently of the computational resources and can be accessed from all such resources at a centre. Key features include the ability to access the same filesystem the same way on all computational resources at a centre, and a unified structure and nomenclature for all centra. Unlike cluster storage which is tightly associated with a single cluster, and thus has a limited life-time, centre storage does not require the users to migrate their own data when clusters are decommissioned, not even when the storage hardware itself is being replaced.<br />
<br />
== Unified environment ==<br />
To make the usage more transparent for SNIC users, a set of environment variables are available on all SNIC resources:<br />
<br />
* <code>SNIC_BACKUP</code> – the user's primary directory at the centre<br>(the part of the centre storage that is backed up)<br />
* <code>SNIC_NOBACKUP</code> – recommended directory for project storage without backup<br>(also on the centre storage)<br />
* <code>SNIC_TMP</code> – recommended directory for best performance during a job<br>(local disk on nodes if applicable)</div>Lars Viklund (HPC2N)https://snicdocs.nsc.liu.se/w/index.php?title=Swestore-dCache&diff=4739Swestore-dCache2013-02-20T07:28:46Z<p>Lars Viklund (HPC2N): /* Download and upload data */</p>
<hr />
<div>[[Category:Storage]]<br />
[[Category:SweStore]]<br />
SNIC is building a storage infrastructure to complement the computational resources.<br />
<br />
Many forms of automated measurements can produce large amounts of data. In scientific areas such as high energy physics (the Large Hadron Collider at CERN), climate modeling, bioinformatics, bioimaging etc., the demands for storage are increasing dramatically. To serve these and other user communities, SNIC has appointed a working group to design a storage strategy, taking into account the needs on many levels and creating a unified storage infrastructure, which is now being implemented.<br />
<br />
Swestore is in collaboration with [http://www.ecds.se/ ECDS], [http://snd.gu.se/ SND], Bioimage Sweden, [http://www.bils.se/ BILS], [http://www.uppnex.uu.se/ UPPNEX],[http://wlcg.web.cern.ch/ WLCG], [http://www.nrm.se/ NaturHistoriska RiksMuseet].<br />
<br />
= National storage =<br />
The aim of the nationally accessible storage is to build a robust, flexible and expandable system that can<br />
be used in most cases where access to large scale storage is needed. To the user it should appear as a single large system,<br />
while it is desirable that some parts of the system are distributed across all SNIC centra to benefit from the advantages<br />
of, among other things, locality and cache effects. The system is intended as a versatile long-term storage system.<br />
<br />
==Supported access protocol==<br />
; Today SweStore support this protocols<br />
: srm://, gsiftp://, http:// (ro), https:// (ro), webdav (rw).<br />
; Coming to support this protocols<br />
: NFS4.1, iRODS<br />
<br />
== Getting access ==<br />
; Apply for storage<br />
: Please follow instructions [[Apply for storage on SweStore|here]]<br />
; Get a client certificate.<br />
: Follow the instructions [[Grid_certificates#Requesting_a_certificate|here]] to get your client certificate. For Terena certificates, please make sure you also [[Requesting_a_grid_certificate_using_the_Terena_eScience_Portal#Exporting Terena certificate for use with Grid tools|export the certificate for use with grid tools]]. For Nordugrid certificates, please make sure to also [[Requesting_a_grid_certificate_from_the_Nordugrid_CA#Installing_the_certificate_in_your_browser|install your client certificate in your browser]].<br />
; Request membership in the SweGrid VO.<br />
: Follow the instructions [[Grid_certificates#Requesting_membership_in_the_SweGrid_VO|here]] to get added to the SweGrid virtual organisation.<br />
<br />
== Download and upload data ==<br />
; Interactive browsing and manipulation of single files<br />
: SweStore is accessible from your web browser, here https://webdav.swegrid.se/. To browse private data you must first install your certificate in your browser (see above). Your data is available at <code><nowiki>https://webdav.swegrid.se/snic/YOUR_PROJECT_NAME</nowiki></code>.<br />
; Upload and delete data interactively or with automation<br />
There are several tools that are capable of using the protocols provided by SweStore national storage.<br />
For interactive usage on SNIC clusters and resources we recommend using the ARC tools.<br />
As an integration point for building scripts and automated systems we suggest using the curl program and library.<br />
: Use the ARC client. Please see the instructions for [[Accessing SweStore national storage with the ARC client]].<br />
: Use cURL. Please see the instructions for [[Accessing SweStore national storage with cURL]].<br />
: Use lftp. Please see the instructions for [[Accessing SweStore national storage with lftp]].<br />
: Use globus-url-copy. Please see the instructions for [[Accessing SweStore national storage with globus-url-copy]].<br />
<br />
== More information ==<br />
* [[SweStore introduction]]<br />
* [http://status.swestore.se/munin/monitor/monitor/ Per Project Monitoring of Swestore usage]<br />
* [[Accessing SweStore national storage with the ARC client]]<br />
<!-- * [[Mounting SweStore national storage via WebDAV|Mounting SweStore national storage via WebDAV (Not recomendated at the moment)]] --><br />
If you have any issues using SweStore please do not hesitate to contact [mailto:swestore-support@snic.vr.se swestore-support].<br />
<br />
== Tools and scripts ==<br />
<br />
There exists a number of tools and utilities developed externally that can be useful. Here are some links:<br />
<br />
* [https://github.com/samuell/arc_tools ARC_Tools] - Convenience scripts for the arc client (Only a recursive rmdir so far).<br />
* [http://sourceforge.net/projects/arc-gui-clients ARC Graphical Clients] - Contains the ARC Storage Explorer (SweStore supported development).<br />
* Transfer script, [http://snicdocs.nsc.liu.se/wiki/SweStore/swstrans_arc swetrans_arc], provided by Adam Peplinski / Philipp Schlatter<br />
* [http://www.nordugrid.org/documents/SWIG-wrapped-ARC-Python-API.pdf Documentation of the ARC Python API (PDF)]<br />
<br />
= Center storage =<br />
Centre storage, as defined by the SNIC storage group, is a storage solution that lives independently of the computational resources and can be accessed from all such resources at a centre. Key features include the ability to access the same filesystem the same way on all computational resources at a centre, and a unified structure and nomenclature for all centra. Unlike cluster storage which is tightly associated with a single cluster, and thus has a limited life-time, centre storage does not require the users to migrate their own data when clusters are decommissioned, not even when the storage hardware itself is being replaced.<br />
<br />
== Unified environment ==<br />
To make the usage more transparent for SNIC users, a set of environment variables are available on all SNIC resources:<br />
<br />
* <code>SNIC_BACKUP</code> – the user's primary directory at the centre<br>(the part of the centre storage that is backed up)<br />
* <code>SNIC_NOBACKUP</code> – recommended directory for project storage without backup<br>(also on the centre storage)<br />
* <code>SNIC_TMP</code> – recommended directory for best performance during a job<br>(local disk on nodes if applicable)</div>Lars Viklund (HPC2N)https://snicdocs.nsc.liu.se/w/index.php?title=Apply_for_storage_on_Swestore&diff=4674Apply for storage on Swestore2013-01-30T14:43:24Z<p>Lars Viklund (HPC2N): language optimisation</p>
<hr />
<div>[[Category:SweStore]]<br />
[[Category:SweStore user guide]]<br />
The SweStore nationally accessible storage is available for researchers financed by VR (which includes all researchers using SNIC compute resources) and FORMA.<br />
<br />
SweStore is in collaboration with [http://www.ecds.se/ ECDS], [http://snd.gu.se/ SND], Bioimage, [http://www.bils.se/ BILS], [http://www.uppnex.uu.se/ UPPNEX],[http://wlcg.web.cern.ch/ WLCG], [http://www.nrm.se/ Naturhistoriska Riksmuseet]. If any of these cover your research area, first read their information on applying for SweStore storage.<br />
<br />
In the future, applications for storage will be handled by each research community, but for now an email to [mailto:swestore-support@snic.vr.se swestore-support@snic.vr.se] will suffice. <br />
<br />
Please include the following information in the application:<br />
* Name of the principal investigator (PI).<br />
* Purpose for the storage: A short description of the project and type of data.<br />
* Required storage capacity: Preferably a maximum size, but if this is not currently determinable, please calculate a starting size and expansion by time period. '''NOTE''' that applications larger than 10TB takes longer to process.<br />
* Suggested project name: This will be used as root directory name for your storage.<br />
# '''NOTE''' that this name is long-lived and will persist. It is not coupled to the lifetime of SNIC compute time allocations.<br />
# The name must be limited to lower-case letters a-z, digits 0-9 and underscores _.</div>Lars Viklund (HPC2N)https://snicdocs.nsc.liu.se/w/index.php?title=Swestore-dCache&diff=4653Swestore-dCache2013-01-14T13:28:13Z<p>Lars Viklund (HPC2N): fix broken links to collaborators (WLCG)</p>
<hr />
<div>[[Category:Storage]]<br />
[[Category:SweStore]]<br />
SNIC is building a storage infrastructure to complement the computational resources.<br />
<br />
Many forms of automated measurements can produce large amounts of data. In scientific areas such as high energy physics (the Large Hadron Collider at CERN), climate modeling, bioinformatics, bioimaging etc., the demands for storage are increasing dramatically. To serve these and other user communities, SNIC has appointed a working group to design a storage strategy, taking into account the needs on many levels and creating a unified storage infrastructure, which is now being implemented.<br />
<br />
Swestore is in collaboration with [http://www.ecds.se/ ECDS], [http://snd.gu.se/ SND], Bioimage Sweden, [http://www.bils.se/ BILS], [http://www.uppnex.uu.se/ UPPNEX],[http://wlcg.web.cern.ch/ WLCG], [http://www.nrm.se/ NaturHistoriska RiksMuseet].<br />
<br />
= National storage =<br />
The aim of the nationally accessible storage is to build a robust, flexible and expandable system that can<br />
be used in most cases where access to large scale storage is needed. To the user it should appear as a single large system,<br />
while it is desirable that some parts of the system are distributed across all SNIC centra to benefit from the advantages<br />
of, among other things, locality and cache effects. The system is intended as a versatile long-term storage system.<br />
<br />
==Supported access protocol==<br />
; Today SweStore support this protocols<br />
: srm://, gsiftp://, http:// (ro), https:// (ro), webdav (rw).<br />
; Coming to support this protocols<br />
: NFS4.1, iRODS<br />
<br />
== Getting access ==<br />
; Apply for storage<br />
: Please follow instructions [[Apply for storage on SweStore|here]]<br />
; Get a client certificate.<br />
: Follow the instructions [[Grid_certificates#Requesting_a_certificate|here]] to get your client certificate. For Terena certificates, please make sure you also [[Requesting_a_grid_certificate_using_the_Terena_eScience_Portal#Exporting Terena certificate for use with Grid tools|export the certificate for use with grid tools]]. For Nordugrid certificates, please make sure to also [[Requesting_a_grid_certificate_from_the_Nordugrid_CA#Installing_the_certificate_in_your_browser|install your client certificate in your browser]].<br />
; Request membership in the SweGrid VO.<br />
: Follow the instructions [[Grid_certificates#Requesting_membership_in_the_SweGrid_VO|here]] to get added to the SweGrid virtual organisation.<br />
<br />
== Download and upload data ==<br />
; Browse and download data<br />
: SweStore is accessible from your web browser, here https://webdav.swegrid.se/. To browse private data you must first install your certificate in your browser (see above). Your data is available at <code><nowiki>https://webdav.swegrid.se/snic/YOUR_PROJECT_NAME</nowiki></code>.<br />
; Upload and delete data<br />
: Use the ARC client. Please see the instructions for [[Accessing SweStore national storage with the ARC client]].<br />
: Use cURL. Please see the instructions for [[Accessing SweStore national storage with cURL]].<br />
: Use lftp. Please see the instructions for [[Accessing SweStore national storage with lftp]].<br />
: Use globus-url-copy. Please see the instructions for [[Accessing SweStore national storage with globus-url-copy]].<br />
<br />
== Examples of storage projects ==<br />
Below are some examples of project that are using SweStore today.<br />
<br />
{|border="1" style="text-align:left; border-collapse: collapse; border-width: 1px; border-style: solid; border-color: #000" class="wikitable sortable" valign=top<br />
!Allocation name<br />
!Size in TB<br />
!class="unsortable"|Project full name<br />
|-<br />
|alice<br />
|400<br />
|<br />
|-<br />
|uppnex<br />
|140<br />
|[https://www.uppnex.uu.se UPPmax NExt Generation Sequencing Cluster & Storage]<br />
|-<br />
|brain_protein_atlas<br />
|10<br />
|Mouse brain protein atlas project<br />
|-<br />
| scims2lab<br />
|20<br />
| Identification of novel gene models by matching mass spectrometry data against 6-frame translations of the human genome<br />
|-<br />
|subatom<br />
|<br />
|Low-energy nuclear theory and experiment<br />
|-<br />
|genomics-gu<br />
|10<br />
|Genomics Core Facility, Sahlgrenska academy at University of Gothenburg.<br />
|-<br />
|Chemo<br />
|5TB<br />
|Genetic interaction networks in human deseas<br />
|-<br />
|cesm1_holocene<br />
|30<br />
|Arctic sea ice in warm climates<br />
|}<br />
<br />
== More information ==<br />
* [[SweStore introduction]]<br />
* [http://status.swestore.se/munin/monitor/monitor/ Per Project Monitoring of Swestore usage]<br />
* [[Accessing SweStore national storage with the ARC client]]<br />
<!-- * [[Mounting SweStore national storage via WebDAV|Mounting SweStore national storage via WebDAV (Not recomendated at the moment)]] --><br />
If you have any issues using SweStore please do not hesitate to contact [mailto:swestore-support@snic.vr.se swestore-support].<br />
<br />
== Tools and scripts ==<br />
<br />
There exists a number of tools and utilities developed externally that can be useful. Here are some links:<br />
<br />
* [https://github.com/samuell/arc_tools ARC_Tools] - Convenience scripts for the arc client (Only a recursive rmdir so far).<br />
* [http://sourceforge.net/projects/arc-gui-clients ARC Graphical Clients] - Contains the ARC Storage Explorer (SweStore supported development).<br />
* Transfer script, [http://snicdocs.nsc.liu.se/wiki/SweStore/swstrans_arc swetrans_arc], provided by Adam Peplinski / Philipp Schlatter<br />
* [http://www.nordugrid.org/documents/SWIG-wrapped-ARC-Python-API.pdf Documentation of the ARC Python API (PDF)]<br />
<br />
= Center storage =<br />
Centre storage, as defined by the SNIC storage group, is a storage solution that lives independently of the computational resources and can be accessed from all such resources at a centre. Key features include the ability to access the same filesystem the same way on all computational resources at a centre, and a unified structure and nomenclature for all centra. Unlike cluster storage which is tightly associated with a single cluster, and thus has a limited life-time, centre storage does not require the users to migrate their own data when clusters are decommissioned, not even when the storage hardware itself is being replaced.<br />
<br />
== Unified environment ==<br />
To make the usage more transparent for SNIC users, a set of environment variables are available on all SNIC resources:<br />
<br />
* <code>SNIC_BACKUP</code> – the user's primary directory at the centre<br>(the part of the centre storage that is backed up)<br />
* <code>SNIC_NOBACKUP</code> – recommended directory for project storage without backup<br>(also on the centre storage)<br />
* <code>SNIC_TMP</code> – recommended directory for best performance during a job<br>(local disk on nodes if applicable)</div>Lars Viklund (HPC2N)https://snicdocs.nsc.liu.se/w/index.php?title=Apply_for_storage_on_Swestore&diff=4652Apply for storage on Swestore2013-01-14T13:27:27Z<p>Lars Viklund (HPC2N): broken link to WLCG</p>
<hr />
<div>[[Category:SweStore]]<br />
[[Category:SweStore user guide]]<br />
Swestore is in collaboration with [http://www.ecds.se ECDS], [http://snd.gu.se SND], Bioimage, [http://www.bils.se/ BILS], [http://www.uppnex.uu.se/ UPPNEX],[http://wlcg.web.cern.ch/ WLCG], [http://www.nrm.se/ Naturhistoriska Riksmuseet]. If any of these cover your research area, first read their information on applying for SweStore storage. In the future, applications for storage will be handled by each research community, but for now an email to [mailto:swestore-support@snic.vr.se swestore-support@snic.vr.se] will suffice. <br />
<br />
Please include the following information:<br />
* Name of the principal investigator (PI).<br />
* Purpose for the storage: A short description of the project and type of data.<br />
* Required storage capacity: Preferably a maximum size, but if this is not currently determinable, please calculate a starting size and expansion by time period. *NOTE* that applications larger than 10TB takes longer to process.<br />
* Suggested project name: This will be used as root directory name for your storage. *NOTE* that this name is long-lived and will persist. It is not coupled to the lifetime of SNIC compute time allocations.</div>Lars Viklund (HPC2N)https://snicdocs.nsc.liu.se/w/index.php?title=Accessing_SweStore_national_storage_with_the_cURL&diff=4475Accessing SweStore national storage with the cURL2012-09-19T13:52:57Z<p>Lars Viklund (HPC2N): Redirecting to avoid duplicates</p>
<hr />
<div>#REDIRECT [[Accessing SweStore national storage with cURL]]</div>Lars Viklund (HPC2N)https://snicdocs.nsc.liu.se/w/index.php?title=Swestore-dCache&diff=4474Swestore-dCache2012-09-19T13:51:40Z<p>Lars Viklund (HPC2N): /* Download and upload data */</p>
<hr />
<div>[[Category:Storage]]<br />
[[Category:SweStore]]<br />
SNIC is building a storage infrastructure to complement the computational resources.<br />
<br />
Many forms of automated measurements can produce large amounts of data. In scientific areas such as high energy physics (the Large Hadron Collider at CERN), climate modeling, bioinformatics, bioimaging etc., the demands for storage are increasing dramatically. To serve these and other user communities, SNIC has appointed a working group to design a storage strategy, taking into account the needs on many levels and creating a unified storage infrastructure, which is now being implemented.<br />
<br />
Swestore is in collaboration with [http://www.ecds.se ECDS], [http://snd.gu.se SND], Bioimage Sweden, [http://www.bils.se BILS], [http://www.uppnex.uu.se UPPNEX],[http://http://lcg.web.cern.ch/lcg/public/ WLCG], [http://www.nrm.se/ NaturHistoriska RiksMuseet].<br />
<br />
= National storage =<br />
The aim of the nationally accessible storage is to build a robust, flexible and expandable system that can<br />
be used in most cases where access to large scale storage is needed. To the user it should appear as a single large system,<br />
while it is desirable that some parts of the system are distributed across all SNIC centra to benefit from the advantages<br />
of, among other things, locality and cache effects. The system is intended as a versatile long-term storage system.<br />
<br />
==Supported access protocol==<br />
; Today SweStore support this protocols<br />
: srm://, gsiftp://, http:// (ro), https:// (ro), webdav (rw).<br />
; Coming to support this protocols<br />
: NFS4.1, iRODS<br />
<br />
== Getting access ==<br />
; Apply for storage<br />
: Please follow instructions [[Apply for storage on SweStore|here]]<br />
; Get a client certificate.<br />
: Follow the instructions [[Grid_certificates#Requesting_a_certificate|here]] to get your client certificate. For Terena certificates, please make sure you also [[Requesting_a_grid_certificate_using_the_Terena_eScience_Portal#Exporting Terena certificate for use with Grid tools|export the certificate for use with grid tools]]. For Nordugrid certificates, please make sure to also [[Requesting_a_grid_certificate_from_the_Nordugrid_CA#Installing_the_certificate_in_your_browser|install your client certificate in your browser]].<br />
; Request membership in the SweGrid VO.<br />
: Follow the instructions [[Grid_certificates#Requesting_membership_in_the_SweGrid_VO|here]] to get added to the SweGrid virtual organisation.<br />
<br />
== Download and upload data ==<br />
; Browse and download data<br />
: SweStore is accessible from your web browser, here https://webdav.swegrid.se/. To browse private data you must first install your certificate in your browser (see above). Your data is available at <code><nowiki>https://webdav.swegrid.se/snic/YOUR_PROJECT_NAME</nowiki></code>.<br />
; Upload and delete data<br />
: Use the ARC client. Please see the instructions for [[Accessing SweStore national storage with the ARC client]].<br />
: Use cURL. Please see the instructions for [[Accessing SweStore national storage with cURL]].<br />
: Use lftp. Please see the instructions for [[Accessing SweStore national storage with lftp]].<br />
: Use globus-url-copy. Please see the instructions for [[Accessing SweStore national storage with globus-url-copy]].<br />
<br />
== Examples of storage projects ==<br />
Below are some examples of project that are using SweStore today.<br />
<br />
{|border="1" style="text-align:left; border-collapse: collapse; border-width: 1px; border-style: solid; border-color: #000" class="wikitable sortable" valign=top<br />
!Allocation name<br />
!Size in TB<br />
!class="unsortable"|Project full name<br />
|-<br />
|alice<br />
|400<br />
|<br />
|-<br />
|uppnex<br />
|140<br />
|[https://www.uppnex.uu.se UPPmax NExt Generation Sequencing Cluster & Storage]<br />
|-<br />
|brain_protein_atlas<br />
|10<br />
|Mouse brain protein atlas project<br />
|-<br />
| scims2lab<br />
|20<br />
| Identification of novel gene models by matching mass spectrometry data against 6-frame translations of the human genome<br />
|-<br />
|subatom<br />
|<br />
|Low-energy nuclear theory and experiment<br />
|-<br />
|genomics-gu<br />
|10<br />
|Genomics Core Facility, Sahlgrenska academy at University of Gothenburg.<br />
|-<br />
|Chemo<br />
|5TB<br />
|Genetic interaction networks in human deseas<br />
|-<br />
|cesm1_holocene<br />
|30<br />
|Arctic sea ice in warm climates<br />
|}<br />
<br />
== More information ==<br />
* [[SweStore introduction]]<br />
* [http://status.swestore.se/munin/monitor/monitor/ Per Project Monitoring of Swestore usage]<br />
* [[Accessing SweStore national storage with the ARC client]]<br />
<!-- * [[Mounting SweStore national storage via WebDAV|Mounting SweStore national storage via WebDAV (Not recomendated at the moment)]] --><br />
If you have any issues using SweStore please do not hesitate to contact [mailto:swestore-support@snic.vr.se swestore-support].<br />
<br />
== Tools and scripts ==<br />
<br />
There exists a number of tools and utilities developed externally that can be useful. Here are some links:<br />
<br />
* [https://github.com/samuell/arc_tools ARC_Tools] - Convenience scripts for the arc client (Only a recursive rmdir so far).<br />
* [http://sourceforge.net/projects/arc-gui-clients ARC Graphical Clients] - Contains the ARC Storage Explorer (SweStore supported development).<br />
* Transfer script, [http://snicdocs.nsc.liu.se/wiki/SweStore/swstrans_arc swetrans_arc], provided by Adam Peplinski / Philipp Schlatter<br />
* [http://www.nordugrid.org/documents/SWIG-wrapped-ARC-Python-API.pdf Documentation of the ARC Python API (PDF)]<br />
<br />
= Center storage =<br />
Centre storage, as defined by the SNIC storage group, is a storage solution that lives independently of the computational resources and can be accessed from all such resources at a centre. Key features include the ability to access the same filesystem the same way on all computational resources at a centre, and a unified structure and nomenclature for all centra. Unlike cluster storage which is tightly associated with a single cluster, and thus has a limited life-time, centre storage does not require the users to migrate their own data when clusters are decommissioned, not even when the storage hardware itself is being replaced.<br />
<br />
== Unified environment ==<br />
To make the usage more transparent for SNIC users, a set of environment variables are available on all SNIC resources:<br />
<br />
* <code>SNIC_BACKUP</code> – the user's primary directory at the centre<br>(the part of the centre storage that is backed up)<br />
* <code>SNIC_NOBACKUP</code> – recommended directory for project storage without backup<br>(also on the centre storage)<br />
* <code>SNIC_TMP</code> – recommended directory for best performance during a job<br>(local disk on nodes if applicable)</div>Lars Viklund (HPC2N)