https://snicdocs.nsc.liu.se/w/api.php?action=feedcontributions&feedformat=atom&user=Jonas+Lindemann+%28LUNARC%29 2026-04-14T18:41:57Z User contributions MediaWiki 1.31.10 https://snicdocs.nsc.liu.se/w/index.php?title=SNIC_Emerging_Technologies&diff=6230 2016-04-04T12:41:40Z

<p>Jonas Lindemann (LUNARC): </p> <hr /> <div>{{project info<br /> |description=Coordinating new and emerging technologies within SNIC.<br /> |fields=Emerging Technologies<br /> |financing=SNIC<br /> |active=yes<br /> |start date=2014-05-01<br /> |end date=<br /> }}<br /> <br /> == Deliverables ==<br /> <br /> * '''[http://next-generation-hpc-desktop.readthedocs.org/en/latest/ Report: Next Generation HPC Desktop]'''<br /> <br /> == Introduction ==<br /> <br /> Currently, many SNIC centres are continuously investigate the market for current developments in emerging technologies and even buy hardware for evaluation. However, in many of these activities user involvement is limited and efforts are often duplicated between centres. This activity aims to coordinate these efforts and competence within SNIC and provide SNIC users and communities with early access to new technologies. The project will also work closely with the user communities to see if they have interests in upcoming technologies.<br /> <br /> Three focus areas have been and are described in the following sections:<br /> <br /> == Storage Technologies (C3SE, HPC2N) ==<br /> <br /> There are many storage technologies employed within SNIC. To be able to deploy solutions that are suitable for different usage scenarios, it is important that there is a continuous evaluation of availa-ble storage technologies. It is also important that prototype solutions are evaluated closely to users and facilities. Typical projects within this focus area could be:<br /> <br /> * File systems for different I/O patterns.<br /> * New storage hardware.<br /> * Higher-level file services.<br /> * Client tools for accessing available resources.<br /> * Integration services.<br /> <br /> '''Objectives and deliverables'''<br /> <br /> * Investigate how existing resources can be facilitated in new ways and how new approaches for users on how to use existing solutions can be facilitated to provide better support for I/O-intensive simulations and work-flows. (C3SE)<br /> * Investigate how the RobinHood service of Lustre v2+ filestystems can be used in conjunction with Tivoli TSM to speed up incremental backups. (C3SE, HPC2N)<br /> * Investigate how the RobinHood service of Lustre v2+ filesystems can be used to provide more fine-grained quotas, ex. for project storage etc. (C3SE)<br /> * One solution that has been used in SNIC for a number of years for long-term storage of re-search data is dCache in combination with TSM for archiving to tape. There are several al-ternatives to using TSM for archiving to tape, which should be further studied both from a performance and cost effectiveness. One example is LTFS [3]. To look at various options for long-term storage is also something that connects well to activities within WLCG and EIS-CAT_3D for which HPC2N are involved in. (HPC2N)<br /> <br /> '''Ongoing activities'''<br /> <br /> Users and groupquota through Littlejohn. Tested in operation. Scans file system and then uses changelogs to update database. Have since discovered a bug in the Lustre for changeloghantering. To be integrated for better project reporting<br /> <br /> Robin Hood and the TSM integration. Based on the changelog is active. Limited progress. Is an activity on the Lustre list.<br /> <br /> == Access-methods and remote visualisation (Lunarc, UPPMAX) ==<br /> <br /> To improve the usability of our HPC resources, many centres have been providing remote desktop solutions to users in an effort to provide a richer user experience in HPC. The service has been a great success in the research groups and the number of users is increasing. Requests for new applications and usage are increasing steadily. <br /> However, the increasing user base and request for new applications and usages has raised questions on how to efficiently scale these services in many dimensions.<br /> <br /> * Monitoring of usage patterns on the desktop servers.<br /> * Monitoring and scaling of distribution networks using different usage patterns and desktop configurations such as spatial resolution.<br /> * How to provide seamless access to hardware accelerated desktop services.<br /> * How to provide interactive access to applications and specific hardware configurations (e.g. GPU enabled nodes) through the batch-systems in a transparent way in the same desktop environment.<br /> <br /> Identifying bottlenecks in remote desktop architectures.<br /> <br /> '''Objectives and deliverables'''<br /> <br /> Remote desktop service aspects:<br /> <br /> * Evaluate different remote-access architectures based on target network specifications.<br /> * Develop best practice guides for configuration and setup of remote desktop services.<br /> * Evaluate techniques for monitoring the load and usage of the servers providing the desktop services, so that the user experience is good. Analyse usage bottlenecks and identify areas for improvement such as easy and (eventually) integrated SFTP file transfer.<br /> * Implement prototypes for providing a scalable and OS independent hardware accelerated back-end to the desktop interface. This can involve taking advantage of single or multiple GPU nodes as a provider for graphical acceleration.<br /> * Developing an on-demand hardware allocation mechanism using the queuing system to enable unique per-session access to unique hardware resources e.g. CPU or accelerator equipped nodes.<br /> * Defining community specific desktops.<br /> <br /> '''Ongoing activities'''<br /> <br /> Lunarc have been improving and testing configuration on the existing Lunarc HPC Desktop. Knowledge from this have been the basis for the design of the upcoming prototype hardware. Prototype hardware for evaluating future desktop environments have been procured from SouthPole. The hardware consists of:<br /> <br /> * NVIDIA K1/K2 evaluation system for investigating an on-demand desktop solution with accelerated graphics support. Solutions evaluated will be hypervisor-based using SLURM to allocate sessions.<br /> * Intel Xeon Clearwell system for evaluating graphics support in this architecture. Could be used for providing cost-effective desktop solutions for HPC resources.<br /> * Commodity graphics card in a server setting.<br /> <br /> Prototype system will be available mid february.<br /> <br /> == Future HPC and Accelerators (PDC, Lunarc, HPC2N) ==<br /> <br /> For future investment of computing resources in SNIC in the coming years (three large and several smaller systems), it is important that there is current specialist knowledge about different types of existing and future CPU architectures. To obtain a good basis for future decision in HPC resources, it is important that the work done at the centres, the SNIC GPU project and activities within PRACE are continued and developed further.<br /> <br /> '''Objectives and deliverables'''<br /> <br /> * Evaluate the use of next generation GPU:s K40 and others in the context of future SNIC resources. (Lunarc, HPC2N)<br /> * Operate and provide support for SNICs investments in the Zorn GPU development resource (PDC)<br /> * Operate and provide support for SNICs investments in the Erik GPU development resource (Lunarc)<br /> * Evaluate the use of next generation Xeon Phi accelerators (Knights Landing) (Lunarc, HPC2N)<br /> <br /> '''Ongoing activities'''<br /> <br /> Lunarc have procured a NVIDIA K80 system which will be installed in Erik. (Availability mid-feb 2015)<br /> <br /> HPC2N is installing Xeon Phi:s in the existing resources (Availability XX-XX-XX)<br /> <br /> == Members ==<br /> {{#ask: [[Category:Person]] [[project::{{PAGENAME}}]]<br /> |?centre<br /> |?role<br /> |?field<br /> }}</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=SNIC_Emerging_Technologies&diff=6229 2016-04-04T12:41:13Z

<p>Jonas Lindemann (LUNARC): </p> <hr /> <div>{{project info<br /> |description=Coordinating new and emerging technologies within SNIC.<br /> |fields=Emerging Technologies<br /> |financing=SNIC<br /> |active=yes<br /> |start date=2014-05-01<br /> |end date=<br /> }}<br /> <br /> == Deliverables ==<br /> <br /> * '''[http://next-generation-hpc-desktop.readthedocs.org/en/latest/ Report: Next Generation HPC Desktop'''<br /> <br /> == Introduction ==<br /> <br /> Currently, many SNIC centres are continuously investigate the market for current developments in emerging technologies and even buy hardware for evaluation. However, in many of these activities user involvement is limited and efforts are often duplicated between centres. This activity aims to coordinate these efforts and competence within SNIC and provide SNIC users and communities with early access to new technologies. The project will also work closely with the user communities to see if they have interests in upcoming technologies.<br /> <br /> Three focus areas have been and are described in the following sections:<br /> <br /> == Storage Technologies (C3SE, HPC2N) ==<br /> <br /> There are many storage technologies employed within SNIC. To be able to deploy solutions that are suitable for different usage scenarios, it is important that there is a continuous evaluation of availa-ble storage technologies. It is also important that prototype solutions are evaluated closely to users and facilities. Typical projects within this focus area could be:<br /> <br /> * File systems for different I/O patterns.<br /> * New storage hardware.<br /> * Higher-level file services.<br /> * Client tools for accessing available resources.<br /> * Integration services.<br /> <br /> '''Objectives and deliverables'''<br /> <br /> * Investigate how existing resources can be facilitated in new ways and how new approaches for users on how to use existing solutions can be facilitated to provide better support for I/O-intensive simulations and work-flows. (C3SE)<br /> * Investigate how the RobinHood service of Lustre v2+ filestystems can be used in conjunction with Tivoli TSM to speed up incremental backups. (C3SE, HPC2N)<br /> * Investigate how the RobinHood service of Lustre v2+ filesystems can be used to provide more fine-grained quotas, ex. for project storage etc. (C3SE)<br /> * One solution that has been used in SNIC for a number of years for long-term storage of re-search data is dCache in combination with TSM for archiving to tape. There are several al-ternatives to using TSM for archiving to tape, which should be further studied both from a performance and cost effectiveness. One example is LTFS [3]. To look at various options for long-term storage is also something that connects well to activities within WLCG and EIS-CAT_3D for which HPC2N are involved in. (HPC2N)<br /> <br /> '''Ongoing activities'''<br /> <br /> Users and groupquota through Littlejohn. Tested in operation. Scans file system and then uses changelogs to update database. Have since discovered a bug in the Lustre for changeloghantering. To be integrated for better project reporting<br /> <br /> Robin Hood and the TSM integration. Based on the changelog is active. Limited progress. Is an activity on the Lustre list.<br /> <br /> == Access-methods and remote visualisation (Lunarc, UPPMAX) ==<br /> <br /> To improve the usability of our HPC resources, many centres have been providing remote desktop solutions to users in an effort to provide a richer user experience in HPC. The service has been a great success in the research groups and the number of users is increasing. Requests for new applications and usage are increasing steadily. <br /> However, the increasing user base and request for new applications and usages has raised questions on how to efficiently scale these services in many dimensions.<br /> <br /> * Monitoring of usage patterns on the desktop servers.<br /> * Monitoring and scaling of distribution networks using different usage patterns and desktop configurations such as spatial resolution.<br /> * How to provide seamless access to hardware accelerated desktop services.<br /> * How to provide interactive access to applications and specific hardware configurations (e.g. GPU enabled nodes) through the batch-systems in a transparent way in the same desktop environment.<br /> <br /> Identifying bottlenecks in remote desktop architectures.<br /> <br /> '''Objectives and deliverables'''<br /> <br /> Remote desktop service aspects:<br /> <br /> * Evaluate different remote-access architectures based on target network specifications.<br /> * Develop best practice guides for configuration and setup of remote desktop services.<br /> * Evaluate techniques for monitoring the load and usage of the servers providing the desktop services, so that the user experience is good. Analyse usage bottlenecks and identify areas for improvement such as easy and (eventually) integrated SFTP file transfer.<br /> * Implement prototypes for providing a scalable and OS independent hardware accelerated back-end to the desktop interface. This can involve taking advantage of single or multiple GPU nodes as a provider for graphical acceleration.<br /> * Developing an on-demand hardware allocation mechanism using the queuing system to enable unique per-session access to unique hardware resources e.g. CPU or accelerator equipped nodes.<br /> * Defining community specific desktops.<br /> <br /> '''Ongoing activities'''<br /> <br /> Lunarc have been improving and testing configuration on the existing Lunarc HPC Desktop. Knowledge from this have been the basis for the design of the upcoming prototype hardware. Prototype hardware for evaluating future desktop environments have been procured from SouthPole. The hardware consists of:<br /> <br /> * NVIDIA K1/K2 evaluation system for investigating an on-demand desktop solution with accelerated graphics support. Solutions evaluated will be hypervisor-based using SLURM to allocate sessions.<br /> * Intel Xeon Clearwell system for evaluating graphics support in this architecture. Could be used for providing cost-effective desktop solutions for HPC resources.<br /> * Commodity graphics card in a server setting.<br /> <br /> Prototype system will be available mid february.<br /> <br /> == Future HPC and Accelerators (PDC, Lunarc, HPC2N) ==<br /> <br /> For future investment of computing resources in SNIC in the coming years (three large and several smaller systems), it is important that there is current specialist knowledge about different types of existing and future CPU architectures. To obtain a good basis for future decision in HPC resources, it is important that the work done at the centres, the SNIC GPU project and activities within PRACE are continued and developed further.<br /> <br /> '''Objectives and deliverables'''<br /> <br /> * Evaluate the use of next generation GPU:s K40 and others in the context of future SNIC resources. (Lunarc, HPC2N)<br /> * Operate and provide support for SNICs investments in the Zorn GPU development resource (PDC)<br /> * Operate and provide support for SNICs investments in the Erik GPU development resource (Lunarc)<br /> * Evaluate the use of next generation Xeon Phi accelerators (Knights Landing) (Lunarc, HPC2N)<br /> <br /> '''Ongoing activities'''<br /> <br /> Lunarc have procured a NVIDIA K80 system which will be installed in Erik. (Availability mid-feb 2015)<br /> <br /> HPC2N is installing Xeon Phi:s in the existing resources (Availability XX-XX-XX)<br /> <br /> == Members ==<br /> {{#ask: [[Category:Person]] [[project::{{PAGENAME}}]]<br /> |?centre<br /> |?role<br /> |?field<br /> }}</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=SNIC_Emerging_Technologies&diff=6228 2016-04-04T12:38:19Z

<p>Jonas Lindemann (LUNARC): </p> <hr /> <div>{{project info<br /> |description=Coordinating new and emerging technologies within SNIC.<br /> |fields=Emerging Technologies<br /> |financing=SNIC<br /> |active=yes<br /> |start date=2014-05-01<br /> |end date=<br /> }}<br /> <br /> Currently, many SNIC centres are continuously investigate the market for current developments in emerging technologies and even buy hardware for evaluation. However, in many of these activities user involvement is limited and efforts are often duplicated between centres. This activity aims to coordinate these efforts and competence within SNIC and provide SNIC users and communities with early access to new technologies. The project will also work closely with the user communities to see if they have interests in upcoming technologies.<br /> <br /> Three focus areas have been and are described in the following sections:<br /> <br /> == Storage Technologies (C3SE, HPC2N) ==<br /> <br /> There are many storage technologies employed within SNIC. To be able to deploy solutions that are suitable for different usage scenarios, it is important that there is a continuous evaluation of availa-ble storage technologies. It is also important that prototype solutions are evaluated closely to users and facilities. Typical projects within this focus area could be:<br /> <br /> * File systems for different I/O patterns.<br /> * New storage hardware.<br /> * Higher-level file services.<br /> * Client tools for accessing available resources.<br /> * Integration services.<br /> <br /> '''Objectives and deliverables'''<br /> <br /> * Investigate how existing resources can be facilitated in new ways and how new approaches for users on how to use existing solutions can be facilitated to provide better support for I/O-intensive simulations and work-flows. (C3SE)<br /> * Investigate how the RobinHood service of Lustre v2+ filestystems can be used in conjunction with Tivoli TSM to speed up incremental backups. (C3SE, HPC2N)<br /> * Investigate how the RobinHood service of Lustre v2+ filesystems can be used to provide more fine-grained quotas, ex. for project storage etc. (C3SE)<br /> * One solution that has been used in SNIC for a number of years for long-term storage of re-search data is dCache in combination with TSM for archiving to tape. There are several al-ternatives to using TSM for archiving to tape, which should be further studied both from a performance and cost effectiveness. One example is LTFS [3]. To look at various options for long-term storage is also something that connects well to activities within WLCG and EIS-CAT_3D for which HPC2N are involved in. (HPC2N)<br /> <br /> '''Ongoing activities'''<br /> <br /> Users and groupquota through Littlejohn. Tested in operation. Scans file system and then uses changelogs to update database. Have since discovered a bug in the Lustre for changeloghantering. To be integrated for better project reporting<br /> <br /> Robin Hood and the TSM integration. Based on the changelog is active. Limited progress. Is an activity on the Lustre list.<br /> <br /> == Access-methods and remote visualisation (Lunarc, UPPMAX) ==<br /> <br /> **Deliverable (Report): [http://next-generation-hpc-desktop.readthedocs.org/en/latest/ Next Generation HPC Desktop]** <br /> <br /> To improve the usability of our HPC resources, many centres have been providing remote desktop solutions to users in an effort to provide a richer user experience in HPC. The service has been a great success in the research groups and the number of users is increasing. Requests for new applications and usage are increasing steadily. <br /> However, the increasing user base and request for new applications and usages has raised questions on how to efficiently scale these services in many dimensions.<br /> <br /> * Monitoring of usage patterns on the desktop servers.<br /> * Monitoring and scaling of distribution networks using different usage patterns and desktop configurations such as spatial resolution.<br /> * How to provide seamless access to hardware accelerated desktop services.<br /> * How to provide interactive access to applications and specific hardware configurations (e.g. GPU enabled nodes) through the batch-systems in a transparent way in the same desktop environment.<br /> <br /> Identifying bottlenecks in remote desktop architectures.<br /> <br /> '''Objectives and deliverables'''<br /> <br /> Remote desktop service aspects:<br /> <br /> * Evaluate different remote-access architectures based on target network specifications.<br /> * Develop best practice guides for configuration and setup of remote desktop services.<br /> * Evaluate techniques for monitoring the load and usage of the servers providing the desktop services, so that the user experience is good. Analyse usage bottlenecks and identify areas for improvement such as easy and (eventually) integrated SFTP file transfer.<br /> * Implement prototypes for providing a scalable and OS independent hardware accelerated back-end to the desktop interface. This can involve taking advantage of single or multiple GPU nodes as a provider for graphical acceleration.<br /> * Developing an on-demand hardware allocation mechanism using the queuing system to enable unique per-session access to unique hardware resources e.g. CPU or accelerator equipped nodes.<br /> * Defining community specific desktops.<br /> <br /> '''Ongoing activities'''<br /> <br /> Lunarc have been improving and testing configuration on the existing Lunarc HPC Desktop. Knowledge from this have been the basis for the design of the upcoming prototype hardware. Prototype hardware for evaluating future desktop environments have been procured from SouthPole. The hardware consists of:<br /> <br /> * NVIDIA K1/K2 evaluation system for investigating an on-demand desktop solution with accelerated graphics support. Solutions evaluated will be hypervisor-based using SLURM to allocate sessions.<br /> * Intel Xeon Clearwell system for evaluating graphics support in this architecture. Could be used for providing cost-effective desktop solutions for HPC resources.<br /> * Commodity graphics card in a server setting.<br /> <br /> Prototype system will be available mid february.<br /> <br /> == Future HPC and Accelerators (PDC, Lunarc, HPC2N) ==<br /> <br /> For future investment of computing resources in SNIC in the coming years (three large and several smaller systems), it is important that there is current specialist knowledge about different types of existing and future CPU architectures. To obtain a good basis for future decision in HPC resources, it is important that the work done at the centres, the SNIC GPU project and activities within PRACE are continued and developed further.<br /> <br /> '''Objectives and deliverables'''<br /> <br /> * Evaluate the use of next generation GPU:s K40 and others in the context of future SNIC resources. (Lunarc, HPC2N)<br /> * Operate and provide support for SNICs investments in the Zorn GPU development resource (PDC)<br /> * Operate and provide support for SNICs investments in the Erik GPU development resource (Lunarc)<br /> * Evaluate the use of next generation Xeon Phi accelerators (Knights Landing) (Lunarc, HPC2N)<br /> <br /> '''Ongoing activities'''<br /> <br /> Lunarc have procured a NVIDIA K80 system which will be installed in Erik. (Availability mid-feb 2015)<br /> <br /> HPC2N is installing Xeon Phi:s in the existing resources (Availability XX-XX-XX)<br /> <br /> == Members ==<br /> {{#ask: [[Category:Person]] [[project::{{PAGENAME}}]]<br /> |?centre<br /> |?role<br /> |?field<br /> }}</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=SNIC_Emerging_Technologies&diff=6037 2015-02-04T13:14:25Z

<p>Jonas Lindemann (LUNARC): /* Future HPC and Accelerators (PDC, Lunarc, HPC2N) */</p> <hr /> <div>{{project info<br /> |description=Coordinating new and emerging technologies within SNIC.<br /> |fields=Emerging Technologies<br /> |financing=SNIC<br /> |active=yes<br /> |start date=2014-05-01<br /> |end date=<br /> }}<br /> <br /> Currently, many SNIC centres are continuously investigate the market for current developments in emerging technologies and even buy hardware for evaluation. However, in many of these activities user involvement is limited and efforts are often duplicated between centres. This activity aims to coordinate these efforts and competence within SNIC and provide SNIC users and communities with early access to new technologies. The project will also work closely with the user communities to see if they have interests in upcoming technologies.<br /> <br /> Three focus areas have been and are described in the following sections:<br /> <br /> == Storage Technologies (C3SE, HPC2N) ==<br /> <br /> There are many storage technologies employed within SNIC. To be able to deploy solutions that are suitable for different usage scenarios, it is important that there is a continuous evaluation of availa-ble storage technologies. It is also important that prototype solutions are evaluated closely to users and facilities. Typical projects within this focus area could be:<br /> <br /> * File systems for different I/O patterns.<br /> * New storage hardware.<br /> * Higher-level file services.<br /> * Client tools for accessing available resources.<br /> * Integration services.<br /> <br /> '''Objectives and deliverables'''<br /> <br /> * Investigate how existing resources can be facilitated in new ways and how new approaches for users on how to use existing solutions can be facilitated to provide better support for I/O-intensive simulations and work-flows. (C3SE)<br /> * Investigate how the RobinHood service of Lustre v2+ filestystems can be used in conjunction with Tivoli TSM to speed up incremental backups. (C3SE, HPC2N)<br /> * Investigate how the RobinHood service of Lustre v2+ filesystems can be used to provide more fine-grained quotas, ex. for project storage etc. (C3SE)<br /> * One solution that has been used in SNIC for a number of years for long-term storage of re-search data is dCache in combination with TSM for archiving to tape. There are several al-ternatives to using TSM for archiving to tape, which should be further studied both from a performance and cost effectiveness. One example is LTFS [3]. To look at various options for long-term storage is also something that connects well to activities within WLCG and EIS-CAT_3D for which HPC2N are involved in. (HPC2N)<br /> <br /> '''Ongoing activities'''<br /> <br /> Users and groupquota through Littlejohn. Tested in operation. Scans file system and then uses changelogs to update database. Have since discovered a bug in the Lustre for changeloghantering. To be integrated for better project reporting<br /> <br /> Robin Hood and the TSM integration. Based on the changelog is active. Limited progress. Is an activity on the Lustre list.<br /> <br /> == Access-methods and remote visualisation (Lunarc, UPPMAX) ==<br /> <br /> To improve the usability of our HPC resources, many centres have been providing remote desktop solutions to users in an effort to provide a richer user experience in HPC. The service has been a great success in the research groups and the number of users is increasing. Requests for new applications and usage are increasing steadily. <br /> However, the increasing user base and request for new applications and usages has raised questions on how to efficiently scale these services in many dimensions.<br /> <br /> * Monitoring of usage patterns on the desktop servers.<br /> * Monitoring and scaling of distribution networks using different usage patterns and desktop configurations such as spatial resolution.<br /> * How to provide seamless access to hardware accelerated desktop services.<br /> * How to provide interactive access to applications and specific hardware configurations (e.g. GPU enabled nodes) through the batch-systems in a transparent way in the same desktop environment.<br /> <br /> Identifying bottlenecks in remote desktop architectures.<br /> <br /> '''Objectives and deliverables'''<br /> <br /> Remote desktop service aspects:<br /> <br /> * Evaluate different remote-access architectures based on target network specifications.<br /> * Develop best practice guides for configuration and setup of remote desktop services.<br /> * Evaluate techniques for monitoring the load and usage of the servers providing the desktop services, so that the user experience is good. Analyse usage bottlenecks and identify areas for improvement such as easy and (eventually) integrated SFTP file transfer.<br /> * Implement prototypes for providing a scalable and OS independent hardware accelerated back-end to the desktop interface. This can involve taking advantage of single or multiple GPU nodes as a provider for graphical acceleration.<br /> * Developing an on-demand hardware allocation mechanism using the queuing system to enable unique per-session access to unique hardware resources e.g. CPU or accelerator equipped nodes.<br /> * Defining community specific desktops.<br /> <br /> '''Ongoing activities'''<br /> <br /> Lunarc have been improving and testing configuration on the existing Lunarc HPC Desktop. Knowledge from this have been the basis for the design of the upcoming prototype hardware. Prototype hardware for evaluating future desktop environments have been procured from SouthPole. The hardware consists of:<br /> <br /> * NVIDIA K1/K2 evaluation system for investigating an on-demand desktop solution with accelerated graphics support. Solutions evaluated will be hypervisor-based using SLURM to allocate sessions.<br /> * Intel Xeon Clearwell system for evaluating graphics support in this architecture. Could be used for providing cost-effective desktop solutions for HPC resources.<br /> * Commodity graphics card in a server setting.<br /> <br /> Prototype system will be available mid february.<br /> <br /> == Future HPC and Accelerators (PDC, Lunarc, HPC2N) ==<br /> <br /> For future investment of computing resources in SNIC in the coming years (three large and several smaller systems), it is important that there is current specialist knowledge about different types of existing and future CPU architectures. To obtain a good basis for future decision in HPC resources, it is important that the work done at the centres, the SNIC GPU project and activities within PRACE are continued and developed further.<br /> <br /> '''Objectives and deliverables'''<br /> <br /> * Evaluate the use of next generation GPU:s K40 and others in the context of future SNIC resources. (Lunarc, HPC2N)<br /> * Operate and provide support for SNICs investments in the Zorn GPU development resource (PDC)<br /> * Operate and provide support for SNICs investments in the Erik GPU development resource (Lunarc)<br /> * Evaluate the use of next generation Xeon Phi accelerators (Knights Landing) (Lunarc, HPC2N)<br /> <br /> '''Ongoing activities'''<br /> <br /> Lunarc have procured a NVIDIA K80 system which will be installed in Erik. (Availability mid-feb 2015)<br /> <br /> HPC2N is installing Xeon Phi:s in the existing resources (Availability XX-XX-XX)<br /> <br /> == Members ==<br /> {{#ask: [[Category:Person]] [[project::{{PAGENAME}}]]<br /> |?centre<br /> |?role<br /> |?field<br /> }}</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=SNIC_Emerging_Technologies&diff=6036 2015-02-04T09:35:42Z

<p>Jonas Lindemann (LUNARC): /* Future HPC and Accelerators (PDC, Lunarc, HPC2N) */</p> <hr /> <div>{{project info<br /> |description=Coordinating new and emerging technologies within SNIC.<br /> |fields=Emerging Technologies<br /> |financing=SNIC<br /> |active=yes<br /> |start date=2014-05-01<br /> |end date=<br /> }}<br /> <br /> Currently, many SNIC centres are continuously investigate the market for current developments in emerging technologies and even buy hardware for evaluation. However, in many of these activities user involvement is limited and efforts are often duplicated between centres. This activity aims to coordinate these efforts and competence within SNIC and provide SNIC users and communities with early access to new technologies. The project will also work closely with the user communities to see if they have interests in upcoming technologies.<br /> <br /> Three focus areas have been and are described in the following sections:<br /> <br /> == Storage Technologies (C3SE, HPC2N) ==<br /> <br /> There are many storage technologies employed within SNIC. To be able to deploy solutions that are suitable for different usage scenarios, it is important that there is a continuous evaluation of availa-ble storage technologies. It is also important that prototype solutions are evaluated closely to users and facilities. Typical projects within this focus area could be:<br /> <br /> * File systems for different I/O patterns.<br /> * New storage hardware.<br /> * Higher-level file services.<br /> * Client tools for accessing available resources.<br /> * Integration services.<br /> <br /> '''Objectives and deliverables'''<br /> <br /> * Investigate how existing resources can be facilitated in new ways and how new approaches for users on how to use existing solutions can be facilitated to provide better support for I/O-intensive simulations and work-flows. (C3SE)<br /> * Investigate how the RobinHood service of Lustre v2+ filestystems can be used in conjunction with Tivoli TSM to speed up incremental backups. (C3SE, HPC2N)<br /> * Investigate how the RobinHood service of Lustre v2+ filesystems can be used to provide more fine-grained quotas, ex. for project storage etc. (C3SE)<br /> * One solution that has been used in SNIC for a number of years for long-term storage of re-search data is dCache in combination with TSM for archiving to tape. There are several al-ternatives to using TSM for archiving to tape, which should be further studied both from a performance and cost effectiveness. One example is LTFS [3]. To look at various options for long-term storage is also something that connects well to activities within WLCG and EIS-CAT_3D for which HPC2N are involved in. (HPC2N)<br /> <br /> '''Ongoing activities'''<br /> <br /> Users and groupquota through Littlejohn. Tested in operation. Scans file system and then uses changelogs to update database. Have since discovered a bug in the Lustre for changeloghantering. To be integrated for better project reporting<br /> <br /> Robin Hood and the TSM integration. Based on the changelog is active. Limited progress. Is an activity on the Lustre list.<br /> <br /> == Access-methods and remote visualisation (Lunarc, UPPMAX) ==<br /> <br /> To improve the usability of our HPC resources, many centres have been providing remote desktop solutions to users in an effort to provide a richer user experience in HPC. The service has been a great success in the research groups and the number of users is increasing. Requests for new applications and usage are increasing steadily. <br /> However, the increasing user base and request for new applications and usages has raised questions on how to efficiently scale these services in many dimensions.<br /> <br /> * Monitoring of usage patterns on the desktop servers.<br /> * Monitoring and scaling of distribution networks using different usage patterns and desktop configurations such as spatial resolution.<br /> * How to provide seamless access to hardware accelerated desktop services.<br /> * How to provide interactive access to applications and specific hardware configurations (e.g. GPU enabled nodes) through the batch-systems in a transparent way in the same desktop environment.<br /> <br /> Identifying bottlenecks in remote desktop architectures.<br /> <br /> '''Objectives and deliverables'''<br /> <br /> Remote desktop service aspects:<br /> <br /> * Evaluate different remote-access architectures based on target network specifications.<br /> * Develop best practice guides for configuration and setup of remote desktop services.<br /> * Evaluate techniques for monitoring the load and usage of the servers providing the desktop services, so that the user experience is good. Analyse usage bottlenecks and identify areas for improvement such as easy and (eventually) integrated SFTP file transfer.<br /> * Implement prototypes for providing a scalable and OS independent hardware accelerated back-end to the desktop interface. This can involve taking advantage of single or multiple GPU nodes as a provider for graphical acceleration.<br /> * Developing an on-demand hardware allocation mechanism using the queuing system to enable unique per-session access to unique hardware resources e.g. CPU or accelerator equipped nodes.<br /> * Defining community specific desktops.<br /> <br /> '''Ongoing activities'''<br /> <br /> Lunarc have been improving and testing configuration on the existing Lunarc HPC Desktop. Knowledge from this have been the basis for the design of the upcoming prototype hardware. Prototype hardware for evaluating future desktop environments have been procured from SouthPole. The hardware consists of:<br /> <br /> * NVIDIA K1/K2 evaluation system for investigating an on-demand desktop solution with accelerated graphics support. Solutions evaluated will be hypervisor-based using SLURM to allocate sessions.<br /> * Intel Xeon Clearwell system for evaluating graphics support in this architecture. Could be used for providing cost-effective desktop solutions for HPC resources.<br /> * Commodity graphics card in a server setting.<br /> <br /> Prototype system will be available mid february.<br /> <br /> == Future HPC and Accelerators (PDC, Lunarc, HPC2N) ==<br /> <br /> For future investment of computing resources in SNIC in the coming years (three large and several smaller systems), it is important that there is current specialist knowledge about different types of existing and future CPU architectures. To obtain a good basis for future decision in HPC resources, it is important that the work done at the centres, the SNIC GPU project and activities within PRACE are continued and developed further.<br /> <br /> '''Objectives and deliverables'''<br /> <br /> * Evaluate the use of next generation GPU:s K40 and others in the context of future SNIC resources. (Lunarc, HPC2N)<br /> * Operate and provide support for SNICs investments in the Zorn GPU development resource (PDC)<br /> * Operate and provide support for SNICs investments in the Erik GPU development resource (Lunarc)<br /> * Evaluate the use of next generation Xeon Phi accelerators (Knights Landing) (Lunarc, HPC2N)<br /> <br /> '''Ongoing activities'''<br /> <br /> Lunarc have procured a NVIDIA K80 system which will be installed in Erik. (Availability mid-feb 2015)<br /> HPC2N is installing Xeon Phi:s in the existing resources (Availability XX-XX-XX)<br /> <br /> == Members ==<br /> {{#ask: [[Category:Person]] [[project::{{PAGENAME}}]]<br /> |?centre<br /> |?role<br /> |?field<br /> }}</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=SNIC_Emerging_Technologies&diff=6035 2015-02-04T09:32:13Z

<p>Jonas Lindemann (LUNARC): /* Future HPC and Accelerators (PDC, Lunarc, HPC2N) */</p> <hr /> <div>{{project info<br /> |description=Coordinating new and emerging technologies within SNIC.<br /> |fields=Emerging Technologies<br /> |financing=SNIC<br /> |active=yes<br /> |start date=2014-05-01<br /> |end date=<br /> }}<br /> <br /> Currently, many SNIC centres are continuously investigate the market for current developments in emerging technologies and even buy hardware for evaluation. However, in many of these activities user involvement is limited and efforts are often duplicated between centres. This activity aims to coordinate these efforts and competence within SNIC and provide SNIC users and communities with early access to new technologies. The project will also work closely with the user communities to see if they have interests in upcoming technologies.<br /> <br /> Three focus areas have been and are described in the following sections:<br /> <br /> == Storage Technologies (C3SE, HPC2N) ==<br /> <br /> There are many storage technologies employed within SNIC. To be able to deploy solutions that are suitable for different usage scenarios, it is important that there is a continuous evaluation of availa-ble storage technologies. It is also important that prototype solutions are evaluated closely to users and facilities. Typical projects within this focus area could be:<br /> <br /> * File systems for different I/O patterns.<br /> * New storage hardware.<br /> * Higher-level file services.<br /> * Client tools for accessing available resources.<br /> * Integration services.<br /> <br /> '''Objectives and deliverables'''<br /> <br /> * Investigate how existing resources can be facilitated in new ways and how new approaches for users on how to use existing solutions can be facilitated to provide better support for I/O-intensive simulations and work-flows. (C3SE)<br /> * Investigate how the RobinHood service of Lustre v2+ filestystems can be used in conjunction with Tivoli TSM to speed up incremental backups. (C3SE, HPC2N)<br /> * Investigate how the RobinHood service of Lustre v2+ filesystems can be used to provide more fine-grained quotas, ex. for project storage etc. (C3SE)<br /> * One solution that has been used in SNIC for a number of years for long-term storage of re-search data is dCache in combination with TSM for archiving to tape. There are several al-ternatives to using TSM for archiving to tape, which should be further studied both from a performance and cost effectiveness. One example is LTFS [3]. To look at various options for long-term storage is also something that connects well to activities within WLCG and EIS-CAT_3D for which HPC2N are involved in. (HPC2N)<br /> <br /> '''Ongoing activities'''<br /> <br /> Users and groupquota through Littlejohn. Tested in operation. Scans file system and then uses changelogs to update database. Have since discovered a bug in the Lustre for changeloghantering. To be integrated for better project reporting<br /> <br /> Robin Hood and the TSM integration. Based on the changelog is active. Limited progress. Is an activity on the Lustre list.<br /> <br /> == Access-methods and remote visualisation (Lunarc, UPPMAX) ==<br /> <br /> To improve the usability of our HPC resources, many centres have been providing remote desktop solutions to users in an effort to provide a richer user experience in HPC. The service has been a great success in the research groups and the number of users is increasing. Requests for new applications and usage are increasing steadily. <br /> However, the increasing user base and request for new applications and usages has raised questions on how to efficiently scale these services in many dimensions.<br /> <br /> * Monitoring of usage patterns on the desktop servers.<br /> * Monitoring and scaling of distribution networks using different usage patterns and desktop configurations such as spatial resolution.<br /> * How to provide seamless access to hardware accelerated desktop services.<br /> * How to provide interactive access to applications and specific hardware configurations (e.g. GPU enabled nodes) through the batch-systems in a transparent way in the same desktop environment.<br /> <br /> Identifying bottlenecks in remote desktop architectures.<br /> <br /> '''Objectives and deliverables'''<br /> <br /> Remote desktop service aspects:<br /> <br /> * Evaluate different remote-access architectures based on target network specifications.<br /> * Develop best practice guides for configuration and setup of remote desktop services.<br /> * Evaluate techniques for monitoring the load and usage of the servers providing the desktop services, so that the user experience is good. Analyse usage bottlenecks and identify areas for improvement such as easy and (eventually) integrated SFTP file transfer.<br /> * Implement prototypes for providing a scalable and OS independent hardware accelerated back-end to the desktop interface. This can involve taking advantage of single or multiple GPU nodes as a provider for graphical acceleration.<br /> * Developing an on-demand hardware allocation mechanism using the queuing system to enable unique per-session access to unique hardware resources e.g. CPU or accelerator equipped nodes.<br /> * Defining community specific desktops.<br /> <br /> '''Ongoing activities'''<br /> <br /> Lunarc have been improving and testing configuration on the existing Lunarc HPC Desktop. Knowledge from this have been the basis for the design of the upcoming prototype hardware. Prototype hardware for evaluating future desktop environments have been procured from SouthPole. The hardware consists of:<br /> <br /> * NVIDIA K1/K2 evaluation system for investigating an on-demand desktop solution with accelerated graphics support. Solutions evaluated will be hypervisor-based using SLURM to allocate sessions.<br /> * Intel Xeon Clearwell system for evaluating graphics support in this architecture. Could be used for providing cost-effective desktop solutions for HPC resources.<br /> * Commodity graphics card in a server setting.<br /> <br /> Prototype system will be available mid february.<br /> <br /> == Future HPC and Accelerators (PDC, Lunarc, HPC2N) ==<br /> <br /> For future investment of computing resources in SNIC in the coming years (three large and several smaller systems), it is important that there is current specialist knowledge about different types of existing and future CPU architectures. To obtain a good basis for future decision in HPC resources, it is important that the work done at the centres, the SNIC GPU project and activities within PRACE are continued and developed further.<br /> <br /> '''Objectives and deliverables'''<br /> <br /> * Evaluate the use of next generation GPU:s K40 and others in the context of future SNIC resources. (Lunarc, HPC2N)<br /> * Operate and provide support for SNICs investments in the Zorn GPU development resource (PDC)<br /> * Operate and provide support for SNICs investments in the Erik GPU development resource (Lunarc)<br /> * Evaluate the use of next generation Xeon Phi accelerators (Knights Landing) (Lunarc, HPC2N)<br /> <br /> '''Ongoing activities'''<br /> <br /> Lunarc have procured a NVIDIA K80 which will be installed in Erik. (Availability mid-feb 2015)<br /> HPC2N is installing Xeon Phi:s in the existing resources (Availability XX-XX-XX)<br /> <br /> == Members ==<br /> {{#ask: [[Category:Person]] [[project::{{PAGENAME}}]]<br /> |?centre<br /> |?role<br /> |?field<br /> }}</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=SNIC_Emerging_Technologies&diff=6034 2015-02-04T09:27:47Z

<p>Jonas Lindemann (LUNARC): </p> <hr /> <div>{{project info<br /> |description=Coordinating new and emerging technologies within SNIC.<br /> |fields=Emerging Technologies<br /> |financing=SNIC<br /> |active=yes<br /> |start date=2014-05-01<br /> |end date=<br /> }}<br /> <br /> Currently, many SNIC centres are continuously investigate the market for current developments in emerging technologies and even buy hardware for evaluation. However, in many of these activities user involvement is limited and efforts are often duplicated between centres. This activity aims to coordinate these efforts and competence within SNIC and provide SNIC users and communities with early access to new technologies. The project will also work closely with the user communities to see if they have interests in upcoming technologies.<br /> <br /> Three focus areas have been and are described in the following sections:<br /> <br /> == Storage Technologies (C3SE, HPC2N) ==<br /> <br /> There are many storage technologies employed within SNIC. To be able to deploy solutions that are suitable for different usage scenarios, it is important that there is a continuous evaluation of availa-ble storage technologies. It is also important that prototype solutions are evaluated closely to users and facilities. Typical projects within this focus area could be:<br /> <br /> * File systems for different I/O patterns.<br /> * New storage hardware.<br /> * Higher-level file services.<br /> * Client tools for accessing available resources.<br /> * Integration services.<br /> <br /> '''Objectives and deliverables'''<br /> <br /> * Investigate how existing resources can be facilitated in new ways and how new approaches for users on how to use existing solutions can be facilitated to provide better support for I/O-intensive simulations and work-flows. (C3SE)<br /> * Investigate how the RobinHood service of Lustre v2+ filestystems can be used in conjunction with Tivoli TSM to speed up incremental backups. (C3SE, HPC2N)<br /> * Investigate how the RobinHood service of Lustre v2+ filesystems can be used to provide more fine-grained quotas, ex. for project storage etc. (C3SE)<br /> * One solution that has been used in SNIC for a number of years for long-term storage of re-search data is dCache in combination with TSM for archiving to tape. There are several al-ternatives to using TSM for archiving to tape, which should be further studied both from a performance and cost effectiveness. One example is LTFS [3]. To look at various options for long-term storage is also something that connects well to activities within WLCG and EIS-CAT_3D for which HPC2N are involved in. (HPC2N)<br /> <br /> '''Ongoing activities'''<br /> <br /> Users and groupquota through Littlejohn. Tested in operation. Scans file system and then uses changelogs to update database. Have since discovered a bug in the Lustre for changeloghantering. To be integrated for better project reporting<br /> <br /> Robin Hood and the TSM integration. Based on the changelog is active. Limited progress. Is an activity on the Lustre list.<br /> <br /> == Access-methods and remote visualisation (Lunarc, UPPMAX) ==<br /> <br /> To improve the usability of our HPC resources, many centres have been providing remote desktop solutions to users in an effort to provide a richer user experience in HPC. The service has been a great success in the research groups and the number of users is increasing. Requests for new applications and usage are increasing steadily. <br /> However, the increasing user base and request for new applications and usages has raised questions on how to efficiently scale these services in many dimensions.<br /> <br /> * Monitoring of usage patterns on the desktop servers.<br /> * Monitoring and scaling of distribution networks using different usage patterns and desktop configurations such as spatial resolution.<br /> * How to provide seamless access to hardware accelerated desktop services.<br /> * How to provide interactive access to applications and specific hardware configurations (e.g. GPU enabled nodes) through the batch-systems in a transparent way in the same desktop environment.<br /> <br /> Identifying bottlenecks in remote desktop architectures.<br /> <br /> '''Objectives and deliverables'''<br /> <br /> Remote desktop service aspects:<br /> <br /> * Evaluate different remote-access architectures based on target network specifications.<br /> * Develop best practice guides for configuration and setup of remote desktop services.<br /> * Evaluate techniques for monitoring the load and usage of the servers providing the desktop services, so that the user experience is good. Analyse usage bottlenecks and identify areas for improvement such as easy and (eventually) integrated SFTP file transfer.<br /> * Implement prototypes for providing a scalable and OS independent hardware accelerated back-end to the desktop interface. This can involve taking advantage of single or multiple GPU nodes as a provider for graphical acceleration.<br /> * Developing an on-demand hardware allocation mechanism using the queuing system to enable unique per-session access to unique hardware resources e.g. CPU or accelerator equipped nodes.<br /> * Defining community specific desktops.<br /> <br /> '''Ongoing activities'''<br /> <br /> Lunarc have been improving and testing configuration on the existing Lunarc HPC Desktop. Knowledge from this have been the basis for the design of the upcoming prototype hardware. Prototype hardware for evaluating future desktop environments have been procured from SouthPole. The hardware consists of:<br /> <br /> * NVIDIA K1/K2 evaluation system for investigating an on-demand desktop solution with accelerated graphics support. Solutions evaluated will be hypervisor-based using SLURM to allocate sessions.<br /> * Intel Xeon Clearwell system for evaluating graphics support in this architecture. Could be used for providing cost-effective desktop solutions for HPC resources.<br /> * Commodity graphics card in a server setting.<br /> <br /> Prototype system will be available mid february.<br /> <br /> == Future HPC and Accelerators (PDC, Lunarc, HPC2N) ==<br /> <br /> For future investment of computing resources in SNIC in the coming years (three large and several smaller systems), it is important that there is current specialist knowledge about different types of existing and future CPU architectures. To obtain a good basis for future decision in HPC resources, it is important that the work done at the centres, the SNIC GPU project and activities within PRACE are continued and developed further.<br /> <br /> '''Objectives and deliverables'''<br /> <br /> * Evaluate the use of next generation GPU:s K40 and others in the context of future SNIC resources. (Lunarc, HPC2N)<br /> * Operate and provide support for SNICs investments in the Zorn GPU development resource (PDC)<br /> * Operate and provide support for SNICs investments in the Erik GPU development resource (Lunarc)<br /> * Evaluate the use of next generation Xeon Phi accelerators (Knights Landing) (Lunarc, HPC2N)<br /> <br /> '''Ongoing activities'''<br /> <br /> Lunarc have procured a NVIDIA K80 which will be installed in Erik. (Availability XX-XX-XX)<br /> HPC2N is installing Xeon Phi:s in the existing resources (Availability XX-XX-XX)<br /> <br /> == Members ==<br /> {{#ask: [[Category:Person]] [[project::{{PAGENAME}}]]<br /> |?centre<br /> |?role<br /> |?field<br /> }}</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=SNIC_Emerging_Technologies&diff=6033 2015-02-04T09:27:28Z

<p>Jonas Lindemann (LUNARC): </p> <hr /> <div>{{project info<br /> |description=Coordinating new and emerging technologies within SNIC.<br /> |fields=Emerging Technologies<br /> |financing=SNIC<br /> |active=yes<br /> |start date=2014-05-01<br /> |end date=<br /> }}<br /> <br /> Currently, many SNIC centres are continuously investigate the market for current developments in emerging technologies and even buy hardware for evaluation. However, in many of these activities user involvement is limited and efforts are often duplicated between centres. This activity aims to coordinate these efforts and competence within SNIC and provide SNIC users and communities with early access to new technologies. The project will also work closely with the user communities to see if they have interests in upcoming technologies.<br /> <br /> Three focus areas have been and are described in the following sections:<br /> <br /> # Access-methods and remote visualisation (Lunarc, UPPMAX)<br /> # Future HPC and Accelerators (Lunarc, PDC, HPC2N)<br /> <br /> == Storage Technologies (C3SE, HPC2N) ==<br /> <br /> There are many storage technologies employed within SNIC. To be able to deploy solutions that are suitable for different usage scenarios, it is important that there is a continuous evaluation of availa-ble storage technologies. It is also important that prototype solutions are evaluated closely to users and facilities. Typical projects within this focus area could be:<br /> <br /> * File systems for different I/O patterns.<br /> * New storage hardware.<br /> * Higher-level file services.<br /> * Client tools for accessing available resources.<br /> * Integration services.<br /> <br /> '''Objectives and deliverables'''<br /> <br /> * Investigate how existing resources can be facilitated in new ways and how new approaches for users on how to use existing solutions can be facilitated to provide better support for I/O-intensive simulations and work-flows. (C3SE)<br /> * Investigate how the RobinHood service of Lustre v2+ filestystems can be used in conjunction with Tivoli TSM to speed up incremental backups. (C3SE, HPC2N)<br /> * Investigate how the RobinHood service of Lustre v2+ filesystems can be used to provide more fine-grained quotas, ex. for project storage etc. (C3SE)<br /> * One solution that has been used in SNIC for a number of years for long-term storage of re-search data is dCache in combination with TSM for archiving to tape. There are several al-ternatives to using TSM for archiving to tape, which should be further studied both from a performance and cost effectiveness. One example is LTFS [3]. To look at various options for long-term storage is also something that connects well to activities within WLCG and EIS-CAT_3D for which HPC2N are involved in. (HPC2N)<br /> <br /> '''Ongoing activities'''<br /> <br /> Users and groupquota through Littlejohn. Tested in operation. Scans file system and then uses changelogs to update database. Have since discovered a bug in the Lustre for changeloghantering. To be integrated for better project reporting<br /> <br /> Robin Hood and the TSM integration. Based on the changelog is active. Limited progress. Is an activity on the Lustre list.<br /> <br /> == Access-methods and remote visualisation (Lunarc, UPPMAX) ==<br /> <br /> To improve the usability of our HPC resources, many centres have been providing remote desktop solutions to users in an effort to provide a richer user experience in HPC. The service has been a great success in the research groups and the number of users is increasing. Requests for new applications and usage are increasing steadily. <br /> However, the increasing user base and request for new applications and usages has raised questions on how to efficiently scale these services in many dimensions.<br /> <br /> * Monitoring of usage patterns on the desktop servers.<br /> * Monitoring and scaling of distribution networks using different usage patterns and desktop configurations such as spatial resolution.<br /> * How to provide seamless access to hardware accelerated desktop services.<br /> * How to provide interactive access to applications and specific hardware configurations (e.g. GPU enabled nodes) through the batch-systems in a transparent way in the same desktop environment.<br /> <br /> Identifying bottlenecks in remote desktop architectures.<br /> <br /> '''Objectives and deliverables'''<br /> <br /> Remote desktop service aspects:<br /> <br /> * Evaluate different remote-access architectures based on target network specifications.<br /> * Develop best practice guides for configuration and setup of remote desktop services.<br /> * Evaluate techniques for monitoring the load and usage of the servers providing the desktop services, so that the user experience is good. Analyse usage bottlenecks and identify areas for improvement such as easy and (eventually) integrated SFTP file transfer.<br /> * Implement prototypes for providing a scalable and OS independent hardware accelerated back-end to the desktop interface. This can involve taking advantage of single or multiple GPU nodes as a provider for graphical acceleration.<br /> * Developing an on-demand hardware allocation mechanism using the queuing system to enable unique per-session access to unique hardware resources e.g. CPU or accelerator equipped nodes.<br /> * Defining community specific desktops.<br /> <br /> '''Ongoing activities'''<br /> <br /> Lunarc have been improving and testing configuration on the existing Lunarc HPC Desktop. Knowledge from this have been the basis for the design of the upcoming prototype hardware. Prototype hardware for evaluating future desktop environments have been procured from SouthPole. The hardware consists of:<br /> <br /> * NVIDIA K1/K2 evaluation system for investigating an on-demand desktop solution with accelerated graphics support. Solutions evaluated will be hypervisor-based using SLURM to allocate sessions.<br /> * Intel Xeon Clearwell system for evaluating graphics support in this architecture. Could be used for providing cost-effective desktop solutions for HPC resources.<br /> * Commodity graphics card in a server setting.<br /> <br /> Prototype system will be available mid february.<br /> <br /> == Future HPC and Accelerators (PDC, Lunarc, HPC2N) ==<br /> <br /> For future investment of computing resources in SNIC in the coming years (three large and several smaller systems), it is important that there is current specialist knowledge about different types of existing and future CPU architectures. To obtain a good basis for future decision in HPC resources, it is important that the work done at the centres, the SNIC GPU project and activities within PRACE are continued and developed further.<br /> <br /> '''Objectives and deliverables'''<br /> <br /> * Evaluate the use of next generation GPU:s K40 and others in the context of future SNIC resources. (Lunarc, HPC2N)<br /> * Operate and provide support for SNICs investments in the Zorn GPU development resource (PDC)<br /> * Operate and provide support for SNICs investments in the Erik GPU development resource (Lunarc)<br /> * Evaluate the use of next generation Xeon Phi accelerators (Knights Landing) (Lunarc, HPC2N)<br /> <br /> '''Ongoing activities'''<br /> <br /> Lunarc have procured a NVIDIA K80 which will be installed in Erik. (Availability XX-XX-XX)<br /> HPC2N is installing Xeon Phi:s in the existing resources (Availability XX-XX-XX)<br /> <br /> == Members ==<br /> {{#ask: [[Category:Person]] [[project::{{PAGENAME}}]]<br /> |?centre<br /> |?role<br /> |?field<br /> }}</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=SNIC_Emerging_Technologies&diff=6032 2015-02-04T09:22:39Z

<p>Jonas Lindemann (LUNARC): </p> <hr /> <div>{{project info<br /> |description=Coordinating new and emerging technologies within SNIC.<br /> |fields=Emerging Technologies<br /> |financing=SNIC<br /> |active=yes<br /> |start date=2014-05-01<br /> |end date=<br /> }}<br /> <br /> Currently, many SNIC centres are continuously investigate the market for current developments in emerging technologies and even buy hardware for evaluation. However, in many of these activities user involvement is limited and efforts are often duplicated between centres. This activity aims to coordinate these efforts and competence within SNIC and provide SNIC users and communities with early access to new technologies. The project will also work closely with the user communities to see if they have interests in upcoming technologies.<br /> <br /> Three focus areas have been defined:<br /> <br /> # Storage Technologies (C3SE, HPC2N)<br /> # Access-methods and remote visualisation (Lunarc, UPPMAX)<br /> # Future HPC and Accelerators (Lunarc, PDC, HPC2N)<br /> <br /> == Storage Technologies ==<br /> <br /> There are many storage technologies employed within SNIC. To be able to deploy solutions that are suitable for different usage scenarios, it is important that there is a continuous evaluation of availa-ble storage technologies. It is also important that prototype solutions are evaluated closely to users and facilities. Typical projects within this focus area could be:<br /> <br /> * File systems for different I/O patterns.<br /> * New storage hardware.<br /> * Higher-level file services.<br /> * Client tools for accessing available resources.<br /> * Integration services.<br /> <br /> '''Objectives and deliverables'''<br /> <br /> * Investigate how existing resources can be facilitated in new ways and how new approaches for users on how to use existing solutions can be facilitated to provide better support for I/O-intensive simulations and work-flows. (C3SE)<br /> * Investigate how the RobinHood service of Lustre v2+ filestystems can be used in conjunction with Tivoli TSM to speed up incremental backups. (C3SE, HPC2N)<br /> * Investigate how the RobinHood service of Lustre v2+ filesystems can be used to provide more fine-grained quotas, ex. for project storage etc. (C3SE)<br /> * One solution that has been used in SNIC for a number of years for long-term storage of re-search data is dCache in combination with TSM for archiving to tape. There are several al-ternatives to using TSM for archiving to tape, which should be further studied both from a performance and cost effectiveness. One example is LTFS [3]. To look at various options for long-term storage is also something that connects well to activities within WLCG and EIS-CAT_3D for which HPC2N are involved in. (HPC2N)<br /> <br /> '''Ongoing activities'''<br /> <br /> Users and groupquota through Littlejohn. Tested in operation. Scans file system and then uses changelogs to update database. Have since discovered a bug in the Lustre for changeloghantering. To be integrated for better project reporting<br /> <br /> Robin Hood and the TSM integration. Based on the changelog is active. Limited progress. Is an activity on the Lustre list.<br /> <br /> == Access-methods and remote visualisation ==<br /> <br /> To improve the usability of our HPC resources, many centres have been providing remote desktop solutions to users in an effort to provide a richer user experience in HPC. The service has been a great success in the research groups and the number of users is increasing. Requests for new applications and usage are increasing steadily. <br /> However, the increasing user base and request for new applications and usages has raised questions on how to efficiently scale these services in many dimensions.<br /> <br /> * Monitoring of usage patterns on the desktop servers.<br /> * Monitoring and scaling of distribution networks using different usage patterns and desktop configurations such as spatial resolution.<br /> * How to provide seamless access to hardware accelerated desktop services.<br /> * How to provide interactive access to applications and specific hardware configurations (e.g. GPU enabled nodes) through the batch-systems in a transparent way in the same desktop environment.<br /> <br /> Identifying bottlenecks in remote desktop architectures.<br /> <br /> '''Objectives and deliverables'''<br /> <br /> Remote desktop service aspects:<br /> <br /> * Evaluate different remote-access architectures based on target network specifications.<br /> * Develop best practice guides for configuration and setup of remote desktop services.<br /> * Evaluate techniques for monitoring the load and usage of the servers providing the desktop services, so that the user experience is good. Analyse usage bottlenecks and identify areas for improvement such as easy and (eventually) integrated SFTP file transfer.<br /> * Implement prototypes for providing a scalable and OS independent hardware accelerated back-end to the desktop interface. This can involve taking advantage of single or multiple GPU nodes as a provider for graphical acceleration.<br /> * Developing an on-demand hardware allocation mechanism using the queuing system to enable unique per-session access to unique hardware resources e.g. CPU or accelerator equipped nodes.<br /> * Defining community specific desktops.<br /> <br /> '''Ongoing activities'''<br /> <br /> Lunarc have been improving and testing configuration on the existing Lunarc HPC Desktop. Knowledge from this have been the basis for the design of the upcoming prototype hardware. Prototype hardware for evaluating future desktop environments have been procured from SouthPole. The hardware consists of:<br /> <br /> * NVIDIA K1/K2 evaluation system for investigating an on-demand desktop solution with accelerated graphics support. Solutions evaluated will be hypervisor-based using SLURM to allocate sessions.<br /> * Intel Xeon Clearwell system for evaluating graphics support in this architecture. Could be used for providing cost-effective desktop solutions for HPC resources.<br /> * Commodity graphics card in a server setting.<br /> <br /> Prototype system will be available mid february.<br /> <br /> == Future HPC and Accelerators ==<br /> <br /> For future investment of computing resources in SNIC in the coming years (three large and several smaller systems), it is important that there is current specialist knowledge about different types of existing and future CPU architectures. To obtain a good basis for future decision in HPC resources, it is important that the work done at the centres, the SNIC GPU project and activities within PRACE are continued and developed further.<br /> <br /> '''Objectives and deliverables'''<br /> <br /> * Evaluate the use of next generation GPU:s K40 and others in the context of future SNIC resources. (Lunarc, HPC2N)<br /> * Operate and provide support for SNICs investments in the Zorn GPU development resource (PDC)<br /> * Operate and provide support for SNICs investments in the Erik GPU development resource (Lunarc)<br /> * Evaluate the use of next generation Xeon Phi accelerators (Knights Landing) (Lunarc, HPC2N)<br /> <br /> '''Ongoing activities'''<br /> <br /> Lunarc have procured a NVIDIA K80 which will be installed in Erik. (Availability XX-XX-XX)<br /> HPC2N is installing Xeon Phi:s in the existing resources (Availability XX-XX-XX)<br /> <br /> == Members ==<br /> {{#ask: [[Category:Person]] [[project::{{PAGENAME}}]]<br /> |?centre<br /> |?role<br /> |?field<br /> }}</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=SNIC_Emerging_Technologies&diff=6031 2015-02-04T08:42:09Z

<p>Jonas Lindemann (LUNARC): </p> <hr /> <div>{{project info<br /> |description=Coordinating new and emerging technologies within SNIC.<br /> |fields=Emerging Technologies<br /> |financing=SNIC<br /> |active=yes<br /> |start date=2014-05-01<br /> |end date=<br /> }}<br /> <br /> Currently, many SNIC centres are continuously investigate the market for current developments in emerging technologies and even buy hardware for evaluation. However, in many of these activities user involvement is limited and efforts are often duplicated between centres. This activity aims to coordinate these efforts and competence within SNIC and provide SNIC users and communities with early access to new technologies. The project will also work closely with the user communities to see if they have interests in upcoming technologies.<br /> <br /> Three focus areas have been defined:<br /> <br /> # Storage Technologies (C3SE, HPC2N)<br /> # Access-methods and remote visualisation (Lunarc, UPPMAX)<br /> # Future HPC and Accelerators (Lunarc, PDC, HPC2N)<br /> <br /> == Storage Technologies ==<br /> <br /> There are many storage technologies employed within SNIC. To be able to deploy solutions that are suitable for different usage scenarios, it is important that there is a continuous evaluation of availa-ble storage technologies. It is also important that prototype solutions are evaluated closely to users and facilities. Typical projects within this focus area could be:<br /> <br /> * File systems for different I/O patterns.<br /> * New storage hardware.<br /> * Higher-level file services.<br /> * Client tools for accessing available resources.<br /> * Integration services.<br /> <br /> '''Objectives and deliverables'''<br /> <br /> * Investigate how existing resources can be facilitated in new ways and how new approaches for users on how to use existing solutions can be facilitated to provide better support for I/O-intensive simulations and work-flows. (C3SE)<br /> * Investigate how the RobinHood service of Lustre v2+ filestystems can be used in conjunction with Tivoli TSM to speed up incremental backups. (C3SE, HPC2N)<br /> * Investigate how the RobinHood service of Lustre v2+ filesystems can be used to provide more fine-grained quotas, ex. for project storage etc. (C3SE)<br /> * One solution that has been used in SNIC for a number of years for long-term storage of re-search data is dCache in combination with TSM for archiving to tape. There are several al-ternatives to using TSM for archiving to tape, which should be further studied both from a performance and cost effectiveness. One example is LTFS [3]. To look at various options for long-term storage is also something that connects well to activities within WLCG and EIS-CAT_3D for which HPC2N are involved in. (HPC2N)<br /> <br /> == Access-methods and remote visualisation ==<br /> <br /> To improve the usability of our HPC resources, many centres have been providing remote desktop solutions to users in an effort to provide a richer user experience in HPC. The service has been a great success in the research groups and the number of users is increasing. Requests for new applications and usage are increasing steadily. <br /> However, the increasing user base and request for new applications and usages has raised questions on how to efficiently scale these services in many dimensions.<br /> <br /> * Monitoring of usage patterns on the desktop servers.<br /> * Monitoring and scaling of distribution networks using different usage patterns and desktop configurations such as spatial resolution.<br /> * How to provide seamless access to hardware accelerated desktop services.<br /> * How to provide interactive access to applications and specific hardware configurations (e.g. GPU enabled nodes) through the batch-systems in a transparent way in the same desktop environment.<br /> <br /> Identifying bottlenecks in remote desktop architectures.<br /> <br /> '''Objectives and deliverables'''<br /> <br /> Remote desktop service aspects:<br /> <br /> * Evaluate different remote-access architectures based on target network specifications.<br /> * Develop best practice guides for configuration and setup of remote desktop services.<br /> * Evaluate techniques for monitoring the load and usage of the servers providing the desktop services, so that the user experience is good. Analyse usage bottlenecks and identify areas for improvement such as easy and (eventually) integrated SFTP file transfer.<br /> <br /> Access to hardware accelerated graphics:<br /> <br /> * Implement prototypes for providing a scalable and OS independent hardware accelerated back-end to the desktop interface. This can involve taking advantage of single or multiple GPU nodes as a provider for graphical acceleration.<br /> <br /> Interactive use of graphical user interfaces:<br /> <br /> * Developing an on-demand hardware allocation mechanism using the queuing system to enable unique per-session access to unique hardware resources e.g. CPU or accelerator equipped nodes.<br /> * Defining community specific desktops.<br /> <br /> '''Ongoing activities'''<br /> <br /> Prototype hardware for evaluating future desktop environments have been procured at Lunarc. The hardware consists of:<br /> <br /> * NVIDIA K1/K2 evaluation system for investigating an on-demand desktop solution with accelerated graphics support. Solutions evaluated will be hypervisor-based using SLURM to allocate sessions.<br /> * Intel Xeon Clearwell system for evaluating graphics support in this architecture. Could be used for providing cost-effective desktop solutions for HPC resources.<br /> * Commodity graphics card in a server setting.<br /> <br /> Prototype system will be available mid february.<br /> <br /> == Members ==<br /> {{#ask: [[Category:Person]] [[project::{{PAGENAME}}]]<br /> |?centre<br /> |?role<br /> |?field<br /> }}</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=SNIC_Emerging_Technologies&diff=6030 2015-02-04T08:27:59Z

<p>Jonas Lindemann (LUNARC): </p> <hr /> <div>{{project info<br /> |description=Coordinating new and emerging technologies within SNIC.<br /> |fields=Emerging Technologies<br /> |financing=SNIC<br /> |active=yes<br /> |start date=2014-05-01<br /> |end date=<br /> }}<br /> <br /> Currently, many SNIC centres are continuously investigate the market for current developments in emerging technologies and even buy hardware for evaluation. However, in many of these activities user involvement is limited and efforts are often duplicated between centres. This activity aims to coordinate these efforts and competence within SNIC and provide SNIC users and communities with early access to new technologies. The project will also work closely with the user communities to see if they have interests in upcoming technologies.<br /> <br /> Three focus areas have been defined:<br /> <br /> # Storage Technologies (C3SE, HPC2N)<br /> # Access-methods and remote visualisation (Lunarc, UPPMAX)<br /> # Future HPC and Accelerators (Lunarc, PDC, HPC2N)<br /> <br /> == Storage Technologies ==<br /> <br /> There are many storage technologies employed within SNIC. To be able to deploy solutions that are suitable for different usage scenarios, it is important that there is a continuous evaluation of availa-ble storage technologies. It is also important that prototype solutions are evaluated closely to users and facilities. Typical projects within this focus area could be:<br /> <br /> * File systems for different I/O patterns.<br /> * New storage hardware.<br /> * Higher-level file services.<br /> * Client tools for accessing available resources.<br /> * Integration services.<br /> <br /> '''Objectives and deliverables'''<br /> <br /> * Investigate how existing resources can be facilitated in new ways and how new approaches for users on how to use existing solutions can be facilitated to provide better support for I/O-intensive simulations and work-flows. (C3SE)<br /> * Investigate how the RobinHood service of Lustre v2+ filestystems can be used in conjunction with Tivoli TSM to speed up incremental backups. (C3SE, HPC2N)<br /> * Investigate how the RobinHood service of Lustre v2+ filesystems can be used to provide more fine-grained quotas, ex. for project storage etc. (C3SE)<br /> * One solution that has been used in SNIC for a number of years for long-term storage of re-search data is dCache in combination with TSM for archiving to tape. There are several al-ternatives to using TSM for archiving to tape, which should be further studied both from a performance and cost effectiveness. One example is LTFS [3]. To look at various options for long-term storage is also something that connects well to activities within WLCG and EIS-CAT_3D for which HPC2N are involved in. (HPC2N)<br /> <br /> == Access-methods and remote visualisation ==<br /> <br /> To improve the usability of our HPC resources, many centres have been providing remote desktop solutions to users in an effort to provide a richer user experience in HPC. The service has been a great success in the research groups and the number of users is increasing. Requests for new applications and usage are increasing steadily. <br /> However, the increasing user base and request for new applications and usages has raised questions on how to efficiently scale these services in many dimensions.<br /> <br /> * Monitoring of usage patterns on the desktop servers.<br /> * Monitoring and scaling of distribution networks using different usage patterns and desktop configurations such as spatial resolution.<br /> * How to provide seamless access to hardware accelerated desktop services.<br /> * How to provide interactive access to applications and specific hardware configurations (e.g. GPU enabled nodes) through the batch-systems in a transparent way in the same desktop environment.<br /> <br /> Identifying bottlenecks in remote desktop architectures.<br /> <br /> '''Objectives and deliverables'''<br /> <br /> Remote desktop service aspects:<br /> <br /> * Evaluate different remote-access architectures based on target network specifications.<br /> * Develop best practice guides for configuration and setup of remote desktop services.<br /> * Evaluate techniques for monitoring the load and usage of the servers providing the desktop services, so that the user experience is good. Analyse usage bottlenecks and identify areas for improvement such as easy and (eventually) integrated SFTP file transfer.<br /> <br /> Access to hardware accelerated graphics:<br /> <br /> * Implement prototypes for providing a scalable and OS independent hardware accelerated back-end to the desktop interface. This can involve taking advantage of single or multiple GPU nodes as a provider for graphical acceleration.<br /> <br /> Interactive use of graphical user interfaces:<br /> <br /> * Developing an on-demand hardware allocation mechanism using the queuing system to enable unique per-session access to unique hardware resources e.g. CPU or accelerator equipped nodes.<br /> * Defining community specific desktops.<br /> <br /> <br /> <br /> == Members ==<br /> {{#ask: [[Category:Person]] [[project::{{PAGENAME}}]]<br /> |?centre<br /> |?role<br /> |?field<br /> }}</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=SNIC_Emerging_Technologies&diff=6029 2015-02-04T08:26:34Z

<p>Jonas Lindemann (LUNARC): </p> <hr /> <div>{{project info<br /> |description=Coordinating new and emerging technologies within SNIC.<br /> |fields=Emerging Technologies<br /> |financing=SNIC<br /> |active=yes<br /> |start date=2014-05-01<br /> |end date=<br /> }}<br /> <br /> Currently, many SNIC centres are continuously investigate the market for current developments in emerging technologies and even buy hardware for evaluation. However, in many of these activities user involvement is limited and efforts are often duplicated between centres. This activity aims to coordinate these efforts and competence within SNIC and provide SNIC users and communities with early access to new technologies. The project will also work closely with the user communities to see if they have interests in upcoming technologies.<br /> <br /> Three focus areas have been defined:<br /> <br /> # Storage Technologies (C3SE, HPC2N)<br /> # Access-methods and remote visualisation (Lunarc, UPPMAX)<br /> # Future HPC and Accelerators (Lunarc, PDC, HPC2N)<br /> <br /> == Storage Technologies ==<br /> <br /> There are many storage technologies employed within SNIC. To be able to deploy solutions that are suitable for different usage scenarios, it is important that there is a continuous evaluation of availa-ble storage technologies. It is also important that prototype solutions are evaluated closely to users and facilities. Typical projects within this focus area could be:<br /> <br /> * File systems for different I/O patterns.<br /> * New storage hardware.<br /> * Higher-level file services.<br /> * Client tools for accessing available resources.<br /> * Integration services.<br /> <br /> === Objectives and deliverables === <br /> <br /> * Investigate how existing resources can be facilitated in new ways and how new approaches for users on how to use existing solutions can be facilitated to provide better support for I/O-intensive simulations and work-flows. (C3SE)<br /> * Investigate how the RobinHood service of Lustre v2+ filestystems can be used in conjunction with Tivoli TSM to speed up incremental backups. (C3SE, HPC2N)<br /> * Investigate how the RobinHood service of Lustre v2+ filesystems can be used to provide more fine-grained quotas, ex. for project storage etc. (C3SE)<br /> * One solution that has been used in SNIC for a number of years for long-term storage of re-search data is dCache in combination with TSM for archiving to tape. There are several al-ternatives to using TSM for archiving to tape, which should be further studied both from a performance and cost effectiveness. One example is LTFS [3]. To look at various options for long-term storage is also something that connects well to activities within WLCG and EIS-CAT_3D for which HPC2N are involved in. (HPC2N)<br /> <br /> == Access-methods and remote visualisation ==<br /> <br /> To improve the usability of our HPC resources, many centres have been providing remote desktop solutions to users in an effort to provide a richer user experience in HPC. The service has been a great success in the research groups and the number of users is increasing. Requests for new applications and usage are increasing steadily. <br /> However, the increasing user base and request for new applications and usages has raised questions on how to efficiently scale these services in many dimensions.<br /> <br /> * Monitoring of usage patterns on the desktop servers.<br /> * Monitoring and scaling of distribution networks using different usage patterns and desktop configurations such as spatial resolution.<br /> * How to provide seamless access to hardware accelerated desktop services.<br /> * How to provide interactive access to applications and specific hardware configurations (e.g. GPU enabled nodes) through the batch-systems in a transparent way in the same desktop environment.<br /> <br /> Identifying bottlenecks in remote desktop architectures.<br /> <br /> === Objectives and deliverables === <br /> <br /> Remote desktop service aspects:<br /> <br /> * Evaluate different remote-access architectures based on target network specifications.<br /> * Develop best practice guides for configuration and setup of remote desktop services.<br /> * Evaluate techniques for monitoring the load and usage of the servers providing the desktop services, so that the user experience is good. Analyse usage bottlenecks and identify areas for improvement such as easy and (eventually) integrated SFTP file transfer.<br /> <br /> Access to hardware accelerated graphics:<br /> <br /> * Implement prototypes for providing a scalable and OS independent hardware accelerated back-end to the desktop interface. This can involve taking advantage of single or multiple GPU nodes as a provider for graphical acceleration.<br /> <br /> Interactive use of graphical user interfaces:<br /> <br /> * Developing an on-demand hardware allocation mechanism using the queuing system to enable unique per-session access to unique hardware resources e.g. CPU or accelerator equipped nodes.<br /> * Defining community specific desktops.<br /> <br /> <br /> <br /> == Members ==<br /> {{#ask: [[Category:Person]] [[project::{{PAGENAME}}]]<br /> |?centre<br /> |?role<br /> |?field<br /> }}</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=User:Jonas_Lindemann_(LUNARC)&diff=6028 2015-02-04T08:21:43Z

<p>Jonas Lindemann (LUNARC): </p> <hr /> <div>{{application expert info<br /> |first name=Jonas<br /> |last name=Lindemann<br /> |centre=LUNARC<br /> |fields=Grid computing;Desktop Environments<br /> |other activities=Director at LUNARC;Coordinator SNIC Emerging Technologies<br /> |image=JonasLindemann.png<br /> |office=John Ericssons väg 1; 221 00 Lund<br /> |phone=(+46)462228162;(+46)707910118<br /> |fte=20<br /> |financing=SNIC<br /> |snic ae financing=<br /> |other ae financing=LU<br /> |general activities=Coordinating SNIC Emerging Technologies;Leading the development of ARC Storage UI;Grid user documentation;Developer of ARC Job Submission Tool;Lunarc Box;Lunarc HPC Desktop<br /> }}<br /> &lt;!--<br /> == Quick facts ==<br /> * Or any other heading. <br /> * This part is clear text and very much up to you.<br /> * Bullet lists are good <br /> * for the lazy<br /> --&gt;<br /> <br /> == Expertise ==<br /> * [[expertise::Grid computing]]<br /> * [[expertise::Python]]<br /> * [[expertise::SciPy]]<br /> * [[expertise::Numpy]]<br /> * [[expertise::Fortran]]<br /> * [[expertise::C++]]<br /> * [[expertise::Structural Mechanics]]<br /> <br /> == Projects ==<br /> * [[project::SNIC Emerging Technologies]]</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=SNIC_Emerging_Technologies&diff=6027 2015-02-04T08:20:27Z

<p>Jonas Lindemann (LUNARC): </p> <hr /> <div>{{project info<br /> |description=Coordinating new and emerging technologies within SNIC.<br /> |fields=Emerging Technologies<br /> |financing=SNIC<br /> |active=yes<br /> |start date=2014-05-01<br /> |end date=<br /> }}<br /> <br /> Currently, many SNIC centres are continuously investigate the market for current developments in emerging technologies and even buy hardware for evaluation. However, in many of these activities user involvement is limited and efforts are often duplicated between centres. This activity aims to coordinate these efforts and competence within SNIC and provide SNIC users and communities with early access to new technologies. The project will also work closely with the user communities to see if they have interests in upcoming technologies.<br /> <br /> Three focus areas have been defined:<br /> <br /> # Storage Technologies (C3SE, HPC2N)<br /> # Access-methods and remote visualisation (Lunarc, UPPMAX)<br /> # Future HPC and Accelerators (Lunarc, PDC, HPC2N)<br /> <br /> Joined project to coordinate the training offered by the [[Centres|SNIC centres]]. An overview on the training events offered or supported by SNIC is available on the [[Training|training page]] of this wiki.<br /> <br /> == Members ==<br /> {{#ask: [[Category:Person]] [[project::{{PAGENAME}}]]<br /> |?centre<br /> |?role<br /> |?field<br /> }}</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=SNIC_ET&diff=6026 2015-02-04T08:19:30Z

<p>Jonas Lindemann (LUNARC): Blanked the page</p> <hr /> <div></div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=SNIC_Emerging_Technologies&diff=6025 2015-02-04T08:19:06Z

<p>Jonas Lindemann (LUNARC): Created page with &quot;{{project info |description=SNIC Emerging Technologies |fields=Emerging Technologies |financing=SNIC |active=yes |start date=2014-05-01 |end date= }} Currently, many SNIC centre...&quot;</p> <hr /> <div>{{project info<br /> |description=SNIC Emerging Technologies<br /> |fields=Emerging Technologies<br /> |financing=SNIC<br /> |active=yes<br /> |start date=2014-05-01<br /> |end date=<br /> }}<br /> <br /> Currently, many SNIC centres are continuously investigate the market for current developments in emerging technologies and even buy hardware for evaluation. However, in many of these activities user involvement is limited and efforts are often duplicated between centres. This activity aims to coordinate these efforts and competence within SNIC and provide SNIC users and communities with early access to new technologies. The project will also work closely with the user communities to see if they have interests in upcoming technologies.<br /> <br /> Three focus areas have been defined:<br /> <br /> # Storage Technologies (C3SE, HPC2N)<br /> # Access-methods and remote visualisation (Lunarc, UPPMAX)<br /> # Future HPC and Accelerators (Lunarc, PDC, HPC2N)<br /> <br /> Joined project to coordinate the training offered by the [[Centres|SNIC centres]]. An overview on the training events offered or supported by SNIC is available on the [[Training|training page]] of this wiki.<br /> <br /> == Members ==<br /> {{#ask: [[Category:Person]] [[project::{{PAGENAME}}]]<br /> |?centre<br /> |?role<br /> |?field<br /> }}</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=SNIC_ET&diff=6024 2015-02-04T08:14:59Z

<p>Jonas Lindemann (LUNARC): </p> <hr /> <div>{{DISPLAYTITLE:SNIC Emerging Technologies}}<br /> <br /> {{project info<br /> |description=SNIC Emerging Technologies<br /> |fields=Emerging Technologies<br /> |financing=SNIC<br /> |active=yes<br /> |start date=2014-05-01<br /> |end date=<br /> }}<br /> <br /> Currently, many SNIC centres are continuously investigate the market for current developments in emerging technologies and even buy hardware for evaluation. However, in many of these activities user involvement is limited and efforts are often duplicated between centres. This activity aims to coordinate these efforts and competence within SNIC and provide SNIC users and communities with early access to new technologies. The project will also work closely with the user communities to see if they have interests in upcoming technologies.<br /> <br /> Three focus areas have been defined:<br /> <br /> # Storage Technologies (C3SE, HPC2N)<br /> # Access-methods and remote visualisation (Lunarc, UPPMAX)<br /> # Future HPC and Accelerators (Lunarc, PDC, HPC2N)<br /> <br /> Joined project to coordinate the training offered by the [[Centres|SNIC centres]]. An overview on the training events offered or supported by SNIC is available on the [[Training|training page]] of this wiki.<br /> <br /> == Members ==<br /> {{#ask: [[Category:Person]] [[project::{{PAGENAME}}]]<br /> |?centre<br /> |?role<br /> |?field<br /> }}</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=SNIC_ET&diff=6023 2015-02-04T08:11:41Z

<p>Jonas Lindemann (LUNARC): </p> <hr /> <div>{{project info<br /> |description=SNIC Emerging Technologies<br /> |fields=Emerging Technologies<br /> |financing=SNIC<br /> |active=yes<br /> |start date=2014-05-01<br /> |end date=<br /> }}<br /> <br /> Currently, many SNIC centres are continuously investigate the market for current developments in emerging technologies and even buy hardware for evaluation. However, in many of these activities user involvement is limited and efforts are often duplicated between centres. This activity aims to coordinate these efforts and competence within SNIC and provide SNIC users and communities with early access to new technologies. The project will also work closely with the user communities to see if they have interests in upcoming technologies.<br /> <br /> Three focus areas have been defined:<br /> <br /> # Storage Technologies (C3SE, HPC2N)<br /> # Access-methods and remote visualisation (Lunarc, UPPMAX)<br /> # Future HPC and Accelerators (Lunarc, PDC, HPC2N)<br /> <br /> Joined project to coordinate the training offered by the [[Centres|SNIC centres]]. An overview on the training events offered or supported by SNIC is available on the [[Training|training page]] of this wiki.<br /> <br /> == Members ==<br /> {{#ask: [[Category:Person]] [[project::{{PAGENAME}}]]<br /> |?centre<br /> |?role<br /> |?field<br /> }}</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=SNIC_ET&diff=6022 2015-02-04T08:11:07Z

<p>Jonas Lindemann (LUNARC): </p> <hr /> <div>{{project info<br /> |description=SNIC Emerging Technologies<br /> |fields=Emerging Technologies<br /> |financing=SNIC<br /> |active=yes<br /> |start date=2014-05-01<br /> |end date=<br /> }}<br /> <br /> Currently, many SNIC centres are continuously investigate the market for current developments in emerging technologies and even buy hardware for evaluation. However, in many of these activities user involvement is limited and efforts are often duplicated between centres. This activity aims to coordinate these efforts and competence within SNIC and provide SNIC users and communities with early access to new technologies. The project will also work closely with the user communities to see if they have interests in upcoming technologies.<br /> <br /> Three focus areas have been defined:<br /> <br /> 1. Storage Technologies (C3SE, HPC2N)<br /> 2. Access-methods and remote visualisation (Lunarc, UPPMAX)<br /> 3. Future HPC and Accelerators (Lunarc, PDC, HPC2N)<br /> <br /> Joined project to coordinate the training offered by the [[Centres|SNIC centres]]. An overview on the training events offered or supported by SNIC is available on the [[Training|training page]] of this wiki.<br /> <br /> == Members ==<br /> {{#ask: [[Category:Person]] [[project::{{PAGENAME}}]]<br /> |?centre<br /> |?role<br /> |?field<br /> }}</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=SNIC_ET&diff=6021 2015-02-04T08:10:50Z

<p>Jonas Lindemann (LUNARC): Created page with &quot;{{project info |description=SNIC Emerging Technologies |fields=Emerging Technologies |financing=SNIC |active=yes |start date=2014-05-01 |end date= }} Currently, many SNIC centre...&quot;</p> <hr /> <div>{{project info<br /> |description=SNIC Emerging Technologies<br /> |fields=Emerging Technologies<br /> |financing=SNIC<br /> |active=yes<br /> |start date=2014-05-01<br /> |end date=<br /> }}<br /> <br /> Currently, many SNIC centres are continuously investigate the market for current developments in emerging technologies and even buy hardware for evaluation. However, in many of these activities user involvement is limited and efforts are often duplicated between centres. This activity aims to coordinate these efforts and competence within SNIC and provide SNIC users and communities with early access to new technologies. The project will also work closely with the user communities to see if they have interests in upcoming technologies.<br /> <br /> Three focus areas have been defined:<br /> <br /> 1. Storage Technologies (C3SE, HPC2N)<br /> 2. Access-methods and remote visualisation (Lunarc, UPPMAX)<br /> 3. Future HPC and Accelerators (Lunarc, PDC, HPC2N)<br /> <br /> Joined project to coordinate the training offered by the [[Centres|SNIC centres]]. An overview on the training events offered or supported by SNIC is available on the [[Training|training page]] of this wiki.<br /> <br /> == Members ==<br /> {{#ask: [[Category:Person]] [[project::{{PAGENAME}}]]<br /> |?centre<br /> |?role<br /> |?field<br /> }}</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=User:Jonas_Lindemann_(LUNARC)&diff=6020 2015-02-04T08:03:26Z

<p>Jonas Lindemann (LUNARC): </p> <hr /> <div>{{application expert info<br /> |first name=Jonas<br /> |last name=Lindemann<br /> |centre=LUNARC<br /> |fields=Grid computing;Desktop Environments<br /> |other activities=Director at LUNARC;Coordinator SNIC Emerging Technologies<br /> |image=JonasLindemann.png<br /> |office=John Ericssons väg 1; 221 00 Lund<br /> |phone=(+46)462228162;(+46)707910118<br /> |fte=20<br /> |financing=SNIC<br /> |snic ae financing=<br /> |other ae financing=LU<br /> |general activities=Coordinating SNIC Emerging Technologies;Leading the development of ARC Storage UI;Grid user documentation;Developer of ARC Job Submission Tool;Lunarc Box;Lunarc HPC Desktop<br /> }}<br /> &lt;!--<br /> == Quick facts ==<br /> * Or any other heading. <br /> * This part is clear text and very much up to you.<br /> * Bullet lists are good <br /> * for the lazy<br /> --&gt;<br /> <br /> == Expertise ==<br /> * [[expertise::Grid computing]]<br /> * [[expertise::Python]]<br /> * [[expertise::SciPy]]<br /> * [[expertise::Numpy]]<br /> * [[expertise::Fortran]]<br /> * [[expertise::C++]]<br /> * [[expertise::Structural Mechanics]]</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=User:Jonas_Lindemann_(LUNARC)&diff=6019 2015-02-04T08:02:30Z

<p>Jonas Lindemann (LUNARC): </p> <hr /> <div>{{application expert info<br /> |first name=Jonas<br /> |last name=Lindemann<br /> |centre=LUNARC<br /> |fields=Grid computing<br /> |other activities=Director at LUNARC;Coordinator SNIC Emerging Technologies<br /> |image=JonasLindemann.png<br /> |office=John Ericssons väg 1; 221 00 Lund<br /> |phone=(+46)462228162;(+46)707910118<br /> |fte=20<br /> |financing=SNIC<br /> |snic ae financing=<br /> |other ae financing=LU<br /> |general activities=Coordinating SNIC Emerging Technologies;Leading the development of ARC Storage UI;Grid user documentation;Developer of ARC Job Submission Tool;Lunarc Box;Lunarc HPC Desktop<br /> }}<br /> &lt;!--<br /> == Quick facts ==<br /> * Or any other heading. <br /> * This part is clear text and very much up to you.<br /> * Bullet lists are good <br /> * for the lazy<br /> --&gt;<br /> <br /> == Expertise ==<br /> * [[expertise::Grid computing]]<br /> * [[expertise::Python]]<br /> * [[expertise::SciPy]]<br /> * [[expertise::Numpy]]<br /> * [[expertise::Fortran]]<br /> * [[expertise::C++]]<br /> * [[expertise::Structural Mechanics]]</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=Preparing_a_client_certificate&diff=5253 2013-07-02T09:04:16Z

<p>Jonas Lindemann (LUNARC): /* Uploading and conversion of the .p12 for your target machine */</p> <hr /> <div>[[Category:Grid computing]]<br /> [[Category:SweGrid user guide]]<br /> [[Category:SweStore]]<br /> [[Category:SweStore user guide]]<br /> <br /> [[Getting started with SweGrid|&lt; Getting started with SweGrid]]<br /> <br /> [[SweStore|&lt; SweStore]]<br /> <br /> Most of the standalone third party tools installed on SNIC resources and your own machine will not be able to use a &lt;tt&gt;.p12&lt;/tt&gt; certificate bundle (or &lt;tt&gt;.pfx&lt;/tt&gt; if you exported from IE), as that format is intended primarily for secure transport and backup of certificates and their private keys.<br /> <br /> Instead of a single &lt;tt&gt;.p12&lt;/tt&gt; file, they expect a pair of files in &lt;tt&gt;.pem&lt;/tt&gt; format, one containing the certificate and the other containing the private key that matches the certificate.<br /> <br /> == Uploading and conversion of the .p12 for your target machine ==<br /> <br /> As the authentication methods for clusters differ, this section will defer to documentations for your particular site when it comes to transferring files to and from the cluster storage.<br /> <br /> The goal is to end up with a &lt;tt&gt;.globus&lt;/tt&gt; directory in your home directory, containing two files named &lt;tt&gt;usercert.pem&lt;/tt&gt; and &lt;tt&gt;userkey.pem&lt;/tt&gt;.<br /> <br /> The instructions below assume that your exported certificate file is named &lt;tt&gt;export.p12&lt;/tt&gt; directly in your home directory. If it's a &lt;tt&gt;.pfx&lt;/tt&gt; or with a different name, change &lt;tt&gt;export.p12&lt;/tt&gt; in the instructions to your actual filename or rename your file to &lt;tt&gt;export.p12&lt;/tt&gt;.<br /> <br /> * Transfer the &lt;tt&gt;export.p12&lt;/tt&gt; file to your home directory on the cluster.<br /> * Get an interactive shell on the login node, via ssh.<br /> * If an .globus directory already exists, rename it with something like<br /> &lt;tt&gt;mv ~/.globus ~/.globus-old&lt;/tt&gt;<br /> * Create the directory with<br /> &lt;tt&gt;mkdir ~/.globus&lt;/tt&gt;<br /> &lt;tt&gt;chmod 0700 ~/.globus&lt;/tt&gt;<br /> * Extract and protect the private key part of &lt;tt&gt;export.p12&lt;/tt&gt;:<br /> openssl pkcs12 -nocerts -in ~/export.p12 -out ~/.globus/userkey.pem<br /> * When asked for import password, specify the password specified when exporting the certificate bundle from your browser. The PEM pass phrase should be a new password that you need to provide whenever using the certificate for tasks like generating a proxy certificate. The output from this command will be similar to the following:<br /> Enter Import Password: *******<br /> MAC verified OK<br /> Enter PEM pass phrase: *******<br /> Verifying - Enter PEM pass phrase: *******<br /> <br /> * Extract the public client certificate part of &lt;tt&gt;export.p12&lt;/tt&gt;:<br /> openssl pkcs12 -clcerts -nokeys -in ~/export.p12 -out ~/.globus/usercert.pem<br /> * The output will be similar to the following:<br /> Enter Import Password: *******<br /> MAC verified OK<br /> * Finally ensure that only your user is allowed to read the private key file. This is important, both for security and due to some tools refusing to use private keys with insufficient restrictions.<br /> chmod 0400 ~/.globus/userkey.pem</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=ARC_client_installation_Mac_OS_X&diff=5153 2013-05-10T10:33:43Z

<p>Jonas Lindemann (LUNARC): </p> <hr /> <div>Installing the ARC Graphical Clients on Mac OS X requires MacPorts. When MacPorts has been installed check out the following 2 repositories to your home directory:<br /> <br /> &lt;pre&gt;<br /> $ svn checkout svn://svn.code.sf.net/p/arc-gui-clients/svn/trunk/macports/globus-ports ./globus-ports<br /> $ svn checkout svn://svn.code.sf.net/p/arc-gui-clients/svn/trunk/macports/arc-ports ./arc-ports<br /> &lt;/pre&gt;<br /> <br /> Next, update add the local repositories to to MacPorts by adding the following lines in the /opt/local/etc/macports/sources.conf before the rsync://rsync.macports… line, as shown below:<br /> <br /> &lt;pre&gt;<br /> file:///Users/[user id]/globus-ports [nosync]<br /> file:///Users/[user id]/arc-ports [nosync]<br /> rsync://rsync.macports.org/release/tarballs/ports.tar [default]<br /> &lt;/pre&gt;<br /> <br /> To make MacPorts aware of the new local repos issue the following commands:<br /> <br /> &lt;pre&gt;<br /> $ cd globus-ports<br /> $ portindex<br /> $ cd ../arc-ports<br /> $ portindex<br /> &lt;/pre&gt;<br /> <br /> To install the ARC client tools issue the following commands:<br /> <br /> &lt;pre&gt;<br /> $ sudo port install nordugrid-arc-client<br /> &lt;/pre&gt;<br /> <br /> The graphical clients can then be installed with:<br /> <br /> &lt;pre&gt;<br /> $ sudo port install arc-gui-clients<br /> &lt;/pre&gt;<br /> <br /> Please note that proxy-generation using the firefox credential store is somewhat buggy. If it doesn't work you can always export your certificate and convert it using the arccert-ui graphical tool. Look in the MacPorts folder in Applications. The other tools can also be found in this folder.</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=ARC_client_installation_Mac_OS_X&diff=5152 2013-05-10T10:32:40Z

<p>Jonas Lindemann (LUNARC): Created page with &quot;Installing the ARC Graphical Clients on Mac OS X requires MacPorts. When MacPorts has been installed check out the following 2 repositories to your home directory: &lt;pre&gt; $ svn c...&quot;</p> <hr /> <div>Installing the ARC Graphical Clients on Mac OS X requires MacPorts. When MacPorts has been installed check out the following 2 repositories to your home directory:<br /> <br /> &lt;pre&gt;<br /> $ svn checkout svn://svn.code.sf.net/p/arc-gui-clients/svn/trunk/macports/globus-ports ./globus-ports<br /> $ svn checkout svn://svn.code.sf.net/p/arc-gui-clients/svn/trunk/macports/arc-ports ./arc-ports<br /> &lt;/pre&gt;<br /> <br /> Next, update add the local repositories to to MacPorts by adding the following lines in the /opt/local/etc/macports/sources.conf before the rsync://rsync.macports… line, as shown below:<br /> <br /> file:///Users/[user id]/globus-ports [nosync]<br /> file:///Users/[user id]/arc-ports [nosync]<br /> rsync://rsync.macports.org/release/tarballs/ports.tar [default]<br /> <br /> To make MacPorts aware of the new local repos issue the following commands:<br /> <br /> $ cd globus-ports<br /> $ portindex<br /> $ cd ../arc-ports<br /> $ portindex<br /> <br /> To install the ARC client tools issue the following commands:<br /> <br /> $ sudo port install nordugrid-arc-client<br /> <br /> The graphical clients can then be installed with:<br /> <br /> $ sudo port install arc-gui-clients<br /> <br /> Please note that proxy-generation using the firefox credential store is somewhat buggy right now. If it doesn't work you can always export your certificate and convert it using the arccert-ui graphical tool. Look in the MacPorts folder in Applications. The other tools can also be found in this folder.</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=Swestore-dCache&diff=5151 2013-04-30T14:17:30Z

<p>Jonas Lindemann (LUNARC): /* Getting access */</p> <hr /> <div>[[Category:Storage]]<br /> [[Category:SweStore]]<br /> SNIC is building a storage infrastructure to complement the computational resources.<br /> <br /> Many forms of automated measurements can produce large amounts of data. In scientific areas such as high energy physics (the Large Hadron Collider at CERN), climate modeling, bioinformatics, bioimaging etc., the demands for storage are increasing dramatically. To serve these and other user communities, SNIC has appointed a working group to design a storage strategy, taking into account the needs on many levels and creating a unified storage infrastructure, which is now being implemented.<br /> <br /> Swestore is in collaboration with [http://www.ecds.se/ ECDS], [http://snd.gu.se/ SND], Bioimage Sweden, [http://www.bils.se/ BILS], [http://www.uppnex.uu.se/ UPPNEX],[http://wlcg.web.cern.ch/ WLCG], [http://www.nrm.se/ NaturHistoriska RiksMuseet].<br /> <br /> = National storage =<br /> The Swestore Nationally Accessible Storage, commonly called just Swestore, is a robust, flexible and expandable long<br /> term storage system aimed at storing large amounts of data produced by various Swedish research projects. It is based on the [http://www.dcache.org dCache]<br /> storage system and is distributed across the SNIC centres [http://www.c3se.chalmers.se/ C3SE], [http://www.hpc2n.umu.se/ HPC2N], [http://www.lunarc.lu.se/ Lunarc],<br /> [http://www.nsc.liu.se/ NSC], [http://www.pdc.kth.se PDC] and [http://www.uppmax.uu.se Uppmax].<br /> <br /> Data is stored in two copies with each copy at a different SNIC centre. This enables the system to cope with a multitude of issues ranging from a simple<br /> crash of a storage element to losing an entire site while stil providing access to the stored data. To protect against silent data corruption the<br /> dCache storage system checksums all stored data and periodically verifies the data using this checksum.<br /> <br /> The system does NOT yet provide protection against user errors like inadvertent file deletions and so on.<br /> <br /> One of the major advantages to the distributed nature of dCache is the excellent aggregated transfer rates possible. This is achieved by bypassing a central node<br /> and having transfers going directly to/from the storage elements if the protocol allows it.<br /> The Swestore Nationally Accessible Storage system can achieve aggregated transfer rates<br /> in excess of 100 Gigabit per second, but in practice this is limited by connectivity to each University (usually 10 Gbit/s) or a limited number of files (typically<br /> max 1 Gbit/s per file/connection).<br /> <br /> ==Access protocols==<br /> ; Currently supported protocols<br /> : GridFTP - gsiftp://gsiftp.swestore.se/<br /> : Storage Resource Manager - srm://srm.swegrid.se/<br /> : Hypertext Transfer Protocol (read-only), Web Distributed Authoring and Versioning - http://webdav.swestore.se/ (unauthenticated), https://webdav.swestore.se/<br /> <br /> ; Protocols in evaluation/development<br /> : NFS4.1, iRODS<br /> <br /> For authentication eScience certificates are used, which provides a higher level of security than legacy username/password schemes.<br /> <br /> == Getting access ==<br /> ; Apply for storage<br /> : Please follow instructions [[Apply for storage on SweStore|here]]<br /> ; Get a client certificate.<br /> : Follow the instructions [[Grid_certificates#Requesting_a_certificate|here]] to get your client certificate. For Terena certificates, please make sure you also [[Exporting_a_client_certificate|export the certificate for use with grid tools]]. For Nordugrid certificates, please make sure to also [[Requesting_a_grid_certificate_from_the_Nordugrid_CA#Installing_the_certificate_in_your_browser|install your client certificate in your browser]].<br /> ; Request membership in the SweGrid VO.<br /> : Follow the instructions [[Grid_certificates#Requesting_membership_in_the_SweGrid_VO|here]] to get added to the SweGrid virtual organisation.<br /> ; Transmit and prepare the certificate.<br /> : In order to use the client certificate on SNIC resources for generating proxy certificates and using command line tools, the certificate needs to be [[Preparing_a_client_certificate|converted into PEM files]] on the target cluster if not already in that format.<br /> <br /> NOTE: If you have installed a Terena-certificate in your browser and you have ARC 3.x installed, there is no need to convert or export the certificate from the browser. The arcproxy command can generate a proxy-certificate from the certificate stored in the Firefox credential store. See also [[Grid_certificates#Creating_a_proxy_certificate_using_the_Firefox.2FThunderbird_credential_store|proxy certificates]].<br /> <br /> == Download and upload data ==<br /> ; Interactive browsing and manipulation of single files<br /> : SweStore is accessible in your web browser in two ways, as a directory index interface at https://webdav.swestore.se/ and with an interactive file manager at https://webdav.swestore.se/browser/. To browse private data you must first install your certificate in your browser (see above). Projects are organized under the &lt;code&gt;/snic&lt;/code&gt; directory as &lt;code&gt;&lt;nowiki&gt;https://webdav.swestore.se/snic/YOUR_PROJECT_NAME/&lt;/nowiki&gt;&lt;/code&gt;.<br /> ; Upload and delete data interactively or with automation<br /> There are several tools that are capable of using the protocols provided by SweStore national storage.<br /> For interactive usage on SNIC clusters we recommend using the ARC tools which should be installed on all SNIC resources.<br /> As an integration point for building scripts and automated systems we suggest using the curl program and library.<br /> : Use the ARC client. Please see the instructions for [[Accessing SweStore national storage with the ARC client]].<br /> : Use lftp. Please see the instructions for [[Accessing SweStore national storage with lftp]].<br /> : Use cURL. Please see the instructions for [[Accessing SweStore national storage with cURL]].<br /> : Use globus-url-copy. Please see the instructions for [[Accessing SweStore national storage with globus-url-copy]].<br /> <br /> == More information ==<br /> * [http://status.swestore.se/munin/monitor/monitor/ Per Project Monitoring of Swestore usage]<br /> <br /> If you have any issues using SweStore please do not hesitate to contact [mailto:swestore-support@snic.vr.se swestore-support].<br /> <br /> == Tools and scripts ==<br /> <br /> There exists a number of tools and utilities developed externally that can be useful. Here are some links:<br /> <br /> * [https://github.com/samuell/arc_tools ARC_Tools] - Convenience scripts for the arc client (Only a recursive rmdir so far).<br /> * [http://sourceforge.net/projects/arc-gui-clients ARC Graphical Clients] - Contains the ARC Storage Explorer (SweStore supported development).<br /> * Transfer script, [http://snicdocs.nsc.liu.se/wiki/SweStore/swstrans_arc swetrans_arc], provided by Adam Peplinski / Philipp Schlatter<br /> * [http://www.nordugrid.org/documents/SWIG-wrapped-ARC-Python-API.pdf Documentation of the ARC Python API (PDF)]<br /> <br /> == Slides and more ==<br /> <br /> [http://docs.snic.se/wiki/Swestore/Lund_Seminar_Apr18 Slides and material from seminar for Lund users on April 18th]<br /> <br /> = Center storage =<br /> Centre storage, as defined by the SNIC storage group, is a storage solution that lives independently of the computational resources and can be accessed from all such resources at a centre. Key features include the ability to access the same filesystem the same way on all computational resources at a centre, and a unified structure and nomenclature for all centra. Unlike cluster storage which is tightly associated with a single cluster, and thus has a limited life-time, centre storage does not require the users to migrate their own data when clusters are decommissioned, not even when the storage hardware itself is being replaced.<br /> <br /> == Unified environment ==<br /> To make the usage more transparent for SNIC users, a set of environment variables are available on all SNIC resources:<br /> <br /> * &lt;code&gt;SNIC_BACKUP&lt;/code&gt; – the user's primary directory at the centre&lt;br&gt;(the part of the centre storage that is backed up)<br /> * &lt;code&gt;SNIC_NOBACKUP&lt;/code&gt; – recommended directory for project storage without backup&lt;br&gt;(also on the centre storage)<br /> * &lt;code&gt;SNIC_TMP&lt;/code&gt; – recommended directory for best performance during a job&lt;br&gt;(local disk on nodes if applicable)</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=Swestore-dCache&diff=5150 2013-04-30T14:16:48Z

<p>Jonas Lindemann (LUNARC): /* Getting access */</p> <hr /> <div>[[Category:Storage]]<br /> [[Category:SweStore]]<br /> SNIC is building a storage infrastructure to complement the computational resources.<br /> <br /> Many forms of automated measurements can produce large amounts of data. In scientific areas such as high energy physics (the Large Hadron Collider at CERN), climate modeling, bioinformatics, bioimaging etc., the demands for storage are increasing dramatically. To serve these and other user communities, SNIC has appointed a working group to design a storage strategy, taking into account the needs on many levels and creating a unified storage infrastructure, which is now being implemented.<br /> <br /> Swestore is in collaboration with [http://www.ecds.se/ ECDS], [http://snd.gu.se/ SND], Bioimage Sweden, [http://www.bils.se/ BILS], [http://www.uppnex.uu.se/ UPPNEX],[http://wlcg.web.cern.ch/ WLCG], [http://www.nrm.se/ NaturHistoriska RiksMuseet].<br /> <br /> = National storage =<br /> The Swestore Nationally Accessible Storage, commonly called just Swestore, is a robust, flexible and expandable long<br /> term storage system aimed at storing large amounts of data produced by various Swedish research projects. It is based on the [http://www.dcache.org dCache]<br /> storage system and is distributed across the SNIC centres [http://www.c3se.chalmers.se/ C3SE], [http://www.hpc2n.umu.se/ HPC2N], [http://www.lunarc.lu.se/ Lunarc],<br /> [http://www.nsc.liu.se/ NSC], [http://www.pdc.kth.se PDC] and [http://www.uppmax.uu.se Uppmax].<br /> <br /> Data is stored in two copies with each copy at a different SNIC centre. This enables the system to cope with a multitude of issues ranging from a simple<br /> crash of a storage element to losing an entire site while stil providing access to the stored data. To protect against silent data corruption the<br /> dCache storage system checksums all stored data and periodically verifies the data using this checksum.<br /> <br /> The system does NOT yet provide protection against user errors like inadvertent file deletions and so on.<br /> <br /> One of the major advantages to the distributed nature of dCache is the excellent aggregated transfer rates possible. This is achieved by bypassing a central node<br /> and having transfers going directly to/from the storage elements if the protocol allows it.<br /> The Swestore Nationally Accessible Storage system can achieve aggregated transfer rates<br /> in excess of 100 Gigabit per second, but in practice this is limited by connectivity to each University (usually 10 Gbit/s) or a limited number of files (typically<br /> max 1 Gbit/s per file/connection).<br /> <br /> ==Access protocols==<br /> ; Currently supported protocols<br /> : GridFTP - gsiftp://gsiftp.swestore.se/<br /> : Storage Resource Manager - srm://srm.swegrid.se/<br /> : Hypertext Transfer Protocol (read-only), Web Distributed Authoring and Versioning - http://webdav.swestore.se/ (unauthenticated), https://webdav.swestore.se/<br /> <br /> ; Protocols in evaluation/development<br /> : NFS4.1, iRODS<br /> <br /> For authentication eScience certificates are used, which provides a higher level of security than legacy username/password schemes.<br /> <br /> == Getting access ==<br /> ; Apply for storage<br /> : Please follow instructions [[Apply for storage on SweStore|here]]<br /> ; Get a client certificate.<br /> : Follow the instructions [[Grid_certificates#Requesting_a_certificate|here]] to get your client certificate. For Terena certificates, please make sure you also [[Exporting_a_client_certificate|export the certificate for use with grid tools]]. For Nordugrid certificates, please make sure to also [[Requesting_a_grid_certificate_from_the_Nordugrid_CA#Installing_the_certificate_in_your_browser|install your client certificate in your browser]].<br /> ; Request membership in the SweGrid VO.<br /> : Follow the instructions [[Grid_certificates#Requesting_membership_in_the_SweGrid_VO|here]] to get added to the SweGrid virtual organisation.<br /> ; Transmit and prepare the certificate.<br /> : In order to use the client certificate on SNIC resources for generating proxy certificates and using command line tools, the certificate needs to be [[Preparing_a_client_certificate|converted into PEM files]] on the target cluster if not already in that format.<br /> <br /> NOTE: If you have installed a Terena-certificate in your browser and you have ARC 3.x installed, there is no need to convert or export the certificate from the browser. The arcproxy command can generate a proxy-certificate from the certificate stored in the Firefox credential store. See also [Grid_certificates#Creating_a_proxy_certificate_using_the_Firefox.2FThunderbird_credential_store|proxy certificates ]<br /> <br /> == Download and upload data ==<br /> ; Interactive browsing and manipulation of single files<br /> : SweStore is accessible in your web browser in two ways, as a directory index interface at https://webdav.swestore.se/ and with an interactive file manager at https://webdav.swestore.se/browser/. To browse private data you must first install your certificate in your browser (see above). Projects are organized under the &lt;code&gt;/snic&lt;/code&gt; directory as &lt;code&gt;&lt;nowiki&gt;https://webdav.swestore.se/snic/YOUR_PROJECT_NAME/&lt;/nowiki&gt;&lt;/code&gt;.<br /> ; Upload and delete data interactively or with automation<br /> There are several tools that are capable of using the protocols provided by SweStore national storage.<br /> For interactive usage on SNIC clusters we recommend using the ARC tools which should be installed on all SNIC resources.<br /> As an integration point for building scripts and automated systems we suggest using the curl program and library.<br /> : Use the ARC client. Please see the instructions for [[Accessing SweStore national storage with the ARC client]].<br /> : Use lftp. Please see the instructions for [[Accessing SweStore national storage with lftp]].<br /> : Use cURL. Please see the instructions for [[Accessing SweStore national storage with cURL]].<br /> : Use globus-url-copy. Please see the instructions for [[Accessing SweStore national storage with globus-url-copy]].<br /> <br /> == More information ==<br /> * [http://status.swestore.se/munin/monitor/monitor/ Per Project Monitoring of Swestore usage]<br /> <br /> If you have any issues using SweStore please do not hesitate to contact [mailto:swestore-support@snic.vr.se swestore-support].<br /> <br /> == Tools and scripts ==<br /> <br /> There exists a number of tools and utilities developed externally that can be useful. Here are some links:<br /> <br /> * [https://github.com/samuell/arc_tools ARC_Tools] - Convenience scripts for the arc client (Only a recursive rmdir so far).<br /> * [http://sourceforge.net/projects/arc-gui-clients ARC Graphical Clients] - Contains the ARC Storage Explorer (SweStore supported development).<br /> * Transfer script, [http://snicdocs.nsc.liu.se/wiki/SweStore/swstrans_arc swetrans_arc], provided by Adam Peplinski / Philipp Schlatter<br /> * [http://www.nordugrid.org/documents/SWIG-wrapped-ARC-Python-API.pdf Documentation of the ARC Python API (PDF)]<br /> <br /> == Slides and more ==<br /> <br /> [http://docs.snic.se/wiki/Swestore/Lund_Seminar_Apr18 Slides and material from seminar for Lund users on April 18th]<br /> <br /> = Center storage =<br /> Centre storage, as defined by the SNIC storage group, is a storage solution that lives independently of the computational resources and can be accessed from all such resources at a centre. Key features include the ability to access the same filesystem the same way on all computational resources at a centre, and a unified structure and nomenclature for all centra. Unlike cluster storage which is tightly associated with a single cluster, and thus has a limited life-time, centre storage does not require the users to migrate their own data when clusters are decommissioned, not even when the storage hardware itself is being replaced.<br /> <br /> == Unified environment ==<br /> To make the usage more transparent for SNIC users, a set of environment variables are available on all SNIC resources:<br /> <br /> * &lt;code&gt;SNIC_BACKUP&lt;/code&gt; – the user's primary directory at the centre&lt;br&gt;(the part of the centre storage that is backed up)<br /> * &lt;code&gt;SNIC_NOBACKUP&lt;/code&gt; – recommended directory for project storage without backup&lt;br&gt;(also on the centre storage)<br /> * &lt;code&gt;SNIC_TMP&lt;/code&gt; – recommended directory for best performance during a job&lt;br&gt;(local disk on nodes if applicable)</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=Swestore-dCache&diff=5149 2013-04-30T14:16:02Z

<p>Jonas Lindemann (LUNARC): /* Getting access */</p> <hr /> <div>[[Category:Storage]]<br /> [[Category:SweStore]]<br /> SNIC is building a storage infrastructure to complement the computational resources.<br /> <br /> Many forms of automated measurements can produce large amounts of data. In scientific areas such as high energy physics (the Large Hadron Collider at CERN), climate modeling, bioinformatics, bioimaging etc., the demands for storage are increasing dramatically. To serve these and other user communities, SNIC has appointed a working group to design a storage strategy, taking into account the needs on many levels and creating a unified storage infrastructure, which is now being implemented.<br /> <br /> Swestore is in collaboration with [http://www.ecds.se/ ECDS], [http://snd.gu.se/ SND], Bioimage Sweden, [http://www.bils.se/ BILS], [http://www.uppnex.uu.se/ UPPNEX],[http://wlcg.web.cern.ch/ WLCG], [http://www.nrm.se/ NaturHistoriska RiksMuseet].<br /> <br /> = National storage =<br /> The Swestore Nationally Accessible Storage, commonly called just Swestore, is a robust, flexible and expandable long<br /> term storage system aimed at storing large amounts of data produced by various Swedish research projects. It is based on the [http://www.dcache.org dCache]<br /> storage system and is distributed across the SNIC centres [http://www.c3se.chalmers.se/ C3SE], [http://www.hpc2n.umu.se/ HPC2N], [http://www.lunarc.lu.se/ Lunarc],<br /> [http://www.nsc.liu.se/ NSC], [http://www.pdc.kth.se PDC] and [http://www.uppmax.uu.se Uppmax].<br /> <br /> Data is stored in two copies with each copy at a different SNIC centre. This enables the system to cope with a multitude of issues ranging from a simple<br /> crash of a storage element to losing an entire site while stil providing access to the stored data. To protect against silent data corruption the<br /> dCache storage system checksums all stored data and periodically verifies the data using this checksum.<br /> <br /> The system does NOT yet provide protection against user errors like inadvertent file deletions and so on.<br /> <br /> One of the major advantages to the distributed nature of dCache is the excellent aggregated transfer rates possible. This is achieved by bypassing a central node<br /> and having transfers going directly to/from the storage elements if the protocol allows it.<br /> The Swestore Nationally Accessible Storage system can achieve aggregated transfer rates<br /> in excess of 100 Gigabit per second, but in practice this is limited by connectivity to each University (usually 10 Gbit/s) or a limited number of files (typically<br /> max 1 Gbit/s per file/connection).<br /> <br /> ==Access protocols==<br /> ; Currently supported protocols<br /> : GridFTP - gsiftp://gsiftp.swestore.se/<br /> : Storage Resource Manager - srm://srm.swegrid.se/<br /> : Hypertext Transfer Protocol (read-only), Web Distributed Authoring and Versioning - http://webdav.swestore.se/ (unauthenticated), https://webdav.swestore.se/<br /> <br /> ; Protocols in evaluation/development<br /> : NFS4.1, iRODS<br /> <br /> For authentication eScience certificates are used, which provides a higher level of security than legacy username/password schemes.<br /> <br /> == Getting access ==<br /> ; Apply for storage<br /> : Please follow instructions [[Apply for storage on SweStore|here]]<br /> ; Get a client certificate.<br /> : Follow the instructions [[Grid_certificates#Requesting_a_certificate|here]] to get your client certificate. For Terena certificates, please make sure you also [[Exporting_a_client_certificate|export the certificate for use with grid tools]]. For Nordugrid certificates, please make sure to also [[Requesting_a_grid_certificate_from_the_Nordugrid_CA#Installing_the_certificate_in_your_browser|install your client certificate in your browser]].<br /> ; Request membership in the SweGrid VO.<br /> : Follow the instructions [[Grid_certificates#Requesting_membership_in_the_SweGrid_VO|here]] to get added to the SweGrid virtual organisation.<br /> ; Transmit and prepare the certificate.<br /> : In order to use the client certificate on SNIC resources for generating proxy certificates and using command line tools, the certificate needs to be [[Preparing_a_client_certificate|converted into PEM files]] on the target cluster if not already in that format.<br /> <br /> NOTE: If you have installed a Terena-certificate in your browser and you have ARC 3.x installed, there is no need to convert or export the certificate from the browser. The arcproxy command can generate a proxy-certificate from the certificate stored in the Firefox credential store. See also [proxy certificates | https://docs.snic.se/wiki/Grid_certificates#Creating_a_proxy_certificate_using_the_Firefox.2FThunderbird_credential_store]<br /> <br /> == Download and upload data ==<br /> ; Interactive browsing and manipulation of single files<br /> : SweStore is accessible in your web browser in two ways, as a directory index interface at https://webdav.swestore.se/ and with an interactive file manager at https://webdav.swestore.se/browser/. To browse private data you must first install your certificate in your browser (see above). Projects are organized under the &lt;code&gt;/snic&lt;/code&gt; directory as &lt;code&gt;&lt;nowiki&gt;https://webdav.swestore.se/snic/YOUR_PROJECT_NAME/&lt;/nowiki&gt;&lt;/code&gt;.<br /> ; Upload and delete data interactively or with automation<br /> There are several tools that are capable of using the protocols provided by SweStore national storage.<br /> For interactive usage on SNIC clusters we recommend using the ARC tools which should be installed on all SNIC resources.<br /> As an integration point for building scripts and automated systems we suggest using the curl program and library.<br /> : Use the ARC client. Please see the instructions for [[Accessing SweStore national storage with the ARC client]].<br /> : Use lftp. Please see the instructions for [[Accessing SweStore national storage with lftp]].<br /> : Use cURL. Please see the instructions for [[Accessing SweStore national storage with cURL]].<br /> : Use globus-url-copy. Please see the instructions for [[Accessing SweStore national storage with globus-url-copy]].<br /> <br /> == More information ==<br /> * [http://status.swestore.se/munin/monitor/monitor/ Per Project Monitoring of Swestore usage]<br /> <br /> If you have any issues using SweStore please do not hesitate to contact [mailto:swestore-support@snic.vr.se swestore-support].<br /> <br /> == Tools and scripts ==<br /> <br /> There exists a number of tools and utilities developed externally that can be useful. Here are some links:<br /> <br /> * [https://github.com/samuell/arc_tools ARC_Tools] - Convenience scripts for the arc client (Only a recursive rmdir so far).<br /> * [http://sourceforge.net/projects/arc-gui-clients ARC Graphical Clients] - Contains the ARC Storage Explorer (SweStore supported development).<br /> * Transfer script, [http://snicdocs.nsc.liu.se/wiki/SweStore/swstrans_arc swetrans_arc], provided by Adam Peplinski / Philipp Schlatter<br /> * [http://www.nordugrid.org/documents/SWIG-wrapped-ARC-Python-API.pdf Documentation of the ARC Python API (PDF)]<br /> <br /> == Slides and more ==<br /> <br /> [http://docs.snic.se/wiki/Swestore/Lund_Seminar_Apr18 Slides and material from seminar for Lund users on April 18th]<br /> <br /> = Center storage =<br /> Centre storage, as defined by the SNIC storage group, is a storage solution that lives independently of the computational resources and can be accessed from all such resources at a centre. Key features include the ability to access the same filesystem the same way on all computational resources at a centre, and a unified structure and nomenclature for all centra. Unlike cluster storage which is tightly associated with a single cluster, and thus has a limited life-time, centre storage does not require the users to migrate their own data when clusters are decommissioned, not even when the storage hardware itself is being replaced.<br /> <br /> == Unified environment ==<br /> To make the usage more transparent for SNIC users, a set of environment variables are available on all SNIC resources:<br /> <br /> * &lt;code&gt;SNIC_BACKUP&lt;/code&gt; – the user's primary directory at the centre&lt;br&gt;(the part of the centre storage that is backed up)<br /> * &lt;code&gt;SNIC_NOBACKUP&lt;/code&gt; – recommended directory for project storage without backup&lt;br&gt;(also on the centre storage)<br /> * &lt;code&gt;SNIC_TMP&lt;/code&gt; – recommended directory for best performance during a job&lt;br&gt;(local disk on nodes if applicable)</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=Grid_certificates&diff=5148 2013-04-30T14:15:06Z

<p>Jonas Lindemann (LUNARC): /* Proxy certificates */</p> <hr /> <div>[[Category:Grid computing]]<br /> [[Category:SweGrid user guide]]<br /> [[Category:SweStore]]<br /> [[Category:SweStore user guide]]<br /> [[Getting started with SweGrid|&lt; Getting started with SweGrid]]&lt;br&gt;<br /> [[SweStore|&lt; SweStore]]<br /> <br /> =Introduction to certificates=<br /> <br /> In order to get access to computer and storage resources on the grid or [[SweStore]] you must have a valid (grid) certificate. This certificate is used instead of a username and password when accessing the resource. The resource have a certificate that tells you that you have contacted the right resource. This is exactly the same mechanism used when you use a web browser to contact your bank.<br /> <br /> A certificate is the similar to a passport in real-life. In the same way you have prove your credentials when you acquire a passport the same is true for a certificate. A third party, the Certificate Authority or CA, that both you and the resource trust has to vouch for your identity and sign your certificate.<br /> <br /> A certificate consist of a public key, some user information and a signature of the CA. In addition to the certificate you have a private key. The private key is secret and should be kept as secure as possible.<br /> <br /> For more information regarding certificates and public key cryptography:<br /> <br /> [http://en.wikipedia.org/wiki/Public-key_cryptography http://en.wikipedia.org/wiki/Public-key_cryptography]<br /> <br /> [http://en.wikipedia.org/wiki/Public_key_certificate http://en.wikipedia.org/wiki/Public_key_certificate]<br /> <br /> [http://www.nordugrid.org/documents/certificate_howto.html http://www.nordugrid.org/documents/certificate_howto.html]<br /> <br /> * The grid certificate and the private key are stored in your web browser and/or located in ~/.globus at the host(s) from where you will be accessing the resource:<br /> usercert.pem<br /> userkey.pem<br /> * The certificate contains your public key, your name and organization and a signature by the CA. It is does not contain any username.<br /> * The certificate is valid for 13 month and should be renewed yearly.<br /> * The private key should be handled with great care. It should only be readable by you and not by the group or others (i.e. ``chmod 400 userkey.pem''). Store the key on trusted computers and transfer the key between computers using encryption (using for example scp).<br /> * On shared file systems make sure that ~/.globus is not readible by everybody:<br /> chmod 700 ~/.globus<br /> and on AFS:<br /> fs sa ~/.globus system:anyuser none<br /> * The private key is encrypted using a passphrase. Anyone that can decrypt the private key will be able to authenticate as you to grid resources. This is similar to the private key in SSH. You must choose a strong passphrase for the private key. This passphrase must not be used anywhere else. You must never ever give away the passphrase to somebody else.<br /> * You should not share the certificate with someone. It's personal. <br /> <br /> For more information regarding certificates and public key cryptography:<br /> <br /> [http://en.wikipedia.org/wiki/Public-key_cryptography http://en.wikipedia.org/wiki/Public-key_cryptography]<br /> [http://en.wikipedia.org/wiki/Public_key_certificate http://en.wikipedia.org/wiki/Public_key_certificate]<br /> <br /> = Requesting a certificate =<br /> <br /> Certificates are issued by a Certificate Authority or CA. For Swedish users there are two relevant CA:s that can issue grid/eScience certificates, Terena and Nordugrid. The Terena CA is preferred if it is available for your university or research group, but many sites has not enabled this service yet. The Nordugrid CA can also be used but requires more manual work by all parties.<br /> <br /> Recommended procedure for each university:<br /> <br /> {| class=&quot;wikitable&quot;<br /> | University<br /> | CA<br /> | Specific instructions<br /> |-<br /> | LU<br /> | Terena CA<br /> | [[LU_Certificate_Information|more...]]<br /> |-<br /> | LiU<br /> | Terena CA<br /> | [[LiU_Certificate_Instructions|more...]]<br /> |-<br /> | CTH<br /> | NorduGrid CA<br /> | [[Chalmers_Certificate_Instructions|more...]]<br /> |-<br /> | GU<br /> | NorduGrid CA<br /> | [[GU_Certificate_Instructions|more...]]<br /> |-<br /> | UU<br /> | Terena CA<br /> | [[UU_Certificate_Instructions|more...]]<br /> |-<br /> | KTH<br /> | Terena CA<br /> | [[KTH_Certificate_Information|more...]]<br /> |-<br /> | SU<br /> | NorduGrid CA<br /> | [[SU_Certificate_Information|more...]]<br /> |-<br /> | KI<br /> | NorduGrid CA<br /> | [[KI_Certificate_Information|more...]]<br /> |-<br /> | UmU<br /> | Terena CA<br /> | [[UmU_Certificate_Information|more...]]<br /> |-<br /> |}<br /> <br /> [[Requesting a grid certificate using the Terena eScience Portal|Instructions for the Terena CA]]<br /> <br /> [[Requesting a grid certificate from the Nordugrid CA|Instructions for the NorduGrid CA (use only if Terena eScience isn't available at your site)]]<br /> <br /> = Requesting membership in the SweGrid VO =<br /> <br /> SweGrid and SweStore resources are currently being allocated for VO:s, virtual organizations, rather than individual users. A VO is basically just a list of users. To be able to use a SweGrid or SweStore resource a membership in the SweGrid VO (virtual organization) and a corresponding subgroup is required. To apply for membership, make sure that the NorduGrid root CA certificate and your personal certificate is installed in the browser. <br /> <br /> The NorduGrid CA cert can be installed by clicking on the following link:<br /> <br /> [http://ca.nordugrid.org/cacrt.crt http://ca.nordugrid.org/cacrt.crt]<br /> <br /> Make sure you check the &quot;Trust this CA to identify web sites.&quot; boxes in the dialog shown.<br /> <br /> <br /> [[File:certinstall.png]]<br /> <br /> <br /> When certificates have been installed in the browser go to the following URL:<br /> <br /> [https://voms.ndgf.org:8443/voms/swegrid.se https://voms.ndgf.org:8443/voms/swegrid.se]<br /> <br /> and follow the instructions. In a couple of hours you will be added to the SweGrid VO. <br /> <br /> To be added to the correct SweGrid project send a mail to [mailto:support@swegrid.se support@swegrid.se] and specify your DN as shown in the Terena portal or from the '''arcproxy --info''' command and which SNIC-project to be added to.<br /> <br /> To be added to the correct Swestore allocation send a mail to [mailto:swestore-support@snic.vr.se swestore-support@snic.vr.se] and specify your DN as shown in the Terena portal or from the '''arcproxy --info''' command and which Swestore allocation to be added to.<br /> <br /> = Proxy certificates =<br /> <br /> Authentication on the grid is done using special short lived ''proxy'' certificates. There are several tools available for creating, checking and destroying these proxy certificates.<br /> <br /> == Creating a proxy certificate ==<br /> <br /> To create a short lived proxy that can be used for authentication with grid services, the '''arcproxy''' command can be used. A 12 hour (default) proxy is created in the following example:<br /> <br /> $ arcproxy<br /> Your identity: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=Kalle Kula<br /> Enter pass phrase for /home/kalle/.globus/userkey.pem:<br /> .++++++<br /> .....++++++<br /> Proxy generation succeeded<br /> Your proxy is valid until: 2011-03-11 03:00:14<br /> <br /> The proxy file itself will be created in the '''/tmp''' directory with the format '''x509up_uid''', where uid is the user id number for your account.<br /> <br /> In some cases a longer lived proxy will be needed. This is achieved using the '''--constraint''' switch. A 24-hour can be created by issuing the following command:<br /> <br /> $ arcproxy --constraint=&quot;validityPeriod=24H&quot;<br /> Your identity: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=Kalle Kula<br /> Enter pass phrase for /home/kalle/.globus/userkey.pem:<br /> ....++++++<br /> .....++++++<br /> Proxy generation succeeded<br /> Your proxy is valid until: 2011-03-11 15:03:19<br /> <br /> == Creating a proxy certificate using the Firefox/Thunderbird credential store ==<br /> <br /> Using the ARC 3.x client tools it is now possible to generate a proxy certificate directly from the Firefox or Thunderbird credential stores. To do this the '''-F''' flag is used as shown in the following example:<br /> <br /> $ arcproxy -F<br /> There are 2 NSS base directories where the certificate, key, and module datbases live<br /> Number 1 is: /Users/lindemann/Library/Application Support/Firefox/Profiles/t22f3aj2.default<br /> Number 2 is: /Users/lindemann/Library/Thunderbird/Profiles/7abb733v.default<br /> Please choose the NSS database you would use (1-2): 1<br /> <br /> Here ARC finds the available Firefox and Thunderbird profile in which the credential stores are stored. Next the passphrase for the credential store is used to unlock the stored credentials:<br /> <br /> NSS database to be accessed: /Users/lindemann/Library/Application Support/Firefox/Profiles/t22f3aj2.default<br /> Enter Password or Pin for &quot;internal (software)&quot;:<br /> <br /> If the passphrase was correct, ARC will list the available certificates in the credential store and ask you for which you would like to use.<br /> <br /> There are 2 user certificates existing in the NSS database<br /> Number 1 is with nickname: Jonas Lindemann xxxxx@lu.se's TERENA ID (Jonas Lindemann xxxxx@lu.se)<br /> expiration time: 2013-06-04 01:59:59<br /> Number 2 is with nickname: Imported Certificate (Jonas Lindemann)<br /> expiration time: 2014-01-18 16:55:52<br /> Please choose the one you would use (1-2): 1<br /> Certificate to use is: Jonas Lindemann xxxxxx@lu.se's TERENA ID<br /> Proxy generation succeeded<br /> Your proxy is valid until: 2013-05-01 04:11:37<br /> <br /> == Checking proxy lifetime ==<br /> <br /> The remaining lifetime of a proxy certificate can be checked using the '''arcproxy''' command with the '''--info''' switch.<br /> <br /> $ arcproxy --info<br /> Subject: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=Kalle Kula/CN=1567862803<br /> Identity: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=Kalle Kula<br /> Time left for proxy: 11 hours 55 minutes<br /> Proxy path: /tmp/x509up_u500<br /> Proxy type: X.509 Proxy Certificate Profile RFC compliant restricted proxy<br /> <br /> In this example the proxy certificate is valid for 11 hours 55 minutes more.<br /> <br /> == Destroying a proxy certificate ==<br /> <br /> A proxy can be destroyed with the '''-r''' or '''--remove''' switch.<br /> <br /> $ arcproxy -r<br /> <br /> or<br /> <br /> $ arcproxy --remove<br /> <br /> = VOMS certificates =<br /> <br /> As long as you are a member of only one VO or VO group, you can<br /> authenticate to a grid service with the regular grid proxy certificate<br /> as defined in the previous section. If you are a member of more than<br /> one VO or VO group you may want to select which membership you want to<br /> be authenticated as. For example, if you are a member of<br /> ''swegrid.se:/swegrid.se/ops'' (operations staff) and<br /> ''swegrid.se:/swegrid.se/bils'' and want to write a file, who should<br /> be the owner? Ops or bils? You need to provide some additional<br /> information. In the grid world this is done with a voms proxy<br /> certificate which basically is a regular proxy certificate but with a<br /> so called voms extension that contains a list of your VO group<br /> memberships (and roles and attributes, which we don't use in<br /> Swegrid/Swestore at the moment).<br /> <br /> '''Please note, if you only have one membership you can skip this section!'''<br /> <br /> The voms extension of the certificate is signed by the virtual<br /> organization management server, or VOMS server. The same VOMS server<br /> you used when applying for the swegrid.se VO membership in the first<br /> place. To enable this signing process you need to add a few<br /> configuration files to your system. First add this to the file<br /> '''/etc/vomses''':<br /> <br /> &quot;swegrid.se&quot; &quot;voms.ndgf.org&quot; &quot;15009&quot; &quot;/O=Grid/O=NorduGrid/CN=host/voms.ndgf.org&quot; &quot;swegrid.se&quot;<br /> <br /> Next create the necessary directories and the file<br /> '''/etc/grid-security/vomsdir/swegrid.se/voms.ndgf.org.lsc''' with the<br /> following contents:<br /> <br /> /O=Grid/O=NorduGrid/CN=host/voms.ndgf.org<br /> /O=Grid/O=NorduGrid/CN=NorduGrid Certification Authority<br /> <br /> == Creating a VOMS proxy ==<br /> <br /> VOMS proxies in ARC1 can be created using the '''arcproxy''' command<br /> and the '''-S''' or '''--voms''' switches as shown in the following<br /> example (if you are a member of the /swegrid.se/ops group. Adjust as<br /> necessary):<br /> <br /> $ arcproxy -S swegrid.se:/swegrid.se/ops<br /> Your identity: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=Kalle Kula<br /> Enter pass phrase for /home/kalle/.globus/userkey.pem:<br /> .....++++++<br /> ............++++++<br /> Contacting VOMS server (named swegrid.se): voms.ndgf.org on port: 15009<br /> Proxy generation succeeded<br /> Your proxy is valid until: 2011-03-10 23:33:06<br /> <br /> <br /> = Signing your e-mail with your certificate =<br /> <br /> First, you will need your grid certificate in PKCS12 format:<br /> == How to transform your certificate from PEM format into PKCS#12 format ==<br /> <br /> This is how you transform your cert into PKCS12 format that can be used within your webbrowser or email send program:<br /> You first will have to change directory into where you created and keep the certificate, historically this is often in ~/.globus<br /> <br /> openssl pkcs12 -export -in usercert.pem -inkey userkey.pem -out cert+key.p12 <br /> <br /> First you will have to enter the password you used for your private key, then you will be asked for a new password to protect the new file. '''cert+key.p12 contains your private key, and is therefore 'lika känslig' as userkey.pem'''. See also [[#Introduction to certificates]]. Security wise the safest way is to delete the PKCS12 file after having imported it into your mail client or browser. Don't forget this.<br /> <br /> Remarks: openssl will either need the variable RANDFILE to be set or that ~/.rnd is writable. So you have to make sure that the current $HOME is yours if you have pagshed away, otherwise the command will fail with ''unable to write 'random state''.<br /> <br /> <br /> === Signing in mew ===<br /> <br /> Mew uses gpgsm. <br /> <br /> &lt;pre&gt;<br /> 1. Import the nordugrid root cert<br /> <br /> 1.1. get 1f0e8352.0 from nordugrid web<br /> <br /> 1.2. gpgsm --import 1f0e8352.0<br /> <br /> 1.2. Make it trusted:<br /> gpgsm --list-keys 2&gt;/dev/null | grep fingerprint | awk '{print $2 &quot; S&quot;}' | grep THE-FINGERPRIT-YOU-WANT &gt;&gt; .gnupg/trustlist.txt<br /> <br /> 2. Add your own key from the cert+key.p12 file in this case<br /> <br /> 2.1 openssl pkcs12 -in cert+key.p12 -out tmp.pem -nokeys<br /> <br /> 2.2. gpgsm --import tmp.pem ; rm tmp.pem<br /> <br /> 2.3. Tell gpgsm not to use revocation lists (bad bad security)<br /> echo disable-crl-checks &gt;&gt; .gnupg/gpgsm.conf<br /> <br /> 3. Test<br /> gpgsm --detach-sign file &gt; sign # should ask for passphrase and give some kind of sign file<br /> <br /> 4. Use:<br /> C-uC-cC-s then enter your email address (must match email in cert) and passphrase<br /> <br /> &lt;/pre&gt;<br /> <br /> === Signing in thunderbird ===<br /> In thunderbird: options/security/digitally sign this message.<br /> <br /> If you do this for the first time and haven't defined yet the certificate to sign with, thunderbird will pop up the according preferences [Account settings/Security], where you can choose between your imported certificates in PKCS12 format.<br /> <br /> In the beginning, of course, you haven't imported any: Click there on the same preferences tab that popped up on [View Certificates]. In the new window that opens you can import the certificate.<br /> <br /> Afterwards you can then choose this certificate to be used for signing and for encryption for this email account.<br /> <br /> Don't forget to actually check that you then really sign the corresponding mail.</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=Swestore-dCache&diff=5147 2013-04-30T14:07:48Z

<p>Jonas Lindemann (LUNARC): /* Getting access */</p> <hr /> <div>[[Category:Storage]]<br /> [[Category:SweStore]]<br /> SNIC is building a storage infrastructure to complement the computational resources.<br /> <br /> Many forms of automated measurements can produce large amounts of data. In scientific areas such as high energy physics (the Large Hadron Collider at CERN), climate modeling, bioinformatics, bioimaging etc., the demands for storage are increasing dramatically. To serve these and other user communities, SNIC has appointed a working group to design a storage strategy, taking into account the needs on many levels and creating a unified storage infrastructure, which is now being implemented.<br /> <br /> Swestore is in collaboration with [http://www.ecds.se/ ECDS], [http://snd.gu.se/ SND], Bioimage Sweden, [http://www.bils.se/ BILS], [http://www.uppnex.uu.se/ UPPNEX],[http://wlcg.web.cern.ch/ WLCG], [http://www.nrm.se/ NaturHistoriska RiksMuseet].<br /> <br /> = National storage =<br /> The Swestore Nationally Accessible Storage, commonly called just Swestore, is a robust, flexible and expandable long<br /> term storage system aimed at storing large amounts of data produced by various Swedish research projects. It is based on the [http://www.dcache.org dCache]<br /> storage system and is distributed across the SNIC centres [http://www.c3se.chalmers.se/ C3SE], [http://www.hpc2n.umu.se/ HPC2N], [http://www.lunarc.lu.se/ Lunarc],<br /> [http://www.nsc.liu.se/ NSC], [http://www.pdc.kth.se PDC] and [http://www.uppmax.uu.se Uppmax].<br /> <br /> Data is stored in two copies with each copy at a different SNIC centre. This enables the system to cope with a multitude of issues ranging from a simple<br /> crash of a storage element to losing an entire site while stil providing access to the stored data. To protect against silent data corruption the<br /> dCache storage system checksums all stored data and periodically verifies the data using this checksum.<br /> <br /> The system does NOT yet provide protection against user errors like inadvertent file deletions and so on.<br /> <br /> One of the major advantages to the distributed nature of dCache is the excellent aggregated transfer rates possible. This is achieved by bypassing a central node<br /> and having transfers going directly to/from the storage elements if the protocol allows it.<br /> The Swestore Nationally Accessible Storage system can achieve aggregated transfer rates<br /> in excess of 100 Gigabit per second, but in practice this is limited by connectivity to each University (usually 10 Gbit/s) or a limited number of files (typically<br /> max 1 Gbit/s per file/connection).<br /> <br /> ==Access protocols==<br /> ; Currently supported protocols<br /> : GridFTP - gsiftp://gsiftp.swestore.se/<br /> : Storage Resource Manager - srm://srm.swegrid.se/<br /> : Hypertext Transfer Protocol (read-only), Web Distributed Authoring and Versioning - http://webdav.swestore.se/ (unauthenticated), https://webdav.swestore.se/<br /> <br /> ; Protocols in evaluation/development<br /> : NFS4.1, iRODS<br /> <br /> For authentication eScience certificates are used, which provides a higher level of security than legacy username/password schemes.<br /> <br /> == Getting access ==<br /> ; Apply for storage<br /> : Please follow instructions [[Apply for storage on SweStore|here]]<br /> ; Get a client certificate.<br /> : Follow the instructions [[Grid_certificates#Requesting_a_certificate|here]] to get your client certificate. For Terena certificates, please make sure you also [[Exporting_a_client_certificate|export the certificate for use with grid tools]]. For Nordugrid certificates, please make sure to also [[Requesting_a_grid_certificate_from_the_Nordugrid_CA#Installing_the_certificate_in_your_browser|install your client certificate in your browser]].<br /> ; Request membership in the SweGrid VO.<br /> : Follow the instructions [[Grid_certificates#Requesting_membership_in_the_SweGrid_VO|here]] to get added to the SweGrid virtual organisation.<br /> ; Transmit and prepare the certificate.<br /> : In order to use the client certificate on SNIC resources for generating proxy certificates and using command line tools, the certificate needs to be [[Preparing_a_client_certificate|converted into PEM files]] on the target cluster if not already in that format.<br /> <br /> NOTE: If you have installed a Terena-certificate in your browser and you have ARC 3.x installed, there is no need to convert or export the certificate from the browser. The arcproxy command can generate a proxy-certificate from the certificate stored in the Firefox credential store.<br /> <br /> == Download and upload data ==<br /> ; Interactive browsing and manipulation of single files<br /> : SweStore is accessible in your web browser in two ways, as a directory index interface at https://webdav.swestore.se/ and with an interactive file manager at https://webdav.swestore.se/browser/. To browse private data you must first install your certificate in your browser (see above). Projects are organized under the &lt;code&gt;/snic&lt;/code&gt; directory as &lt;code&gt;&lt;nowiki&gt;https://webdav.swestore.se/snic/YOUR_PROJECT_NAME/&lt;/nowiki&gt;&lt;/code&gt;.<br /> ; Upload and delete data interactively or with automation<br /> There are several tools that are capable of using the protocols provided by SweStore national storage.<br /> For interactive usage on SNIC clusters we recommend using the ARC tools which should be installed on all SNIC resources.<br /> As an integration point for building scripts and automated systems we suggest using the curl program and library.<br /> : Use the ARC client. Please see the instructions for [[Accessing SweStore national storage with the ARC client]].<br /> : Use lftp. Please see the instructions for [[Accessing SweStore national storage with lftp]].<br /> : Use cURL. Please see the instructions for [[Accessing SweStore national storage with cURL]].<br /> : Use globus-url-copy. Please see the instructions for [[Accessing SweStore national storage with globus-url-copy]].<br /> <br /> == More information ==<br /> * [http://status.swestore.se/munin/monitor/monitor/ Per Project Monitoring of Swestore usage]<br /> <br /> If you have any issues using SweStore please do not hesitate to contact [mailto:swestore-support@snic.vr.se swestore-support].<br /> <br /> == Tools and scripts ==<br /> <br /> There exists a number of tools and utilities developed externally that can be useful. Here are some links:<br /> <br /> * [https://github.com/samuell/arc_tools ARC_Tools] - Convenience scripts for the arc client (Only a recursive rmdir so far).<br /> * [http://sourceforge.net/projects/arc-gui-clients ARC Graphical Clients] - Contains the ARC Storage Explorer (SweStore supported development).<br /> * Transfer script, [http://snicdocs.nsc.liu.se/wiki/SweStore/swstrans_arc swetrans_arc], provided by Adam Peplinski / Philipp Schlatter<br /> * [http://www.nordugrid.org/documents/SWIG-wrapped-ARC-Python-API.pdf Documentation of the ARC Python API (PDF)]<br /> <br /> == Slides and more ==<br /> <br /> [http://docs.snic.se/wiki/Swestore/Lund_Seminar_Apr18 Slides and material from seminar for Lund users on April 18th]<br /> <br /> = Center storage =<br /> Centre storage, as defined by the SNIC storage group, is a storage solution that lives independently of the computational resources and can be accessed from all such resources at a centre. Key features include the ability to access the same filesystem the same way on all computational resources at a centre, and a unified structure and nomenclature for all centra. Unlike cluster storage which is tightly associated with a single cluster, and thus has a limited life-time, centre storage does not require the users to migrate their own data when clusters are decommissioned, not even when the storage hardware itself is being replaced.<br /> <br /> == Unified environment ==<br /> To make the usage more transparent for SNIC users, a set of environment variables are available on all SNIC resources:<br /> <br /> * &lt;code&gt;SNIC_BACKUP&lt;/code&gt; – the user's primary directory at the centre&lt;br&gt;(the part of the centre storage that is backed up)<br /> * &lt;code&gt;SNIC_NOBACKUP&lt;/code&gt; – recommended directory for project storage without backup&lt;br&gt;(also on the centre storage)<br /> * &lt;code&gt;SNIC_TMP&lt;/code&gt; – recommended directory for best performance during a job&lt;br&gt;(local disk on nodes if applicable)</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=Swestore-dCache&diff=4900 2013-04-19T07:41:14Z

<p>Jonas Lindemann (LUNARC): </p> <hr /> <div>[[Category:Storage]]<br /> [[Category:SweStore]]<br /> SNIC is building a storage infrastructure to complement the computational resources.<br /> <br /> Many forms of automated measurements can produce large amounts of data. In scientific areas such as high energy physics (the Large Hadron Collider at CERN), climate modeling, bioinformatics, bioimaging etc., the demands for storage are increasing dramatically. To serve these and other user communities, SNIC has appointed a working group to design a storage strategy, taking into account the needs on many levels and creating a unified storage infrastructure, which is now being implemented.<br /> <br /> Swestore is in collaboration with [http://www.ecds.se/ ECDS], [http://snd.gu.se/ SND], Bioimage Sweden, [http://www.bils.se/ BILS], [http://www.uppnex.uu.se/ UPPNEX],[http://wlcg.web.cern.ch/ WLCG], [http://www.nrm.se/ NaturHistoriska RiksMuseet].<br /> <br /> = National storage =<br /> The Swestore Nationally Accessible Storage, commonly called just Swestore, is a robust, flexible and expandable long<br /> term storage system aimed at storing large amounts of data produced by various Swedish research projects. It is based on the [http://www.dcache.org dCache]<br /> storage system and is distributed across the SNIC centres [http://www.c3se.chalmers.se/ C3SE], [http://www.hpc2n.umu.se/ HPC2N], [http://www.lunarc.lu.se/ Lunarc],<br /> [http://www.nsc.liu.se/ NSC], [http://www.pdc.kth.se PDC] and [http://www.uppmax.uu.se Uppmax].<br /> <br /> Data is stored in two copies with each copy at a different SNIC centre. This enables the system to cope with a multitude of issues ranging from a simple<br /> crash of a storage element to losing an entire site while stil providing access to the stored data. To protect against silent data corruption the<br /> dCache storage system checksums all stored data and periodically verifies the data using this checksum.<br /> <br /> The system does NOT yet provide protection against user errors like inadvertent file deletions and so on.<br /> <br /> One of the major advantages to the distributed nature of dCache is the excellent aggregated transfer rates possible. This is achieved by bypassing a central node<br /> and having transfers going directly to/from the storage elements if the protocol allows it.<br /> The Swestore Nationally Accessible Storage system can achieve aggregated transfer rates<br /> in excess of 100 Gigabit per second, but in practice this is limited by connectivity to each University (usually 10 Gbit/s) or a limited number of files (typically<br /> max 1 Gbit/s per file/connection).<br /> <br /> ==Access protocols==<br /> ; Currently supported protocols<br /> : GridFTP - gsiftp://gsiftp.swestore.se/<br /> : Storage Resource Manager - srm://srm.swegrid.se/<br /> : Hypertext Transfer Protocol (read-only), Web Distributed Authoring and Versioning - http://webdav.swestore.se/ (unauthenticated), https://webdav.swestore.se/<br /> <br /> ; Protocols in evaluation/development<br /> : NFS4.1, iRODS<br /> <br /> For most of the access protocols the form of authentication is not username/password but X.509 client certificates, typically acquired from TCS eScience.<br /> <br /> == Getting access ==<br /> ; Apply for storage<br /> : Please follow instructions [[Apply for storage on SweStore|here]]<br /> ; Get a client certificate.<br /> : Follow the instructions [[Grid_certificates#Requesting_a_certificate|here]] to get your client certificate. For Terena certificates, please make sure you also [[Exporting_a_client_certificate|export the certificate for use with grid tools]]. For Nordugrid certificates, please make sure to also [[Requesting_a_grid_certificate_from_the_Nordugrid_CA#Installing_the_certificate_in_your_browser|install your client certificate in your browser]].<br /> ; Request membership in the SweGrid VO.<br /> : Follow the instructions [[Grid_certificates#Requesting_membership_in_the_SweGrid_VO|here]] to get added to the SweGrid virtual organisation.<br /> ; Transmit and prepare the certificate.<br /> : In order to use the client certificate on SNIC resources for generating proxy certificates and using command line tools, the certificate needs to be [[Preparing_a_client_certificate|converted into PEM files]] on the target cluster if not already in that format.<br /> <br /> == Download and upload data ==<br /> ; Interactive browsing and manipulation of single files<br /> : SweStore is accessible in your web browser in two ways, as a directory index interface at https://webdav.swestore.se/ and with an interactive file manager at https://webdav.swestore.se/browser/. To browse private data you must first install your certificate in your browser (see above). Projects are organized under the &lt;code&gt;/snic&lt;/code&gt; directory as &lt;code&gt;&lt;nowiki&gt;https://webdav.swestore.se/snic/YOUR_PROJECT_NAME/&lt;/nowiki&gt;&lt;/code&gt;.<br /> ; Upload and delete data interactively or with automation<br /> There are several tools that are capable of using the protocols provided by SweStore national storage.<br /> For interactive usage on SNIC clusters we recommend using the ARC tools which should be installed on all SNIC resources.<br /> As an integration point for building scripts and automated systems we suggest using the curl program and library.<br /> : Use the ARC client. Please see the instructions for [[Accessing SweStore national storage with the ARC client]].<br /> : Use lftp. Please see the instructions for [[Accessing SweStore national storage with lftp]].<br /> : Use cURL. Please see the instructions for [[Accessing SweStore national storage with cURL]].<br /> : Use globus-url-copy. Please see the instructions for [[Accessing SweStore national storage with globus-url-copy]].<br /> <br /> == More information ==<br /> * [http://status.swestore.se/munin/monitor/monitor/ Per Project Monitoring of Swestore usage]<br /> * [[Accessing SweStore national storage with the ARC client]]<br /> &lt;!-- * [[Mounting SweStore national storage via WebDAV|Mounting SweStore national storage via WebDAV (Not recomendated at the moment)]] --&gt;<br /> If you have any issues using SweStore please do not hesitate to contact [mailto:swestore-support@snic.vr.se swestore-support].<br /> <br /> == Tools and scripts ==<br /> <br /> There exists a number of tools and utilities developed externally that can be useful. Here are some links:<br /> <br /> * [https://github.com/samuell/arc_tools ARC_Tools] - Convenience scripts for the arc client (Only a recursive rmdir so far).<br /> * [http://sourceforge.net/projects/arc-gui-clients ARC Graphical Clients] - Contains the ARC Storage Explorer (SweStore supported development).<br /> * Transfer script, [http://snicdocs.nsc.liu.se/wiki/SweStore/swstrans_arc swetrans_arc], provided by Adam Peplinski / Philipp Schlatter<br /> * [http://www.nordugrid.org/documents/SWIG-wrapped-ARC-Python-API.pdf Documentation of the ARC Python API (PDF)]<br /> <br /> == Slides and more ==<br /> <br /> [http://docs.snic.se/wiki/Swestore/Lund_Seminar_Apr18 Slides and material from seminar for Lund users on April 18th]<br /> <br /> = Center storage =<br /> Centre storage, as defined by the SNIC storage group, is a storage solution that lives independently of the computational resources and can be accessed from all such resources at a centre. Key features include the ability to access the same filesystem the same way on all computational resources at a centre, and a unified structure and nomenclature for all centra. Unlike cluster storage which is tightly associated with a single cluster, and thus has a limited life-time, centre storage does not require the users to migrate their own data when clusters are decommissioned, not even when the storage hardware itself is being replaced.<br /> <br /> == Unified environment ==<br /> To make the usage more transparent for SNIC users, a set of environment variables are available on all SNIC resources:<br /> <br /> * &lt;code&gt;SNIC_BACKUP&lt;/code&gt; – the user's primary directory at the centre&lt;br&gt;(the part of the centre storage that is backed up)<br /> * &lt;code&gt;SNIC_NOBACKUP&lt;/code&gt; – recommended directory for project storage without backup&lt;br&gt;(also on the centre storage)<br /> * &lt;code&gt;SNIC_TMP&lt;/code&gt; – recommended directory for best performance during a job&lt;br&gt;(local disk on nodes if applicable)</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=Swestore/Lund_Seminar_Apr18&diff=4897 2013-04-18T21:37:29Z

<p>Jonas Lindemann (LUNARC): /* Links */</p> <hr /> <div>== Slides ==<br /> <br /> [[File:Swestore_slides_sem_apr15.pdf]]<br /> <br /> == Links ==<br /> <br /> [http://docs.snic.se/wiki/Swestore General information on SweStore]<br /> <br /> [http://docs.snic.se/wiki/Apply_for_storage_on_SweStore Applying for storage]<br /> <br /> [http://docs.snic.se/wiki/Grid_certificates#Requesting_a_certificate Applying for certificate]<br /> <br /> [http://download.nordugrid.org/repos-13.02.html Installing NorduGrid Client]<br /> <br /> [http://youtu.be/I-tNFvSIaRU Video - Generating a Terena certificate]<br /> <br /> [http://youtu.be/E9D24PZDK_k Video – Exporting Terena Certificate - Part 1]<br /> <br /> [http://youtu.be/jDq774WeF_Y Video – Exporting Terena Certificate - Part 2]<br /> <br /> [http://arc-gui-clients.sourceforge.net/ ARC Storage Explorer]<br /> <br /> == proxy_transfer scripts ==<br /> <br /> The following script can be used in conjunction with the installed proxy_use script (installed on Platon and Alarik)<br /> <br /> Usage:<br /> <br /> &lt;pre&gt;<br /> proxy_upload [hostname] [username]<br /> &lt;/pre&gt;<br /> <br /> This generates a proxy certificates on your local machine and transfers it to the temp-directory on the remote resource using a unique filename.<br /> <br /> On the remote machine:<br /> <br /> &lt;pre&gt;<br /> proxy_use<br /> &lt;/pre&gt;<br /> <br /> This command will look in the /tmp dir for uploaded proxy_certs for your username and rename the file to the standard ARC proxy format.<br /> <br /> == proxy_upload (local machine) ==<br /> <br /> &lt;pre&gt;#!/bin/bash<br /> <br /> if [ $# -ne 2 ]<br /> then<br /> echo &quot;Usage: `basename $0` hostname username&quot;<br /> exit $E_BADARGS<br /> fi<br /> <br /> proxyPath=/tmp/x509up_u$UID<br /> <br /> echo &quot;Generating proxy certificate.&quot;<br /> arcproxy --proxy=$proxyPath<br /> echo<br /> <br /> if [ -e $proxyPath ] ; then<br /> echo &quot;Found generated proxy certificate : $proxyPath&quot;<br /> else<br /> echo &quot;Could not find any proxy certificate.&quot;<br /> return -1<br /> fi<br /> <br /> uuid=`uuidgen`<br /> <br /> remoteProxyPath=/tmp/x509_up_$2_$uuid<br /> <br /> echo<br /> echo &quot;Uploading proxy certificate to $1.&quot;<br /> scp -p -q $proxyPath $2@$1:$remoteProxyPath<br /> <br /> echo<br /> echo &quot;-------------------------------------------------------------&quot;<br /> echo &quot;To use the uploaded proxy on $1, issue the&quot;<br /> echo &quot;following command:&quot;<br /> echo<br /> echo &quot;proxy_use&quot;<br /> echo &quot;-------------------------------------------------------------&quot;<br /> echo&lt;/pre&gt;<br /> <br /> == proxy_use (on remote machine) ==<br /> <br /> Available on Platon and Alarik<br /> <br /> &lt;pre&gt;<br /> #!/bin/env python<br /> <br /> import sys, os<br /> <br /> from datetime import datetime<br /> <br /> tempDir = &quot;/tmp&quot;<br /> <br /> def findProxyFiles():<br /> userName = os.environ[&quot;LOGNAME&quot;]<br /> allFiles = os.listdir(tempDir)<br /> proxyFiles = []<br /> <br /> for dirEntry in allFiles:<br /> fullPath = os.path.join(tempDir, dirEntry)<br /> if os.path.isfile(fullPath):<br /> if fullPath.find(&quot;x509_up_%s&quot; % userName)!=-1:<br /> proxyFiles.append(fullPath)<br /> <br /> return proxyFiles<br /> <br /> <br /> <br /> if __name__ == &quot;__main__&quot;:<br /> <br /> stdProxyFilename = os.path.join(tempDir, &quot;x509up_u%s&quot; % os.getuid())<br /> proxyCertExists = False<br /> <br /> if os.path.isfile(stdProxyFilename):<br /> print(&quot;Proxy certificate %s exists.&quot; % stdProxyFilename)<br /> proxyCertExists = True<br /> else:<br /> print(&quot;No existing proxy certificate %s found. &quot; % stdProxyFilename)<br /> <br /> proxyFiles = findProxyFiles()<br /> proxyDict = {}<br /> <br /> for proxyFilename in proxyFiles:<br /> info = os.stat(proxyFilename)<br /> proxyDict[info.st_ctime] = proxyFilename<br /> <br /> sortedProxyKeys = proxyDict.keys()<br /> sortedProxyKeys.sort()<br /> sortedProxyKeys.reverse()<br /> <br /> if proxyCertExists:<br /> proxyCount = 1<br /> else:<br /> proxyCount = 0<br /> <br /> for timeStamp in sortedProxyKeys:<br /> if (proxyCount == 0):<br /> datetime = datetime.fromtimestamp(timeStamp)<br /> os.rename(proxyDict[timeStamp], stdProxyFilename)<br /> print(&quot;Created %s from uploaded proxy %s.&quot; % (stdProxyFilename, proxyDict[timeStamp]))<br /> else:<br /> print(&quot;Removing old uploaded proxy %s.&quot; % proxyDict[timeStamp])<br /> os.remove(proxyDict[timeStamp])<br /> proxyCount += 1<br /> <br /> if len(proxyFiles) == 0:<br /> print(&quot;No uploaded proxy files found. Please upload proxy files using proxy_upload.&quot;)<br /> &lt;/pre&gt;</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=Swestore/Lund_Seminar_Apr18&diff=4896 2013-04-18T21:35:44Z

<p>Jonas Lindemann (LUNARC): /* proxy_use */</p> <hr /> <div>== Slides ==<br /> <br /> [[File:Swestore_slides_sem_apr15.pdf]]<br /> <br /> == Links ==<br /> <br /> [http://docs.snic.se/wiki/Swestore General information on SweStore]<br /> <br /> [http://docs.snic.se/wiki/Apply_for_storage_on_SweStore Applying for storage]<br /> <br /> [http://docs.snic.se/wiki/Grid_certificates#Requesting_a_certificate Applying for certificate]<br /> <br /> [http://download.nordugrid.org/repos-13.02.html Installing NorduGrid Client]<br /> <br /> == proxy_transfer scripts ==<br /> <br /> The following script can be used in conjunction with the installed proxy_use script (installed on Platon and Alarik)<br /> <br /> Usage:<br /> <br /> &lt;pre&gt;<br /> proxy_upload [hostname] [username]<br /> &lt;/pre&gt;<br /> <br /> This generates a proxy certificates on your local machine and transfers it to the temp-directory on the remote resource using a unique filename.<br /> <br /> On the remote machine:<br /> <br /> &lt;pre&gt;<br /> proxy_use<br /> &lt;/pre&gt;<br /> <br /> This command will look in the /tmp dir for uploaded proxy_certs for your username and rename the file to the standard ARC proxy format.<br /> <br /> == proxy_upload (local machine) ==<br /> <br /> &lt;pre&gt;#!/bin/bash<br /> <br /> if [ $# -ne 2 ]<br /> then<br /> echo &quot;Usage: `basename $0` hostname username&quot;<br /> exit $E_BADARGS<br /> fi<br /> <br /> proxyPath=/tmp/x509up_u$UID<br /> <br /> echo &quot;Generating proxy certificate.&quot;<br /> arcproxy --proxy=$proxyPath<br /> echo<br /> <br /> if [ -e $proxyPath ] ; then<br /> echo &quot;Found generated proxy certificate : $proxyPath&quot;<br /> else<br /> echo &quot;Could not find any proxy certificate.&quot;<br /> return -1<br /> fi<br /> <br /> uuid=`uuidgen`<br /> <br /> remoteProxyPath=/tmp/x509_up_$2_$uuid<br /> <br /> echo<br /> echo &quot;Uploading proxy certificate to $1.&quot;<br /> scp -p -q $proxyPath $2@$1:$remoteProxyPath<br /> <br /> echo<br /> echo &quot;-------------------------------------------------------------&quot;<br /> echo &quot;To use the uploaded proxy on $1, issue the&quot;<br /> echo &quot;following command:&quot;<br /> echo<br /> echo &quot;proxy_use&quot;<br /> echo &quot;-------------------------------------------------------------&quot;<br /> echo&lt;/pre&gt;<br /> <br /> == proxy_use (on remote machine) ==<br /> <br /> Available on Platon and Alarik<br /> <br /> &lt;pre&gt;<br /> #!/bin/env python<br /> <br /> import sys, os<br /> <br /> from datetime import datetime<br /> <br /> tempDir = &quot;/tmp&quot;<br /> <br /> def findProxyFiles():<br /> userName = os.environ[&quot;LOGNAME&quot;]<br /> allFiles = os.listdir(tempDir)<br /> proxyFiles = []<br /> <br /> for dirEntry in allFiles:<br /> fullPath = os.path.join(tempDir, dirEntry)<br /> if os.path.isfile(fullPath):<br /> if fullPath.find(&quot;x509_up_%s&quot; % userName)!=-1:<br /> proxyFiles.append(fullPath)<br /> <br /> return proxyFiles<br /> <br /> <br /> <br /> if __name__ == &quot;__main__&quot;:<br /> <br /> stdProxyFilename = os.path.join(tempDir, &quot;x509up_u%s&quot; % os.getuid())<br /> proxyCertExists = False<br /> <br /> if os.path.isfile(stdProxyFilename):<br /> print(&quot;Proxy certificate %s exists.&quot; % stdProxyFilename)<br /> proxyCertExists = True<br /> else:<br /> print(&quot;No existing proxy certificate %s found. &quot; % stdProxyFilename)<br /> <br /> proxyFiles = findProxyFiles()<br /> proxyDict = {}<br /> <br /> for proxyFilename in proxyFiles:<br /> info = os.stat(proxyFilename)<br /> proxyDict[info.st_ctime] = proxyFilename<br /> <br /> sortedProxyKeys = proxyDict.keys()<br /> sortedProxyKeys.sort()<br /> sortedProxyKeys.reverse()<br /> <br /> if proxyCertExists:<br /> proxyCount = 1<br /> else:<br /> proxyCount = 0<br /> <br /> for timeStamp in sortedProxyKeys:<br /> if (proxyCount == 0):<br /> datetime = datetime.fromtimestamp(timeStamp)<br /> os.rename(proxyDict[timeStamp], stdProxyFilename)<br /> print(&quot;Created %s from uploaded proxy %s.&quot; % (stdProxyFilename, proxyDict[timeStamp]))<br /> else:<br /> print(&quot;Removing old uploaded proxy %s.&quot; % proxyDict[timeStamp])<br /> os.remove(proxyDict[timeStamp])<br /> proxyCount += 1<br /> <br /> if len(proxyFiles) == 0:<br /> print(&quot;No uploaded proxy files found. Please upload proxy files using proxy_upload.&quot;)<br /> &lt;/pre&gt;</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=Swestore/Lund_Seminar_Apr18&diff=4895 2013-04-18T21:35:20Z

<p>Jonas Lindemann (LUNARC): /* proxy_upload */</p> <hr /> <div>== Slides ==<br /> <br /> [[File:Swestore_slides_sem_apr15.pdf]]<br /> <br /> == Links ==<br /> <br /> [http://docs.snic.se/wiki/Swestore General information on SweStore]<br /> <br /> [http://docs.snic.se/wiki/Apply_for_storage_on_SweStore Applying for storage]<br /> <br /> [http://docs.snic.se/wiki/Grid_certificates#Requesting_a_certificate Applying for certificate]<br /> <br /> [http://download.nordugrid.org/repos-13.02.html Installing NorduGrid Client]<br /> <br /> == proxy_transfer scripts ==<br /> <br /> The following script can be used in conjunction with the installed proxy_use script (installed on Platon and Alarik)<br /> <br /> Usage:<br /> <br /> &lt;pre&gt;<br /> proxy_upload [hostname] [username]<br /> &lt;/pre&gt;<br /> <br /> This generates a proxy certificates on your local machine and transfers it to the temp-directory on the remote resource using a unique filename.<br /> <br /> On the remote machine:<br /> <br /> &lt;pre&gt;<br /> proxy_use<br /> &lt;/pre&gt;<br /> <br /> This command will look in the /tmp dir for uploaded proxy_certs for your username and rename the file to the standard ARC proxy format.<br /> <br /> == proxy_upload (local machine) ==<br /> <br /> &lt;pre&gt;#!/bin/bash<br /> <br /> if [ $# -ne 2 ]<br /> then<br /> echo &quot;Usage: `basename $0` hostname username&quot;<br /> exit $E_BADARGS<br /> fi<br /> <br /> proxyPath=/tmp/x509up_u$UID<br /> <br /> echo &quot;Generating proxy certificate.&quot;<br /> arcproxy --proxy=$proxyPath<br /> echo<br /> <br /> if [ -e $proxyPath ] ; then<br /> echo &quot;Found generated proxy certificate : $proxyPath&quot;<br /> else<br /> echo &quot;Could not find any proxy certificate.&quot;<br /> return -1<br /> fi<br /> <br /> uuid=`uuidgen`<br /> <br /> remoteProxyPath=/tmp/x509_up_$2_$uuid<br /> <br /> echo<br /> echo &quot;Uploading proxy certificate to $1.&quot;<br /> scp -p -q $proxyPath $2@$1:$remoteProxyPath<br /> <br /> echo<br /> echo &quot;-------------------------------------------------------------&quot;<br /> echo &quot;To use the uploaded proxy on $1, issue the&quot;<br /> echo &quot;following command:&quot;<br /> echo<br /> echo &quot;proxy_use&quot;<br /> echo &quot;-------------------------------------------------------------&quot;<br /> echo&lt;/pre&gt;<br /> <br /> == proxy_use ==<br /> <br /> &lt;pre&gt;<br /> #!/bin/env python<br /> <br /> import sys, os<br /> <br /> from datetime import datetime<br /> <br /> tempDir = &quot;/tmp&quot;<br /> <br /> def findProxyFiles():<br /> userName = os.environ[&quot;LOGNAME&quot;]<br /> allFiles = os.listdir(tempDir)<br /> proxyFiles = []<br /> <br /> for dirEntry in allFiles:<br /> fullPath = os.path.join(tempDir, dirEntry)<br /> if os.path.isfile(fullPath):<br /> if fullPath.find(&quot;x509_up_%s&quot; % userName)!=-1:<br /> proxyFiles.append(fullPath)<br /> <br /> return proxyFiles<br /> <br /> <br /> <br /> if __name__ == &quot;__main__&quot;:<br /> <br /> stdProxyFilename = os.path.join(tempDir, &quot;x509up_u%s&quot; % os.getuid())<br /> proxyCertExists = False<br /> <br /> if os.path.isfile(stdProxyFilename):<br /> print(&quot;Proxy certificate %s exists.&quot; % stdProxyFilename)<br /> proxyCertExists = True<br /> else:<br /> print(&quot;No existing proxy certificate %s found. &quot; % stdProxyFilename)<br /> <br /> proxyFiles = findProxyFiles()<br /> proxyDict = {}<br /> <br /> for proxyFilename in proxyFiles:<br /> info = os.stat(proxyFilename)<br /> proxyDict[info.st_ctime] = proxyFilename<br /> <br /> sortedProxyKeys = proxyDict.keys()<br /> sortedProxyKeys.sort()<br /> sortedProxyKeys.reverse()<br /> <br /> if proxyCertExists:<br /> proxyCount = 1<br /> else:<br /> proxyCount = 0<br /> <br /> for timeStamp in sortedProxyKeys:<br /> if (proxyCount == 0):<br /> datetime = datetime.fromtimestamp(timeStamp)<br /> os.rename(proxyDict[timeStamp], stdProxyFilename)<br /> print(&quot;Created %s from uploaded proxy %s.&quot; % (stdProxyFilename, proxyDict[timeStamp]))<br /> else:<br /> print(&quot;Removing old uploaded proxy %s.&quot; % proxyDict[timeStamp])<br /> os.remove(proxyDict[timeStamp])<br /> proxyCount += 1<br /> <br /> if len(proxyFiles) == 0:<br /> print(&quot;No uploaded proxy files found. Please upload proxy files using proxy_upload.&quot;)<br /> &lt;/pre&gt;</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=Swestore/Lund_Seminar_Apr18&diff=4894 2013-04-18T21:34:49Z

<p>Jonas Lindemann (LUNARC): /* proxy_upload script */</p> <hr /> <div>== Slides ==<br /> <br /> [[File:Swestore_slides_sem_apr15.pdf]]<br /> <br /> == Links ==<br /> <br /> [http://docs.snic.se/wiki/Swestore General information on SweStore]<br /> <br /> [http://docs.snic.se/wiki/Apply_for_storage_on_SweStore Applying for storage]<br /> <br /> [http://docs.snic.se/wiki/Grid_certificates#Requesting_a_certificate Applying for certificate]<br /> <br /> [http://download.nordugrid.org/repos-13.02.html Installing NorduGrid Client]<br /> <br /> == proxy_transfer scripts ==<br /> <br /> The following script can be used in conjunction with the installed proxy_use script (installed on Platon and Alarik)<br /> <br /> Usage:<br /> <br /> &lt;pre&gt;<br /> proxy_upload [hostname] [username]<br /> &lt;/pre&gt;<br /> <br /> This generates a proxy certificates on your local machine and transfers it to the temp-directory on the remote resource using a unique filename.<br /> <br /> On the remote machine:<br /> <br /> &lt;pre&gt;<br /> proxy_use<br /> &lt;/pre&gt;<br /> <br /> This command will look in the /tmp dir for uploaded proxy_certs for your username and rename the file to the standard ARC proxy format.<br /> <br /> == proxy_upload ==<br /> <br /> &lt;pre&gt;#!/bin/bash<br /> <br /> if [ $# -ne 2 ]<br /> then<br /> echo &quot;Usage: `basename $0` hostname username&quot;<br /> exit $E_BADARGS<br /> fi<br /> <br /> proxyPath=/tmp/x509up_u$UID<br /> <br /> echo &quot;Generating proxy certificate.&quot;<br /> arcproxy --proxy=$proxyPath<br /> echo<br /> <br /> if [ -e $proxyPath ] ; then<br /> echo &quot;Found generated proxy certificate : $proxyPath&quot;<br /> else<br /> echo &quot;Could not find any proxy certificate.&quot;<br /> return -1<br /> fi<br /> <br /> uuid=`uuidgen`<br /> <br /> remoteProxyPath=/tmp/x509_up_$2_$uuid<br /> <br /> echo<br /> echo &quot;Uploading proxy certificate to $1.&quot;<br /> scp -p -q $proxyPath $2@$1:$remoteProxyPath<br /> <br /> echo<br /> echo &quot;-------------------------------------------------------------&quot;<br /> echo &quot;To use the uploaded proxy on $1, issue the&quot;<br /> echo &quot;following command:&quot;<br /> echo<br /> echo &quot;proxy_use&quot;<br /> echo &quot;-------------------------------------------------------------&quot;<br /> echo&lt;/pre&gt;<br /> <br /> == proxy_use ==<br /> <br /> &lt;pre&gt;<br /> #!/bin/env python<br /> <br /> import sys, os<br /> <br /> from datetime import datetime<br /> <br /> tempDir = &quot;/tmp&quot;<br /> <br /> def findProxyFiles():<br /> userName = os.environ[&quot;LOGNAME&quot;]<br /> allFiles = os.listdir(tempDir)<br /> proxyFiles = []<br /> <br /> for dirEntry in allFiles:<br /> fullPath = os.path.join(tempDir, dirEntry)<br /> if os.path.isfile(fullPath):<br /> if fullPath.find(&quot;x509_up_%s&quot; % userName)!=-1:<br /> proxyFiles.append(fullPath)<br /> <br /> return proxyFiles<br /> <br /> <br /> <br /> if __name__ == &quot;__main__&quot;:<br /> <br /> stdProxyFilename = os.path.join(tempDir, &quot;x509up_u%s&quot; % os.getuid())<br /> proxyCertExists = False<br /> <br /> if os.path.isfile(stdProxyFilename):<br /> print(&quot;Proxy certificate %s exists.&quot; % stdProxyFilename)<br /> proxyCertExists = True<br /> else:<br /> print(&quot;No existing proxy certificate %s found. &quot; % stdProxyFilename)<br /> <br /> proxyFiles = findProxyFiles()<br /> proxyDict = {}<br /> <br /> for proxyFilename in proxyFiles:<br /> info = os.stat(proxyFilename)<br /> proxyDict[info.st_ctime] = proxyFilename<br /> <br /> sortedProxyKeys = proxyDict.keys()<br /> sortedProxyKeys.sort()<br /> sortedProxyKeys.reverse()<br /> <br /> if proxyCertExists:<br /> proxyCount = 1<br /> else:<br /> proxyCount = 0<br /> <br /> for timeStamp in sortedProxyKeys:<br /> if (proxyCount == 0):<br /> datetime = datetime.fromtimestamp(timeStamp)<br /> os.rename(proxyDict[timeStamp], stdProxyFilename)<br /> print(&quot;Created %s from uploaded proxy %s.&quot; % (stdProxyFilename, proxyDict[timeStamp]))<br /> else:<br /> print(&quot;Removing old uploaded proxy %s.&quot; % proxyDict[timeStamp])<br /> os.remove(proxyDict[timeStamp])<br /> proxyCount += 1<br /> <br /> if len(proxyFiles) == 0:<br /> print(&quot;No uploaded proxy files found. Please upload proxy files using proxy_upload.&quot;)<br /> &lt;/pre&gt;</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=Swestore/Lund_Seminar_Apr18&diff=4893 2013-04-18T21:29:21Z

<p>Jonas Lindemann (LUNARC): /* proxy_upload script */</p> <hr /> <div>== Slides ==<br /> <br /> [[File:Swestore_slides_sem_apr15.pdf]]<br /> <br /> == Links ==<br /> <br /> [http://docs.snic.se/wiki/Swestore General information on SweStore]<br /> <br /> [http://docs.snic.se/wiki/Apply_for_storage_on_SweStore Applying for storage]<br /> <br /> [http://docs.snic.se/wiki/Grid_certificates#Requesting_a_certificate Applying for certificate]<br /> <br /> [http://download.nordugrid.org/repos-13.02.html Installing NorduGrid Client]<br /> <br /> == proxy_upload script ==<br /> <br /> &lt;pre&gt;#!/bin/bash<br /> <br /> if [ $# -ne 2 ]<br /> then<br /> echo &quot;Usage: `basename $0` hostname username&quot;<br /> exit $E_BADARGS<br /> fi<br /> <br /> proxyPath=/tmp/x509up_u$UID<br /> <br /> echo &quot;Generating proxy certificate.&quot;<br /> arcproxy --proxy=$proxyPath<br /> echo<br /> <br /> if [ -e $proxyPath ] ; then<br /> echo &quot;Found generated proxy certificate : $proxyPath&quot;<br /> else<br /> echo &quot;Could not find any proxy certificate.&quot;<br /> return -1<br /> fi<br /> <br /> uuid=`uuidgen`<br /> <br /> remoteProxyPath=/tmp/x509_up_$2_$uuid<br /> <br /> echo<br /> echo &quot;Uploading proxy certificate to $1.&quot;<br /> scp -p -q $proxyPath $2@$1:$remoteProxyPath<br /> <br /> echo<br /> echo &quot;-------------------------------------------------------------&quot;<br /> echo &quot;To use the uploaded proxy on $1, issue the&quot;<br /> echo &quot;following command:&quot;<br /> echo<br /> echo &quot;proxy_use&quot;<br /> echo &quot;-------------------------------------------------------------&quot;<br /> echo&lt;/pre&gt;</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=Swestore/Lund_Seminar_Apr18&diff=4892 2013-04-18T21:29:01Z

<p>Jonas Lindemann (LUNARC): /* proxy_transfer package */</p> <hr /> <div>== Slides ==<br /> <br /> [[File:Swestore_slides_sem_apr15.pdf]]<br /> <br /> == Links ==<br /> <br /> [http://docs.snic.se/wiki/Swestore General information on SweStore]<br /> <br /> [http://docs.snic.se/wiki/Apply_for_storage_on_SweStore Applying for storage]<br /> <br /> [http://docs.snic.se/wiki/Grid_certificates#Requesting_a_certificate Applying for certificate]<br /> <br /> [http://download.nordugrid.org/repos-13.02.html Installing NorduGrid Client]<br /> <br /> == proxy_upload script ==<br /> <br /> &lt;nowiki&gt;#!/bin/bash<br /> <br /> if [ $# -ne 2 ]<br /> then<br /> echo &quot;Usage: `basename $0` hostname username&quot;<br /> exit $E_BADARGS<br /> fi<br /> <br /> proxyPath=/tmp/x509up_u$UID<br /> <br /> echo &quot;Generating proxy certificate.&quot;<br /> arcproxy --proxy=$proxyPath<br /> echo<br /> <br /> if [ -e $proxyPath ] ; then<br /> echo &quot;Found generated proxy certificate : $proxyPath&quot;<br /> else<br /> echo &quot;Could not find any proxy certificate.&quot;<br /> return -1<br /> fi<br /> <br /> uuid=`uuidgen`<br /> <br /> remoteProxyPath=/tmp/x509_up_$2_$uuid<br /> <br /> echo<br /> echo &quot;Uploading proxy certificate to $1.&quot;<br /> scp -p -q $proxyPath $2@$1:$remoteProxyPath<br /> <br /> echo<br /> echo &quot;-------------------------------------------------------------&quot;<br /> echo &quot;To use the uploaded proxy on $1, issue the&quot;<br /> echo &quot;following command:&quot;<br /> echo<br /> echo &quot;proxy_use&quot;<br /> echo &quot;-------------------------------------------------------------&quot;<br /> echo&lt;/nowiki&gt;</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=Swestore/Lund_Seminar_Apr18&diff=4891 2013-04-18T21:25:29Z

<p>Jonas Lindemann (LUNARC): /* Links */</p> <hr /> <div>== Slides ==<br /> <br /> [[File:Swestore_slides_sem_apr15.pdf]]<br /> <br /> == Links ==<br /> <br /> [http://docs.snic.se/wiki/Swestore General information on SweStore]<br /> <br /> [http://docs.snic.se/wiki/Apply_for_storage_on_SweStore Applying for storage]<br /> <br /> [http://docs.snic.se/wiki/Grid_certificates#Requesting_a_certificate Applying for certificate]<br /> <br /> [http://download.nordugrid.org/repos-13.02.html Installing NorduGrid Client]<br /> <br /> == proxy_transfer package ==</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=Swestore/Lund_Seminar_Apr18&diff=4890 2013-04-18T21:25:17Z

<p>Jonas Lindemann (LUNARC): /* Links */</p> <hr /> <div>== Slides ==<br /> <br /> [[File:Swestore_slides_sem_apr15.pdf]]<br /> <br /> == Links ==<br /> <br /> [http://docs.snic.se/wiki/Swestore General information on SweStore]<br /> [http://docs.snic.se/wiki/Apply_for_storage_on_SweStore Applying for storage]<br /> [http://docs.snic.se/wiki/Grid_certificates#Requesting_a_certificate Applying for certificate]<br /> [http://download.nordugrid.org/repos-13.02.html Installing NorduGrid Client]<br /> <br /> == proxy_transfer package ==</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=Swestore/Lund_Seminar_Apr18&diff=4889 2013-04-18T21:23:09Z

<p>Jonas Lindemann (LUNARC): /* Slides */</p> <hr /> <div>== Slides ==<br /> <br /> [[File:Swestore_slides_sem_apr15.pdf]]<br /> <br /> == Links ==<br /> <br /> == proxy_transfer package ==</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=Swestore/Lund_Seminar_Apr18&diff=4888 2013-04-18T21:21:21Z

<p>Jonas Lindemann (LUNARC): </p> <hr /> <div>== Slides ==<br /> <br /> [[File:Example.jpg]]<br /> <br /> == Links ==<br /> <br /> == proxy_transfer package ==</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=File:Swestore_slides_sem_apr15.pdf&diff=4887 2013-04-18T21:20:00Z

<p>Jonas Lindemann (LUNARC): </p> <hr /> <div></div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=Swestore/Lund_Seminar_Apr18&diff=4886 2013-04-18T21:15:04Z

<p>Jonas Lindemann (LUNARC): </p> <hr /> <div>== Slides ==<br /> <br /> == Links ==<br /> <br /> == proxy_transfer package ==</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=Swestore/Lund_Seminar_Apr18&diff=4885 2013-04-18T21:14:49Z

<p>Jonas Lindemann (LUNARC): Created page with &quot;= Seminar: Using the national storage system (SweStore) = == Slides == == Links == == proxy_transfer package ==&quot;</p> <hr /> <div>= Seminar: Using the national storage system (SweStore) =<br /> <br /> == Slides ==<br /> <br /> == Links ==<br /> <br /> == proxy_transfer package ==</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=Swestore-dCache&diff=4275 2012-07-30T12:41:41Z

<p>Jonas Lindemann (LUNARC): /* Download and upload data */</p> <hr /> <div>[[Category:Storage]]<br /> [[Category:SweStore]]<br /> SNIC is building a storage infrastructure to complement the computational resources.<br /> <br /> Many forms of automated measurements can produce large amounts of data. In scientific areas such as high energy physics (the Large Hadron Collider at CERN), climate modeling, bioinformatics, bioimaging etc., the demands for storage are increasing dramatically. To serve these and other user communities, SNIC has appointed a working group to design a storage strategy, taking into account the needs on many levels and creating a unified storage infrastructure, which is now being implemented.<br /> <br /> Swestore is in collaboration with [http://www.ecds.se ECDS], [http://snd.gu.se SND], Bioimage Sweden, [http://www.bils.se BILS], [http://www.uppnex.uu.se UPPNEX],[http://http://lcg.web.cern.ch/lcg/public/ WLCG], [http://www.nrm.se/ NaturHistoriska RiksMuseet].<br /> <br /> = National storage =<br /> The aim of the nationally accessible storage is to build a robust, flexible and expandable system that can<br /> be used in most cases where access to large scale storage is needed. To the user it should appear as a single large system,<br /> while it is desirable that some parts of the system are distributed across all SNIC centra to benefit from the advantages<br /> of, among other things, locality and cache effects. The system is intended as a versatile long-term storage system.<br /> <br /> == Getting access ==<br /> ; Apply for storage<br /> : Please follow instructions [[Apply for storage on SweStore|here]]<br /> ; Get a client certificate.<br /> : Follow the instructions [[Grid_certificates#Requesting_a_certificate|here]] to get your client certificate. For Terena certificates, please make sure you also [[Requesting_a_grid_certificate_using_the_Terena_eScience_Portal#Exporting Terena certificate for use with Grid tools|export the certificate for use with grid tools]]. For Nordugrid certificates, please make sure to also [[Requesting_a_grid_certificate_from_the_Nordugrid_CA#Installing_the_certificate_in_your_browser|install your client certificate in your browser]].<br /> ; Request membership in the SweGrid VO.<br /> : Follow the instructions [[Grid_certificates#Requesting_membership_in_the_SweGrid_VO|here]] to get added to the SweGrid virtual organisation.<br /> <br /> == Download and upload data ==<br /> ; Browse and download data<br /> : SweStore is accessible from your web browser, here http://webdav.swegrid.se/. To browse private data you must first install your certificate in your browser (see above). Your data is available at &lt;code&gt;&lt;nowiki&gt;http://webdav.swegrid.se/snic/YOUR_PROJECT_NAME&lt;/nowiki&gt;&lt;/code&gt;.<br /> ; Upload and delete data<br /> : Use the ARC client. Please see the instructions for [[Accessing SweStore national storage with the ARC client]].<br /> : Use cURL. Please see the instructions for [[Accessing SweStore national storage with the cURL]].<br /> : Use lftp. Please see the instructions for [[Accessing SweStore national storage with lftp]].<br /> : Use globus-url-copy. Please see the instructions for [[Accessing SweStore national storage with globus-url-copy]].<br /> <br /> == Examples of storage projects ==<br /> Below are some examples of project that are using SweStore today.<br /> <br /> {|border=&quot;1&quot; style=&quot;text-align:left; border-collapse: collapse; border-width: 1px; border-style: solid; border-color: #000&quot; class=&quot;wikitable sortable&quot; valign=top<br /> !Allocation name<br /> !Size in TB<br /> !class=&quot;unsortable&quot;|Project full name<br /> |-<br /> |alice<br /> |400<br /> |<br /> |-<br /> |uppnex<br /> |140<br /> |[https://www.uppnex.uu.se UPPmax NExt Generation Sequencing Cluster &amp; Storage]<br /> |-<br /> |brain_protein_atlas<br /> |10<br /> |Mouse brain protein atlas project<br /> |-<br /> | scims2lab<br /> |20<br /> | Identification of novel gene models by matching mass spectrometry data against 6-frame translations of the human genome<br /> |-<br /> |subatom<br /> |<br /> |Low-energy nuclear theory and experiment<br /> |-<br /> |genomics-gu<br /> |10<br /> |Genomics Core Facility, Sahlgrenska academy at University of Gothenburg.<br /> |-<br /> |Chemo<br /> |5TB<br /> |Genetic interaction networks in human deseas<br /> |-<br /> |cesm1_holocene<br /> |30<br /> |Arctic sea ice in warm climates<br /> |}<br /> <br /> == More information ==<br /> * [[SweStore introduction]]<br /> * [http://status.swestore.se/munin/monitor/monitor/ Per Project Monitoring of Swestore usage]<br /> * [[Accessing SweStore national storage with the ARC client]]<br /> &lt;!-- * [[Mounting SweStore national storage via WebDAV|Mounting SweStore national storage via WebDAV (Not recomendated at the moment)]] --&gt;<br /> If you have any issues using SweStore please do not hesitate to contact [mailto:swestore-support@snic.vr.se swestore-support].<br /> <br /> == Tools and scripts ==<br /> <br /> There exists a number of tools and utilities developed externally that can be useful. Here are some links:<br /> <br /> * [https://github.com/samuell/arc_tools ARC_Tools] - Convenience scripts for the arc client (Only a recursive rmdir so far).<br /> * [http://sourceforge.net/projects/arc-gui-clients ARC Graphical Clients] - Contains the ARC Storage Explorer (SweStore supported development).<br /> * Transfer script, [http://snicdocs.nsc.liu.se/wiki/SweStore/swstrans_arc swetrans_arc], provided by Adam Peplinski / Philipp Schlatter<br /> * [http://www.nordugrid.org/documents/SWIG-wrapped-ARC-Python-API.pdf Documentation of the ARC Python API (PDF)]<br /> <br /> = Center storage =<br /> Centre storage, as defined by the SNIC storage group, is a storage solution that lives independently of the computational resources and can be accessed from all such resources at a centre. Key features include the ability to access the same filesystem the same way on all computational resources at a centre, and a unified structure and nomenclature for all centra. Unlike cluster storage which is tightly associated with a single cluster, and thus has a limited life-time, centre storage does not require the users to migrate their own data when clusters are decommissioned, not even when the storage hardware itself is being replaced.<br /> <br /> == Unified environment ==<br /> To make the usage more transparent for SNIC users, a set of environment variables are available on all SNIC resources:<br /> <br /> * &lt;code&gt;SNIC_BACKUP&lt;/code&gt; – the user's primary directory at the centre&lt;br&gt;(the part of the centre storage that is backed up)<br /> * &lt;code&gt;SNIC_NOBACKUP&lt;/code&gt; – recommended directory for project storage without backup&lt;br&gt;(also on the centre storage)<br /> * &lt;code&gt;SNIC_TMP&lt;/code&gt; – recommended directory for best performance during a job&lt;br&gt;(local disk on nodes if applicable)</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=Accessing_Swestore_with_globus-url-copy&diff=4274 2012-07-30T12:41:09Z

<p>Jonas Lindemann (LUNARC): </p> <hr /> <div>Copying a single file to a local file:<br /> <br /> globus-url-copy gsiftp://srm.swegrid.se/snic/myproject/myfile file:///home/myname/myfile<br /> <br /> Copying a single file to a directory:<br /> <br /> globus-url-copy gsiftp://srm.swegrid.se/snic/myproject/myfile gsiftp://srm.swegrid.se/snic/myproject/mydir/<br /> <br /> Copying a directory recursively to a local directory (verbose -v):<br /> <br /> globus-url-copy -v -cd -r gsiftp://srm.swegrid.se/snic/myproject/test2/ file:///home/myname/test2/<br /> <br /> Listing files in a directory:<br /> <br /> globus-url-copy -list gsiftp://srm.swegrid.se/snic/myproject/<br /> <br /> P.S. Please note the trailing slashes for directories.</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=Accessing_Swestore_with_globus-url-copy&diff=4273 2012-07-30T12:39:04Z

<p>Jonas Lindemann (LUNARC): Created page with &quot;Copying a single file to a local file: globus-url-copy gsiftp://srm.swegrid.se/ops/jonas/arc-gui-clients-0.2.1.1.tar.gz file:///home/jonas/test.tar.gz Copying a single file to ...&quot;</p> <hr /> <div>Copying a single file to a local file:<br /> <br /> globus-url-copy gsiftp://srm.swegrid.se/ops/jonas/arc-gui-clients-0.2.1.1.tar.gz file:///home/jonas/test.tar.gz<br /> <br /> Copying a single file to a directory:<br /> <br /> globus-url-copy gsiftp://srm.swegrid.se/ops/jonas/arc-gui-clients-0.2.1.1.tar.gz gsiftp://srm.swegrid.se/ops/jonas/test2/<br /> <br /> Copying a directory recursively to a local directory (verbose -v):<br /> <br /> globus-url-copy -v -cd -r gsiftp://srm.swegrid.se/ops/jonas/test2/ file:///home/jonas/test2/<br /> <br /> Listing files in a directory:<br /> <br /> globus-url-copy -list gsiftp://srm.swegrid.se/ops/jonas/<br /> <br /> P.S. Please note the trailing slashes for directories.</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=Swestore-dCache&diff=4272 2012-07-30T12:37:57Z

<p>Jonas Lindemann (LUNARC): /* Download and upload data */</p> <hr /> <div>[[Category:Storage]]<br /> [[Category:SweStore]]<br /> SNIC is building a storage infrastructure to complement the computational resources.<br /> <br /> Many forms of automated measurements can produce large amounts of data. In scientific areas such as high energy physics (the Large Hadron Collider at CERN), climate modeling, bioinformatics, bioimaging etc., the demands for storage are increasing dramatically. To serve these and other user communities, SNIC has appointed a working group to design a storage strategy, taking into account the needs on many levels and creating a unified storage infrastructure, which is now being implemented.<br /> <br /> Swestore is in collaboration with [http://www.ecds.se ECDS], [http://snd.gu.se SND], Bioimage Sweden, [http://www.bils.se BILS], [http://www.uppnex.uu.se UPPNEX],[http://http://lcg.web.cern.ch/lcg/public/ WLCG], [http://www.nrm.se/ NaturHistoriska RiksMuseet].<br /> <br /> = National storage =<br /> The aim of the nationally accessible storage is to build a robust, flexible and expandable system that can<br /> be used in most cases where access to large scale storage is needed. To the user it should appear as a single large system,<br /> while it is desirable that some parts of the system are distributed across all SNIC centra to benefit from the advantages<br /> of, among other things, locality and cache effects. The system is intended as a versatile long-term storage system.<br /> <br /> == Getting access ==<br /> ; Apply for storage<br /> : Please follow instructions [[Apply for storage on SweStore|here]]<br /> ; Get a client certificate.<br /> : Follow the instructions [[Grid_certificates#Requesting_a_certificate|here]] to get your client certificate. For Terena certificates, please make sure you also [[Requesting_a_grid_certificate_using_the_Terena_eScience_Portal#Exporting Terena certificate for use with Grid tools|export the certificate for use with grid tools]]. For Nordugrid certificates, please make sure to also [[Requesting_a_grid_certificate_from_the_Nordugrid_CA#Installing_the_certificate_in_your_browser|install your client certificate in your browser]].<br /> ; Request membership in the SweGrid VO.<br /> : Follow the instructions [[Grid_certificates#Requesting_membership_in_the_SweGrid_VO|here]] to get added to the SweGrid virtual organisation.<br /> <br /> == Download and upload data ==<br /> ; Browse and download data<br /> : SweStore is accessible from your web browser, here http://webdav.swegrid.se/. To browse private data you must first install your certificate in your browser (see above). Your data is available at &lt;code&gt;&lt;nowiki&gt;http://webdav.swegrid.se/snic/YOUR_PROJECT_NAME&lt;/nowiki&gt;&lt;/code&gt;.<br /> ; Upload and delete data<br /> : Use the ARC client. Please see the instructions for [[Accessing SweStore national storage with the ARC client]].<br /> : Use cURL. Please see the instructions for [[Accessing SweStore national storage with the cURL]].<br /> : Use lftp. Please see the instructions for [[Accessing SweStore national storage with lftp]].<br /> <br /> == Examples of storage projects ==<br /> Below are some examples of project that are using SweStore today.<br /> <br /> {|border=&quot;1&quot; style=&quot;text-align:left; border-collapse: collapse; border-width: 1px; border-style: solid; border-color: #000&quot; class=&quot;wikitable sortable&quot; valign=top<br /> !Allocation name<br /> !Size in TB<br /> !class=&quot;unsortable&quot;|Project full name<br /> |-<br /> |alice<br /> |400<br /> |<br /> |-<br /> |uppnex<br /> |140<br /> |[https://www.uppnex.uu.se UPPmax NExt Generation Sequencing Cluster &amp; Storage]<br /> |-<br /> |brain_protein_atlas<br /> |10<br /> |Mouse brain protein atlas project<br /> |-<br /> | scims2lab<br /> |20<br /> | Identification of novel gene models by matching mass spectrometry data against 6-frame translations of the human genome<br /> |-<br /> |subatom<br /> |<br /> |Low-energy nuclear theory and experiment<br /> |-<br /> |genomics-gu<br /> |10<br /> |Genomics Core Facility, Sahlgrenska academy at University of Gothenburg.<br /> |-<br /> |Chemo<br /> |5TB<br /> |Genetic interaction networks in human deseas<br /> |-<br /> |cesm1_holocene<br /> |30<br /> |Arctic sea ice in warm climates<br /> |}<br /> <br /> == More information ==<br /> * [[SweStore introduction]]<br /> * [http://status.swestore.se/munin/monitor/monitor/ Per Project Monitoring of Swestore usage]<br /> * [[Accessing SweStore national storage with the ARC client]]<br /> &lt;!-- * [[Mounting SweStore national storage via WebDAV|Mounting SweStore national storage via WebDAV (Not recomendated at the moment)]] --&gt;<br /> If you have any issues using SweStore please do not hesitate to contact [mailto:swestore-support@snic.vr.se swestore-support].<br /> <br /> == Tools and scripts ==<br /> <br /> There exists a number of tools and utilities developed externally that can be useful. Here are some links:<br /> <br /> * [https://github.com/samuell/arc_tools ARC_Tools] - Convenience scripts for the arc client (Only a recursive rmdir so far).<br /> * [http://sourceforge.net/projects/arc-gui-clients ARC Graphical Clients] - Contains the ARC Storage Explorer (SweStore supported development).<br /> * Transfer script, [http://snicdocs.nsc.liu.se/wiki/SweStore/swstrans_arc swetrans_arc], provided by Adam Peplinski / Philipp Schlatter<br /> * [http://www.nordugrid.org/documents/SWIG-wrapped-ARC-Python-API.pdf Documentation of the ARC Python API (PDF)]<br /> <br /> = Center storage =<br /> Centre storage, as defined by the SNIC storage group, is a storage solution that lives independently of the computational resources and can be accessed from all such resources at a centre. Key features include the ability to access the same filesystem the same way on all computational resources at a centre, and a unified structure and nomenclature for all centra. Unlike cluster storage which is tightly associated with a single cluster, and thus has a limited life-time, centre storage does not require the users to migrate their own data when clusters are decommissioned, not even when the storage hardware itself is being replaced.<br /> <br /> == Unified environment ==<br /> To make the usage more transparent for SNIC users, a set of environment variables are available on all SNIC resources:<br /> <br /> * &lt;code&gt;SNIC_BACKUP&lt;/code&gt; – the user's primary directory at the centre&lt;br&gt;(the part of the centre storage that is backed up)<br /> * &lt;code&gt;SNIC_NOBACKUP&lt;/code&gt; – recommended directory for project storage without backup&lt;br&gt;(also on the centre storage)<br /> * &lt;code&gt;SNIC_TMP&lt;/code&gt; – recommended directory for best performance during a job&lt;br&gt;(local disk on nodes if applicable)</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=Swestore-dCache&diff=4271 2012-07-30T12:37:11Z

<p>Jonas Lindemann (LUNARC): /* Download and upload data */</p> <hr /> <div>[[Category:Storage]]<br /> [[Category:SweStore]]<br /> SNIC is building a storage infrastructure to complement the computational resources.<br /> <br /> Many forms of automated measurements can produce large amounts of data. In scientific areas such as high energy physics (the Large Hadron Collider at CERN), climate modeling, bioinformatics, bioimaging etc., the demands for storage are increasing dramatically. To serve these and other user communities, SNIC has appointed a working group to design a storage strategy, taking into account the needs on many levels and creating a unified storage infrastructure, which is now being implemented.<br /> <br /> Swestore is in collaboration with [http://www.ecds.se ECDS], [http://snd.gu.se SND], Bioimage Sweden, [http://www.bils.se BILS], [http://www.uppnex.uu.se UPPNEX],[http://http://lcg.web.cern.ch/lcg/public/ WLCG], [http://www.nrm.se/ NaturHistoriska RiksMuseet].<br /> <br /> = National storage =<br /> The aim of the nationally accessible storage is to build a robust, flexible and expandable system that can<br /> be used in most cases where access to large scale storage is needed. To the user it should appear as a single large system,<br /> while it is desirable that some parts of the system are distributed across all SNIC centra to benefit from the advantages<br /> of, among other things, locality and cache effects. The system is intended as a versatile long-term storage system.<br /> <br /> == Getting access ==<br /> ; Apply for storage<br /> : Please follow instructions [[Apply for storage on SweStore|here]]<br /> ; Get a client certificate.<br /> : Follow the instructions [[Grid_certificates#Requesting_a_certificate|here]] to get your client certificate. For Terena certificates, please make sure you also [[Requesting_a_grid_certificate_using_the_Terena_eScience_Portal#Exporting Terena certificate for use with Grid tools|export the certificate for use with grid tools]]. For Nordugrid certificates, please make sure to also [[Requesting_a_grid_certificate_from_the_Nordugrid_CA#Installing_the_certificate_in_your_browser|install your client certificate in your browser]].<br /> ; Request membership in the SweGrid VO.<br /> : Follow the instructions [[Grid_certificates#Requesting_membership_in_the_SweGrid_VO|here]] to get added to the SweGrid virtual organisation.<br /> <br /> == Download and upload data ==<br /> ; Browse and download data<br /> : SweStore is accessible from your web browser, here http://webdav.swegrid.se/. To browse private data you must first install your certificate in your browser (see above). Your data is available at &lt;code&gt;&lt;nowiki&gt;http://webdav.swegrid.se/snic/YOUR_PROJECT_NAME&lt;/nowiki&gt;&lt;/code&gt;.<br /> ; Upload and delete data<br /> : Use the ARC client. Please see the instructions for [[Accessing SweStore national storage with the ARC client]].<br /> : Use cURL. Please see the instructions for [[Accessing SweStore national storage with the cURL]].<br /> : Use lftp. Please see the instructions for [[Accessing SweStore national storage with the lftp]].<br /> <br /> == Examples of storage projects ==<br /> Below are some examples of project that are using SweStore today.<br /> <br /> {|border=&quot;1&quot; style=&quot;text-align:left; border-collapse: collapse; border-width: 1px; border-style: solid; border-color: #000&quot; class=&quot;wikitable sortable&quot; valign=top<br /> !Allocation name<br /> !Size in TB<br /> !class=&quot;unsortable&quot;|Project full name<br /> |-<br /> |alice<br /> |400<br /> |<br /> |-<br /> |uppnex<br /> |140<br /> |[https://www.uppnex.uu.se UPPmax NExt Generation Sequencing Cluster &amp; Storage]<br /> |-<br /> |brain_protein_atlas<br /> |10<br /> |Mouse brain protein atlas project<br /> |-<br /> | scims2lab<br /> |20<br /> | Identification of novel gene models by matching mass spectrometry data against 6-frame translations of the human genome<br /> |-<br /> |subatom<br /> |<br /> |Low-energy nuclear theory and experiment<br /> |-<br /> |genomics-gu<br /> |10<br /> |Genomics Core Facility, Sahlgrenska academy at University of Gothenburg.<br /> |-<br /> |Chemo<br /> |5TB<br /> |Genetic interaction networks in human deseas<br /> |-<br /> |cesm1_holocene<br /> |30<br /> |Arctic sea ice in warm climates<br /> |}<br /> <br /> == More information ==<br /> * [[SweStore introduction]]<br /> * [http://status.swestore.se/munin/monitor/monitor/ Per Project Monitoring of Swestore usage]<br /> * [[Accessing SweStore national storage with the ARC client]]<br /> &lt;!-- * [[Mounting SweStore national storage via WebDAV|Mounting SweStore national storage via WebDAV (Not recomendated at the moment)]] --&gt;<br /> If you have any issues using SweStore please do not hesitate to contact [mailto:swestore-support@snic.vr.se swestore-support].<br /> <br /> == Tools and scripts ==<br /> <br /> There exists a number of tools and utilities developed externally that can be useful. Here are some links:<br /> <br /> * [https://github.com/samuell/arc_tools ARC_Tools] - Convenience scripts for the arc client (Only a recursive rmdir so far).<br /> * [http://sourceforge.net/projects/arc-gui-clients ARC Graphical Clients] - Contains the ARC Storage Explorer (SweStore supported development).<br /> * Transfer script, [http://snicdocs.nsc.liu.se/wiki/SweStore/swstrans_arc swetrans_arc], provided by Adam Peplinski / Philipp Schlatter<br /> * [http://www.nordugrid.org/documents/SWIG-wrapped-ARC-Python-API.pdf Documentation of the ARC Python API (PDF)]<br /> <br /> = Center storage =<br /> Centre storage, as defined by the SNIC storage group, is a storage solution that lives independently of the computational resources and can be accessed from all such resources at a centre. Key features include the ability to access the same filesystem the same way on all computational resources at a centre, and a unified structure and nomenclature for all centra. Unlike cluster storage which is tightly associated with a single cluster, and thus has a limited life-time, centre storage does not require the users to migrate their own data when clusters are decommissioned, not even when the storage hardware itself is being replaced.<br /> <br /> == Unified environment ==<br /> To make the usage more transparent for SNIC users, a set of environment variables are available on all SNIC resources:<br /> <br /> * &lt;code&gt;SNIC_BACKUP&lt;/code&gt; – the user's primary directory at the centre&lt;br&gt;(the part of the centre storage that is backed up)<br /> * &lt;code&gt;SNIC_NOBACKUP&lt;/code&gt; – recommended directory for project storage without backup&lt;br&gt;(also on the centre storage)<br /> * &lt;code&gt;SNIC_TMP&lt;/code&gt; – recommended directory for best performance during a job&lt;br&gt;(local disk on nodes if applicable)</div>

Jonas Lindemann (LUNARC) https://snicdocs.nsc.liu.se/w/index.php?title=Accessing_Swestore_with_cURL&diff=4270 2012-07-30T12:31:52Z

<p>Jonas Lindemann (LUNARC): </p> <hr /> <div>This guide outlines the procedure for using cURL to access files through the WebDav door of dCache.<br /> <br /> == Essential parameters ==<br /> <br /> --capath /etc/grid-security/certificates<br /> The certificate bundle provided through --capath is required in order for cURL to accept the server certificates the door presents. If the certificate bundle is not available, the -k flag may be passed to allow untrusted server certificates.<br /> <br /> --cert /tmp/x509up_u1234<br /> --cert (or -E) names the proxy certificate generated by arcproxy or similar tools, which is a single PEM file consisting of the client certificate, the proxy key and the proxy certificate. The name will vary based on the user issuing it.<br /> grid-proxy-init (and thus arcproxy) will put the certificate in /tmp by default and name it according to the pattern x509up_u&lt;NumericUID&gt;. The -out parameter to grid-proxy-init takes a location to store the certificate in if the default is not sufficient.<br /> <br /> --location<br /> --location (or -L) instructs cURL to follow HTTP redirects, in this case the 302 redirects that the dCache door uses to direct clients to different storage nodes.<br /> <br /> == Sample invocations ==<br /> <br /> Downloads the file 'file-to-download.ext':<br /> curl --location --capath /etc/grid-security/certificates --cert /tmp/x509up_u1234 -O https://webdav.swegrid.se/target/path/file-to-download.ext<br /> <br /> Upload the file 'source.file' as 'uploaded.ext':<br /> curl --location --capath /etc/grid-security/certificates --cert /tmp/x509up_u1234 -T ~/source.file https://webdav.swegrid.se/target/path/uploaded.ext<br /> <br /> = Credits =<br /> <br /> This guide was written by Lars Viklund</div>

Jonas Lindemann (LUNARC)