Difference between revisions of "Building CESM models on the NSC triolith system"

From SNIC Documentation
Jump to: navigation, search
 
(24 intermediate revisions by the same user not shown)
Line 1: Line 1:
This page describes the building of the CESM1 ([http://www.cesm.ucar.edu/models/cesm1.1/ CESM1.1]) model on NSC's triolith system ([http://www.nsc.liu.se/systems/triolith/ triolith]). Triolith consists of 1200 HP SL230s compute nodes is equipped with two Intel E5-2660 (2.2 GHz Sandybridge) processors with 8 cores each, i.e 16 cores per node. Triolith is equipped with a fast interconnect (Infiniband from Mellanox (FDR IB, 56 Gb/s) in a 2:1 blocking configuration) for high performance for parallel applications.
+
This page describes the building the CESM series of models, including CCSM4 (e.g. [http://www.cesm.ucar.edu/models/ccsm4.0/ CCSM4.0], [http://www.cesm.ucar.edu/models/cesm1.1/ CESM1.1], [http://www.cesm.ucar.edu/models/cesm1.2/ CESM1.2]) on NSC's triolith system ([http://www.nsc.liu.se/systems/triolith/ triolith]). Triolith consists of 1200 HP SL230s compute nodes is equipped with two Intel E5-2660 (2.2 GHz Sandybridge) processors with 8 cores each, i.e 16 cores per node. Triolith is equipped with a fast interconnect (Infiniband from Mellanox (FDR IB, 56 Gb/s) in a 2:1 blocking configuration) for high performance for parallel applications.
  
 
Triolith is a SNIC-funded system, and computing time on Triolith is exclusively allocated to SNIC projects (Large, Medium and Small). See [http://www.snic.vr.se/apply-for-resources] and [http://www.nsc.liu.se/start/apply/] for information on how to apply for a SNIC project. Contact: support@nsc.liu.se.
 
Triolith is a SNIC-funded system, and computing time on Triolith is exclusively allocated to SNIC projects (Large, Medium and Small). See [http://www.snic.vr.se/apply-for-resources] and [http://www.nsc.liu.se/start/apply/] for information on how to apply for a SNIC project. Contact: support@nsc.liu.se.
 +
  
 
;CESM1 unix group on triolith
 
;CESM1 unix group on triolith
  
If you are going to run the CESM1 model on triolith, you should join the 'cesm1' unix group on triolith. This is not absolutely necessary but being a member allows access to shared storage space on /nobackup/global where pre-downloaded initial data and ancillary files are stored. Email support@nsc.liu.se to request membership to the cesm1 unix group.
+
If you are going to run one of the CESM models on triolith, you should join the 'cesm_input-data' storage project on Triolith (snic2014-5-61). This is not absolutely necessary but being a member allows access to shared storage space on <tt>/proj/cesm_input-data</tt> where pre-downloaded initial data and ancillary files are stored. To join the 'cesm_input-data' project, log into SUPR (https://supr.snic.se/) and click the 'View and Manage Projects' button, the request membership to the snic2014-5-61 project.
 +
 
 +
 
 +
;The CESM models (incl. CCSM4) - Getting the source code
 +
 
 +
CESM is a free, open source model, but a short [http://www.cesm.ucar.edu/models/cesm1.1/register/register_cesm1.1.cgi registeration] is required before access to the subversion repository is granted. The source code, build and run scripts can be downloaded as described in the CESM1 [http://www.cesm.ucar.edu/models/cesm1.1/cesm/doc/usersguide/book1.html user guide]. To list the available model versions, after CESM registration (on triloith):
 +
 
 +
:<code>svn list https://svn-ccsm-release.cgd.ucar.edu/model_versions</code>
 +
 
 +
IMPORTANT NOTE: Starting with CESM1.2.1, the Subversion repository path has changed (https://svn-ccsm-models.cgd.ucar.edu/cesm1/release_tags)
 +
 
 +
The source files should be downloaded into the <tt>/home</tt> file system. This ensures that your code, and configuration files are backed up.
 +
 
 +
 
 +
;Building and configuring the model
 +
 
 +
As, an example. to create a CESM1.2.2 version in some base folder <tt><cesm-base></tt>:
 +
 
 +
1. Change to the appropriate folder
 +
 
 +
:<code>cd <cesm-base></code>
 +
 
 +
2. Checkout version 1.2.2
 +
 
 +
:<code>svn co https://svn-ccsm-models.cgd.ucar.edu/cesm1/release_tags/cesm1_2_2 cesm1.2.2</code>
 +
 
 +
3. Files containing triolith specific machine environment variables have been created and are stored in the folder /software/apps/CESM/cesm1.2.2/ (ccsm4.0, cesm1.0.3, cesm1.0.4, cesm1.1.1 and cesm1.2.0 config files also available). These files should be copied to the appropriate location in your local distribution. Based on the steps above:
 +
 
 +
 
 +
:<code>cd <cesm-base>/cesm1.2.2/scripts/</code>
 +
:<code>cp /software/apps/CESM/cesm1.2.2/create_newcase .</code>
 +
 
 +
 
 +
:<code>cd <cesm-base>/cesm1.2.2/scripts/ccsm_utils/Case.template</code>
 +
:<code>cp /software/apps/CESM/cesm1.2.2/config_definition.xml .</code>
 +
 
 +
 
 +
Copy the remaining files to <cesm-base>/cesm1.2.2/scripts/ccsm_utils/Machine. e.g. for CESM1.1 and above
 +
 
 +
:<code>cd <cesm-base>/cesm1.2.2/scripts/ccsm_utils/Machines</code>
 +
:<code>cp /software/apps/CESM/cesm1.2.2/config_compilers.xml .</code>
 +
:<code>cp /software/apps/CESM/cesm1.2.2/config_machines.xml .</code>
 +
:<code>cp /software/apps/CESM/cesm1.2.2/config_pes.xml .</code>
 +
:<code>cp /software/apps/CESM/cesm1.2.2/env_mach_specific.triolith .</code>
 +
:<code>cp /software/apps/CESM/cesm1.2.2/mkbatch.triolith .</code>
 +
 
 +
For CCSM4.0 and CESM1.0.X, e.g.
 +
 
 +
:<code>cd <cesm-base>/ccsm4.0/scripts/ccsm_utils/Machines</code>
 +
:<code>cp /software/apps/CESM/cesm1.2.2/config_machines.xml .</code>
 +
:<code>cp /software/apps/CESM/cesm1.2.2/config_pes.xml .</code>
 +
:<code>cp /software/apps/CESM/cesm1.2.2/env_machopts.triolith .</code>
 +
:<code>cp /software/apps/CESM/cesm1.2.2/mkbatch.triolith .</code>
 +
 
  
;CESM1 - Getting the source code
+
''Notes:''
 +
* CESM1.0.3, the file <tt>ecosys_mod.F90</tt>, also found in the /software/apps/CESM/cesm1.0.3 must be copied to the <your-CESM-case>/SourceMods/src.pop2 folder before compiling. This file corrects a compile bug in the ocean model.
 +
* CCSM4.0, the file <tt>ncdio.F90</tt>, also found in the /software/apps/CESM/ccsm4.0 must be copied to the <cesm-base>/models/lnd/clm/src/main/ folder. This file corrects a small compile bug in the land model.
  
CEMS1 is a free, open source model, but a short [http://www.cesm.ucar.edu/models/cesm1.1/register/register_cesm1.1.cgi registeration] is required before access to the subversion repository is granted. The source code, build and run scripts can be downloaded as described in the CESM1 [http://www.cesm.ucar.edu/models/cesm1.1/cesm/doc/usersguide/book1.html user guide]. The files should be downloaded into the /home file system. This ensures that your code, and configuration files are backed up. To create a CESM1.1.1 version in /home/$USER/Projects/cesm1.1.1:
 
  
1) Change to the appropriate folder
+
4. CESM1 can now be built in the usual way as described in the CESM1 users guide (see [http://www.cesm.ucar.edu/models/cesm1.1/cesm/doc/usersguide/c588.html Creating and Setting Up A Case]) using 'triolith' for the machine designation. '''Note''': when using the 'create_newcase' script, you have to add an additional variable 'ccsm_out'. This defines the folder where the model is compiled and where output is written, and must correspond to an appropriate Triolith project storage area (e.g. /proj/bolinc/users/x_abcde/cesm1.2.2)
  
:cd Projects
+
For example, to create a new 'f19_g16' resolution case using the 'B' compset:
  
2) List available model versions:
+
:<code>cd <cesm-base>/cesm1.2.2/scripts/</code>
 +
:<code>./create_newcase -case ../cases/lbtest-f19_g16-B -res f19_g16 -compset B -mach triolith -pecount M -ccsm_out /proj/bolinc/users/${USER}/cesm1.2.2</code>
  
:svn list https://svn-ccsm-release.cgd.ucar.edu/model_versions
+
5. Configure and build your model:
  
3) Checkout version 1.1.1
+
:<code>cd <cesm-base>/cesm1.2.2/cases/lbtest-f19_g16-B</code>
 +
:<code>./cesm_setup</code>
 +
:<code>./lbtest-f19_g16-B.build</code>
  
:svn co https://svn-ccsm-release.cgd.ucar.edu/model_versions/cesm1_1_1 cesm1.1.1
+
''Note:'' for CCSM4 CESM1.0.X and NorESM use
  
4) Files containing triolith specific machine environment variables have been created and are stored in the folder /nobackup/global/proj/cesm1/triolith_config_cesm1.1.1 (triolith_config_cesm1.0.4 config files also available). These files should be copied to the appropriate location in your local distribution. Based on the steps above:
+
:<code>cd <cesm-base>/cssm4.0/cases/lbtest-f19_g16-B</code>
 +
:<code>./configure -case</code>
 +
:<code>./lbtest-f19_g16-B.build</code>
  
:cd /home/$USER/Projects/cesm1.1.1/scripts/ccsm_utils/Machines
+
6. Edit the env_run.xml and the $CASE.run files for the particular simulation you wish to make, then submit your job:
:cp /nobackup/global/proj/cesm1/triolith_config_cesm1.1.1/* .
 
  
5) CESM1 can now be built in the usual way as described in the CESM1 users guide (see [http://www.cesm.ucar.edu/models/cesm1.1/cesm/doc/usersguide/c588.html Creating and Setting Up A Case]) using 'triolith' for the machine designation. For example, to create a new 'f19_g16' resolution case using the 'B' compset:
+
:<code>sbatch ./lbtest-f19_g16-B.run</code>
  
:cd /home/$USER/Projects/cesm1.1.1/scripts/
 
:./create_newcase -case ../cases/lbtest-f19_g16-B -res f19_g16 -compset B -mach triolith
 
  
6) Configure and build your model:
 
  
:cd /home/$USER/Projects/cesm1.1.1/cases/lbtest-f19_g16-B
+
;Model load balancing
:./cesm_setup
 
:./lbtest-f19_g16-B.build
 
  
7) Edit the env_run.xml file for the particular simulation you wish to make, then submit your job:
+
Load balancing of the models (laying out pe configurations e.g. [http://www.cesm.ucar.edu/models/cesm1.1/cesm/doc/usersguide/x1565.html load balancing a case]) has been completed for a limited set of model configurations (resolution f19_g16, compset B). Using the 'pecount' keyword when invoking <tt>.create_newcase</tt> (see 4. above) will create a load balanced configuration of the model (res f19_g16, compest B). pecount options are 'S' (small), 'M' (medium) and 'L' (large): see [[EC-Earth and CESM model timings]]. For assistance with load balancing alternative configurations (resolutions / compsets), email: support@nsc.liu.se.
  
:sbatch ./lbtest-f19_g16-B.run
+
{{set category list|field|Climate research}}

Latest revision as of 09:46, 28 May 2015

This page describes the building the CESM series of models, including CCSM4 (e.g. CCSM4.0, CESM1.1, CESM1.2) on NSC's triolith system (triolith). Triolith consists of 1200 HP SL230s compute nodes is equipped with two Intel E5-2660 (2.2 GHz Sandybridge) processors with 8 cores each, i.e 16 cores per node. Triolith is equipped with a fast interconnect (Infiniband from Mellanox (FDR IB, 56 Gb/s) in a 2:1 blocking configuration) for high performance for parallel applications.

Triolith is a SNIC-funded system, and computing time on Triolith is exclusively allocated to SNIC projects (Large, Medium and Small). See [1] and [2] for information on how to apply for a SNIC project. Contact: support@nsc.liu.se.


CESM1 unix group on triolith

If you are going to run one of the CESM models on triolith, you should join the 'cesm_input-data' storage project on Triolith (snic2014-5-61). This is not absolutely necessary but being a member allows access to shared storage space on /proj/cesm_input-data where pre-downloaded initial data and ancillary files are stored. To join the 'cesm_input-data' project, log into SUPR (https://supr.snic.se/) and click the 'View and Manage Projects' button, the request membership to the snic2014-5-61 project.


The CESM models (incl. CCSM4) - Getting the source code

CESM is a free, open source model, but a short registeration is required before access to the subversion repository is granted. The source code, build and run scripts can be downloaded as described in the CESM1 user guide. To list the available model versions, after CESM registration (on triloith):

svn list https://svn-ccsm-release.cgd.ucar.edu/model_versions

IMPORTANT NOTE: Starting with CESM1.2.1, the Subversion repository path has changed (https://svn-ccsm-models.cgd.ucar.edu/cesm1/release_tags)

The source files should be downloaded into the /home file system. This ensures that your code, and configuration files are backed up.


Building and configuring the model

As, an example. to create a CESM1.2.2 version in some base folder <cesm-base>:

1. Change to the appropriate folder

cd <cesm-base>

2. Checkout version 1.2.2

svn co https://svn-ccsm-models.cgd.ucar.edu/cesm1/release_tags/cesm1_2_2 cesm1.2.2

3. Files containing triolith specific machine environment variables have been created and are stored in the folder /software/apps/CESM/cesm1.2.2/ (ccsm4.0, cesm1.0.3, cesm1.0.4, cesm1.1.1 and cesm1.2.0 config files also available). These files should be copied to the appropriate location in your local distribution. Based on the steps above:


cd <cesm-base>/cesm1.2.2/scripts/
cp /software/apps/CESM/cesm1.2.2/create_newcase .


cd <cesm-base>/cesm1.2.2/scripts/ccsm_utils/Case.template
cp /software/apps/CESM/cesm1.2.2/config_definition.xml .


Copy the remaining files to <cesm-base>/cesm1.2.2/scripts/ccsm_utils/Machine. e.g. for CESM1.1 and above

cd <cesm-base>/cesm1.2.2/scripts/ccsm_utils/Machines
cp /software/apps/CESM/cesm1.2.2/config_compilers.xml .
cp /software/apps/CESM/cesm1.2.2/config_machines.xml .
cp /software/apps/CESM/cesm1.2.2/config_pes.xml .
cp /software/apps/CESM/cesm1.2.2/env_mach_specific.triolith .
cp /software/apps/CESM/cesm1.2.2/mkbatch.triolith .

For CCSM4.0 and CESM1.0.X, e.g.

cd <cesm-base>/ccsm4.0/scripts/ccsm_utils/Machines
cp /software/apps/CESM/cesm1.2.2/config_machines.xml .
cp /software/apps/CESM/cesm1.2.2/config_pes.xml .
cp /software/apps/CESM/cesm1.2.2/env_machopts.triolith .
cp /software/apps/CESM/cesm1.2.2/mkbatch.triolith .


Notes:

  • CESM1.0.3, the file ecosys_mod.F90, also found in the /software/apps/CESM/cesm1.0.3 must be copied to the <your-CESM-case>/SourceMods/src.pop2 folder before compiling. This file corrects a compile bug in the ocean model.
  • CCSM4.0, the file ncdio.F90, also found in the /software/apps/CESM/ccsm4.0 must be copied to the <cesm-base>/models/lnd/clm/src/main/ folder. This file corrects a small compile bug in the land model.


4. CESM1 can now be built in the usual way as described in the CESM1 users guide (see Creating and Setting Up A Case) using 'triolith' for the machine designation. Note: when using the 'create_newcase' script, you have to add an additional variable 'ccsm_out'. This defines the folder where the model is compiled and where output is written, and must correspond to an appropriate Triolith project storage area (e.g. /proj/bolinc/users/x_abcde/cesm1.2.2)

For example, to create a new 'f19_g16' resolution case using the 'B' compset:

cd <cesm-base>/cesm1.2.2/scripts/
./create_newcase -case ../cases/lbtest-f19_g16-B -res f19_g16 -compset B -mach triolith -pecount M -ccsm_out /proj/bolinc/users/${USER}/cesm1.2.2

5. Configure and build your model:

cd <cesm-base>/cesm1.2.2/cases/lbtest-f19_g16-B
./cesm_setup
./lbtest-f19_g16-B.build

Note: for CCSM4 CESM1.0.X and NorESM use

cd <cesm-base>/cssm4.0/cases/lbtest-f19_g16-B
./configure -case
./lbtest-f19_g16-B.build

6. Edit the env_run.xml and the $CASE.run files for the particular simulation you wish to make, then submit your job:

sbatch ./lbtest-f19_g16-B.run


Model load balancing

Load balancing of the models (laying out pe configurations e.g. load balancing a case) has been completed for a limited set of model configurations (resolution f19_g16, compset B). Using the 'pecount' keyword when invoking .create_newcase (see 4. above) will create a load balanced configuration of the model (res f19_g16, compest B). pecount options are 'S' (small), 'M' (medium) and 'L' (large): see EC-Earth and CESM model timings. For assistance with load balancing alternative configurations (resolutions / compsets), email: support@nsc.liu.se.