Using SweGrid resources - Revision history

Niklas Edmundsson (HPC2N): Swestore documentation moved

2023-02-08T10:19:40Z

Swestore documentation moved

Show changes

Jonas Lindemann (LUNARC): /* Cleaning job data on resources */

2012-04-10T14:24:41Z

‎Cleaning job data on resources

Jonas Lindemann (LUNARC): /* Retrieving finished jobs */

2012-04-10T14:24:02Z

‎Retrieving finished jobs

Jonas Lindemann (LUNARC): /* Killing running jobs */

2012-04-10T14:23:05Z

‎Killing running jobs

Jonas Lindemann (LUNARC): /* Job status information */

2012-04-10T14:22:22Z

‎Job status information

Jonas Lindemann (LUNARC): /* Job submission */

2012-04-10T14:19:59Z

‎Job submission

Jonas Lindemann (LUNARC): /* Job submission */

2012-04-10T14:19:08Z

‎Job submission

Joel Hedlund (NSC) at 13:02, 28 September 2011

2011-09-28T13:02:18Z

Joel Hedlund (NSC) at 12:30, 28 September 2011

2011-09-28T12:30:18Z

Torben Rasmussen (NSC) at 13:26, 8 July 2011

2011-07-08T13:26:47Z

@@ Line 430: / Line 430: @@
 == Cleaning job data on resources ==
-The output files and logs from finished jobs are kept a couple of days on the resource and will be eventually erased automatically. If you are not interested in downloading results from jobs or want to remove old job results, the '''ngclean''' command can instruct the resources to clean the data from the jobs. '''ngclean''' works in the same way as '''ngkill'''. Cleaning a job using a job id can be achieved in the following way:
+The output files and logs from finished jobs are kept a couple of days on the resource and will be eventually erased automatically. If you are not interested in downloading results from jobs or want to remove old job results, the '''arcclean''' command can instruct the resources to clean the data from the jobs. '''ngclean''' works in the same way as '''arckill'''. Cleaning a job using a job id can be achieved in the following way:
-<pre>[jonas@localhost ex4]$ ngclean gsiftp://siri.lunarc.lu.se:2811/jobs/1388112973341962029409606
+<pre>[jonas@localhost ex4]$ arcclean gsiftp://siri.lunarc.lu.se:2811/jobs/1388112973341962029409606
 Proxy subject name: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=Jonas Lindemann/CN=792268717
 Proxy valid to: 2011-02-10 21:11:16
@@ Line 440: / Line 440: @@
 Job names can also be used:
-<pre>[jonas@localhost ex4]$ ngclean job0001
+<pre>[jonas@localhost ex4]$ arcclean job0001
 Proxy subject name: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=Jonas Lindemann/CN=792268717
 Proxy valid to: 2011-02-10 21:11:16
@@ Line 448: / Line 448: @@
 All jobs can be cleaned by using the '''-a''' switch:
-<pre>[jonas@localhost ex4]$ ngclean -a
+<pre>[jonas@localhost ex4]$ arcclean -a
 Proxy subject name: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=Jonas Lindemann/CN=792268717
 Proxy valid to: 2011-02-10 21:11:16
@@ Line 455: / Line 455: @@
 Deleting job: gsiftp://siri.lunarc.lu.se:2811/jobs/1394612973341971718841954
 Jobs processed: 2, deleted: 2</pre>
-Custom job lists can also be used by specifying the job list file using the '''-i'''.
+Custom job lists can also be used by specifying the job list file using the '''-j'''.

@@ Line 355: / Line 355: @@
 == Retrieving finished jobs ==
-When a job has finished executing on a grid resource the job output files and results can be downloaded using the '''ngget''' command. The general syntax:
+When a job has finished executing on a grid resource the job output files and results can be downloaded using the '''arcget''' command. The general syntax:
-<pre>ngget [options] [jobid|jobname]</pre>
+<pre>arcget [options] [jobid|jobname]</pre>
-Retrieving a single job is accomplished by using '''ngget''' and the job identifier:
+Retrieving a single job is accomplished by using '''arcget''' and the job identifier:
-<pre>[user@localhost ex2]$ ngget gsiftp://siri.lunarc.lu.se:2811/jobs/126181296507986385553351
+<pre>[user@localhost ex2]$ arcget gsiftp://siri.lunarc.lu.se:2811/jobs/126181296507986385553351
 Results stored at /home/user/usersguide/examples/ex2/126181296507986385553351
 Jobs processed: 1, successfuly downloaded: 1</pre>
 Retrieveing a job by the job name attributes can be done by specifying the job name instead of the job id. In the following example the job description had the '''jobName''' attribute set to '''job0001''':
-<pre>[user@localhost ex3]$ ngget job0001
+<pre>[user@localhost ex3]$ arcget job0001
 Results stored at /home/user/usersguide/examples/ex3/69012965093911208764313
 Jobs processed: 1, successfuly downloaded: 1</pre>
 Downloading all jobs in the '''$HOME/.ngjobs''' file is done by using the '''-a''' switch:
-<pre>[user@localhost ex3]$ ngget -a
+<pre>[user@localhost ex3]$ arcget -a
 Results stored at /home/user/usersguide/examples/ex3/131291296508001121515985
 Results stored at /home/user/usersguide/examples/ex3/133541296508018329511759
 Jobs processed: 2, successfuly downloaded: 2</pre>
-By default downloaded jobs are stored in directories with the same name as the last part of the job id. A job with the job id '''gsiftp://siri.lunarc.lu.se:2811/jobs/126181296507986385553351''' will be stored in the directory '''126181296507986385553351''' in the same directory as the '''ngget''' command was executed. This behavior can be changed using the '''-dir''' switch. Using the '''-dir''' switch will place create the downloaded job directories in the directory specified by the switch. The following example will download all jobs to the '''job_sweep1''' directory:
+By default downloaded jobs are stored in directories with the same name as the last part of the job id. A job with the job id '''gsiftp://siri.lunarc.lu.se:2811/jobs/126181296507986385553351''' will be stored in the directory '''126181296507986385553351''' in the same directory as the '''arcget''' command was executed. This behavior can be changed using the '''-dir''' switch. Using the '''-dir''' switch will place create the downloaded job directories in the directory specified by the switch. The following example will download all jobs to the '''job_sweep1''' directory:
-<pre>[user@localhost ex3]$ ngget -a -dir ./job_sweep1
+<pre>[user@localhost ex3]$ arcget -a -dir ./job_sweep1
 Results stored at ./job_sweep1/198512965106411445114039
 Results stored at ./job_sweep1/221412965106421926028190
@@ Line 384: / Line 384: @@
 It is also possible to use the job name as the job directory by using the '''-j''' switch as shown in the following example:
-<pre>[user@localhost ex3]$ ngget -a -j -dir ./job_sweep2
+<pre>[user@localhost ex3]$ arcget -a -j -dir ./job_sweep2
 Results stored at ./job_sweep2/job0001
 Results stored at ./job_sweep2/job0002
@@ Line 390: / Line 390: @@
 Results stored at ./job_sweep2/job0004
 Jobs processed: 4, successfuly downloaded: 4</pre>
 == Killing running jobs ==

@@ Line 392: / Line 392: @@
 == Killing running jobs ==
-If for some reason you need to kill any of the jobs submitted to a resource the '''ngkill''' command can be used. In the most basic form the command takes a job id or a job name as input. Killing a job using a job id is shown below:
+If for some reason you need to kill any of the jobs submitted to a resource the '''arckill''' command can be used. In the most basic form the command takes a job id or a job name as input. Killing a job using a job id is shown below:
-<pre>[jonas@localhost ex4]$ ngkill gsiftp://siri.lunarc.lu.se:2811/jobs/312511297331801573492658
+<pre>[jonas@localhost ex4]$ arckill gsiftp://siri.lunarc.lu.se:2811/jobs/312511297331801573492658
 Proxy subject name: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=Jonas Lindemann/CN=792268717
 Proxy valid to: 2011-02-10 21:11:16
@@ Line 403: / Line 403: @@
 killing a job using a job name is done in a similar procedure:
-<pre>[jonas@localhost ex4]$ ngkill job0001
+<pre>[jonas@localhost ex4]$ arckill job0001
 Proxy subject name: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=Jonas Lindemann/CN=792268717
 Proxy valid to: 2011-02-10 21:11:16
@@ Line 412: / Line 412: @@
 To kill all runnig jobs, the '''-a''' switch can be used, which is illustrated in the following example:
-<pre>[jonas@localhost ex4]$ ngkill -a
+<pre>[jonas@localhost ex4]$ arckill -a
 Proxy subject name: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=Jonas Lindemann/CN=792268717
 Proxy valid to: 2011-02-10 21:11:16
@@ Line 425: / Line 425: @@
 Deleting job: gsiftp://siri.lunarc.lu.se:2811/jobs/1329512973323431857144927
 Jobs processed: 4, killed: 4, deleted: 4</pre>
-Custom job lists can also be used by specifying the job list file using the '''-i''' switch.
+Custom job lists can also be used by specifying the job list file using the '''-j''' switch.
 == Cleaning job data on resources ==

@@ Line 299: / Line 299: @@
 == Job status information ==
-The status of the submitted job can be queried using the '''ngstat''' command which is similar to the '''qstat''' or '''showq''' commands on normal computational resources. The command takes a job id as input and queries the resource for information on the status of the job. This is shown in the following examples:
+The status of the submitted job can be queried using the '''arcstat''' command which is similar to the '''qstat''' or '''showq''' commands on normal computational resources. The command takes a job id as input and queries the resource for information on the status of the job. This is shown in the following examples:
-<pre>[user@localhost ex1]$ ngstat gsiftp://siri.lunarc.lu.se:2811/jobs/261871296472351384107384
+<pre>[user@localhost ex1]$ arcstat gsiftp://siri.lunarc.lu.se:2811/jobs/261871296472351384107384
 Job gsiftp://siri.lunarc.lu.se:2811/jobs/261871296472351384107384
    Status: FINISHED
@@ Line 307: / Line 307: @@
 This shows the status and exit code of the job, in this case that it has been executed and returned with an exit code of 0.
-To query all submitted jobs found in the '''$HOME/.ngjobs''' file, the '''-a''' switch can be used. A typical output from this command is show below:
+To query all submitted jobs found in the '''$HOME/.arc/jobs.xml''' file, the '''-a''' switch can be used. A typical output from this command is show below:
-<pre>[user@localhost ex1]$ ngstat -a
+<pre>[user@localhost ex1]$ arcstat -a
 Job gsiftp://siri.lunarc.lu.se:2811/jobs/2817512964675921399075910
    Status: FINISHED
@@ Line 324: / Line 324: @@
 If more information on the job is needed, the '''-l''' switch can be used to provide additional information on the job as in the following example:
-<pre>[user@localhost ex1]$ ngstat -l gsiftp://siri.lunarc.lu.se:2811/jobs/261871296472351384107384
+<pre>[user@localhost ex1]$ arcstat -l gsiftp://siri.lunarc.lu.se:2811/jobs/261871296472351384107384
 Job gsiftp://siri.lunarc.lu.se:2811/jobs/261871296472351384107384
    Status: FINISHED
@@ Line 348: / Line 348: @@
    Entry valid from: 2011-01-31 12:43:07
    Entry valid to: 2011-01-31 12:44:37</pre>
-The '''ngstat''' command can also used to query status of jobs from job lists created with the '''-o''' switch in the '''ngsub''' command. In '''ngstat''' this is accomplished using the '''-i''' switch:
+The '''arcstat''' command can also used to query status of jobs from job lists created with the '''-j''' switch in the '''arcsub''' command.
-<pre>[user@localhost ex1]$ ngstat -a -i job_sweep1</pre>
+<pre>[user@localhost ex1]$ arcstat -a -j job_sweep1</pre>
-The options timeout ('''-t'''), debug ('''-d''') and '''-c''' options can be used in the same way as in the '''ngsub''' command.
+The options timeout ('''-t'''), debug ('''-d''') and '''-c''' options can be used in the same way as in the '''arcsub''' command.
 == Retrieving finished jobs ==

@@ Line 204: / Line 204: @@
 == Job submission ==
-When the job description has been created and any additional needed files have been setup, the job can be submitted to a grid resource. In ARC jobsubmission is done using the '''ngsub''' command. The command is similar to the '''qsub''' command found on non-grid resources. The job submission procedure can be described in the following steps:
+When the job description has been created and any additional needed files have been setup, the job can be submitted to a grid resource. In ARC jobsubmission is done using the '''arcsub''' command. The command is similar to the '''qsub''' command found on non-grid resources. The job submission procedure can be described in the following steps:
 # Parse XRSL definition.
@@ Line 233: / Line 233: @@
 <pre>[user@localhost ex1]$ arcsub ex1.xrls
 Job submitted with jobid: gsiftp://siri.lunarc.lu.se:2811/jobs/2817512964675921399075910</pre>
-If the submission is succesful the command displays the job id of the job. The job id is a URL which uniquely identfies the job and the resource to which it was submitted. The ngsub command also stores submitted job id:s in the '''$HOME/.arc/jobs.xml''' file for later use by other commands.
+If the submission is succesful the command displays the job id of the job. The job id is a URL which uniquely identfies the job and the resource to which it was submitted. The arcsub command also stores submitted job id:s in the '''$HOME/.arc/jobs.xml''' file for later use by other commands.
 The see more output from the job submission the '''-d''' flag can be used with a parameter for the level of debug output. It is often enough to use '''-d 1''' to get more useful information as in the following example:
-<pre>[user@localhost ex1]$ ngsub -d 1 ex1.xrls
+<pre>[user@localhost ex1]$ arcsub -d 1 ex1.xrls
 Proxy subject name: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=User/CN=1212121
 Proxy valid to: 2011-01-31 22:32:58

@@ Line 210: / Line 210: @@
 # Submit job to selected resource. Transferring any files local to the submission directory (if any).
-The general syntax of the '''ngsub''' command is as follows:
+The general syntax of the '''arcsub''' command is as follows:
-<blockquote>ngsub [options]
+<blockquote>arcsub [options] [filename]
 </blockquote>
-The most important option is the '''-f''' option which defines the job description file to be used for job submission.
+The most important option is the "filename" option which defines the job description file to be used for job submission.
 To illustrate the job submission process we use the following example descriptions and scripts.
@@ Line 231: / Line 231: @@
 The simples form of submission of this job is shown in the following example:
-<pre>[user@localhost ex1]$ ngsub -f ex1.xrls
+<pre>[user@localhost ex1]$ arcsub ex1.xrls
 Job submitted with jobid: gsiftp://siri.lunarc.lu.se:2811/jobs/2817512964675921399075910</pre>
-If the submission is succesful the command displays the job id of the job. The job id is a URL which uniquely identfies the job and the resource to which it was submitted. The ngsub command also stores submitted job id:s in the '''$HOME/.ngjobs''' file for later use by other commands.
+If the submission is succesful the command displays the job id of the job. The job id is a URL which uniquely identfies the job and the resource to which it was submitted. The ngsub command also stores submitted job id:s in the '''$HOME/.arc/jobs.xml''' file for later use by other commands.
 The see more output from the job submission the '''-d''' flag can be used with a parameter for the level of debug output. It is often enough to use '''-d 1''' to get more useful information as in the following example:
-<pre>[user@localhost ex1]$ ngsub -d 1 -f ex1.xrls
+<pre>[user@localhost ex1]$ ngsub -d 1 ex1.xrls
 Proxy subject name: /O=Grid/O=NorduGrid/OU=lunarc.lu.se/CN=User/CN=1212121
 Proxy valid to: 2011-01-31 22:32:58
@@ Line 249: / Line 249: @@
 In some cases the information system on some resources is overloaded. This means that the job submission can get stuck waiting for a response. To limit the time waiting for non-responsive sites, the '''-t''' flag can be used to set a timeout in seconds. The following example shows a job submission with the '''-t''' flag set to 20 seconds:
-<pre>[user@localhost ex1]$ ngsub -t 20 -f ex1.xrls</pre>
+<pre>[user@localhost ex1]$ arcsub -t 20 -f ex1.xrls</pre>
 It is also possible to bypass the resource brokering and submit a job directly to a resource using the '''-c''' switch. The '''-c''' switch takes a hostname for the resource as input and will only submit to this resource. The '''-c''' switch can be given repeatedly to submit to multiple resources. It is also possible to reject a specific cluster using the switch by adding a minus sign in front of the hostname. The following examples illustrate different options of using this switch.
 Job submission directly to the resource given by '''siri.lunarc.lu.se''':
-<pre>[user@localhost ex1]$ ngsub -f ex1.xrls -c siri.lunarc.lu.se</pre>
+<pre>[user@localhost ex1]$ arcsub -c siri.lunarc.lu.se ex1.xrsl</pre>
 Job submission to all available resources `except` '''siri.lunarc.lu.se''':
-<pre>ngsub -f ex1.xrls -c -siri.lunarc.lu.se</pre>
+<pre>arcsub -c -siri.lunarc.lu.se ex1.xrls </pre>
-Instead of using the default '''$HOME/.ngjobs''' job file a user defined job list file can be specified using the '''-o''' switch. This can be useful when submitting a number of jobs in a parameter sweep:
+Instead of using the default '''$HOME/.arc/jobs.xml''' job file a user defined job list file can be specified using the '''-j''' switch. This can be useful when submitting a number of jobs in a parameter sweep:
-<pre>[user@localhost ex1]$ ngsub -o job_sweep1 -f ex1.xrsl</pre>
+<pre>[user@localhost ex1]$ arcsub -j job_sweep1 ex1.xrsl</pre>
-Additional options for the '''ngsub''' command can be found by using the '''-h''' switch:
+Additional options for the '''arcsub''' command can be found by using the '''-h''' switch:
-<pre>[user@localhost ex1]$ ngsub -h
+<pre>[user@localhost ex1]$ arcsub -h
-Usage: ngsub [options] [filename ...]
+Usage:
-Options:
+The arcsub command is used for submitting jobs to Grid enabled computing
-   -c, -cluster   [-]name       explicity select or reject a specific cluster
+resources.
-   -C, -clustlist [-]filename   list of clusters to select or reject
-   -g, -giisurl   url           url to a GIIS
+Help Options:
-  -G, -giislist  filename      list of GIIS urls
+  -?, --help                        Show help options
-  -e, -xrsl      xrslstring    xrslstring describing the job to be submitted
-   -f, -file      filename      xrslfile describing the job to be submitted
+Application Options:
-   -o, -joblist   filename      file where the jobids will be stored
+  -h, --help                        Show help options
-   -D  -dryrun                  add dryrun option
+   -c, --cluster=[-]name             explicitly select or reject a specific resource
-      -dumpxrsl                do not submit - dump transformed xrsl to
+   -g, --index=[-]name               explicitly select or reject an index server
-                                 stdout
+   -e, --jobdescrstring=string       jobdescription string describing the job to be submitted
-   -t, -timeout   time          timeout in seconds (default 20)
+   -f, --jobdescrfile=string         jobdescription file describing the job to be submitted
-   -d, -debug     debuglevel    from -3 (quiet) to 3 (verbose) - default 0
+   -j, --joblist=filename            the file storing information about active jobs (default ~/.arc/jobs.xml)
    ...</pre>
 == Job status information ==

@@ Line 1: / Line 1: @@
 [[Category:Grid computing]]
-[[Category:Getting started with SweGrid]]
+[[Category:SweGrid user guide]]
 [[Getting started with SweGrid|< Getting started with SweGrid]]

← Older revision		Revision as of 12:30, 28 September 2011
Line 1:		Line 1:
	[[Category:Grid computing]]		[[Category:Grid computing]]
		+	[[Category:Getting started with SweGrid]]
	[[Getting started with SweGrid\|< Getting started with SweGrid]]		[[Getting started with SweGrid\|< Getting started with SweGrid]]

← Older revision		Revision as of 13:26, 8 July 2011
Line 1:		Line 1:
−	[[Category:Grid]]	+	[[Category:Grid computing]]
	[[Getting started with SweGrid\|< Getting started with SweGrid]]		[[Getting started with SweGrid\|< Getting started with SweGrid]]