Batch Jobs at QM

Basic usage

This section describes how to run batch jobs.
Batch jobs are jobs that run without user intervention. Typically these are big compute jobs, or a large collection of jobs, producing results to a file.
On the QMUL Atlas Cluster Sun Grid Engine is used to schedule and control batch jobs. In its most basic operation, a job is described by a shell script, which is submitted to the sun grid engine (SGE) scheduler. The scheduler runs the job as soon as resources are available and informs the user when the job is complete.

Useful links

The following links might be useful:
http://docs.sun.com/app/docs/doc/817-6117 Sun Microsystems Documentation in html or the same Manual as a pdf from QMUL server from here.

Requirements for a job

• A job must be a shell script.
  
• It may call a binary executable.
  
• The binary should be valid for the machine on which it runs.

Simple example

The following simple example will run a simple test shell job.
Log on to any submit host ( heppc100-003 heppc200-211) and do

source /opt/sge/default/common/settings.sh

and copy /opt/sge/examples/jobs/simple.sh into your working directory. Examine the contents of this script and you will see that it prints the date, then pauses for twenty seconds, before finally printing the date and time again and exiting.
Submit the job to the scheduler using:

qsub -cwd simple.

and take note of the job id which was returned.
Take a look at the list of jobs on the queuing system by running qstats
If you run qstat a few times, you should see your jobs status change from queued/waiting (qw) to transfering(t) to running (r) as the job is allocated to a machine and run. When the job is completed it will be removed from the qstat output. If your job status stays at queued/waiting for a long time it is likely that all queues which are of an appropriate type for your job are in use and you will have to wait until resources are free to run your job. If you take a look in your directory, you should see the job output and error files, containing stdout and stderr output, named simple.ojob_id and simple.ejob_id respectively.

Requesting resources

The example above is fine for a short job which does not have any special requirements (eg. large amounts of memory, long run time, multiple cpus or specialised software licences). To ensure that the job executes on an appropriate machine you must specify the resources that it requires.
Resources can be requested on the command line, using options to qsub, or may be embedded into the job script. If you edit the simple.sh script run in the previous example, you will see it contains 2 lines near the beginning:

#$ -S /bin/bash  #$ -N Sleeper

These are sun grid engine options. They are comments (indicated by the # at the beggining of the line) and so will be ignored by the command interpreter when the job is run, but are picked up by the qsub command when the job is submitted. Any line starting #$ can contain sge qsub options.
The two options in the simple.sh file set the command interpreter to /bin/bash and set the job name (the name displayed by qstat) to "Sleeper".
If these lines were not embedded in the job script they could be specified on the command line at job submision:

qsub -cwd -S /bin/bash -N Sleeper simple.sh

Most sge options can be included on the command line or in a job file.
There are a huge number of options available for qsub, a complete list can be found by running man qsub on a console on a WRG machine. The most commonly used is the -l option which lists requirements for the job (target architecture, job runtime, memory requirements etc...) and which has the format:

-l resource=value,.....

Resources which are supported include:

• arch Machine architecture required to run job, supported type for the time being is lx24.
• h_rt Hard run time limit, maximum amount of cpu time the job requires, the format is hours:minutes:seconds. 
• h_rss Amount of memory required by your job. 

Example

To request a job run on a machine which will provide 2GB memory and allow the job to run for up to 2 days:

qsub -cwd -h_rt=48:0:0,h_rss=2G simple.sh

Other useful qsub options

• Working directory: -cwd This option instructs SGE to place output and error files from the job into your current directory (the directory you are in when you run qsub). If you do not use this option or -o/-e output files will be written to your home directory

Showwing running jobs

Then qstat command provides a list of jobs which are currently running or waiting for resources.

Deleting a job.

You can delete a job using the qdel command:

qdel job_id

You can find the job id using the qstat command.

Writing job scripts

Supported Command Line Interpreters (shells)

Jobs are submitted as shell scripts, by default bash is used, however, you can use csh by putting:

#!/bin/csh #$ -S /bin/csh

at the beginning of your shell script. You could also use an interpreted language (perl or python for example) if you wish.

Job environment variables

SGE provides a number of useful environment variables to the job when it runs. To get a list of all variables, add the command "env" then submit the simple.sh job used in the previous examples and take a look at the output file.
A very useful variable is $TMPDIR, if your job needs temporary/scratch file space then use this directory rather than your home file system as the performance of the local disk can be up to twice that of the network shares.