PBS Information for the CGL compute Server

Introduction
Queue Structure
How to Create a Command File
How to Submit a Job
How to Submit OpenMP Jobs
How to Submit MPI Jobs
How to Check Job Status
How to Remove and Kill Jobs
How to Use Checkpoint/Restart

Introduction

The Computational Genetics Laboratory compute server resources consist of cgls1 (16-CPU Sun Fire 6800 and 12 CPUs for pbs) and cs0 (8-CPU Sun Fire v890 and 7 CPUs for pbs).

PBS is a queueing system for submitting serial and parallel jobs. It matches job requirements with the machine's resources by use of a command file. It ensures that the machines are used fairly and to their full potential and that resources don't go unused.

Queue Structure

The following table gives a summary of the enforced limits on memory and number of processors.

Queue Architecture Max Wall Clock Time per Job Max Number of CPUs per Job

sun 24 hrs 12 cpus

Queue	Architecture	Max Wall Clock Time per Job	Max Number of CPUs per Job
sun		24 hrs	12 cpus

How to Create a Command File

All pbs jobs must be submitted with a command file. The following example is a PBS script that will submit a single processor job that uses 2000 MB of memory and is estimated to run up to 24 hours.

  #PBS -l mem=2000mb,ncpus=1,walltime=24:00:00
  #PBS -m abe
  #PBS -M your_email@umn.edu
  
    cd /home/cgl/your_cgl_username/whatever_folder

    module load bioinformatics

    blastall -p blastn -d nt -a 1 -i test.seq -o test_nt.out

By using the -m abe option, PBS will send email to you if the job is aborted, beginning to run, and ended. If you use the option line:

#PBS -M your_email@umn.edu

you will received your job status email in whatever you specified email address.

How to Submit a Job

You may use PBS to submit jobs from cgls1.

There are two major ways to submit jobs. One is by using the qsub command. The other is by using the xpbs graphical user interface to PBS.

qsub

The qsub command is used to submit a job to the queuing system. qsub takes a job submission script that contains PBS commands and the commands necessary to run your job.
For example, if your submission script is "myscript," you could submit it as follows:
```
         module load pbs
         qsub myscript
```
xpbs

The xbps command allows you to select the resources you want by clicking on buttons and selecting options. One helpful hint is you can select all the resources you need and save the settings to a PBS submission text file that can be used with the qsub command.

How to Submit OpenMP Jobs

The following is a 4-processors OpenMP job. The wall time represents the amount of time the job takes to complete. So, if you specified 00:30:00, the job would need to finish in 30 minutes.

  #PBS -l ncpus=4,mem=1gb,walltime=00:30:00
  #PBS -m abe
  #PBS -M your_email@umn.edu

  cd /home/cgl/your_cgl_username/whatever_folder
  setenv OMP_NUM_THREADS 4

  ./a.out

  # end of example script

How to Submit MPI Jobs

The following is a 4-processor Message Passing Interface job. It requests the same resources as the above job.

  #PBS -l ncpus=4,mem=1gb,walltime=00:30:00
  #PBS -m abe
  #PBS -M your_email@umn.edu

    cd /home/cgl/your_cgl_username/whatever_folder

    mprun -np 4 ./a.out

  # end of example script

How to Check Job Status

You can check job status using the qstat command.

Alternatively, you can use the showq command. showq is provided with the Maui Scheduler, the job scheduler that we use in the CGL.

You can also use command: qstat -f job_number | grep walltime to figure out walltime of your job.

How to Remove and Kill Jobs

Jobs are killed or removed from the queuing system by using the qdel command. There is a man page for qdel that lists the options you can use with it. If you wish to signal a running job, you may use the qsig command. Again, please see the man page for information. Here is a quick example of how to kill job 1974.cfs as listed above.

qdel 1974.cfs