The Operating System on the Power4 systems is AIX 5.3.
If desired, you can specify the type of nodes that you want your job to run on. Specifying the type of node is optional. If you do not specify the type of node, it will use whatever processors are available. This will not present a problem for serial jobs, but may be an inefficient use of the system for parallel jobs. Not specifying a feature may mean a parallel jobs will run on nodes with different processor speeds leading to faster processors waiting for results from slower processors. For benchmarking codes, we recommend you specify the type of node. Nodes are specified through features. The following features are available:
| Feature Name | Node Type | Number of Nodes with this Feature | LoadLeveler Specification |
|---|---|---|---|
| Regatta | 1.3 GHz Regatta | 2 | #@ requirements = (Feature == "Regatta") |
| FastRegatta | 1.7 GHz Regatta | 2 | #@ requirements = (Feature == "FastRegatta") |
| p655 | 1.5 GHz p655 | 13 | #@ requirements = (Feature == "p655") |
| Fastp655 | 1.7 GHz p655 | 10 | #@ requirements = (Feature == "Fastp655") |
The IBM Power4 system has separate home directories and scratch spaces from the IBM SP.
| Node Type | Large page | Memory for large pages | Total Memory | Memory Available* |
|---|---|---|---|---|
| 1.3 GHz 32-processor Regatta | | | | |
| 1.3 GHz 24-processor Regatta | | | | |
| 1.7 GHz 32-processor Regatta | | | | |
| 1.7 GHz 30-processor Regatta | | | | |
| 1.5 GHz p655 | | | | |
| 1.7 GHz p655 | | | |
* This includes memory for large pages. If you are not using large pages, subtract the amount of memory reserved for large pages from this number.
| Wall Time | Nodes |
|---|---|
| 300 hours |
|
| 150 hours |
|
| 24 hours |
|
You can only login to the interactive node, which is one of 8-processor p655 nodes. This node is used for compiling and job submission. To run on the nodes of the Power4 system, you have to submit your job to the queue.
To login:
ssh -X -l username regatta.msi.umn.eduYou only need to use -l if your login name is different than the login name you use on your own workstation.
/scratch1 : 800 GB
/scratch2 : 800 GB
/scratch3 : 800 GB
/scratch4 : 800 GB
Fortran compiler level 7.1.1.2 is installed and fortran codes can be compiled
using xlf or xlf90 for serial codes and mpxlf and mpxlf90 for MPI parallel
codes. Please look at the man page for compiler options.
Examples:
Fortran 77: xlf -O3 -qtune=pwr4 -qarch=pwr4 prog.f
Fortran 90: xlf90 -O3 -qtune=pwr4 -qarch=pwr4 prog.f
Fortran 95: xlf95 -O3 -qtune=pwr4 -qarch=pwr4 prog.f
Thread-safe: xlf_r, xlf90_r and xlf95_r
MPI : mpxlf
MPI Thread-safe: mpxlf_r
C:
C compiler level 5.0.2.2 is installed and C codes can be compiled using xlc
for serial codes and mpcc for mpi parallel code.
Examples:
C: xlc
Thread-safe C: xlc_r
MPI C: mpcc
MPI Thread-safe C: mpcc_r
We have only one queue for the entire system. You do not need to specify a queue class in your LoadLeveler script. As mentioned above, you can specify a feature to specify the type of node that your job will use.
Maximum number of jobs: The maximum number of jobs that a person can have in the queue on the IBM Power4 is 32.
Example of Loadleveler script to submit a serial batch job
#!/bin/csh
#@ initialdir = /homes/r30/runesha/TESTS
#@ output = mm_serial.output
#@ error = mm_serial.error
#@ wall_clock_limit = 10:00
#@ resources = ConsumableMemory(500)
#@ queue
./a.out
No feature is specified in the above example, so the job will run an any
available processor on the system. Example of Loadleveler script to submit a parallel MPI batch job
#!/bin/csh
#@ initialdir = /homes/r30/runesha/TESTS
#@ output = mpi_$(jobid).out
#@ error = mpi_$(jobid).err
#@ job_type = parallel
#@ wall_clock_limit = 20:00
#@ resources = ConsumableMemory(700)
#@ network.MPI = css0,shared,IP
#@ blocking = unlimited
#@ total_tasks = 4
#@ requirements = (Feature == "Regatta")
#@ node_usage = shared
#@ queue
poe ./a.out
In this example the feature "Regatta" is specified so the parallel job will run only on the 1.3 Ghz Regatta nodes. Here is an example on using the Loadleveler feature "FastRegatta" in a script.
Example of a script to submit a MPI batch job with the Loadleveler feature "FastRegatta"
#!/bin/csh
#@ initialdir = /homes/r30/runesha/TESTS
#@ output = mpi_$(jobid).out
#@ error = mpi_$(jobid).err
#@ job_type = parallel
#@ wall_clock_limit = 20:00
#@ resources = ConsumableMemory(700)
#@ blocking = unlimited
#@ total_tasks = 4
#@ requirements = (Feature == "FastRegatta")
#@ node_usage = shared
#@ queue
poe ./a.out
A similar modification in the Loadleveler script is needed in order to use the "p655" feature.
Example of Loadleveler script to submit an OpenMP batch job
#!/bin/csh
#@ initialdir = /homes/r30/runesha/TESTS
#@ output = OpenMP_test.out
#@ error = OpenMp_test.err
#@ job_type = parallel
#@ wall_clock_limit = 30:00
#@ resources = ConsumableMemory(500)
#@ node = 1
#@ tasks_per_node = 4
#@ node_usage = shared
#@ queue
setenv OMP_NUM_THREADS 4
./a.out
Loadleveler commands
- Submitting a job: llsubmit test.ll
- Monitoring job's status: llq
- Canceling a job: llcancel job_id
where test.ll is your LoadLeveler script and job_id is the job id
that is returned when you submit a job to the queue.