On master and all the nodes, this should start the PBS execution daemon. The action will return no value or message if it is successful. Should the system complain at this point, try changing directories to /sbin and starting Mom from there.

2.2 The Server

There need only be one server, and it should be built on master. Run the command:

# pbs_server -t create

Again, if successful, it should return no value. If this is not the first time you are starting the scheduler, you need only type:

# pbs_server

for the program to begin. If you are attempting to rebuild the system from scratch (as will be discussed in later chapters) you may get the error message, ``server database already exists. Proceed Y/N?'' You will have to proceed in order for PBS to run, but the server will continue to use the same database as it did before (meaning that any previous jobs you had running or options you had set will be reinstated).

Assuming that all proceeds normally, you may run:

# qmgr -c "set server scheduling=true"

You should now be able to start, enable and configure the scheduler.

2.3 The Scheduler

At the command line run:

# pbs_sched

With the scheduler activated, it is possible to evoke the Queue Manager (Qmgr). The qmgr command allow you to create and modify servers and queues for use within PBS. Without at least one active queue to run jobs and a server to moderate tasks for the queue, PBS is little more than a powerless daemon trapped within the system. To begin the process that will unleash them upon your cluster type:

# qmgr

This will start the queue manager. You will see:

qmgr
Max open servers: 4
Qmgr:

One server already exists-the clienthost ``master''. Therefore, the first order of business is to create a queue. To create a queue by the name of ``rque'', for example, one would type:

Qmgr: c q rque@master
queue_type=execution

Queues can be of either type=execution or type=scheduling. There must be at least one execution queue for jobs to run. In addition the queue has to know to which server it is bound, hence the @master prefix affixed to rque.

Now the server needs to be made aware of the queue insofar as knowing where to direct jobs by default. If multiple queues exists, it is possible to submit jobs to one queue specifically. So, for example, if rque, dque and mque were all @master, I could submit my job specifically to mque.

In any case the server requires a default queue. Assuming that you have already created rque@master, run:

Qmgr: s s master
default_queue=rque@master

At this point you can start and enable your queue:

Qmgr: s q rque@master enabled=true
Qmgr: s q rque@master started=true

At this point all that remains to be done before jobs can be submitted is to tell both the server and the queue(s) how many jobs can be submitted to them at a time. Do this with the attribute max_running. For example:

Qmgr: s s master max_running=10
Qmgr: s q rque@master max_running=10

At this point your system should be ready to receive, queue, run jobs, and deliver the output. Run ``xpbs'' at the command line to start a gui that will allow you to monitor the status of your queues and the jobs within those queues. Run ``xpbsmon'' to bring up a color-coated display of the nodes which are online, busy, available, etc.

If you have any problems seeing either of the GUIs, or if a user has a preference regarding color and/or font, many of those options may be set in the ~./xpbsrc file. More information on how to set xpbs preferences may be found on page 57 of the administrator's manual.

2. 31 Further Queue Attributes

Experience has shown that, in general, the fewer attributes assigned to a single queue or server, the better. Setting resource defaults regarding job size, cycles or CPU time can cause jobs to be rejected right and left. Unless severe problems arise as a result of jobs monopolizing nodes for weeks at a time, it is suggested that resource limits and defaults remain unset.

On the other hand, the query_other_jobs attribute can be extremely helpful, especially if there are groups of users on the system with a vested interested in the progress of each others' jobs. This attribute allows users to qstat, or view the status of jobs submitted by another users. To activate it, set:

Qmgr: s s master query_other_jobs=true

2. 32 Unsetting Queue Attributes, Queues, and Killing Jobs

Attributes may also be unset, as needed. If, for instance, a job is stuck in a queue, one solution in order to get things moving along again is to make another queue. In this case, in order prevent jobs from being submitted to the queue with the stopped job, you change the default queue server settings. Before you set it, though, you have to unset it. For example:

unset server master default_queue

Remember always to name objects @master, or @their respective server, or you will encounter errors like:

Qmgr: s s default_queue=dque
qmgr obj= svr=default: Unauthorized Request

Simply naming the applicable server, even if there is no other server, will solve the problem:

Qmgr: s s master
default_queue=dque@master
Qmgr:

Rque can then be disabled, or destroyed, as the case may be. To delete a queue, run:

Qmgr: delete queue dque@master

Once in a while you will take a completely valid action, with an entirely correct syntax, and the system will still complain. One example is:

Qmgr: delete queue dque@master
qmgr obj=dque svr=master: Cannot delete busy queue

If the queue is busy a user with administrative privileges may attempt to delete the job in question with the qsig command, which, when it takes an argument of 0 signals a running job to quit immediately.

Problems may arise when the job itself is still in the queue, but is not running. To see if a queued job is running, run xpbs and look under the Jobs section for the column labeled``S'' (``Status''). If the letter in the column is not an R, the job is not running and therefore cannot be killed with either the qsig command, or manually via the delete button in the GUI (xpbs).

Once in a while a job will fail during execution, and will remain in the queue with a Status of R (Running), despite its processes being halted. Jobs of this type can take up CPU time and lock up nodes, preventing them from being used for other jobs. PBS sees part of its jobs as ``protecting the job against system crashes''. As a result, if a system or process goes down, PBS will often keep the job in queue, with the status of running. Furthermore it will ignore attempts to manually deactivate these jobs. Stopping and restarting PBS will not serve to clear the queue, nor will rebooting the entire system, as PBS is specifically designed to preserve jobs through a system shutdown. The only viable recourse that the Aquila team has identified in the contingency of a job being irrevocably stuck in a queue is to -- quite literally -- kill off the server. That is, if the status of a job is reading as Q, E, or R for an extended period but is clearing making no progress, you can kill off PBS completely by sending all PBS daemons the kill -9 PID signal, and then erasing, completely, the PBS home directory. Of course, this requires then running the configuration script and, essentially, remaking and reinstalling PBS on the master node. On the upside, this is a fairly swift procedure, especially if you wrote a script to do your initial configuration. If the PBS/home/server_priv/nodes file and the PBS/home/mom_priv/config file have been copied elsewhere, they can simply be recopied back into the appropriate directories after the reinstallation. If you wait long enough, and kill off and bring back the server a few times, zombied jobs will sometimes simply disappear of their own accord. How they do this is unknown. This sort of exorcism works in mysterious and unpredictable ways, if it works at all. In general, it has been our experience you must erase the PBS/home directory in usr/spool, for upon restarting the server with pbs_server -t create command, you will get an error stating that the a server database already exists. Part of that database, apparently, is a complete record of all jobs that were running before the server was halted. Unless you erase all records of the server database, you will be right back where you started.

This is an issue of insufficient amount of available, allocated global memory. You can increase the amount of memory by setting the environment variable P4_GLOBMEMSIZE (in bytes). Since the request for 17640040 bytes failed, one can reset the value of P4_GLOBMEMSIZE to 20000000 bytes with the following additions to the script:

For C Shell :

setenv P4_GLOBMEMSIZE 20000000

P4_GLOBMEMSIZE=20000000
export P4_GLOBMEMSIZE

Alternatively, if you have failed to create a maximum number of running processes (s q rque@master max_running=12 for example) you will get a message of qsub: Job exceeds queue resource limits. Simply setting max_running for the server and queue(s) to some value will ameliorate that issue.

Each of these has numerous options which can make your life easier once you become acquainted with the system.

• qsub - takes the file name of a shell script to run
• qstat - no arguments works fine
• qdel - takes the job number to delete as an argument you can find this number using qstat.

While you can submit jobs a variety of ways, perhaps the most convenient is to createt a shell script that passes the PBS arguments. Here are 2 such examples -- one for a serial job one for a parallel.

Example One:

#!/bin/bash
#PBS -N Xi(r)
#PBS -k eo
#PBS -m a
#PBS -M mwhite@astron.berkeley.edu
#PBS -l nodes=1:ppn=1
#
cd /home/mwhite/treepm/lcdm300/
#
../GalFake/pnt_corrfn_mesh 300.0
tpmsph_1.0000.gal > galcorr.log
#

Example Two:

#!/bin/bash
#PBS -N Prop
#PBS -k eo
#PBS -m a
#PBS -M mwhite@astron.berkeley.edu
#PBS -l nodes=7:ppn=2
#
cd /home/mwhite/treepm/lcdm300/
#
/usr/local/bin/mpirun -np 14 \
-machinefile $PBS_NODEFILE \
./properties tpmsph_1.0000 16 300.0 \
0.3 0.0 0.7 > tpmsph_1.0000.prop
#

PBS_NODEFILE is an environment variable which will be set by PBS to point to a file which lists the nodes that that job has been allocated.

The format of the shell script is the usual. It starts with

#!/bin/bash

then you can have any number of lines beginning with #PBS. These are treated as comments by the shell, but contain information for PBS. The most common arguments are:

• #PBS -N Name The name of the job. This will appear in the qstat output.
• #PBS -k eo This directs PBS to keep stderr and stdout. These go into files in your home directory called (usually) jobname.o??? and jobname.e??? where ??? is the job number. If you don't specify this you get the stdout/stderr only upon completion of the job.
• #PBS -m a Send mail if the job aborts. Could also ask for E-mail upon start and finish, but then you get a lot of E-mail.
• #PBS -M name The E-mail address to send mail to, e.g. marc@deep.berkeley.edu
• #PBS -l nodes=1:ppn=1 This is the key line, it gives the resources needed by the job. In this case 1 node(s) and 1 processor per node. If you need the whole box specify ppn=2. You aren't required to use all of the processors, but PBS will make sure no other jobs are sent to that node since no processors are "free". This is useful if you need all 2Gb of RAM for example.

Following the PBS arguments are the shell script commands. In these exampels the scripts begin by "cd"ing to the working directory. As qsub remembers from where you submitted the job, this isn't strictly necessary. This is a useful technique, however, as the scripts and the relavant data often reside in different places.

Job Submission

In the above example, the job "prop.sh" runs a halo property finder on the 512^3 simulation using 14 processors (the whole cluster at present). The job "xi.sh" runs a serial code to compute the correlation function for a fake galaxy catalogue produced from this simulation. To run the latter, for example, one would type at the command line:

$ qsub xi.sh
99.master

When the job is successfully submitted, it will return a value that corresponds to the job ID numer (99 in this case). To observe your job in the queue type:

$ qstat
Job id           Name             User 
           Time Use S Queue
---------------- ---------------- -------- - -----
99.master        Xi(r)           
mwhite                  0 R dque

Job 99, named Xi(r) submitted mwhite is running. So far it has used 0 time, since it has just started. It has a status of R, running, and is doing so in queue dque. The queue to which jobs are submitted may be specified with the -q option. Since it wasn't specified, one may assume that dque is the default queue.

To delete the job from the queue one may use:

$ qdel 99

which, if successful, returns no output. One could also say "qdel 99.master". Now, if you try and view the queue, you will see that the queue is empty:

$ qstat

Stdout and Stderr

PBS creates files log files with messages pertaining to the run of the job. For example:

-rw-------    1 mwhite   mwhite  95 May  7 18:08 Xi(r).o99
-rw-------    1 mwhite   mwhite   0 May  7 18:07 Xi(r).e99

Containing stdout and stderr for job 99. The output contains simply:

[mwhite@master ~]$ more "Xi(r).o99"
Warning: no access to tty (Bad file descriptor).
Thus no job control in this shell.
Terminated

This reveals, essentially, that the job was terminated. The first two lines always appear and can be safely ignored.

* above section submitted by Martin White

If you encounter difficulty more than 16 processes in PBS, despite having an adequate number of servers built in qmgr, it could possibly be an issue with MPI. By default mpirun will use at most the number of processors specified in the machines list for each node (up to 16 per machine, 16 TOTAL in our case). This can be controlled/altered with the MPI_MAX_CLUSTER_SIZE environmental variable.

6.2 SSH

When PBS was first brought up on Aquila, ssh and secure copy were being used for intranetwork communication. This proved to be both cumbersome and complicated. Additionally, SSH and MPI did not play especially well together, despite efforts on behalf of the system administrators to put public and private keys in all the proper places. When all was said and done it proved easier to put rsh and rcp on the system and leave ssh and scp on master only for purposes of communication with the outside world.

6.3 MISC

The author is this document is aware of the existence of an .rpm package for installation on systems running RedHat, but has no advice to provide in that regard, as the Aquila cluster ultimately ran SuSE as a primary OS.