Control queues

About this task

Queues are controlled by:

Procedure

Parent topic: Working with Queues

Close a queue

Procedure

Run badmin qclose: 
badmin qclose normal
Queue <normal> is closed

When a user tries to submit a job to a closed queue the following message is displayed:

bsub -q normal ...
normal: Queue has been closed

Open a queue

Procedure

Run badmin qopen: 
badmin qopen normal
Queue <normal> is opened

Deactivate a queue

Procedure

Run badmin qinact: 
badmin qinact normal
Queue <normal> is inactivated

Activate a queue

Procedure

Run badmin qact: 
badmin qact normal
Queue <normal> is activated

Log a comment when controlling a queue

Procedure

  1. Use the -C option of badmin queue commands qclose, qopen, qact, and qinact to log an administrator comment in lsb.events. 
    badmin qclose -C "change configuration" normal

    The comment text change configuration is recorded in lsb.events.

    A new event record is recorded for each queue event. For example:

    badmin qclose -C "add user" normal

    followed by

    badmin qclose -C "add user user1" normal

    will generate records in lsb.events:

    "QUEUE_CTRL" "7.0 1050082373 1 "normal" 32185 "lsfadmin" "add user"
"QUEUE_CTRL" "7.0 1050082380 1 "normal" 32185 "lsfadmin" "add user user1"
  2. Use badmin hist or badmin qhist to display administrator comments for closing and opening hosts. 
    badmin qhist
Fri Apr  4 10:50:36: Queue <normal> closed by administrator <lsfadmin> change configuration.

    bqueues -l also displays the comment text:

    bqueues -l normal
QUEUE: normal
 -- For normal low priority jobs, running only if hosts are lightly loaded.  Th is is the default queue.
 
PARAMETERS/STATISTICS
PRIO NICE STATUS          MAX JL/U JL/P JL/H NJOBS  PEND   RUN SSUSP USUSP  RSV
 30   20  Closed:Active     -    -    -    -     0     0     0     0     0    0 
Interval for a host to accept two jobs is 0 seconds
 THREADLIMIT
       7 
 
SCHEDULING PARAMETERS
            r15s   r1m  r15m   ut      pg    io   ls    it    tmp    swp    mem 
loadSched   -     -     -     -       -     -    -     -     -      -      - 
loadStop    -     -     -     -       -     -    -     -     -      -      -
 
              cpuspeed    bandwidth  
loadSched          -            -  
loadStop           -            -  
 
JOB EXCEPTION PARAMETERS
              OVERRUN(min) UNDERRUN(min) IDLE(cputime/runtime)
Threshold         -         2             -       
Jobs              -         0             - 
 
USERS:  all users 
HOSTS:  all 
RES_REQ:  select[type==any] 
 
ADMIN ACTION COMMENT: "change configuration"

Configure dispatch windows

About this task

A dispatch window specifies one or more time periods during which batch jobs are dispatched to run on hosts. Jobs are not dispatched outside of configured windows. Dispatch windows do not affect job submission and running jobs (they are allowed to run until completion). By default, queues are always Active; you must explicitly configure dispatch windows in the queue to specify a time when the queue is Inactive.

To configure a dispatch window:

Procedure

  1. Edit lsb.queues
  2. Create a DISPATCH_WINDOW keyword for the queue and specify one or more time windows. 
    Begin Queue 
QUEUE_NAME   = queue1 
PRIORITY     = 45 
DISPATCH_WINDOW = 4:30-12:00 
End Queue
  3. Reconfigure the cluster:
    1. Run lsadmin reconfig.
    2. Run badmin reconfig.
  4. Run bqueues -l to display the dispatch windows.

Configure run windows

About this task

A run window specifies one or more time periods during which jobs dispatched from a queue are allowed to run. When a run window closes, running jobs are suspended, and pending jobs remain pending. The suspended jobs are resumed when the window opens again. By default, queues are always Active and jobs can run until completion. You must explicitly configure run windows in the queue to specify a time when the queue is Inactive.

To configure a run window:

Procedure

  1. Edit lsb.queues.
  2. Create a RUN_WINDOW keyword for the queue and specify one or more time windows. 
    Begin Queue QUEUE_NAME   = queue1 PRIORITY     = 45 RUN_WINDOW = 4:30-12:00 End Queue
  3. Reconfigure the cluster:
    1. Run lsadmin reconfig.
    2. Run badmin reconfig.
  4. Run bqueues -l to display the run windows.

Add a queue

Procedure

  1. Log in as the LSF administrator on any host in the cluster.
  2. Edit lsb.queues to add the new queue definition.

    You can copy another queue definition from this file as a starting point; remember to change the QUEUE_NAME of the copied queue.

  3. Save the changes to lsb.queues.
  4. Run badmin reconfig to reconfigure mbatchd.

    Adding a queue does not affect pending or running jobs.

Example

Example queue:

Begin Queue 
QUEUE_NAME = normal 
PRIORITY = 30 
STACKLIMIT= 2048 
DESCRIPTION = For normal low priority jobs, running only if hosts are lightly loaded. 
QJOB_LIMIT = 60     # job limit of the queue 
PJOB_LIMIT = 2     # job limit per processor 
ut = 0.2 
io = 50/240 
USERS = all 
HOSTS = all  
NICE = 20 
End Queue

Remove a queue

About this task

Important:

Before removing a queue, make sure there are no jobs in that queue.

If there are jobs in the queue, move pending and running jobs to another queue, then remove the queue. If you remove a queue that has jobs in it, the jobs are temporarily moved to a queue named lost_and_found. Jobs in the lost_and_found queue remain pending until the user or the LSF administrator uses the bswitch command to switch the jobs into an existing queue. Jobs in other queues are not affected.

Procedure

  1. Log in as the LSF administrator on any host in the cluster.
  2. Close the queue to prevent any new jobs from being submitted. 
    badmin qclose night
Queue night is closed
  3. Move all pending and running jobs into another queue.

    Below, the bswitch -q night argument chooses jobs from the night queue, and the job ID number 0 specifies that all jobs should be switched:

    bjobs -u all -q night
JOBID USER  STAT  QUEUE FROM_HOST   EXEC_HOST   JOB_NAME   SUBMIT_TIME 
5308  user5  RUN   night    hostA     hostD         job5  Nov 21 18:16 
5310  user5 PEND   night    hostA     hostC        job10  Nov 21 18:17
 
bswitch -q night idle 0
Job <5308> is switched to queue <idle> 
Job <5310> is switched to queue <idle>
  4. Edit lsb.queues and remove or comment out the definition for the queue being removed.
  5. Save the changes to lsb.queues.
  6. Run badmin reconfig to reconfigure mbatchd.

Restrict host use by queues

About this task

You may want a host to be used only to run jobs that are submitted to specific queues. For example, if you just added a host for a specific department such as engineering, you may only want jobs submitted to the queues engineering1 and engineering2 to be able to run on the host.

Procedure

  1. Log on as root or the LSF administrator on any host in the cluster.
  2. Edit lsb.queues, and add the host to the HOSTS parameter of specific queues. 
    Begin Queue 
QUEUE_NAME = queue1 
... 
HOSTS=mynewhost hostA hostB 
... 
End Queue
  3. Save the changes to lsb.queues.
  4. Use badmin ckconfig to check the new queue definition. If any errors are reported, fix the problem and check the configuration again.
  5. Run badmin reconfig to reconfigure mbatchd.
  6. If you add a host to a queue, the new host will not be recognized by jobs that were submitted before you reconfigured. If you want the new host to be recognized, you must use the command badmin mbdrestart.

Add queue administrators

About this task

Queue administrators are optionally configured after installation. They have limited privileges; they can perform administrative operations (open, close, activate, inactivate) on the specified queue, or on jobs running in the specified queue. Queue administrators cannot modify configuration files, or operate on LSF daemons or on queues they are not configured to administer.

To switch a job from one queue to another, you must have administrator privileges for both queues.

Procedure

In the lsb.queues file, between Begin Queue and End Queue for the appropriate queue, specify the ADMINISTRATORS parameter, followed by the list of administrators for that queue. Separate the administrator names with a space. You can specify user names and group names. 
Begin Queue 
ADMINISTRATORS = User1 GroupA 
End Queue