About preemptive scheduling

Preemptive scheduling takes effect when two jobs compete for the same job slots. If a high-priority job is pending, LSF can suspend a lower-priority job that is running, and then start the high-priority job instead. For this to happen, the high-priority job must be pending in a preemptive queue (a queue that can preempt other queues), or the low-priority job must belong to a preemptable queue (a queue that can be preempted by other queues).

If multiple slots are required, LSF can preempt multiple jobs until sufficient slots are available. For example, one or more jobs can be preempted for a job that needs multiple job slots.

A preempted job is resumed as soon as more job slots become available; it does not necessarily have to wait for the preempting job to finish.

Preemptive queue

Jobs in a preemptive queue can preempt jobs in any queue of lower priority, even if the lower-priority queues are not specified as preemptable.

Preemptive queues are more aggressive at scheduling jobs because a slot that is not available to a low-priority queue may be available by preemption to a high-priority queue.

Preemptable queue

Jobs in a preemptable queue can be preempted by jobs from any queue of a higher priority, even if the higher-priority queues are not specified as preemptive.

When multiple preemptable jobs exist (low-priority jobs holding the required slots), and preemption occurs, LSF preempts a job from the least-loaded host.

Resizable jobs

Resize allocation requests are not able take advantage of the queue-based preemption mechanism to preempt other jobs. However, regular pending jobs are still able to preempt running resizable jobs, even while they have a resize request pending. When a resizable job is preempted and goes to the SSUSP state, its resize request remains pending and LSF stops scheduling it until it returns back to RUN state.

  • New pending allocation requests cannot make use of preemption policy to get slots from other running or suspended jobs.

  • Once a resize decision has been made, LSF updates its job counters to be reflected in future preemption calculations. For instance, resizing a running preemptable job from 2 slots to 4 slots, makes 4 preemptable slots for high priority pending jobs.

  • If a job is suspended, LSF stops allocating resources to a pending resize request.

  • When a preemption decision is made, if job has pending resize request and scheduler already has made an allocation decision for this request, LSF cancels the allocation decision.

  • If a preemption decision is made while a job resize notification command is running, LSF prevents the suspend signal from reaching the job.

Scope

By default, preemptive scheduling does not apply to jobs that have been forced to run (using brun) or backfill and exclusive jobs.

Limitations

Description

Exclusive jobs

Jobs requesting exclusive use of resources cannot preempt other jobs.

Jobs using resources exclusively cannot be preempted.

Backfill jobs

Jobs backfilling future advance reservations cannot be preempted.

brun

Jobs forced to run with the command brun cannot be preempted.

Default behavior (preemptive scheduling not enabled)

With preemptive scheduling enabled (preemptive queue)

With preemptive scheduling enabled (preemptable queue)

Parent topic: Preemptive Scheduling