lsmake

runs make tasks in parallel

Synopsis

lsmake [-m "host_name [num_cores] [host_name [num_cores]] ...]"] [-a seconds] [-c num_tasks] [-E] [-G debug_level] [-T] [-u] [-V] [-x num_retries] [-y] [makeoption ...] [target ...]
lsmake [-R res_req] [-j max_cores] [-a seconds] [-c num_tasks] [-E] [-G debug_level] [-T] [-u] [-V] [-x num_retries] [-y] [makeoption ...] [target ...]
lsmake [-h]

Description

Runs make tasks in parallel on LSF hosts. Sets the environment variables on the remote hosts when lsmake first starts.

By default, uses the local host, uses only one core, starts only one task in each core, processes sub-makes sequentially, allows 1 second buffer time to compensate for file system latency, and does not retry if the job fails. lsmake is a modified version of GNU make.

Options

-a seconds

When commands in a target finish, commands in a dependent target wait the specified time before starting on a different host. This delay allows time for the shared file system to synchronize client and server, and compensates for file system latency. By default, the delay is 1 second. Slower file systems require a longer delay.

If the dependent target's commands start on the same execution host, there is no delay.

If retries are enabled with -x, the interval between retries also depends on the delay time.

-c num_tasks

Starts the specified number of tasks concurrently on each core. If you specify too many tasks, you could overload a host.

-E

Sets the environment variables for every task sent remotely.

This is necessary when make files change or override the environment variables they inherit at startup.

-G debug_level

Enables debugging, specify the debug level.

-j max_cores

Uses multiple cores, selecting the best available. Specify the maximum number of cores to use.

Not compatible with -m "host_name [num_cores] [host_name [num_cores]] ..."

Ignored if you use bsub to run lsmake.

-m "host_name [num_cores] [host_name [num_cores]] ..."

Uses the specified hosts. To use multiple cores on a host, specify the number of cores after the host name.

Not compatible with -R res_req and -j max_cores.

Ignored if you use bsub to run lsmake.

-R res_req

Uses only hosts that satisfy the specified resource requirements.

When you specify -R but not -j, uses one core on one host that satisfies the resource requirements.

If the group of hosts that match the selection string includes the submission host, the submission host will always be selected, and the policies defined by the order string only affect the other hosts.

Not compatible with -m "host_name [num_cores] [host_name [num_cores]] ..."

Ignored if you use bsub to run lsmake.

-T

Enables output tagging to prefix the task ID of the sender to the parallel task output data.

-u

Creates the data file lsmake.dat and updates it each second, tracking the number of tasks running over time.

This is useful if you want to export the data to third-party charting applications.

-V

Verbose mode. Prints the names of the hosts used.

-x num_retries

If the command fails, retries the command the specified number of times (for example, if the number of retries is 1, the command is attempted twice before exiting). This is useful to compensate for file system latency and minor errors.

The interval between retries increases exponentially with each retry attempt. The time between the initial, failed attempt and the first retry is equal to 1 second by default, or equal to the buffer time specified by -a. For subsequent attempts, the interval between attempts is doubled each time.

-y

Displays summary information after the job is done.

makeoption ...

Specifies standard GNU make options. Note: -j and -R are not supported as a GNU make options, see the lsmake options -j max_cores and -R res_req. See GNU documentation for detailed descriptions of other options. This version of lsmake supports GNU Make version 3.81, which includes the following options:

  • -b,-m

    Ignored for compatibility.

  • -B, --always-make

    Unconditionally make all targets.

  • -C dir, --directory=dir

    Change directory before reading the makefile.

  • -d

    Print all debugging information.

  • --debug[=options]

    Print basic debugging information, or specify what types of information to print (all, basic, verbose, implicit, jobs, makefile).

  • -e, --environment-overrides

    Environment variables override makefiles.

  • -f file, --file=file, --makefile=file

    Specify the makefile.

  • -h, --help

    Print usage and exit.

  • -i, --ignore-errors

    Ignore errors.

  • -I dir, --include-dir=dir

    Search a directory for included makefiles.

  • -k, --keep-going

    Keep going when some targets cannot be made.

  • -l [n], --load-average[=n], --max-load[=n]

    Obsolete. Load limit.

  • -L , --check-symlink-times

    Target file modification time considers the timestamp of symbolic links also.

  • -n, --just-print, --dry-run, --recon

    Print instead of executing.

  • -o file, --old-file=file, --assume-old=file

    Do not remake the old file.

  • -p, --print-data-base

    Print make’s internal database.

  • -q, --question

    Question mode, return exit status.

  • -r, --no-builtin-rules

    Disable the built-in implicit rules.

  • --no-builtin-variables

    Disable the built-in variable settings. The make option -R is not supported, it conflicts with the lsmake option -R res_req.

  • -s, --silent, --quiet

    Silent mode, do not echo commands.

  • -S, --no-keep-going, --stop

    Turns off -k.

  • -t, --touch

    Touch targets (just change modification time) instead of remaking them.

  • -v, --version

    Print the version number of make and exit.

  • -w, --print-directory

    Print the current directory.

  • --no-print-directory

    Turn off -w, even if it was turned on implicitly.

  • -W file, --what-if=file, --new-file=file, --assume-new=file

    Always consider the file to be new (do not change modification time).

  • --warn-undefined-variables

    Warn when an undefined variable is referenced.

target ...

Specifies targets to make.

Output: -y

Total Run Time

Total lsmake job run time, in the format hh:mm:ss

Most Concurrent Tasks

Maximum number of tasks that ran simultaneously; compare to Total Slots Allocated and Tasks Allocated per Slot to determine if parallel execution may have been limited by resource availability.

Retries Allowed

Maximum number of retries allowed (set by lsmake -x option)

Hosts and Number of Slots Allocated

The output is a single line showing each name and number pair separated by spaces, in the format:host_name number_slots [host_name number_slots] ...

Tasks Allowed per Slot

Maximum number of tasks allowed per slot (set by lsmake -c option)

Total Slots Allocated

Total number of slots actually allocated (may be limited by lsmake -j or lsmake -m options)

Output: -u

The lsmake.dat file is a simple text file, consisting of two values separated by a comma. The first value is the time in the format hh:mm:s, the second value is the number of tasks running at that time, for example:

23:13:39,2

The file is updated with a new line of information every second.

Limitations

If a submake in a makefile specifies options which are specific to lsmake, they are ignored. Only the command line options are used. The resource requirements of tasks in the remote task list are not considered when dispatching tasks.

See also

lstcsh(1), gmake(1)