Ways to improve the performance of IBM Platform Make:
Tune your makefile and increase parallelism
Process subdirectories in parallel
Adjust the number of tasks run depending on the file server load
Ensure tasks always run on the best cores available at the time
Compensate for file system latency
Analyze resource usage to improve performance and efficiency
You do not need to modify your makefile to use IBM Platform Make, but reorganizing the contents of the makefile to increase the parallelism might reduce the running time.
The smallest unit that IBM Platform Make runs in parallel is a single make rule. If your makefile has rules that include many steps, or rules that contain shell loops to build sub-parts of your project, IBM Platform Make runs the steps serially.
Increase the parallelism in your makefile by breaking up complex rules into groups of simpler rules. Steps that must run in sequence can use make dependencies to enforce the order. IBM Platform Make can then find more subtasks to run in parallel.
Whenever a command depends on results of a previous command, running the commands on different hosts may result in errors due to file system latency. The -x and -a options are two ways to prevent problems. Use -x to automatically rerun a command that has failed for any reason. Use -a when you have dependent targets that may run on different hosts, and you need to allow extra time in between for file synchronization. By default, the dependent target (if it runs on a different host) starts after a delay of 1 second.
For any target, the retry feature (-x) is useful to compensate for file system latency and minor errors. With this feature enabled, the system automatically reruns any command that fails. You control how many times the same command should be rerun (for example, if the number of retries is 1, the command is attempted twice before exiting).
For dependent targets, the -a option is most useful. Ideally, dependent targets run sequentially on the same execution host, and files generated or modified by the previous target are available immediately. However, the dependent target may run on a different host (if the first host is busy running another command, or the target has multiple dependencies). If you notice errors in these cases, use -a to define a larger buffer time to compensate for file system latency. By default, the buffer time is 1 second.
This feature allows time for the shared file system to synchronize client and server. When commands in a target finish, commands in a dependent target wait the specified time before starting on a different host. If the dependent target's commands start on the same execution host, there is no delay. Slower file systems require a longer delay, so configure this based on network performance at your site.
If retry is enabled, this buffer time also affects the timing of retry attempts. The interval between retries increases exponentially with each retry attempt. The time between the initial, failed attempt and the first retry is equal to the buffer time. For subsequent attempts, the interval between attempts is doubled each time.
For example, if the buffer time defined by -a is 3 seconds and the number of retries defined by -x is 4, the system will wait 3 seconds before the first retry, then wait 6 seconds for the second retry, then 12 seconds, then 24, and exit if the 4th retry fails. However, if the dependent target can start on the same execution host at any time before exiting, it does so immediately, because the delay between retries is only enforced when the dependent target runs on a different host.
When you run lsmake, you can use the summay (-y) and usage (-u) options to learn if resources are being used efficiently and if resource availability may be limiting performance.
Use -y to display information about the job run time, hosts and slots allocated, and the highest number of tasks that ran in parallel. With this information, you can know if you requested more slots than the job actually needed.
Summary output:
Total Run Time - Total lsmake job run time, in the format hh:mm:ss
Most Concurrent Tasks - Maximum number of tasks that ran simultaneously; compare to Total Slots Allocated and Tasks Allowed per Slot to determine if parallel execution may have been limited by resource availability
Retries Allowed - Maximum number of retries allowed (set by lsmake -x option)
Hosts and Number of Slots Allowed - Execution hosts allocated, and the number of slots allocated on each. The output is a single line showing each name and number pair separated by spaces, in the format: host_name number_of_slots
Tasks Allowed per Slot - Maximum number of tasks allowed per slot (set by lsmake -c option)
Total Slots Allocated - Total number of slots actually allocated (may be limited by lsmake -j or -m options)
Use -u to generate a data file tracking the number of tasks running over time, which tells you how many slots were actually used and when they were needed. This file is useful if you want to export the data to third-party charting applications.
lsmake.dat file format:
The file is a simple text file, each line consists of just two values, separated by a comma. The first value is the time in the format hh:mm:ss, the second value is the number of tasks running at that time, for example:
23:13:39,2
The file is updated with a new line of information every second.