About LSF on IBM EGO
Scalability—EGO enhances LSF scalability. Currently, the LSF scheduler has to deal with a large number of jobs. EGO provides management functionality for multiple schedulers that co-exist in one EGO environment. In LSF 9, although only a single instance of LSF is available on EGO, the foundation is established for greater scalability in follow-on releases that will allow multiple instances of LSF on EGO.
Robustness—In previous releases, LSF functioned as both scheduler and resource manager. EGO decouples these functions, making the entire system more robust. EGO reduces or eliminates downtime for LSF users while resources are added or removed.
Reliability—In situations where service is degraded due to noncritical failures such as sbatchd or RES, by default, LSF does not automatically restart the daemons. The EGO Service Controller can monitor all LSF daemons and automatically restart them if they fail. Similarly, the EGO Service Controller can also monitor and restart other critical processes such as lmgrd.
Additional scheduling functionality—EGO provides the foundation for EGO-enabled SLA, which provides LSF with additional and important scheduling functionality.
Centralized management and administration framework.
Single reporting framework—across various application heads built around EGO.
What is IBM EGO?
Enterprise Grid Orchestrator (EGO) allows developers, administrators, and users to treat a collection of distributed software and hardware resources on a shared computing infrastructure (cluster) as parts of a single virtual computer.
EGO assesses the demands of competing business services (consumers) operating within a cluster and dynamically allocates resources so as to best meet a company's overriding business objectives. These objectives might include
Reducing the time or the cost of providing key business services
Maximizing the revenue generated by existing computing infrastructure
Configuring, enforcing, and auditing service plans for multiple consumers
Ensuring high availability and business continuity through disaster scenarios
Simplifying IT management and reducing management costs
Consolidating divergent and mixed computing resources into a single virtual infrastructure that can be shared transparently between many business users
IBM EGO also provides a full suite of services to support and manage resource orchestration. These include cluster management, configuration and auditing of service-level plans, resource facilitation to provide fail-over if a master host goes down, monitoring and data distribution.
EGO is only sensitive to the resource requirements of business services; EGO has no knowledge of any run-time dynamic parameters that exist for them. This means that EGO does not interfere with how a business service chooses to use the resources it has been allocated.
How IBM EGO works
IBM Platform products work in various ways to match business service (consumer) demands for resources with an available supply of resources. While a specific clustered application manager or consumer (for example, an LSF cluster) identifies what its resource demands are, IBM EGO is responsible for supplying those resources. IBM EGO determines the number of resources each consumer is entitled to, takes into account a consumer’s priority and overall objectives, and then allocates the number of required resources (for example, the number of slots, virtual machines, or physical machines).
Once the consumer receives its allotted resources from IBM EGO, the consumer applies its own rules and policies. How the consumer decides to balance its workload across the fixed resources allotted to it is not the responsibility of EGO.
So how does IBM EGO know the demand? Administrators or developers use various EGO interfaces (such as the SDK or CLI) to tell EGO what constitutes a demand for more resources. When Platform LSF identifies that there is a demand, it then distributes the required resources based on the resource plans given to it by the administrator or developer.
For all of this to happen smoothly, various components are built into IBM EGO. Each EGO component performs a specific job.
IBM EGO components
IBM EGO comprises a collection of cluster orchestration software components. The following figure shows overall architecture and how these components fit within a larger system installation and interact with each other:
Key EGO concepts
- Consumers
A consumer represents an entity that can demand resources from the cluster. A consumer might be a business service, a business process that is a complex collection of business services, an individual user, or an entire line of business.
- EGO resources
Resources are physical and logical entities that can be requested by a client. For example, an application (client) requests a processor (resource) in order to run.
Resources also have attributes. For example, a host has attributes of memory, processor utilization, operating systems type, etc.
- Resource distribution tree
The resource distribution tree identifies consumers of the cluster resources, and organizes them into a manageable structure.
- Resource groups
Resource groups are logical groups of hosts. Resource groups provide a simple way of organizing and grouping resources (hosts) for convenience; instead of creating policies for individual resources, you can create and apply them to an entire group. Groups can be made of resources that satisfy a specific requirement in terms of OS, memory, swap space, CPU factor and so on, or that are explicitly listed by name.
- Resource distribution plans
The resource distribution plan, or resource plan, defines how cluster resources are distributed among consumers. The plan takes into account the differences between consumers and their needs, resource properties, and various other policies concerning consumer rank and the allocation of resources.
The distribution priority is to satisfy each consumer's reserved ownership, then distribute remaining resources to consumers that have demand.
- Services
A service is a self-contained, continuously running process that accepts one or more requests and returns one or more responses. Services may have multiple concurrent service instances running on multiple hosts. All IBM EGO services are automatically enabled by default at installation.
Run egosh to check service status.
If EGO is disabled, the egosh command cannot find ego.conf or cannot contact vemkd (not started), and the following message is displayed:You cannot run the egosh command because the administrator has chosen not to enable EGO in lsf.conf: LSF_ENABLE_EGO=N.- EGO user accounts
A user account is a IBM Platform system user who can be assigned to any role for any consumer in the tree. User accounts include optional contact information, a name, and a password.