TWS LoadLeveler: Using and Administering

798 Pages·2008·7.65 MB·English

(cid:1)(cid:2)(cid:3)
Tivoli Workload Scheduler LoadLeveler

Using and Administering

Version 3 Release 5

SA22-7881-08

(cid:1)(cid:2)(cid:3)
Tivoli Workload Scheduler LoadLeveler

Using and Administering

Version 3 Release 5

SA22-7881-08
Note

Beforeusingthisinformationandtheproductitsupports,readtheinformationin“Notices”onpage745.

NinthEdition(November2008)

Thiseditionappliestoversion3,release5,modification0ofIBMTivoliWorkloadSchedulerLoadLeveler(product

numbers5765-E69and5724-I23)andtoallsubsequentreleasesandmodificationsuntilotherwiseindicatedinnew

editions.ThiseditionreplacesSA22-7881-07.Significantchangesoradditionstothetextandillustrationsare

indicatedbyaverticalline(|)totheleftofthechange.

IBMwelcomesyourcomments.Aformforreaders’commentsmaybeprovidedatthebackofthispublication,or

youcansendyourcommentstotheaddress:

InternationalBusinessMachinesCorporation

Department58HA,MailStationP181

2455SouthRoad

Poughkeepsie,NY12601-5400

UnitedStatesofAmerica

FAX(UnitedStates&Canada):1+845+432-9405

FAX(OtherCountries):

YourInternationalAccessCode+1+845+432-9405

IBMLink™ (UnitedStatescustomersonly):IBMUSM10(MHVRCFS)

Internete-mail:[e-mail]

Ifyouwantareply,besuretoincludeyourname,address,andtelephoneorFAXnumber.

Makesuretoincludethefollowinginyourcommentornote:
v Titl eand o rdernum ber ofthispu bl icatio n
v Page num bero rtopicre la tedt oyourcomm ent

WhenyousendinformationtoIBM,yougrantIBManonexclusiverighttouseordistributetheinformationinany

wayitbelievesappropriatewithoutincurringanyobligationtoyou.

©Copyright1986,1987,1988,1989,1990,1991bytheCondorDesignTeam.

©CopyrightInternationalBusinessMachinesCorporation1986,2008.Allrightsreserved.USGovernmentUsers

RestrictedRights-Use,duplicationordisclosurerestrictedbyGSAADPScheduleContractwithIBMCorp.
Contents

Figures. . . . . . . . . . . . . . . ix LoadLevelerforAIXandLoadLevelerforLinux

compatibility . . . . . . . . . . . . . . 35

Tables . . . . . . . . . . . . . . . xi RestrictionsforLoadLevelerforLinux . . . . 36

FeaturesnotsupportedinLoadLevelerforLinux 36

RestrictionsforLoadLevelerforAIXand
About this information. . . . . . . . xiii
LoadLevelerforLinuxmixedclusters . . . . 37
Whoshouldusethisinformation. . . . . . . xiii

Conventionsandterminologyusedinthis
Part 2. Configuring and managing
information . . . . . . . . . . . . . . xiii
the T W S LoadLeve ler e nvironmen t . 39
Prerequisiteandrelatedinformation. . . . . . xiv

Howtosendyourcomments . . . . . . . . xv

Chapter 4. Configuring the LoadLeveler
Summary of changes . . . . . . . . xvii environm e nt . . . . . . . . . . . . 41

Modifyingaconfigurationfile . . . . . . . . 42

Part 1. Overview of TWS DefiningLoadLeveleradministrators. . . . . . 43
Load Le veler con ce pts an d operation 1 Defining aLoadLevele rcluster . . . . . . . . 44

Choosingascheduler . . . . . . . . . . 44

Settingnegotiatorcharacteristicsandpolicies . . 45
Chapter 1. What is LoadLeveler? . . . . 3
Specifyingalternatecentralmanagers. . . . . 46
LoadLevel erb asics . . . . . . . . . . . . 4 Definingn etworkch aracter istics . . . . . . 47
LoadLeveler: Anet wo rk job m ana ge me nta nd Specifyin gfileand directoryloca tio ns . . . . 47
schedulingsy st em . . . . . . . . . . . . 4 Configurin gre cord ingactivi tyandlog fil es. . . 48
Jobdefin ition . . . . . . . . . . . . . 5 Settingupfi lesystem monitor ing . . . . . . 54
Ma chinedefin it ion . . . . . . . . . . . 6 DefiningL oad Lev elerma chinechara ct eri stic s . . . 54
HowLoadL evelersch edu le sjo bs . . . . . . . 7 Defini ngjobclasses thataL oadLevelerma ch in e
How LoadLeveler daemons proce ss job s . . . . . 8 willacce pt. . . . . . . . . . . . . . 55
Th emasterdae mon . . . . . . . . . . . 9 Spec ifying h ow m an yjo bs am ac hin ec an ru n . . 55
The Schedd daemon . . . . . . . . . . 10 Definingsecu ritym echan isms . . . . . . . . 56
The startdd aemon . . . . . . . . . . . 11 Config uringLo adLevelerto u se clu ste rse cu rity
The negoti atordaem on . . . . . . . . . 13 services. . . . . . . . . . . . . . . 57
The kbdddaem on . . . . . . . . . . . 14 Definingus age po lic ies for co nsu m abl ere so urc es. . 60
The gsmo nitordae m on . . . . . . . . . 14 Enabling suppo rtforbu lkd atatransfer andrCxt
TheLo adLeveler jobcycle . . . . . . . . . 16 blocks . . . . . . . . . . . . . . . . 61
L oadLevelerjo bs tates. . . . . . . . . . 19 Gather in gjo ba cc oun tin gd ata . . . . . . . . 61
Consumableres our ces. . . . . . . . . . . 22 Collecti ng jobresource data o ns eri ala nd pa ral lel
Consuma bleresourc e sa nd AI XW o rkl oad jobs . . . . . . . . . . . . . . . . 62
Manager . . . . . . . . . . . . . . 24 | Coll ec tin ga cco un ting in for ma tio nf or rec urr ing
Overviewof re ser vat ion s. . . . . . . . . . 25 || jobs . . . . . . . . . . . . . . . . 63
Fairshare sc hedulingove r vie w. . . . . . . . 27 Coll ec tin ga cco un ting d ata fo rre ser vat ion s. . . 63

Collectingjobresourcedatabasedonmachines 64
Chapter 2. Getting a quick start using Collecting job resource data based on events . . 64

the default configuration . . . . . . . 29 Collectingjobresourceinformationbasedonuser
Wha tyounee dtoknowbeforey o ub eg in . . . . 29 accounts . . . . . . . . . . . . . . 65
Using the defau lt configu ration files . . . . . . 29 Collectin gt he acc oun tin gi nfo rm ati on and
LoadL eve lerforL inuxquicksta rt . . . . . . . 30 storingiti nto files . . . . . . . . . . . 66
Quickins talla tion . . . . . . . . . . . 30 Produc ing acc ount in gr epo rts . . . . . . . 66
Quick configuratio n . . . . . . . . . . 30 Correlating AIXandL oadLev ele ra cco un ting
Quick verification . . . . . . . . . . . 30 records. . . . . . . . . . . . . . . 66
Post-insta llationcons ide rat ion s. . . . . . . . 31 64-bits u ppo rt for ac cou nti ng fun cti on s . . . . 67
StartingLoad Leveler . . . . . . . . . . 31 Exam ple:Setti ngu pjobaccou ntingfile s. . . . 67
Location ofdirectorie sf oll ow ing in stal lat ion . . 32 Managingjo bstatus thr oug hcontrolex pres s ion s. . 68
Howco ntro lexpr essionsa ffectjob s . . . . . 69

Chapter 3. What operating systems are Trackingjobprocesses. . . . . . . . . . . 70

QueryingmultipleLoadLevelerclusters. . . . . 71
supported by LoadLeveler?. . . . . . 35
Handlingswitch-tableerrors. . . . . . . . . 72

iii
Providingadditionaljob-processingcontrolsthrough | ConfiguringLoadLevelertosupportdata

installationexits. . . . . . . . . . . . . 72 || staging . . . . . . . . . . . . . . 114

Controllingthecentralmanagerschedulingcycle 73 Usinganexternalscheduler . . . . . . . . 115

HandlingDCEsecuritycredentials . . . . . 74 ReplacingthedefaultLoadLevelerscheduling

HandlinganAFStoken . . . . . . . . . 75 algorithmwithanexternalscheduler . . . . 116

Filteringajobscript . . . . . . . . . . 76 Customizingtheconfigurationfiletodefinean

Writingprologandepilogprograms . . . . . 77 externalscheduler. . . . . . . . . . . 118

Usingyourownmailprogram . . . . . . . 81 Stepsforgettinginformationaboutthe

LoadLevelercluster,itsmachines,andjobs . . 118

Chapter 5. Defining LoadLeveler Assigningresourcesanddispatchingjobs . . . 122
resource s to admin ister . . . . . . . 83 Example:Cha ngingsch edul ertypes. . . . . . 126

Preemptingandresumingjobs . . . . . . . 126
Stepsformodifyinganadministrationfile . . . . 83
Overviewofpreemption . . . . . . . . 127
Definingmachines . . . . . . . . . . . . 84
Planningtopreemptjobs . . . . . . . . 128
Planningconsiderationsfordefiningmachines . 85
Stepsforconfiguringaschedulertopreempt
Machinestanzaformatandkeywordsummary 86
jobs . . . . . . . . . . . . . . . 130
Examples:Machinestanzas . . . . . . . . 86
ConfiguringLoadLevelertosupportreservations 131
Definingadapters . . . . . . . . . . . . 86
Stepsforconfiguringreservationsina
Configuringdynamicadapters . . . . . . . 87
LoadLevelercluster . . . . . . . . . . 132
ConfiguringInfiniBandadapters . . . . . . 87
StepsforintegratingLoadLevelerwiththeAIX
Adapterstanzaformatandkeywordsummary 88
WorkloadManager . . . . . . . . . . . 137
Examples:Adapterstanzas . . . . . . . . 89
LoadLevelersupportforcheckpointingjobs . . . 139
Definingclasses . . . . . . . . . . . . . 89
Checkpointkeywordsummary . . . . . . 139
Usinglimitkeywords . . . . . . . . . . 89
Planningconsiderationsforcheckpointingjobs 140
Allowinguserstouseaclass . . . . . . . 92
AIXcheckpointandrestartlimitations . . . . 141
Classstanzaformatandkeywordsummary . . 92
Namingcheckpointfilesanddirectories . . . 145
Examples:Classstanzas . . . . . . . . . 93
Removingoldcheckpointfiles. . . . . . . 146
Definingusersubstanzasinclassstanzas . . . . 94
LoadLevelerschedulingaffinitysupport . . . . 146
Examples:Substanzas . . . . . . . . . . 95
ConfiguringLoadLevelertousescheduling
Definingusers . . . . . . . . . . . . . 97
affinity . . . . . . . . . . . . . . 147
Userstanzaformatandkeywordsummary. . . 97
LoadLevelermulticlustersupport. . . . . . . 148
Examples:Userstanzas . . . . . . . . . 98
ConfiguringaLoadLevelermulticluster . . . 150
Defininggroups. . . . . . . . . . . . . 99
|| Scale-acrossschedulingwithmulticlusters. . . 153
Groupstanzaformatandkeywordsummary . . 99
LoadLevelerBlueGenesupport . . . . . . . 155
Examples:Groupstanzas. . . . . . . . . 99
ConfiguringLoadLevelerBlueGenesupport 157
Definingclusters . . . . . . . . . . . . 100
BlueGenereservationsupport. . . . . . . 159
Clusterstanzaformatandkeywordsummary 100
BlueGenefairshareschedulingsupport . . . 159
Examples:Clusterstanzas . . . . . . . . 100
BlueGeneheterogeneousmemorysupport . . 160

BlueGenepreemptionsupport . . . . . . 160
Chapter 6. Performing additional
BlueGene/LHTCpartitionsupport. . . . . 160
administrator tasks. . . . . . . . . 103
Usingfairsharescheduling. . . . . . . . . 160

Settinguptheenvironmentforparalleljobs . . . 104 Fairshareschedulingkeywords . . . . . . 161

Schedulingconsiderationsforparalleljobs . . 104 Reconfiguringfairshareschedulingkeywords 163

Stepsforreducingjoblaunchoverheadfor Example:threegroupsshareaLoadLeveler

paralleljobs. . . . . . . . . . . . . 105 cluster. . . . . . . . . . . . . . . 164

Stepsforallowinguserstosubmitinteractive Example:twothousandstudentssharea

POEjobs. . . . . . . . . . . . . . 106 LoadLevelercluster . . . . . . . . . . 165

Settingupaclassforparalleljobs . . . . . 106 Queryinginformationaboutfairshare
|| Striping w he nsom en etworks fail . . . . . 107 schedulin g . . . . . . . . . . . . . 166

Settingupaparallelmasternode. . . . . . 108 Resettingfairsharescheduling . . . . . . 166

ConfiguringLoadLevelertosupportMPICH Savinghistoricdata . . . . . . . . . . 166

jobs . . . . . . . . . . . . . . . 108 Restoringsavedhistoricdata . . . . . . . 167

ConfiguringLoadLevelertosupportMVAPICH Procedureforrecoveringajobspool. . . . . . 167

jobs . . . . . . . . . . . . . . . 108

ConfiguringLoadLevelertosupport Chapter 7. Using LoadLeveler’s GUI to

MPICH-GMjobs . . . . . . . . . . . 109 perform ad minist rator tasks . . . . . 169

UsingtheBACKFILLscheduler . . . . . . . 110
Job-relatedadministrativeactions. . . . . . . 169
TipsforusingtheBACKFILLscheduler . . . 112
Machine-relatedadministrativeactions. . . . . 172
Example:BACKFILLscheduling . . . . . . 113

|| Datastaging. . . . . . . . . . . . . . 113

iv TWSLoadLeveler:UsingandAdministering
Part 3. Submitting and managing Checkpointingajob . . . . . . . . . . . 232

TWS LoadLeveler jobs . . . . . . 177
Chapter 10. Example: Using

commands to build, submit, and
Chapter 8. Building and submitting
manage jobs. . . . . . . . . . . . 235
jobs . . . . . . . . . . . . . . . 179

Buildingajobcommandfile . . . . . . . . 179
Chapter 11. Using LoadLeveler’s GUI
Usingmultiplestepsinajobcommandfile . . 180
to build, submit, and manage jobs . . 237
Examples:Jobcommandfiles . . . . . . . 181

Editingjobcommandfiles . . . . . . . . . 185 Buildingjobs . . . . . . . . . . . . . 237

Definingresourcesforajobstep . . . . . . . 185 Editingthejobcommandfile . . . . . . . . 249

|| Submittingjobsrequestingdatastaging . . . . 186 Submittingajobcommandfile . . . . . . . 250

Workingwithcoscheduledjobsteps. . . . . . 187 Displayingandrefreshingjobstatus. . . . . . 251

Submittingcoscheduledjobsteps. . . . . . 187 SortingtheJobswindow . . . . . . . . . 252

Determiningpriorityforcoscheduledjobsteps 187 Changingthepriorityofyourjobs . . . . . . 253

Supportingpreemptionofcoscheduledjobsteps 187 Placingajobonhold. . . . . . . . . . . 253

CoscheduledjobstepsandcommandsandAPIs 188 Releasingtheholdonajob. . . . . . . . . 253

Terminationofcoscheduledsteps. . . . . . 188 Cancelingajob. . . . . . . . . . . . . 254

Usingbulkdatatransfer. . . . . . . . . . 188 Modifyingconsumableresourcesandotherjob

Preparingajobforcheckpoint/restart . . . . . 190 attributes. . . . . . . . . . . . . . . 254

Preparingajobforpreemption . . . . . . . 193 Takingacheckpoint . . . . . . . . . . . 254

Submittingajobcommandfile . . . . . . . 193 Addingajobtoareservation . . . . . . . . 255

Submittingajobusingasubmit-onlymachine 194 Removingajobfromareservation . . . . . . 255

Workingwithparalleljobs . . . . . . . . . 194 Displayingandrefreshingmachinestatus . . . . 255

StepforcontrollingwhetherLoadLevelercopies SortingtheMachineswindow. . . . . . . . 257

environmentvariablestoallexecutingnodes. . 195 Findingthelocationofthecentralmanager . . . 257

Ensuringthatparalleljobsinaclusterrunon Findingthelocationofthepublicscheduling

thecorrectlevelsofPEandLoadLeveler machines. . . . . . . . . . . . . . . 258

software . . . . . . . . . . . . . . 195 Findingthetypeofschedulerinuse. . . . . . 258

Task-assignmentconsiderations . . . . . . 196 SpecifyingwhichjobsappearintheJobswindow 258

Submittingjobsthatusestriping . . . . . . 198 SpecifyingwhichmachinesappearinMachines

RunninginteractivePOEjobs . . . . . . . 203 window . . . . . . . . . . . . . . . 259

RunningMPICH,MVAPICH,andMPICH-GM SavingLoadLevelermessagesinafile . . . . . 259

jobs . . . . . . . . . . . . . . . 204

Examples:Buildingparalleljobcommandfiles 207 Part 4. TWS LoadLeveler

Obtainingstatusofparalleljobs . . . . . . 212
interfaces reference . . . . . . . 261
Obtainingallocatedhostnames . . . . . . 212

Workingwithreservations . . . . . . . . . 213
Chapter 12. Configuration file
Understandingthereservationlifecycle . . . 214

Creatingnewreservations . . . . . . . . 216 reference . . . . . . . . . . . . . 263

Submittingjobstorununderareservation . . 218 Configurationfilesyntax . . . . . . . . . 263
Removingb oun dj obs fromth e reservation . . 220 Numerical and alphab etic al con sta nts . . . . 264
Querying existing rese rvatio ns . . . . . . 221 Mathemati calo perators . . . . . . . . . 264
Modifying existing reservation s . . . . . . 221 64-bitsupport forconfig u rat ion fil ek eyw or ds
Canceling existing reservations . . . . . . 222 ande xpression s . . . . . . . . . . . 264
Submittingjo bsreque stingschedu lin ga ffin ity . . 222 Configu rationfilek eyw or dd es crip tio ns . . . . 265
Submitting and monitoring jobsinaL oadLeve ler User-definedk eyw ords . . . . . . . . . . 313
multicluste r . . . . . . . . . . . . . . 223 LoadLeveler variables . . . . . . . . . . 314
Stepsfor s ubm it ting jo bs in aL oad Le vel er Variables tousefor set tin gd ate s . . . . . . 319
multi clus terenvironm ent . . . . . . . . 224 Variables to use for setting times . . . . . . 320
Submittingand monitoringB lu eG en ejo bs . . . 226

Chapter 13. Administration file
Chapter 9. Managing submitted jobs 229
reference . . . . . . . . . . . . . 321

Queryingthestatusofajob . . . . . . . . 229
Administrationfilestructureandsyntax . . . . 321
Workingwithmachines . . . . . . . . . . 230
Stanzacharacteristics. . . . . . . . . . 323
Displayingcurrentlyavailableresources . . . . 230
Syntaxforlimitkeywords . . . . . . . . 324
Settingandchangingthepriorityofajob . . . . 230
64-bitsupportforadministrationfilekeywords 325
Example:Howdoesajob’spriorityaffect
Administrationfilekeyworddescriptions . . . . 327
dispatchingorder?. . . . . . . . . . . 231

Placingandreleasingaholdonajob . . . . . 232

Cancelingajob. . . . . . . . . . . . . 232

Contents v
Chapter 14. Job command file llstatus-Querymachinestatus . . . . . . . 512

reference . . . . . . . . . . . . . 357 llsubmit-Submitajob . . . . . . . . . . 531

llsummary-Returnjobresourceinformationfor
Jobcommandfilesyntax . . . . . . . . . 357
accounting . . . . . . . . . . . . . . 535
Serialjobcommandfile . . . . . . . . . 357

Paralleljobcommandfile . . . . . . . . 358
Chapter 17. Application programming
Syntaxforlimitkeywords . . . . . . . . 358
64-bits upp ortfo rjobcom m and fi lek ey wo rds 358 interface s (A PIs) . . . . . . . . . . 541

Jobcommandfilekeyworddescriptions . . . . 359 64-bitsupportfortheLoadLevelerAPIs . . . . 543

Jobcommandfilevariables. . . . . . . . 399 LoadLevelerforAIXAPIs . . . . . . . . 543

Run-timeenvironmentvariables . . . . . . 400 LoadLevelerforLinuxAPIs . . . . . . . 544

Jobcommandfileexamples . . . . . . . 401 AccountingAPI . . . . . . . . . . . . 544

GetHistorysubroutine . . . . . . . . . 545

Chapter 15. Graphical user interface llacctvaluserexit . . . . . . . . . . . 547

(GUI) reference. . . . . . . . . . . 403 CheckpointingAPI . . . . . . . . . . . 548

ckptsubroutine. . . . . . . . . . . . 549
StartingtheGUI . . . . . . . . . . . . 403
ll_ckptsubroutine. . . . . . . . . . . 550
SpecifyingGUIoptions . . . . . . . . . 404
ll_init_ckptsubroutine . . . . . . . . . 553
TheLoadLevelermainwindow . . . . . . 404
ll_set_ckpt_callbackssubroutine . . . . . . 555
GettinghelpusingtheGUI. . . . . . . . 405
ll_unset_ckpt_callbackssubroutine . . . . . 556
DifferencesbetweenLoadLeveler’sGUIand
ConfigurationAPI. . . . . . . . . . . . 557
othergraphicaluserinterfaces. . . . . . . 406
ll_config_changedsubroutine . . . . . . . 558
GUItypographicconventions . . . . . . . 406
ll_read_configsubroutine . . . . . . . . 559
64-bitsupportfortheGUI . . . . . . . . 407
DataaccessAPI . . . . . . . . . . . . 560
CustomizingtheGUI. . . . . . . . . . . 407
UsingthedataaccessAPI . . . . . . . . 560
SyntaxofanXloadlfile . . . . . . . . . 407
UnderstandingtheLoadLevelerdataaccess
Modifyingwindowsandbuttons. . . . . . 408
objectmodel. . . . . . . . . . . . . 561
Creatingyourownpull-downmenus . . . . 409
UnderstandingtheBlueGeneobjectmodel . . 562
CustomizingfieldsontheJobswindowandthe
UnderstandingtheClassobjectmodel . . . . 562
Machineswindow. . . . . . . . . . . 409
UnderstandingtheClusterobjectmodel . . . 563
Modifyinghelppanels . . . . . . . . . 410
UnderstandingtheFairshareobjectmodel. . . 563

UnderstandingtheJobobjectmodel. . . . . 564
Chapter 16. Commands . . . . . . . 411
UnderstandingtheMachineobjectmodel . . . 565
llacctmrg-Collectmachinehistoryfiles . . . . 413
UnderstandingtheMClusterobjectmodel. . . 566
llbind-Bindjobstepstoareservation . . . . . 415
UnderstandingtheReservationsobjectmodel 566
llcancel-Cancelasubmittedjob . . . . . . . 421
UnderstandingtheWlmstatobjectmodel . . . 567
llchres-Changeattributesofareservation . . . 424
ll_deallocatesubroutine . . . . . . . . . 568
llckpt-Checkpointarunningjobstep . . . . . 430
ll_free_objssubroutine . . . . . . . . . 569
llclass-Queryclassinformation . . . . . . . 433
ll_get_datasubroutine . . . . . . . . . 570
llclusterauth-Generatespublicandprivatekeys 438
ll_get_objssubroutine . . . . . . . . . 624
llctl-ControlLoadLevelerdaemons. . . . . . 439
ll_next_objsubroutine . . . . . . . . . 627
llextRPD-ExtractdatafromanRSCTpeerdomain 443
ll_querysubroutine . . . . . . . . . . 628
llfavorjob-Reordersystemqueuebyjob . . . . 447
ll_reset_requestsubroutine . . . . . . . . 629
llfavoruser-Reordersystemqueuebyuser . . . 449
ll_set_requestsubroutine . . . . . . . . 630
llfs-Fairshareschedulingqueriesandoperations 450
ExamplesofusingthedataaccessAPI . . . . 633
llhold-Holdorreleaseasubmittedjob . . . . 454
ErrorhandlingAPI . . . . . . . . . . . 639
llinit-InitializemachinesintheLoadLeveler
ll_errorsubroutine. . . . . . . . . . . 640
cluster. . . . . . . . . . . . . . . . 457
FairshareschedulingAPI . . . . . . . . . 641
llmkres-Makeareservation . . . . . . . . 459
ll_fair_sharesubroutine . . . . . . . . . 642
llmodify-Changeattributesofasubmittedjob
ReservationAPI . . . . . . . . . . . . 643
step . . . . . . . . . . . . . . . . 464
ll_bindsubroutine. . . . . . . . . . . 645
llmovejob-Moveasingleidlejobfromthelocal
ll_change_reservationsubroutine. . . . . . 648
clustertoanothercluster . . . . . . . . . 470
ll_init_reservation_paramsubroutine . . . . 652
llmovespool-Movejobrecords . . . . . . . 472
ll_make_reservationsubroutine . . . . . . 653
llpreempt-Preemptasubmittedjobstep . . . . 474
ll_remove_reservationsubroutine. . . . . . 658
llprio - Change the user priority of submitted job || ll_remove_reservation_ xtndsubro u tin e. . . . 660
steps . . . . . . . . . . . . . . . . 477
SubmitAPI . . . . . . . . . . . . . . 663
llq-Queryjobstatus. . . . . . . . . . . 479
llfree_job_infosubroutine . . . . . . . . 664
llqres-Queryareservation. . . . . . . . . 500
llsubmitsubroutine . . . . . . . . . . 665
llrmres-Cancelareservation . . . . . . . . 508
monitor_programuserexit. . . . . . . . 667
llrunscheduler-Runthecentralmanager’s
WorkloadmanagementAPI . . . . . . . . 668
schedulingalgorithm. . . . . . . . . . . 511
ll_clustersubroutine . . . . . . . . . . 669

vi TWSLoadLeveler:UsingandAdministering
ll_cluster_authsubroutine . . . . . . . . 671 HowdoIfindmyremotejob? . . . . . . 716

ll_controlsubroutine . . . . . . . . . . 673 Whywon’tmyremotejobrun? . . . . . . 717

ll_modifysubroutine . . . . . . . . . . 677 Whydoesllq-Xallshownojobsrunningwhen

ll_move_jobsubroutine . . . . . . . . . 681 therearejobsrunning? . . . . . . . . . 717

ll_move_spoolsubroutine . . . . . . . . 683 TroubleshootinginaBlueGeneenvironment. . . 717

ll_preemptsubroutine . . . . . . . . . 686 WhydoallofmyBlueGenejobsfaileven

ll_preempt_jobssubroutine. . . . . . . . 688 thoughllstatusshowsthatBlueGeneispresent? 718

ll_run_schedulersubroutine . . . . . . . 691 WhydoesllstatusshowthatBlueGeneis

ll_start_job_extsubroutine . . . . . . . . 692 absent? . . . . . . . . . . . . . . 718

ll_terminate_jobsubroutine. . . . . . . . 696 WhydidmyBlueGenejobfailwhenthejob

wassubmittedtoaremotecluster? . . . . . 718
Appendix A. Troubleshooting | Why doesllmk res o rllchre sreturn ″I nsu ffi cien t
LoadLeve ler . . . . . . . . . . . . 699 | resou rcest omeet the reques t″fora BlueGene

| reservationwhenresourcesappeartobe
Frequentlyaskedquestions. . . . . . . . . 699
|| available?. . . . . . . . . . . . . . 719
Whywon’tLoadLevelerstart?. . . . . . . 700
Helpfulhints . . . . . . . . . . . . . 719
Whywon’tmyjobrun?. . . . . . . . . 700
Scalingconsiderations . . . . . . . . . 719
Whywon’tmyparalleljobrun? . . . . . . 703
Hintsforrunningjobs . . . . . . . . . 720
Whywon’tmycheckpointedjobrestart? . . . 704
Hintsforusingmachines . . . . . . . . 723
Whywon’tmysubmit-onlyjobrun? . . . . 705
HistoryfilesandSchedd . . . . . . . . 724
Whywon’tmyjobrunonaclusterwithboth
GettinghelpfromIBM . . . . . . . . . . 724
AIXandLinuxmachines? . . . . . . . . 705

| Whywon’tmyjobrunwhenschedulingaffinity
Appendix B. Sample command output 725
|| isenabledonx86andx86_64systems?. . . . 705
W hydoes ajo bst ayin thePe nding(or Sta rtin g) llclass-lcom ma ndoutput listing. . . . . . . 725
state? . . . . . . . . . . . . . . . 706 llq-lc om mandout putlist ing . . . . . . . . 727
What ha pp en sto ru nn ing jo bs wh en am ac hin e llq -l command output listing fo ra B lue Ge ne
goesd own? . . . . . . . . . . . . . 706 ena b ledsystem . . . . . . . . . . . . . 729
Why won’t m yj obs ru nt hat w ere dir ect ed to an llq-l-x comman do utp ut lis ting . . . . . . . 730
idlep ool? . . . . . . . . . . . . . 708 llst at us -lcomman doutpu tlistin g . . . . . . 733
Wha thapp en sif th ece nt ral ma nag er isn ’t llstatus -l -bcomma ndoutp utlisti ng . . . . . 733
opera ting? . . . . . . . . . . . . . 708 llstatus -B co mmandou tputli sting . . . . . . 735
HowdoIr ec ove rr eso urc es allo ca ted by a llstatus -P command output listing . . . . . . 736
Sched dm achine? . . . . . . . . . . . 710 llsumm ary -l-xcomm ando utputl ist ing . . . . 736
Whyca n’tIfinda c ore fi leo nL in ux? . . . . 710 llsummary -l -x command output listing fo ra Blu e
Why amI s eeing i ncon siste nci esinm yl lfs Gene-enabl ed sy stem. . . . . . . . . . . 738

output? . . . . . . . . . . . . . . 711
Whydo n’t Is ee my jo bw he nI iss ue the llq Appendix C. LoadLeveler port usage 741

command? . . . . . . . . . . . . . 711

Whathappensiferrorsarefoundinmy Accessibility features for TWS

configurationoradministrationfile?. . . . . 711 LoadLeveler . . . . . . . . . . . . 743

Otherquestions . . . . . . . . . . . 712
Accessibilityfeatures. . . . . . . . . . . 743
Troubleshootinginamulticlusterenvironment . . 714
Keyboardnavigation. . . . . . . . . . . 743
HowdoIdetermineifIaminamulticluster
IBMandaccessibility. . . . . . . . . . . 743
environment? . . . . . . . . . . . . 714

HowdoIdeterminehowmymulticluster
Notices . . . . . . . . . . . . . . 745
environmentisdefinedandwhatarethe
inboundand ou tbound host sdefi ned for each Trademark s . . . . . . . . . . . . . . 746

cluster? . . . . . . . . . . . . . . 714
Whyis my m ult iclu ste re nvi ron me nt no t Glossary . . . . . . . . . . . . . 749

enabled? . . . . . . . . . . . . . . 714

HowdoIfindlogmessagesfrommy Index . . . . . . . . . . . . . . . 753

multicluster-definedinstallationexits? . . . . 715

Whywon’tmyremotejobbesubmittedor

moved? . . . . . . . . . . . . . . 715

WhydidtheCLUSTER_REMOTE_JOB_FILTER

notupdatethejobwithallofthestatementsI

defined? . . . . . . . . . . . . . . 716

Contents vii
viii TWSLoadLeveler:UsingandAdministering