Maui/Torque Job defered

今天在cluster上提交任务，发现提交之后一直显示处于站队状态（Q）。换了一个node之后发现可以正常运行。cluster的配置是一个head node带了10个child node， 所有Maui和TORQUE的配置均在head node上。版本信息：

Ubuntu 12.04.4 LTS

Torque PBS 2.5.12

Maui 3.3.1

 

qstat的状态

# qstat

Job id                    Name             User            Time Use S Queue

------------------------- ---------------- --------------- -------- - -----

66625.head               testpy19         qz                  0 Q temp

66626.child09            testpy19         qz                  0 R temp

 

追踪任务显示没有给相应的job分配任何资源

 

# checkjob 66625

checking job 66625

 

State: Idle  EState: Deferred

Creds:  user:qz  group:qz  class:batch  qos:DEFAULT

WallTime: 00:00:00 of 1:00:00:00

SubmitTime: Wed Oct 14 16:52:37

 (Time Queued  Total: 00:00:31  Eligible: 00:00:00)

 

 Total Tasks: 1

 

 Req[0]  TaskCount: 1  Partition: ALL

 Network: [NONE]  Memory >= 0  Disk >= 0  Swap >= 0

 Opsys: [NONE]  Arch: [NONE]  Features: [1][ppn=1]

 NodeCount: 1

 

 IWD: [NONE]  Executable:  [NONE]

 Bypass: 0  StartCount: 0

 PartitionMask: [ALL]

 Flags:       RESTARTABLE

 

 job is deferred.  Reason:  NoResources  (cannot create reservation for job '66625' (intital reservation attempt)

 )

 Holds:    Defer  (hold reason:  NoResources)

 PE:  1.00  StartPriority:  1

 cannot select job 66625 for partition DEFAULT (job hold active)

 

一开始怀疑是queue的配置或者maui的配置问题，但是因为只有在head node上有问题，其他child node都运行良好，所以问题不在配置。下一步检查各个node

 

# checknode head

checking node head

State:      Down  (in current state for 00:00:00)

Configured Resources: PROCS: 24  MEM: 15G  SWAP: 16G  DISK: 1M

Utilized   Resources: PROCS: 24

Dedicated  Resources: [NONE]

Opsys:         linux  Arch:      [NONE]

Speed:      1.00  Load:       0.120

Network:    [DEFAULT]

Features:   [temp][normal][mpi][long][bigmem]

Attributes: [Batch]

Classes:    [temp 24:24][normal 24:24][mpi 24:24][long 24:24]

 

Total Time:   INFINITY  Up:   INFINITY (96.56%)  Active:   INFINITY (42.95%)

 

Reservations:

NOTE:  no reservations on node

 

# checknode child09

checking node child09

 

State:      Idle  (in current state for 00:40:17)

Configured Resources: PROCS: 12  MEM: 31G  SWAP: 47G  DISK: 1M

Utilized   Resources: SWAP: 5290M

Dedicated  Resources: [NONE]

Opsys:         linux  Arch:      [NONE]

Speed:      1.00  Load:       0.000

Network:    [DEFAULT]

Features:   [temp][normal][mpi][long]

Attributes: [Batch]

Classes:    [temp 12:12][normal 12:12][mpi 12:12][long 12:12]

 

Total Time:   INFINITY  Up:   INFINITY (98.61%)  Active:   INFINITY (17.80%)

 

Reservations:

NOTE:  no reservations on node

 

很明显，head node 没有正常工作，但是pbsnodes显示head node的状态是free。仔细对比了两个node的状态之后，发现head node没有任何session (nsession=0)， 而且有一条错误信息表明spool 文件系统已满……

# pbsnodes head

head

     state = free

     np = 24

     properties = normal,bigmem,long,mpi,temp

     ntype = cluster

     status = rectime=1444852351,varattr=,jobs=,state=free,netload=124602597243261,gres=,message=ERROR: torque spool filesystem full,loadave=0.00,ncpus=24,physmem=264108356kb,availmem=276266268kb,totmem=295356736kb,idletime=128,nusers=0,nsessions=0,uname=Linux mobs-head 3.5.0-45-generic #68~precise1-Ubuntu SMP Wed Dec 4 16:18:46 UTC 2013 x86_64,opsys=linux

     gpus = 0

 

# pbsnodes child09

child09

     state = free

     np = 12

     properties = normal,long,mpi,temp

     ntype = cluster

     status = rectime=1444852532,varattr=,jobs=,state=free,netload=77659293583815,gres=,loadave=0.00,ncpus=12,physmem=32901268kb,availmem=43868400kb,totmem=49285264kb,idletime=9836391,nusers=4,nsessions=10,sessions=510 1075 1101 1217 1233 1260 1295 1423 10483 12024,uname=Linux mobs-child09 3.5.0-45-generic #68~precise1-Ubuntu SMP Wed Dec 4 16:18:46 UTC 2013 x86_64,opsys=linux

     gpus = 0

 

于是，删掉没有用的文件…… 大约26G……

# rm /var/spool/torque/server_logs/*

# rm /var/spool/torque/undelivered/*

 

重启pbs_mom之后，一切正常

# ps aux | grep pbs_mom

root      1169  0.0  0.0  32792 22428 ?        SLsl 15:32   0:00 pbs_mom -p

root     46849  0.0  0.0   9392   944 pts/1    S+   15:57   0:00 grep --color=auto pbs_mom

# kill -9 1169

# pbs_mom -p