PBS的配置

http://grid.tsinghua.edu.cn/home/liulk/wiki/computer/pbsconfig.html

内容提纲：

安装前的要求

安装

PBS的配置

起动PBS.

创建队列.

PBS Script samples

 

安装前的要求

如果是单机跑跑pbs, 那么什么都不要求. 如果你想在机群上pbs,那么再进行pbs安装之前你需要保证一些其他的软件都能工作.

• NFS 这个是为了在机群之间共享一个目录, 使得个个机器都能访问.
• Nis 这是是为了在机群上实现统一的用户认证.

安装

pbs的安装比较简单, 应该说是一个标准的linux的tarball安装方式. 安装过程主要有如下几步:

• 下载pbs的源码包 OpenPBS-2.3.12sc2.tar。
• 解压缩软件包

  $tar zxvpf OpenPBS-2.3.12sc2.tar
  

• 进入相应的目录, 配置,make

  $cd SPBS-1.0.0
  $./configure --enable-docs --disable-gui
  $make
  

  注意上面的选项

   --enable-docs 指定要编译文档
   --disable-gui 指定去掉图形界面. 因为图形界面要求需要tck支持, 而且似乎必须是tcl-8.0的版本, 高了会出错
   --enable-scp  先使用scp,再使用rcp进行数据的复制.
   

• 切换成root用户, 安装.

  $su
  #make install
  

  默认情况下, PBS会安装再/usr/spoole/PBS目录下. 该目录在./configure 的时候指定 --prefix=/you/want/dir来更改. 建议对于初次安装的人采用默认的设置.

PBS的配置

一般来说, PBS系统必须有一个server和至少一个mom, server负责作业的提交, mom接受server的控制,负责作业的执行. 假设pbs的根目录为 $PBS_HOME. server节点为CGSP48, mom节点为CGSP48和CGSP47,也就是CGSP48既作为共享server也作为mom, 而CGSP47只作为mom.

1. 配置server_name 编辑所有的节点的 $PBS_HOME/server_name,在里面写入选定的server的主机名. 例如

   CGSP48
   

2. 配置节点. 在CGSP48的 $PBS_HOME/server_priv目录下建立nodes文件.

   touch nodes
   

   在nodes文件写入你所有mom节点的名称:

   CGSP48 np=2
   CGSP47 np=3
   

   其中的np代表的是虚拟处理器的个数. 实际上也就是该节点最多可以同时跑多少个任务.
3. 配置mom节点,使其接受sever节点的控制. 编辑每个mom节点的 $PBS_HOME/mom_priv目录下的config文件(没有就自己建立). 写入如下信息

   $logevent 0x1ff
   $clienthost CGSP48
   

   其中:

   $logevent 指定的日志的级别, 基本上不用修改. 默认就很好了
   $clienthost 指定的server的地址
   

起动PBS.

pbs起动最好按如下顺序. mom, sever, sched. 而且必须以root身份起动. 命令如下:

# /usr/local/sbin/pbs_mom
# /usr/local/sbin/pbs_server -t create
# /usr/local/sbin/pbs_sched

其中, -t create 在第一次起动的时候要用, 用于创建一些初始化必要的环境. 以后起动就不在需要了.

创建队列.

PBS中的队列分为两种类型, 执行队列和路由队列. 下面是一个创建队列的脚本.

# 
# Create and define queue verylong 
# 
create queue verylong 
set queue verylong queue_type = Execution 
set queue verylong Priority = 40 
set queue verylong max_running = 10 
set queue verylong resources_max.cput = 72:00:00 
set queue verylong resources_min.cput = 12:00:01 
set queue verylong resources_default.cput = 72:00:00 
set queue verylong enabled = True 
set queue verylong started = True 
# 
# Create and define queue long 
# 
create queue long 
set queue long queue_type = Execution 
set queue long Priority = 60 
set queue long max_running = 10 
set queue long resources_max.cput = 12:00:00 
set queue long resources_min.cput = 02:00:01 
set queue long resources_default.cput = 12:00:00 
set queue long enabled = True 
set queue long started = True 
# 
# Create and define queue medium 
# 
create queue medium 
set queue medium queue_type = Execution 
set queue medium Priority = 80 
set queue medium max_running = 10 
set queue medium resources_max.cput = 02:00:00 
set queue medium resources_min.cput = 00:20:01 
set queue medium resources_default.cput = 02:00:00 
set queue medium enabled = True 
set queue medium started = True 
# 
# Create and define queue small 
# 
create queue small 
set queue small queue_type = Execution 
set queue small Priority = 100 
set queue small max_running = 10 
set queue small resources_max.cput = 00:20:00 
set queue small resources_default.cput = 00:20:00 
set queue small enabled = True 
set queue small started = True                   
# 
# Create and define queue default 
# 
create queue default 
set queue default queue_type = Route 
set queue default max_running = 10 
set queue default route_destinations = small 
set queue default route_destinations += medium 
set queue default route_destinations += long 
set queue default route_destinations += verylong 
set queue default enabled = True 
set queue default started = True 
# 
# Set server attributes. 
# 
set server scheduling = True 
set server max_user_run = 6 
set server acl_host_enable = True 
set server acl_hosts = * 
set server default_queue = default 
set server log_events = 63 
set server mail_from = adm 
set server query_other_jobs = True 
set server resources_default.cput = 01:00:00 
set server resources_default.neednodes = 1 
set server resources_default.nodect = 1 
set server resources_default.nodes = 1 
set server scheduler_iteration = 60 
set server default_node = 1#shared

该脚本定义了verylong long medium small 四个作业队列 和 一个default路由队列. 里面的几个比较重要的属性:

 enabled 指示作业队列可用. 也就是可以往里面添加新的作业了.
 actived 指示作业队列处于活动状态, 可以参与调度了. 好象是这个样子的. 反正两个都设置为true就可以了.
 sever的scheduling属性指示server开始调度. 这个属性非常的重要.
         为了这个属性, 我浪费了一个晚上. 因为只要该属性没有设置,那么服务器中的所有的作业就一直处于Q状态.
 

到此为止PBS的配置基本就完成了. 当然. PBS还有很多的参数可以设置. 但那就不是这个基本配置想给大家的. 大家自己去查PBS的手册把.

PBS Script samples

#LJRS -S /bin/bash
#LJRS -o script.out
#LJRS -j oe
#LJRS -q dpool
#LJRS -l nodes=8:ppn=1      （nodes：计算节点数，ppn：计算节点CPU数）
#LJRS -l walltime=48:00:00  （用户估计的最大计算时间，超时系统会自动中断作业）

limit -s unlimited
TMPFILE=`whoami`_mpich_gm.tmp
sed 's/c/g/g' $LJRS_NODEFILE > /tmp/$TMPFILE
GM_NODEFILE=/tmp/$TMPFILE
echo Working directory is $LJRS_O_WORKDIR
cd $LJRS_O_WORKDIR
echo Runing on host `hostname`
echo Starting Time is `date`
echo Directory is `pwd`
echo This jobs runs on the following processors:
echo `cat $GM_NODEFILE`
NPROCS=`wc -l < $GM_NODEFILE`
echo This job has allocated $NPROCS nodes

## User Parallel Program ###########

mpirun -v -machinefile $GM_NODEFILE -np $NPROCS ~/my_parallel.exe > out （修改为用户的并行作业执行命令）

####################################

rm -f /tmp/$TMPFILE
echo Ending Time is `date`

##########################################################################
#
# Script for submitting parallel Gaussian 03 jobs to the cluster.
#
###########################################################################
# To use this script, first make the following changes:
#
#  1. Add the G03 .com filename to "jobname=", but leave out the suffix .com.
#      Example:  jobname=input
#  2. Add the same filename to #LJRS -o with the suffix .err.
#      Example:  #LJRS -o water03.err
#  3. Specify the number of nodes(nodes = x) and processors per node
#      (ppn = y) needed for the job. Note that each job should typically use
#      both processors on each node(i.e., ppn = 2). Therefore, set nodes equal
#      to the total number of processors divided by 2.
#      Example: If the job needs 16 processors total, then:
#                #LJRS -l nodes=8:ppn=2
#     Note that the appropriate number of processors must be requested in
#      the .com file. In the above example, %nprocl=16 must be specified
#      before the job command line(i.e., the line beginning with "#").
#      (Note: Either the command nprocl or nproclinda can be used.)
#      If only one processor is required, then %nprocl does not need to be
#      specified. In the .g03 file, one processor is specified as:
#                #LJRS -l nodes=4:ppn=2
#  4. Set the maximum length of time the job will use.
#      Example:  #LJRS -l walltime=24:00:00
#  5. Make sure the input (.com) and submitting script (.g03) files are in
#      the same directory.
#
#  Submit the script using "qsub input.g03".
###########################################################################
# Lines that begin with #LJRS are PBS directives (not comments).
# True comments begin with "# " (i,e., # followed by a space).
###########################################################################

#LJRS -S /bin/bash
#LJRS -o water333.err
#LJRS -j oe
#LJRS -q dpool
#LJRS -l nodes=4:ppn=2        (与input.com文件保持一致）
#LJRS -l walltime=860:00:00  （用户估计的最大计算时间，超时系统会自动中断作业）
#LJRS -V

cat "$LJRS_NODEFILE"

#############################################################################
#  -S: shell the job will run under
#  -o: name of the queue error filename
#  -j: merges stdout and stderr to the same file
#  -l: resources required by the job: number of nodes and processors per node
#  -l: resources required by the job: maximun job time length
#############################################################################

# Define variable "jobname".

jobname=input                  （g03输入文件名，不包括.com扩展名）
username=`whoami`

ulimit -s unlimited

# Make a directory in scr and copy .com and .g03 file to there.

GAUSS_RUNDIR=/scratch/${username}
if [ ! -a $GAUSS_RUNDIR ]; then
   echo "Scratch directory $GAUSS_RUNDIR created."
   mkdir -p $GAUSS_RUNDIR
fi

cp $LJRS_O_WORKDIR/${jobname}.* $GAUSS_RUNDIR

ORIG_LJRS_O_WORKDIR=${LJRS_O_WORKDIR}
LJRS_O_WORKDIR=${GAUSS_RUNDIR}
cd $LJRS_O_WORKDIR

# Setup for Gaussian 03:
# =======================
# Make a scratch directory if it doesn't already exist.

GAUSS_SCRDIR=/scratch/${username}/${jobname}
if [ ! -a $GAUSS_SCRDIR ]; then
   echo "Scratch directory $GAUSS_SCRDIR created."
   mkdir -p $GAUSS_SCRDIR
fi
export GAUSS_SCRDIR
echo "Using $GAUSS_SCRDIR for temporary Gaussian 03 files."


# Define the location where Gaussian was installed and run
# a setup script, g03.profile.

g03root=/export/local/g03
source $g03root/g03/bsd/g03.profile
#source /export/local/g03/g03/bsd/g03.profile

# Define PATH to include location of LINDA

PATH=$PATH:/export/local/g03/g03/linda7.1/intel-linux2.4/bin

# Define node list

#echo $LJRS_NODEFILE $LJRS_JOBID > /tmp/g03log
sed 's/c/g/g' $LJRS_NODEFILE > $GAUSS_SCRDIR/tsnet.nodes
#cat /tmp/$2 > $LJRS_NODEFILE

G03_NODEFILE="$GAUSS_SCRDIR/tsnet.nodes"

GAUSS_LFLAGS="-mp 2 -nodefile $G03_NODEFILE"

# Export variable list

export PATH g03root GAUSS_LFLAGS

echo pbs nodefile:

cat $G03_NODEFILE

#Run a Gaussian command file, water03.com, redirecting output
#to a file, water03.log

echo "Starting Gaussian run at" `date`

time g03l < $GAUSS_RUNDIR/${jobname}.com >$GAUSS_RUNDIR/${jobname}.log

echo "Finished Gaussian run at" `date`

LJRS_O_WORKDIR=${ORIG_LJRS_O_WORKDIR}

echo $LJRS_O_WORKDIR

mv $GAUSS_RUNDIR/${jobname}.* $LJRS_O_WORKDIR
mv $GAUSS_RUNDIR/*.chk $LJRS_O_WORKDIR
echo "$GAUSS_SCRDIR"

rm -Rf $GAUSS_SCRDIR