shell 编程

SSHFS
挂在远程文件系统 基于ssh的
不用在多个文件中切换了
方便查看日志



--------------------

watch 命令
实时刷新执行程序
watch -d free 实时刷新内存变化

-----------

查看apache配置文件的时候有很多注释，需要从大量的#号中找有用的配置，挺费神的，可以用sed命令将 #号和换行过滤掉

sed -n '/#/!p' httpd.conf | sed -n '/^$/!p' | less

----------------------

awk vs sed
我觉着，如果是按行操作，就统统都用 sed
如果是不仅按行，还要按列操作，就统统都用 awk


在文本处理方面,perl相当于awk/sed/grep的一个高度概括
sed/awk/grep 的好处在于，它们大多数都是单一文件的软件，因此在小系统上要比 Perl 好使，Perl 动辄数十M……


检测服务器CPU的性能

#!/bin/sh

case "$ENV" in
"DP10 Prod")
  SERVERS="g1u2201 g1u2202 g2u1596 g2u1597 g1u2203 g1u2204 g1u2205 g1u2206 g2u1598 g2u1599 g2u1600 g2u1601";;
"Perf ITG")
  SERVERS="g2u1495 g2u1496 g2u1497";;
"Perf2 ITG")
  SERVERS="g4u1905 g4u1906 g4u1907 g4u1908";;
"Perf2 ITG Web")
  SERVERS="g4t2135g g4t2136g";;
"POC-Cloud-c0007615")
  SERVERS="c0007615";;
esac

shell_file=/tmp/$BUILD_TAG.txt

cat > $shell_file <<EOF
sleep JOB_COUNT
vmstat $INTERVAL $((MINUTES*60/INTERVAL))
EOF

trap "rm $shell_file" INT

/home/zfeng/remote/tools/remote_shell.sh $shell_file "$SERVERS"

#!/bin/sh

SHELL_TXT=$(cat $1)
shift
HOSTS="$@"
JOB_COUNT=1

echo "#==="
echo "# Following Shell will by run on hosts $HOSTS (JOB_COUNT is variable)"
echo "#---"
echo "${SHELL_TXT/JOB_COUNT/JOB_COUNT(${JOB_COUNT})}"
echo "# End"

KILL_TXT="echo 'Kill Jobs' "

for host in $HOSTS
do
ssh $host "

hostname
id
pwd
${SHELL_TXT/JOB_COUNT/${JOB_COUNT}}
exit
" 2>&1 | awk -v host=$host '{print host "|" strftime("%F %T", systime())"> " $0; fflush();}' &
KILL_TXT="$KILL_TXT ; kill %$JOB_COUNT"
JOB_COUNT=$((JOB_COUNT+1))
done

trap "$KILL_TXT" INT
wait

ProcessMonitor

cat > $shell_file <<EOF
COUNT=$((MINUTES*60/INTERVAL))
while [[ COUNT -gt 0 ]]; do
ps -ef| grep httpd | awk '{print \$(NF-5)}' | grep /opt | sort | uniq -c
let COUNT=COUNT-1
if $Once; then
  break
fi
sleep ${INTERVAL}
done
EOF

正则表达式
元字符 * . ^  $ [] \ 
用grep 命令来测试
注意 *只是匹配前面一个普通字符的0次或多次重复
hel*o  能匹配 hello  不能匹配 helxxxo  需要用   hel.*o 来匹配

^行首  $行尾

[]匹配字符集合  [0123456789] [0-9]
注意 ^出现在[]中 就变成了取反  [^b-d]  不在bd范围之间的字符

\ 反斜杠 转义字符

\<\>  精确匹配
\<the\>  them they 不匹配  在grep中使用时需要 加上双引号

正则表达式扩展
?  + () |

通配 和 元字符的意义不完全相同
  * ? ^
用ls 命令来测试

grep 全称 Global search Regular Expression and Print out
-c 匹配数量
-n 显示行号 
-v 不包含模式
-i    不区分大小写
-r 迭代子目录
-E 或字符  grep -vE "#|^$"  httpd.conf  查看非# 空行
-F 不支持正则表达式，按照字符串字面意思匹配


统计consumer的日志错误信息

#!/bin/bash

# Uncomment to next line to Debug
# set -x

# Script parses all consumer logs and displays a sorted list of exceptions and their frequency for Today.

# Variables

Mailto=[e-mail]
Today=$(date +"%Y-%m-%d")
Logs=/Bdata/prodlogs/dp10pro/consumer/g*/spf*/*.txt
Regex="\[hpsc\]".*"The portlet with title"
File=/tmp/tmp.$$
First=$(grep -ih  "$Today".*"$Regex" $Logs | head -1 | awk '{print $1,$2}')
Last=$(grep -ih  "$Today".*"$Regex" $Logs | tail -1 | awk '{print $1,$2}')

echo "Parsing logs, use ctrl -c to cancel ..."

echo "First log entry seen at "$First", last at "$Last"" > $File
echo "" >> $File

grep -ihA1  "$Today".*"$Regex" $Logs  | awk '{ if (NR % 3) printf("%s ", $0); else printf("%s\n", $0) }' | cut -d "," -f3,6 | sort | uniq -c | sort -rn >> $File

echo "Done. Emailing result...";
uuencode $File "Consumer.log" | mailx -s "DP10 Consumer log extract for $Today attached" $Mailto


# Clean up
rm -f $File

exit

sample

First log entry seen at 2014-03-09 00:00:52,392, last at 2014-03-09 09:42:43,001

    164  Warranty Check Main, failed to render. com.vignette.portal.portlet.website.InvocationFailedException: Portlet Render Failed. --
     10  PSI Content Results, failed to render. com.vignette.portal.portlet.website.InvocationFailedException: Portlet Render Failed. --
      3  SWD Related Links, failed to render. com.vignette.portal.portlet.website.PortletTimedOutException: The portlet with the UID
      3  SWD Message of the Day, failed to render. com.vignette.portal.portlet.website.PortletTimedOutException: The portlet with the UID
      3  SWD Left Promo Graphics, failed to render. com.vignette.portal.portlet.website.InvocationFailedException: Portlet Render Failed. --
      3  PSI SWD Options Menu, failed to render. com.vignette.portal.portlet.website.PortletTimedOutException: The portlet with the UID
      2  PSI Product Options Menu, failed to render. com.vignette.portal.portlet.website.InvocationFailedException: Portlet Render Failed. --
      2  PSI Message of the Day, failed to render. com.vignette.portal.portlet.website.PortletTimedOutException: The portlet with the UID
      2  PSI Home Right Promo Graphics, failed to render. com.vignette.portal.portlet.website.PortletTimedOutException: The portlet with the UID
      2  PSI Home Left Promo Graphics, failed to render. com.vignette.portal.portlet.website.PortletTimedOutException: The portlet with the UID
      1  Warranty Check Main, failed to render. com.vignette.portal.portlet.website.InvocationFailedException: Portlet Render Failed. 
      1  PSI Product Selector, failed to render. com.vignette.portal.portlet.website.InvocationFailedException: Portlet Render Failed. --