Shell Scrap

Table of Contents

Flow Control

if

  • syntax
    # First form
    
    if condition ; then
        commands
    fi
    
    # Second form
    
    if condition ; then
        commands
    else
        commands
    fi
    
    # Third form
    
    if condition ; then
        commands
    elif condition ; then
        commands
    fi
    
  • Example
    if [ -f .bash_profile ]; then
        echo "You have a .bash_profile. Things are fine."
    else
        echo "Yikes! You have no .bash_profile!"
    fi
    
  • Condition

    Expression Description

    • -d file True if file is a directory.
    • -e file True if file exists.
    • -f file True if file exists and is a regular file.
    • -L file True if file is a symbolic link.
    • -r file True if file is a file readable by you.
    • -w file True if file is a file writable by you.
    • -x file True if file is a file executable by you.
    • file1 -nt file2 True if file1 is newer than (according to modification time) file2
    • file1 -ot file2 True if file1 is older than file2
    • -z string True if string is empty.
    • -n string True if string is not empty.
    • string1 = string2 True if string1 equals string2.
    • string1 != string2 True if string1 does not equal string2.
    • -a (AND) and -o (OR). The -a and -o operators are similar to the && and || operators used with exit statuses.

case

  • syntax
    case word in
        patterns ) statements ;;
    esac
    
  • example
    #!/bin/bash
    
    echo -n "Enter a number between 1 and 3 inclusive > "
    read character
    case $character in
        1 ) echo "You entered one."
            ;;
        2 ) echo "You entered two."
            ;;
        3 ) echo "You entered three."
            ;;
        * ) echo "You did not enter a number"
            echo "between 1 and 3."
    esac
    

Loops

  • while
    • example
      #!/bin/bash
      
      number=0
      while [ $number -lt 10 ]; do
          echo "Number = $number"
          number=$((number + 1))
      done
      
  • until

    The until command works exactly the same way, except the block of code is repeated as long as the condition is false.

    • Example
      #!/bin/bash
      
      number=0
      until [ $number -ge 10 ]; do
          echo "Number = $number"
          number=$((number + 1))
      done
      
  • for
    • syntax
      for variable in words; do
          statements
      done
      

Keyboard Input

read

  • example
    #!/bin/bash 
    
    echo -n "Enter some text > "
    read text
    echo "You entered: $text"
    
  • option -t (time)

    The -t option followed by a number of seconds provides an automatic timeout for the read command.

    #!/bin/bash
    echo -n "Hurry up and type something! > "
    if read -t 3 response; then
        echo "Great, you made it in time!"
    else
        echo "Sorry, you are too slow!"
    fi
    
  • option -s

    The -s option causes the user's typing not to be displayed.

Parameters

$#

the shell maintains a variable called $# that contains the number of items on the command line

shift

shift is a shell builtin that operates on the positional parameters. Each time you invoke shift, it "shifts" all the positional parameters down by one. $2 becomes $1, $3 becomes $2, $4 becomes $3, and so on.

Errors and Signals and Traps

Exit status

  • Checking the exit status

    First, you can examine the contents of the $? environment variable. $? will contain the exit status of the last command executed.

    [me] $ true; echo $?
    0
    [me] $ false; echo $?
    1
    

Creating safe temporary files

The preferred technique is to write them in a local directory such as ~/tmp (a tmp subdirectory in the user's home directory.) If you must write files in /tmp, you must take steps to make sure the file names are not predictable. Predictable file names allow an attacker to create symbolic links to other files that the attacker wants you to overwrite.

example

TEMP_FILE=$TEMP_DIR/printfile.$$.$RANDOM
  • $$ shell variable to embed the process id (pid) of the program
  • $RANDOM shell variable to append a random number to the file name

Debug

Watching your script run

  • Method1
    #!/bin/bash -x
    
  • Method2

    you can use the set command within your script to turn tracing on and off. Use set -x to turn tracing on and set +x to turn tracing off. For example.:

    #!/bin/bash
    
    number=1
    
    set -x
    if [ $number = "1" ]; then
        echo "Number equals 1"
    else
        echo "Number does not equal 1"
    fi
    set +x
    

trap command

其基本的语法是:

trap 'command' signal

其中signal是要捕获的信号,command是捕获到指定的信号之后,所要执行的命令。可以用kill –l命令看到系统中全部可用的信号名,捕获信号后所执行的命令可以是任何一条或多条合法的shell语句,也可以是一个函数名。

shell脚本在执行时,会产生三个所谓的“伪信号”,(之所以称之为“伪信号”是因为这三个信号是由shell产生的,而其它的信号是由操作系统产生的),通过使用trap命令捕获这三个“伪信号”并输出相关信息对调试非常有帮助。

EXIT  从一个函数中退出或整个脚本执行完毕 
ERR   当一条命令返回非零状态时(代表命令执行不成功) 
DEBUG 脚本中每一条命令执行之前
  • example
    • ERR
    ERRTRAP()
         {
           echo "[LINE:$1] Error: Command or function exited with status $?"
         }
         foo()
         {
           return 1;
         }
         trap 'ERRTRAP $LINENO' ERR
        abc
        foo
    
    • DEBUG
    #!/bin/bash
    trap 'echo “before execute line:$LINENO, a=$a,b=$b,c=$c”' DEBUG
    a=1
    if [ "$a" -eq 1 ]
    then
        b=2
    else
        b=1
    fi
    c=3
    echo "end"
    

tee command

tee命令会从标准输入读取数据,将其内容输出到标准输出设备,同时又可将内容保存成文件。

运行这个脚本,实际输出的却不是本机的ip地址,而是广播地址,这时我们可以借助tee命令,输出某些中间结果,将上述脚本片段修改为:

ipaddr=`/sbin/ifconfig | grep 'inet addr:' | grep -v '127.0.0.1'
| tee temp.txt | cut -d : -f3 | awk '{print $1}'`
echo $ipaddr

之后,将这段脚本再执行一遍,然后查看temp.txt文件的内容:

$ cat temp.txt
inet addr:192.168.0.1  Bcast:192.168.0.255  Mask:255.255.255.0

use debug hook

使用DEBUG宏来控制是否要输出调试信息,在shell脚本中我们同样可以使用这样的机制,如下列代码所示:

if [ “$DEBUG” = “true” ]; then
echo “debugging”  #此处可以输出调试信息
fi
DEBUG()
{
    if [ "$DEBUG" = "true" ]; then
        $@  
    fi
}
a=1
DEBUG echo "a=$a"

command

-n 只读取shell脚本,但不实际执行 -x 进入跟踪方式,显示所执行的每一条命令 -c "string" 从strings中读取命令 {sh -c 'a=1;b=2;let c=$a+$b;echo "c=$c"'}

  • 对"-x"选项的增强

    几个shell内置的环境变量:

    • $LINENO 代表shell脚本的当前行号,类似于C语言中的内置宏_LINE__
    • $FUNCNAME 函数的名字,类似于C语言中的内置宏_func__,但宏_func_只能代表当前所 在的函数名,而$FUNCNAME的功能更强大,它是一个数组变量,其中包含了整个调用链上所有的函数的名字,故变量${FUNCNAME1}代表shell脚本当前正在执行的函数的名字,而变量${FUNCNAME2}则代表调用函数${FUNCNAME1} 的函数的名字,余者可以依此类推。
    • $PS4 $PS4的值将被显示在“-x”选项输出的每一条命令的前面
    $ export PS4='+{$LINENO:${FUNCNAME[0]}} '
    

Parameter expansion

Simple usage

$PARAMETER
${PARAMETER}

Indirection

${PARAMETER} The referenced parameter is not PARAMETER itself, but the parameter whose name is stored as the value of PARAMETER. If the parameter PARAMETER has the value "TEMP", then ${!PARAMETER} will expand to the value of the parameter named TEMP

Case modification

${PARAMETER^} #modifies the first character to uppercase,
${PARAMETER^^} # all characters are converted
${PARAMETER,} #  , operator to lowercase
${PARAMETER,,}
${PARAMETER~} #~ reverses the case of first letter of words in the variabl
${PARAMETER~~}

Variable name expansion

${!PREFIX*} #This expands to a list of all set variable names beginning with the string PREFIX. 
${!PREFIX@}

Substring removal

${PARAMETER#PATTERN}  #The operator "#" will try to remove the shortest text matching the pattern, while "##" tries to do it with the longest text matching. 
${PARAMETER##PATTERN}
${PARAMETER%PATTERN} #except that Bash now tries to match the pattern from the end of the string
${PARAMETER%%PATTERN}

Search and replace

${PARAMETER/PATTERN/STRING}
${PARAMETER//PATTERN/STRING}
${PARAMETER/PATTERN} #removed
${PARAMETER//PATTERN}

Anchoring Additionally you can "anchor" an expression: A # (hashmark) will indicate that your expression is matched against the beginning portion of the string, a % (percent-sign) will do it for the end portion.

MYSTRING=xxxxxxxxxx
echo ${MYSTRING/#x/y}  # RESULT: yxxxxxxxxx
echo ${MYSTRING/%x/y}  # RESULT: xxxxxxxxxy

String length

${#PARAMETER}

Substring expansion

${PARAMETER:OFFSET}
${PARAMETER:OFFSET:LENGTH}

Use a default value

If the parameter PARAMETER is unset (never was defined) or null (empty), this one expands to WORD, otherwise it expands to the value of PARAMETER, as if it just was ${PARAMETER}. If you omit the : (colon), like shown in the second form, the default value is only used when the parameter was unset, not when it was empty.

${PARAMETER:-WORD}

${PARAMETER-WORD}

Assign a default value

This one works like the using default values, but the default text you give is not only expanded, but also assigned to the parameter, if it was unset or null. Equivalent to using a default value, when you omit the : (colon), as shown in the second form, the default value will only be assigned when the parameter was unset.

${PARAMETER:=WORD}

${PARAMETER=WORD}

Use an alternate value

This form expands to nothing if the parameter is unset or empty. If it is set, it does not expand to the parameter's value, but to some text you can specify:

${PARAMETER:+WORD}

${PARAMETER+WORD}
echo "The Java application was installed and can be started.${JAVAPATH:+ NOTE: JAVAPATH seems to be set}"

Display error if null or unset

If the parameter PARAMETER is set/non-null, this form will simply expand it. Otherwise, the expansion of WORD will be used as appendix for an error message:

  • an interactive shell has $? to a non-zero value
  • a non-interactive shell exits with a non-zero exit code
${PARAMETER:?WORD}

${PARAMETER?WORD}

Arithmetic and command expansion

#Arithmetic expansion
$(( EXPRESSION ))
$[ EXPRESSION ]  #Please do not use the second form $[ … ]! It's deprecated. 
#Command substitution
$( COMMAND )
` COMMAND `

Brace expansion

{string1,string2,...,stringN}
{<START>..<END>}
<PREAMBLE>{........}
{........}<POSTSCRIPT>
<PREAMBLE>{........}<POSTSCRIPT>

aa


String Operators

Command

!$ !$是一个特殊的环境变量,它代表了上一个命令的最后一个字符串。如:你可能会这样: $mkdir mydir $mv mydir yourdir $cd yourdir 可以改成: $mkdir mydir \(mv !\) yourdir \(cd !\) sudo !! 以root的身份执行上一条命令 。场景举例:比如Ubuntu里用apt-get安装软件包的时候是需要root身份的,我们经常会忘记在apt-get前加sudo。每次不得不加上sudo再重新键入这行命令,这时可以很方便的用sudo !!完事。(陈皓注:在shell下,有时候你会输入很长的命令,你可以使用!xxx来重复最近的一次命令,比如,你以前输入过,vi /where/the/file/is, 下次你可以使用 !vi 重得上次最近一次的vi命令。) cd –回到上一次的目录 。场景举例:当前目录为/home/a,用cd ../b切换到/home/b。这时可以通过反复执行cd –命令在/home/a和/home/b之间来回方便的切换。(陈皓注:cd ~ 是回到自己的Home目录,cd ~user,是进入某个用户的Home目录)‘ALT+.’ or ‘<ESC> .’热建alt+. 或 esc+. 可以把上次命令行的参数给重复出来。 ^oldnew 替换前一条命令里的部分字符串。场景:echo "wanderful",其实是想输出echo "wonderful"。只需要ao就行了,对很长的命令的错误拼写有很大的帮助。(陈皓注:也可以使用 !!:gs/old/new) du -s * | sort -n | tail 列出当前目录里最大的10个文件。 :w !sudo tee % 在vi中保存一个只有root可以写的文件 date -d@1234567890 时间截转时间 > file.txt 创建一个空文件,比touch短。 mtr coolshell.cn mtr命令比traceroute要好。在命令行前加空格,该命令不会进入history里。 echo “ls -l” | at midnight 在某个时间运行某个命令。 curl -u user:pass -d status=”Tweeting from the shell” http://twitter.com/statuses/update.xml 命令行的方式更新twitter。 curl -u username –silent “https://mail.google.com/mail/feed/atom” | perl -ne ‘print “\t” if <name>; print “$2\n” if <(title|name)>(.*)<\\1>/;’检查你的gmail未读邮件 ps aux | sort -nk +4 | tail 列出头十个最耗内存的进程 man ascii 显示ascii码表。场景:忘记ascii码表的时候还需要google么?尤其在天朝网络如此“顺畅”的情况下,就更麻烦在GWF多应用一次规则了,直接用本地的man ascii吧。 ctrl-x e 快速启动你的默认编辑器(由变量$EDITOR设置)。 netstat –tlnp 列出本机进程监听的端口号。(陈皓注:netstat -anop 可以显示侦听在这个端口号的进程) tail -f path/to/file.log | sed '/Finished: SUCCESS$ q' 当file.log里出现Finished: SUCCESS时候就退出tail,这个命令用于实时监控并过滤log是否出现了某条记录。 ssh user@server bash < /path/to/local/script.sh 在远程机器上运行一段脚本。这条命令最大的好处就是不用把脚本拷到远程机器上。 ssh user@host cat /path/to/remotefile | diff /path/to/localfile - 比较一个远程文件和一个本地文件 net rpc shutdown -I ipAddressOfWindowsPC -U username%password 远程关闭一台Windows的机器 screen -d -m -S somename ping myrouter 后台运行一段不终止的程序,并可以随时查看它的状态。-d -m参数启动“分离”模式,-S指定了一个session的标识。可以通过-R命令来重新“挂载”一个标识的session。更多细节请参考screen用法 man screen。 wget –random-wait -r -p -e robots=off -U mozilla http://www.example.com 下载整个www.example.com网站。(注:别太过分,大部分网站都有防爬功能了:)) curl ifconfig.me 当你的机器在内网的时候,可以通过这个命令查看外网的IP。 convert input.png -gravity NorthWest -background transparent -extent 720×200 output.png 改一下图片的大小尺寸 lsof –i 实时查看本机网络服务的活动状态。 vim scp://username@host//path/to/somefile vim一个远程文件 python -m SimpleHTTPServer 一句话实现一个HTTP服务,把当前目录设为HTTP服务目录,可以通过http://localhost:8000%E8%AE%BF%E9%97%AE 这也许是这个星球上最简单的HTTP服务器的实现了。 history | awk '{CMD[$2]++;count++;} END { for (a in CMD )print CMD[a] " " CMD[a]/count*100 "% " a }' | grep -v "./" | column -c3 -s " " -t | sort -nr | nl | head -n10 (陈皓注:有点复杂了,history|awk ‘{print $2}’|awk ‘BEGIN {FS=”|”} {print $1}’|sort|uniq -c|sort -rn|head -10) 这行脚本能输出你最常用的十条命令,由此甚至可以洞察你是一个什么类型的程序员。

getopts

Note that getopts is not able to parse GNU-style long options (–myoption) or XF86-style long options (-myoption)!

  • OPTIND Holds the index to the next argument to be processed. This is how getopts "remembers" its own status between invocations. Also usefull to shift the positional parameters after processing with getopts. OPTIND is initially set to 1, and needs to be re-set to 1 if you want to parse anything again with getopts
  • OPTARG This variable is set to any argument for an option found by getopts. It also contains the option flag of an unknown option.
  • OPTERR (Values 0 or 1) Indicates if Bash should display error messages generated by the getopts builtin. The value is initialized to 1 on every shell startup - so be sure to always set it to 0 if you don't want to see annoying messages!
#The base-syntax for getopts is:
getopts OPTSTRING VARNAME [ARGS...]

OPTSTRING tells getopts which options to expect and where to expect arguments (see below)
VARNAME tells getopts which shell-variable to use for option reporting
ARGS    tells getopts to parse these optional words instead of the positional parameters

compute float number

result=$(echo 1 23 | awk '{printf("%.2f", ($1/$2)*100)}')
=>result=4.35

a=12.34
b=23.45
echo "$a*$b"|bc

a=12.34
b=23.45
echo "scale=1;$a/$b"|bc #保留小数点精度只对除法有效
.5
echo "scale=2;$a/$b"|bc
.52

positional parameters

$FUNCNAME        the function name (attention: inside a function, $0 is still the $0 of the shell, not the function name)
$1 … $9        the argument list elements from 1 to 9
${10} … ${N}   the argument list elements beyond 9 (note the parameter expansion syntax!)
$*      all positional parameters except $0
$@      all positional parameters except $0
$#      the number of arguments, not counting $0

loop through the positional parameters

numargs=$#
for ((i=1 ; i <= numargs ; i++))
do
    echo "$1"
    shift
done
#or
for arg
do
    echo "$arg"
done
#while but it has the disadvantage to stop when $1 is empty (null-string)
while [ "$1" ]
do
    echo "$1"
    shift
done
# run as long as $1 is defined
while [ "${1+defined}" ]; do
  echo "$1"
  shift
done

mass usage

$*      $1 $2 $3 … ${N}
$@      $1 $2 $3 … ${N}
"$*"    "$1c$2c$3c…c${N}" # where 'c' is the first character of IFS.
"$@"    "$1" "$2" "$3""${N}"

Range Of Positional Parameters

${@:START:COUNT}
${*:START:COUNT}
"${@:START:COUNT}"
"${*:START:COUNT}"

多个文件总大小

cat filelist | xargs du -cb

regular expression

匹配中文字符的正则表达式:[u4e00-u9fa5]
评注:匹配中文还真是个头疼的事,有了这个表达式就好办了

匹配双字节字符(包括汉字在内):[^x00-xff]
评注:可以用来计算字符串的长度(一个双字节字符长度计2,ASCII字符计1)

匹配空白行的正则表达式:ns*r
评注:可以用来删除空白行

匹配HTML标记的正则表达式:<(S*?)[^>]*>.*?</1>|<.*? />
评注:网上流传的版本太糟糕,上面这个也仅仅能匹配部分,对于复杂的嵌套标记依旧无能为力

匹配首尾空白字符的正则表达式:^s*|s*$
评注:可以用来删除行首行尾的空白字符(包括空格、制表符、换页符等等),非常有用的表达式

匹配Email地址的正则表达式:w+([-+.]w+)*@w+([-.]w+)*.w+([-.]w+)*
评注:表单验证时很实用

匹配网址URL的正则表达式:[a-zA-z]+://[^s]*
评注:网上流传的版本功能很有限,上面这个基本可以满足需求

匹配帐号是否合法(字母开头,允许5-16字节,允许字母数字下划线):^[a-zA-Z][a-zA-Z0-9_]{4,15}$
评注:表单验证时很实用

匹配国内电话号码:d{3}-d{8}|d{4}-d{7}
评注:匹配形式如0511-4405222或021-87888822

匹配腾讯QQ号:[1-9][0-9]{4,}
评注:腾讯QQ号从10000开始

匹配中国邮政编码:[1-9]d{5}(?!d)
评注:中国邮政编码为6位数字

匹配身份证:d{15}|d{18}
评注:中国的身份证为15位或18位

匹配ip地址:d+.d+.d+.d+

评注:提取ip地址时有用

匹配特定数字:
^[1-9]d*$  //匹配正整数
^-[1-9]d*$ //匹配负整数
^-?[1-9]d*$  //匹配整数
^[1-9]d*|0$ //匹配非负整数(正整数+ 0)
^-[1-9]d*|0$  //匹配非正整数(负整数+ 0)
^[1-9]d*.d*|0.d*[1-9]d*$  //匹配正浮点数
^-([1-9]d*.d*|0.d*[1-9]d*)$ //匹配负浮点数
^-?([1-9]d*.d*|0.d*[1-9]d*|0?.0+|0)$ //匹配浮点数
^[1-9]d*.d*|0.d*[1-9]d*|0?.0+|0$  //匹配非负浮点数(正浮点数+ 0)
^(-([1-9]d*.d*|0.d*[1-9]d*))|0?.0+|0$  //匹配非正浮点数(负浮点数+ 0)
评注:处理大量数据时有用,具体应用时注意修正

匹配特定字符串:
^[A-Za-z]+$  //匹配由26个英文字母组成的字符串
^[A-Z]+$  //匹配由26个英文字母的大写组成的字符串
^[a-z]+$  //匹配由26个英文字母的小写组成的字符串
^[A-Za-z0-9]+$  //匹配由数字和26个英文字母组成的字符串
^w+$  //匹配由数字、26个英文字母或者下划线组成的字符串

touch

touch all folders in a directory

find . -maxdepth 1 -mindepth 1 -type d -exec touch {} \+

touch all files in a directory

find * -exec touch {} \;
# set time to 29th of Feb 2012.
find * -exec touch -t 201202291000 {} \;

Footnotes:

1

DEFINITION NOT FOUND.

2

DEFINITION NOT FOUND.

Author: Shi Shougang

Created: 2015-03-07 Sat 15:38

Emacs 24.3.1 (Org mode 8.2.10)

Validate