Shell scripts

Debugging scripts

For Bash, run the script with -xv command-line option, and use set -x (turn on echo; to turn off, set +x) and set -v (verbose mode) in the script.

For Tcsh, run the script with -XV command-line option, and use set echo and set verbose (use unset to turn them off) in the script.

Bash compatibility

On most Linux systems, /bin/sh is bash, but this is not always true for other UN*X systems. To avoid compatiblity problems, one better adheres to the standard sh features. Read here about bash extensions and how to avoid them.

Special character in quotation marks

"Interpreted" means the special character has special meaning.

"Requires \" means a backslash is required to escape the special meaning.

See FAQ on shell quotes for details.

Special character"..."'....'no quotation marks
newline Requires \Requires \Requires \
space Requires \
tab Requires \
! Requires \Requires \Requires \
# Requires \
$ InterpretedRequires \
& Requires \
( Requires \
) Requires \
{ Requires \
} Requires \
| Requires \
< Requires \
> Requires \
* Requires \
? Requires \
[ Requires \
] Requires \
; Requires \
' InterpretedRequires \
" InterpretedRequires \
` InterpretedRequires \

Multi-line quoting

Compare the following two examples in Bash 
$ greeting="Hello,
> World,  $USER, `date`"
$ echo "$greeting"
Hello
World,  jerry, Fri Sep  1 13:48:12 EDT 2000
$ echo $greeting
Hello World, jerry, Fri Sep 1 13:48:12 EDT 2000
Within the double quotes, the shell variable $USER is expanded, but the shell doesn't use the spaces and newlines in the variable as argument separators.

Without the double quotes, the shell variable $USER is also expanded, but the shell treat spaces and newlines as argument separators. For example, also note there are two spaces between World, and $USER and between Sep and 1 in the original input, but in the output there is only one space.

Control flows

For bash, see here for details.

BashTcshPerl
if [ expr ]
then
  commands
elif [ expr ]
then
  commands
else
  commands
fi
Note the location of then and the spaces between [, expr, and ].

To put then on the same line as if or elif, add ; after ]. 

if ( expr ) then
  commands
else if ( expr ) then
  commands
else
  commands
endif
Note the spaces between (, expr, and ). 
if (expr) {
  command;
}
elsif (expr) {
  command;
}
else {
  command;
}

case "$var" in
pattern1)
  commands
  ;;
pattern2)
  commands
  ;;
*)
  commands
  ;;
esac
See here for patterns. 
switch ("$var")
case pattern1:
  commands
  breaksw
case pattern2:
  commands
  breaksw
default:
  commands
  breaksw
endsw
(Perl has no switch/case) 
for var in word_list
do
  commands
done

foreach var (word_list)
  commands
end
Special commands:

break: Exit the loop
continue: Re-evaluate expr 

for var in pattern
do
  commands
done
In this second form of for loop, var will be file names which matches pattern. 
foreach var (pattern)
  commands
end

for ((var=1 ; var <= 100 ; var++))
do
  commands
done

while [ expr ]
do
  commands
done

while ( expr )
  commands
end
Special commands:

break: Exit the loop
continue: Re-evaluate expr 

while (expr) {
  ### redo always comes here
  commands
}  continue {
  ### next always comes here
  commands
  ### then back to the top and re-evaluate expr
}
### last always comes here
Special commands:

last: Exit the loop
next: Jump to the continue block if there is one, and then re-evaluate expr.
redo: Jump to the begining of the loop block without evaluating expr again.

Arithmetic expressions

BashTcsh
i=$((i+1))
$((i+=1))
let i+=1
$((i++))
let i++
i=`expr $i + 1`
Note:
  • No spaces around =
  • No spaces anywhere in the expression if one uses the let form, unless the expression is quoted by ".
  • Must be spaces if one uses the `expr ...` form. 
@ i = $i + 1
@ i += 1
@ i++
Note there must be a space between @ and whatever follow it. Also note the spaces in each case.

Checking files or other conditions

One can also see the man page of command test or here.

For bash, see here and here for details.

Return true ifBashTcshPerl
Grouping the expr\( expr \) (expr)
expr is false ! expr! expr! expr
Both expressions are trueexpr1 -a expr2expr1 && expr2expr1 && expr2
At least one of the expressions is trueexpr1 -o expr2expr1 || expr2expr1 || expr2
The string is non-empty-n str
The string is empty-z str
The variable is not defined-z $var! defined($var)
The variable is non-empty-n "$var"

(note the double quotes)

Two strings are equalstr1 = str2str1 == str2str1 eq str2
Two strings are not equalstr1 != str2str1 != str2str1 ne str2
The string matches patternstring =~ pattern

(this is a Bash 3.x feature and it should be used within double brackets [[ ]] )

string =~ pattern
The variable matches pattern"$var" =~ pattern

(note the double quotes)

The string doesn't match patternstring !~ pattern
Two integer expressions are equalint1 -eq int2int1 == int2int1 == int2
Two integer expressions are not equalint1 -ne int2int1 != int2int1 != int2
int1 <= int2int1 -le int2int1 <= int2int1 <= int2
int1 < int2int1 -lt int2int1 < int2int1 < int2
int1 >= int2int1 -ge int2int1 >= int2int1 >= int2
int1 > int2int1 -gt int2int1 > int2int1 > int2
Two files are symbolic/hard links of each otherfile1 -ef file2
file1 is newer
(has larger modifcation timestamp)		file1 -nt file2(stat(file1))[9] > stat(file2))[9]
file1 is olderfile1 -ot file2(stat(file1))[9] < stat(file2))[9]
file is block special -b file -b file
file is character special -c file -c file
file is a directory -d file-d file-d file
file exists (could be a directory) -e file-e file-e file
file is a regular file -f file-f file-f file
file is a symbolic link -L file-L file-l file
file is owned by the user -o file-o file-o file
file is a named pipe -p file -p file
file is readable -r file-r file-r file
file has a size greater than 0-s file-s file-s file
file is writable -w file-w file-w file
file is executable -x file-x file-x file
file has size 0-z file-z file-z file
file is text -T file
file is binary -B file

Manipulating variables

When referring a variable it's better to use curly braces { and } to avoid issues.

For bash, also read the "parameter substitution".

OperationBashTcsh
Set a local variablevar=valueset var=value
Set an environmental variableexport var=valuesetenv var value
Test if variable is set${?var}-n "$var"
${?var}
Define a read-only local variabledeclare -r var=valueset -r var=value
Set a variable to be read-onlydeclare -r varset -r var
If variable is not set
use default		${var-default}
If variable is not set or null
use default		${var:-default}
If variable is not set
set it to default and use the value		${var=default}
If variable is not set or null
set it to default and use the value		${var:=default}
If variable is set
use instead, otherwise, empty string		${var+instead}
If variable is set or not null
use instead, otherwise, empty string		${var:+instead}
If variable is set
use the value, otherwise, print msg		${var?msg}
If variable is set or not null
use the value, otherwise, print msg		${var:?msg}
If variable contains a full file name,
get the path		${var%/*}$var:h
If variable contains a full file name,
get the root (i.e. without extension)		${var%.*}$var:r
If variable contains a full file name,
get the extension		${var##*.}$var:e
If variable contains a full file name,
get the file name but not the path		${var##*/}$var:t
Make above file name
operations global		(Add g before the single
character modifier h,r,e,t)
Get the value of variable with pattern
(shortest match) removed from the beginning		${var#pattern}
Get the value of variable with pattern
(longest match) removed from the beginning		${var##pattern}
Get the value of variable with pattern
(shortest match) removed from the end		${var%pattern}
Get the value of variable with pattern
(longest match) removed from the end		${var%%pattern}
Get the value of variable starting
at the n-th character 		${var:n}
Get the value of variable starting
at the n-th character and for length m 		${var:n:m}`expr $var n m`
Replace the first occurrence of pattern
with replacement

replacement is optional

${var/pattern/replacement}
(Same as above, but make the operation global)${var//pattern/replacement}
Convert the first occurrence of pattern
to upper/lower case

pattern is optional

${var^pattern}

${var,pattern}

(Same as above, but make the operation global)${var^^pattern}

${var,,pattern}

String length${#var}${%var}
Array size${#var}

Special variables

When referring a variable it's better to use curly braces { and } to avoid issues.

MeaningBashTcshPerl
Name of the script being run${0}${0}$0
PID of the script being run$$$$$$
PID of the subshell$BASHPID
PID of last job run in background$!$!$!
Number of command-line arguments$#$#
$#argv		scalar(@ARGV)
n-th command-line argument passed to
the script being run (n starts from 1)		${n}$argv[n]$ARGV[n-1]
All command-line arguments,
seen as a single word		"$*""$*"
All command-line arguments,
broken into words		$*$*
All command-line arguments,
seen as original, quoted words		"$@""$@"
Exit status of last command executed$?$status
$?		$?
Final argument of the last command executed$_
Command line of the last command executed$_

Wildcard patterns/Globbing

Note that wildcard patterns/globbing are not regular expressions. In GNU utilities, globbing is implemented using fnmatch or glob library calls.

Wildcard Shells Meaning
* All Match zero or more characters.

For example, a* matches the files a, ab, abc, abc.d, and so on.

? All Match exactly one character.

For example, a? matches aa, ab, ac, etc.

[12..a..z] All Match any character listed in the brackets.

For example, a[ab] matches aa or ab.

[a-z] All Match all characters between a and z.

For example, a[0-9] matches a0, a1, and so on, up to a9.

[!ab..z] Bash Match any character that does not appear within the brackets.

For example, a[!0-9] doesn't match a0 but does match aa.

[^ab..z] Tcsh (Same as above)
{word1,word2...} Bash, Tcsh Match word1, word2, etc. For example, a_{dog,cat,horse} matches the filenames a_dog, a_cat, and a_horse.
?(x|y|z) Bash Match 0 or 1 instance of any of the specified patterns.

For example, w?(abc)w matches ww or wabcw. Also, ?(foo|bar) matches only foo, bar, and the empty string.

In Bash, this works only if you've set the extglob option using shopt.

*(x|y|z) Bash Match 0 or more instances of any of the specified patterns.

For example, w*(abc)w matches ww, wabcw, wabcabcw, etc. Also, *(foo|bar) matches foo, bar, foobarfoo, etc., as well as the empty string.

In Bash, this works only if you've set the extglob option using shopt.

+(x|y|z) Bash Match 1 or more instances of any of the specified patterns.

For example, w+(abc)w matches wabcw, wabcabcw, etc. Also, +(foo|bar) matches foo, bar, foobarfoo, etc.

In Bash, this works only if you've set the extglob option using shopt.

@(x|y|z) Bash Match exactly 1 of any of the specified patterns.

For example, @(foo|bar) matches foo or bar.

In Bash, this works only if you've set the extglob option using shopt.

!(x|y|z) Bash Match anything that doesn't contain any of the specified patterns.

For example, w!(abc)w doesn't match wabcw or wabcabcw, but it does match practically anything else that begins or ends with w. Also, !(foo|bar) matches all strings except foo and bar.

In Bash, this works only if you've set the extglob option using shopt.

Array variables

OperationBashTcshPerl
Define an array 
array=( 'Debian Linux' 'Redhat Linux' Ubuntu Linux )
 
set array=( 'Debian Linux' 'Redhat Linux' Ubuntu Linux )
 
@array=( "Debian Linux","Redhat Linux","Ubuntu","Linux" );
Get array size 
${#array[@]}
 
$#array
 
scalar(@array)
$#array+1
Get array elements 
$array[0]
${array[${i}]}

$array[1]
$array[2-3]
$array[2-]
$array[$#array]
$array[$i]
Note the index starts from 1, not 0! 
$array[0]
$array[2..3]
$array[-1]
$array[$#array]
The index -1 gives the last element of the array.
Add to an array 
array=( "${array[@]}" "Arch Linux" )
array=( "Arch Linux" "${array[@]}" )

set array=($array "Arch Linux")
set array=( "Arch Linux" $array )

push(@array, "Arch Linux");
unshift(@array, "Arch Linux");

Here documents

In Bash and Tcsh, a here document is defined as 
command <<DelimiterString
text..
more text...
...
DelimiterString
where DelimiterString is any string of choice, and the second occurrence of DelimiterString must be at the first column of the line.

The text.. can contain variables, and variable/parameter substitution will take place. To disable this, quote the first occurrence of DelimiterString.

Here documents can be used to comment out a block of code in the script: 

: <<"COMMENTED"
script..
more script...
...
COMMENTED
where the colon : is the do-nothing null command.

Redirection

OperationBash
Redirect both stdout and stderr to file foo &> foo

> foo 2>&1

Redirect stdout to file foo and stderr to file bar > foo 2> bar
Redirect file descriptor n to file foo n> foo

If n=1, n can be omitted.

Redirect file descriptor n to file descriptor m n>&m
Redirect stdout to file descriptor m >&m

Controlling shell behavior

OperationBashTcsh
Enable/disable clobbering files in redirection operations set +o noclobber
set -o noclobber		 unset noclobber
set noclobber
Enable/disable pressing Ctrl-D to exit set +o ignoreeof
set -o ignoreeof		 unset ignoreeof
set ignoreeof
Disable automatic path name expansion set -o noglob set noglob
Disable no-matching error in automatic path name expansion set nonomatch
Display every command before executing it set -o verbose set echo
Enable verbose mode set -o xtrace set verbose
Set history size to 100 export HISTSIZE=100 set history=100
Display resource limits ulimit -a limit
Display Linux kernel parameters sysctl -a
sysconf -l		 sysctl -a
sysconf -l
Set Linux personality setarch setarch
Set max core dump size to 0 ulimit -c 0 limit coredumpsize 0
Set max size of a file which can be created to unlimited ulimit -f unlimited unlimit filesize
Set max per-process CPU time to 10 seconds ulimit -t 10 limit cputime 10
Set max number of concurrent processes to unlimited ulimit -u unlimited unlimit maxproc
Set max number of file descriptors to unlimited ulimit -n unlimited unlimit descriptors
Set max size of physical memory (which can be allocated) to unlimited ulimit -m unlimited unlimit memoryuse
Set max size of virtual memory (which can be allocated) to unlimited ulimit -v unlimited unlimit vmemoryuse
Set max stack size to unlimited ulimit -s unlimited unlimit stacksize

Customizing the prompt

BashTcsh
Set up the prompt export PS1="..." set prompt="..."
Current working directory \w $CWD or %/
Current working directory,
with one's home directory by ~		 \W $CWD:t or %~
Use # as the prompt character for root user \$ %#
Host name (up to the first .) \h %m
User name \u %n
Current history number \! %h
Time of day \T %p