]> An in-depth exploration of the art of shell scripting Mendel Cooper

thegrendel@theriver.com

3.7.3 28 November 2005 3.5 04 June 2005 mc 'BOXBERRY' release: Important Update. 3.6 28 Aug 2005 mc 'POKEBERRY' release: Bugfix Update. 3.7 23 Oct 2005 mc 'WHORTLEBERRY' release: Bugfix Update. This tutorial assumes no previous knowledge of scripting or programming, but progresses rapidly toward an intermediate/advanced level of instruction . . . all the while sneaking in little snippets of UNIX wisdom and lore. It serves as a textbook, a manual for self-study, and a reference and source of knowledge on shell scripting techniques. The exercises and heavily-commented examples invite active reader participation, under the premise that the only way to really learn scripting is to write scripts. This book is suitable for classroom use as a general introduction to programming concepts. The latest update of this document, as an archived, bzip2-ed tarball including both the SGML source and rendered HTML, may be downloaded from the author's home site. A pdf version is also available. See the change log for a revision history. For Anita, the source of all the magic The shell is a command interpreter. More than just the insulating layer between the operating system kernel and the user, it's also a fairly powerful programming language. A shell program, called a script, is an easy-to-use tool for building applications by gluing together system calls, tools, utilities, and compiled binaries. Virtually the entire repertoire of UNIX commands, utilities, and tools is available for invocation by a shell script. If that were not enough, internal shell commands, such as testing and loop constructs, give additional power and flexibility to scripts. Shell scripts lend themselves exceptionally well to administrative system tasks and other routine repetitive jobs not requiring the bells and whistles of a full-blown tightly structured programming language. Herbert Mayer No programming language is perfect. There is not even a single best language; there are only languages well suited or perhaps poorly suited for particular purposes. A working knowledge of shell scripting is essential to anyone wishing to become reasonably proficient at system administration, even if they do not anticipate ever having to actually write a script. Consider that as a Linux machine boots up, it executes the shell scripts in /etc/rc.d to restore the system configuration and set up services. A detailed understanding of these startup scripts is important for analyzing the behavior of a system, and possibly modifying it. Writing shell scripts is not hard to learn, since the scripts can be built in bite-sized sections and there is only a fairly small set of shell-specific operators and options These are referred to as builtins, features internal to the shell. to learn. The syntax is simple and straightforward, similar to that of invoking and chaining together utilities at the command line, and there are only a few rules to learn. Most short scripts work right the first time, and debugging even the longer ones is straightforward. A shell script is a quick and dirty method of prototyping a complex application. Getting even a limited subset of the functionality to work in a shell script is often a useful first stage in project development. This way, the structure of the application can be tested and played with, and the major pitfalls found before proceeding to the final coding in C, C++, Java, or Perl. Shell scripting hearkens back to the classic UNIX philosophy of breaking complex projects into simpler subtasks, of chaining together components and utilities. Many consider this a better, or at least more esthetically pleasing approach to problem solving than using one of the new generation of high powered all-in-one languages, such as Perl, which attempt to be all things to all people, but at the cost of forcing you to alter your thinking processes to fit the tool. When not to use shell scripts Resource-intensive tasks, especially where speed is a factor (sorting, hashing, etc.) Procedures involving heavy-duty math operations, especially floating point arithmetic, arbitrary precision calculations, or complex numbers (use C++ or FORTRAN instead) Cross-platform portability required (use C or Java instead) Complex applications, where structured programming is a necessity (need type-checking of variables, function prototypes, etc.) Mission-critical applications upon which you are betting the ranch, or the future of the company Situations where security is important, where you need to guarantee the integrity of your system and protect against intrusion, cracking, and vandalism Project consists of subcomponents with interlocking dependencies Extensive file operations required (Bash is limited to serial file access, and that only in a particularly clumsy and inefficient line-by-line fashion) Need native support for multi-dimensional arrays Need data structures, such as linked lists or trees Need to generate or manipulate graphics or GUIs Need direct access to system hardware Need port or socket I/O Need to use libraries or interface with legacy code Proprietary, closed-source applications (shell scripts put the source code right out in the open for all the world to see) If any of the above applies, consider a more powerful scripting language -- perhaps Perl, Tcl, Python, Ruby -- or possibly a high-level compiled language such as C, C++, or Java. Even then, prototyping the application as a shell script might still be a useful development step. We will be using Bash, an acronym for Bourne-Again shell and a pun on Stephen Bourne's now classic Bourne shell. Bash has become a de facto standard for shell scripting on all flavors of UNIX. Most of the principles this book covers apply equally well to scripting with other shells, such as the Korn Shell, from which Bash derives some of its features, Many of the features of ksh88, and even a few from the updated ksh93 have been merged into Bash. and the C Shell and its variants. (Note that C Shell programming is not recommended due to certain inherent problems, as pointed out in an October, 1993 Usenet post by Tom Christiansen.) What follows is a tutorial on shell scripting. It relies heavily on examples to illustrate various features of the shell. The example scripts work -- they've been tested, insofar as was possible -- and some of them are even useful in real life. The reader can play with the actual working code of the examples in the source archive (scriptname.sh or scriptname.bash), By convention, user-written shell scripts that are Bourne shell compliant generally take a name with a .sh extension. System scripts, such as those found in /etc/rc.d, do not conform to this nomenclature. give them execute permission (chmod u+rx scriptname), then run them to see what happens. Should the source archive not be available, then cut-and-paste from the HTML, pdf, or text rendered versions. Be aware that some of the scripts presented here introduce features before they are explained, and this may require the reader to temporarily skip ahead for enlightenment. Unless otherwise noted, the author of this book wrote the example scripts that follow. Larry Wall Shell programming is a 1950s juke box . . . In the simplest case, a script is nothing more than a list of system commands stored in a file. At the very least, this saves the effort of retyping that particular sequence of commands each time it is invoked. &ex1; There is nothing unusual here, only a set of commands that could just as easily be invoked one by one from the command line on the console or in an xterm. The advantages of placing the commands in a script go beyond not having to retype them time and again. The script becomes a tool, and can easily be modified or customized for a particular application. &ex1a; Now that's beginning to look like a real script. But we can go even farther . . . &ex2; Since you may not wish to wipe out the entire system log, this version of the script keeps the last section of the message log intact. You will constantly discover ways of refining previously written scripts for increased effectiveness. The sha-bang sha-bang ( #! #!) at the head of a script tells your system that this file is a set of commands to be fed to the command interpreter indicated. The #! is actually a two-byte Some flavors of UNIX (those based on 4.2BSD) take a four-byte magic number, requiring a blank after the ! -- #! /bin/sh. magic number magic number, a special marker that designates a file type, or in this case an executable shell script (type man magic for more details on this fascinating topic). Immediately following the sha-bang is a path name. This is the path to the program that interprets the commands in the script, whether it be a shell, a programming language, or a utility. This command interpreter then executes the commands in the script, starting at the top (line following the sha-bang line), ignoring comments. The #! line in a shell script will be the first thing the command interpreter (sh or bash) sees. Since this line begins with a #, it will be correctly interpreted as a comment when the command interpreter finally executes the script. The line has already served its purpose - calling the command interpreter. If, in fact, the script includes an extra #! line, then bash will interpret it as a comment. #!/bin/bash echo "Part 1 of script." a=1 #!/bin/bash # This does *not* launch a new script. echo "Part 2 of script." echo $a # Value of $a stays at 1. #!/bin/sh #!/bin/bash #!/usr/bin/perl #!/usr/bin/tcl #!/bin/sed -f #!/usr/awk -f Each of the above script header lines calls a different command interpreter, be it /bin/sh, the default shell (bash in a Linux system) or otherwise. This allows some cute tricks. #!/bin/rm # Self-deleting script. # Nothing much seems to happen when you run this... except that the file disappears. WHATEVER=65 echo "This line will never print (betcha!)." exit $WHATEVER # Doesn't matter. The script will not exit here. Also, try starting a README file with a #!/bin/more, and making it executable. The result is a self-listing documentation file. (A here document using cat is possibly a better alternative -- see ). Using #!/bin/sh, the default Bourne shell in most commercial variants of UNIX, makes the script portable to non-Linux machines, though you sacrifice Bash-specific features. The script will, however, conform to the POSIX Portable Operating System Interface, an attempt to standardize UNIX-like OSes. The POSIX specifications are listed on the Open Group site. sh standard. Note that the path given at the sha-bang must be correct, otherwise an error message -- usually Command not found -- will be the only result of running the script. #! can be omitted if the script consists only of a set of generic system commands, using no internal shell directives. The second example, above, requires the initial #!, since the variable assignment line, lines=50, uses a shell-specific construct. Note again that #!/bin/sh invokes the default shell interpreter, which defaults to /bin/bash on a Linux machine. This tutorial encourages a modular approach to constructing a script. Make note of and collect boilerplate code snippets that might be useful in future scripts. Eventually you can build quite an extensive library of nifty routines. As an example, the following script prolog tests whether the script has been invoked with the correct number of parameters. E_WRONG_ARGS=65 script_parameters="-a -h -m -z" # -a = all, -h = help, etc. if [ $# -ne $Number_of_expected_args ] then echo "Usage: `basename $0` $script_parameters" # `basename $0` is the script's filename. exit $E_WRONG_ARGS fi Many times, you will write a script that carries out one particular task. The first script in this chapter is an example of this. Later, it might occur to you to generalize the script to do other, similar tasks. Replacing the literal (hard-wired) constants by variables is a step in that direction, as is replacing repetitive code blocks by functions. Having written the script, you can invoke it by sh scriptname, Caution: invoking a Bash script by sh scriptname turns off Bash-specific extensions, and the script may therefore fail to execute. or alternatively bash scriptname. (Not recommended is using sh <scriptname, since this effectively disables reading from stdin within the script.) Much more convenient is to make the script itself directly executable with a chmod. Either: chmod 555 scriptname (gives everyone read/execute permission) A script needs read, as well as execute permission for it to run, since the shell needs to be able to read it. or chmod +rx scriptname (gives everyone read/execute permission) chmod u+rx scriptname (gives only the script owner read/execute permission) Having made the script executable, you may now test it by ./scriptname. Why not simply invoke the script with scriptname? If the directory you are in ($PWD) is where scriptname is located, why doesn't this work? This fails because, for security reasons, the current directory is not by default included in a user's $PATH. It is therefore necessary to explicitly invoke the script in the current directory with a ./scriptname. If it begins with a sha-bang line, invoking the script calls the correct command interpreter to run it. As a final step, after testing and debugging, you would likely want to move it to /usr/local/bin (as root, of course), to make the script available to yourself and all other users as a system-wide executable. The script could then be invoked by simply typing scriptname [ENTER] from the command line. System administrators often write scripts to automate common tasks. Give several instances where such scripts would be useful. Write a script that upon invocation shows the time and date, lists all logged-in users, and gives the system uptime. The script then saves this information to a logfile. # # special character # comment Lines beginning with a # (with the exception of #!) are comments. # This line is a comment. Comments may also occur following the end of a command. echo "A comment will follow." # Comment here. # ^ Note whitespace before # Comments may also follow whitespace at the beginning of a line. # A tab precedes this comment. A command may not follow a comment on the same line. There is no method of terminating the comment, in order for live code to begin on the same line. Use a new line for the next command. Of course, an escaped # in an echo statement does not begin a comment. Likewise, a # appears in certain parameter substitution constructs and in numerical constant expressions. echo "The # here does not begin a comment." echo 'The # here does not begin a comment.' echo The \# here does not begin a comment. echo The # here begins a comment. echo ${PATH#*:} # Parameter substitution, not a comment. echo $(( 2#101011 )) # Base conversion, not a comment. # Thanks, S.C. The standard quoting and escape characters (" ' \) escape the #. Certain pattern matching operations also use the #. ; ; special character ; separator Permits putting two or more commands on the same line. echo hello; echo there if [ -x "$filename" ]; then # Note that "if" and "then" need separation. # Why? echo "File $filename exists."; cp $filename $filename.bak else echo "File $filename not found."; touch $filename fi; echo "File test complete." Note that the ; sometimes needs to be escaped. ;; ;; special character case ;; case "$variable" in abc) echo "\$variable = abc" ;; xyz) echo "\$variable = xyz" ;; esac . . special character . dot command source Equivalent to source (see ). This is a bash builtin. . . special character . filename part of a filename When working with filenames, a dot is the prefix of a hidden file, a file that an ls will not normally show. bash$ touch .hidden-file bash$ ls -l total 10 -rw-r--r-- 1 bozo 4034 Jul 18 22:04 data1.addressbook -rw-r--r-- 1 bozo 4602 May 25 13:58 data1.addressbook.bak -rw-r--r-- 1 bozo 877 Dec 17 2000 employment.addressbook bash$ ls -al total 14 drwxrwxr-x 2 bozo bozo 1024 Aug 29 20:54 ./ drwx------ 52 bozo bozo 3072 Aug 29 20:51 ../ -rw-r--r-- 1 bozo bozo 4034 Jul 18 22:04 data1.addressbook -rw-r--r-- 1 bozo bozo 4602 May 25 13:58 data1.addressbook.bak -rw-r--r-- 1 bozo bozo 877 Dec 17 2000 employment.addressbook -rw-rw-r-- 1 bozo bozo 0 Aug 29 20:54 .hidden-file When considering directory names, a single dot represents the current working directory, and two dots denote the parent directory. bash$ pwd /home/bozo/projects bash$ cd . bash$ pwd /home/bozo/projects bash$ cd .. bash$ pwd /home/bozo/ The dot often appears as the destination (directory) of a file movement command. bash$ cp /home/bozo/current_work/junk/* . . . special character . character match match single character When matching characters, as part of a regular expression, a dot matches a single character. " "STRING" preserves (from interpretation) most of the special characters within STRING. See also . ' 'STRING' preserves all special characters within STRING. This is a stronger form of quoting than using ". See also . , The comma operator links together a series of arithmetic operations. All are evaluated, but only the last one is returned. let "t2 = ((a = 9, 15 / 3))" # Set "a = 9" and "t2 = 15 / 3" \ A quoting mechanism for single characters. \X escapes the character X. This has the effect of quoting X, equivalent to 'X'. The \ may be used to quote " and ', so they are expressed literally. See for an in-depth explanation of escaped characters. / Separates the components of a filename (as in /home/bozo/projects/Makefile). This is also the division arithmetic operator. ` The `command` construct makes available the output of command for assignment to a variable. This is also known as backquotes or backticks. : : special character : null command true endless loop This is the shell equivalent of a NOP (no op, a do-nothing operation). It may be considered a synonym for the shell builtin true. The : command is itself a Bash builtin, and its exit status is true (0). : echo $? # 0 Endless loop: while : do operation-1 operation-2 ... operation-n done # Same as: # while true # do # ... # done Placeholder in if/then test: if condition then : # Do nothing and branch ahead else take-some-action fi Provide a placeholder where a binary operation is expected, see and default parameters. : ${username=`whoami`} # ${username=`whoami`} Gives an error without the leading : # unless "username" is a command or builtin... Provide a placeholder where a command is expected in a here document. See . Evaluate string of variables using parameter substitution (as in ). : ${HOSTNAME?} ${USER?} ${MAIL?} # Prints error message #+ if one or more of essential environmental variables not set. Variable expansion / substring replacement. In combination with the > redirection operator, truncates a file to zero length, without changing its permissions. If the file did not previously exist, creates it. : > data.xxx # File "data.xxx" now empty. # Same effect as cat /dev/null >data.xxx # However, this does not fork a new process, since ":" is a builtin. See also . In combination with the >> redirection operator, has no effect on a pre-existing target file (: >> target_file). If the file did not previously exist, creates it. This applies to regular files, not pipes, symlinks, and certain special files. May be used to begin a comment line, although this is not recommended. Using # for a comment turns off error checking for the remainder of that line, so almost anything may appear in a comment. However, this is not the case with :. : This is a comment that generates an error, ( if [ $x -eq 3] ). The : also serves as a field separator, in /etc/passwd, and in the $PATH variable. bash$ echo $PATH /usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/sbin:/usr/sbin:/usr/games ! ! special character ! not logical not The ! operator inverts the exit status of the command to which it is applied (see ). It also inverts the meaning of a test operator. This can, for example, change the sense of equal ( = ) to not-equal ( != ). The ! operator is a Bash keyword. In a different context, the ! also appears in indirect variable references. In yet another context, from the command line, the ! invokes the Bash history mechanism (see ). Note that within a script, the history mechanism is disabled. * * special character * wild card globbing regular expression The * character serves as a wild card for filename expansion in globbing. By itself, it matches every filename in a given directory. bash$ echo * abs-book.sgml add-drive.sh agram.sh alias.sh The * also represents any number (or zero) characters in a regular expression. * * special character * multiplication exponentiation arithmetic operator In the context of arithmetic operations, the * denotes multiplication. A double asterisk, **, is the exponentiation operator. ? ? special character ? test operator test token Within certain expressions, the ? indicates a test for a condition. In a double parentheses construct, the ? serves as a C-style trinary operator. See . In a parameter substitution expression, the ? tests whether a variable has been set. ? ? special character ? wild card globbing regular expression The ? character serves as a single-character wild card for filename expansion in globbing, as well as representing one character in an extended regular expression. $ $ special character $ variable substitution var1=5 var2=23skidoo echo $var1 # 5 echo $var2 # 23skidoo A $ prefixing a variable name indicates the value the variable holds. $ $ special character $ regular expression end of line In a regular expression, a $ addresses the end of a line of text. ${} ${} special character ${} parameter substitution $* $@ $* special character $* $@ positional parameters $@ $? $? special character ? exit status variable exit status The $? variable holds the exit status of a command, a function, or of the script itself. $$ $$ special character $$ process ID variable process ID The $$ variable holds the process ID of the script in which it appears. () (a=hello; echo $a) A listing of commands within parentheses starts a subshell. Variables inside parentheses, within the subshell, are not visible to the rest of the script. The parent process, the script, cannot read variables created in the child process, the subshell. a=123 ( a=321; ) echo "a = $a" # a = 123 # "a" within parentheses acts like a local variable. Array=(element1 element2 element3) {xxx,yyy,zzz..} special character {} brace expansion {xxx,yyy,zzz,...} cat {file1,file2,file3} > combined_file # Concatenates the files file1, file2, and file3 into combined_file. cp file22.{txt,backup} # Copies "file22.txt" to "file22.backup" A command may act upon a comma-separated list of file specs within braces. The shell does the brace expansion. The command itself acts upon the result of the expansion. Filename expansion (globbing) applies to the file specs between the braces. No spaces allowed within the braces unless the spaces are quoted or escaped. echo {file1,file2}\ :{\ A," B",' C'} file1 : A file1 : B file1 : C file2 : A file2 : B file2 : C {} {} special character {} block of code Also referred to as an inline group, this construct, in effect, creates an anonymous function. However, unlike a function, the variables in a code block remain visible to the remainder of the script. bash$ { local a; a=123; } bash: local: can only be used in a function a=123 { a=321; } echo "a = $a" # a = 321 (value inside code block) # Thanks, S.C. The code block enclosed in braces may have I/O redirected to and from it. &ex8; &rpmcheck; Unlike a command group within (parentheses), as above, a code block enclosed by {braces} will not normally launch a subshell. Exception: a code block in braces as part of a pipe may run as a subshell. ls | { read firstline; read secondline; } # Error. The code block in braces runs as a subshell, #+ so the output of "ls" cannot be passed to variables within the block. echo "First line is $firstline; second line is $secondline" # Will not work. # Thanks, S.C. {} \; Mostly used in find constructs. This is not a shell builtin. The ; ends the -exec option of a find command sequence. It needs to be escaped to protect it from interpretation by the shell. [ ] [] special character [ ] test Test expression between [ ]. Note that [ is part of the shell builtin test (and a synonym for it), not a link to the external command /usr/bin/test. [[ ]] [[]] special character [[ ]] test Test expression between [[ ]] (shell keyword). See the discussion on the [[ ... ]] construct. [ ] [ ] special character array_element[ ] array element In the context of an array, brackets set off the numbering of each element of that array. Array[1]=slot_1 echo ${Array[1]} [ ] [ ] special character character range regular expression As part of a regular expression, brackets delineate a range of characters to match. (( )) (( )) special character (( )) integer comparison Expand and evaluate integer expression between (( )). See the discussion on the (( ... )) construct. > &> >& >> < > >& >> < special character > special character >& special character >> special character < redirection scriptname >filename redirects the output of scriptname to file filename. Overwrite filename if it already exists. command &>filename redirects both the stdout and the stderr of command to filename. command >&2 redirects stdout of command to stderr. scriptname >>filename appends the output of scriptname to file filename. If filename does not already exist, it will be created. (command)> <(command) In a different context, the < and > characters act as string comparison operators. In yet another context, the < and > characters act as integer comparison operators. See also . << <<< < > < special character < > ASCII comparison > veg1=carrots veg2=tomatoes if [[ "$veg1" < "$veg2" ]] then echo "Although $veg1 precede $veg2 in the dictionary," echo "this implies nothing about my culinary preferences." else echo "What kind of dictionary are you using, anyhow?" fi \< \> \< regular expression \< > word boundary > bash$ grep '\<the\>' textfile | | special character | pipe Passes the output of previous command to the input of the next one, or to the shell. This is a method of chaining commands together. echo ls -l | sh # Passes the output of "echo ls -l" to the shell, #+ with the same result as a simple "ls -l". cat *.lst | sort | uniq # Merges and sorts all ".lst" files, then deletes duplicate lines. A pipe, as a classic method of interprocess communication, sends the stdout of one process to the stdin of another. In a typical case, a command, such as cat or echo, pipes a stream of data to a filter (a command that transforms its input) for processing. cat $filename1 $filename2 | grep $search_word The output of a command or commands may be piped to a script. #!/bin/bash # uppercase.sh : Changes input to uppercase. tr 'a-z' 'A-Z' # Letter ranges must be quoted #+ to prevent filename generation from single-letter filenames. exit 0 Now, let us pipe the output of ls -l to this script. bash$ ls -l | ./uppercase.sh -RW-RW-R-- 1 BOZO BOZO 109 APR 7 19:49 1.TXT -RW-RW-R-- 1 BOZO BOZO 109 APR 14 16:48 2.TXT -RW-R--R-- 1 BOZO BOZO 725 APR 20 20:56 DATA-FILE The stdout of each process in a pipe must be read as the stdin of the next. If this is not the case, the data stream will block, and the pipe will not behave as expected. cat file1 file2 | ls -l | sort # The output from "cat file1 file2" disappears. A pipe runs as a child process, and therefore cannot alter script variables. variable="initial_value" echo "new_value" | read variable echo "variable = $variable" # variable = initial_value If one of the commands in the pipe aborts, this prematurely terminates execution of the pipe. Called a broken pipe, this condition sends a SIGPIPE signal. >| >| special character >| redirection force noclobber This will forcibly overwrite an existing file. || || special character || or logical operator In a test construct, the || operator causes a return of 0 (success) if either of the linked test conditions is true. & A command followed by an & will run in the background. bash$ sleep 10 & [1] 850 [1]+ Done sleep 10 Within a script, commands and even loops may run in the background. &bgloop; A command run in the background within a script may cause the script to hang, waiting for a keystroke. Fortunately, there is a remedy for this. && && special character && and logical operator In a test construct, the && operator causes a return of 0 (success) only if both the linked test conditions are true. - Option flag for a command or filter. Prefix for an operator. COMMAND -[Option1][Option2][...] ls -al sort -dfu $filename set -- $variable if [ $file1 -ot $file2 ] then echo "File $file1 is older than $file2." fi if [ "$a" -eq "$b" ] then echo "$a is equal to $b." fi if [ "$c" -eq 24 -a "$d" -eq 47 ] then echo "$c equals 24 and $d equals 47." fi - - special character - redirection from/to stdin/stdout (cd /source/directory && tar cf - . ) | (cd /dest/directory && tar xpvf -) # Move entire file tree from one directory to another # [courtesy Alan Cox <a.cox@swansea.ac.uk>, with a minor change] # 1) cd /source/directory Source directory, where the files to be moved are. # 2) && "And-list": if the 'cd' operation successful, then execute the next command. # 3) tar cf - . The 'c' option 'tar' archiving command creates a new archive, # the 'f' (file) option, followed by '-' designates the target file as stdout, # and do it in current directory tree ('.'). # 4) | Piped to... # 5) ( ... ) a subshell # 6) cd /dest/directory Change to the destination directory. # 7) && "And-list", as above # 8) tar xpvf - Unarchive ('x'), preserve ownership and file permissions ('p'), # and send verbose messages to stdout ('v'), # reading data from stdin ('f' followed by '-'). # # Note that 'x' is a command, and 'p', 'v', 'f' are options. # Whew! # More elegant than, but equivalent to: # cd source/directory # tar cf - . | (cd ../dest/directory; tar xpvf -) # # Also having same effect: # cp -a /source/directory/* /dest/directory # Or: # cp -a /source/directory/* /source/directory/.[^.]* /dest/directory # If there are hidden files in /source/directory. bunzip2 linux-2.6.13.tar.bz2 | tar xvf - # --uncompress tar file-- | --then pass it to "tar"-- # If "tar" has not been patched to handle "bunzip2", # this needs to be done in two discrete steps, using a pipe. # The purpose of the exercise is to unarchive "bzipped" kernel source. Note that in this context the - is not itself a Bash operator, but rather an option recognized by certain UNIX utilities that write to stdout, such as tar, cat, etc. bash$ echo "whatever" | cat - whatever Where a filename is expected, - redirects output to stdout (sometimes seen with tar cf), or accepts input from stdin, rather than from a file. This is a method of using a file-oriented utility as a filter in a pipe. bash$ file Usage: file [-bciknvzL] [-f namefile] [-m magicfiles] file... By itself on the command line, file fails with an error message. Add a - for a more useful result. This causes the shell to await user input. bash$ file - abc standard input: ASCII text bash$ file - #!/bin/bash standard input: Bourne-Again shell script text executable Now the command accepts input from stdin and analyzes it. The - can be used to pipe stdout to other commands. This permits such stunts as prepending lines to a file. Using diff to compare a file with a section of another: grep Linux file1 | diff file2 - Finally, a real-world example using - with tar. &ex58; Filenames beginning with - may cause problems when coupled with the - redirection operator. A script should check for this and add an appropriate prefix to such filenames, for example ./-FILENAME, $PWD/-FILENAME, or $PATHNAME/-FILENAME. If the value of a variable begins with a -, this may likewise create problems. var="-n" echo $var # Has the effect of "echo -n", and outputs nothing. - A cd - command changes to the previous working directory. This uses the $OLDPWD environmental variable. Do not confuse the - used in this sense with the - redirection operator just discussed. The interpretation of the - depends on the context in which it appears. - Minus sign in an arithmetic operation. = Assignment operator a=28 echo $a # 28 In a different context, the = is a string comparison operator. + Addition arithmetic operator. In a different context, the + is a Regular Expression operator. + Option flag for a command or filter. Certain commands and builtins use the + to enable certain options and the - to disable them. % Modulo (remainder of a division) arithmetic operation. In a different context, the % is a pattern matching operator. ~ This corresponds to the $HOME internal variable. ~bozo is bozo's home directory, and ls ~bozo lists the contents of it. ~/ is the current user's home directory, and ls ~/ lists the contents of it. bash$ echo ~bozo /home/bozo bash$ echo ~ /home/bozo bash$ echo ~/ /home/bozo/ bash$ echo ~: /home/bozo: bash$ echo ~nonexistent-user ~nonexistent-user ~+ This corresponds to the $PWD internal variable. ~- This corresponds to the $OLDPWD internal variable. =~ This operator was introduced with version 3 of Bash. ^ ^ special character ^ regular expression beginning of line In a regular expression, a ^ addresses the beginning of a line of text. Control Characters A control character is a CONTROL + key combination. Control characters are not normally useful inside a script. Ctl-B Backspace (nondestructive). Ctl-C Break. Terminate a foreground job. Ctl-D Log out from a shell (similar to exit). EOF (end of file). This also terminates input from stdin. When typing text on the console or in an xterm window, Ctl-D erases the character under the cursor. When there are no characters present, Ctl-D logs out of the session, as expected. In an xterm window, this has the effect of closing the window. Ctl-G BEL (beep). On some old-time teletype terminals, this would actually ring a bell. Ctl-H Rubout (destructive backspace). Erases characters the cursor backs over while backspacing. #!/bin/bash # Embedding Ctl-H in a string. a="^H^H" # Two Ctl-H's (backspaces). echo "abcdef" # abcdef echo -n "abcdef$a " # abcd f # Space at end ^ ^ Backspaces twice. echo -n "abcdef$a" # abcdef # No space at end Doesn't backspace (why?). # Results may not be quite as expected. echo; echo Ctl-I Horizontal tab. Ctl-J Newline (line feed). In a script, may also be expressed in octal notation -- '\012' or in hexadecimal -- '\x0a'. Ctl-K Vertical tab. When typing text on the console or in an xterm window, Ctl-K erases from the character under the cursor to end of line. Ctl-L Formfeed (clear the terminal screen). This has the same effect as the clear command. Ctl-M Carriage return. #!/bin/bash # Thank you, Lee Maschmeyer, for this example. read -n 1 -s -p $'Control-M leaves cursor at beginning of this line. Press Enter. \x0d' # Of course, '0d' is the hex equivalent of Control-M. echo >&2 # The '-s' makes anything typed silent, #+ so it is necessary to go to new line explicitly. read -n 1 -s -p $'Control-J leaves cursor on next line. \x0a' # '0a' is the hex equivalent of Control-J, linefeed. echo >&2 ### read -n 1 -s -p $'And Control-K\x0bgoes straight down.' echo >&2 # Control-K is vertical tab. # A better example of the effect of a vertical tab is: var=$'\x0aThis is the bottom line\x0bThis is the top line\x0a' echo "$var" # This works the same way as the above example. However: echo "$var" | col # This causes the right end of the line to be higher than the left end. # It also explains why we started and ended with a line feed -- #+ to avoid a garbled screen. # As Lee Maschmeyer explains: # -------------------------- # In the [first vertical tab example] . . . the vertical tab #+ makes the printing go straight down without a carriage return. # This is true only on devices, such as the Linux console, #+ that can't go "backward." # The real purpose of VT is to go straight UP, not down. # It can be used to print superscripts on a printer. # The col utility can be used to emulate the proper behavior of VT. exit 0 Ctl-Q Resume (XON). This resumes stdin in a terminal. Ctl-S Suspend (XOFF). This freezes stdin in a terminal. (Use Ctl-Q to restore input.) Ctl-U Erase a line of input, from the cursor backward to beginning of line. In some settings, Ctl-U erases the entire line of input, regardless of cursor position. Ctl-V When inputting text, Ctl-V permits inserting control characters. For example, the following two are equivalent: echo -e '\x0a' echo <Ctl-V><Ctl-J> Ctl-V is primarily useful from within a text editor. Ctl-W When typing text on the console or in an xterm window, Ctl-W erases from the character under the cursor backwards to the first instance of whitespace. In some settings, Ctl-W erases backwards to first non-alphanumeric character. Ctl-Z Pause a foreground job. Whitespace Whitespace consists of either spaces, tabs, blank lines, or any combination thereof. A linefeed (newline) is also a whitespace character. This explains why a blank line, consisting only of a linefeed, is considered whitespace. In some contexts, such as variable assignment, whitespace is not permitted, and results in a syntax error. Blank lines have no effect on the action of a script, and are therefore useful for visually separating functional sections. $IFS, the special variable separating fields of input to certain commands, defaults to whitespace. To preserve whitespace within a string or in a variable, use quoting. Variables are how programming and scripting languages represent data. They appear in arithmetic operations and manipulation of quantities, in string parsing, and they are indispensable for working in the abstract with symbols -- tokens that represent something else. A variable is nothing more than a label assigned to a location or set of locations in computer memory holding an item of data. The name of a variable is a placeholder for its value, the data it holds. Referencing its value is called variable substitution. $ $ variable $ variable substitution Let us carefully distinguish between the name of a variable and its value. If variable1 is the name of a variable, then $variable1 is a reference to its value, the data item it contains. bash$ variable=23 bash$ echo variable variable bash$ echo $variable 23 The only time a variable appears naked -- without the $ prefix -- is when declared or assigned, when unset, when exported, or in the special case of a variable representing a signal (see ). Assignment may be with an = (as in var1=27), in a read statement, and at the head of a loop (for var2 in 1 2 3). Enclosing a referenced value in double quotes (" ") does not interfere with variable substitution. This is called partial quoting, sometimes referred to as weak quoting. Using single quotes (' ') causes the variable name to be used literally, and no substitution will take place. This is full quoting, sometimes referred to as strong quoting. See for a detailed discussion. Note that $variable is actually a simplified alternate form of ${variable}. In contexts where the $variable syntax causes an error, the longer form may work (see , below). &ex9; An uninitialized variable has a null value - no assigned value at all (not zero!). Using a variable before assigning a value to it will usually cause problems. It is nevertheless possible to perform arithmetic operations on an uninitialized variable. echo "$uninitialized" # (blank line) let "uninitialized += 5" # Add 5 to it. echo "$uninitialized" # 5 # Conclusion: # An uninitialized variable has no value, however #+ it acts as if it were 0 in an arithmetic operation. # This is undocumented (and probably non-portable) behavior. See also . = = variable assignment the assignment operator (no space before and after) Do not confuse this with = and -eq, which test, rather than assign! Note that = can be either an assignment or a test operator, depending on context. &ex15; &ex16; Variable assignment using the $(...) mechanism (a newer method than backquotes). This is actually a form of command substitution. # From /etc/rc.d/rc.local R=$(cat /etc/redhat-release) arch=$(uname -m) Unlike many other programming languages, Bash does not segregate its variables by type. Essentially, Bash variables are character strings, but, depending on context, Bash permits integer operations and comparisons on variables. The determining factor is whether the value of a variable contains only digits. &intorstring; Untyped variables are both a blessing and a curse. They permit more flexibility in scripting (enough rope to hang yourself!) and make it easier to grind out lines of code. However, they permit errors to creep in and encourage sloppy programming habits. The burden is on the programmer to keep track of what type the script variables are. Bash will not do it for you. local variables variable local variables visible only within a code block or function (see also local variables in functions) environmental variables variable environmental variables that affect the behavior of the shell and user interface In a more general context, each process has an environment, that is, a group of variables that hold information that the process may reference. In this sense, the shell behaves like any other process. Every time a shell starts, it creates shell variables that correspond to its own environmental variables. Updating or adding new environmental variables causes the shell to update its environment, and all the shell's child processes (the commands it executes) inherit this environment. The space allotted to the environment is limited. Creating too many environmental variables or ones that use up excessive space may cause problems. bash$ eval "`seq 10000 | sed -e 's/.*/export var&=ZZZZZZZZZZZZZZ/'`" bash$ du bash: /usr/bin/du: Argument list too long (Thank you, Stéphane Chazelas for the clarification, and for providing the above example.) If a script sets environmental variables, they need to be exported, that is, reported to the environment local to the script. This is the function of the export command. A script can export variables only to child processes, that is, only to commands or processes which that particular script initiates. A script invoked from the command line cannot export variables back to the command line environment. Child processes cannot export variables back to the parent processes that spawned them. --- positional parameters parameter positional arguments passed to the script from the command line: $0, $1, $2, $3 . . . $0 is the name of the script itself, $1 is the first argument, $2 the second, $3 the third, and so forth. The process calling the script sets the $0 parameter. By convention, this parameter is the name of the script. See the manpage for execv. After $9, the arguments must be enclosed in brackets, for example, ${10}, ${11}, ${12}. The special variables $* and $@ denote all the positional parameters. &ex17; Bracket notation for positional parameters leads to a fairly simple way of referencing the last argument passed to a script on the command line. This also requires indirect referencing. args=$# # Number of args passed. lastarg=${!args} # Or: lastarg=${!#} # (Thanks, Chris Monson.) # Note that lastarg=${!$#} doesn't work. Some scripts can perform different operations, depending on which name they are invoked with. For this to work, the script needs to check $0, the name it was invoked by. There must also exist symbolic links to all the alternate names of the script. See . If a script expects a command line parameter but is invoked without one, this may cause a null variable assignment, generally an undesirable result. One way to prevent this is to append an extra character to both sides of the assignment statement using the expected positional parameter. variable1_=$1_ # Rather than variable1=$1 # This will prevent an error, even if positional parameter is absent. critical_argument01=$variable1_ # The extra character can be stripped off later, like so. variable1=${variable1_/_/} # Side effects only if $variable1_ begins with an underscore. # This uses one of the parameter substitution templates discussed later. # (Leaving out the replacement pattern results in a deletion.) # A more straightforward way of dealing with this is #+ to simply test whether expected positional parameters have been passed. if [ -z $1 ] then exit $E_MISSING_POS_PARAM fi # However, as Fabian Kreutz points out, #+ the above method may have unexpected side-effects. # A better method is parameter substitution: # ${1:-$DefaultVal} # See the "Parameter Substition" section #+ in the "Variables Revisited" chapter. --- &ex18; --- shift command shift The shift command reassigns the positional parameters, in effect shifting them to the left one notch. $1 <--- $2, $2 <--- $3, $3 <--- $4, etc. The old $1 disappears, but $0 (the script name) does not change. If you use a large number of positional parameters to a script, shift lets you access those past 10, although {bracket} notation also permits this. &ex19; The shift command works in a similar fashion on parameters passed to a function. See . " special character " ' special character ' quote \ special character \ escape Quoting means just that, bracketing a string in quotes. This has the effect of protecting special characters in the string from reinterpretation or expansion by the shell or shell script. (A character is special if it has an interpretation other than its literal meaning, such as the wild card character -- *.) bash$ ls -l [Vv]* -rw-rw-r-- 1 bozo bozo 324 Apr 2 15:05 VIEWDATA.BAT -rw-rw-r-- 1 bozo bozo 507 May 4 14:25 vartrace.sh -rw-rw-r-- 1 bozo bozo 539 Apr 14 17:11 viewdata.sh bash$ ls -l '[Vv]*' ls: [Vv]*: No such file or directory In everyday speech or writing, when we quote a phrase, we set it apart and give it special meaning. In a Bash script, when we quote a string, we set it apart and protect its literal meaning. Certain programs and utilities reinterpret or expand special characters in a quoted string. An important use of quoting is protecting a command-line parameter from the shell, but still letting the calling program expand it. bash$ grep '[Ff]irst' *.txt file1.txt:This is the first line of file1.txt. file2.txt:This is the First line of file2.txt. Note that the unquoted grep [Ff]irst *.txt works under the Bash shell. Unless there is a file named first in the current working directory. Yet another reason to quote. (Thank you, Harald Koenig, for pointing this out. Quoting can also suppress echo's appetite for newlines. bash$ echo $(ls -l) total 8 -rw-rw-r-- 1 bozo bozo 130 Aug 21 12:57 t222.sh -rw-rw-r-- 1 bozo bozo 78 Aug 21 12:57 t71.sh bash$ echo "$(ls -l)" total 8 -rw-rw-r-- 1 bozo bozo 130 Aug 21 12:57 t222.sh -rw-rw-r-- 1 bozo bozo 78 Aug 21 12:57 t71.sh When referencing a variable, it is generally advisable to enclose its name in double quotes. This prevents reinterpretation of all special characters within the quoted string -- the variable name It also has side-effects on the value of the variable (see below) -- except $, ` (backquote), and \ (escape). Encapsulating ! within double quotes gives an error when used from the command line. This is interpreted as a history command. Within a script, though, this problem does not occur, since the Bash history mechanism is disabled then. Of more concern is the inconsistent behavior of \ within double quotes. bash$ echo hello\! hello! bash$ echo "hello\!" hello\! bash$ echo -e x\ty xty bash$ echo -e "x\ty" x y (Thank you, Wayne Pollock, for pointing this out.) Keeping $ as a special character within double quotes permits referencing a quoted variable ("$variable"), that is, replacing the variable with its value (see , above). Use double quotes to prevent word splitting. Word splitting, in this context, means dividing a character string into a number of separate and discrete arguments. An argument enclosed in double quotes presents itself as a single word, even if it contains whitespace separators. variable1="a variable containing five words" COMMAND This is $variable1 # Executes COMMAND with 7 arguments: # "This" "is" "a" "variable" "containing" "five" "words" COMMAND "This is $variable1" # Executes COMMAND with 1 argument: # "This is a variable containing five words" variable2="" # Empty. COMMAND $variable2 $variable2 $variable2 # Executes COMMAND with no arguments. COMMAND "$variable2" "$variable2" "$variable2" # Executes COMMAND with 3 empty arguments. COMMAND "$variable2 $variable2 $variable2" # Executes COMMAND with 1 argument (2 spaces). # Thanks, Stéphane Chazelas. Enclosing the arguments to an echo statement in double quotes is necessary only when word splitting or preservation of whitespace is an issue. &weirdvars; Single quotes (' ') operate similarly to double quotes, but do not permit referencing variables, since the special meaning of $ is turned off. Within single quotes, every special character except ' gets interpreted literally. Consider single quotes (full quoting) to be a stricter method of quoting than double quotes (partial quoting). Since even the escape character (\) gets a literal interpretation within single quotes, trying to enclose a single quote within single quotes will not yield the expected result. echo "Why can't I write 's between single quotes" echo # The roundabout method. echo 'Why can'\''t I write '"'"'s between single quotes' # |-------| |----------| |-----------------------| # Three single-quoted strings, with escaped and quoted single quotes between. # This example courtesy of Stéphane Chazelas. Escaping is a method of quoting single characters. The escape (\) preceding a character tells the shell to interpret that character literally. With certain commands and utilities, such as echo and sed, escaping a character may have the opposite effect - it can toggle on a special meaning for that character. used with echo and sed \n \n escaped character \n newline means newline \r \r escaped character \r carriage return means return \t \t escaped character \t tabulation means tab \v \v escaped character \v vertical tabulation means vertical tab \b \b escaped character \b backspace means backspace \a \a escaped character \a alert beep flash means alert (beep or flash) \0xx \0xx escaped character \0xx octal ASCII translates to the octal ASCII equivalent of 0xx &escaped; See for another example of the $' ' string expansion construct. \" \" escaped character \" quote gives the quote its literal meaning echo "Hello" # Hello echo "\"Hello\", he said." # "Hello", he said. \$ \$ escaped character \$ dollar gives the dollar sign its literal meaning (variable name following \$ will not be referenced) echo "\$variable01" # results in $variable01 \\ \\ escaped character \\ double backslash gives the backslash its literal meaning echo "\\" # Results in \ # Whereas . . . echo "\" # Invokes secondary prompt from the command line. # In a script, gives an error message. The behavior of \ depends on whether it is itself escaped, quoted, or appearing within command substitution or a here document. # Simple escaping and quoting echo \z # z echo \\z # \z echo '\z' # \z echo '\\z' # \\z echo "\z" # \z echo "\\z" # \z # Command substitution echo `echo \z` # z echo `echo \\z` # z echo `echo \\\z` # \z echo `echo \\\\z` # \z echo `echo \\\\\\z` # \z echo `echo \\\\\\\z` # \\z echo `echo "\z"` # \z echo `echo "\\z"` # \z # Here document cat <<EOF \z EOF # \z cat <<EOF \\z EOF # \z # These examples supplied by Stéphane Chazelas. Elements of a string assigned to a variable may be escaped, but the escape character alone may not be assigned to a variable. variable=\ echo "$variable" # Will not work - gives an error message: # test.sh: : command not found # A "naked" escape cannot safely be assigned to a variable. # # What actually happens here is that the "\" escapes the newline and #+ the effect is variable=echo "$variable" #+ invalid variable assignment variable=\ 23skidoo echo "$variable" # 23skidoo # This works, since the second line #+ is a valid variable assignment. variable=\ # \^ escape followed by space echo "$variable" # space variable=\\ echo "$variable" # \ variable=\\\ echo "$variable" # Will not work - gives an error message: # test.sh: \: command not found # # First escape escapes second one, but the third one is left "naked", #+ with same result as first instance, above. variable=\\\\ echo "$variable" # \\ # Second and fourth escapes escaped. # This is o.k. Escaping a space can prevent word splitting in a command's argument list. file_list="/bin/cat /bin/gzip /bin/more /usr/bin/less /usr/bin/emacs-20.7" # List of files as argument(s) to a command. # Add two files to the list, and list all. ls -l /usr/X11R6/bin/xsetroot /sbin/dump $file_list echo "-------------------------------------------------------------------------" # What happens if we escape a couple of spaces? ls -l /usr/X11R6/bin/xsetroot\ /sbin/dump\ $file_list # Error: the first three files concatenated into a single argument to 'ls -l' # because the two escaped spaces prevent argument (word) splitting. The escape also provides a means of writing a multi-line command. Normally, each separate line constitutes a different command, but an escape at the end of a line escapes the newline character, and the command sequence continues on to the next line. (cd /source/directory && tar cf - . ) | \ (cd /dest/directory && tar xpvf -) # Repeating Alan Cox's directory tree copy command, # but split into two lines for increased legibility. # As an alternative: tar cf - -C /source/directory . | tar xpvf - -C /dest/directory # See note below. # (Thanks, Stéphane Chazelas.) If a script line