% date: 2018-03-08 POSIX Shell Scripting Survival Guide ==================================== Authors: ulcer License: CC-BY-SA 4.0 Published: 2018-03-08 Updated: 2018-05-08 Link: gopher://sdf.org/0/users/ulcer/kb/kb/article-script-survival.md Mirror: https://uuuuu.github.io/article-script-survival.txt Buy me cookies: https://patreon.com/ulcer Table of contents: 1. Introduction 1.1. Note on shells: bourne shell syntax, bashisms and POSIX 1.2. Tools of trade: BSD userland, GNU coreutils 1.3. Shell scripting limitations 2. Scripting basics 2.1. echo vs printf and few more details 2.2. Variables, eval and quotation 2.3. Conditions and loops: if, while 2.4. Options parsing and validation: getopt vs getopts, case 2.5. Life without arrays 2.6. Change file 2.7. Check lock 3. Commentary on tools 3.1. Working with pipes: stdbuf and pv 3.2. Notes on grep 3.3. Notes on sed 3.4. Notes on awk 3.5. Notes on portable syntax 3.6. Notes on UTF-8 compatibility 3.7. Working with XML and HTML: curl, tidy, xmlstarlet and others 3.8. Working with JSON: jq 3.9. Working with CSV: miller 4. Advanced topics 4.1. Reading user input 4.2. Internal field separator 4.3. Command line options manupulation 4.4. Nested constructions difficulties and recursion 4.5. Libraries and trap cascading 4.6. Debugging 4.7. Testing and deploying in new environment 4.8. Networking with shell 4.9. Paste safety 5. Further reading 6. References 7. Changelog % body 1. Introduction --------------- While it is true that tasks shell solves are of limited scope, using POSIX shell/toolset you may still do plenty of day-to-day and administration/deployment/maintenance work done not caring much about platform you use - BSD or GNU. This guide was motivated by watching fellow SDFers doing common mistakes and assumes you know how to do "hello world" . It also should provide you with answer to question "how do I solve real life problems with this junk". Given number of historical tool alternatives and spreaded functionality, it's no obvious question. This guide is highly opiniated, when possible link with reasoning provided. ### 1.1. Note on shells: bourne shell syntax, bashisms and POSIX Since this guide assumes you know some basics, it was probably bash. If you didn't really dive into it (except for arrays), you should be aware that there's minimal number of differences, called bashisms [1], holding you from POSIX compliant syntax (which is subset of language features you may successfully run under any major contemporary shells without extra movements). This may also drop question about bash incompatibilities under different versions. Same bashisms curse follows ksh users. Other reason for getting rid of bashisms is real slowness of bash. While it's good for interactive usage, there is better choice, as zsh holds ultimate position at command completion. Famous ShellShock vulnerability also suggests necessity of less bloated shell for system tasks. You may check common bashisms list at [2] and use "checkbashisms" script bundled in Debian "devscripts" package [3]. Closest to POSIX standard shells are Almquist shell successors: dash and busybox ash. I usually develop scripts in dash, which is default Debian system shell. Bash usually doesn't need even minor changes for dash scripts, but it can be run in "bash --posix" mode. Zsh provides bare minimum default compatibility with POSIX syntax, requiring "zsh -c 'emulate sh; sh'" to run POSIX scripts. As you noticed, POSIX sh has one major drawback: it has no arrays. Therefore there were csh, ksh, bash and so on. One thing you should know looking at shell alternative, csh (tcsh, if you like) is dead branch [4] [5]. If you ask which shell is best compromise between POSIX compliance and bare minimum improvement - arrays, it'd be probably mksh (MirBSD ksh) which is already default Android shell. Alternative syntaxes not compatible with POSIX sh: rc, fish and zsh. Fish is contemporary effort to fish bash problems, which doesn't focus on executation efficiency. You may surely take a look at rc publication and plan9 ecosystem to spot problematic parts of traditional shell [6]. Zsh while being not compatible with POSIX syntax offers significantly improved over bash interactive experience and best currently available completion system. Shells are largely bloatware. Example comparison of manpages size between shells (using "MANWIDTH=80 manpage dash | wc -l"): dash - 1590, bash - 5742, rc - 1143. Don't be surprised when you hit bug: dash at the moment I extensively studied it (using Debian 7 version) had empty "trap" output and buggy printf operation. So don't be afraid of trying your portable work on multiple shells (e.g. dash and bash), if you're not sure of shell misbehaviour. ### 1.2. Tools of trade: BSD userland, GNU coreutils Part about GNU/Linux in linux name is about GNU ecosystem, including coreutils, like "cat", "sed", "tr" and so on. Relevant part of BSD world is called userland. Embedded systems often utilize busybox, which bundles all these tools under single binary with often too restricted functions even for POSIX. You may expect any of these tools to fully support UTF-8, to be adherent to POSIX standard and to be bug free. Maybe in the future, so prepare to switch tools on occasion. Following "do one thing and do it well", you should be aware of existing tools not to waste time attempting to solve your problem with wrong tool (like replacing newlines with sed). ### 1.3. Shell scripting limitations Small subset of shell scripting deficiencies you should be aware: - speed. Lack of arrays and fancy operations with firing subshell each time you need to do something contributes to general slowness. Yet individual operations (be it grep or awk) provide great speed benefit over scripting languages - subshells consume variables: it's hard to make piped constructs to return multiple values and impossible to carry exception-like messages, and so forth - untyped and unstructured values: shell efficiency can be extended to topic of typed/binary data flows but that's another story Following comment in Structural Regular Expressions, "the silent limits placed on line lengths by most tools can be frustrating." [7] In general, deeper you'll dive with shell scripting, more limitations you'll discover. Given all subtle details in every small facility support (and even development) is a burden. 2. Scripting basics ------------------- As promised in title, it's more like guide, so I won't duplicate detailed answers and only give solutions in order suitable for learning. ### 2.1. echo vs printf and few more details To make story short, there are differences between build-in and standalone echo implementations, there are subtle backspace and option parsing defails, and you shouldn't use echo for anything other than fixed string (preferable without backslash and other special chars). For other cases you got printf. [8] It's usually done like this: $ a=$( printf "%a\n" "${var}" | something ) Note that command substitution consumes trailing newline, so depending on what you do you may need trailing line returned back. Piping to while loop is quite common pattern: $ a=$( seq 1 5 ) ; printf "%s" "$a" | \ while IFS= read -r var; do printf "%s\n" "${var}" done Which gives only 4 result lines, as we haven't write 'printf "%s\n"'. That said, you shouldn't use backticks for command substitution, always favouring $() notation [9]. Also note that "read" by default reacts on backspaces, which is turned off with "-r" switch. For whole lines you may also use line(1) utility. In case you need sole newline character, it's done like this: $ nl=$(printf "\n "); nl="${nl% }" There's also no way to expand $'\' bashisms, so for sake of being univarsal you just printf to variable in octal. You may see redirections in certain order, which is important. Like this: $ date >/dev/null 2&>1 One thing you should remember about them: they take precedence in order they are written, so writing "2>&1 >/dev/null" won't disable stderr. ### 2.2. Variables, eval and quotation Let's go back to string printing example: $ printf "%s\n" "${var}" As you see, I wrote variable inside of curly braces inside of quotes, as that being the only safe way of using variable (quotes usage and variable expansion "${}" being separate topics). [10] Regarding "${}" safety consider next examples: $ class="core"; var="path"; core_path="${HOME}" $ eval dest=\"\$$class_$var\" $ echo "Destination: ${dest}" Which won't work until you cover "class" from second line in braces (because it's recognized as "class_"). See next example with unquoted variable evaluation: $ AWK=mawk AWK_OPTS="-W interactive" $ { while :; do date; sleep 1; done; } | $AWK $AWK_OPTS '{print}' - Next example splits unquoted QUERY_STRING according to IFS into positional params, available with $1, $2 and so on: $ set -- ${QUERY_STRING} All you have to know about "eval" operation - it best way of shooting oneself in the foot, because all your newlines, semicolons and all kinds of expansions take place. Don't do it without extreme reason and proper quotation. Unix filenames may include newlines. All the fuzz for proper quotation, "xargs -0" and such is about safety from crashes and other malicious actions (e.g. with "while read line" loops). Make it a rule to quote variables in double quotes any place you use them to prevent at least IFS split. Double quotes around command substitution ("$()") are not necessary. One note for variables scope. POSIX doesn't define "local" for local variables in functions, but you may find it in literally any shell around. In other case just use unique names and preserve shared variables (like IFS) in backups. ### 2.3. Conditions and loops: if, while Let's look at syntax from this example: [ ! -f "${list_raw}" ] || [ "${TMWW_DRYRUN}" != "yes" \ -a $(( $(date +%s) - $( stat ${STATSEC} "${list_raw}" ) )) \ -gt "${TMWW_DELTA}" 2>/dev/null ] && \ fetch_all When you write conditions, "[" is equivalent to "test" built-in (with "]" being optional decoration). It's quite powerful operator, but error messages too often lack problem cause. First of all, there's only "=" being correct (which is frequent typo). Only correct syntax for shell built-in calculations (arithmetic expansion) is "$(())": "$ i=$(( $i + 1 ))". More complex calculations are solved using expr(1), specifically when you need string manipulation functions without resorting to mastodon like awk. Thing you may often see in scripts is ":" which is equivalent to "NOP" machine instruction and does exactly nothing. Like this: $ while :; do date; sleep 1; done $ : ${2?aborting: missing second param} $ : >flush_or_create_file Few words about "if" statement. Tests like '[ "x$a" -eq "x" ]' are archaic, related to earliest shells and absolutely useless nowadays. With tests written as "test -n" or "test -z" you shouldn't ever think if variable is "empty" or "unset", but something like '[ "$b" = "" ]' is good too. "while read" piped constructs with external calls are most slow part contributing to general scripts speed. They also suffer from lack of values with newline support. Being pretty obscure case, it still may be addressed with xargs [11]: ... | tr '\n' '\0' | xargs -0 -n1 ### 2.4. Options parsing and validation: getopt vs getopts, case General note on options notation: there are short options like "-a", which can be written concatenated, like "-ab 'value'" depending on how smart your option parser is, and GNU-style long options, like "--version" and "--help" (this two are most ubiquitous for GNU tools). When you need explicitly tell option parser to stop accept options, there's handy "--" empty option: $ kill -- "-${pgid}" $ random_input | grep -- "${filter}" Note that "${filter}" variable in last example may start with dash, so it's always good to put "--" beforehand. The only "getopt" you should use is shell embedded "getopts". If it happens you need long options for thing like shell script, you really should reevaluate right tool for your task. [12] Common pattern for switching "shift" usage. NOTE: if you struggle for "yes/no" and other interactive facilities in your script, remember that you loose all scripting/piping benefits of unix filter-type programs Now we came close to "case" instruction; there are few hints to care about. First, when you treat cases, make sure you place empty case before "*" wildcard (note unquoted values): case "$s" in foo) echo bar ;; '') echo empty ;; *) echo else ;; esac You may also do basic validation: case "${input}" in *[!0-9]*) echo "Not a number" ; ;; esac These checks are limited with glob patterns (same you use in interactive sessions, like "rm *~"), so you should grep/sed for any stronger validation. ### 2.5. Life without arrays If you can't rely on kind of mksh for array support, then there's still life there. Most probably your data is line oriented, which you append/sort/uniq. Let's query it: $ a=$( seq 1 3 ); seq 40 60 | grep -F "$a" If you need to search field separated data (key-value or any kind of CSV) per line, fast lookup is done with "join" utility. Here, "storage" file is ordered by first column CSV with 1st column to be queried, second file is ordered term per line: $ join -t ',' storage request The simple way of walking string of words is parameter substitution. Let's for example split word from word list: $ array="alpha beta gamma" $ head="${array%% *}"; array="${array#* }" NOTE: if you can't remember which one for prefix and suffix, "#" is on the left (under "3") on IBM keyboard, which is prefix, thus "%" being suffix (you're reading this in english anyway). See how you can split hostname from a link: $ link="gopher://sdf.org/0/users/ulcer/" $ hostname="${link#*://}"; hostname="${hostname%%/*}" The other includes splitting by IFS and is prone to errors. See "4.2 Internal field separator" chapter. The rest rely on printf and pipes: $ a=$( printf "%s\n" "$a" | grep -v "exclude_me" ) $ result=$( printf "%s\n" "${date}"; while read line; do ... done; ) Size of variable you operate is limited to size of argument you pass to underlying exec(3) call [13]. Usually it's order of hundred of KBs. When you don't want to use awk's "getline" while-cycles, usual practice is feeding awk with multiple files and comparing end of first with NR==FNR check: $ echo www-data | awk 'NR==FNR{split($0,a,":"); b[a[1]]=$0; next} \ {print 123, b[$1]}' /etc/passwd - Furthermore, jump between files listed on command line is done with "nextfile" awk command, which is not POSIX but is widely supported (e.g. with gawk and mawk). In order to pass more data to awk(1) with line oriented data you may send it as awk variable: $ a=$( seq 1 10; ); echo | mawk -v a="$a" \ 'END{split(a,f,"\n"); for (i in f) print f[i]}' But if you think you need arrays of arrays, kinds of linked lists and so on it's time to reevaluate if you 'd be able to read this script again if you solve everything with shell/awk. ### 2.6. Change file File edition is not that trivial as you may expect. Simplest way is e.g. inplace sed editing with GNU sed "-i" switch. But what if you don't have GNU tools or want to edit file with awk? Usual template looks like this [14]: inplace() { local input tmp=$( mktemp ) [ $? -ne 0 ] && { echo failed creating temp file; exit 1; } trap "rm -f '${tmp}'" 0 input="$1" shift "$@" <"${input}" >"${tmp}" && cat "${tmp}" >"${input}" rm -f "${tmp}" } inplace "target_file" sed "s/foo/bar/" You may certainly use mv or cp, but cat here is most safe option as it preserves permissions and hard links. This is behavior ed(1) provides. Regarding attributes: cat is surely safest way of writing changes. One more or less complex example includes sharing files for creation/removal/write access between group members, which includes 2770 mode on directory, umask 002 and proper FS ACL settings. cat is only utility which won't break permissions on modification. In such environments you may additionally check files with "chmod +w". Pay attention to pending disk writes with sync utility [15]. ### 2.7. Check lock Depending on system preferences, /var/lock or /var/run/lock can be available for unpriviliged locks. Locking using mkdir(1) is preferred to other methods because it's atomic operation (no separate "check lock", then "create lock" operations) and is native. mkdir "$1" 2>/dev/null || { echo locked; exit 1; } Prepare parent path with separate call, as "mkdir -p" will ignore errors on existing target directory. 3. Commentary on tools ---------------------- When you lack arrays (and given shell execution speed) you are kind of forced to piping data to specialized filter. Luckily there are enough of them nowadays. ### 3.1. Working with pipes: stdbuf and pv You may essentially expect realtime output in piped constructs, but result depends on line buffering capability of used tools. stdbuf(1) is a wrapper which forces target tool to output string when it's ready [16]. This feature is best employed when you mangle heavy datasets. Try something like this to get better idea: $ while :; do date; sleep 1; done | mawk -W interactive '{print}' - Fast grep won't help you if you pass results over tool without line buffering. Almost every tool in your standard toolset (grep, cut, tr, etc.) requires tweaks or external wrapper. Line buffering support per tool: - grep: GNU grep has "--line-buffered" switch - mawk: has "-W interactive" switch - sed: has "-u" switch - cat: OK - gawk, nawk, tr, cut: require stdbuf wrapper Solutions outside of linux vary. See [17] for detailed explanations and solutions comparison (e.g. TCL expect "unbuffer" program). For awk "fflush()" (POSIX) may be attempted. pv(1) helps when you want to know quantity of data passed over pipe, measure network speed with netcat and so on [18]. mbuffer(1) can be used as alternative to pv, exclusive for buffering tasks. ### 3.2. Notes on grep Looks like in 2018 you can't market "grep" without "data science", "big data" and other buzzwords. Ok, here are few hints for faster grep. First of all, "-F" switch stops interpretting pattern as regular expression, which means faster grep. Other thing to consider - setting "LANG=C" instead of UTF locale and utilization of your multiple CPU cores. See relevant publications [19] for xargs(1) (see "-P" switch) and parallel(1) (note "--linebuffer" switch). Also don't forget about "-m" switch if you know you need only few matches. Again, remember about "grep --" when your search argument may start from dash. In case of fuzzy search term, there's approximate grep agrep(1). This tool searches regex patterns for given levelnstein distance (which means all kinds of letter rearrangement, like mixing, dropping or placing extra letter). Try agrep on next example to get idea: wesnoth # agrep -0 wesnoth wesnoht # agrep -1 wesnoth westnorth # agrep -2 wesnoth western # agrep -3 wesnoth GNU grep is also good for searching binary data: $ LANG=C grep -obUaP "PK\x03\x04" chrome_extension.crx Which can also be performed with binwalk(1). And if you just want to peek at file for readable strings, there's coreutils strings(1) tool. ### 3.3. Notes on sed Most important thing you should remember about sed - it's a tool and not programming language. People may write brainfuck interpreters in sed - just leave them alone. You probably know about legacy of basic and extended versions of regular expressions (try man 7 regex on Linux) - "Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems" //Jamie Zawinski. Non of these being POSIX or GNU knows about non-greedy quantifier. Usual workaround for look-ahead feature for single character looks like this: $ echo "foo https://www.ietf.org/standards/ bar" | \ sed 's|http[s]://\([^/]*\)/[^ ]*|\1|' foo www.ietf.org bar For non-greedy replacement of character sequence, target sequence is first replaced with unique for given input character and then followed previous example. If you want to avoid sed extended regexp flag inconsistency, usual thing lacking in basic regexp is "?" quantifier, which can be emulated with "\(x|\)" kind of expression (sed prefers leftmost matching part). GNU sed is unicode aware and supports lower/upper case conversion which works like this: $ echo Aa Bb Cc | sed 's/\(..\) \(..\) \(..\)/\L\1 \U\2 \E\3/' aa BB Cc Depending on implementation sed may force you to write each command under separate "-e" option and can be strict about closing semicolons ";", like here: $ sed -n '/marker/{p;q;}' example Safe assumption about sed is it doesn't know about escape sequences so you have to put desired character either literally (e.g. tab character) or from shell variable, like "tabchar=$( printf \011 )". ### 3.4. Notes on awk Which awk? Short answer is: gawk when you need UTF-8 support, mawk when you don't. Relevant literature: mawk vs gawk and other languages speed comparison [20]. mawk 1.3.3 which is most widely distributed version suffers from number of sore bugs, like lack of regex quantifiers "{}" support. Few survival hints for awk: - delete array: split ("", array) - variable passed as parameter to function: should be array, like global[0], to be mutable: $ echo | mawk 'function a(){b=5} { b=2; a(); print b}' 5 $ echo | mawk 'function a(c){c=5} { b=2; a(b); print b}' 2 $ echo | mawk 'function a(c){c[0]=5} { b[0]=2; a(b); print b[0]}' 5 ### 3.5. Notes on portable syntax When you want to know if tool is available, most portable way is using "command -v" which should be built-in: $ AWK=$( command -v mawk 2>/dev/null ); AWK="${AWK:-awk}" Sometimes you have to access user local directories, which FHS (Filesystem Hierarchy Standard) doesn't cover. Two well established paths are "~/.config" and "~/.local/share". There is also X Desktop Group approach of searching ~/.config/user-dirs.dirc with xdg-user-dir(1) with XDG_CONFIG_HOME and XDG_CACHE_HOME corresponding to previously mentioned dirs. Graceful query of local configuration path may look like this, but it's most certainly just "~/.config/": $ config="${${XDG_CONFIG_HOME:-${HOME}/.config}/ACMETOOL/config}" Temporary files can be attempted at TMPDIR (which is POSIX), XDG_RUNTIME_DIR (which is relatively recent addition), and then resorted to /tmp. Most common GNU/BSD discrepancies: - sed extended regex switch: "sed -r" vs "sed -E" - print file in reverse order: "tac" vs "tail -r" - file modification date in seconds since epoch: "stat -c %Z" vs "stat -f %Uc" BSD naming of GNU utils starts with "g", like "gawk": "gdate", "gsed", etc. BSD date is different from GNU date in some details. Further examples for ISO8601 works well for both date versions: $ date -u +'%Y-%m-%dT%H:%M:%SZ' $ echo '2018-03-05T19:24:21Z' | date +%s Last but not least, your sort(1) results surprisingly may vary between systems depending on LC_COLLATE setting. ### 3.6. Notes on UTF-8 compatibility gawk works with UTF-8 by default. awk language wasn't crafted with multibyte encoding in mind, so there could be problems if you work with binary data. By default sprintf("%c", 150) will print multibyte char like "0xC2 0x96" and fail to compare string if you pass 0x96 from shell variable like 'printf "\226"' in dash. gawk here requires "-b" switch, mawk works well by default. GNU sed is unicode aware. Other seds support vary; without proper support you won't get it working at all because of improper single char "." length. GNU "wc -c" still doesn't count UTF chars properly, tr also doesn't work here, so you have to resort to sed "y" command with GNU sed. You shouldn't rely on shell "${#var}" expansion as it still often displays number of bytes and not characters. ### 3.7. Working with XML and HTML: curl, tidy and xmlstarlet This is relevant to web page scraping. There are two URL encodings you may encounter: punycode and percent encoding. Punycode of "xn--" prefix covers internationalized domains and can be converted back and forth with idn(1). Percent encoding operations (urldecode and urlencode): $ echo 'a%20b+c' | sed "s|+| |g;s|%|\\\\x|g" | xargs -L1 printf "%b" $ printf "%s" test | od -An -tx1 | tr ' ' '%' | xargs printf "%s" Pages are usually grabbed with curl(1) or wget(1) with wget being more suitable for mirroring/recursive download. curl invocation with user agent set and requesting compressed data: $ curl -H 'Accept-encoding: gzip' | gunzip - $ curl -H 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) \ AppleWebKit/537.36 (KHTML, like Gecko) \ Chrome/40.0.2214.85 Safari/537.36' Picky servers require referer and user agent to be set. See also curl manpage for "retry" set of options for responsible downloads. If you fetch data repeatedly within some seconds, curl is responsible for timeout delays, where not a single options ensures wget to run in defined time. Don't forget to strip CR characters with "tr -d '\r'". When you're done fetching, it's time to convert your HTML source to uniformed XML. tidy(1) is corrects and reformats HTML to well-formed XHTML [21]: $ tidy -iqw0 -asxml input.html >output.xml When you finished scalpeling sample input, you may validate and pretty print it with xmllint(1), bundled with libxml(3): $ xmllint --format ugly.xml >pretty.xml Now it's time to finally make queries. xmlstarlet(1) "sel" command provides compact way of writing XSLT right on command line: $ xmlstarlet sel --help XML tree structure inspection command is handy for unfamiliar input: $ xmlstarlet el -u Finally, you may omit schema definition by using default schema shortcut "_": $ xmlstarlet sel -t -v "//_:p[@content]" One notable limitation of xmlstarlet is input file size. It doesn't provide SAX parser and will fail on files of few hundred MB size. Most important, W3 maintains set of C tools dedicated for XML/HTML works, like hxtoc TOC creation, hxpipe conversion to YAML-like structure, more usable for awk(1) parsing, hxselect for quarying with CSS selector and others [22]. ### 3.8. Working with JSON: jq JSON (RFC4627) does what it has to - serializes data in text representation without much bloat, so you can read it without kind of decoder [23]. It lacks standard validation methods, but work has been done in that direction [24]. Best tool to mutate and query JSON data is jq, which is already popular enough. I should mention that YAML with entity per line also lacks bloat of XML and is just perfect for parsing with awk without extra tools. jq is also tolerant to JSON lines input (each line is valid json expression) [25], which is compromise for mungling it with grep/awk and jq and solves disgusting problem of root braces {}. Besides, it's great for parallelization. Just an example jq call to rename field "a" to "b": $ jq -c 'with_entries(if .key == "a" then {key:"e",value} \ else . end)' ### 3.9. Working with CSV: miller Most common predecessor for CSV data is some flavour of spreadsheet. There is ubiquitour python-based csvkit, but probably more correct would be Gnumeric supplementary converter [26]: $ ssconvert input.xls output.csv Now that we have our data ready, let's process it. Without quoted values, usual tools are enough: awk with "-F" flag, cut, sort with "-k" flag. Handy shortcut for skipping CSV fields on search: $ field="[^,${tabchar}]+[,${tabchar}]" $ egrep -m1 "^${field}${field}${search_string}[,${tabchar}]" See "4.2 Internal field separator" chapter for more examples for CSV. miller(1) [27] jumps in straight where jq got its place: it's easy to drag around single binary, it combines speed of awk with features of popular scripting languages CSV tools. Recently (around 2016?) it finally got double quoted RFC4180 compliant CSV support. Depending on your language preferences, there are too much of CSV processing tools. Two of such are csvkit and q (both in python). While csvkit being obvious, q is way to query CSV with SQL [28]. 4. Advanced topics ------------------ Next problems are not related to average scripting needs. As was stated before, think twice about right choice of tool. ### 4.1. Reading user input You may attach GNU readline capabilities for any input using rlwrap, just note that it'll obviously store input history (consult rlwrap manpage). Here is Debian specific example for running SDF chat program: $ rlwrap annotate-output ssh ulcer@sdf.org 'com 2>/dev/null' For timeout of line/char reading routins, most portable solution includes stty with dd. In other case you may get "line -t", coreutils timeout(1) or bash "read -t" working. ### 4.2. Internal field separator Which you usually shouldn't touch, because this path carries only pitfalls. IFS is composed of characters, which act as field separator for unquoted values in eval loop. First IFS char works for output too. IFS can be handy when you are certain about proper composition of your input. Any user provided input without proper quotation carries possibility of execution of arbitrary code. Let's see how it works: $ tab=$( printf '\011' ) $ get_element() { eval printf \"\%s\" \$$(( $1 + 1 )) ; } $ get_csv() { local id IFS; id="$1"; shift; IFS=",${tab}" get_element $id $* ; } $ get_csv 3 'foo,\\$(date);date \$(date),bar' bar $ print_csv() { local IFS; IFS=,; printf "%s" "$*"; } $ print_csv a b c a,b,c Such code doesn't outsource tasks to external calls and especially pipes of external calls, but itself doesn't provide greater advantage over e.g. printf to cut(1) though. IFS is often used in read cycles: # read whole line while IFS= read -r line; do printf "%s\n" "${line}" done # read CSV fields with residue going into $c3 debug() { printf "<%s>\n" "$@"; } while IFS=, read -r c1 c2 c3; do debug "$c1" "$c2" "$c3" done /dev/null; then echo >&2 "Exporting: $1" plugin_array="${plugin_array} $1" . "$LIBPATH/$1" fi } Your very first inlined script will rise question about parent trap handlers preservation. Here is proposed solution: # $1 -- trap name (to prevent duplicates) # $2 -- trap command # $3 $* -- signals trap_add() { # stay silent on incorrect options and on fail to set trap [ -z "$3" ] && return printf "%s" "${trap_array}" | grep -qw "$1" 2>/dev/null && return 0 trap_array="${trap_array} $1 " trap_cmd="$2"; shift 2 for trap_signal in "$@"; do trap -- "$( extract_trap_cmd() { printf '%s\n' "$3"; } eval extract_trap_cmd $( trap | \ sed -n "H;/^trap --/h;/${trap_signal}$/{x;p;q;}" ) printf '%s\n' "${trap_cmd}" );" "${trap_signal}" || return done } # debug trap_add test 'echo 123' INT TERM EXIT trap_add test2 'date' INT TERM EXIT trap Never forget to test your code chunks in multiple shells, as e.g. dash and mksh do not provide POSIX exception for treating command substitution with empty trap as not called from subshell, where it should dump existing signal handlers (so former example will work only in bash/zsh and require preservation of all trapped signal handlers in dedicated variables for being portable). ### 4.6. Debugging Debugging shell scripts is done either with "set -x" shell option to output executed commands with timestamp, which also works well for performance troubleshooting, or with debug printfs all around the code. Usual debug routine is 'printf "debug \"%s\"\n" "$@"', which expands to param per line. hexdump(1) may be missed on target machine. Alternatives are xxd(1) often distributed with vim(1) or od(1), which od being most ubiquitous. Problem cases can be much obscure. One common pitfall is forgotten ";" inside of "$()"/"{}"/"()" constructs with closing parentheses being on same line with last instruction. ### 4.7. Testing and deploying in new environment shtest [29] is a single file in POSIX shell without any dependency. It takes different approach from unit tests in that is doesn't force you to write anything. You copy commands to be tested with some prefix (by default with "$", like any documentation does), run "shtest -r", which records those commands output and exit statuses and finally run tests in new environment (or under different shell) to get diff output if something went wrong. Also shtest tests are easily embeddable in markdown documentation. shtest was inspired by similar cram tool in python [31], just that it doesn't need python to run. For more systematic approach see BATS [32], which can output TAP-compatible messages, suitable for continuous integration. ### 4.8. Networking with shell This one is completely esoteric part, which carries little practical value. With already mentioned curl, doing its job for HTTP (and GOPHER!) protocols, it's horribly inefficient but yet possible to do simple tasks with binary data flow under just shell and awk. You still need few specialized tools. tcpick(8) encodes byte streams into hex [33]: /usr/bin/stdbuf -oL /usr/sbin/tcpick -i eth0 \ "src $1 && port 5122 && \ tcp[((tcp[12:1] & 0xf0) >> 2):2] = 0x9500" -yH | awk ... Actually, you may do full-fledged networking with just netcat and sh [34]. This one was functional online game client. Netcat stream was read word by word with dd, expect-like behavior and binary packets parsing were done in pure shell. It still serves as good example on scalpeling binary data with only shell. ### 4.9. Paste safety It's survival guide after all, so let's step away from target subject and look at how you actually do scripting. Obviously, noone reads mans and writes code by copying parts from web. See this question [35] for details and at the very least inspect your paste with "xsel -o | hd". 5. Further reading ------------------ POSIX standard is your best friend [36]. You should obviously see manpages. Dash and mawk manpages are pretty excellent and compact for learning corresponding topics and using as language reference. For GNU bloatware important info is often contained within GNU info(1) infopages (which means man pages miss what you may seek, take a look at e.g. sed). This format should certainly diy one day but so far it is accessible with info(1) or somewhat abandoned pinfo(1) for colored output. And of course is available at [37]. Then take a look at next resources: - comp.unix.questions / comp.unix.shell archived FAQ, which contains still relevant answers for particular scripting topics [38]. - harmful.cat-v.org [39] And finally few of individual widely circulated pages: - Rich's POSIX sh tricks, covering advanced topics [40] - Sculpting text with regex, grep, sed, awk [41] - 7 command-line tools for data science [42] Bash FAQ, which still covers lot of newbie POSIX-related pitfalls: [43] 6. References ------------- 1. Introduction [1] Ubuntu Wiki, "Dash as /bin/sh" https://wiki.ubuntu.com/DashAsBinSh [2] Greg's Wiki, "How to make bash scripts work in dash" https://mywiki.wooledge.org/Bashism [3] checkbashisms - check for bashisms in /bin/sh scripts https://manpages.debian.org/testing/devscripts/checkbashisms.1.en.html [4] Tom Christiansen, "Csh Programming Considered Harmful", 1996-10-06 http://harmful.cat-v.org/software/csh [5] Bruce Barnett, "Top Ten Reasons not to use the C shell", 2009-06-28 http://www.grymoire.com/unix/CshTop10.txt [6] Tom Duff, "Rc — The Plan 9 Shell" http://doc.cat-v.org/plan_9/4th_edition/papers/rc [7] Rob Pike, "Structural Regular Expressions", 1987 http://doc.cat-v.org/bell_labs/structural_regexps/se.pdf 2. Scripting basics [8] Stéphane Chazelas. "Why is printf better than echo?" https://unix.stackexchange.com/a/65819 [9] Why is $(...) preferred over `...` (backticks)? http://mywiki.wooledge.org/BashFAQ/082 [10] Fred Foo. "When do we need curly braces around shell variables?" https://stackoverflow.com/a/8748880 [11] Tobia. Make xargs execute the command once for each line of input https://stackoverflow.com/a/28806991 [12] Dennis Williamson. "Cross-platform getopt for a shell script" https://stackoverflow.com/a/2728625 [13] Graeme. "What defines the maximum size for a command single argument?" https://unix.stackexchange.com/a/120842 [14] William Pursell. "sed edit file in place" https://stackoverflow.com/a/12696585 [15] Wikipedia - sync (Unix) https://en.wikipedia.org/wiki/Sync_(Unix) 3. Commentary on tools [16] Pádraig Brady, "stdio buffering", 2006-05-26 http://www.pixelbeat.org/programming/stdio_buffering/ [17] Aaron Digulla. "Turn off buffering in pipe" https://unix.stackexchange.com/questions/25372/turn-off-buffering-in-pipe [18] Martin Streicher, "Speaking UNIX: Peering into pipes", 2009-11-03 https://www.ibm.com/developerworks/aix/library/au-spunix_pipeviewer/index.html [19] Adam Drake, "Command-line Tools can be 235x Faster than your Hadoop Cluster", 2014-01-18 https://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html [20] Brendan O'Connor, "Don’t MAWK AWK – the fastest and most elegant big data munging language!", 2012-10-25 https://brenocon.com/blog/2009/09/dont-mawk-awk-the-fastest-and-most-elegant-big-data-munging-language/ [21] The granddaddy of HTML tools, with support for modern standards https://github.com/htacg/tidy-html5 [22] HTML and XML manipulation utilities https://www.w3.org/Tools/HTML-XML-utils/README [23] Douglas Crockford, "JSON: The Fat-Free Alternative to XML", 2006-12-06 http://www.json.org/fatfree.html [24] JSON Schema is a vocabulary that allows you to annotate and validate JSON documents http://json-schema.org/ [25] JSON Lines http://jsonlines.org/ [26] The Gnumeric Manual, "Converting Files" https://help.gnome.org/users/gnumeric/stable/sect-files-ssconvert.html.en [27] Miller is like awk, sed, cut, join, and sort for CSV https://johnkerl.org/miller/doc/index.html [28] Run SQL directly on CSV files http://harelba.github.io/q/ 4. Advanced topics [29] Useless Use of Cat Award http://porkmail.org/era/unix/award.html#uucaletter [30] shtest - run command line tests https://github.com/uuuuu/shtest [31] Cram is a functional testing framework based on Mercurial's unified test format https://bitheap.org/cram/ [32] Bats: Bash Automated Testing System https://github.com/sstephenson/bats [33] tcpick with awk example https://github.com/uuuuu/tmww/blob/master/utils/accsniffer [34] shamana - tmwa ghetto bot engine made with POSIX shell https://github.com/uuuuu/shamana [35] Sam Hocevar. "How can I protect myself from this kind of clipboard abuse?" https://security.stackexchange.com/questions/39118/how-can-i-protect-myself-from-this-kind-of-clipboard-abuse 5. Further reading [36] The Open Group Base Specifications Issue 7, 2016 Edition http://pubs.opengroup.org/onlinepubs/9699919799/ [37] GNU Coreutils https://www.gnu.org/software/coreutils/manual/html_node/index.html [38] Unix - Frequently Asked Questions http://www.faqs.org/faqs/unix-faq/faq/ [39] Encyclopedia of things considered harmful http://harmful.cat-v.org/ [40] Rich’s sh (POSIX shell) tricks http://www.etalabs.net/sh_tricks.html [41] Matt Might: Sculpting text with regex, grep, sed, awk http://matt.might.net/articles/sculpting-text/ [42] Jeroen Janssens, "7 command-line tools for data science", 2013-09-19 http://jeroenjanssens.com/2013/09/19/seven-command-line-tools-for-data-science.html [43] Greg's Wiki, "Bash Pitfalls" http://mywiki.wooledge.org/BashPitfalls 7. Changelog ------------ 2018-03-08 initial release 2018-03-29 ADD missed to mention W3 hx* tools from html-xml-utils ADD few newbie awk hints 2018-04-11 ADD portable urlencode/urldecode, fast querying with "join" 2018-05-08 FIX example in 3.4 "Notes on awk" about mutable variable passed as parameter