[HN Gopher] Bash Error Handling
___________________________________________________________________
Bash Error Handling
Author : sohkamyung
Score : 212 points
Date : 2020-10-09 06:53 UTC (1 days ago)
(HTM) web link (wizardzines.com)
(TXT) w3m dump (wizardzines.com)
| major505 wrote:
| This is usefull. I know a guy who didnt treat unset vairable, so
| in the script where he would remove with rm -rf /$dir-old-back
| his script removed all directories. The problem is that all his
| backups were in an external drive mounted in a directory, so he
| removed all his backups also. A hell month for him .
| loevborg wrote:
| `-o pipefail` is not without its gotchas bash
| -eo pipefail -c "cat /etc/services | head -1; echo never got
| here"
| aidenn0 wrote:
| I hate pipefail.
|
| I find set -e also has some gotchas. '|| die' added to the end
| of each line is uglier but more obvious in many cases.
| [deleted]
| stonewareslord wrote:
| What is the gotcha here? You're telling it to pipefail and exit
| on fail, so it exits.
| tux1968 wrote:
| The gotcha is perhaps failing to recognize that cat will
| actually fail in this simple pipeline because its output is
| truncated "unexpectedly" by head exiting after completing its
| job properly. (Maybe you could argue that head should be a
| good citizen and read its input to completion)
|
| Where this will work as expected: bash -eo
| pipefail -c "head < /etc/services -1|cat ; echo got here"
| arendtio wrote:
| I like the notion of 'set -e' and at the same time I hate it.
|
| First, because it behaves inconsistently across shells/versions
| [1] and second, because it doesn't always work as expected. For
| example, when you depend on the 'set -e' behavior in a function
| and _call_ the function from within a condition, the 'set -e'
| has no effect at all. So you better don't count on 'set -e'.
|
| But don't expect me to follow my own advice, as not using 'set
| -e' isn't a good option either...
|
| [1] https://www.in-ulm.de/~mascheck/various/set-e/
| chubot wrote:
| Yup, I am fixing this with Oil:
| https://news.ycombinator.com/item?id=24740842 (2 of 4 problems
| fixed so far, more problems welcome)
|
| Try it out on your shell scripts and let me know what happens
| :) OSH has the "broken" POSIX/bash behavior to maintain
| compatibility, while Oil opts you in to the better error
| semantics.
| JeremyBanks wrote:
| https://dev.to/banks/stop-ignoring-errors-in-bash-3co5
| polyrand wrote:
| In Bash 4.4+ you can use: shopt -s
| inherit_errexit 2>/dev/null || true
|
| To make subprocesses inherit the errexit flag.
| chinigo wrote:
| Lifesaver! Without this, error handling in bash functions is a
| gigantic, unintuitive pain. Thank you for bringing this to my
| attention.
| lazyant wrote:
| I'd add TRAP DEBUG (running line by line) to the toolset
| harisund wrote:
| I thought people shouldn't post anything about bash on HN? The
| minute you post something about bash immediately you will draw
| out a whole bunch of folks from the wood works talking about how
| bash sucks and should never be used for anything more than 3 or 4
| lines and how they replaced bash with python or some thing else,
| immediately in turn drawing out a bunch of other folks talking
| about how bash should be replaced with power shell and how you
| can parse objects better ... .
| exdsq wrote:
| I had a really fun project earlier in the year prototyping a
| load testing tool for a blockchain in Bash while 4 other
| developers wrote a 'better' one in Haskell. Bash can get
| results quickly, although it's not maintainable! Still, a
| decent kloc or two of bash with performance results within the
| sprint.
| heresie-dabord wrote:
| I had become annoyed by the Python bigots who will tell us
| about how easy to read their language is because its notation
| is clean, how all its functionality is "intuitive", how any
| combination of Python-based Rube-Goldberg Machine systems is
| the best.
|
| But then came the PowerShell people...
| michaelcampbell wrote:
| Any programming topic spawns a bunch of comments about
| readability, or examples of code-golf. It's the nature of
| things.
| asicsp wrote:
| Others in this on-going series:
|
| * https://wizardzines.com/comics/environment-variables
|
| * https://wizardzines.com/comics/brackets-cheatsheet
|
| * https://wizardzines.com/comics/bash-quotes
|
| * https://wizardzines.com/comics/bash-if-statements/
| GolDDranks wrote:
| I almost always write POSIX shell instead of bash for
| compatibility; it would be nice to see collections for tips and
| tricks specifically for POSIX shell. I know, for example, that -o
| pipefail doesn't exist in plain POSIX shell. I wonder what's are
| the best practices when you can't use it.
| peterwwillis wrote:
| Read the _ash_ or _dash_ man pages and just use what they give
| you. That 's pretty much Bourne shell which is pretty much
| POSIX.
| still_grokking wrote:
| > -o pipefail doesn't exist in plain POSIX shell
|
| I think this changed lately[1]. No clue where's implemented,
| though.
|
| [1] https://www.austingroupbugs.net/view.php?id=789
| m463 wrote:
| ok some low-grade tips:
|
| cleanup="true"
|
| later: if $cleanup; then a ; else b ; fi
|
| or slightly uglier $cleanup && rm -f mytempfile
|
| I like one-line functions: die() { echo "$@"
| 1>&2; exit 1; }
| asicsp wrote:
| > _it would be nice to see collections for tips and tricks
| specifically for POSIX shell_
|
| Check out https://freebsdfrau.gitbook.io/serious-shell-
| programming/
| sigzero wrote:
| > it would be nice to see collections for tips and tricks
| specifically for POSIX shell
|
| https://www.shellscript.sh/
|
| There is a Facebook group and I have emailed the author as well
| with questions. Nice guy.
| fpoling wrote:
| I also write my scripts to stay /bin/sh compatible. If this is
| not enough, then a real scripting language should be used, not
| bash.
|
| But I very much agree that lack of pipefail is painful. If I
| know that output on the left of the pipe is small, I read it
| into a variable and then use printf | right part. If the output
| can be big, I use a helper function to emulate it that I copy-
| paste.
| tpoacher wrote:
| I'm surprised nobody's mentioned
| [expect](https://likegeeks.com/expect-command/) yet in this
| thread :)
| x87678r wrote:
| bash is not fit for any moderately complex task.
|
| I dont understand its popularity. Normally sane people who like
| unit testing, CICD, SOLID principles, quality tools end up with a
| bunch of crappy scripts holding everything together. Please
| avoid.
| peterwwillis wrote:
| It's fantastic for simple tasks, which is why so many people
| use it. It also turns out that many complex tasks can be
| reduced to a collection of simple tasks, otherwise known as "a
| bunch of crappy scripts".
|
| I encourage all my teams to avoid built-in CI/CD features and
| plugins and just script what they want in a Docker container.
| It ends up being easier to maintain, breaks less often, and is
| more portable.
| chubot wrote:
| https://www.oilshell.org/blog/2018/01/28.html#shouldnt-we-di...
| pferde wrote:
| Also known as "bash strict mode". It's a godsend if you're
| unlucky enough to have to deal with bash scripting.
|
| http://redsymbol.net/articles/unofficial-bash-strict-mode/
|
| EDIT: The comic strip would be better in three rows of two panels
| - a row for each set flag.
| pzmarzly wrote:
| Another useful flag is -x, which enables tracing (Bash will print
| each command before executing it).
|
| You can disable it again with set +x (same goes for +e, +u and
| afaik +o pipefail)
|
| Also, please use shellcheck - https://www.shellcheck.net/
|
| EDIT: Also, please don't modify scripts while they are running.
| aidenn0 wrote:
| Here's a workaround for modifying scripts while they are
| running; it works because function bodies are read and parsed
| entirely when they are defined. main() {
| script goes here exit } main "$@"
| 0xmohit wrote:
| Specifying PS4 when using the -x flag can be even more helpful
| while debugging. The variable PS4 denotes the value is the
| prompt printed before the command line is echoed when the -x
| option is set and defaults to : followed by space.
|
| A number of useful debugging tips are listed at
| <https://wiki.bash-hackers.org/scripting/debuggingtips>.
| _where wrote:
| If it's not by default there's a reason. Bash is literally
| running commands in a shell session. Think terminal session. When
| a command fails, would you want the terminal session to end?
| That'd be annoying.
|
| Same theory for unset variable. Referencing an undefined variable
| shouldn't break your session. Why initialize it anyway? It's more
| code to change if you don't use it if you have to initialize it
| when it might not be needed. And, you'd have to call the script
| with A= just to check A wasn't defined, and in the process now
| you have A assigned to an empty string, instead of only
| defaulting to one when called, which uses more memory and
| execution time.
|
| The pipeline doesn't die because && and || and parens are
| seriously helpful for one-liners.
|
| Don't think of it as a script. Think of it as a script for a
| shell.
| pwdisswordfish4 wrote:
| Sometimes the only reason is backwards compatibility.
| IshKebab wrote:
| A compelling argument for why Bash scripts shouldn't exist at
| all.
| chubot wrote:
| https://www.oilshell.org/blog/2018/01/28.html#shouldnt-we-
| di...
| Waterluvian wrote:
| Thinking of it that way, it becomes clear why you would want to
| switch to something like Python for more complex workflows.
|
| Bash is brilliantly useful for a lot of things. I'm not bashing
| it.
| cryptonector wrote:
| Except `set -e` is stupid because it is disabled all the way down
| when a command occurs in a conditional, meaning this:
| function foo { false; true; } foo || echo foo failed
|
| prints nothing!
|
| From the bash manual page:
|
| | If a compound command other than a subshell returns a non-zero
| status because a command failed while -e was being ignored, the
| shell does not exit.
|
| POSIX says the same thing, so this is true of all POSIX-y shells.
|
| This means you really have to check for errors you care about,
| and `set -e` is useless. Ugh!
| mehrdadn wrote:
| Wait until you realize set -e; (false && true);
| echo hi
|
| does nothing, but set -e; false && true ; echo
| hi
|
| prints hi.
| cryptonector wrote:
| That one is OK. What's NOT OK is that conditionals disable set
| -e in functions that appear in the conditionals, so:
| function foo { false; true; } foo || echo foo failed
|
| prints nothing.
| mehrdadn wrote:
| Ouch, that's horrible. Thanks!
| chubot wrote:
| Yeah this is a variation on problem 2 I mentioned here, except
| with subshells rather than functions.
|
| https://news.ycombinator.com/item?id=24740842
|
| The problem actually has more to do with the definition of $?
| than the set -e behavior itself. And the fact that POSIX
| specifies that the error a the LHS of && is ignored (a
| fundamental confusion between true/false and success/error)
|
| The exit code of the function is not what you expect, or the
| exit code of the subshell is not what you expect.
|
| I made a note of it on the bug ... still thinking about what
| the solution to that one is.
|
| (The other solutions are inherit_errexit, more_errexit, and a
| "catch" builtin.)
| Sebb767 wrote:
| It's a bit surprising at first, but I'd like to think most
| people deeper into bash are aware of Subshells.
| mehrdadn wrote:
| Merely knowing about when subshells occur doesn't explain
| this. You need to know that && and || suppress -e.
| alanbernstein wrote:
| I know $() invokes a subshell, but does () ?
|
| My guess was that the () affects the order of operations
| between ; and &&, so the first line is three commands, while
| the second line is two.
| asicsp wrote:
| See https://www.gnu.org/savannah-
| checkouts/gnu/bash/manual/bash....
|
| "Placing a list of commands between parentheses causes a
| subshell environment to be created"
|
| "Placing a list of commands between curly braces causes the
| list to be executed in the current shell context"
| gravitas wrote:
| It's horribly more complex than just that, the person
| you're replying to is "more correct" because of the rules
| of `set -e` which is critical in this thread; a singular
| command followed by && is treated differently than a
| compound command. Pull up `man bash` and search
| `/set.*abef` to read the rather long and involved
| paragraph for the `-e` option, running in a subshell is
| only part of it.
| asicsp wrote:
| >I know $() invokes a subshell, but _does ()_ ?
|
| >My guess was that the _() affects the order of
| operations_ between ; and &&, so the first line is three
| commands, while the second line is two.
|
| Emphasis mine. My understanding was that the question is
| simply about whether () invokes a subshell or not
| (irrespective of set -e)
| aidenn0 wrote:
| It's a subshell. Order of operations is unchanged, and if
| you just want a block of commands you can use { }
| exdsq wrote:
| Other tips I've found after a year or two of using Bash for many
| things it shouldn't be used for:
|
| - Use shellcheck (static analysis/linter)
| https://www.shellcheck.net/
|
| - Use shunit2 (unit tests) https://github.com/kward/shunit2
|
| - Use 'local' or 'readonly' to annotate your variables
|
| - Trap ctrl+c to gracefully exit (details here
| https://www.tothenew.com/blog/foolproof-your-bash-script-som...)
|
| - Stick to long-form options for readability (--delete over -d
| for example)
|
| - #!/usr/bin/env > #!/bin/bash for portability
|
| - Consider setting variable values in a file and importing it at
| the top of your script to improve refactoring
|
| - You will eventually forget how your scripts work - seriously
| consider if Bash is your best option for anything that needs to
| last a while!
| txutxu wrote:
| > - #!/usr/bin/env > #!/bin/bash for portability
|
| Yes... but do not take that as a "cargo cult script shebang".
|
| If you're a sysadmin writing a script for a company with 2k
| linux servers, that has a policy of "we only use linux version
| Foo X"... and we do not use other bash in the system than
| /bin/bash (no bash compiled by hand, no multiple versions of
| bash, etc)... then portability via "env" does not make sense.
|
| If you have two laptops and a raspberry at home, with debian or
| arch, and you write a script for yourself... then portability
| via "env" does not make sense.
|
| And last but not least... using env is slower.
|
| See: strace -fc /bin/bash -c ':'
|
| Vs strace -fc /usr/bin/env bash -c ':'
|
| On my system, that's 92 syscalls and 3 errors, Vs 152 syscalls
| and 8 errors.
|
| Just to start procesing.
|
| Diferent levels of system bloat (environment, library paths,
| etc) can give different results than my example.
|
| And as others said... if you're not using GNU/bash syntax and
| the script is really simple, the best for portbility is to go
| with /bin/sh. strace -fc /bin/sh -c ':'
|
| On my system 41 syscalls and 1 error... (and less RAM, CPU and
| pagefaults).
|
| If you're not using associative arrays, array indexes, non
| POSIX builtin options, and other bash extensions... if the
| script is just to join a few commands and variables... it pays
| the effort to write it in simple sh, both, for portability and
| performance.
| asicsp wrote:
| Some more useful links:
|
| * https://mywiki.wooledge.org/BashFAQ
|
| * https://mywiki.wooledge.org/BashGuide/Practices
|
| * https://mywiki.wooledge.org/BashPitfalls
|
| * https://devmanual.gentoo.org/tools-reference/bash/index.html
| amarshall wrote:
| Note that /bin/bash isn't completely portable (though /bin/sh
| is, that's not Bash) as some systems have it at
| /usr/local/bin/bash.
| m463 wrote:
| That's what i've learned... #!/bin/sh is generally what I use
| and it survives porting.
|
| If I need more power I usually switch to python. (Probably
| not relevant to a bash discussion though, sorry)
| laumars wrote:
| You're just reiterating the GPs point of using env instead of
| calling bash directly from the shebang:
| #!/usr/bin/env bash
|
| is more portable than #!/bin/bash
|
| Though personally if I was worried about portability then I'd
| just write the script in vanilla sh instead.
| spurgu wrote:
| > using env instead
|
| If this method is superior, why hasn't it caught on with
| Bash I wonder (it's considered standard with Python and
| Node at least)?
|
| > Though personally if I was worried about portability then
| I'd just write the script in vanilla sh instead.
|
| Agreed.
| laumars wrote:
| > _If this method is superior, why hasn 't it caught on
| with Bash I wonder (it's considered standard with Python
| and Node at least)?_
|
| Because if you need a portable bash then you target sh.
| But if you want a portable Python then you still target
| Python.
| josefx wrote:
| > But if you want a portable Python then you still target
| Python
|
| I am still supporting systems that came with Python 2.
| You get portable Python the same way you get portable
| bash: build and deploy the interpreter with your code.
| laumars wrote:
| You're missing my point. If you're targeting Python then
| you need Python installed however you don't always know
| where that executable might live. Whereas if you're
| targeting shell scripts then you can always fallback to
| regular Bourne shell if you need portability and that
| should always have an executable or simlink in /bin/sh.
|
| The Python 2 problem is a whole other topic :)
| shakna wrote:
| I believe this statement agrees with you:
|
| > #!/usr/bin/env > #!/bin/bash for portability
|
| Perhaps you misread it? env bash is greater for portability
| than bin/bash.
| jhardy54 wrote:
| Oh, I misread that as an arrow pointing from 'before' to
| 'after'. Makes sense now!
| williamdclt wrote:
| You're not the only one, I spent a minute thinking about
| a discussion explaining why the `env` way was better, I
| was going to have a rant about people giving
| contradictory advice for "portability"!
| l0b0 wrote:
| I'm working on a book about Bash scripting which is currently
| in the review phase, and it includes most of these. For
| graceful exit I recommend `trap cleanup EXIT` rather than
| specifically trapping SIGINT, mostly because the special exit
| signal is triggered no matter why the script is interrupted. I
| wouldn't normally recommend pulling out variables into a
| separate files until those variables are used by more than one
| script. I'd be interested in the rationale for why that helps
| refactoring.
| corytheboyd wrote:
| Huge +1 to using long form options in scrips, even if you're
| the sole maintainer of the script. Also if you have a command
| that takes many flags, breaking them out onto new lines can
| help keep it readable
| jancsika wrote:
| > - Trap ctrl+c to gracefully exit (details here
| https://www.tothenew.com/blog/foolproof-your-bash-script-
| som...)
|
| I'd rather that most devs don't touch that signal. Using that
| binding and having a GUI or CLI program continue hanging
| because the dev screwed up the cleanup is a real pain. And
| someone writing a Bash script is highly likely for doing
| something "very clever" with that signal to make my life
| harder.
|
| Or if you're going to do something with it, at least make it
| clear you're trolling me. Show me a text add that forces me to
| choose my favorite Korean boy band before I can exit, or
| something in that vein.
| jmholla wrote:
| In that case, you could not catch CTRL-C in the cleanup.
| nemetroid wrote:
| Agree that developers should be very careful about messing up
| Ctrl-C. However, as others have pointed out, it can make
| sense for long-running processes (especially in cases where
| there's an intermediate result that can be output instead of
| the final result). I think a good compromise is to only ever
| trap Ctrl-C once, so that a double Ctrl-C always successfully
| interrupts.
| exdsq wrote:
| That's fair! But sometimes I want to have a hook that
| basically says "Are you sure?" to catch mistakes in the wrong
| terminal or something. One thing I wrote recently took
| several hours to run and it'd suck to accidentally close it
| because I type without looking.
| gkfasdfasdf wrote:
| This is a great list. Also while reading about 'readonly' bash
| variables I ran across this amazing project which lets you call
| native functions from bash [0]. My mind is spinning from the
| possibilities...
|
| [0]: https://github.com/taviso/ctypes.sh
| mehrdadn wrote:
| Do you want to trap INT, or EXIT?
| exdsq wrote:
| EXIT is trapped in the same way as 0, it's something that
| happens when your shell exits. Ctrl+C sends the SIGINT signal
| but you can catch it with INT too. You want to do the latter
| because your gracefully exiting the script, if you want to
| have some cleanup after that you could trap EXIT (for
| deleting tmp files or something).
| jwilk wrote:
| Writing good SIGINT handler is a bit tricky. You should re-
| raise the signal, so that bash notices that the program was
| actually interrupted:
|
| https://www.cons.org/cracauer/sigint.html (section "How to
| be a proper program")
|
| Most of the time, especially for short lived script, adding
| signal handlers is not worth the trouble.
| hivacruz wrote:
| I would also add shfmt, to format scripts:
| https://github.com/mvdan/sh
| JdeBP wrote:
| I wonder how much Clippy is to blame for the visual motif used in
| this comic strip.
|
| Common motifs, elsewhere, for a shell are a dollar sign and an
| underscore or a greater than sign and an underscore. (The latter
| is somewhat odd for a shell, given that it more resembles the
| prompts on Microsoft/IBM command interpreters, and not the PS1
| prompts of Unix shells, which are commonly dollar symbols,
| hashes, or percent signs rather than greater than.)
|
| * https://www.redbubble.com/i/sticker/zsh-by-zoerab/20363330.E...
|
| * https://commons.wikimedia.org/wiki/File:Bash_Logo_black_and_...
|
| * https://commons.wikimedia.org/wiki/File:PowerShell_5.0_icon....
|
| * https://icon-library.net/icon/commands-icon-5.html
|
| * https://icon-library.com/icon/bash-icon-10.html
|
| * https://dribbble.com/shots/6101482-Bash-Automation
| chubot wrote:
| I'm working on fixing errexit / set -e in Oil:
|
| https://github.com/oilshell/oil/issues/709
|
| Summary of problems:
|
| 1. The "if myfunc" problem -- error checking is skipped. This is
| specified by POSIX, but it's undesirable.
|
| 2. The f() { test -d /tmp && echo "exists" } problem. The exit
| code of the function is probably not what you expect!
|
| 3. the local x=$(false) problem. This happens in all shells
| because of the definition of $?. (local is a shell builtin with
| its own exit code, not part of the language) This one is already
| fixed with shopt -s more_errexit in Oil.
|
| 4. the x=$(false) problem. This is a bash problem with command
| substitution. For example, dash and zsh don't have this problem.
| Test case: bash -c 'set -e; x=$(false); echo
| should not get here'
|
| Newer versions of bash fix it with inherit_errexit. Oil also
| implements inherit_errexit and turns it on by default! (in Oil,
| not OSH)
|
| -----
|
| So 1 and 2 relate to the confusing combination of shell functions
| and errexit.
|
| And 3 and 4 relate to command subs. Oil has fixed these with opt-
| in OSH shell options (on by default in Oil), but not 1 and 2.
|
| If you know of any other problems, please chime in on the bug!
| a1369209993 wrote:
| > 2. The f() { test -d /tmp && echo "exists" } problem. The
| exit code of the function is probably not what you expect!
|
| The exit code is 0 (assuming /tmp/, stdout, /bin/test and
| /bin/echo are all working correctly; with /tmp1 it's 1), as
| expected; is this referencing a bug in sh and/or bash that I've
| fixed locally and then forgotten about?
|
| (Also, I'm pretty sure it should be: f() { test
| -d /tmp && echo "exists"; }
|
| unless the parse error for missing ';' was your point (I
| haven't bothered to fix that one, but maybe Oil has).)
___________________________________________________________________
(page generated 2020-10-10 23:00 UTC)