[HN Gopher] How to do things safely in Bash (2018)
___________________________________________________________________
How to do things safely in Bash (2018)
Author : soheilpro
Score : 103 points
Date : 2021-04-03 15:12 UTC (7 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| belter wrote:
| Every time I think of writing some Bash I spend some time here:
| https://mywiki.wooledge.org/BashPitfalls and then ...I use
| something else.
| renewiltord wrote:
| Embarrassingly, instead of fixing the code, I fix the data for
| bash. I just make sure my input is well-formed haha. I don't
| think I have any files or folders with spaces in them on my
| computer.
|
| It may not be the way but it is a way.
| lpapez wrote:
| I tend to do this as well.
|
| This is the way.
| maxrev17 wrote:
| See the same debate ensue each time with bash - use python, they
| shout. Perhaps the overriding factor here is familiarity? Past
| looping and some basic flow control I'm guessing most people run
| out of being comfortable in bash and can get things done quicker
| in their fave Lang?
| kaba0 wrote:
| It's not about being comfortable, but about the million ways
| your shell script will fail. A file has a space in the name?
| Good chance your script failed. No file founs? Same. Multiple
| files found after expansion? Same.
| nunez wrote:
| If you're gonna Bash, definitely use shellcheck for everything.
| If you're writing Bash in vim, I'd highly recommend installing
| Syntastic and enabling the checker below:
|
| ``` let g:syntastic_sh_checkers = ["shellcheck", "-e", "SC1090"]
| ```
| geocrasher wrote:
| I've been shell scripting intermittently for a long time and have
| written some bash scripts that have seen a lot of use. This
| document really showed me how bad some of my programming is. But
| then again, the first thing I tell people when demonstrating one
| of these programs: "I am not a programmer. I am a bash scripter.
| And, a bad one." The stuff still works. But I would feel a lot
| better if I were doing things the right way. I'll be reviewing
| this next time I write a bigger s/program/script/
| inetknght wrote:
| Shellcheck helped me learn a lot about bash.
|
| https://www.shellcheck.net/
| antegamisou wrote:
| Here's a great resource for proper Bash syntax
|
| http://mywiki.wooledge.org/BashGuide
| pwdisswordfish8 wrote:
| I've recently taken to not bothering with shell scripts at all. I
| just use Python instead. The type system, while it's by no means
| state of the art, is still miles ahead of the stringly-typed
| bash. The libraries are excellent; much can be done with the
| standard library alone. The scripts are much faster (even though
| the language isn't particularly tuned for performance), since I
| don't need to spawn a subprocess for each little string
| manipulation task. And if I ever need to 'shell out' to an
| external process, there's always the subprocess module that is
| much more robust than 'set -e' ever was.
|
| Seriously, if you can avoid it, just don't write shell scripts at
| all. Use a real programming language instead.
| yeowMeng wrote:
| > The [python] scripts are much faster...
|
| Pipelines are performant; many algorithms can be expressed as
| pipelines. Bash has many built-in string manipulation
| mechanisms which do not require a sub shell.
|
| Why do people who subscribe to the "avoid shell scripts at all
| costs" ideology feel so passionately about prescribing to
| others?
|
| I like python, but if I am in a rush it's shell.
| neolog wrote:
| > Why do people who subscribe to the "avoid shell scripts at
| all costs" ideology feel so passionately about prescribing to
| others?
|
| Are you asking why people make recommendations in general?
| marcosdumay wrote:
| Pipelines are a kernel feature that can be created on Python
| too. You'll just need to abstract it in a function.
| IshKebab wrote:
| 100% agree. This article should just be "rewrite your script in
| Python". It's not the best scripting language ever, but it
| works fairly well and it's about 100 times better than Bash.
| lanstin wrote:
| All the deployment things are just wrappers to shell commands
| so if you want to help people debug user-data.sh, cron,
| ansible, docker files, or puppet, you need to understand shell.
| Also, it is better to have something in bash as code than not
| have it be fully automated. I draw the line at arrays tho. If
| you want arrays don't do bash. But if you want to just run a
| bunch of commands on your compute, bash is there.
|
| One thing I have started doing for stuff that need to go via
| ssh is I just do a here doc for an embedded shell script that I
| scp to the remote and then just use ssh to invoke it. The
| quoting and pseudotty stuff just got to be too dumb to
| understand.
|
| Also, and maybe this is just for non-interactive scripts, I
| find everything from writing them to using them is a lot easier
| if you take pains to make them idempotent, so you can run them
| one or 100 times with the same effect.
| stormbrew wrote:
| This, and I mean _very specifically_ this with python always as
| the go-to replacement, comes up every single time anyone even
| utters the words "shell" or "bash" and every time all I can
| think is that the people saying it aren't writing the same
| kinds of shell scripts I ever see, use, or wind up writing.
|
| And every time I've ever seen something that's allegedly "a
| shell script replaced with a python script" it's way more
| complicated than an equivalent shell script would have been.
|
| I just wonder, really, what "a shell script" even is to you if
| not something where you need to "shell out to external
| processes"?
| dragonwriter wrote:
| > I just wonder, really, what "a shell script" even is to you
| if not something where you need to "shell out to external
| processes"?
|
| A _replacement_ for a shell script may not itself be a shell
| script.
|
| A fairly typical python-as-shell-replacement I have that
| (among other things) creates/manipulates test resources in
| AWS corresponding to git feature branches, shells out _less_
| than it would if it was a typical shell language (using boto3
| rather than shelling to the AWS CLI, etc.) but still shells
| out to git. But it would be _replacing_ a shell script even
| if I had a git-repository-interaction library that eliminated
| the need to shell out.
| blabitty wrote:
| I wonder too. Things like file and directory manipulations
| are way more complicated in native Python and I find the
| exceptions harder to troubleshoot than native commands. I'm
| not even going to get into pythons version and dependency
| hell, suffice it to say I have more confidence in a standard
| version of bash and standard utilities generally being
| present on a modern Linux system. What I've seen from
| pythonistas as reasoning for this replacememt love is usually
| one or a combination of a few things - a claimed write once
| run anywhere ability that I think is generally fantasy, a
| belief that Python is a "real" language and shell is not
| which usually just means "I don't like shell or think it is
| unfashionable", or a belief that Python code can be made more
| reusable. The person in question almost always has a
| background of feature developer that is now doing devops type
| tasks.
|
| All that said I like python and there are tasks I absolutely
| prefer it for, usually anything to do with scraping some web
| API that returns a complex piece of JSON or xml.
| kubanczyk wrote:
| Here is a novel idea.
|
| I am allowed to call 'mv' or 'cp' binaries from bash.
|
| I am also allowed to call 'mv' or 'cp' binaries from python
| with subprocess!
|
| (Edit: bad example binaries, 'df' or 'tar' would be
| probably better.)
|
| I must choose not to install bash libraries for a bash
| script beyond builtins.
|
| I can choose not to install python libraries beyond
| builtins.
|
| It becomes a really simple tradeoff.
|
| Nobody forces you to step out of python's stdlib. No extra
| dependencies. And you can still reap benefits of a proper
| programming language.
| fillest wrote:
| It is OK to call whatever one needs actually - just
| minimize the usage of Bash language features (Python
| provides most of them). It is not complicated and
| requires no dependencies, here's a self-contained helper:
| https://gist.github.com/fillest/8d64f8fa0cdb1745bfc9c683c
| f39...
| marcosdumay wrote:
| Hum, no those people are probably writing the same kind of
| script you do. Their Python versions are indeed longer and
| more verbose.
|
| I always recommend Python anyway if you care about edge cases
| (what is not always). That longer script handles them on the
| obvious way, while the short and more readable shell script
| does something absolutely crazy every time a detail is
| different from planed. And if you try to correctly handle the
| edge cases in Bash (you'll fail), you'll get something much
| more complicated than the Python version anyway.
| chubot wrote:
| _Shouldn 't scripts over 100 lines be rewritten in Python or
| Ruby?_
|
| http://www.oilshell.org/blog/2021/01/why-a-new-shell.html#sh...
|
| tl;dr I think the main skill that is missing is being able to
| write a Python script that fits well within a shell script. The
| shell script often invokes tools written in other languages
| like C, Rust, JavaScript, etc. (most of which you didn't write)
|
| Good book on this: https://www.amazon.com/UNIX-Programming-
| Addison-Wesley-Profe...
|
| Online for free: http://www.catb.org/esr/writings/taoup/html/
| Supersaiyan_IV wrote:
| At least shell follows POSIX standard which is more than one
| can say about Python. Have witnessed self-proclaimed "shell
| replacing" python scripts invoking commands, and never checking
| the exit code. Debugging hell.
| jrockway wrote:
| Same. I feel like I spend 45 minutes every time I write a shell
| script to determine simple things like "is this environment
| variable set". Stack Overflow always has 100 answers on how to
| do this, and every one has someone replying "well that doesn't
| work if..."
|
| In general, after 20 years of using Unix, I'm starting to sour
| on the "everything is a string" and "streams are just newline
| separated plain-text records". The shell is a pretty good REPL,
| and glue like seq, find, xargs, grep, etc. are very good. But
| honestly, I find myself reaching for purpose-built tools rather
| than ad-hoc pipelines more and more. fdfind does more of what I
| want on average than find; rg does more of what I want than
| 'find | xargs grep'. I also find myself using more and more
| structured data, and write more jq pipelines than I do bash
| pipelines. Finally, I am getting more and more frustrated at
| basic shell mechanics like history handling. I have 5 terminals
| open on average, and I don't understand why C-r in one won't
| find history in another (I know why, of course, but I don't
| like it). I'm getting pretty close to just writing my own shell
| and toolchain. I feel like if the Unix shell needed to be
| fundamentally reimagined, someone would have already done it.
| But they haven't. They just hacked Unicode and remote RPCs into
| shell prompts so that pressing "enter" on an empty shell prompt
| takes 25 seconds to run. I'm not sure why people are so focused
| on the polish and not the core inadequacies, but they are, and
| I feel like I'm going to have to fix that myself in the next
| couple years...
| chubot wrote:
| Have you tried any of the shells here?
|
| https://github.com/oilshell/oil/wiki/Alternative-Shells
|
| Discussed recently:
| https://news.ycombinator.com/item?id=26121592
|
| A lot of them are "fundamentally reimagining" shell --
| actually I'd say MOST of them are; whether that's good or bad
| depends on the user's POV.
|
| Oil is reimagining shell, but also providing a graceful
| upgrade path. Out of all the shells I'd say it's most focused
| on the fundamental language and runtime, and less on the
| interactive issues (right now).
| jwilk wrote:
| A portable way to test if the variable is set:
| test -n "${VAR+x}"
| arielb1 wrote:
| When calling shell from bash (to use pipes etc.), I found it
| useful to pass variables via the environment, like this:
| def safe_call(command, **keywds): return
| subprocess.check_call(f'set -euo pipefail; {command}',
| shell=True, env=keywds) safe_call('command1 --
| "$bar" | command2 --baz="$baz" | command3 > "$output_file"',
| bar=bar, baz=baz, output_file=output_file)
| arielb1 wrote:
| The main difficulty I have with this method is that I don't
| have an easy way to pass a Python array to bash as an array of
| parameters.
| xonix wrote:
| I personally have found that AWK is a surprisingly good
| alternative to shells for scripts bigger than average.
|
| Why?
|
| 1. Portability (AWK is a part of POSIX)
|
| 2. Very clean syntax
|
| 3. Powerful associative arrays
|
| 4. Super-powerful string manipulations facilities
|
| 5. Easy shell interoperability
|
| As an example here is a simplistic build tool [1] I've developed
| in AWK. As you can check [2] it runs unchanged under
| Linux/macOS/Win (via Git Bash).
|
| [1] https://github.com/xonixx/makesure/blob/main/makesure.awk
|
| [2] https://github.com/xonixx/makesure/actions/runs/702594092
| drran wrote:
| AWK is good, but Perl is better. Perl is good, but Python is
| cleaner. Python is good, but Go is better. Go is good, but Rust
| is faster. Rust is good, but shell is simpler. Shell is good,
| but AWK is better...
| hutrdvnj wrote:
| I think the best way to write things safely is to not write them
| in Bash in the first place. Better pick a strongly typed language
| of your choice.
| macintux wrote:
| How many strongly typed languages are:
|
| + Installed everywhere
|
| + Even within 20% as concise as bash
| effie wrote:
| Indeed. Python does not win over bash in either category, and
| it is slower.
|
| I wish there was a super-fast shell-like scripting language
| everywhere that could serve as a general programming
| language. Like Perl, but simpler rules and readable syntax.
| But there is nothing. Who would have thought, I may have to
| learn Perl eventually.
| kaba0 wrote:
| How is it slower/matter at all? Goddamn bash calls an
| external process in if branches, so if a bash script is
| enough for a given process, than running python inside a vm
| inside a vm inside a vm will be still fast enough.
| Scripting doesn't require performance.
| drran wrote:
| IMHO, we need more than one language in the shell: one to
| orchestrate launching of applications, plumbing, and
| processing of their inputs and outputs, another one for
| computations with integers, floats, complex values, arrays,
| matrices, another one for text processing and parsing,
| another one for argument parsing and default values,
| another one for error handling and recovery, another one
| for pattern matching and state machines, another one for
| object-oriented programming, another one for network
| programming, another one for record-oriented programming,
| another one for security and debugging, another one for
| container management, another one for system manipulation,
| upgrading, recovering, another one for documentation, and
| so on, with advanced way to share variables and data
| between these languages (environment variables on
| steroids).
|
| It's not possible to implement such monstrosity as one
| simple, portable, small, fast, and secure binary.
| [deleted]
| gdavisson wrote:
| I agree with almost everything in this guide, but it has a couple
| of recommendations that create new problems (while solving
| others):
|
| The nullglob option ('shopt -s nullglob') makes things like 'for
| f in _.txt ' work right when there are no matching files, but
| make other commands like 'grep somepattern _.txt' change their
| behavior in ways that can cause serious trouble. grep (and many
| other commands), given no files as input, will read from standard
| input (up to the EOF); if it's running interactively and input
| hasn't been redirected, this causes the script to hang for no
| discernible reason. If input _has_ been redirected, it steals
| input that was presumably meant to be read by some other command.
| In my opinion, the problems this causes are more serious than
| what it solves.
|
| The errexit option ('set -e' or 'shopt -s errexit') can both fail
| to exit when you expect/want it to (several such situations are
| described in the guide) and also exit when you don't expect/want
| it to. The guide mentions suppressing unexpected exits with '||
| true', but it's not always obvious when this is needed, and it's
| not even consistent between versions of bash. There's a good
| parable about this (and some examples) at
| http://mywiki.wooledge.org/BashFAQ/105
|
| I really don't like trying to predict and work around
| unpredictable features like this; I'd much rather deal with
| explicit error handling, like 'commandThatMightFail || { echo
| "Aaaargh" >&2; exit 1; }'
| dang wrote:
| Discussed at the time:
|
| _Safe ways to do things in bash_ -
| https://news.ycombinator.com/item?id=17057596 - May 2018 (240
| comments)
| alblue wrote:
| If you're interested in writing safe shell scripts then check out
| shellcheck:
|
| https://github.com/koalaman/shellcheck
|
| If you're interested I've written a git hook for it that runs a
| check when you git commit:
|
| https://github.com/alblue/scripts/blob/main/shellcheck-pre-c...
|
| You should also check out her Google shell script style guide:
|
| https://google.github.io/styleguide/shellguide.html
| woudsma wrote:
| Shellcheck and the VS Code Shellcheck extension[1] are really
| good. I never write a bash script without it.
|
| Built-in autofixes and direct links to extensive documentation
| on each rule, with easy to grasp examples. It's just awesome.
|
| [1]:
| https://marketplace.visualstudio.com/items?itemName=timonwon...
| montroser wrote:
| This all has its place, and I learned a good few things from the
| article. But also, in many situations when you may not need to
| deal with completely arbitrary input, it can be nice to shed all
| of the "correct" syntax and just optimize for readability.
| nas wrote:
| The fact that you have to use quoting nearly everywhere is a
| design flaw in the Borne shell. Some shells, like Plan 9's rc,
| for example, don't expand after variable substantiation. They
| have an operator to call if you want to explicitly force
| expansion. That's so much cleaner and less error prone.
| rustshellscript wrote:
| zsh is also doing the variable substitution better than bash.
| FYI, I just released rust_cmd_lib 1.0 recently, which can do
| variable substitution without any quotes:
| https://github.com/rust-shell-script/rust_cmd_lib
| chubot wrote:
| Oil is Bourne compatible, but has a mode to opt you into the
| better behavior. Example: osh$ empty=''
| osh$ x='name with spaces.mp3'
|
| This is like Bourne shell: $ argv $empty $x
| ['name', 'with', 'spaces.mp3'] # omit empty and split
| $ argv "$empty" "$x" ['', 'name with
| spaces.mp3'] # unchanged
|
| Opt into better behavior, also available with bin/oil:
| $ shopt --set oil:basic $ argv $empty $x
| ['', 'name with spaces.mp3'] # no splitting/elision
|
| If you want to omit empty strings, you can use the maybe()
| function, which returns a 0 or 1 length array for SPLICING with
| @: $ argv @maybe(empty) "$x" ['name
| with spaces.mp3'] # omitted empty string
|
| Example of splicing arrays: $ array=("foo $x"
| '1 2') $ argv $empty @array ['', 'foo name
| with spaces.mp3', '1 2']
|
| This is called "Simple Word Evaluation":
| https://www.oilshell.org/release/latest/doc/simple-word-eval...
|
| Feedback appreciated!
|
| (Interestingly zsh also doesn't split words, but it silently
| removes empty strings).
| enriquto wrote:
| > The fact that you have to use quoting nearly everywhere is a
| design flaw in the Borne shell
|
| I disagree. This can be considered a design flaw in other
| places, like filesystems that allow filenames with spaces and
| other idiotic complexity-inducing things.
___________________________________________________________________
(page generated 2021-04-03 23:01 UTC)