[HN Gopher] Shellcheck finds bugs in your shell scripts
___________________________________________________________________
Shellcheck finds bugs in your shell scripts
Author : mooreds
Score : 211 points
Date : 2023-11-23 19:16 UTC (3 hours ago)
(HTM) web link (www.shellcheck.net)
(TXT) w3m dump (www.shellcheck.net)
| DamonHD wrote:
| Nice: I have learnt some things from this on the very first
| production /bin/sh script that I pointed it at, and I've been
| hacking such scripts since the 80s!
| Dries007 wrote:
| Shameless self-plug:
|
| I very much live by the "fix all warnings before you commit" (or
| at least before you merge), so I have Shellcheck and a bunch of
| other linters set up in my pre-commit configurations. But the
| majority of the shell in most of my projects ends up embedded in
| .gitlab-ci.yml files, where it's hard to check. So I made a
| wrapper that does that automatically:
| https://pypi.org/project/glscpc/.
|
| It uses the Shellcheck project and some magic to give you the
| Shellcheck remarks with (mostly) accurate line-numbers.
| brirec wrote:
| I'd love to see a project that would do this, but more
| generally.
|
| I don't use GitLab CI, but I do use a good handful of other
| file types that essentially inline shell scripts. Dockerfiles,
| GitHub Actions, and Justfiles, just to name a couple.
|
| Usually, and almost exclusively for the sake of ShellCheck, I
| make a point of putting anything more complex than a couple of
| commands into their own shell scripts that I call from the
| inlined script in my Dockerfile.
|
| (This pattern also helps me keep my CI from being too locked
| into GitHub Actions)
| stouset wrote:
| This is the way
| iamjackg wrote:
| Absolutely agree. The main downside of that pattern is that
| it doesn't work with jobs included from other projects in
| GitLab CI, since the job runs in the context of the project
| that imported it and therefore can't find the script in its
| original repo. Huge bummer.
| vinnymac wrote:
| Bundle up configuration and scripts, and now we have
| containerized CI infrastructure.
| vinnymac wrote:
| I'd rather see CI services add a mode that would enforce all
| scripts must live in separate files, rather than inline.
|
| It's really not necessary to support inline except for single
| lines that are very short (under 30 chars).
| Piraty wrote:
| so much this. it will help CI not become a holy, super-
| hard-to-debug, unreproducible mammoth. write scripts, call
| them from CI
| Dries007 wrote:
| I would say: stick to straight command sequences with
| variables, avoid if/for/while/functions etc. Subshells and
| pipes only for trivial things.
|
| Setting something like a 30 char limit will inevitably lead
| to code-golfing to fit a mess into the 30 char limit.
|
| But that becomes harder to automatically check, that's why
| you should still have good peer reviews based on written
| standards somewhere.
| plorkyeran wrote:
| I am not generally a fan of Xcode Cloud's design, but this
| is one thing I think it gets very right. Rather than let
| you specify the build actions in the CI settings, it
| invokes scripts with fixed names in the `ci_scripts`
| directory of your repo if they exist. There just isn't a
| mechanism for creating jobs which aren't something that you
| can build locally and test independently of Xcode Cloud.
| verdverm wrote:
| You might like Dagger, building images with code, uses the
| same buildkit engine under the hood
|
| No more linear Dockerfile, use the powers of your preferred
| language
| marcosnils wrote:
| Hi there, Dagger contributor here. We're solving this exact
| same problem by allowing you to encode your CI/CD pipelines
| in your favorite programming language and run them the same
| way locally and/or any CI provider (Gitlab, Github Actions,
| Jenkins, etc).
|
| We're very active in our Discord server if you have any
| questions! https://discord.gg/invite/dagger-io
| Dries007 wrote:
| We've been experimenting with Dagger here, as part of the
| alternative to writing glscpc actually, but I'm not
| convinced it's ready to replace Gitlab CI.
|
| Dagger has one very big downside IMO: It does not have
| native integration with Gitlab, so you end up having to use
| Docker-in-Docker and just running dagger as a job in your
| pipeline.
|
| This causes a number of issues:
|
| It clumps all your previously separated steps into a single
| step in the Gitlab pipeline. That doesn't matter too much
| for console output (although it does when your different
| steps should run on different runners), but is very
| annoying if you use Gitlab CI's built in parsing of
| junit/coverage/... files, since you now have extra layers
| of context to dig trough when tests fail etc. Plus not all
| of these allow for multiple input files, so now you have to
| add extra merging steps.
|
| If your job already uses Docker-in-Docker for something,
| you have to be careful not to end up with Docker-in-Docker-
| in-Docker situations, or container name conflicts if you
| just pass through the DOCKER_HOST variable.
|
| The one thing that would make this worth it is being able
| to run the pipelines locally to debug, but I've just
| written quick-and-dirty scripts to do that every time I've
| needed it. For example, running the test job in our
| pipeline on every Python version:
| https://gitlab.com/Qteal/oss/gitlabci-shellcheck-
| precommit/-...
| Dries007 wrote:
| Generalizing this is non-trivial (I tried initially) but I'm
| sure others can build in the same principles.
|
| I think this comes close for Dockerfiles:
| https://hadolint.github.io/hadolint/ Just have to write a
| pre-commit hook for it.
| iamjackg wrote:
| Oh, that's really cool. I was trying to solve this from a
| different perspective a while ago: I wanted to add some pre-
| processing that would take a "normal" shell script and render
| it to the `script` part of the corresponding job at build time,
| the advantage being that you still have everything self-
| contained in the gitlab-ci job.
|
| I stopped working on it because dealing with shell shenanigans
| in the GitLab CI runner environment is such a miserable
| experience that we're in the process of moving all our jobs to
| python scripts.
| Dries007 wrote:
| Yea, Pythonifying the scripts is also generally my
| preferences the moment they become somewhat complex. But even
| then it's nice that you can be reasonably sure you're not
| forgetting quotes around variables or using bash constructs
| where you only have sh.
| ulrischa wrote:
| Saves a lot of time for tons of legacy shell scripts
| hapulala89 wrote:
| I have a colleague that writes alot of shellscripts and there
| is an ongoing discussion if shellcripts or scripting languages
| like python is better.
| agumonkey wrote:
| i remember trying to rewrite so fragile scraping bash script
| in python, and even with some effort to use nice libs and
| create some cool helpers it ended up as long and not much
| more solid
|
| bash is infectious in the bad sense :D
| pram wrote:
| It's perfectly fine for glue type stuff in a CI pipeline imo.
| There's frankly no easier way to work with files.
| sneed_chucker wrote:
| Python's subprocess/shell-out story is just bad enough that I
| still find myself writing a lot of shell scripts if the task
| at hand warrants more than 2-3 subprocess calls.
|
| Realistically, Perl or Ruby would fill this role fine, but I
| hate adding another language to a project just for that
| purpose.
| pletnes wrote:
| Agree, and also you have to write a few lines of shell to
| get your python going (and same for node or ruby or
| whatever).
| ovex wrote:
| From a security point of view, Python is better because it is
| less of a footgun. So if you expose an interface to untrusted
| users, you should use Python because its behavior is more
| intuitive. An arithmetic expansion or missing quotes do not
| easily become a vulnerability in Python.
| Schnitz wrote:
| We ported all shell scripts to Python at a company that I've
| worked for. Scripts just kept getting longer and more
| complex. As a language Python is great, super fast and easy
| to code in, very little inherent complexity. The reason I
| wouldn't choose Python again is distribution. It's easy
| enough in Docker, you can just bite the bullet and vendor the
| same Python in all Docker containers. Mac was a pain though,
| pyenv etc all had their own issues and collisions with
| homebrew and dependency management with pip is a hassle as
| big as npm. A real bummer given how well Python works as a
| language for scripting.
| wittekm wrote:
| Shellcheck is great, but dealing with source/imports is suuuch a
| pain. Not their fault sh is a nightmare.
| johnchristopher wrote:
| Well, it's possible to do this: # shellcheck
| source=./deployment/deployment-example.env . "${1}"
|
| But I see how it's a pain point when you have multiple subshell
| scripts and files to source.
| eddtries wrote:
| I also recommend https://github.com/bach-sh/bach when you have to
| use Bash for things long enough it probably shouldn't be!
| seb1204 wrote:
| The page is thanking Mercedes Benz? That came unexpected.
| popcalc wrote:
| https://github.com/orgs/mercedes-benz/sponsoring
|
| They're sponsoring quite a few devs. Caddy, curl, and SeaweedFS
| notably.
| belval wrote:
| That gives me a new found respect for Mercedes-benz.
| frizlab wrote:
| But not zsh scripts, sadly
| cglong wrote:
| zsh was originally supported, but unceremoniously removed:
| https://github.com/koalaman/shellcheck/issues/298
|
| I've had great experiences with this tool, but, for some
| reason, this issue always makes me question taking too great a
| dependency on it.
| rascul wrote:
| There is also a bash language server.
|
| https://github.com/bash-lsp/bash-language-server/
| lolc wrote:
| My take is that bugs in sh scripts are best avoided by not
| programming in sh. So my preferred tool for sh linting is git-rm.
| It's not always possible, but driving down the sh line count sure
| helps against bugs from this language and its weird expansion
| rules.
|
| Most people don't even know the language has expansion rules and
| write stuff that accidentally works after the fourth try. This
| lang wants to become obsolete.
| dimitar wrote:
| I agree, but some bash can be unavoidable. I've found that even
| trivial looking bash can be helped with shell-check; this is a
| testament to the issues in bash more than anything.
| jzwinck wrote:
| What bash is unavoidable? The aliases and functions you
| define in your personal shell, sure. But what else?
| auselen wrote:
| Piping stuff, job control?
| synergy20 wrote:
| there are many cases that you have to use sh scripts, e.g. many
| IoT devices, embedded devices etc where python etc are just
| huge and slow.
| uxp8u61q wrote:
| If you wouldn't program it in Python, you wouldn't program it
| in sh either. You don't run shell scripts on embedded
| devices! Typically, an embedded device runs _one_ program
| that basically acts as the whole OS for the device. There 's
| no kernel or userspace to speak of. You're directly
| interfacing with the hardware, and that's it.
| pletnes wrote:
| This just isn't true. Smart TVs run some linux/android, so
| do car infotainment, the list goes on.
|
| Old tumble dryers, vacuum cleaners - sure.
| uxp8u61q wrote:
| These appliances can run python scripts just fine, then.
| Try to read the whole context instead of focusing on one
| part of the comment. I'm writing in the context of an
| appliance that can't run python script. That means the
| resources are heavily constrained.
| treis wrote:
| Eh, there's like a 10x difference in speed between Python
| and Java/Go and probably like a 100x between Python and
| shell stuff. Definitely some devices in that range that
| can do shell stuff but not Python.
| cjaybo wrote:
| Are you implying that Bash is 10x faster than Java or Go?
| treis wrote:
| Not Bash but the libraries they call out to.
| lachlan_gray wrote:
| This probably explains why Mercedes-Benz is on the list
| of sponsors
| IshKebab wrote:
| Properly designed IoT devices wouldn't have Bash at all in my
| book.
| kkfx wrote:
| Did you know properly designed IoT devices on sale?
| Personally I have some IoT at home to automate the home
| itself especially for p.v. self-consumption and the best I
| was able to find and integrate can be described as crap...
| I failed to fined anything else...
|
| A simple example: I like to have some electricity
| switch/breakers automation, the best I've found are from
| Shelly Pro series, witch have a not really useful webserver
| built-in and not really useful APIs the rest are even worse
| having no wired versions at all. Why the hell not offer
| manual breaker + two wires for modbus so I do not need to
| fit ethernet wires and power in the same place?
|
| Why just finding classic ModBUS-tcp/MQTT wired devices is
| so hard?
|
| Things meant to be integrated does not need shiny UIs, need
| effective ways to integrate them, simple and reliable coms.
| My VMC witch is not a dirty cheap device have mobus
| support, unfortunately even the vendor do not know a full
| list of all registry and many of them does works
| "sometimes" like "write a 1 to switch from heating to
| passive ventilation", "sometimes works", so to integrate it
| I need to check if the command was "accepted" after 30"
| then re-check it after 40 because sometimes it flip back
| for unknown reasons... And the list is long...
| morelisp wrote:
| Not strident enough. Properly designed Ts wouldn't have I
| at all in my book.
| diego_sandoval wrote:
| Possibly dumb question: Why not write it in a compiled
| language like C or Go?
| synergy20 wrote:
| because it's a script, we have bash and c on a linux and we
| need both, same to embedded devices.
| paulddraper wrote:
| Because then you need a computer and build process.
|
| Or, pre compile it across all target platforms and have an
| install process.
| kkfx wrote:
| Personally because it's quick. Sometimes I just need to
| automate some set of CLI commands... Of course sometimes
| things evolve and it's time to replace the script with
| something more easy to handle at the new scale, but for
| simple stuff, meaning something that can fit a single page
| or two they are far quicker.
|
| BTW in a broad topic: a classic system with a user-
| programming language as the topmost human computer
| automation/interface is obviously better, but we have had
| such systems in the past and commercial reasons have push
| them to oblivion so...
| justapassenger wrote:
| Bash is perfect for interacting with console tools.
|
| If that's what you need to do, C/go won't only be much
| longer code, but also much harder and error prone. Command
| line tools are complex to deal with and there's no magic
| bullet language for that.
| bluGill wrote:
| Bash is usful for 100 lines scripts that do little logic and
| mostly chain together various commands. Setup the right CC
| variables and call make.
| leosarev wrote:
| I say ten. Ten lines maximum
| paulddraper wrote:
| Okay, let's say that you want a script that counts the number
| of lines in files versioned by git.
|
| You'd write a Python3 script I assume? With subprocess?
| jzwinck wrote:
| Toy examples should not guide larger decisions. And even if
| that trivial script you describe is really what goes into
| production today, tomorrow someone will modify it and
| introduce a quoting bug or a poorly-done command line option
| facility or whatever.
| paulddraper wrote:
| Huh?
|
| What makes this a plaything?
|
| Did you respond to the right comment?
| jzwinck wrote:
| You asked about:
|
| > a script that counts the number of lines in files
| versioned by git
|
| I'm saying that is not a realistic production program
| that most of us would need.
|
| If you want it as a personal utility to use in your own
| shell, absolutely you can use bash. I'm responding to the
| idea that such a trivial script would have long term use
| in production.
| jenscow wrote:
| To mitigate that, recently there was something posted on HN
| that checks your shell script for those types of bugs.
|
| However, let's not produce any software at all, in case
| someone introduces a bug in it later.
| tgv wrote:
| Don't use bash, don't use C, don't use C++, don't use Python,
| don't use Javascript, don't use Ruby, ...
| paulddraper wrote:
| Don't use computers.
|
| Only winning move
| fooker wrote:
| There are two kinds of languages, ones that everyone complains
| about, and those that nobody uses.
| devnullbrain wrote:
| Everyone uses Python
| spoiler wrote:
| People complain about various python and its ecosystem's
| quirks all the time, though!
| justapassenger wrote:
| If you have to interact with console tools, nothing beats bash.
|
| If you don't have to, you should never use it.
| Alupis wrote:
| One does not program in bash. It is a scripting language -
| there's a difference, even if subtle.
|
| Just like any other tool, commit the time to learn it instead
| of just complaining it's hard.
|
| Developers tend to think they can write amazing things with
| minimal effort and then curse the tool/lang when things turn
| out different.
|
| The world runs on C and bash scripts... and it's just fine.
| goombacloud wrote:
| To spot more common problems I recommend: alias
| shellcheck='shellcheck -o all -e SC2292 -e SC2250'
| throw0101a wrote:
| SC2292: Prefer [[ ]] over [ ] for tests in Bash/Ksh.
|
| * https://www.shellcheck.net/wiki/SC2292
|
| SC2250: Prefer putting braces around variable references
| (${my_var}) even when not strictly required.
|
| * https://www.shellcheck.net/wiki/SC2250
| ovex wrote:
| Recently, I found a privilege escalation vulnerability in a shell
| script as a result of arithmetic expansion (similar to the one
| described at https://research.nccgroup.com/2020/05/12/shell-
| arithmetic-ex...). For example, $((1 + ENV_VAR)) allows you to
| inject code if you can control $ENV_VAR.
|
| Unfortunately, shellcheck did not catch that. At least not with
| the default settings. But if you are implementing anything
| remotely security-critical, you should not be using shell anyway.
| mmsc wrote:
| Shellcheck is great. Unfortunately, its checks pale at the
| idiosyncrasies of per-version bashism.
|
| For example: set u ignored_users=()
| for i in "${ignored_users[@]}"; do echo "$i" done
|
| passes shellcheck's checks, however bash <= 4.3 will crash with
| "bash: ignored_users[@]: unbound variable". Therefore, set -u
| isn't available to use in this (valid) use-case.
|
| Shellcheck also doesn't catch the expansion of variables as key
| names in testing assoc arrays: declare -A
| my_array un='$anything' [[ -v my_array["$un"] ]]
| && return 1
|
| will will fail as "my_array: bad array subscript" because "$un"
| gets expanded to "$anything", which on a second pass, gets
| expanded to "", making the check [[ -v my_array[] ]]. Even worse,
| a value of un='$(huh)'
|
| actually gets executed: [[ -v my_array["$un"] ]]
| && return 1 -bash: huh: command not found
|
| Here's another one: in versions older than 4.3 (maybe?) these -v
| checks don't even work: $ declare -A my_array
| $ my_array["key"]=1 $ [[ -v 'my_array["key"]' ]] && echo
| exists $ [[ -v my_array["key"] ]] && echo exists $ [[
| -v $my_array["key"] ]] && echo exists $ [[ -v
| "$my_array["key"]" ]] && echo exists $ bash --version
| GNU bash, version 4.2.46(1)-release (x86_64-redhat-linux-gnu)
|
| I've recently been documenting some of this on my website:
| https://joshua.hu/more-fun-with-bash-ssh-and-ssh-keygen-vers...
| throw0101a wrote:
| Lots of mentions of this:
|
| * https://news.ycombinator.com/from?site=shellcheck.net
|
| with the last large-scale discussion (301 points; 54 comments)
| being in 2021:
|
| * https://news.ycombinator.com/item?id=27030504
| pvg wrote:
| It's actually 'follow-up dupe' of this
| https://news.ycombinator.com/item?id=38387464 where it comes up
| repeatedly.
| hyllos wrote:
| I've turned some time ago a build and deploy script (single
| production server) some bash scripts into Haskell using Turtle
| [1]. What I enjoyed was the ability to reduce redundancies
| significantly. It was significantly shorter code afterwards.
|
| [1] https://hackage.haskell.org/package/turtle
| mrkeen wrote:
| I recently tried Turtle but ended up throwing it out in favour
| of typed-process.
|
| Afaik a Turtle program has a single current directory, which
| makes it hard when you want to run concurrent jobs that need to
| be executed from particular directories. I partially solved the
| problem by using locks/queues/workers. But it got too much for
| me when Turtle started failing due to its current directory
| being deleted.
|
| In contrast, typed-process lets you spawn separate processes,
| and execute within a working dir (rather than needing to _cd_
| there), so it works great for big, complicated workflows.
|
| And it also has good support for OverloadedStrings, which means
| you can generally copy & paste what you would have typed into
| bash, and it just works.
|
| I also use the _interpolate_ package (with QuasiQuotes) to make
| the raw strings nicer in the source code, but it 's not
| compatible with hlint, so I'm thinking of looking for a
| different package for string-handling.
| mr-wendel wrote:
| Some tips of my own:
|
| - It's almost always preferable to put `-u` (nounset) in your
| shebang to cause using undeclared variables to be an error. The
| only exception I typically run across is expansion of arrays
| using "${arr[@]}" syntax -- if the array is empty, this is
| considered unbound.
|
| - You can use `-n` (noexec) as a poor-man's attempt at a dry-run,
| as this will prevent execution of commands.
|
| - Also handy is `-e` (errexit), but you must take care to observe
| that essentially, this only causes "naked" commands that fail to
| cause an exit. Personally, I prefer to avoid this and append `||
| fail "..."` to commands liberally.
| Calzifer wrote:
| > to put `-u` (nounset) in your shebang
|
| Any particular reason why in the shebang instead of set -u?
|
| > The only exception I typically run across is expansion of
| arrays using "${arr[@]}" syntax
|
| In Bash? Works for me. Edit: another comment mentions it as
| well. Seem to behave better in newer versions of Bash and only
| problematic in <= 4.3
| https://news.ycombinator.com/item?id=38397241 $
| bash -uc 'unset x; echo "=> ${x[@]}"' => $ bash -uc
| 'x=(); echo "=> ${x[@]}"' => $ bash -uc 'x=(); echo
| "=> ${x[0]}"' bash: x[0]: unbound variable
|
| Zsh does not like the first example but both should support:
| $ bash -uc 'unset x; echo "=> ${x[@]:-null}"' => null
|
| > Also handy is `-e` (errexit),
|
| It is unfortunately very confusion with functions. Made me like
| it less over the years.
| mr-wendel wrote:
| Using the shebang just helps highlight the fact that the rule
| is in use globally, but otherwise has no advantage to using
| `set -u`.
|
| The clarifications on `-u` and arrays are useful. I'm
| definitely used to assuming newer (... non-ancient?) versions
| of Bash are what is available.
| Xophmeister wrote:
| Using `set -u` is more portable. If your shebang is
| `/usr/bin/env bash`, which it probably should be, then you
| can't add additional command line arguments in Linux with
| older coreutils. macOS supports additional arguments,
| regardless, and in Linux, coreutils 8.30 added the `-S`
| option to `env` to get around this problem.
| augusto-moura wrote:
| The problem with "${arr[@]}" only exists on bash 3 and before,
| since bash 4, [@] will never throw unbound variables even in
| cases where the variable is truly undefined. This is still a
| problem however, because macOS, to this day, still installs
| bash v3 by default and doesn't update it automatically
| (absolute madness, the last release of bash 3 it's from 20
| years ago!).
|
| In any case, you can workaround expanding empty arrays throwing
| unbound by using the ${var+alter} expansion
| echo "${arr+${arr[@]}}"
| mmsc wrote:
| The problem with "${arr[@]}" only exists on bash 3 and
| before, since bash 4
|
| 4.4 fixed it: $ bash --version GNU
| bash, version 4.3.48(1)-release (x86_64-pc-linux-gnu) $
| declare -A arr $ set -u $ "${arr[@]}"
| -bash: arr[@]: unbound variable
| augusto-moura wrote:
| Ah, that's true, I couldn't recall which version fixed it.
| Usually I assume v4 because any other distro automatically
| updates to the latest v4 version or latest version af all.
|
| macOS is the only one out there missing on this. The other
| big feature that was only added after bash 3 and is missing
| on mac is associative arrays
| jamespwilliams wrote:
| Shellcheck is a godsend
|
| https://github.com/jamespwilliams/strictbash, I wrote this little
| wrapper a while back that you can use as a shebang for scripts.
| It runs shellcheck for you before the script executes, so it's
| not possible to run the script at all if there are failures. It
| also sets all the bash "strict mode" [0] flags.
|
| [0] http://redsymbol.net/articles/unofficial-bash-strict-mode/
| w10-1 wrote:
| Shellcheck is great, but requires some investment to tailor to
| your style
|
| Disable default checks or enable optional ones using directives:
| https://www.shellcheck.net/wiki/Directive
|
| The error checks can be pretty arcane:
| https://github.com/koalaman/shellcheck/wiki/Checks
|
| I appreciate that the text for each check is brief and usually
| includes a suggestion. I end up disabling 26xx's a lot (for
| unquoted variables to be interpreted as multiple values).
|
| Python is probably the best alternative to bash, but Swift is
| getting surprisingly good.
|
| With shwift[1] you get NIO/async APIs, operator overloading for
| shell-like locutions, and trivial access to existing executables:
| /// Piping between two executables try await echo("Foo",
| "Bar") | sed("s/Bar/Baz/") /// Piping to a builtin
| try await echo("Foo", "Bar") | map {
| $0.replacingOccurrences(of: "Bar", with: "Baz") }
|
| Scripts can easily be configured with libraries and run pre-
| compiled by using clutch[2].
|
| For cross-platform use, be sure only use libraries on all
| platforms (i.e., not Foundation). It's a pain, but at least the
| error shows up typically at compile-time instead of run-time.
|
| [1] - [shwift](https://github.com/GeorgeLyon/Shwift)
|
| [2] - [clutch - any Swift scripts in a common
| nest](https://github.com/swift-nest/clutch)
| 1vuio0pswjnm7 wrote:
| What about finding bugs in other peoples' shell scripts.
___________________________________________________________________
(page generated 2023-11-23 23:00 UTC)