[HN Gopher] Bash $* and $@ (2017)
___________________________________________________________________
Bash $* and $@ (2017)
Author : oftenwrong
Score : 179 points
Date : 2020-01-12 18:00 UTC (4 hours ago)
(HTM) web link (eklitzke.org)
(TXT) w3m dump (eklitzke.org)
| pletnes wrote:
| I highly recommend this site. It's very nitpicky, and that's the
| only way to write somewhat robust shell scripts.
| https://mywiki.wooledge.org/BashGuide
| dwheeler wrote:
| It's not just bash, this is true for all POSIX shells (including
| dash, bash, ksh, and so on).
|
| If you're doing a lot of complex calculations, shells are the
| wrong tool for the job. But if it's a relatively small program
| whose primary task is invoking other programs on a Unix-like
| system, shells are still a decent choice. The biggest problems
| with shells are handled by using shellcheck, so if you're writing
| shell scripts, use shellcheck.
| ljm wrote:
| Man, one of the worst debugging experiences of my life was when
| we built an extensigble build tool, based on Bash. Most of it
| worked, but there was always an inconsistency between $* and $@
| and there would be PRs that swapped those values around, back and
| forth.
|
| They were both totally valid; we just hadn't agreed on a calling
| convention for the tool so people were trying to fix their
| individual problems based on their own habits.
| a1369209993 wrote:
| > there was always an inconsistency between $* and $@
|
| There is never a legitimate reason to use $* . Even in the rare
| cases where you _want_ those semantics (hint: you don 't), you
| should use something like "$(join ' ' "$@")" instead.
|
| > They were both totally valid;
|
| No, they weren't.
| mikelward wrote:
| They are not both totally valid.
|
| You almost always want "$@".
|
| If you're using $* , you're not supporting arguments that
| contain spaces, and you'll break if arguments contain
| wildcards. If you're using "$*" , you're treating multiple
| arguments as a single argument.
|
| https://unix.stackexchange.com/a/41595/3169
| baby wrote:
| That's why I hate bash and Makefiles. The syntax is just so
| cryptic that if you don't write/read bashfiles/Makefile for a
| while it's just impossible to get back into it.
| eikenberry wrote:
| I agree to an extent about Makefiles, though I still think they
| are useful as long as you keep to a simple subset of their
| functionality.
|
| Shell scripting is a different matter. One of the great things
| about shell scripting is that you can never forget it because
| you are using it constantly to interact with your system. The
| fact that it is something you use constantly and can just take
| and stick in a file and re-use is one reason why shell scripts
| are so popular.
| 3xblah wrote:
| https://www.in-ulm.de/~mascheck/various/bourne_args
| jblow wrote:
| Why are we still using this in 2020?
| bori5 wrote:
| bash is ubiquitous, and for me who grew up pre-python, which I
| probably should start learning for bash like tasks.
| jblow wrote:
| Toxic waste sites were ubiquitous too, but we put in the
| effort to clean them up.
| grayed-down wrote:
| I'm game. What's your plan?
| yjftsjthsd-h wrote:
| Because nobody has written a good (universal) alternative.
| POSIX sh is available everywhere and is extremely well suited
| to gluing things together.
| HorstG wrote:
| If you need a portable replacement, there is Perl. Available
| since the last century. If you don't care about obscure
| Unixes, there is also Python and a whole bunch of other
| scripting languages on more modern systems.
| baby wrote:
| It never matters in practice, just bootstrap your environment
| by installing python and use a python script.
| peterwwillis wrote:
| What do you bootstrap the python environment with?
| jblow wrote:
| Obviously, but then the answer is, why? What the hell is our
| problem?
|
| Even if you think shell scripting is a good idea (which I
| don't), just fix all the obviously dumb toxic stuff like this
| and put out a shell that is minimally different with only
| semantic fixes. Emit warnings now for toxic semantics but
| still support them, and in a couple of years, turn off that
| support for good.
| gpvos wrote:
| _> in a couple of years, turn off that support for good._
|
| Make that at least 50 years, if not 100. You have no
| control over every place where shell scripts are run.
| peterwwillis wrote:
| I think the problem is the people who don't understand the
| difference between "obviously dumb toxic stuff" and
| "features".
|
| This thread is all about the difference between $* and $@.
| The difference is explained in the man pages for most
| shells, but since most programmers don't read instructions,
| they often need blog posts to explain to them how a
| documented feature of a language works.
|
| I highly recommend the _dash_ man page
| (http://man7.org/linux/man-pages/man1/dash.1.html) as a
| concise explanation of portable shell syntax. From the
| _dash_ man page, under _Special Parameters_ :
| * Expands to the positional parameters, starting
| from one. When the expansion occurs
| within a double-quoted string it
| expands to a single field with the value of each parameter
| separated by the first character of the IFS variable, or
| by a <space> if IFS is unset. @
| Expands to the positional parameters, starting from one.
| When the expansion occurs within double-quotes, each posi-
| tional parameter expands as a separate argument. If there
| are no positional parameters, the expansion of @ generates
| zero arguments, even when @ is double-quoted. What this
| basically means, for example, is if $1 is "abc" and $2 is
| "def ghi", then "$@" expands to the two arguments:
| "abc" "def ghi"
|
| It turns out we didn't need a blog post to explain it,
| because it's in the manual that nobody reads. But we should
| definitely complain about how this crafty, unusual piece of
| obviously dumb toxic stuff works, because how were you
| supposed to know to RTFM?
|
| To answer your question "why still in 2020?", it's because
| these are independent features people needed. Sometimes
| people wanted the $* semantics, and sometimes the $@
| semantics. So both exist. It's up to you to learn how the
| system works and use it properly.
|
| It's not like Python doesn't also have weird edge cases
| that you won't know until you learn the whole language.
| I've seen people spend hours futzing about with lambdas and
| list comprehensions to try to fix a bug, which I addressed
| by just rewriting the expressions as regular-old loops and
| data structures. Bash isn't uniquely bad, it has warts like
| everything else. Take out the warts you don't want and
| someone else will complain that they're missing.
| shuspect wrote:
| Just do it.
| sullyj3 wrote:
| I feel like "busy working on a different programming
| language" is one of the better excuses for not doing
| this.
| lallysingh wrote:
| http://tldp.org/LDP/abs/html/
| teddyh wrote:
| _Advanced Bash-Scripting Guide_ , working link:
|
| https://www.tldp.org/LDP/abs/html/
|
| (Your link, without "www", gives me a certificate error.)
| lallysingh wrote:
| It was http. How'd you get a cert error?
| mattbillenstein wrote:
| Oh man, wish I'd seen this a few days ago - ended up doing
| something ugly using printf and xargs...
| gpvos wrote:
| Use perl instead.
| mattbillenstein wrote:
| WORSE
| mikelward wrote:
| You should try stackoverflow or unix.stackexchange.com.
|
| e.g. my answer https://unix.stackexchange.com/a/41595/3169
| mattbillenstein wrote:
| Hmm, still not able to do what I want with this -- I want to
| take an argument like:
|
| foo.sh --clean bar.yml
|
| And actually run something like:
|
| blah -e '{"clean": true}' bar.yml
|
| where -e and the thing in '' are two separate args...
| acdha wrote:
| If you're writing shell scripts you should have
| https://www.shellcheck.net/ in your editor and pre-commit hooks
| to catch common footguns.
|
| Even then, my threshold for "this should be Python" has shrunk
| over the years: I used to say "greater than one screen of code"
| but now it's more like ">1 branch point or any non-trivial scalar
| variable".
| npmaile wrote:
| second this. We recently added it to a project at my company
| (https://github.com/homedepot/spingo) as part of a github
| action and it is awesome. A quick search for the specific code
| in the shellcheck wiki reveals the problem and a solution. I've
| had no real issues with it yet.
| skocznymroczny wrote:
| To be honest, for most of my scripting needs I usually start
| with Python directly and just go mass os.system() or
| commands.get_output() calls. Later on I refactor into
| subprocess.Popen as needed
| 3xblah wrote:
| shellcheck /scriptsdir/script
|
| I noticed this in the output. ^-- SC2148:
| Tips depend on target shell and yours is unknown. Add a
| shebang.
|
| Being new to shellcheck, not familar with options or what it
| does, so I hastily and erroneously typed:
| shellcheck -shell=bash script
|
| Note I learned UNIX via NetBSD. I prefer and use their version
| of ash _for both interactive and scripting use_.1 I never got
| used to "--" GNU-style long options. I sometimes type a single
| "-" out of habit. Anyway, here is the output I got from
| shellcheck: Unknown shell: hell=bash
|
| I agree with shellcheck.
|
| Although there may be some irony in the fact it cannot sort out
| it own argument parsing.
|
| 1. I do not use other scripting languages such as Python, Perl,
| Ruby, etc. That means, e.g., for quick and dirty one-offs and
| prototyping, I can omit the shebang. Debian's "dash" scripting
| shell is derived from NetBSD's ash, the one I choose for
| interactive use.
| therein wrote:
| Sounds like it would be content with `-sbash`.
| inetknght wrote:
| The error message is quite clear: SC2148:
| Tips depend on target shell and yours is unknown. Add a
| shebang.
|
| If you google what a shebang is, the top link for me is a
| Wikipedia article on the subject [0]. A shebang is basically
| just a line (always the first line) in a file which tells the
| operating system what program to invoke to execute the
| script. There are different shells beyond just bash, so
| shellcheck wants to know which flavor the shell is written
| for and uses the shebang to figure it out.
|
| I always have the top of my shell scripts with a shebang,
| even if the script isn't intended to be directly executed.
|
| Pick the user's bash from PATH environment:
| #!/usr/bin/env bash
|
| Or specify a specific bash: #!/bin/bash
|
| Or use whatever plain-shell is installed:
| #!/bin/sh
|
| Or maybe it's a Python script:
| #!/usr/bin/env python3
|
| Or it's a text file: #!/usr/bin/env vi
|
| If you're not using shebangs then you're probably writing
| your scripts wrongly.
|
| [0] https://en.wikipedia.org/wiki/Shebang_(Unix)
| [deleted]
| [deleted]
| kylek wrote:
| My rule of thumb is- if I think I'm going to need to use an
| array in bash, I'm probably doing something wrong and better
| just use python.
| pinopinopino wrote:
| I don't know, I often miss the pipe operator in python, which
| makes certain things much easier. I once created a small
| class to simulate this a bit, it allows you to use + as pipe
| operator. Wouldn't use it in production though, was just fun
| to write it. https://pastebin.com/eQacwLj7
| swiley wrote:
| Doesn't python have channels? The pipe is essentially a
| channel combined with a close message (a null object I
| guess) Otherwise you can do something similar with
| iterators and function composition. I guess the syntax
| isn't quite as easy to read if you're not used to function
| composition?(which seems surprisingly difficult for some
| beginners)
| fiddlerwoaroof wrote:
| Shell pipes are fairly low overhead because they are an
| os primitive.
| andrewshadura wrote:
| Tried python-sh?
| ehsankia wrote:
| Why not use | as the pipe operator? Just use __or__ instead
| of __add__
|
| I have done something similar too once in a shell-like
| Python CLI I worked on. Can also use __ror__ to be able to
| pass primitives into your commands like `[1, 2, 3] |
| SumCmd() | PrintCmd()`
|
| I do agree that typing something like that, especially when
| working in a shell, is much nicer than having to go back
| and forth all the time to type `print(sum([1,2,3]))`
|
| Here it is mentioned in a talk about Python Aesthetics by
| Brandon Rhodes:
| https://www.youtube.com/watch?v=x-kB2o8sd5c&t=8m24s
| sverhagen wrote:
| I suppose I had a more forgiving day, the other day, and told
| someone the threshold was hundred lines. Anyway, _some value_.
|
| It just challenged my colleague to make the code denser, to
| stay within the given limit. _Sigh._
| BurningFrog wrote:
| As someone who likes small fonts and big screens, for me a
| screen is often a lot more than 100 lines.
| kjeetgill wrote:
| I don't think this is like the 80 character line length
| that's about screen size. This 100 line limit is framed as
| a quick and dirty for heuristic for script complexity.
| michaelmrose wrote:
| What about some sort of complexity checking more precise than
| lines of code? What about trying to write one thing you would
| normally write in shell with something else per unit of time.
| mysterydip wrote:
| A common problem that happened to a coworker was he made a
| quick bash script for something simple, then kept adding "just
| one more" thing with sunk cost fallacy not wanting to take the
| time to rewrite it. Eventually the monstrosity created was too
| difficult to debug and it had to be rewritten in a different
| language.
| koala_man wrote:
| I keep posting this, but my favorite rule of thumb came from a
| Google dev infra engineer who said that "every Python script
| over 100 lines should be rewritten in bash, because at least
| then you won't fool yourself into thinking it's production
| quality"
| echelon wrote:
| That's a cute heuristic, but I think the better practice is
| to distrust scripts without tests as they can quickly diverge
| from the rest of the codebase.
|
| It's still better to script in Python or Ruby than Bash.
| Nobody understands Bash. It's even more mysterious than Perl.
| fiddlerwoaroof wrote:
| I'd rather write bash for orchestration than nearly
| anything else: bash is designed to make coordinating
| processes easy, something very few programming languages
| have managed to do.
| laumars wrote:
| The thing that gets me about all the new shells and shell
| scripting languages popping up these days is they loosely
| seem to fall into 2 categories:
|
| 1. more emphasis traditional programming paradigms (be it
| JS, LISP, Python, whatever) which leaves a platform that
| is arguably a better designed language but however is a
| poorer REPL environment for it. Bash works because it's
| terse and terseness is actually preferable for "write
| many, read once" style environments like an interactive
| command prompt.
|
| 2. or they spend so much effort supporting POSIX/Bash --
| including their warts -- that they end up poorer
| scripting languages.
|
| I think what we really need isn't to rewrite all our
| shell scripts in Python but rather better shells. Ones
| which work with existing muscle memory but isn't afraid
| to break compatibility for the sake of eliminating a few
| footguns. Shells that can straddle both the
| aforementioned objectives without sacrificing the other.
| But there doesn't seem to be many people trying this (I
| can only think of a couple off hand).
| jhasse wrote:
| What are your thoughts about fish?
| fiddlerwoaroof wrote:
| Does fish still have the issue where pipelines aren't
| really concurrent?
| fiddlerwoaroof wrote:
| If you haven't tried scripting in zsh, I'd give it a go.
| zsh has a lot of safety improvements and quality of life
| features.
|
| https://news.ycombinator.com/item?id=22029662
| laumars wrote:
| It's also subject to many of the same footguns as Bash so
| I'd put that into the 2nd camp (re my previous post).
|
| Not that I'm taking anything away from zsh. It is a nice
| shell. But I think we can do even better considering how
| dependant we still are on shells for day to day stuff.
| fiddlerwoaroof wrote:
| I realize that it has similar footguns, however reading
| through their info pages, I was surprised by how many
| they just decided to fix, unless you explicitly turn on
| compatibility mode.
| fiddlerwoaroof wrote:
| > Zsh had arrays from the start, and its author opted for
| a saner language design at the expense of backward
| compatibility. In zsh (under the default expansion rules)
| $var does not perfom word splitting; if you want to store
| a list of words in a variable, you are meant to use an
| array; and if you really want word splitting, you can
| write $=var.
|
| https://unix.stackexchange.com/a/26672
| yodsanklai wrote:
| The Google shell style guide [1] says this:
|
| "If you are writing a script that is more than 100 lines
| long, you should probably be writing it in Python instead.
| Bear in mind that scripts grow. Rewrite your script in
| another language early to avoid a time-consuming rewrite at a
| later date."
|
| [1] https://google.github.io/styleguide/shell.xml
| a3n wrote:
| Related, the longer your shell script gets, the greater
| potential for accumulated foot-guns. When you finally do
| the rewrite, will you be bug-compatible? And once
| discovered, will you be confident on which side the bug
| lies?
| HocusLocus wrote:
| "If you are writing a script that is more than 100 lines
| long, you should probably be writing it in _Perl_ instead.
| "
|
| This is true, especially of Python scripts.
| acemarke wrote:
| I'm a big fan of using Plumbum [0] for writing more complex
| shell-script-like logic as Python.
|
| [0] https://plumbum.readthedocs.io/en/latest/
| moopling wrote:
| Wow this looks great, thanks!
| gdevenyi wrote:
| I still find python incredibly inconvenient for pipelining a
| ton of unix style command line tools together. Bash still wins
| there.
| scbrg wrote:
| Another rule of thumb would be: "How many years from now do you
| still want this to work?" If I run a shell script I wrote ten
| years ago, it works. If I run a Python script I wrote ten years
| ago, it's quite likely to fail with a SyntaxError.
|
| This, I say as someone who _loves_ Python and I use it as my
| primary language both privately and at work. But I have to
| admit, Python scripts do not really age well.
| np_tedious wrote:
| I'm really struggling to think of an example that doesn't
| involve py2 --> py3. Can you share a few?
| thennegah wrote:
| Lol, we just made a bash function using this.
|
| // echo_and_run() { echo "$*" ; "$@" ; }
|
| Logs the cmd before running.
| gcmeplz wrote:
| You can also use `set -xv` to get nice debugging logs
|
| https://www.gnu.org/software/bash/manual/html_node/The-Set-B...
| throw7 wrote:
| At one time, I did think python would replace my bash shell.
|
| Then I tried it. Either I'm an old dog or I was naive.
|
| Both are true, I think.
| HocusLocus wrote:
| This is priceless. Countless times I have learned -- then later
| forgotten -- to use "$@" ... especially in cygwin's Windows
| 'spaces in filenames' territory
| orev wrote:
| At this point in time (i.e. 21st century), any *nix script or
| program that doesn't handle spaces in files names is woefully
| buggy. Maybe 20 years ago this was excusable, but not now.
| Spaces can reasonably be expected to be in filenames in all
| systems.
|
| Support for other "special" characters in filenames (e.g.
| newlines), however, could still be debatable.
| StillBored wrote:
| I might say that allowing spaces (newlines, quotes, asterisk
| , and various other "control" characters) in the filenames is
| the real bug. Sure its cool that you can put anything you
| want in a filename, but do you really need them? Particularly
| on a command line oriented OS? If the entire OS's experience
| was windows explorer style interactions or C binary strings
| like manipulation then fine.
|
| This causes nothing but problems, all to avoid a simple
| character filter..
|
| https://www.tecmint.com/manage-linux-filenames-with-
| special-...
|
| See how many "errors" you can find in that article. There are
| a bunch, consider the touch *, example if your using rm...
| joshuaissac wrote:
| A space is a completely normal character that users quite
| reasonably expect to be able to have in filenames. Treating
| it as a regular character is not a bug. There are lots of
| Linux programs that can deal with spaces in filenames
| already.
| Sharlin wrote:
| It just seems that the shell semantics (and to an extend
| those of other Unix tools) are specifically designed to strip
| off a level of quoting/escaping and perform word splitting
| the moment you lose your focus for a bit if you're even aware
| of all the arcane rules in the first place.
| enriquto wrote:
| > any *nix script or program that doesn't handle spaces in
| files names is woefully buggy.
|
| On the contrary! It is filesystems that support plain spaces
| in filenames that are broken. Filenames are variable names.
| Allowing separators in them is bonkers. I make a point of
| carefully crafting my scripts to wreak havoc whenever a user
| has spaces in their filenames.
| michaelmrose wrote:
| File names are an interface used by normal people to name
| files. Teaching everyone to name their files differently
| seems unlikely to succeed.
|
| There is no particular reason your shell can't distinguish
| between a space inside a filename and the space between
| tokens in output. The fact that you have to do anything at
| all yourself is a bug.
| enriquto wrote:
| > File names are an interface used by normal people to
| name files. Teaching everyone to name their files
| differently seems unlikely to succeed.
|
| Sure. But there's no reason why typing the spacebar on a
| GUI to input your filename should produce a file with a
| plain space on the filesystem. It could be a unicode non-
| breaking space, for example.
| a1369209993 wrote:
| Or, you know, a underscore, as has been de-facto standard
| for deacades?
| anoncake wrote:
| That would just make working with files programmatically
| harder. The issue isn't that file names contain spaces
| but that we use programming languages that use spaces to
| separate fields/entries instead of proper data
| structures.
| enriquto wrote:
| Do these languages allow spaces in their variable names?
|
| Filenames _are_ the variable names of shell scripting.
| oalae5niMiel7qu wrote:
| Wrong. Shell scripts have actual variables. They start
| with $. Filenames are one possible _value_ you might
| store in these variables.
| thanatropism wrote:
| Wat
| anoncake wrote:
| GREATI~1.DEA
| hapless wrote:
| It's 2020. Friends don't let friends write shell scripts.
| BossingAround wrote:
| When I first came into the world of SWE, I thought "why would
| anyone use Bash nowadays when we have Python?"
|
| A colleague answered me: "If we left, there's around 500 people
| in this building who could support my Bash script, and around
| 50 Python people."
|
| It kind of stuck with me. At this point, I feel like Bash is
| one of the common tongues between all tech roles (that deal
| with Linux, that is).
|
| Python, while nice, is a lot more niche, since you really have
| to be into development to know Python, while every sysadmin
| worth their salt can debug a Bash script. And, of course, every
| SWE, TSE, DOE, .... worth their salt know Bash scripts as well.
|
| If you want to get a job (in the Linux land), bash scripting is
| typically an "of course".
| dnpp123 wrote:
| This highly depends on the building you are in, I guess.
| baby wrote:
| woot, this is definitely wrong in my experience. Most people
| can read/write python to some degree, and even if you don't
| know python you can quickly ramp up to understand a config
| file or a simple program.
|
| On the other hand most people don't write bash scripts and
| usually have to deal with them when encountering legacy
| systems or languages.
| etaioinshrdlu wrote:
| me: What if there was a programming language as inconsistent
| and terrible as English, maybe it would be uniquely easy for
| humans to understand?
|
| bfox (wrote bash): Nah, Bash disproves that.
| zbentley wrote:
| larry wall (wrote perl): hold my beer
| krackers wrote:
| Applescript takes the cake there. Instead of standard
| indexing/dereferencing rules in the object hierarchy you
| instead have this nested mess of "tell X .... end tell"
| arpa wrote:
| I swear this is my last shell script.
| humblebee wrote:
| What do you recommend instead?
| HorstG wrote:
| There are lots of those footguns in shellscript. One should
| always try to avoid any shell and rather use python, tcl, perl or
| powershell. Any criticism one might have about insecure and
| broken by design languages apply doubly to shell.
|
| A short list of possible problems (of course depending on the
| shell in question):
|
| spaces in filenames
|
| newlines in filenames
|
| nonprintables in filenames
|
| empty variables and their expansion ([ x$foo = "xsomething" ])
|
| errors in pipes
|
| environment madness
|
| /bin/bash ?= /bin/sh
|
| Arrays or the lack of it
|
| Space separates lists as arrays
|
| #!bash vs. #!/bin/bash vs. #!/usr/bin/env bash vs.
| #!/usr/sfw/bin/bash vs. ...
|
| Unwritable and unreadable control structures (if [], case,
| &&,...)
|
| Information leaks via ps
|
| and many others...
|
| Never use shell except to search for and invoke a sensible
| language. And anything is more sensible, including C, Perl,
| brainfuck and Basic.
| TheDong wrote:
| I believe your diatribe is misplaced.
|
| There are quite a few pitfalls in shell scripting. You can
| considerably reduce them by limiting yourself to only being
| compatible with modern versions of bash and settings things
| like pipefail, nounset, etc etc.
|
| I do agree that in general a good programming language will be
| a better option.
|
| > anything is more sensible, including C, Perl, brainfuck and
| Basic
|
| I do disagree with that however. A 5 line bash script may be
| 500 lines of C, will take a hundred times longer to write, and
| may contain memory safety issues (which the bash script at
| least wouldn't).
|
| I know brainfuck is hyperbolic so I won't argue against that.
| Something with no filesystem or process forking abilities
| obviously can't be used for any real task.
|
| I think perl and basic have just as bad syntax as bash though,
| if not worse. Basic's penchant for "GOTO" is awful, perl's
| syntax as a whole is just as peculiar as bash's in many places.
|
| I guess my overall point is that bash is usually not a good
| option compared to modern languages, but it's a darn sight
| better than you give it credit for. I think it still has its
| place for 5 or 10 liners that are easy to express and read in
| bash and don't need any abstractions beyond what coreutils
| provide.
| HorstG wrote:
| I agree that 5 to 10 lines might be a sensible upper limit
| where a shell can safely be used.
|
| Basic does have Goto, but modern dialects do have all the
| usual control structures. Perl has weird syntax, but far less
| dangerous footguns: e.g. there are proper arrays, as opposed
| to many shells. One can distinguish between an empty and an
| undefined string. One can declare variables and there is the
| notion of data types. There are even things like taint mode.
| In shell, you can't even properly iterate over a directory
| without nasty surprises.
|
| Same in C. Yes, there are memory safety problems, but those
| are outnumbered by far by shellscripts exploitable via some
| expansion or variable injection. Its just that thankfully
| nobody uses shellskripts as network services, so you don't
| see as many reports about that.
|
| And yes, brainfuck was there as hyperbole. But I truly
| believe that there are very few things worse than shell for
| programming.
| diegocg wrote:
| Many of these crazy corner cases are the reason why I switched
| to fish. Eg, in fish all variables are arrays, so arguments are
| passed in the $argv array, and number of elements in the array
| is "count $argv" (in bash the number of arguments is passed in
| $#, but the hieroglyphic required to count the number of
| elements in an array is ${#var[@]} )
|
| $PATH is just an array where each path is an element, so
| removing or adding a path equals to adding/removing an element
| from the array which is trivial.
|
| Iterating over files names with spaces works exactly as
| expected and no IFS tweaking is needed (fun thing, in bash,
| doing for i in * .foobar; do echo $i; done; in a directory with
| no files with that extension will return the string "
| __*.foobar "). No "[" craziness, nicer syntax, etc.
|
| Unfortunately fish still lacks other important features (eg. no
| set -e equivalent).
|
| There is an space for sane Unix shells but everybody has
| settled on bash and changing the status quo is difficult.
| AmericanChopper wrote:
| I like powershell, but it has its own arsenal of footguns for
| you to use. Things like non-terminating errors and the
| sometimes confounding behaviour of automatic variables comes to
| mind.
| ulrikrasmussen wrote:
| Ick. I have written my fair share of bash, and stuff like this is
| very common. Most things in bash are just inherently non-
| compositional and/or is riddled with weird corner cases that you
| just have to know about in order to not shoot yourself in the
| foot. This document [0] made the rounds on HN a while back, and
| it has, together with the associated tool, been something that I
| have regularly consulted whenever I have had to do anything non-
| trivial with bash (anything that has to deal with arguments to
| commands is already non-trivial to get right).
|
| [0]
| https://github.com/anordal/shellharden/blob/master/how_to_do...
| [deleted]
| marcacohen wrote:
| Here's an easy way to see how this works. Run this script:
|
| echo dollar-star: for i in $ _; do echo $i; done echo dollar-at:
| for i in $@; do echo $i; done echo quoted-dollar-star: for i in
| "$_"; do echo $i; done echo quoted-dollar-at: for i in "$@"; do
| echo $i; done
|
| ./x.sh a "b c" d dollar-star: a b c d dollar-at: a b c d quoted-
| dollar-star: a b c d quoted-dollar-at: a b c d
| _kst_ wrote:
| https://news.ycombinator.com/formatdoc
|
| Blank lines separate paragraphs. Text surrounded by asterisks
| is italicized, if the character after the first asterisk isn't
| whitespace.
|
| Text after a blank line that is indented by two or more spaces
| is reproduced verbatim. (This is intended for code.)
|
| Urls become links, except in the text field of a submission.
| yesenadam wrote:
| HN gobbled your stars.
| TheDong wrote:
| The article doesn't mention it, but very similar syntax is also
| used for arrays.
|
| For example: arr=(a b c) arr+=(d)
| ls "${arr[@]}" # ls "a" "b" "c" "d" ls "${arr[*]}" # ls
| "a b c d"
|
| This has quite nice symmetry with the fact that the 1st argument
| is "$1", and you replace the number with these symbols, and for
| arrays you access elements with "${arr[1]}", and again replace
| the number with the same symbols for the same behaviour.
|
| If you do a lot of bash scripting, arrays are invaluable.
| useragent86 wrote:
| Indeed, and actual Bash scripting (as opposed to plain POSIX
| sh) is much more pleasant. Used properly, arrays make it easy
| to build commands with arguments and finally run them, e.g.
| #!/bin/bash command_args=( --foo bar
| --baz "buzz buzz" ) [[ $frob_option ]] &&
| command_args+=(--frob frab) echo command_name
| "${command_args[@]}" "other" "arg" # Echoes:
| # command_name --foo bar --baz "buzz buzz" --frob frab other
| arg
| [deleted]
| j1elo wrote:
| Thing is, if the script is basically the glue between
| incantations of multiple other commands (which is basically the
| intended use case of shell scripting), then replacing that with
| Python[1] is just adding lots of boilerplate code for no real
| improvements in functionality. I still agree with a strict limit
| on the acceptable complexity, though.
|
| Most if not all my shell scripts are just piping executions of
| external commands. I find all the code needed to properly run a
| process and process its output is much easier with the UNIX
| toolbox and a couple of pipe commands, than having to handle all
| those input/output buffers, command execution modes, etc in any
| other shell script language.
|
| OTOH Plumbum [2] has been mentioned here, and it seems fantastic
| for that use case. But I think the issue is obvious, in that it
| took a conversation in HN to raise awareness of this tool: it is
| not officially promoted, or recommended even, as the solution for
| replacing shell scripting, so it is kind of obscure (unless you
| are actively into the language or somehow by chance end up
| getting to know about it, that is)
|
| There is also the thing about choosing Python to replace Bash
| scripts would force having to install Python in all of the
| project's Docker images, while a short POSIX script works as-is.
|
| [1]: Saying Python because that's the most common suggestion for
| replacing Bash.
|
| [2]: https://plumbum.readthedocs.io/en/latest/
| lalaland1125 wrote:
| I highly recommend that people learn how to use
| https://docs.python.org/3/library/subprocess.html as a
| replacement for Bash. It's a little more work upfront, but it's
| vastly more maintainable.
| ben509 wrote:
| I disagree. Here's a warning from subprocess[1]:
|
| > Use communicate() rather than .stdin.write, .stdout.read or
| .stderr.read to avoid deadlocks due to any of the other OS pipe
| buffers filling up and blocking the child process.
|
| The trouble with `communicate()` is it only handles very simple
| cases, basically, your output has to fit in memory. Same
| problem exists in asyncio.[2]
|
| Yes, you can usually work around this. That doesn't mean it's a
| good replacement; whereas complex pipelines are so trivial in
| bash that any user-defined function can be used in a pipeline,
| subprocess generally forces you to do the dumb thing and create
| a mess of temporary files.
|
| And that's not even considering you're writing 10 times as much
| code than you would to accomplish the same task.
|
| [1]:
| https://docs.python.org/3/library/subprocess.html#subprocess...
|
| [2]: https://docs.python.org/3/library/asyncio-
| subprocess.html#as...
| roryrjb wrote:
| Shell scripting is absolutely still relevant. The rule of thumb
| should not be about length but about complexity, specifically if
| you absolutely need something like a real array or a hash then
| move onto a different language. Use shellcheck and avoid bash for
| scripting. I use bash or pdksh interactively but stick to POSIX
| shell for scripting. I am finding myself writing POSIX shell all
| the time and having great success with it.
| pletnes wrote:
| Why not bash? It's in most places, even if POSIX is even more
| general. And it does add some nice scripting features.
|
| I use zsh for interactive and bash for scripts for the same
| reasons as you, though.
| roryrjb wrote:
| Yeah don't get me wrong the bashisms are useful, but I'm
| hopping between OpenBSD, FreeBSD and Linux (and perhaps
| sharing scripts with my macOS-using colleagues) and although
| bash is available on all those platforms and more, POSIX
| shell will work out of the box without any further
| configuration.
| fiddlerwoaroof wrote:
| I write my scripts in zsh, because its extensions are most
| useful in scripts, and it's not too hard just to make it
| available everywhere you need to work (e.g. you can compile
| it to be installed in ~/zsh and just copy the installation to
| your home directory on any machine you need to use.
|
| Things like array-linked variables ($path is an array version
| of $PATH and modifications to one propagate to the other),
| associative arrays (dictionaries in python) and a handful of
| other really nice tools (e.g. saner white space handling)
| make going back to bash for scripts unpleasant.
| smichel17 wrote:
| Because when you find yourself needing bash-specific
| features, alarm bells should be going off. POSIX sh is good
| at reminding you to KISS and delegate to other tools.
| gameswithgo wrote:
| it is, but should it be?
| clort wrote:
| This is not Bash specific, this is basic POSIX shell:
|
| https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V...
|
| Bash also incorporates the POSIX shell, but has extensions. This
| is fine if you are using it, but if you are writing a script
| which may need to run on another system, its better to keep it to
| POSIX.
|
| (edit: URL - thanks userbinator)
| [deleted]
| userbinator wrote:
| You probably meant to link to the shell section of POSIX?
|
| https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V...
| gpvos wrote:
| In the far past, it used to be necessary to use ${1+"$@"} because
| of some shells that didn't handle "$@" properly when it was
| empty.
| _kst_ wrote:
| I've run into that. There was a problem with the OSF/1 /bin/sh
| that caused "$@" to expand to a single empty argument if there
| are no arguments, rather than to an empty list as it should.
|
| I just now removed a workaround for that problem from one of my
| scripts, 17 years after I added it.
|
| https://en.wikipedia.org/wiki/OSF/1
| floatingatoll wrote:
| The only time you should use $* is inside a debug message like
| "unable to open $* ($!)". If you're passing around arguments,
| always use "$@".
|
| If you know enough bash to disagree, you know enough bash to use
| the third case safely :)
| eikenberry wrote:
| I was going to say the same thing (so I will). The rule is just
| always use "$@" and then you have only one case to reason about
| and it is what you want 99.99% of the time anyways.
| dnautics wrote:
| I think it's also reasonable to use in ssh and I believe su,
| which are commands that expect a single string as their inner
| command parameter.
| floatingatoll wrote:
| I can count on ten fingers the number of times in twenty
| years I've worked with another bash coder who did the su and
| ssh cases correctly without triggering escaping bugs. It's
| not any insult on them, but it's almost always done
| incorrectly and happens to work due to the absence of
| whitespace and backslashes, leading to eventual bugs (that
| Shellcheck won't always catch). Given: #
| ARGV=( "one two", "three four" )
|
| It's probably safe to recommend "$@" for use with su _only
| when_ you use -c correctly, as you're locally specifying the
| args without any further IFS interference. But $* isn't
| usable: # CORRECT su root -c 'rm
| "$@"' -- "$@" rm "one two" "three four"
| # incorrect su root -c "rm \"$@\"" rm one two
| three four # wrong arguments # incorrect
| su root -c "rm" "$@" rm # -c
| doesn't use arguments # incorrect su
| root -c 'rm "$@"' "$@" rm "three four" #
| loses the first argument (?!) # incorrect:
| su root "rm" "$*" rm one two three four # wrong
| arguments # incorrect: su root "rm $*"
| "rm one two three four" # command not found
|
| It's probably safe to recommend "$@" for use with ssh _only
| when_ using printf %q to ensure that you escape your
| arguments for their transit through ssh to the remote host,
| as otherwise the arguments get corrupted by the extra layer
| of shell processing. $* isn 't usable here either:
| # CORRECT ssh remote -- 'rm '"$(printf '%q ' "$@")"
| rm "one two" "three four" # incorrect
| ssh remote 'rm '$(printf '%q ' "$*") rm "one two
| three four" # wrong arguments # incorrect
| ssh remote rm "$@" rm one two three four # wrong
| arguments # incorrect ssh remote "rm
| \"$@\"" rm "one two three four" # wrong arguments
| # incorrect ssh remote 'rm "$@"' "$@" rm one
| two three four # wrong arguments # incorrect:
| ssh remote "rm" "$*" rm one two three four # wrong
| arguments # incorrect: ssh remote "rm"
| "$*" rm one two three four # wrong arguments
|
| EDIT: Shellcheck misses 3 of the 4 broken su cases, but
| catches all of the broken ssh cases. (And produced a warning
| I disagreed with in one of the complete examples, but in the
| spirit of things, added double quotes to silence it.)
| chubot wrote:
| Oil [1] supports all of this old syntax to run existing shell
| scripts, but has new syntax which is more convenient.
|
| - You can write @ARGV instead of "$@".
|
| - You can write @myarray instead of "${myarray[@]}"
|
| (Related: _Thirteen Incorrect Ways and Two Awkward Ways to Use
| Arrays_ https://www.oilshell.org/blog/2016/11/06.html )
|
| Example: oil$ var myarray = @('has spaces' foo)
| oil$ var s = $'has\ttabs' # function to print an
| array element on each line oil$ lines() { for x in @ARGV;
| do echo $x; done } # pass 3 args -- 2 from myarray
| and 1 from s oil$ lines @myarray $s has spaces
| foo has tabs
|
| [1] https://www.oilshell.org/
| dzidol wrote:
| https://xkcd.com/927/
| chubot wrote:
| Unlike every other alternative shell, Oil runs existing bash
| scripts to avoid this problem.
___________________________________________________________________
(page generated 2020-01-12 23:00 UTC)