[HN Gopher] Advanced Bash-Scripting Guide (2014)
___________________________________________________________________
Advanced Bash-Scripting Guide (2014)
Author : jolux
Score : 103 points
Date : 2022-08-19 16:28 UTC (6 hours ago)
(HTM) web link (tldp.org)
(TXT) w3m dump (tldp.org)
| [deleted]
| [deleted]
| uhtred wrote:
| Cue all the "when your program gets longer than one line you
| should consider using python" comments.
| gpderetta wrote:
| To be fair you can do a _lot_ in a bash one liner.
| dmtroyer wrote:
| doesn't mean you should, for the sake of the next person.
| vlunkr wrote:
| My rule is when you start needing functions or arrays you
| should consider using something else. Bash is really neat and
| easy to get started with, but maintaining a large codebase is
| terrible.
| metadat wrote:
| 100% sane advice on the "should" front, but I don't always
| follow the shoulds..
|
| Bash is just too damn fast to whip things up in (even at the
| risk of endless footguns). Build up a pipeline of commands,
| translate into script, done. It's powerful and can scale if
| common conventions and good practices are used.
|
| For easier debugging I always add a simple flag along the lines
| of: if $1 = -v then set -x; shift
|
| Obviously shellscripts are a terrible or highly suspect idea
| when production-grade transactional integrity is critical, but
| in many cases the swiftness of shell development outweighs the
| cons when the goal is to Get Things Done.
|
| The shell is a pretty sweet interface.
| cjvirtucio wrote:
| it's so much easier to glue tools together with it. I'd end
| up writing more code if I had used python's subprocess
| package or ruby's IO package. as long as I don't need a
| complex data structure of some sort.
| bryanlarsen wrote:
| If your primary consideration is that it's generally available
| on a random developer's machine, what are the options? IOW out
| of the box on OSX and the major Linux distributions, and
| readily available on Windows.
|
| Python is pretty safe, but your code would have to work with
| both 2 & 3 and not pull in any external dependencies. And
| recommended practices have changed so drastically on how to
| install Python over the years that a random dev could have
| theirs in a fairly weird state.
|
| Perl would work, but hacked-together Perl has the similar
| drawbacks to hacked-together Bash, especially when it's me
| doing the coding.
|
| Ruby has problems similar to Python.
|
| Make & awk have problems similar to Perl & Bash.
|
| Therefore my latest rust package contains bash scripts.
| vaporup wrote:
| TCL
| js2 wrote:
| Last week I wrote a bash script that ended up being about 250
| lines. It was mostly running other processes. For technical
| reasons, it needed to re-invoke itself a few times via sudo.
|
| When I was done I looked over it and thought: this is a nice
| piece of code right now. It's well documented. It's self
| contained. It passes my tests. I could maintain this code.
|
| But it made use of quite a few bash'isms: arrays and so forth.
| I wasn't confident in the ability of my colleagues to keep it
| in good state.
|
| So I rewrote it in Python. The Python version ended up being a
| little longer, but that was mostly due to formatting
| difference. The Python version in addition I could reformat
| with black and isort, I added type annotations so I could test
| it with mypy.
|
| (I had formatting the shell version with shfmt and checked it
| with shellcheck, but still.)
|
| The Python version is much easier to read, and I think will be
| more maintainable.
|
| I consider myself competent in both languages so it's a
| courtesy to my colleagues if nothing else to ship the Python
| version.
| aaaaaaaaaaab wrote:
| _Yawn..._
|
| Python sucks for building processing pipelines.
|
| How many lines of Python code would you need to implement this?
| paste -d \\n <(foo ...) <(bar ...) | while read -r f && read -r
| b; do baz "$f" "$b" done
|
| Mind you, `foo` and `bar` run in parallel.
| slt2021 wrote:
| replace bash one-liner with python? No, thanks
|
| replace 200+ bash script with multiple subroutines in python?
| yes
| dmtroyer wrote:
| as someone who is maintaining mountains of legacy bash, for
| all that is holy, yes. or Go. or Rust. anything.
| idispatch wrote:
| Mind you, before this goes to prod please add error handling
| and logging. What happens when bar fails? What about baz?
| Now, please rewrite this so your team mates can maintain and
| extend. Yawn, I'll wait for the PR.
| aaaaaaaaaaab wrote:
| Gotcha, thanks for the feedback! We should also totally A/B
| test this! I'll make sure that this ties in with the
| quarterly OKRs, too! I hope I'll get promoted to E6 for
| that sweet 10% raise next cycle :fingers_crossed:
| alganet wrote:
| I am on the portable shell team. It is hard to write for all
| shells, but totally worth it. If we all did it, reusing shell
| code would be easier.
|
| We never left the hell caused by the war between shells to
| provide interactive features. They were all feature creeped for
| terminal usage and the scripting API was left to rot.
|
| I get the people that say we should use python or something else
| instead. It won't work though if that language is not capable of
| replacing the shell _as a build dependency_. As long as we need
| the shell to build stuff, we're stuck to it.
|
| Dockerfiles, GitHub Actions YAMLs, all these tools accept shell
| input. This is shiny new software that could have picked any
| language for their proprietary DSLs, but the shell was chosen.
|
| What the shell needs is what happened to JavaScript around the
| 2000s. Different interpreters started to align, standards were
| written, reusable libraries started to pop up everywhere, and now
| it runs in our brains. Before all of that, JavaScript was merely
| a cute toy to animate pages.
| js2 wrote:
| The one thing I really miss in portable shell code is arrays.
|
| There's a place reserved in hell for whoever thought it was a
| good idea to design a build system around shell code in yaml
| files.
|
| I'm in the process of moving thousands of lines of such code
| into standalone scripts that then get called from the yaml
| files. Keep the yaml as minimal as possible. Then I can
| shellcheck the shell code or easily port bits to other
| languages as needed.
| [deleted]
| alganet wrote:
| You can reimplement arrays using pure shell!
|
| I saw it first here: https://github.com/shellfire-
| dev/core/blob/master/variable/a...
|
| For many cases, this is even better than bash native arrays.
| You can pass these as arguments and preserve the structure,
| while the bash ones will autoexpand if you try that.
|
| I ended up cooking my own array implementation later, but it
| is a bit more unorthodox (I create extra vars with numeric
| pointers to handle the values). Let me know if you want to
| see it, it's not complete but it works.
|
| I'm not very fond of YAML either, I feel your pain :(
| js2 wrote:
| I knew I should have qualified my comment about arrays
| with: "without using a hack such as storing the values in a
| string with a custom delimiter." Those aren't really
| arrays.
|
| What happens if the delimiter appears among the values you
| want to store? This is a pretty big disclaimer:
|
| > Do not use arrays with file paths if there's any chance
| they could contain \r.
| alganet wrote:
| That's why I did my own implementation using pointers.
|
| var lorem = text_create "Hello"
|
| var foo = array_create
|
| array_push $foo $lorem
|
| `var`, `array_push` and `array_create` are shell
| functions working together. They create pointers and
| return just integers. You don't even have to quote the
| variable.
|
| The code above should create the following variables:
|
| _123="Hello" _124=1 _124_0=_123 foo=_124 lorem=_123
|
| This way, I can even implement more complex data
| structures.
|
| The numbers start from zero and the program will stop if
| the global variable counter overflows. My current
| implementation also has a primitive GC (ARC-based), which
| could be used to avoid that (but I didn't, yet).
|
| Despite all of this runtime processing and jerry rigging,
| I have some evidence that it could be faster than typical
| shell scripts. One single subshell or path lookup is
| often more expensive than these fake pointer arrays.
| These strucutres have zero subshells and run on PATH='',
| just with builtins. The subshell only becomes faster if
| the volume of data is larger than a few kilobytes (when
| the cost of forking and exec'in is worth the payoff).
|
| I know this is far from ideal. I should try to put these
| ideas in a new interpreter, not hack them on existing
| ones (JS calls them polyfills, maybe I should do that).
| If I did another interpreter though, it would end up like
| the fish shell (nice, fast, doesn't compile the kernel
| and no one uses it).
|
| Maybe I should probably stick to bash, ksh and zsh that
| offers the most complete set of features. Sounds like
| "best viewed in internet explorer" though.
| dotancohen wrote:
| This is a good guide for Bash, but I wouldn't use the code as-is
| in production. For example, the following code is recommended for
| cleaning up a log file and leaving only the last 50 lines:
| tail -n $lines messages > mesg.temp mv mesg.temp messages
|
| That code has a clear race condition and will lose lines on a
| busy server.
| metadat wrote:
| Yes, use `mktemp', just as you would in Python, Go, Rust, C, or
| Java.
| metadat wrote:
| Actually, to protect against the clobber of the original
| file, you could use a flocker lock on the process to
| serialize the operations. Otherwise it will be unsafe.
| wswope wrote:
| How would you fix that race condition?
| formerly_proven wrote:
| Have the writing process do it.
|
| Which kind of reminds me of that product running on a pseudo-
| distributed NoSQL database built out of text files with
| "record per line" structure and sort of a compacting garbage
| collector for these files to remove dead records. And there's
| just absolutely nothing that can go wrong with that.
| metadat wrote:
| Lots of good refreshers, the cat + heredoc combo is powerful but
| lots of cryptically subtle variations in behavior between:
|
| <<EOF (expands variables etc)
|
| <<'EOF' (doesn't expand variables etc)
|
| <<-EOF (trims leading tabs)
|
| Am I missing any more of the tricks?
| 0xbadcafebee wrote:
| https://linux.die.net/man/1/bash "Here Documents" and "Here
| Strings" sections.
| dotopotoro wrote:
| Trimming version does not trim whitespace indentation. I guess
| point goes to "tabs" camp.
| metadat wrote:
| Correct, though you could pipe to sed and trim leading spaces
| with relative ease:
|
| sed 's/^ *//' <<EOF
|
| Which I'd prefer anyways because it's less cryptic. I try to
| stay away from the obscure, rarely used notations because
| I'll always have to look it up in StackOverflow anytime I
| come back to it. These can be confusingly sharp edges.
| alganet wrote:
| You can do it in compound commands as well:
|
| while read -r LINE
|
| do printf %s "$LINE"
|
| done <<FOO
|
| foo
|
| bar
|
| baz
|
| FOO
|
| This is faster than using cat (one less path lookup, one less
| fork, one less exec, read is builtin), works on all shells.
|
| There is a <<-'EOF' as well (trims left tabs and does not
| expand vars).
|
| Another cool trick that only bash and zsh have is `printf -v
| NAME`, which prints to a variable instead of stdout. You can
| use it to avoid subshells (one less fork for each subshell
| avoided).
| stjohnswarts wrote:
| Doesn't this get posted like every other month?
| 0xbadcafebee wrote:
| The best guide to using Bash is to read the manual
| (https://linux.die.net/man/1/bash).
|
| Nobody who has ever used Bash has read the whole manual. It is
| actually good and useful, please read it. Yes, it is very big.
| But you will be using this shell for the rest of your life (even
| if you use Zsh, you will end up using other people's Bash
| scripts). Consider it an investment in your future.
|
| ...and now that I say that - try to restrict shell scripts to
| POSIX semantics (https://betterprogramming.pub/24-bashism-to-
| avoid-for-posix-...). POSIX is very simple and most shells can
| run it. If you need a Bashism, try Bash v3 first so it works on a
| Mac without Homebrew. Simple is better/portabler than complex. If
| you need something Bash doesn't provide, use any language that
| the majority of your team knows well.
|
| If you want to test a specific version of Bash, there's
| containers @ https://hub.docker.com/_/bash (Macs use v3.2.57).
| Test POSIX scripts with _`bash --posix`_. Definitely use
| Shellcheck.
| jpitz wrote:
| Not everyone has portability as a first-class requirement for
| their scripts. For 95% of what I write, bash 4 is entirely
| appropriate.
| rascul wrote:
| > The best guide to using Bash is to read the manual
| (https://linux.die.net/man/1/bash).
|
| Here's the link to the latest official manual on GNU's site:
|
| https://www.gnu.org/software/bash/manual/html_node/index.htm...
| joenoob wrote:
| > try to restrict shell scripts to POSIX semantics
|
| I have to admit, while i admire the idea of and understand the
| reasoning behind striving for POSIX-compatibility, i never
| couldn't fully get past viewing this as some kind of
| hypothetical exercise. The chance that any script i (or
| presumably most people who get this advice on pages like SO)
| will ever write will have to run on a 25 years old Tru64
| machine or even a modern Mac, BSD, QNX, ... is relatively
| small. Truth is (in my case), most of the scripts will never
| even run on anything but the system they were written on. As i
| said, i admire the idealism though.
| ithrow wrote:
| A quick ToC for the bash manual: 'man bash | grep ^[A-Z]'
| chasil wrote:
| Careful readers of the manual will find this commentary from
| the bash maintainers: $ man bash | sed -n
| '/BUGS/,/^$/p' BUGS It's too big and too
| slow.
|
| It is at this point that such a reader will discover the
| reasons behind Debian dash.
| stjohnswarts wrote:
| I think that fish is supplanting zsh as the "developer's"
| shell. Bash is like C though, gonna be here for another century
| at least.
| asojfdowgh wrote:
| > Test POSIX scripts with `bash --posix`. Definitely use
| Shellcheck.
|
| Except that doesn't disable all bashisms so you end up leaving
| in bashisms as a result, which is what caused all the issues
| for debian trying to migrate to dash
| dijit wrote:
| >Nobody who has ever used Bash has read the whole manual.
|
| Bold claim and patently untrue because _I have_ read the whole
| manual.
|
| However:
|
| 1) Things you don't use often leak out of your brain at a rate
| that is absurd.
|
| 2) Different versions have different features and there's wide
| variations between versions on different platforms, you can't
| actually depend on a lot of bash stuff existing, so instead
| it's usually better to target POSIX sh
| jagged-chisel wrote:
| I, too, experience Swiss-cheese-brain.
|
| I once thought I'd solve this by taking notes, organizing a
| journal, and writing ... Well, it was a manual. That had the
| same problem as other manuals - too long, not gonna read.
|
| Full circle.
| nocman wrote:
| Your note taking, organizing and writing will have helped
| you retain a lot more info for a longer period of time. It
| wasn't wasted effort. Still, there is a such thing as "too
| much". Higher degrees of success are achieved by regularly
| evaluating where you are in the "too little" to "too much"
| spectrum.
| dejj wrote:
| Don't forget to shellcheck1 your script before checking it in.
| Also available in VSCode, IntelliJ, and others.
|
| [1] https://www.shellcheck.net/
| nixcraft wrote:
| I also like, BASH Frequently Asked Questions
| https://mywiki.wooledge.org/BashFAQ and linter
| https://github.com/koalaman/shellcheck
| michaelsbradley wrote:
| Bash Hackers Wiki is also a nice resource: https://wiki.bash-
| hackers.org/
| ufo wrote:
| Sometimes I pause to think how we're held back by needing to make
| our shell scripts backwards compatible with Bash or POSIX,
| because that's what is installed by default. Shell scripting
| would be nicer if the shell could be less crufty. For example,
| get rid of all the footguns about expanding unquoted variables...
| arinlen wrote:
| > _Sometimes I pause to think how we 're held back by needing
| to make our shell scripts backwards compatible with Bash or
| POSIX, because that's what is installed by default._
|
| I think you're missing the forest for the trees.
|
| Bash is installed by default because Bash is specified in an
| international standard dubbed the Portable Operating System
| Interface (POSIX). Your bash scripts are not backwards
| compatible. Your bash scripts are compatible, and portable, and
| standardized, because that's their point, and the whole point
| of POSIX.
|
| No one is held back by bash or POSIX. You are free to use
| anything that suits your fancy. Feel free to whip out scripts
| in Python or Perl or Ruby. Some people do, and last time I
| checked some operating systems like macOS shipped their
| interpreters by default. Odds are you are free to easily
| install those interpreters in those who don't.
| redleader55 wrote:
| I'm getting a "404 No Found" on my mobile. Is this supposed to be
| META?
|
| Edit: it's a genuine question, I have no idea what everyone else
| is seeing.
| dejj wrote:
| Backup here:
| http://web.archive.org/web/20220819180012/https://tldp.org/L...
| Cwizard wrote:
| I'm getting a 404
___________________________________________________________________
(page generated 2022-08-19 23:00 UTC)