[HN Gopher] Scripting with Go
___________________________________________________________________
Scripting with Go
Author : synergy20
Score : 114 points
Date : 2022-03-11 16:26 UTC (6 hours ago)
(HTM) web link (bitfieldconsulting.com)
(TXT) w3m dump (bitfieldconsulting.com)
| mftb wrote:
| It took me a while to find the link to the library "script" and
| it's repo - https://github.com/bitfield/script
| synergy20 wrote:
| https://golangexample.com/making-it-easy-to-write-shell-like...
| with quick examples, everything is pipe based just like
| bash|fish|zsh|csh
| 0xbadcafebee wrote:
| Go script: p := Exec("man bogus")
| p.SetError(nil) output, err := p.String()
| fmt.Println(output)
|
| Shell script: man bogus
|
| ......I'm gonna stick with shell scripts.
| nunez wrote:
| There was a recent discussion in /r/golang about chaining methods
| like this. The OP had a hard time writing tests for their
| implementation since each part of the chain returned interfaces
| that were relied on by following links within. The top answer in
| the thread was that Golang wasn't really meant to make "Promise-"
| type workflows possible due to it leaning heavily on code being
| intentional and explicit.
|
| I agree with them.
|
| One of the problems I have with things like LINQ or Promises is
| that they are immensely powerful but can be really difficult to
| debug when things go wrong because of implicit errors within the
| pipeline bubble up the stack without providing location context.
| While the standard boilerplate around errors in Go is annoying, I
| actually prefer it for two reasons:
|
| 1. It provides exactly where things went wrong while looking at
| an exception trace, and 2. From a readability perspective, it is
| much easier to understand what the author of the code meant to do
| and, more importantly, what the follow-on error s mean.
| Sometimes, errors aren't actually errors.
|
| Either way, OCaml/Haskell/F#'s approach with match expressions
| and the `Result` type is the best way to deal with this, IMO. You
| get the best of both worlds: explicit declaration with a very
| expressive (triangular-like) "shape" to the code.
|
| So something like this: var data string =
| script.File("test.txt").Match("Error").CountLines()
|
| Can be expressed like this: var foundLines int32
| = 0 var numLines int32 = match file.Open("test.txt") with
| | Error e -> return e | Ok f -> match f.Readlines() with
| | Error e -> return e | Ok lines -> match lines with
| | /Error.*/ -> foundLines++ | _ -> // do nothign
| return foundLines
|
| instead of: foundLines := 0 lines, err :=
| ioutil.ReadFile("test.txt") if err != nil { return
| 0, err } for _, l := range lines { re, err :=
| regexp.MustCompile(`Error.*`) if err != nil {
| return 0, err } matches :=
| re.FindStringSubmatch(`Error.*`) if len(matches) > 0 {
| foundLines++ } } return foundLines, nil
|
| This way, you can see exactly where the error occurred but can
| still see that `numLines` is generated through a pipeline.
|
| As far as the library itself, I personally wouldn't use something
| like this, though I see the appeal. I turn to statically-typed
| languages like Golang when a Bash script becomes too kludgey to
| stand on its own (usually when I need to begin relying on mild
| data structures) and when I care about type safety and
| portability more than what Ruby or Python can give me. When I'm
| writing a Go program, I want something that's testable that I
| know can work anywhere. With Ruby or Python, unless I'm
| distributing the script as a Docker image, I have to worry about
| versions, environments, etc., none of which are pleasant.
|
| However, writing the error boilerplate is annoying, and I can see
| developers who want to write something quick but spend 99% of
| their time in Go using this to get something done with a tool
| they know well. I've seen similar things in the Java world; hell,
| that's the reason why Groovy and Kotlin exist!
|
| TL;DR: The "wrong thing" in the Bitfield article isn't
| necessarily wrong; their library is useful, but niche; all hail
| pattern-matching and the Result type.
| hankchinaski wrote:
| i am not sure i would personally use this library/approach when
| scripting, a few reasons:
|
| Using a third-party libraries as core part of your infrastructure
| (ci/cd scripts, provisioning, automation) implies a greater risk
| to potential security issues, "framework tax" ie. having to
| comply, learn, document, debug its custom APIs, having to deal
| with potential limitations, issues that either needs to be fixed
| upstream or result in the library being forked and therefore
| maintained in house. I would rather either put together a set of
| bash commands or - if the problem entails a more comprehensive
| endeavour with greater complexity - put together a in-house
| tool/library where i can make the right compromises from day one
| kkfx wrote:
| Honestly I'm not convinced. I understand the need of a so-called
| "real" programming language ready available, that's what all
| classic system have had, from SmallTalk workstation to LispM, but
| for them there is not just the language, there is the complete
| ecosystem build with the same language, so an user made script
| have no difference than a system part of the user desktop.
|
| Unix decide to "simply" creating a system with a system language
| and an user system with an user language, the shell. I do not
| like much unix approach after having used Emacs a bit, but I do
| understand it. On contrary I always fails to "craft scripts" in
| "real" programming languages no matter what. I've tried in the
| past to "go Perl", "go Python", "go TCL", yes I can write scripts
| with them, I have written some etc but if I need something quick
| I go for the shell, zsh to be more precise (tried others, all
| failed at a certain point, from bash to elvish passing through
| oil shell and few others) or being in Emacs (EXWM is my WM/DE)
| Emacs itself depending on the case.
|
| I read various similar article for a language or another but in
| the long run see no colleagues really choose a programming
| language behind shell itself...
| mkdirp wrote:
| I'll have to check this out properly later. 10 years ago go devs
| rejected the idea of introducing support hashbangs and now we're
| left with having to use gorun[0], meaning, people are unlikely to
| use go for script.
|
| I hope the go devs will reconsider. I'd love to be able to use go
| for scripting. But as it stands, it's a sad state of affairs
| because you have to rely on hacks.
|
| [0] https://github.com/erning/gorun/
| throwaway894345 wrote:
| I'm not sure what definition you have for "scripts", but I'm
| fine with running `go run main.go` or whatever. I don't need my
| Go files to be executable (half the time I just run bash
| scripts via `bash script.sh` anyway because that _always_
| works). The biggest impediment is the ceremony for
| subprocessing out, but I 'm sure that could be abstracted
| behind a library with a more pleasant veneer than os/exec, and
| even if not whatever you lose in subprocess ergonomics you make
| up for in static typing, not needing to Google/Stack Overflow
| everything, and general absence of weird quirks.
| moondev wrote:
| you can now directly "go run" any import path
| go run github.com/mikefarah/yq/v3@3.4.1
|
| It also works fine in shebang that expects input of script
| filepath at $1. For example "test.yaml"
| #!/usr/bin/env -S go run github.com/mikefarah/yq/v3@3.4.1 r
| --- some: yaml: here
|
| chmod +x test.yaml then ./test.yaml some.yaml
|
| returns test
| shimst3r wrote:
| While I see the benefit of this approach, I'm often baffled why
| people want to go either 100 % POSIX builtins or 100 % scripting
| language.
|
| The biggest benefit of the shell is its clearly defined input and
| output (and error) interfaces. Most programming languages can
| read from and write to stdin, stdout, and stderr.
|
| Why not use it and stick to KISS, replacing one cumbersome POSIX
| utility at a time, suites for the task? Then you don't need to
| chain methods using less idiomatic code. But then you wouldn't
| need these kind of libraries either.
| cogman10 wrote:
| I think it boils down to the friction of starting and stopping
| external processes.
|
| For example, you could in your scripting language use `find` to
| search for files in a folder and do something with them, but
| why do that when your language of choice almost certainly has
| globbing capabilities? You could grep a file for a line, but
| why do that when you can use your language's inbuilt regex
| system?
|
| At least from the scripting side, the reason I tend to push
| more towards using the language and less towards using external
| processes is because most scripting languages can do what those
| external processes do in one go.
|
| Perhaps it would make more sense if I were better at defining
| scope :)
| unfocussed_mike wrote:
| Cross-platform deployment is why I switched from script plus
| utilities to a go binary.
|
| I did manage to make a Windows batch file that replicated the
| functionality of a linux/mac bash script, but configuring it
| was no fun for customers on any platform, and then there were
| the utilities themselves to deploy.
|
| The replacement binary has a very small platform-dependent
| aspect, and I am not held back by the limits of batch files
| when trying to achieve feature parity.
|
| It might be doable to deploy a powershell script, but then
| there's installation work to do on the unix side instead.
| pphysch wrote:
| Let one-liners be one-liners, and bring in Go/etc when it
| ceases to be a one-liner. But not before.
| jjtheblunt wrote:
| I agree in principle, but mastering the syntactic quirk-fest of
| bash and other shells is really a bit weird, in that surprises
| arise at runtime.
|
| maybe that's the compelling use of scripting with a statically
| typed, thus compile-time at least partially low-hanging-fruit-
| error-checked, language?
| qbasic_forever wrote:
| We really need a 'Bash: The Good Parts' book like Doug
| Crockford did for Javascript. IMHO bash and the shell are in
| a state that Javascript was ~2005--an old language/tool full
| of complexities but that can be sharpened and honed down to
| something beautiful.
|
| IMHO writing procedural style code with lots of if, loops,
| etc. in the shell can quickly turn into an anti-pattern. Try
| to stick to simple functions that are chained together in
| pipelines. The only loop is typically one that processes
| arguments and that's it.
| b215826 wrote:
| > _We really need a 'Bash: The Good Parts' book like Doug
| Crockford did for Javascript._
|
| Bash is incredibly less complex than Javascript and there
| is such a resource: the "Bash guide" [1] and "Bash
| pitfalls" [2] are both excellent resources that teach you
| how to use Bash properly.
|
| [1] http://mywiki.wooledge.org/BashGuide
|
| [2] http://mywiki.wooledge.org/BashPitfalls
| hnlmorg wrote:
| I completely agree.
|
| This is the approach that I took with murex. It comes with a
| stack of builtins for json, jsonlines, yaml, toml, csv,
| sqlite3, and others. So you have all the power of data types
| and intelligent management of structured data but also works
| with regular POSIX CLI commands without any additional
| boilerplate code when swapping between murex code and coreutils
| (et al). So one could say I've spent a fair amount of time
| studying this point you've raised :)
| klabb3 wrote:
| https://github.com/lmorg/murex
|
| (Don't be afraid to self promote!)
|
| One obvious benefit over what's suggested in the article is
| that you can use it interactively first, with autocomplete
| and such goodies, and transition smoothly to a script later.
| freedomben wrote:
| GP has promoted murex a few times on HN recently, which
| doesn't bother me, but if I were them I'd be sensitive to
| not wanting to overdo it as well.
| cyberge99 wrote:
| Is murex open source? A quick search of murex shell presented
| me with beautiful seashells.
| zto wrote:
| A quick Google for murex cli -sea yields:
| https://murex.rocks and https://github.com/lmorg/murex,
| looks like a promising tool
| darrenf wrote:
| Not to mention the URL in @hnlmorg's HN profile!
| gkfasdfasdf wrote:
| Something which the shell script has which the go implementation
| lacks (unless I missed it) is concurrency. When you have a shell
| pipeline like 'cmd1 | cmd2 | cmd3' those cmds are actually
| started at the same time and run in parallel. This is really
| great for performance - you get multiple cores working on the
| problem with very little effort.
| skybrian wrote:
| This Go library looks like a good one, with the caveats that it
| hasn't reached 1.0 yet, and does have a couple of dependencies
| that I didn't trace further.
|
| At one time jQuery was popular and this seems like a similar
| thing, but for files?
|
| It's small enough that if you're worried about "other people's
| code" you could fork it and maintain it yourself.
| shadowofneptune wrote:
| This article focused entirely on replacing the Unix shell. Does
| this library also work for the Windows shell or PowerShell (iirc
| PowerShell is POSIX)? I could see the value of having a single
| script in a Go repository which works for all build systems.
| vessenes wrote:
| I like me some bash scripting. And I do a lot of it. And I write
| a good amount of Go.
|
| I read the intent of the original package as a bit different than
| how it's presented here; a pain point in using go for systems
| programming on Posix systems with GNU tooling is interacting with
| all the other really excellent tools there. There's an awful lot
| of cruft to go spawn a shell script in Go without this lib.
|
| So, I cannot imagine using this to replace a short-ish bash
| script, it's just too heavyweight to add go development into the
| workflow. But, I can easily imagine using this when I want to do
| some simple-to-medium-scale pipelined text processing from the
| middle of a go program.
| unfocussed_mike wrote:
| > But, I can easily imagine using this when I want to do some
| simple-to-medium-scale pipelined text processing from the
| middle of a go program.
|
| Yeah, this is where I see it.
|
| I have a cross-platform utility in customer use that replaces a
| bash script or a .bat file that did a simple job with curl and
| a database client but was tricky for customers to configure and
| deploy; I replaced it with an interactively-configurable go
| command.
|
| In the middle of it is the replacement pipe; this library would
| help here. Might use it in the rewrite.
| GordonS wrote:
| Thing is, bash is everywhere (or, at least a POSIX shell is) -
| for some use cases that matters, because you can't control what
| is available on the target machine. Even python isn't
| guaranteed - OK python is _fairly_ ubiquitous, but what
| version?
|
| Shell scripting is far from perfect, and sometimes it does take
| a while to figure out how to do something that would be easy in
| another language - but very often, she'll scripting is "good
| enough".
| qbasic_forever wrote:
| It's interesting that go text templates have shell-like pipelines
| but that never bled over into the rest of the language:
| https://pkg.go.dev/text/template#hdr-Pipelines
|
| It seems like a shame that this kind of power and expressiveness
| is reserved only for generating text.
| flakiness wrote:
| (Mostly trolling) You must be missing extension method that other
| modern languages have, let alone the pipe operator of Elixir, R,
| etc. ;-)
| jerf wrote:
| The places I've "shell scripted with Go" involved also using
| some other code I already had in Go. The referenced page
| doesn't show it, but being able to do a bit of quick shell
| scripting-type manipulation on a file, then feed it to the Go
| JSON decoder to get Go objects, call a few methods or
| something, then maybe manipulate it a bit more on the way out,
| is sort of thing that is the real killer feature, IMHO.
|
| The space of "shell-like libraries", not least of which is
| shell itself, is so rich it's hard to beat out everything else
| on its own merits. But having something as easy as shell that
| integrates back into your Go ecosystem is very convenient.
|
| And I'm sure similar things are true for the other libraries in
| other languages.
|
| So I would personally not present this as "here's something
| awesome Go can uniquely do", but, "if you already have Go, be
| aware of this tool/technique".
| flakiness wrote:
| Yeah, I agree with your point and can see the usefulness.
| Sorry for trolling, just couldn't resist :-/
| jniedrauer wrote:
| I may be in the minority here, but I personally prefer the "wrong
| answer" highlighted at the beginning of this article. Scripts are
| source code. They go in the repo, and they have the same
| standards applied to them as any other source code. I would much
| prefer that the code be explicit and rely on few if any third
| party libraries. I have go scripts that have been functioning in
| production for half a decade now without modification. They are
| as "self documenting" as any other go code, and I do not require
| esoteric knowledge about a third party library to re-familiarize
| myself with them.
| fhood wrote:
| I think you are missing the point. The overarching motivation
| here is "Normally one would do this in bash, but I want to use
| go instead. Is there any way for me to do that while retaining
| the aspects of bash that make it good for these tasks?" And the
| author provides a solution. Dunno why you would do that, but
| not really important.
| _adamb wrote:
| I agree. It would be much more readable if it simply had a few
| comments too (something that can't really exist in a 1-liner)
| and is infinitely flexible (ex: for each match, also make an
| API call).
| chubot wrote:
| FWIW a lot of other host languages are more DSL-like, and have
| similar libraries:
|
| https://github.com/oilshell/oil/wiki/Internal-DSLs-for-Shell
|
| It looks like this particular one relies on method chaining
| script.Args().Concat().Match("Error").First(10).Stdout()
| tedunangst wrote:
| Seems like this could pair well with something like yaegi.
| mbreese wrote:
| If you're willing to add a new binfmt to your system, here's
| another method for using golang to write "scripts" (executed with
| gorun). It's not the same as a shebang/hashbang, but it works.
|
| https://blog.cloudflare.com/using-go-as-a-scripting-language...
___________________________________________________________________
(page generated 2022-03-11 23:00 UTC)