[HN Gopher] Parsing JSON at the CLI: A Practical Introduction to...
___________________________________________________________________
Parsing JSON at the CLI: A Practical Introduction to jq and more
Author : sequoia
Score : 77 points
Date : 2020-12-21 19:22 UTC (3 hours ago)
(HTM) web link (sequoia.makes.software)
(TXT) w3m dump (sequoia.makes.software)
| loloquwowndueo wrote:
| Thanks!
|
| I found the most value in some of the additional tricks shown in
| the article (like the "pretty print JSON stored in the clipboard)
| over the introduction/tutorial aspect; there are hundreds of
| these "look, I figured out how to do a handful of interesting
| things in jq so let me share with you all" tutorials floating
| around and most of the basics are redundant among those by now.
|
| If jq's documentation had some basic tutorials it would probably
| be a better authoritative source for this information.
|
| Until then, as said, thanks for preparing and sharing this!
| blueline wrote:
| the jq manual has tons of examples
| https://stedolan.github.io/jq/manual/
| jolux wrote:
| I've been frequently derided for demoing it but I should write
| something about how to do similar tricks with PowerShell, as I
| have found it easier to learn than jq and an underrated tool in
| dealing with JSON.
|
| The first example: ConvertFrom-Json $USERX |
| ConvertTo-Json | Set-Clipboard
|
| Getting properties by name: ConvertFrom-Json
| $ORDER | select order*
| sequoia wrote:
| You definitely should! Is it alright if I copy this comment
| onto my blog underneath this post in the "comments" section?
|
| If you _do_ write such a post make sure to let me know so I can
| link it from mine! You could write a response "here's how I'd
| do all the stuff in that post in Powershell with no extra
| tools" that would be very cool :)
| jolux wrote:
| I'll try to remember!
| vips7L wrote:
| Most of PowerShell is underrated when it comes to pipelines in
| shell.
|
| I've completely rid myself of GNU core utils with just pwsh.
| the_cat_kittles wrote:
| why? are you forced to work on windows?
| djeiasbsbo wrote:
| Iirc powershell is cross platform.
| dopylitty wrote:
| PowerShell is supported on Linux and macOS as well [1]
|
| I completely agree with the parent that PowerShell blows
| the existing shell options out of the water. Having
| pipelines with objects rather than just text makes
| everything so much easier. Instead of spending an hour
| futzing with awk or regex or applicatioons like jq to parse
| values from command results you can just access what you
| want directly and get on with your work.
|
| 1. https://docs.microsoft.com/en-
| us/powershell/scripting/instal...
| Arnavion wrote:
| I haven't used pwsh in a few years, but does `ConvertTo-Json`
| still have that terrible `-Depth` parameter that you have to
| remember to set so that it doesn't mangle your JSON? A quick
| web search indicates that it does.
|
| Edit: Yes, it does. '{ "a": { "b": { "c": {
| "d": { "e": 5 } } } } }' | ConvertFrom-Json | ConvertTo-Json
| { "a": { "b": { "c":
| "@{d=}" } } }
|
| So you have to remember to set `-Depth 100` for every
| invocation of `ConvertTo-Json`. You also can't set it higher
| because it has a hard-coded limit of 100, so hopefully you
| never deal with JSON that has more nesting than that.
|
| Yeesh. I like PS, but this one commandlet's design has always
| baffled me.
| joshgel wrote:
| I recently wrote a short example based gist for a specific health
| care use case (evaluating FHIR data) using jq.
|
| https://gist.github.com/joshgel/12082d23a75feaab5d405db31981...
| sequoia wrote:
| > But, this prints all 317 resource types on a new line, not
| super helpful.
|
| > So, let's group by and count:
|
| > cat 1396-Ledner.json | jq '.entry[].resource.resourceType' |
| sort | uniq -c | sort -nr This gives me a list that looks like
| this:
|
| This is so funny! We came up with some of the exact same
| combinations of tools (jq + sort, uniq, wc etc.). I mean, it
| makes sense so I shouldn't be surprised!
| eloycoto wrote:
| And this post didn't mentioned custom functions, where things are
| getting more interesting :-) https://acalustra.com/jq-functions-
| modules-and-dotfiles-tips...
| brundolf wrote:
| At that point why not write a Python script?
| naniwaduni wrote:
| jq is a much, much more expressive language than Python at
| applying similar operations to the list of inputs (with
| fanout!).
| eloycoto wrote:
| I'm not disagree with you, but in this case all stored in
| custom dotfiles, as functions, easy to use IMHO than a custom
| Python script.
|
| But I use a lot JQ, so it's super easy for me ;-)
| johnorourke wrote:
| +1 for jq. It is so useful for mucking about with API responses.
| I used it to write a bash script that acts as a NAGIOS plugin for
| spotting changes in our Digital Ocean server and DNS configs -
| curl to pull from the API, jq to strip out the bits we don't care
| for, and sed to put actual newlines in instead of \n. We had
| Cisco as a customer, and infrastructure change-monitoring was one
| of their infosec team's requirements... that was part of our
| solution.
| heleninboodler wrote:
| I needed this tutorial 6 months ago (and 6 months before that,
| and 6 months before that). :D Highly recommend looking at and
| maybe including "gron"[1] in this as a very nice complement to
| jq. It fills in some use cases in a very straightforward way that
| are pretty cumbersome in jq, such as finding a field deeply
| nested in an optional parent.
|
| [1] https://github.com/tomnomnom/gron
| donut wrote:
| Agreed, gron is a fantastic tool.
|
| Earler today I was looking at some deeply nested structures
| which had leaf nodes with a field named "fileURL", and values
| that were mostly "https://" but some were "http://". I needed
| to see how many, etc.
|
| cat file.json | gron | grep fileURL | grep http | grep -v https
|
| ... and presto, I had only four such nodes.
|
| Would've been a ton more work to get there with just jq.
| sequoia wrote:
| oh wow cool I didn't know about gron!! Is it alright if I add
| this comment to the bottom of my post?
| heleninboodler wrote:
| Of course, although I wouldn't be surprised if a jq expert
| comes along and gives an easy way to deal with deeply nested
| children of an optional parent, making my comment moot. :D
| sequoia wrote:
| Ha, I never let that stop me! You can do some of the stuff
| in my post other ways, like using the `--jsonpath` switch
| in kubectl, but 1. I prefer learning jq rather than the
| idiosyncratic syntax/flag of each tool & 2. Let 1000
| flowers bloom. I figure its OK if there's more than one way
| to do something!
| fiatjaf wrote:
| Shameless plug: https://github.com/fiatjaf/awesome-jq
___________________________________________________________________
(page generated 2020-12-21 23:00 UTC)