[HN Gopher] Show HN: jql - Easier jq Alternative with a Lispy Sy...
___________________________________________________________________
Show HN: jql - Easier jq Alternative with a Lispy Syntax Written in
Go
Author : cube2222
Score : 138 points
Date : 2020-01-07 16:00 UTC (7 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| cookiecaper wrote:
| To be frank the ascendance of jq has always seemed weird to me.
| There are many alternatives that felt much more natural, as this
| thread demonstrates.
|
| JSONata remains pretty obscure but it's been my favorite JSON
| query/mangling DSL for a while now: https://github.com/jsonata-
| js/jsonata.
|
| jmespath is also quite natural, but I think it was hurt by
| reference implementation in Python, meaning it's quite slow.
| http://jmespath.org/
| rajangdavis wrote:
| jq is simple to install, can handle large datasets (have used
| it to parse GB's of JSON data), and works well with bash/shell.
| pololee wrote:
| I have used fx for a few days and enjoyed it.
| https://github.com/antonmedv/fx
|
| Highlights - interactive mode - Use full power of JavaScript.
|
| $ curl ... | fx '.filter(x => x.startsWith("a"))'
|
| - Access all lodash (or ramda, etc) methods by using .fxrc file.
|
| $ curl ... | fx '_.groupBy("commit.committer.name")'
| '_.mapValues(_.size)'
| cube2222 wrote:
| I've seen fx recently and it looks very interesting. jql is
| obviously not as feature-rich, but you can achieve
| interactivity with fzf! echo '' | fzf --print-
| query --preview-window wrap --preview 'cat test.json | jql {q}'
| xvilka wrote:
| There is more universal alternative to jq that supports also
| protobuf, msgpack, cbor, etc - rq[1].
|
| [1] https://github.com/dflemstr/rq
| cube2222 wrote:
| Thanks, didn't know about it, looks interesting!
|
| Though from the tutorial I don't really understand how I can
| use it to transform data, other than converting between
| formats.
| dan-robertson wrote:
| Looks like query support was dropped.
| dan-robertson wrote:
| I tend end to struggle with this sort of lispy syntax as I use a
| regular shell (rather than one in emacs) which makes balancing
| parens hard, and the syntax is otherwise verbose. If I'm using a
| query tool like this it will be interactively writing a one-liner
| so I want syntax that's easy to get right (balancing parens by
| hand is hard), easy in the usual case (eg I think it should
| accept some way to use symbols to name fields in a record because
| typing and balancing double quotes all of the time is a waste of
| time) and easy to append to (I will run a query, press up, move
| to the end of the query, and modify. In this case I think moving
| to the end involves counting parens).
|
| Perhaps I would want some implicit pipe or threading at the top
| level (which could be turned off with some option for scripts I
| guess).
|
| Another helpful feature could be adding the magic closing paren:
| ] which will close all open parens to the top level. E.g. these
| two lines are equivalent: (foo (bar (baz) whizz))
| (foo (bar (baz) whizz]
|
| A completely different (and less powerful) paradigm I like is one
| of match-format where the pattern looks a lot like what it's
| trying to match. In this case most functions and fancy
| transformations are impossible. So you could write something like
| this to extract the list of countries: j
| '{countries: %%}'
|
| Or to extract each country's name: j '{countries:
| [ .*, { name: %% }, .*]}'
|
| (Presumably you would have syntax for trying to match against
| each array element). (I also would make commas optional but I'm
| trying to make this syntax obvious).
|
| Or to list the keys of the 0th element: j
| '{countries: [ { %%: . }, .* ]}'
|
| Or to get name and population: j '{countries: [
| .*, { name: %name, population: %pop }]}'
|
| The output might look like: { "name": "Poland",
| "population": "38000000" }
|
| ----------------
|
| An alternative query tool which I haven't really seen before
| would be one which is interactive rather than repeatedly run: you
| pipe your json (or whatever) into it and it spins up some kind of
| (curses) gui shoeing you your data with a nice interface to
| interactively write your query and see its results. It could
| finish by outputting a command line to run the query non-
| interactively.
|
| The closest thing to this I know of is a small programming by
| example demo that came out of Microsoft research. The goal was to
| figure out how to extract some desired data from some big blob of
| json (or maybe html, I don't remember). The user would see the
| json in some web gui and could click on the data they wanted and
| the app would try to figure out the "most obvious" query to get
| it. If they clicked on the country name, I think it would
| probably suggest a query for all the countries' names (but maybe
| that would be the second option). I think it would also try to be
| clever about grouping, so if the user picked name and population
| it would hopefully figure out that they should come in pairs
| rather than being two independent lists that might not be the
| same length.
| cube2222 wrote:
| Thanks so much for so much feedback!
|
| There's already an issue to automatically add trailing parens
| and I definitely think it's a good idea, as balancing parens
| really is annoying in-shell. I didn't think of it before.
|
| I decided that adding interactivity would be too much work, but
| you could definitely put together a oneliner (or alias) which
| achieves that using fzf: https://paweldu.dev/posts/fzf-live-
| repl/
|
| Overall I encourage you to try write a tool that has the syntax
| you described, as it sounds plausible! It kinda looks a little
| bit like graphql, so maybe that would fit?
| dan-robertson wrote:
| I wonder if it's worth writing some read line macros (and
| instructions for setting them up) for eg inserting ( ) <move-
| cursor-backwards>. (Bound to M-( in emacs) I think they help
| for simple sexp editing. I also wish I had something like
| "run this command but when it's done show me a prompt with
| the same command and my cursor in the same place" but I have
| no idea if it's easy
| cube2222 wrote:
| I didn't know about readline macros, thanks for mentioning
| them.
|
| I'll think about it.
| cube2222 wrote:
| I've written the interactivity oneliner: echo
| '' | fzf --print-query --preview-window wrap --preview 'cat
| test.json | jql {q}'
| mharju wrote:
| Looks cool!
|
| I'm using https://github.com/borkdude/babashka with with
| https://github.com/borkdude/jet and a few aliases along with curl
| for the same effect.
|
| The syntax of jq is so hard to remember a familiar lisp always
| beats it :)
| cube2222 wrote:
| Nice, this looks great, and I love Clojure too, so I'll make
| sure to check it out!
| lcfcjs2 wrote:
| Syntax is too confusing, I'll stick with jq thanks
| onionisafruit wrote:
| Nice project. I'm a heavy jq user and still find myself regularly
| tinkering around with queries on https://jqplay.org. It would be
| nice if there were a similar page for jql, even if it is just a
| local server.
| cube2222 wrote:
| I was thinking of an interactive terminal app, but that would
| be much more work.
|
| Though you can surely put a oneliner together which achieves
| the same effect using fzf, as per:
| https://paweldu.dev/posts/fzf-live-repl/
| preek wrote:
| Here's an interactive implementation (in Emacs):
| https://github.com/200ok-ch/counsel-jq
| cube2222 wrote:
| Done: echo '' | fzf --print-query --preview-
| window wrap --preview 'cat test.json | jql {q}'
| moondev wrote:
| Personally I like https://github.com/mikefarah/yq
|
| It also converts between yaml and json with ease for writing and
| reading
| spikepuppet wrote:
| I'm super excited to sink my teeth into this. I usually have to
| use jq in intense bursts and no matter what i end up back in the
| manual because it just get's so damn confusing.
|
| From a quick look on the repo, this seems so much simpler.
| Thanks!!!
| cube2222 wrote:
| I'm happy that you're so enthusiastic! Make sure to let me know
| what you think about it after having tried it out!
| preek wrote:
| If you're into jq, here's an interface so you can have live
| feedback (in Emacs): https://github.com/200ok-ch/counsel-jq
| cube2222 wrote:
| You can probably also use an analogue to: echo
| '' | fzf --print-query --preview-window wrap --preview 'cat
| test.json | jql {q}'
|
| Something close to: echo '' | fzf --print-query
| --preview-window wrap --preview 'cat test.json | jq {q}'
| michaelmrose wrote:
| If this was lispy then (elem "countries" (elem 0)
|
| Would be an error.
|
| Expression evaluation ought to be inside out.
| cube2222 wrote:
| Why?
|
| It is, it's just that everything returns a function! The
| topmost function will in the end be called with the json as an
| argument.
|
| Check out these parts of the README:
| https://github.com/cube2222/jql#attention
| https://github.com/cube2222/jql#type-cheatsheet
|
| (elem 0) returns a function which takes a JSON array as input
| and returns its first element.
| michaelmrose wrote:
| Can you find many lisps or even any lisps that evaluate like
| that? A familiar syntax that actually works different from
| any lisp in existence may be more hindrance than help.
| murat124 wrote:
| echo '' | fzf --print-query --preview-window wrap --preview 'cat
| test.json | jql {q}'
|
| Trivial note. echo instead of echo '' or echo "".
| $> [[ "$(echo)" == "$(echo '')" ]] && echo $? || echo $? $>
| 0
| fiatjaf wrote:
| Shameless plug to my list of interesting `jq` stuff:
| https://github.com/fiatjaf/awesome-jq
|
| It even features a complete rewrite of the actual jq in Go.
| EdSchouten wrote:
| So I guess we all know https://jsonnet.org, right? It's
| advertised as a data templating language. Interestingly enough,
| you can also use it as a querying tool. The first four examples
| on the jql page can be translated to Jsonnet as follows:
| $ jsonnet -e "(import 'test.json').countries[0]" $
| jsonnet -e "(import 'test.json').countries[0:2]" $
| jsonnet -e "[x.name for x in (import 'test.json').countries]"
| $ jsonnet -e "std.objectFields((import
| 'test.json').countries[0])"
|
| With that in mind, I never bothered to learn how to use a tool
| like jq, rq, jql, etc.
| ljm wrote:
| My favourite is yq, because there are two competing and utterly
| incompatible implementations of it. Just to add onto the
| shitshow that YAML can be.
| FragenAntworten wrote:
| Another option which looks interesting:
| https://github.com/jmespath/jp
|
| I haven't used it yet, but the syntax seems simpler than some.
| edaemon wrote:
| Thanks for the link, this straightforward syntax is what I've
| been after for dealing with JSON on the command line.
| hnlmorg wrote:
| Shameless plug but I've also written a language (murex) for
| shell scripting which can be used for this too:
| open test.json -> [ countries ] open test.json -> [[
| /countries/0 ]] open test.json -> [ countries ] -> [ 0
| 2 ]
|
| Single brackets only return the next level deep but can
| retrieve multiple items (and negative integers to count from
| the end of an array). Double brackets are to specify a path. So
| [[/foo/bar]] is literally the same as [foo]->[bar]
|
| Lastly the 4th example: open test.json -> [
| countries ] -> foreach c { echo $c[name] }
|
| Unfortunately this one doesn't output it in JSON. If you needed
| that you could chain another command to reformat it:
| open test.json -> [ countries ] -> foreach c { echo $c[name] }
| -> cast str -> format json
|
| This works because murex can auto-convert between lists, JSON,
| YAML, CSV and a few other structured formats. However it's fair
| to say it is a lot more verbose than jsonnet and jql in that
| last example.
|
| Github repo: https://github.com/lmorg/murex
|
| I've been using this as my primary shell for a few years now
| and, like yourself, I've never bothered to learn jq because of
| that.
|
| NB all of the above example are running from inside the murex
| interactive command line (like bash). If you wanted to run it
| like jq/jql then you'd need to do the same sort of thing as
| bash: murex -c 'open test.json -> [ countries
| ]'
| cube2222 wrote:
| This is actually the best alternative to jq for me I've seen so
| far! (other than jql obviously ;) ) Thanks for posting!
| jaib8 wrote:
| I haven't used jsonnet before but is it possible to read via
| stdin? for eg to parse a json response from a rest endpoint in
| jq I would do something like: $ curl "..." | jq .abc
| theamk wrote:
| > Language Reference
|
| > TODO: This page still needs to be written.
|
| Looks like it is not ready for prime time yet... (Not to
| mention that it seems to have no support for stdin input,
| multiple input files, and json lines format - which are my main
| uses cases for jq)
| cube2222 wrote:
| Hey, author here. Christmas Eve, 22 pm, after having eaten with
| my family, I was browsing HN (because what else could you be
| doing then), and stumbled upon this comment:
| https://news.ycombinator.com/item?id=21860107
|
| It inspired me to create an alternative to jq with more of a
| Lispy syntax, as I think the original is awesome but also fairly
| cryptic for anything more advanced than single field selection.
|
| Overall it was a fun few-day project, and was also very
| educational in terms of writing a parser in Go. (I used goyacc
| before in the sql parser of OctoSQL[1], however, that one is
| copied from vitess, so I've never built one from scratch, only
| customised an existing one. It's really pleasant overall and the
| code is very simple, so I encourage you to take a look[2].
|
| I'd love to hear any feedback, comments or potential improvements
| you can think of.
|
| [1]:https://github.com/cube2222/octosql
|
| [2]:https://github.com/cube2222/jql/tree/master/jql/parser
| jsd1982 wrote:
| Looks good but there seems to be a missing explanation gap
| between switching from `(elem "countries")` and dropping the
| `elem`s to just `("countries")`. Why is `elem` able to be
| dropped? Is it the default function or something? I didn't
| follow that big jump in the readme.
| cube2222 wrote:
| It's explained right below, I think I messed up the order
| there.
|
| Is this understandable? "You can see that elem is the most
| used function, and in fact it's what you'll usually be using
| when munging data, so there's a shortcut. If you put a value
| in function name position, it implicitly converts it to an
| elem."
|
| EDIT: Fixed the order.
| daxelrod wrote:
| Nicely done!
|
| I scratched a similar itch by creating Jowl[1], which uses
| JavaScript one-liners with the Lodash library to transform
| JSON.
|
| Like you, I wanted syntax that was more familiar to me than
| jq's, and like you, I wanted to use a language that was built
| for data structure transformation.
|
| However, I offloaded all of the actual hard work to an existing
| JavaScript runtime, and to an existing library for data
| structure transformation, meaning I could punt on the parser
| and language design; the hard parts that you've taken on. I
| have this bookmarked to read through in more detail later this
| week.
|
| [1]: https://github.com/daxelrod/jowl
| cube2222 wrote:
| Sounds great! I'll make sure to check out out!
|
| My approach was caused by the fact that I wanted to have a
| minimal language which only contains what really is necessary
| and is as regular as possible, to be very simple as a result.
| downerending wrote:
| Looks great!
|
| As an old Lisper, I did get a wry chuckle out of the fact that
| you managed to avoid Lisp for a tool that uses Lispy syntax.
| :-)
| cube2222 wrote:
| Thanks!
|
| Yup, I felt the irony myself creating it, as I actually love
| clojure!
|
| But I'm most comfortable with Go, and it results with a
| single binary for any of the major platforms with very quick
| startup time (as illustrated by the small benchmark), so
| those were huge advantages.
| useragent86 wrote:
| I love the idea of a Lispy version of jq, but as a Lisper,
| the examples feel bewildering to me. The fact that (elem)
| takes an optional second argument, whose default value is
| 'identity, seems familiar, but then I try to understand
| this example, and it just doesn't make sense to me:
| (elem "countries" (elem (keys) (elem "name")))
|
| I think part of the problem is, IIUC, this is constructing
| a pipeline, like (->) in Clojure, but is using function
| call syntax rather than pipeline syntax, and also mixing in
| a function call with (keys). I feel like it's a very non-
| Lispy language masquerading as Lisp by wearing parens.
|
| Instead, if you wrote it as real Lisp function calls, and
| provided a real pipeline macro, it would be much more Lispy
| and, I think, much easier to understand. For example,
| instead of: (elem "countries" (elem
| (keys) (elem "name")))
|
| Either of these alternative syntaxes: (->
| "countries" (key "name")) (key "name"
| ("countries"))
|
| Would return the same result: ["Poland",
| "United States", "Germany"]
|
| While being more concise and more Lispy.
|
| I also wouldn't mind using an anaphoric 'mapcar, something
| like: (-> "countries" (map-> (key
| "name"))
|
| Otherwise it may be unclear which functions implicitly map
| across entries and which don't.
|
| My two cents. Thanks for sharing your work.
| cube2222 wrote:
| Thanks for the feedback!
|
| Have you checked out the type cheatsheet? (keys) is a
| function call!
|
| The function you pass as last to elem is kind of a
| continuation. It's a function which transforms the output
| before returning.
|
| All you're doing writing a jql query is composing one big
| function.
|
| With values in function call position, it's just that
| (keys) gets evaluated to a value (the keys of the map)
| and that is used to index the json. That's why keys
| wouldn't work, you need the parentheses to make it a
| function call.
|
| This may sound complex at first but I find it gets
| intuitive quickly.
|
| EDIT: Check out this comment thread on reddit as I think
| the commenter may have had similar problems: https://www.
| reddit.com/r/golang/comments/ehnsz5/jql_json_que...
| useragent86 wrote:
| > Have you checked out the type cheatsheet?
|
| I'm afraid that that does not help me understand how
| (elem) works at all. From my perspective, I don't need a
| "type cheatsheet," I need elem's docstring that explains
| what arguments it expects and what it returns. I'm not
| even thinking about value types yet, and I don't know
| what grammar that cheatsheet is written in, nor why I
| would need to read one to be able to use what seems like
| the most basic function jql has, the elem function.
|
| To write clear, useful documentation, you need to put
| yourself in the shoes of a user who has never seen your
| project before, who has not walked the miles you have to
| arrive at the solutions you have, and take them quickly
| to the same destination.
|
| The readme, in general, is another issue. It's...not
| ideal.
|
| The first part tries to sound cute with stuff like, "Hey
| there!" and, "remember? That's explicitly not why we're
| here. But it aids understanding of the more complex
| examples, so stay with me just a little bit longer!".
|
| But I'm not a little kid who's bored in math class, so
| writing like that turns me off quickly. Having to read
| through long prose like, "Ok, let's check it out now, but
| first things first, you have to install it:" instead of
| simply a heading titled "Install:" feels like I'm wasting
| my time, and I quickly skip ahead or move on to the next
| HN article.
|
| What I'm really looking for is a short intro and a table
| of contents with sections like: Intro, Examples,
| Tutorial, Reference, FAQ.
|
| You might think of it like this: the project is your
| baby, but just like in real life, no one thinks your
| baby's as cute as you do, and, no, we don't want to see
| the baby pictures. ;)
| cube2222 wrote:
| I actually don't think like that. I even have another
| fairly successful open source project whose readme is
| probably more to your liking, and is nothing like this
| one: https://github.com/cube2222/octosql
|
| Everybody has different taste. I like README's like that
| as they add an entertainment value. Overall I've got two
| kinds of feedback about the README: 1. I really like the
| project and really liked the readme, great fun to read!
| 2. The readme was very confusing on top of the already
| confusing query language.
|
| And based on that I'm planning to keep it unchanged for
| now. I added the type cheatsheet to make case 2 at least
| a little bit better.
|
| As for the quick examples, you can just scroll through
| the code blocks (as each one contains input with its
| corresponding output).
| useragent86 wrote:
| Since you're discarding feedback kind #2, and all of the
| potential users it represents, in favor of entertainment
| value, I'd say that you _do_ think like that. ;)
| Comedians write lots of jokes, but when they stand up and
| tell them in front of an audience, they find out which
| ones work and which ones don 't. How they respond to that
| feedback determines their success. Hey, it's your
| project.
| cube2222 wrote:
| I'm definitely not discarding it! I'm open to writing
| additional clarifying paragraphs. (that's why I wrote the
| type cheat sheet). I'm also not discarding feedback kind
| #1 :)
|
| I'm thinking of adding another one which shows how
| standard map/filter notation maps to jql queries.
|
| Anyways, thanks for the feedback again, I do appreciate
| it!
| useragent86 wrote:
| Regarding CPS: I think that expecting users to write
| queries in CPS is going to be very limiting. It's missing
| a big opportunity to use simple funcall and/or pipeline
| syntax, with which many more users are familiar. Even
| among Lispers, CPS isn't that popular. But Lisp, in
| general, could make jq-like syntax much more regular and
| usable.
| cube2222 wrote:
| I find it gets intuitive without being familiar with CPS
| (I wasn't before).
|
| In practice, thanks to this, the resulting queries end up
| matching the input JSON structure very well.
| mst wrote:
| I was seeing it as roughly: (elem x f)
|
| is sugar for (filter-on f (elem x))
|
| or so? (filter-on imaginary thing but hopefully obvious
| enough as pseudocode)
| cube2222 wrote:
| Almost! It's more like (map f (elem x JSON)), where map
| gets changed to apply, if (elem x JSON) results in a
| single value (non-collection).
| yamafaktory wrote:
| Hi, awesome work! Love the idea of using S-expression for the
| query syntax! Funny fact: we share the same project name - even
| though the core language and the query syntax are different -
| see https://github.com/yamafaktory/jql.
| cube2222 wrote:
| Hey, looks really nice! I see we had similar goals of
| simplicity in mind, though with very different approaches.
|
| Sorry for the name duplication! I thought I'm sharing the
| name only with the jira API jql.
|
| EDIT: I'd love to see some benchmarks. I expect your jql to
| greatly outperform mine and jq, considering it's written in
| Rust!
| yamafaktory wrote:
| No worries regarding the naming, it's what makes open-
| source friendly! You can see some benches here e.g.
| https://travis-ci.org/yamafaktory/jql/jobs/618374730#fold-
| st... but those actually more like regression tests that
| are triggered for pull requests to compare them against
| master (and I should use Github actions probably here now
| instead of Travis). Thanks for your reply!
| markandrewj wrote:
| This is fairly short talk, but it might interesting to some
| people.
|
| https://youtu.be/PS_9pyIASvQ [Serious Programming with jq?! A
| practical and Purely Functional Programming Language!]
| cube2222 wrote:
| Thanks for posting! I'll make sure to watch it!
| markandrewj wrote:
| No problem. I hope it is helpful and interesting for you.
| ldd wrote:
| only tangentially related, I made a VS Code extension to help out
| with jq development[0].
|
| Also, jq supports modules. which is awesome!
|
| [0]https://marketplace.visualstudio.com/items?itemName=ldd-
| vs-c...
___________________________________________________________________
(page generated 2020-01-07 23:00 UTC)