[HN Gopher] Show HN: DTL: a language and JavaScript lib to trans...
___________________________________________________________________
Show HN: DTL: a language and JavaScript lib to transform and
manipulate data
DTL is a project that began it's life as part of a another piece of
software for a startup I founded a few years ago. For that project
I needed a way to describe how to rewrite data in a portable way. I
needed to be able to create the rules on the fly and store them in
a database, I needed them to be able to describe transformations I
hadn't thought about, but also needed them to be safe and
predictable. Though the startup didn't survive, the language I made
was so useful to me I felt I had to extract it and make it usable
to everyone. DTL is the result. Though the npm module is relatively
new, the language itself has been in use in production systems for
years. Over the past couple of years I have been working to make it
more accessible and useful to newcomers. Though it's really
powerful, I tried to make it easy to use and simple to understand
so that you can get up to speed quickly and use only as much as you
need. To summarize: DTL is a javascript module and related CLI
tools that are really handy for transforming data from one format
to another. It's made to allow you to specify your transformations
as data (JSON by default) which means they are easily shared from
frontend to backend and vice versa, as well as making them easily
stored in databases, etc. It can be used as part of your project to
transform data between APIs, between the frontend and your
database, etc. and can do simple mappings as well as complex
calculations. It can also be used for validation and is really
handy for extracting useful information from large / complex
datasets (there are some great examples of this you can try on the
website). The CLI tool (dtl) is like jq on sterooids, allowing you
to slice, dice and remap csv, yaml, json or even plaintext data
doing anything you can describe in a DTL transform. If you ever
wished you could `grep` in complex data structures, today is your
lucky day. :) I'd love any feedback you have and if you think of
anything it doesn't have that it should, I'd love to hear that too.
Author : jk0ne
Score : 31 points
Date : 2022-04-20 15:20 UTC (7 hours ago)
(HTM) web link (getdtl.org)
(TXT) w3m dump (getdtl.org)
| hyuuu wrote:
| how does this compare to jsonata?
|
| https://jsonata.org/
| jk0ne wrote:
| I am not super-familiar with jsonata, so someone else can
| probably comment better here. That said, I think the biggest
| difference is Jsonata is really oriented towards querying, as
| evidenced by it's xpath oriented syntax. DTL can do querying,
| but it's really made for data transformation, as evidenced by
| it's more JSON oriented syntax.
|
| I also think DTL is significantly more approachable and
| familiar and therefore easy to learn and make use of. (I think
| xpath is terribly obtuse and unintuitive, personally) DTL is
| also made to be self-contained. A single entrypoint in your
| javascript code, `DTL->apply_transform()' can process any
| transform you can create.
|
| DTL is extremely powerful, but it's deceptively simple, and you
| only need to learn as much as you need for the task at hand. It
| has no real special syntax you haven't encountered before, and
| it has no special modes. It has the capability to do things
| like map / reduce / grouping, etc. But it accomplishes this in
| a very functional-form, requiring no special syntax beyond what
| you would already use to get a string length, for example. DTL
| also has a number of builtins that are very helpful when
| remapping data that you would otherwise have to implement
| yourself. And the entirety of any DTL transform can go wherever
| you can put a JSON block, including in your db or a config
| file, which makes it significantly easier to make use of in
| your projects.
|
| DTL also has a very sophisticated CLI tool, allowing you to
| make direct use of it without writing any code whatsoever, so
| you can start working with your data immediately, without
| having to learn much at all. DTL also has a great REPL that
| lets you play with the syntax, complete with full help within
| the REPL.
|
| In a general sense, DTL predates Jsonata by several years
| (originally created in 2012) and has been in use that entire
| time, though I only published to npm a little over a year ago.
|
| SO.. TL;DR:
|
| * Made for transforming complex data in and out, not just
| querying
|
| * Much easier to use and more familiar
|
| * Really useful builtin functions
|
| * more approachable, use as much or as little as you need
|
| * Embeddable
|
| * Great CLI tools
|
| But, to be fair, I am a little biased. ;)
| [deleted]
| mattewong wrote:
| Looks cool. But why use this instead of jq? You mention it's
| "like jq on steroids"; could you provide an example of a real-
| world use case where it solves a problem that couldn't be solved
| with jq?
| jk0ne wrote:
| There are a number of areas where DTL can be used where jq
| can't. The obvious ones are that DTL is a library and is
| therefore usable directly in your own code, both frontend and
| backend.
|
| In terms of the command line, I'm sure advanced users of jq
| could get almost, if not all of DTL's command line
| functionality out of jq, that said I think DTL would make it a
| lot easier. DTL, I think, provides a more approachable syntax,
| especially when you are doing more than just extracting data as
| it already exists. The DTL builtins of `grep` `group` are great
| examples, as are `chain` `derive` and the various set-
| operations present in DTL.
|
| Add to that that DTL is functional in nature and allows you to
| specify your own transforms as controls to the various
| functions, you can easily get very sophisticated with your
| remapping in ways that would be much more difficult (maybe
| impossible) with jq.
|
| In terms of an example, the one most readily available I have
| got turned into a DTL test (in dtl-expressions-advanced.js in
| the test suite), which is to perform a soundex search on
| textual input data. DTL doesn't support soundex natively, but
| it was trivial to create it with a DTL transform, and because
| it's json, it can be imported into any transformation object
| you might need to use it in.
|
| And because DTL is a module AND a cli, you can move your
| transforms back and forth between your code and your command
| line without re-translation, which I have found to be immensely
| useful... but YMMV.
| stevage wrote:
| jq is a DSL that is IMHO pretty hard to learn and remember. I
| kept trying to use it for years. I would find using a JS
| library much easier.
| mst wrote:
| This is really quite fascinating.
|
| Can it handle JSON with trailing commas and similar? jq hating
| that has been something of a bugbear of mine.
|
| (also, are you the jayk I remember from back in the day?)
| jk0ne wrote:
| Thanks.
|
| It can. The cli can handle json5 (https://json5.org/) so
| trailing commas and comments are a-ok.
|
| And yes, re: jayk. Almost certainly. :)
|
| It has been an ambition of mine to turn this into a c lib to
| parse/process the syntax into an AST so it can be used in
| multiple languages... but I haven't been able to set the time
| aside for that particular batch of work, yet.
___________________________________________________________________
(page generated 2022-04-20 23:00 UTC)