Title: Nushell: Introduction to a new kind of shell
Author: Solène
Date: 31 October 2022
Tags: openbsd nixos nushell shell
Description: This article covers a gentle introduction to the shell
nushell
# What is nushell
Let me introduce you to a nice project I found while lurking on the
Internet. It's called nushell and is a non-POSIX shell, so most of
your regular shells knowledge (zsh, bash, ksh, etc…) can't be applied
on it, and using it feels like doing functional programming.
It's a good tool for creating robust data manipulation pipelines, you
can think of it like a mix of a shell which would include awk's power,
behave like a SQL database, and which knows how to import/export
XML/JSON/YAML/TOML natively.
You may want to try nushell only as a tool, and not as your main shell,
it's perfectly fine.
With a regular shell, iterating over a command output can be complex
when it involves spaces or newlines, for instance, that's why `find`
and `xargs` have a `-print0` parameter to have a special delimited
between "items", but it doesn't compose well with other tools. Nushell
handles correctly this situation as its manipulates the data using
indexed entries, given you correctly parsed the input at the beginning.
(HTM) Nushell official project page
(HTM) Nushell documentation website
# How to get it
Nushell is a rust program, so it should work on every platform where
Rust/Cargo are supported. I packaged it for OpenBSD, so it's available
on -current (and will be in releases after 7.3 is out), the port could
be used on 7.2 with no effort.
With Nix, it's packaged under the name `nushell`, the binary name is
`nu`.
For other platforms, it's certainly already packaged, otherwise you can
find installation instructions to build it from sources.
(HTM) Nushell documentation: Building nushell from sources
# Configuration
At first run, you are prompted to use default configuration files, I'd
recommend accepting, you will have files created in
`~/.config/nushell/`.
The only change I made from now is to make Tab completion
case-sensitive, so `D[TAB]` completes to `Downloads` instead of asking
between `dev` and `Downloads`. Look for `case_sensitive_completions` in
`.config/nushell/config.nu` and set it to `true`.
# Examples
If you are like me, and you prefer learning by doing instead of reading
a lot of documentation, I prepared a bunch of real world use case you
can experiment with. The documentation is still required to learn the
many commands and syntax, but examples are a nice introduction.
## Getting help
Help from nushell can be parsed directly with nu commands, it's
important to understand where to find information about commands.
Use `help a-command` to learn from a single command:
```script
> help help
Display help information about commands.
Usage:
> help {flags} ...(rest)
Flags:
-h, --help - Display this help message
-f, --find <String> - string to find in command names, usage, and search terms
[cut so it's not too long]
```
Use `help commands` to list all available commands (I'm limiting to 5
between there are a lot of commands)
```script
help commands | last 5
╭───┬─────────────┬────────────────────────┬───────────┬───────────┬────────────┬───────────────────────────────────────────────────────────────────────────────────────┬──────────────╮
│ # │ name │ category │ is_plugin │ is_custom │ is_keyword │ usage │ search_terms │
├───┼─────────────┼────────────────────────┼───────────┼───────────┼────────────┼───────────────────────────────────────────────────────────────────────────────────────┼──────────────┤
│ 0 │ window │ filters │ false │ false │ false │ Creates a sliding window of `window_size` that slide by n rows/elements across input. │ │
│ 1 │ with-column │ dataframe or lazyframe │ false │ false │ false │ Adds a series to the dataframe │ │
│ 2 │ with-env │ env │ false │ false │ false │ Runs a block with an environment variable set. │ │
│ 3 │ wrap │ filters │ false │ false │ false │ Wrap the value into a column. │ │
│ 4 │ zip │ filters │ false │ false │ false │ Combine a stream with the input │ │
╰───┴─────────────┴────────────────────────┴───────────┴───────────┴────────────┴───────────────────────────────────────────────────────────────────────────────────────┴──────────────╯
```
Add `sort-by category` to list them... sorted by category.
```
help commands | sort-by category
```
Use `where category == filters` to only list commands from the
`filters` category.
```
help commands | where category == filters
```
Use `find foobar` to return lines containing `foobar`.
```
help commands | find insert
```
## General examples
### Converting a data structure into another
This is just an example from YAML to JSON, but you can convert much
more formats into other formats.
```
open dev/home-impermanence/tests/impermanence.yml | to json
{
"directories":
[
"Documents",
"Downloads",
"Datastore/Music",
"Datastore",
"Datastore/",
"Datastore/Music/Band1",
".config",
"foo/bar",
"foo/bar/hello"
],
"size": "500m",
"files":
[
".Xdefaults",
".profile",
".xsession",
]
}
```
### Parsing sysctl output
```
sysctl -a | parse -r "(?<key>.*?)=(?<value>.*)"
```
Because the output would be too long, here is how you get 10 random
keys from sysctl.
```
sysctl -a | parse -r "(?<key>.*?)=(?<value>.*)" | shuffle | last 10 | sort-by key
╭───┬─────────────────────────────────────────────────┬──────────╮
│ # │ key │ value │
├───┼─────────────────────────────────────────────────┼──────────┤
│ 0 │ fs.quota.reads │ 0 │
│ 1 │ net.core.high_order_alloc_disable │ 0 │
│ 2 │ net.ipv4.conf.all.drop_gratuitous_arp │ 0 │
│ 3 │ net.ipv4.conf.default.rp_filter │ 2 │
│ 4 │ net.ipv4.conf.lo.disable_xfrm │ 1 │
│ 5 │ net.ipv4.conf.lo.forwarding │ 0 │
│ 6 │ net.ipv4.ipfrag_low_thresh │ 3145728 │
│ 7 │ net.ipv6.conf.all.ioam6_id │ 65535 │
│ 8 │ net.ipv6.conf.all.router_solicitation_interval │ 4 │
│ 9 │ net.mptcp.enabled │ 1 │
╰───┴─────────────────────────────────────────────────┴──────────╯
```
### Recursively convert FLAC files to OPUS
A complicated task using a regular shell, recursively find files
matching a pattern and then run a given command on each of them, in
parallel. Which is exactly what you need if you want to convert your
music library into another format, let's convert everything from FLAC
to OPUS in this example.
In the following command line, we will look for every `.flac` file in
the subdirectories, then run in parallel using `par-each` the command
`ffmpeg` on it, from its current name to the old name with `.flac`
changed to `.opus`.
The `let convert` and `| complete` commands are used to store the
output of each command into a result table, and store it in the
variable `convert` so we can query it after the job is done.
```
let convert = (ls **/*flac | par-each { |file| do -i { ffmpeg -i $file.name ($file.name | str replace flac opus) } | complete })
```
Now, we have a structure in `convert` that contains the columns
`stdout`, `stderr` and `exit_code`, so we can look if all the commands
did run correctly using the following query.
```
$convert | where exit_code != 0
```
### Synchronize a music library to a compressed one
I had a special need for my phone and my huge music library, I wanted
to have a lower quality version of it synced with syncthing, but I
needed this to be easy to update when adding new files.
It takes all the music files in `/home/user/Music/` and creates a 64K
opus file in `/home/user/Stream/` by keeping the same file tree
hierarchy, and if the opus destination file exists it's skipped.
```nushell
cd /home/user/Music/
let dest = "/home/user/Stream/"
let convert = (ls **/* |
where name =~ ".(mp3|flac|opus|ogg)$" |
where name !~ "(Audiobook|Piano)" |
par-each {
|file| do -i {
let new_name = ($file.name | str replace -r ".(flac|ogg|mp3)" ".opus")
if (not ([$dest, $new_name] | str join | path exists)) {
mkdir ([$dest, ($file.name | path dirname)] | str join)
ffmpeg -i $file.name -b:a 64K ([$dest, $new_name] | str join)
} | complete
}
})
$convert
```
### Convert PDF/CBR/CBZ pages into webp and CBZ archives
I have a lot of digitalized books/mangas/comics, this conversion is a
handy operation reducing the size of the files by 40% (up to 70%).
```
def conv [] {
if (ls | first | get name | str contains ".jpg") {
ls *jpg | par-each {|file| do -i { cwebp $file.name -o ($file.name | str replace jpg webp) } | complete }
rm *jpg
}
if (ls | first | get name | str contains ".ppm") {
ls *ppm | par-each {|file| do -i { cwebp $file.name -o ($file.name | str replace ppm webp) } | complete }
rm *ppm
}
}
ls * | each {|file| do -i {
if ($file.name | str contains ".cbz") { unzip $file.name -d ./pages/ } ;
if ($file.name | str contains ".cbr") { unrar e -op./pages/ $file.name } ;
if ($file.name | str contains ".pdf") { mkdir pages ; pdfimages $file.name pages/page } ;
cd pages ; conv ; cd ../ ; ^zip -r $"($file.name).webp.cbz" pages ; rm -fr pages
} }
```
### Parse gnu tar output
```
〉tar vtf nushell.tgz | parse -r "(.*?) (.*?)\/(.*?)\\s+(.*?) (.*?) (.*?) (.*)" | rename mode owner group size date time path
╭───┬────────────┬────────┬───────┬───────┬────────────┬───────┬────────────────────╮
│ # │ mode │ owner │ group │ size │ date │ time │ path │
├───┼────────────┼────────┼───────┼───────┼────────────┼───────┼────────────────────┤
│ 0 │ drwxr-xr-x │ solene │ wheel │ 0 │ 2022-10-30 │ 16:45 │ nushell │
│ 1 │ -rw-r--r-- │ solene │ wheel │ 519 │ 2022-10-30 │ 13:41 │ nushell/Makefile │
│ 2 │ -rw-r--r-- │ solene │ wheel │ 29304 │ 2022-10-29 │ 18:49 │ nushell/crates.inc │
│ 3 │ -rw-r--r-- │ solene │ wheel │ 75003 │ 2022-10-29 │ 13:16 │ nushell/distinfo │
│ 4 │ drwxr-xr-x │ solene │ wheel │ 0 │ 2022-10-30 │ 00:00 │ nushell/pkg │
│ 5 │ -rw-r--r-- │ solene │ wheel │ 337 │ 2022-10-29 │ 18:52 │ nushell/pkg/DESCR │
│ 6 │ -rw-r--r-- │ solene │ wheel │ 14 │ 2022-10-29 │ 18:53 │ nushell/pkg/PLIST │
╰───┴────────────┴────────┴───────┴───────┴────────────┴───────┴────────────────────╯
```
### Opening spreadsheets
```
〉open --raw freq.ods | from ods | get Sheet1 | headers
╭───┬─────────────┬──────────────┬───────────┬─────────┬───────────────┬────────────┬───────┬─────────┬─────────┬──────────╮
│ # │ Policy │ Compile time │ Idle time │ column3 │ Compile power │ Idle power │ Total │ column8 │ column9 │ column10 │
├───┼─────────────┼──────────────┼───────────┼─────────┼───────────────┼────────────┼───────┼─────────┼─────────┼──────────┤
│ 0 │ powersaving │ 1123.00 │ 0.00 │ │ 5.90 │ 0.00 │ 5.90 │ │ │ │
│ 1 │ auto │ 871.00 │ 252.00 │ │ 5.60 │ 0.74 │ 6.34 │ │ 0.44 │ 6.94 │
╰───┴─────────────┴──────────────┴───────────┴─────────┴───────────────┴────────────┴───────┴─────────┴─────────┴──────────╯
```
We can format new strings from columns values.
```
〉open --raw freq.ods | from ods | get Sheet1 | headers | each {|row| do { echo $"($row.Policy) = ($row.'Compile power' + $row.'Idle power') Watts" } }
╭───┬─────────────────────────╮
│ 0 │ powersaving = 5.9 Watts │
│ 1 │ auto = 6.34 Watts │
╰───┴─────────────────────────╯
```
### Filter and sort a JSON
There is a website listing packages that can be updated on OpenBSD at
https://portroach.openbsd.org, it provides json of data for rendering.
We can use this data to sort which maintainer has the most up to date
percentage, but only if they manage more than 30 packages.
```
fetch https://portroach.openbsd.org/json/totals.json | get results | where total > 30 | sort-by percentage
```
## NixOS examples
### Query profiles packages
```
nix profile list | parse "{index} {flake} {source} {store}"
╭───┬───────────────────────────────────────────────────────┬──────────────────────────────────────────────────────────────────────────────────┬─────────────────────────────────────────────────────────────────╮
│ # │ flake │ source │ store │
├───┼───────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────────┤
│ 0 │ flake:nixpkgs#legacyPackages.x86_64-linux.libreoffice │ path:/nix/store/iw3xi0bfszikb0dmyywp7pm590jvbqvs-source?lastModified=1663494472& │ /nix/store/1m6wp1pznhf2nrvs7xwmvig5x3nspq0j-libreoffice-7.2.6.2 │
│ │ │ narHash=sha256-fSowlaoXXWcAM8m9wA6u+eTJJtvruYHMA+Lb%2ftFi%2fqM=&rev=f677051b8dc0 │ │
│ │ │ b5e2a9348941c99eea8c4b0ff28f#legacyPackages.x86_64-linux.libreoffice │ │
│ 1 │ flake:nixpkgs#legacyPackages.x86_64-linux.dino │ path:/nix/store/9cj1830pvd88lrwmmxw65achd3lw2q9n-source?lastModified=1667050928& │ /nix/store/ljhn4n1q5pk7wr337v681m1h39jp5l2y-dino-0.3.0 │
│ │ │ narHash=sha256-xOn0ZgjImIyeecEsrjxuvlW7IW5genTwvvnDQRFncB8=&rev=fdebb81f45a1ba2c │ │
│ │ │ 4afca5fd9f526e1653ad0949#legacyPackages.x86_64-linux.dino │ │
╰───┴───────────────────────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────┴─────────────────────────────────────────────────────────────────╯
```
### Query flakes
```
nix flake show --json | from json
╭────────────────┬───────────────────╮
│ defaultPackage │ {record 5 fields} │
│ packages │ {record 5 fields} │
╰────────────────┴───────────────────╯
nix flake show --json | from json | get packages
╭────────────────┬───────────────────╮
│ aarch64-darwin │ {record 2 fields} │
│ aarch64-linux │ {record 2 fields} │
│ i686-linux │ {record 2 fields} │
│ x86_64-darwin │ {record 2 fields} │
│ x86_64-linux │ {record 2 fields} │
╰────────────────┴───────────────────╯
nix flake show --json | from json | get packages.x86_64-linux
╭───────────────┬───────────────────╮
│ nix-dev-html │ {record 2 fields} │
│ nix-dev-pyenv │ {record 3 fields} │
╰───────────────┴───────────────────╯
```
### Parse a flake.lock file
```
> open flake.lock | from json | get nodes.nixpkgs.locked
╭──────────────┬─────────────────────────────────────────────────────╮
│ lastModified │ 1663494472 │
│ narHash │ sha256-fSowlaoXXWcAM8m9wA6u+eTJJtvruYHMA+Lb/tFi/qM= │
│ path │ /nix/store/iw3xi0bfszikb0dmyywp7pm590jvbqvs-source │
│ rev │ f677051b8dc0b5e2a9348941c99eea8c4b0ff28f │
│ type │ path │
╰──────────────┴─────────────────────────────────────────────────────╯
```
## OpenBSD examples
### Parse /etc/fstab
```
> open /etc/fstab | from ssv -m 1 -n | rename device mountpoint fs options freq passno
_────┬────────────────────┬─────────────────┬──────┬───────────────────────────────────────────┬──────┬────────_
│ # │ device │ mountpoint │ fs │ options │ freq │ passno │
├────┼────────────────────┼─────────────────┼──────┼───────────────────────────────────────────┼──────┼────────┤
│ 0 │ 55a6c21017f858cb.b │ none │ swap │ sw │ __ │ __ │
│ 1 │ 55a6c21017f858cb.a │ / │ ffs │ rw,noatime,softdep │ 1 │ 1 │
│ 2 │ 55a6c21017f858cb.l │ /home │ ffs │ rw,noatime,wxallowed,softdep,nodev,nosuid │ 1 │ 2 │
│ 3 │ 55a6c21017f858cb.d │ /tmp │ ffs │ rw,noatime,softdep,nodev,nosuid │ 1 │ 2 │
│ 4 │ 55a6c21017f858cb.f │ /usr │ ffs │ rw,noatime,softdep,nodev │ 1 │ 2 │
│ 5 │ 55a6c21017f858cb.g │ /usr/X11R6 │ ffs │ rw,noatime,softdep,nodev │ 1 │ 2 │
│ 6 │ 55a6c21017f858cb.h │ /usr/local │ ffs │ rw,noatime,softdep,wxallowed,nodev │ 1 │ 2 │
│ 7 │ 55a6c21017f858cb.k │ /usr/obj │ ffs │ rw,noatime,softdep,nodev,nosuid │ 1 │ 2 │
│ 8 │ 55a6c21017f858cb.j │ /usr/src │ ffs │ rw,noatime,softdep,nodev,nosuid │ 1 │ 2 │
│ 9 │ 55a6c21017f858cb.e │ /var │ ffs │ rw,noatime,softdep,nodev,nosuid │ 1 │ 2 │
│ 10 │ afebb2a83a449265.b │ /build │ ffs │ rw,noatime,softdep,wxallowed,nosuid │ 1 │ 2 │
│ 11 │ afebb2a83a449265.a │ /build/pobj │ ffs │ rw,noatime,softdep,nodev,wxallowed,nosuid │ 1 │ 2 │
│ 12 │ 55a6c21017f858cb.b │ /build/pobj_mfs │ mfs │ -s1G,wxallowed,noatime,rw │ 0 │ 0 │
╰────┴────────────────────┴─────────────────┴──────┴───────────────────────────────────────────┴──────┴────────_
```
### Parse /var/log/messages
```
open /var/log/messages | parse -r "(?<date>\\w+ \\d+ \\d+:\\d+:\\d+) (?<hostname>\\w+) (?<program>\\w+)\\[?(?<pid>\\d+)?\\]?: (?<message>.*)"
╭───┬─────────────────┬──────────┬────────────┬───────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ # │ date │ hostname │ program │ pid │ message │
├───┼─────────────────┼──────────┼────────────┼───────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 0 │ Oct 31 10:27:32 │ fx6 │ collectd │ 55258 │ uc_update: Value too old: name = fx6openbsd/swap/swap-free; value time = 1667208452.108; last cache update = 1667208452.108; │
│ 1 │ Oct 31 10:43:02 │ fx6 │ collectd │ 55258 │ uc_update: Value too old: name = fx6openbsd/swap/percent-free; value time = 1667209382.102; last cache update = 1667209382.102; │
│ 2 │ Oct 31 11:00:01 │ fx6 │ syslogd │ 4629 │ restart │
│ 3 │ Oct 31 11:05:26 │ fx6 │ pkg_delete │ │ Removed helix-22.08.1 │
│ 4 │ Oct 31 11:05:29 │ fx6 │ pkg_add │ │ Added helix-22.08.1 │
│ 5 │ Oct 31 11:16:49 │ fx6 │ pkg_add │ │ Added llvm-13.0.0p3 │
│ 6 │ Oct 31 11:20:18 │ fx6 │ pkg_add │ │ Added clang-tools-extra-13.0.0p2 │
│ 7 │ Oct 31 11:20:32 │ fx6 │ pkg_add │ │ Added bash-5.2.2 │
│ 8 │ Oct 31 11:20:34 │ fx6 │ pkg_add │ │ Added fzf-0.34.0 │
│ 9 │ Oct 31 11:21:01 │ fx6 │ pkg_delete │ │ Removed fzf-0.34.0 │
╰───┴─────────────────┴──────────┴────────────┴───────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
```
### Parse pkg_info output
```
pkg_info | str trim | parse -r "(?<package>.*?)-(?<version>[a-zA-Z0-9\\.]*?) (?<description>.*)" | str trim description
╭────┬───────────────────┬────────────┬────────────────────────────────────────────────────╮
│ # │ package │ version │ description │
├────┼───────────────────┼────────────┼────────────────────────────────────────────────────┤
│ 0 │ athn-firmware │ 1.1p4 │ firmware binary images for athn(4) driver │
│ 1 │ collectd │ 5.12.0 │ system metrics collection engine │
│ 2 │ curl │ 7.85.0 │ transfer files with FTP, HTTP, HTTPS, etc. │
│ 3 │ gettext-runtime │ 0.21p1 │ GNU gettext runtime libraries and programs │
│ 4 │ intel-firmware │ 20220809v0 │ microcode update binaries for Intel CPUs │
│ 5 │ inteldrm-firmware │ 20220913 │ firmware binary images for inteldrm(4) driver │
│ 6 │ kakoune │ 2021.11.08 │ modal code editor with a focus on interactivity │
│ 7 │ libgcrypt │ 1.10.1p0 │ crypto library based on code used in GnuPG │
│ 8 │ libgpg-error │ 1.46 │ error codes for GnuPG related software │
│ 9 │ libiconv │ 1.17 │ character set conversion library │
│ 10 │ libstatgrab │ 0.91p5 │ system statistics gathering library │
│ 11 │ libxml │ 2.10.3 │ XML parsing library │
│ 12 │ libyajl │ 2.1.0 │ small JSON library written in ANSI C │
│ 13 │ nghttp2 │ 1.50.0 │ library for HTTP/2 │
│ 14 │ nushell │ 0.70.0 │ a new kind of shell │
│ 15 │ obsdfreqd │ 1.0.3 │ userland daemon to manage CPU frequency │
│ 16 │ quirks │ 6.42 │ exceptions to pkg_add rules and cache │
│ 17 │ rsync │ 3.2.5pl0 │ mirroring/synchronization over low bandwidth links │
│ 18 │ ttyplot │ 1.4p0 │ realtime plotting utility for terminals │
│ 19 │ vmm-firmware │ 1.14.0p0 │ firmware binary images for vmm(4) driver │
│ 20 │ xz │ 5.2.7 │ LZMA compression and decompression tools │
│ 21 │ yash │ 2.52 │ POSIX-compliant command line shell │
╰────┴───────────────────┴────────────┴────────────────────────────────────────────────────╯
```
# Conclusion
Nushell is very fun, it's terribly different from regular shells, but
it comes with a powerful language and tooling. I always liked shells
because of pipes commands, allowing to construct a complex
transformation/analysis step by step, and easily inspect any step, or
be able to replace a step by another.
With nushell, it feels like I finally have a better tool to create more
reliable, robust, portable and faster command pipelines. The learning
curve didn't feel too hard, but maybe it's because I'm already used to
functional programming.