[HN Gopher] Moreutils: A collection of Unix tools that nobody th...
___________________________________________________________________
Moreutils: A collection of Unix tools that nobody thought to write
long ago
Author : escot
Score : 279 points
Date : 2022-04-15 18:16 UTC (4 hours ago)
(HTM) web link (joeyh.name)
(TXT) w3m dump (joeyh.name)
| sundarurfriend wrote:
| The "What's included" section direly needs either clearer/longer
| descriptions, or at least links to the tools' own pages (if they
| have them) where their use case and usage is explained. I've
| understood a lot more about (some of) the tools from the comments
| here than from the page - and I'd likely have skipped over these
| very useful tools if not for these comments!
| sundarurfriend wrote:
| Ok, longer descriptions from the tools' man pages:
|
| ---
|
| chronic runs a command, and arranges for its standard out and
| standard error to only be displayed if the command fails (exits
| nonzero or crashes). If the command succeeds, any extraneous
| output will be hidden.
|
| A common use for chronic is for running a cron job. Rather than
| trying to keep the command quiet, and having to deal with mails
| containing accidental output when it succeeds, and not verbose
| enough output when it fails, you can just run it verbosely
| always, and use chronic to hide the successful output.
|
| ---
|
| combine combines the lines in two files. Depending on the
| boolean operation specified, the contents will be combined in
| different ways:
|
| and Outputs lines that are in file1 if they are also present in
| file2.
|
| not Outputs lines that are in file1 but not in file2.
|
| or Outputs lines that are in file1 or file2.
|
| xor Outputs lines that are in either file1 or file2, but not in
| both files.
|
| The input files need not be sorted
|
| ---
|
| ifdata can be used to check for the existence of a network
| interface, or to get information about the interface, such as
| its IP address. Unlike ifconfig or ip, ifdata has simple to
| parse output that is designed to be easily used by a shell
| script.
|
| ---
|
| lckdo: Now that util-linux contains a similar command named
| flock, lckdo is deprecated, and will be removed from some
| future version of moreutils.
|
| ---
|
| mispipe: mispipe pipes two commands together like the shell
| does, but unlike piping in the shell, which returns the exit
| status of the last command; when using mispipe, the exit status
| of the first command is returned.
|
| Note that some shells, notably bash, do offer a pipefail
| option, however, that option does not behave the same since it
| makes a failure of any command in the pipeline be returned, not
| just the exit status of the first.
|
| ---
|
| pee: [my own description: `pee cmd1 cmd2 cmd3` takes the data
| from the standard input, sends copies of it to the commands
| cmd1, cmd2, and cmd3 (as their stdin), aggregates their outputs
| and provides that at the standard output.]
|
| ---
|
| sponge, ts and vipe have been described in other comments in
| this thread. (And I've also skipped some easier-to-understand
| ones like errno and isutf8 for the sake of length.)
|
| ---
|
| zrun: Prefixing a shell command with "zrun" causes any
| compressed files that are arguments of the command to be
| transparently uncompressed to temp files (not pipes) and the
| uncompressed files fed to the command.
|
| The following compression types are supported: gz bz2 Z xz lzma
| lzo
|
| [One super cool thing the man page mentions is that if you
| create a link named z<programname> eg. zsed, with zrun as the
| link target, then when you run `zsed XYZ`, zrun will read its
| own program name, and execute 'zrun sed XYZ' automatically.]
|
| ---
| caymanjim wrote:
| They come with good manpages if you install them. I don't
| disagree that a little more detail on the linked web page might
| help people decide whether or not to install, but in keeping
| with how these are well-behaved oldschool-style Unix commands,
| they also come with man pages, which is more than I can say for
| most command line tools people make today (the ones that assume
| you have Unicode glyphs and color terminals [and don't check if
| they're being piped or offer --no-color] and don't accept
| -h/--help and countless other things that kids these days have
| eschewed).
| caymanjim wrote:
| I was prepared to mock this before I even clicked, but I have to
| say this looks like a nice set of tools that follow the ancient
| Unix philosophy of "do one thing, play nice in a pipeline, stfu
| if you have nothing useful to say". Bookmarking this to peer at
| until I internalize the apps. There's even an Ubuntu package for
| them.
|
| I don't think it's a good idea to rely on any of these being
| present. If you write a shell script to share and expect them to
| be there, you aren't being friendly to others, but for
| interactive command line use, I'm happy to adopt new tools.
| nonrandomstring wrote:
| > Bookmarking this to peer at until I internalize the apps.
| There's even an Ubuntu package for them.
|
| Ditto. But I will probably forget they exist and go do the same
| old silly kludges with subshells and redirections. May I ask if
| anyone has a technique for introducing new CL tools into their
| everyday workflow?
|
| It helps if things have good man, apropos and help responses,
| but the problem is not _how_ new tools function, rather
| remembering that they exist at all.
|
| Sometimes I think I want a kind of terminal "clippy" that says:
|
| "Looks like you're trying to match using regular expressions,
| would you like me to fzf that for you?"
|
| Then again, maybe not.
| bcbrown wrote:
| > May I ask if anyone has a technique for introducing new CL
| tools into their everyday workflow?
|
| Pick one tool a year. Write it on a sticky note on your
| monitor. Every time you're doing something on the command
| line, ask yourself "would $TOOL be useful here?".
|
| You're not going to have high throughput on learning new
| tools this way, but it'll be pretty effective for the tools
| you do learn.
| chrisshroba wrote:
| My two cents on this is that if you do something enough that
| one of these tools is a good tool for it, it'll quickly
| become a habit to use the tool. And if there's a rare case
| where one of these tools would have been useful but you
| forgot it existed, you're probably not wasting _too_ much
| time using a hackier solution.
|
| That being said, I've been meaning to add a Linux and Mac
| install.sh script to my dotfiles repo for installing all my
| CLI tools, and that could probably serve as a good reminder
| of all the tools you've come across over the years that might
| provide some value.
| _dain_ wrote:
| >I don't think it's a good idea to rely on any of these being
| present. If you write a shell script to share and expect them
| to be there, you aren't being friendly to others, but for
| interactive command line use, I'm happy to adopt new tools.
|
| Isn't that a shame though? Where does it say in the UNIX
| philosophy that the canon should be closed?
| Delk wrote:
| It's not that different than refraining from using non-POSIX
| syntax in shell scripts that are meant to be independent of a
| specific flavour of unix, or sticking with standard C rather
| than making assumptions that are only valid in one compiler.
|
| There are shades of grey, of course. Bash is probably
| ubiquitous enough that it may not be a big issue if a script
| that's meant to be universal depends on it, as long as the
| script explicitly specifies bash in the shebang. Sometimes
| some particular functionality is not technically part of a
| standard but is widely enough supported in practice.
| Sometimes the standards (either formal or de facto) are
| expanded to include new functionality, and that's of course
| totally fine, but it's not likely to be a very quick process
| because there are almost certainly going to be differing
| opinions on what should be part of the core and what
| shouldn't.
|
| Either way, sometimes you want to write for the lowest common
| denominator, and moreutils certainly aren't common enough
| that they could be considered part of that.
| caymanjim wrote:
| I think it's fine if you're on a dev team that decides to
| include these tools in its shared toolkit, but none of these
| rise to the level that I think warrants them being a
| dependency for a broadly-distributed shell script. There are
| slightly-less-terse alternatives for most of the
| functionality that only rely on core utilities. I don't think
| it's being a good citizen to say "go install moreutils and
| its dozen components because I wanted to use sponge instead
| of >output.txt".
| pjungwir wrote:
| These are necessarily bash functions, not executables, but here
| are two tools I'm proud of, which seem similar in spirit to vidir
| & vipe: # Launch $EDITOR to let you edit your
| env vars. function viset() { if [ -z "$1"
| ]; then echo "USAGE: viset THE_ENV_VAR"
| exit 1 else declare -n ref=$1
| f=$(mktemp) echo ${!1} > $f $EDITOR $f
| ref=`cat $f` export $1 fi }
| # Like viset, but breaks up the var on : first, # then
| puts it back together after you're done editing. #
| Defaults to editing PATH. # # TODO: Accept a
| -d/--delimiter option to use something besides :.
| function vipath() { varname="${1:-PATH}"
| declare -n ref=$varname f=$(mktemp) echo
| ${!varname} | tr : "\n" > $f $EDITOR $f
| ref=`tr "\n" : < $f` export $varname }
|
| Mostly I use vipath because I'm too lazy to figure out why tmux
| makes rvm so angry. . . .
|
| I guess a cool addition to viset would be to accept more than one
| envvar, and show them on multiple lines. Maybe even let you edit
| your entire env if you give it zero args. Having autocomplete-on-
| tab for viset would be cool too. Maybe even let it interpret
| globs so you can say `viset AWS*`.
|
| Btw I notice I'm not checking for an empty $EDITOR. That seems
| like it could be a problem somewhere.
| I_complete_me wrote:
| I only install moreutils for vidir which is simply brilliant IMO
| teddyh wrote:
| I prefer Emacs dired, where pressing C-x C-q starts editing the
| opened directory (including any inserted subdirectories).
| [deleted]
| VTimofeenko wrote:
| One alias I always do is "vidir" -> "vidir --verbose" so that
| it would tell me what it's doing
| zzo38computer wrote:
| I have this installed and have used some of these sometimes; it
| is good. (I do not use all of them, though)
| WalterBright wrote:
| > errno: look up errno names and descriptions
|
| I like this. Reminds me of a couple very useful things I've done:
|
| 1. Add a -man switch to command line programs. This causes a
| browser to be opened on the web page for the program. For
| example: dmd -man
|
| opens https://dlang.org/dmd-windows.html in your default browser.
|
| 2. Fix my text editor to recognize URLs, and when clicking on the
| URL, open a browser on it. This silly little thing is amazingly
| useful. I used to keep bookmarks in an html file which I would
| bring up in a browser and then click on the bookmarks. It's so
| much easier to just put them in a plain text file as plain text.
| I also use it for source code, for example the header for code
| files starts with: /* * Takes a token
| stream from the lexer, and parses it into an abstract syntax
| tree. * * Specification: $(LINK2
| https://dlang.org/spec/grammar.html, D Grammar) *
| * Copyright: Copyright (C) 1999-2020 by The D Language
| Foundation, All Rights Reserved * Authors: $(LINK2
| http://www.digitalmars.com, Walter Bright) * License:
| $(LINK2 http://www.boost.org/LICENSE_1_0.txt, Boost License 1.0)
| * Source: $(LINK2
| https://github.com/dlang/dmd/blob/master/src/dmd/parse.d,
| _parse.d) * Documentation:
| https://dlang.org/phobos/dmd_parse.html * Coverage:
| https://codecov.io/gh/dlang/dmd/src/master/src/dmd/parse.d
| */
|
| and I'll also use URLs in the source code to reference the spec
| on what the code is implementing, and to refer to closed bug
| reports that the code fixes.
|
| Very, very handy!
| HellsMaddy wrote:
| I find sponge really useful. Have you ever wanted to process a
| file through a pipeline and write the output back to the file? If
| you do it the naive way, it won't work, and you'll end up losing
| the contents of your file: awk '{do_stuff()}'
| myfile.txt | sort -u | column --table > myfile.txt
|
| The shell will truncate myfile.txt before awk (or whatever
| command) has a chance to read it. So you can use sponge instead,
| which waits until it reads EOF to truncate/write the file.
| awk '{do_stuff()}' myfile.txt | sort -u | column --table | sponge
| myfile.txt
| mzs wrote:
| If you're using awk anyway: awk
| 'BEGIN{system("rm myfile.txt")}{do_stuff()}' <myfile.txt | sort
| -u | column --table > myfile.txt
|
| In general though: <foo { rm -f foo && wc >foo
| }
| jwilk wrote:
| Your awk pipeline is racy.
| indigodaddy wrote:
| Never heard of sponge. Isn't it just the same as tee ?
| halostatue wrote:
| It's part of the moreutils collection and is better
| considered a buffered redirect than anything similar to
| `tee`.
| indigodaddy wrote:
| Ah yes I glanced too quickly over the surface here. It does
| look more like redirection. Will have to look at it more.
| Appreciate your helpful response vs the downvoters and the
| one unhelpful/snarky response.
| AshamedCaptain wrote:
| The point is not to truncate the file immediately when it is
| open for output, and before the input side has had time to
| slurp it.
| binwiederhier wrote:
| I urge you to actually read the link before commenting.
| sponge is the front and center example of the link you are
| commenting on.
| indigodaddy wrote:
| The downvoting here is the equivalent of getting shamed for
| "asking a stupid question at work."
|
| Yes I should have done a bit more homework but shaming for
| asking a clarifying question is unreasonable. Those of you
| who have the downvote trigger-finger can and should do
| better.
| HellsMaddy wrote:
| I agree with you, the downvotes are unnecessary, it was
| actually a good question.
|
| tee actually does sorta work for this sometimes, but it's
| not guaranteed to wait until EOF. For example I tested with
| a 10 line file where I ran `sort -u file.txt | tee
| file.txt` and it worked fine. But I then tried a large json
| file `jq . large.json | tee large.json` and the file was
| truncated before jq finished reading it.
| thanatos519 wrote:
| 'sort -o file' does the same thing but I like the generic
| 'sponge'. Not sure why it's a binary, since it's basically this
| shell fragment (if you don't bother checking for errors etc):
|
| (cat > $OUT.tmp; mv -f $OUT.tmp $OUT)
|
| Hmmm ... "When possible, sponge creates or updates the output
| file atomically by renaming a temp file into place. (This
| cannot be done if TMPDIR is not in the same filesystem.)"
|
| My shell fragment already beats sponge on this feature!
| caymanjim wrote:
| I can see how this is handy, but it's also dangerous and likely
| to bite you in the ass more than not. I think sponge is great,
| but I think your example is dangerous. If you make a mistake
| along the way, unless it's a fatal error, you're going to lose
| your source data. Typo a grep and you're screwed.
| justsomehnguy wrote:
| Agree.
|
| Sure there could be some situations where it could be handy,
| like in some auomated scenarios, but most of the time it is
| not a big deal to write foo data1 data2
| bar data2
| HellsMaddy wrote:
| Right. You should always test the command first. If the data
| is critical, use a temporary file instead. I usually use this
| in scripts so I don't have to deal with cleanup.
| sedatk wrote:
| > If the data is critical, use a temporary file instead
|
| Use a temporary file always. Sponge process may be
| interrupted, and you end up with a half-complete
| /etc/passwd in return.
| rcoveson wrote:
| Couldn't `mv` or `cp` from the temp file to `/etc/passwd`
| be interrupted as well? I think the only way to do it
| atomically is a temporary file on the same filesystem as
| `/etc`, followed by a rename. On most systems `/tmp` will
| be a different filesystem from `/etc`.
| AdamJacobMuller wrote:
| mv can't, or, more correctly the rename system call can
| not.
|
| rename is an atomic operation from any modern
| filesystem's perspective, you're not writing new data,
| you're simply changing the name of the existing file, it
| either succeeds or fails.
|
| Keep in mind that if you're doing this, mv (the command
| line tool) as opposed to the `rename` system call, falls
| back to copying if the source and destination files are
| on different filesystems since you can not really mv a
| file across filesystems!
|
| In order to have truly atomic writes you need to:
|
| open a new file on the same filesystem as your
| destination file
|
| write contents
|
| call fsync
|
| call rename
|
| call sync (if you care about the file rename itself never
| being reverted).
|
| This is some very naive golang code (from when I barely
| knew golang) for doing this which has been running in
| production since I wrote it without a single issue: https
| ://github.com/AdamJacobMuller/atomicxt/blob/master/file..
| .
| JJMcJ wrote:
| Not clear on need for fsync and sync.
|
| Are those for networked like NTFS or just as security
| against crashes.
|
| Logically on a single system there would be no effect
| assuming error free filesystem operation. Unless I'm
| missing something.
| antihero wrote:
| Why not write it to a different file
| Quekid5 wrote:
| This is why I usually just use a temporary directory and do
| a quick git init . git add .
| git commit -m "wip"
|
| ... and proceed from there. So many ways to screw up ad hoc
| data processing using shell and the above can be a life
| saver. (Along with committing along the way, ofc.)
|
| EDIT: Doesn't work if you have huuuuge files, obviously...
| but you should perhaps be using different tools for that
| anyway.
| chungy wrote:
| You might like to try using src, a simple single-file VC:
| http://www.catb.org/esr/src/
| uuyi wrote:
| Been there done that. Good advice.
|
| In my case I could recreate the original file but it took 90
| minutes to scrape the remote API to do it again...
| dan-robertson wrote:
| I usually used tac | tac for a stupid way of pausing in a
| pipeline. Though it doesn't work in this case. A typical use is
| if you want to watch the output of something slow-to-run
| changing but watch doesn't work for some reason, eg:
| while : ; do ( tput reset run-slow-command )
| | tac | tac done
| alerighi wrote:
| It would also be useful in cases when the file must be written
| with root privileges (but the shell is run as a normal user)
|
| Now you have to use `tee` for that, that is fine but if you
| don't want to echo the file back to the terminal you have to do
| command | sudo tee file > /dev/null
|
| with this you can simply do command | sudo
| sponge file
| leni536 wrote:
| Can't you just use `cat`?
| jaden wrote:
| I thought the same thing. What's wrong with this approach?
| cat myfile.txt | awk '{do_stuff()}' | sort -u | column
| --table > new-myfile.txt
| HellsMaddy wrote:
| cat is doing nothing useful here. Your command is
| equivalent to awk '{do_stuff()}'
| myfile.txt | sort -u | column --table > new-myfile.txt
|
| The only thing you've changed is that you're sending the
| output to a new file. That's fine, but it's what sponge
| is avoiding.
| ben0x539 wrote:
| You cannot use `sudo cat` to open a file with root
| privileges, because `sudo cat > foo` means "open file foo
| with your current privileges, then run `sudo cat` passing
| the file to it", and the whole root thing only happens
| after you already tried and failed to open the file.
|
| I seem to get this wrong roughly once a week.
| gorgoiler wrote:
| ... | sudo tee foo
| readme wrote:
| sudo sh -c "cat > file"
| HellsMaddy wrote:
| Cat won't write the file. If you mean `command | sudo cat >
| file.txt`, that won't work because the redirection is still
| happening in the non-root shell. You could do `command |
| sudo sh -c "cat > file.txt"` but that's rather verbose.
| justinsaccount wrote:
| protip: don't run this awk '{do_stuff()}'
| myfile.txt | sort -u | column --table | sponge > myfile.txt
| jamespwilliams wrote:
| Are there any legitimate reasons to have a particular file both
| as an input to a pipe and as an output? I wonder whether a
| shell could automatically "sponge" the pipe's output if it
| detected that happening.
| mjochim wrote:
| sed --in-place has one file as both input and output. It's
| not really different from any pipe of commands where the
| input and output files are the same. But sed also makes a
| copy of the file before overwriting it - per default.
| teawrecks wrote:
| Yeah, it seems like the kind of command that you only need
| because of a quirk in how the underlying system happens to
| work. Not something that should pollute the logic of the
| command, imo. I would expect a copy-on-write filesystem to be
| able to do this automatically for free.
| caymanjim wrote:
| Do people really use copy-on-write filesystems though? I
| mean it'd be great if that were a default, but I rarely
| encounter them, and when I do, it's only because someone
| intentionally set it up that way. In 30+ years of using
| Unix systems, I can't even definitively recall one of them
| having a copy-on-write filesystem in place. Which is insane
| considering I used VAX/VMS systems before that and it was
| standard there.
| r3trohack3r wrote:
| FreeBSD uses ZFS by default, which is copy on write post-
| snapshot.
| mjochim wrote:
| Huh? Btrfs is copy on write and it's definitely being
| used.
| paulmd wrote:
| > I would expect a copy-on-write filesystem to be able to
| do this automatically for free.
|
| this is an artifact of how handles work (in relation to
| concurrency), not the filesystem.
|
| copy-on-write still guarantees a consistent view of the
| data, so if you write on one handle you're going to clobber
| the data on the other, _because that 's what's in the
| file_.
|
| what you really want is an operator which says "I want this
| handle to point to the original snapshot of this data even
| if it's changed in the meantime", which a CoW filesystem
| _could_ do, but you 'd need some additional semantics here
| (different access-mode flag?) which isn't trivially granted
| just by using a CoW filesystem underneath.
| Arnavion wrote:
| The shell can't detect that unless the file is also used as
| input via `<`. So it couldn't do that in HellsMaddy's
| example, since the filename is given as an arg to awk instead
| of being connected to its stdin.
| RulerOf wrote:
| >I find sponge really useful.
|
| When I found `sponge`, I couldn't help but wonder where it had
| been all of my life. It's nice to be able to modify an in-place
| file with something other than `sed`.
| JoshTriplett wrote:
| I use a few of these regularly:
|
| ts timestamps each line of the input, which I've found convenient
| for ad-hoc first-pass profiling in combination with verbose print
| statements in code: the timestamps make it easy to see where long
| delays occur.
|
| errno is a _great_ reference tool, to look up error numbers by
| name or error names by number. I use this for two purposes.
| First, for debugging again, when you get an errno numerically and
| want to know which one it was. And second, to search for the
| right errno code to return, in the list shown by errno -l.
|
| And finally, vipe is convenient for quick one-off pipelines,
| where you know you need to tweak the input at one point in the
| pipeline but you know the nature of the tweak would take less
| time to do in an editor than to write the appropriate tool
| invocation to do it.
| ducktective wrote:
| I actually installed moreutils just for errno, but I got
| disappointed. Here it the whole `errno -l` : http://ix.io/3VeV
|
| Like, none of the everyday tools I use produce exit codes which
| correspond to these explanations.
|
| For my scripts, I just return 1 for generic erros and 2 for bad
| usage. I wished I could be more specific in that and adhered to
| _some standard_.
| JoshTriplett wrote:
| errno codes aren't used in the exit codes of command-line
| tools; they're used in programmatic APIs like syscalls and
| libc functions.
|
| There's no standard for process exit codes, other than "zero
| for success, non-zero for error".
| zzo38computer wrote:
| Some programs do follow a specification (sysexits.h) where
| numbers 64 and higher are used for common uses, such as
| usage error, software error, file format error, etc.
| ithkuil wrote:
| Aldo when the oomkiller process kills another process it
| causes it to exit with a well known exit code: 137
| saagarjha wrote:
| That's just 128+SIGKILL (9).
| teddyh wrote:
| That's not an exit code; that is a fake exit code which
| your shell made up, since your shell has no other way to
| tell you that a process did not exit at all, but was
| killed by a signal (9, i.e. SIGKILL). This is, again, not
| an actual exit code, since the process did not actually
| exit.
|
| See here for why you can't use an exit status of
| 128+signal to fake a "killed by signal" state, either:
|
| https://www.cons.org/cracauer/sigint.html
| teddyh wrote:
| That is not entirely true. There _is_ sysexits.h, which
| tried to standardize some exit codes (originally meant for
| mail filters). It is used by, for instance, argp_parse().
|
| https://sourceware.org/git/?p=glibc.git;a=blob;f=misc/sysex
| i...
| drewg123 wrote:
| Its a shame errno is even needed, and is one of my pet peeves
| about linux.
|
| On linux, errno.h is fragmented across several places because
| errnos are _different_ on different architectures. I think this
| started because when Linus did the initial alpha port, he
| bootstrapped Linux using a DEC OSF /1 userland, which meant
| that he had to use the DEC OSF/1 BSD-derived values of errno
| rather than the native linux ones so that they would run
| properly. I'm not sure why this wasn't cleaned up before it
| made it into the linux API on alpha.
|
| At least on FreeBSD, determining what errno means what is just
| a grep in /usr/include/sys/errno.h. And it uses different
| errnos for different ABIs (eg, linux binaries get their normal
| errnos, not the FreeBSD ones).
| tedunangst wrote:
| Freebsd doesn't have man errno??
| hgomersall wrote:
| Are they written in rust? If not, it's just more programmes
| someone is going you have to rewrite at some point _sigh_.
| jedberg wrote:
| My favorite missing tool of all time is `ack` [0]. It's grep if
| grep were made now. I use it all the time, and it's the first
| thing I install on a new system.
|
| It has a basic understanding of common text file structures and
| also directories, which makes it super powerful.
|
| [0] https://beyondgrep.com
| renewiltord wrote:
| ripgrep is a modern written-in-Rust[0] equivalent that I really
| like.
|
| 0: I like this because it's much easier to edit IMHO
| outworlder wrote:
| What about the Silver Searcher? It's even faster than ack
|
| https://github.com/ggreer/the_silver_searcher
| makeworld wrote:
| What about ripgrep? It's even faster than the Silver Searcher
|
| https://github.com/BurntSushi/ripgrep
| jedberg wrote:
| Had never heard of it before, but I'll definitely check it
| out now. Thanks!
| usr1106 wrote:
| I have searched ts for a long time. Used it many years ago but
| forgot the exact name of the tool and package. No search machine
| could find it or I just entered to wrong search words.
| topher200 wrote:
| The one I use the most is `vipe`.
|
| > vipe: insert a text editor into a pipe
|
| It's useful when you want to edit some input text before passing
| it to a different function. For example, if I want to delete many
| of my git branches (but not all my git branches):
| $ git branch | vipe | xargs git branch -D
|
| `vipe` will let me remove some of the branch names from the list,
| before they get passed to the delete command.
| LukeShu wrote:
| I vaguely recall in ~2010 coming across a Plan 9 manpage(?)
| that seemed to imply that Emacs could work that way in a pipe
| (in reference to complex/bloated tools on non-Plan 9 systems),
| but that wasn't true of any version of Emacs I'd ever used.
| e40 wrote:
| Yep, set EDITOR to emacsclient and vipe will use emacs.
| unkulunkulu wrote:
| I mostly rely on fzf for stuff like this nowadays. You can
| replace vipe with fzf --multi for example and get non-inverted
| behavior with fuzzy search.
|
| More to it, not in a pipe (because of poor ^C behavior), but
| using a hokey in zsh to bring up git branch | fzf, select any
| number of branches I need and put them on command line, this is
| extremely composable.
| bombcar wrote:
| This is useful - I usually either use an intermediate file or a
| bunch of grep -v
| orblivion wrote:
| And what if you decide mid-vipe "oh crap I don't want to do
| this anymore"? In the case of branches to delete you could just
| delete every line. In other cases maybe not?
| goodside wrote:
| To be fair, that scenario would be just as bad or worse
| without vipe.
|
| Also, you can construct your pipeline so that a blank file
| (or some sentinel value) returned from vipe means "abort". A
| good example of this is when git opens your editor for
| interactive merging -- deleting all lines cancels the whole
| thing.
| orblivion wrote:
| Yeah you could just as well say "oh crap I didn't mean to
| do that" after finishing a non-interactive command.
| However, at least knowing my own lazy tendencies, I could
| imagine feeling comfortable hitting <enter> on this command
| without a final careful review, because part of me thinks
| that I can still back out, since the interactive part isn't
| finished yet.
|
| But maybe not. I haven't tried it yet (and it does seem
| really useful).
| saagarjha wrote:
| Send a SIGTERM to the editor, maybe?
| Beltalowda wrote:
| And there's always the power button on your computer :-)
| orblivion wrote:
| Kind of ugly, but yeah, that's what I'd imagine doing.
| throwaway09223 wrote:
| It will depend on the commands in question. The entire unix
| pipeline is instantiated in parallel, so the commands
| following vipe will already be running and waiting on stdin.
|
| You could kill them before exiting the editor, if that's what
| you want. Or you could do something else.
|
| The other commands in the pipeline are run by the parent
| shell, not vipe, so handling this would not be vipe specific.
| delusional wrote:
| Depends on your editor i suppose, but vipe checks the exit
| code of it [1]. In vim you can exit with an error using :cq
|
| [1]: https://github.com/pgdr/moreutils/blob/f4a811d0a1fafb3d7
| b0e7...
| foresto wrote:
| Exiting vim with :cq is also handy for backing out of git
| commits.
| _joel wrote:
| Working with *nix for over 25 years and I've only just heard of
| sponge.
| PaulDavisThe1st wrote:
| The need for ifdata(1) has become even more acute with the
| essential replacement of ifconfig(1) by ip(1), an even more
| inscrutable memory challange. However, it would be even nicer if
| its default action when not given an interface name was do
| <whatever> for all discovered interfaces.
| teddyh wrote:
| Do not "ip -brief address show" and "ip -brief link show" serve
| as suitable replacements for most common uses of ifdata(1)? The
| ip(8) command even supports JSON output using "-json" instead
| of "-brief".
| zwayhowder wrote:
| TIL. Thanks.
|
| One does have to wonder though, why isn't -brief the default
| and the current default set to -verbose or -long. I look at
| -brief on either command and it has all the information I am
| ever looking for.
| Filligree wrote:
| Yes, I suppose, but I can never remember those flags.
| wang_li wrote:
| Amazing how many new things come about because people don't
| know or don't remember how to use the existing things.
| teddyh wrote:
| ifdata is not meant for interactive use.
| hedora wrote:
| Well, they're strictly worse for interactive use than
| ifconfig, from what I can tell from your comment.
| teddyh wrote:
| We were discussing ifdata, not ifconfig. From the
| documentation, ifdata is explicitly meant for use in
| scripts. And in scripts, using "ip -brief" or "ip -json ...
| | jq ..." may well be suitable replacements for ifdata.
| PaulDavisThe1st wrote:
| Sure, they do.
|
| In fact, I take it back. ifdata(1) is not in any way a
| replacement for ifconfig(1) for most things. The problem is
| that just running ifconfig with no arguments showed you
| everything, which was generally perfect for interactive use.
| Now to get _any_ information from ip(1) you have to remember
| an argument name. If you do this a lot, it 's almost
| certainly fine. If you do it occasionally, it's horrible.
| ChrisGranger wrote:
| It's been a _very_ long time since this happened, but in my early
| days of using Linux, I experienced naming collisions with both
| sponge and parallel, and at the time I didn 't know how to
| resolve them. I don't remember which other sponge there was, but
| I imagine most Linux users are familiar with GNU parallel at this
| point.
| JulianWasTaken wrote:
| moreutils indeed has some great utils, but a minor annoyance it
| causes is still shipping a `parallel` tool which is relatively
| useless, but causes confusion for new users or conflict (for
| package managers) with the way way way more indispensable GNU
| parallel.
| yesenadam wrote:
| When I installed moreutils v0.67 with macports just now it
| said: moreutils has the following notes:
| The binary parallel is no longer in this port; please install
| the port parallel instead.
|
| i.e. GNU parallel
| omoikane wrote:
| `parallel` seems redundant because it appears that `xargs -P`
| can accomplish the same effect, except the "-l maxload" option.
| Davertron wrote:
| So just today I was wondering if there was a cli tool (or maybe a
| clever use of existing tools...) that could watch the output of
| one command for a certain string, parse bits of that out, and
| then execute another command with that parsed bit as input. For
| example, I have a command I run that spits out a log line with a
| url on it, I need to usually manually copy out that url and then
| paste it as an arg to my other command. There are other times
| when I simply want to wait for something to start up (you'll
| usually get a line like "Dev server started on port 8080") and
| then execute another command.
|
| I know that I could obviously grep the output of the first
| command, and then use sed or awk to manipulate the line I want to
| get just the url, but I'm not sure about the best way to go about
| the rest. In addition, I usually want to see all the output of
| the first command (in this case, it's not done executing, it
| continues to run after printing out the url), so maybe there's a
| way to do that with tee? But I usually ALSO don't want to
| intermix 2 commands in the same shell, i.e. I don't want to just
| have a big series of pipes, Ideally I could run the 2 commands
| separately in their own terminals but the 2nd command that needs
| the url would effectively block until it received the url output
| from the first command. I have a feeling maybe you could do this
| with named pipes or something but that's pretty far out of my
| league...would love to hear if this is something other folks have
| done or have a need for.
| ufo wrote:
| A named pipe sounds like a good way to fulfill your requirement
| of having the command runs on separate shells.. In the first
| terminal, shove the output of commend A into the named pipe. In
| the second terminal, have a loop that reads from the named pipe
| line by line and invokes command B with the appropriate
| arguments.
|
| You can create a named pipe using "mkfifo", which creates a
| pipe "file" with the specified name. Then, you can tell your
| programs to read and write to the pipe the same way you'd tell
| them to read and write from a normal file. You can use "<" and
| ">" to redirect stdout/stderr, or you can pass the file name if
| it's a program that expects a file name.
| nix0n wrote:
| There's a tool for this type of thing called Expect[0], written
| in TCL.
|
| [0] https://core.tcl-lang.org/expect/index
| abbeyj wrote:
| In one terminal, run: $ mkfifo myfifo $
| while true; do sed -rune 's/^Dev server started on port
| (.*)/\1/p' myfifo | xargs -n1 -I{} echo "Execute other command
| here with argument {}"; done
|
| In the other terminal, run your server and tee the output to
| the fifo you just created: $ start_server | tee
| myfifo
| sillysaurusx wrote:
| Thanks for this!
| teddyh wrote:
| I'd solve your exact problem like this:
|
| 1. Run one command with output to a file, possibly in the
| background. Since you want to watch the output, run "tail
| --follow=name filename.log".
|
| 2. In a second terminal, run a second tail --follow on the same
| log file but pipe the output to a command sequence to find and
| extract the URL, and then pipe _that_ into a shell while loop;
| something like "while read -r url; do do-thing-with "$url";
| done".
| sillysaurusx wrote:
| I used Python subprocess module for this.
|
| ... good luck, is my best advice. It's not straightforward to
| handle edge cases.
| marcodiego wrote:
| Long time ago, a colleague of mine created "evenmoreutils" :
| https://github.com/rudymatela/evenmoreutils
| aendruk wrote:
| See also the related announcement [1] on the most recent
| "Volunteer Responsibility Amnesty Day" [2].
|
| [1]:
| https://joeyh.name/blog/entry/Volunteer_Responsibility_Amnes...
|
| [2]: https://www.volunteeramnestyday.net/
| mc4ndr3 wrote:
| How does sponge compare with tee?
| mftb wrote:
| I have used vidir from this collection quite a bit. If you're a
| vi person, it's makes it quite convenient to use vi/vim for
| renaming whole directories full of files.
| tails4e wrote:
| I wrote a utility like ts before, called it teetime, was thrilled
| with my pun. It was quiteand useful when piping stdout from a
| compute heavy tool (multi hour EDA tool run) as you could see by
| the delta time between logs what the most time consuming parts
| were.
| escot wrote:
| Using `vipe` you can do things like: $ pbpaste |
| vipe | pbcopy
|
| Which will open your editor so you can edit whatever is in your
| clipboard.
| IshKebab wrote:
| Just realised what the pb stands for. Did they really not think
| of clipboard? Who came up with the "clipboard" name?
| Delk wrote:
| Wikipedia says "clipboard" was coined by Larry Tesler (in the
| 70's?)
| zarzavat wrote:
| The pasteboard, like many things in OS X, is from NextStep.
| As for why they called it a pasteboard and not a clipboard, I
| have no idea, presumably someone thought it would be more
| descriptive.
| [deleted]
| BoneZone wrote:
| I chuckled at pee
| theandrewbailey wrote:
| I outright laughed that its juxtaposed with sponge.
| jchw wrote:
| How might one use sponge in a way that shell redirection wouldn't
| be more fully-featured? The best I can currently think of is that
| it's less cumbersome to wrap (for things like sudo.)
| JoshTriplett wrote:
| Sponge exists for cases where shell redirection wouldn't work,
| namely where you want the source and sink to be the same file.
| If you write: somecmd < somefile | othercmd |
| anothercmd > somefile
|
| the output redirection will truncate the file before it can get
| read as input.
|
| Sponge "soaks up" all the output before writing any of it, so
| that you can write pipelines like that:
| somecmd < somefile | othercmd | anothercmd | sponge somefile
| jchw wrote:
| Thank you, I missed this bit of nuance. That indeed would be
| useful, and now the example makes a lot more sense.
| scbrg wrote:
| This _can_ be done with regular shell redirection, even
| though I wouldn 't recommend it. Easy to get wrong, and
| fairly opaque: $ cat foo foo
| bar baz $ ( rm foo && grep ba > foo ) < foo
| $ cat foo bar baz $
| figital wrote:
| here's a little wrapper around i made around "find" which i
| always have to install on every new box i manage ....
|
| https://github.com/figital/fstring
|
| (just shows you more useful details about what is found)
| eichin wrote:
| heh. I use `chronic` all the time, `ifdata` in some scripts that
| predate Linux switching to `ip`. I occasionally use `sponge` for
| things but it's almost always an alternative to doing something
| correctly :-)
|
| Looking at the other comments, I suspect one of the difficulties
| in finding a new maintainer will be that lots of people use 2 or
| 3 commands from it, but nobody uses the _same_ 2 or 3, and
| actually caring about all of them is a big stretch...
| suprjami wrote:
| Have loved this collection for a long time. I use errno almost
| daily.
| moralestapia wrote:
| >pee: tee standard input to pipes
|
| Nice tool, great name.
| theteapot wrote:
| Some of this looks useful, but a non exhaustive critique - from
| non expert - of some of the rest:
|
| > chronic: runs a command quietly unless it fails
|
| Isn't that just `command >/dev/null`?
|
| > ifdata: get network interface info without parsing ifconfig
| output
|
| `ip link show <if>`?
|
| > isutf8: check if a file or standard input is utf-8
|
| `file` for files. For stdin, when is it not utf8 - unless you've
| got some weird system configuration?
|
| > lckdo: execute a program with a lock held
|
| `flock`?
| compsciphd wrote:
| > > chronic: runs a command quietly unless it fails
|
| > Isn't that just `command >/dev/null`?
|
| no. that just shows your stderr, not stdout if it failed. and
| you get stderr even if it doesn't fail.
| ninkendo wrote:
| > > chronic: runs a command quietly unless it fails > Isn't
| that just `command >/dev/null`?
|
| Often times you want to run a command silently (like in a build
| script), but if it fails with a nonzero exit status, you want
| to then display not only the stderr but the stdout as well.
| I've written makefile hacks in the past that do this to silence
| overly-chatty compilers where we don't really care what it's
| outputting unless it fails, in which case we want all the
| output. It would've been nice to have this tool at the time to
| avoid reinventing it.
| arendtio wrote:
| I guess that chronic doesn't just show stderr, but also stdout
| if the command fails. If I am not mistaken, your example would
| hide the stdout output, even when the command fails.
___________________________________________________________________
(page generated 2022-04-15 23:00 UTC)