[HN Gopher] Watchman: Execute a command when something changes
___________________________________________________________________
Watchman: Execute a command when something changes
Author : ColinWright
Score : 200 points
Date : 2022-10-05 12:50 UTC (10 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| majodev wrote:
| Very happy user of https://github.com/watchexec/watchexec (rust),
| can recommend.
| maximilianroos wrote:
| Very much agree. And the maintainer has been super responsive.
| loevborg wrote:
| Yes, this is the one I reach for always. It works reliably
| AFAICT, runs on all platforms, is simple enough and still has
| the necessary features.
| st0le wrote:
| Is there a similar tool that can trigger a command when the
| output of command changes or the stdout has some kind of string?
|
| I understand this is shell script territory, but a cross-platform
| tool would be ideal.
| pletnes wrote:
| If you like python or work on mac/win/linux, you might like
| https://github.com/samuelcolvin/watchfiles
|
| Uses rust for performance, and is usable from the terminal or
| from python.
| amelius wrote:
| From my experience, these types of notification handlers are very
| fragile and tend to fail every so often.
|
| Do not use if your life depends on it.
| remram wrote:
| From my experience, they break _every time_ if you are making
| changes with tools that replace the file rather than change it
| in-place, like most text editors.
| Brian_K_White wrote:
| At least with inotify, ie, incron, this is just a matter of
| setting a less stupid watch.
|
| You can ask the kernel to watch for several different
| specific kinds of fs events, either at the file or directory
| level.
|
| Detecting when a file is either created, or closed from
| writing, is no problem at all. Ie, handles the case of
| writing a file normally, editing a file, and renaming a file
| into exitence without having opened it for writing (that
| happened while it had some other name you weren't watching).
| This would handle unlinking and recreating the same as
| editing or uploading.
|
| You can watch specifically for the close event only, and even
| more specifically for close-from-write-mode to detect updates
| but ignore until the update is done. And at the same time you
| can also watch for creation, which handles creating a file
| under some other name and then renaming it into existence
| under the name you care about without the name you care about
| ever having a close-from-write event.
|
| That would fire off your script when a filename you were
| watching was unlinked and re-created instead of edited in-
| place, and then it's up to your script to handle the case
| where it's a "new" file that is really just replacing an old
| file.
|
| I don't even see why that should be any kind of special
| problem. Are some tools doing something like maintaining ooen
| file handles or something for every watched file??? That
| would break by unlink-create, but that would be insane.
|
| Perhaps these wrappers that try to simplify things are as
| usual just a way to break the thing they are trying to
| simplify. Just misguided crap.
|
| If the feature is baked into an ide or media server and
| failing to handle that case, well they don't _have_ to fail
| to handle that case. The subsystem totally supplies the
| functionality.
|
| For something random adhoc where you're writing your own
| incrontab and handler script, it's no problem at all.
| diffxx wrote:
| Yeah, this is difficult, though not impossible, to get right
| with the inotify/kqueue api. Windows and mac do provide apis
| for recursive directory monitoring but a cross platform tool
| will have to solve this problem. At a high level, the way to
| do it is to create an in memory representation of the file
| system that caches a watch handle for every file. When a
| deletion of a file is detected, you must create a watch on
| the parent directory, if there wasn't one already. Then you
| should be able to detect the ensuing create. To make this
| more concrete, the problem that a file watcher needs to solve
| is the problem of keeping its in memory representation of the
| file system consistent with the actual file system. Watch
| events are just a useful side effect of this process.
|
| The other fun part is that there is often a lag between the
| deletion and the create in the text editor case so it is
| necessary to defer triggering events when a deletion is
| detected and wait a little while to see if there is a
| corresponding creation. Otherwise you may rerun your command
| that depends on the file before it exists.
|
| It is possible to get like a 99+% solution to this problem
| without polling but it is a lot harder than what these simple
| tools, including entr do. The upshot is that file monitoring
| should be looked at as a lowest common denominator solution.
| A better solution is to build automatic command running into
| the text editor itself.
| amelius wrote:
| In that case, you might want to monitor the directory too,
| not just the file. I believe this is even how it is supposed
| to work.
| remram wrote:
| Yeah, and then filter out the events in the directory to
| find out when the correct file changed, etc. That is
| exactly what I would want the tool to abstract for me.
| dabears wrote:
| Inotify can drop events when there is a burst of file changes.
| You can bump up the max number of queued events though. For
| example, if you're watching a git repo and switch to an old
| branch that has 1500+ changed files then inotify will likely miss
| some with a default configuration.
| sproketboy wrote:
| stoplying1 wrote:
| Watchman works great, but the CLI UX is... really kinda dreadful
| in my opinion. Trying to get it usable in diverse scenarios kinda
| reveals that the author has a very particular way it's meant to
| be used and it seems needlessly frustrating to use it otherwise.
| david2ndaccount wrote:
| I wrote a similar tool for macOS, using libdispatch.
|
| https://github.com/drpriver/macwatch
| benjaminjosephw wrote:
| This project wraps `inotifywait`[0] which I didn't know about
| until just now. Really useful utility, thanks for sharing!
|
| [0] - https://www.man7.org/linux/man-
| pages/man1/inotifywait.1.html
| dec0dedab0de wrote:
| I'm so used to making snap judgements on complex projects, and
| seeing that every file was last committed 8 years ago is usually
| a giant red flag to me.
|
| However, this is a single bash script, which definitely could be
| stable and forgotten about. Something like this would be more
| appealing as a posix shell script, but if you're using bash
| anyway and not trying to distribute, then stable and simple goes
| a long way.
| e-dant wrote:
| Those considering doing something like this (such as the author)
| might consider using my library:
|
| https://github.com/e-dant/watcher
|
| It hooks into system APIs where viable. (Otherwise, it uses
| std::filesystem.)
|
| It's meant to be as or more:
|
| - Easy to use
|
| - Efficient
|
| Than all similar libraries.
| wmichelin wrote:
| I have been using reflex https://github.com/cespare/reflex for
| the same thing in some of our docker projects
| athorax wrote:
| +1 for reflex, love being able to have a config for development
|
| Example .reflex.conf to run one command when an openapi spec
| changes and another when any go files change:
| -g "spec.yaml" -- bash -c 'make' -sr "\.go$" -- go run
| cmd/app/main.go
|
| Then run it with: reflex -d fancy -c
| .reflex.conf
| Brian_K_White wrote:
| incron
| jimpudar wrote:
| I haven't seen anybody here mention that you can do this with
| systemd path units [0],[1].
|
| I know not everybody uses Linux (or loves systemd as much as I
| do) but it's a great solution if you already use systemd.
|
| [0] https://www.redhat.com/sysadmin/introduction-path-units
|
| [1]
| https://www.freedesktop.org/software/systemd/man/systemd.pat...
| IncRnd wrote:
| What does this add to inotifywait? That does exactly the same as
| this project, which uses inotifywait to reimplement inotifywait.
|
| For example: dnf install inotify-tools or
| apt-get install inotify-tools
| cortesoft wrote:
| It looks like a simple bash wrapper around inotifywait to
| provide a simpler interface for a subset of the functionality.
| You don't have to remember any flags.
|
| Not super useful, but not useless.
| masukomi wrote:
| ifnotifywait doesn't exist on macOS, so in addition to that not
| being an option on macos, this would be something you could
| write tools that a team who used multiple platforms, or a
| person with setups on multiple os's, could use. I suspect this
| might even be usable on windows.
|
| [edit: fswatch looks very promising, and may be a better choice
| than either ifnotifywait or watchman]
| awinder wrote:
| This wraps inotifywait, so it doesn't really address cross-
| platform needs.
|
| The unique features in this over straight inotifywait is
| plugging together the command you want to execute, and some
| event dedupe logic. Grand majority of the code is around
| colorized logs, command docs, and some options handling code
| which mostly just gets piped to inotifywait.
| wwweston wrote:
| Discovered fswatch recently, it's proved _very_ useful, my
| favorite use being replacing a few situations where I was
| using sshfs with fswatch+rsync: no network lag on save /load,
| can even work offline, but everything still makes its way to
| the network destination!
| williamcotton wrote:
| I've been using fswatch for a project that supports both
| linux and darwin. Here's an example from a Makefile of how
| I use it to re-run tests on file change:
| test-watch: make --no-print-directory test || :
| fswatch --event Updated -o test/*.c test/*.h src/ | xargs
| -n1 -I{} make --no-print-directory test
| brigandish wrote:
| This one? https://github.com/emcrisostomo/fswatch
| IncRnd wrote:
| You are confused about how the tool works. This project uses
| inotifywait from inotify tools to reimplement what
| inotifywait already does. The linked article page at github
| clearly says this.
|
| Adding a new layer atop inotify tools doesn't make that
| package automatically run on new platforms. There are already
| tools on macos that use the native FSEvents API for watching
| for file changes and executing scripts.
| [deleted]
| EvanAnderson wrote:
| Does anybody have a good "execute a command when nothing changes"
| tool? I have some situations where I'd like to take action if,
| say, a file hasn't had its mtime updated in 24 hours.
| pitaj wrote:
| Does it need to be precise? I'd probably just use an hourly
| cron job.
| yabones wrote:
| Not sure if there's a way to do it instantly, but one way is to
| just run a find command in a cron job every X hours:
| find /foo/bar -type f -mtime +1 | xargs foobar
| lloeki wrote:
| I've been using fswatch to great effect. The big advantage is
| that it's truly cross-platform, falling back to polling if no fs
| notification system is available as long as stat(2) works.
|
| Very composable as you just pipe its output to whatever you want
| (typically a while read do end), so exclusion is a
| awk/sed/perl/grep/rg away, and command can be as simple or
| complex as it needs to while the tool itself stays lean.
|
| Cool features include -o to just be notified that something
| changed (e.g to fire up rsync), and ability to batch changes.
|
| Also the command is a front end to libfswatch, so one can use
| that directly if shelling is to be foregone.
|
| https://github.com/emcrisostomo/fswatch
| chmaynard wrote:
| > If you are installing fswatch using a package manager and you
| would like the PDF manual to be bundled into the package,
| please send a feature request to the package maintainer.
|
| I can't find the PDF manual in the repo. Do I need to contact
| the author and say please?
| bin_bash wrote:
| this project hasn't had a commit in 8 years and only has a
| handful of stars. It confusingly shares the name with a _much_
| more popular tool from Meta--though the use case is slightly
| different.
|
| There are other much more popular tools that do the same thing,
| like entr, watchexec, and nodemon. Why is this posted?
| [deleted]
| Chris2048 wrote:
| Funnily enough, the real value of this post is the thread
| detailing all the alternatives...
| TheRealPomax wrote:
| Small correction: watchman is a tool made and released by
| Facebook, not made and released by Meta.
|
| (yes, one owns the other, but Facebook employees writing and
| releasing code for Facebook are on Facebook's payroll, not
| Meta's payroll)
| bin_bash wrote:
| No, there is no such thing as "Facebook's Payroll". It's all
| Meta. I'm a former Meta employee. Watchman is used heavily
| across the family of apps. It's not like Alphabet that's a
| holding company for different smaller companies.
|
| Watchman not yet being branded with "Meta" is a case of
| laziness. It doesn't indicate it's limited to the Blue app.
| tartuffe78 wrote:
| I'm gonna try to use this to revert the .xcscheme file changes
| that Xcode makes when I switch between various Xcode versions on
| my machine. These happen in 10+ submodules everyday so this is
| great!
| m00dy wrote:
| I wrote something very similar.
|
| https://github.com/m00dy/fwe
| akras14 wrote:
| Edit: it may have been a different watchman, but same point still
| applies.
|
| It's been a few years, so perhaps it got addressed, but this
| library used to introduce a "memory leak" by design, by infinity
| storing pointers to files being watched.
|
| It was a pain to isolate (https://techtldr.com/simple-guide-to-
| finding-a-javascript-me...)
|
| Ultimately I ended up fixing it by writing my own version using
| native system calls available on Node.
|
| The moral of the story, this library is great if you are building
| cross platform dev tool. It is not (at least wasn't at the time)
| great for long lived processes that target only one type of
| operating system.
| vinay_ys wrote:
| Its a simple shell script wrapper on top of inotify-tools. I
| suppose its biggest contribution is a more appealing name -
| watchman.
| [deleted]
| VyseofArcadia wrote:
| I've seen about a dozen versions of this now. So my question is,
| who watches the watchmen?
| cjcampbell wrote:
| Came here for this. Thank you.
| koolba wrote:
| If it's running in a shell, it'd be the terminal session's
| shell. If it's running as a daemon, in a traditional Unix
| environment, it'd be the init process (pid 0).
| Brian_K_White wrote:
| Inotify is a kernel feature.
|
| Neither init or a shell or any other userspace code. The
| userspace code like in inotifywait or incron just tells the
| kernel what it wants the kernel to watch, and then exits,
| gone, no further parent, not even init.
|
| Ok now that I mentioned incron..., incron being a daemon is
| managed by init, but incron is not watching any files, not
| even it's own config files. incron just reads it's incrontab
| files and registers the desired events with the kernel, and
| then performs the desired actions when the kernel triggers
| it. It notices when it's own config files change the same way
| as any other inotify job, it asks the kernel to tell it when
| they change.
|
| incron is like cron where you write a crontab-like file that
| specifies a command of your choosing. Pretty much the same as
| this thing minus some colorful display.
|
| I guess the point of this thing would be if it provides a
| consistent wrapper interface over the various fs watching
| mechanisms on different OS's.
|
| But for linux, I've already been using incron to do this job
| efficiently for ages.
| [deleted]
| benniomars wrote:
| Nice. I've been using yardman that's written for NodeJS and Crow
| written in Go. Nice to have something that's just bash.
| pyuser583 wrote:
| Very cool
| BrainVirus wrote:
| This command seems like a no-brainer that should be in all OSes
| by default.
|
| It's mind-boggling that most systems don't have a standard tool
| for observing file changes. The only language I know of that has
| a platform-agnostic API for this is Factor.
| dspillett wrote:
| The required kernel hooks exist in pretty much any common OS
| these days, it is a user-space tool that is sometimes missing.
|
| It may not be installed by default, but inotifywait is
| available in common Linux distributions, usually in a package
| called something like ionotify-tools, and has been for over a
| decade-ana-half IIRC. It'll work under WSL on Windows too,
| though only for ext4 devices not bits of the Windows filesystem
| made available to Linux.
|
| I can't speak to what other OSs include by default, but as
| every major OS has a different API for how to register a
| listener & how it gets messages no built-in tool is going to be
| cross-platform. There are third party tools which present more
| cross-platform consistency, most notably
| https://github.com/emcrisostomo/fswatch#readme (also available
| in common Linux distros, just an _apt install_ away in Debian
| for instance).
| jitl wrote:
| Not to be confused with Facebook's file watch daemon, also names
| watchman, which does the same sort of thing but is more
| complicated. There's a bunch of tools that integrate Facebook's
| watchman for more efficient change tracking.
|
| Advantages of Facebook's watchman:
|
| 1. Implements efficient file system event watches on macOS and
| Windows
|
| 2. IPC/daemon system reduces resource use because overlapping
| watches/triggers don't use more inotify slots.
|
| 3. Denounces / waits for changes to settle
|
| 4. Client libraries in a few different languages for scripting
| purposes
|
| https://facebook.github.io/watchman/
| [deleted]
| malkia wrote:
| Also support for other OS's - like Windows.
| giantg2 wrote:
| I have just used some Power Shell scripts for stuff like this
| in the past on Windows.
| dec0dedab0de wrote:
| Also not to be confused with watchdog and it's command
| watchmedo, which is what I thought this was when I first read
| the headline. I don't know what the differences are, but I use
| watchdog to restart my local celery workers when I make
| changes. It's scripted out so I don't ever use the command
| directly.
|
| https://pypi.org/project/watchdog/
|
| https://github.com/gorakhargosh/watchdog
|
| EDIT: Now I'm wondering how often I miss out on something
| because I confuse it for something I already use. Maybe there
| is room for a "things I use" or "things I know about" browser
| plugin.
| ghassanmas wrote:
| Also to note "watchdog" the term. Is used also in
| hardware/software context. Usually as a way to mitgate or
| recover system faulate. I first I heard of this term while
| taking in a course about PIC microcontroller[1].
|
| > The Watchdog Timer in the PIC16F819 The watchdog timer can
| be a real source of pain and can also make a PIC system
| incredibly robust and reliable. But what exactly is the
| watchdog timer? Simply put, the watchdog timer is a hardware
| timer in the PIC that, if not constantly reset by the
| software, will cause the PIC to reset. This feature can be
| incredibly useful if the PIC should hang due to a hardware or
| software issue and guarantees that the PIC will restart from
| the beginning. Not only does it reset the system, it also
| flags a bit that can be used to determine if the system just
| crashed.
|
| Ref: https://maker.pro/pic/tutorial/how-to-get-started-with-
| pic-m...
|
| [1]: PIC microcontroller, is a small hardware simliar to
| arduino but more on a lower level.
| damagednoob wrote:
| Also not to be confused with watchexec (my tool of choice for
| executing commands on file changes):
|
| https://github.com/watchexec/watchexec
| HelloNurse wrote:
| 5. Windows is supported.
| martincmartin wrote:
| If you sign up for notifications on both github projects, you
| can watch the watchmen.
| graypegg wrote:
| It's a good day when I can get a laugh out of HN heh
| Manu40 wrote:
| But then, who watches the watcher?
| adhesive_wombat wrote:
| Microsoft, via the analytics on GitHub.
| MengerSponge wrote:
| Ok, but then who watches the watchmen watcher?
| [deleted]
| Manu40 wrote:
| Hardly comforting, but okay.
| jinnko wrote:
| Q
| Semiapies wrote:
| Alternately, Facebook's project is not to be confused with this
| significantly older one.
| amenghra wrote:
| Both projects are from around 2014.
| GRBurst wrote:
| For those use-cases I use entr (https://github.com/clibs/entr).
| In what sense does watchman differ to it? Didn't see anything
| related to this at first glance :-)
| DotaFan wrote:
| OP did say he did this for learning purpose.
| everybodyknows wrote:
| The page linked above says:
|
| > WARNING: This is a (possibly outdated and/or unmaintained)
| fork of https://github.com/eradman/entr .
| [deleted]
| [deleted]
| basemi wrote:
| I've used `incron`[0] for that purpose, with success. Packaged
| for most distros I think
|
| [0]https://github.com/ar-/incron
| thomasfl wrote:
| Shameless plug for my own take on file watchers. The filewatcher
| command line utility is available as a ruby gem. It exports shell
| variables, so you can print name of files that changes this way:
| $ filewatcher *.js 'echo $FILENAME'
|
| It recently reached 1 million downloads. The current maintainer,
| Alex Popov, is a very skilled developer working from Moscow.
| Zamicol wrote:
| Here was our take on the problem written in Go.
|
| https://github.com/Cyphrme/watch
|
| { "path_to_watch":"example_command_to_execute_on_change.sh", }
___________________________________________________________________
(page generated 2022-10-05 23:00 UTC)