Simple `watchman` example commands / quickstart

.00-watchman.md

watchman basic example

I found the watchman docs hard to follow / bereft of useful examples. Below is my attempt at a quickstart guide; see ryan-williams/watchman-helpers for some aliases and other scripts.

• 0. Install watchman
• 1. Configure a "watch" on the current directory
  • 1b. List currently watched directories
  • 1c. Remove a watched directory
• 2. Set up a "trigger": continuously run a command in response to changes
  • 2a. rsync a directory to a remote host
    • Is there a simpler way to trigger on all files?
    • Factoring command to a file
  • 2b. List active triggers for current directory
  • 2c. Delete trigger
• 3. Debugging / Monitoring
  • 3a. watchman-wait: print changed file names noticed by watchman
  • 3b. View logs from triggered scripts (and other watchman events)
    • Find watchman logfile
    • Tail the log

0. Install watchman

See installation docs.

1. Configure a "watch" on the current directory

watchman watch .

(watch docs)

1b. List currently watched directories

watchman watch-list
# {
#     "version": "2023.02.20.00",
#     "roots": [
#         "$PWD"
#     ]
# }

This should just show the directory you told it to watch above (I've substituted $PWD here, it will be a literal absolute path on your system).

(watch-list docs)

1c. Remove a watched directory

watchman watch-del .

(watch-del docs)

2. Set up a "trigger": continuously run a command in response to changes

2a. rsync a directory to a remote host

In this example, we'll rsync the current directory to a remote host

# Set up params
HOST=ec2                                       # configure this SSH host in your ~/.ssh/config
DST_DIR="`basename`"                           # remote directory to mirror this one to; `basename` will target a directory with the same name on the remote host (under your $HOME directory on that host)
trigger_name=ec2-sync                          # arbitrary name, used for `trigger-del` later
trigger_patterns=('**/*' '**/.*')              # trigger on all files in the current directory
trigger_cmd=(rsync -avzh ./ "$HOST:$DST_DIR")  # rsync the current directory to the remote host/dir

# Create the "trigger". It will begin running immediately and continuously (in response to file changes in the current directory)
watchman -- trigger "$PWD" "$trigger_name" "${trigger_patterns[@]}" -- "${trigger_cmd[@]}"

Is there a simpler way to trigger on all files?

The patterns ('**/*' '**/.*') are the only way I've found to get all files recursively under the watched root (including "hidden" files and directories, that begin with a .). However, when creating the trigger I've had the watchman logs print an error like:

trigger <dir>:<name> failed: posix_spawnp: Argument list too long

Apparently there is an inital run attempt that includes all eligible files as positional arguments. Maybe there's a better way, or workaround 🤷‍♂️. The "Simple Pattern Syntax" page doesn't seem to say, and single globs like ('' '.')` (Bash array syntax, as above) did not trigger for changes in subfolders.

Factoring command to a file

If your command gets more complex, you might want to factor it out to its own file, e.g.:

sync.sh

#!/usr/bin/env bash

# Configure this SSH host in your ~/.ssh/config
HOST=ec2
# Remote directory to mirror the current directory to. As written here, will target a directory
# with the same `basename` on the remote host (under your `$HOME` directory on that host)
DST_DIR="$(basename "$(dirname "${BASH_SOURCE[0]}")")"

# List some exclude paths here to pass to `rsync`
excludes=(.DS_Store __pycache__ .ipynb_checkpoints .pytest_cache '*.egg-info')
exclude_args=()
for exclude in "${excludes[@]}"; do
    exclude_args+=(--exclude "$exclude")
done

rsync -avzh "$@" "${exclude_args[@]}" ./ "$HOST:$DST_DIR/"

Then make sure to mark it as executable, and pass an absolute path to that script to watchman -- trigger …:

chmod 755 sync.sh
watchman -- trigger "$PWD" "$trigger_name" "${trigger_patterns[@]}" -- "$PWD/sync.sh"

(trigger docs)

2b. List active triggers for current directory

watchman trigger-list .
# {
#     "version": "2023.02.20.00",
#     "triggers": [
#         {
#             "append_files": true,
#             "name": "ec2-sync",
#             "stdin": [ "name", "exists", "new", "size", "mode" ],
#             "expression": [ "anyof", [ "match", "*", "wholename" ] ],
#             "command": [ "rsync", "-avzh", "./", "$HOST:$DST_DIR" ]
#         }
#     ]
# }

(trigger-list docs)

2c. Delete trigger

watchman trigger-del . "$trigger_name"

(trigger-del docs)

3. Debugging / Monitoring

3a. watchman-wait: print changed file names noticed by watchman

watchman-wait -m0 .

• -m0 causes this to run indefinitely (by default, it exits after one event, i.e. -m1)
• . watches the current directory

(watchman-wait docs

3b. View logs from triggered scripts (and other watchman events)

Find watchman logfile

logfile="$(ps aux | grep watchman | grep -o -- '--logfile=\S*' | awk -F= '{ print $2 }')"

• For me (on macOS, with watchman installed by Homebrew]()), this is /opt/homebrew/var/run/watchman/ryan-state/log.
• Thanks to this StackOverflow answer
• "Where are the logs" docs were not helpful

Tail the log

Continuously stream new output to the logfile:

tail -f "$logfile"

.watchman-rc

#!/usr/bin/env bash

append_to_path "$(dirname "${BASH_SOURCE[0]}")"

alias wm="watchman"

alias wmx="watchman-filter-exec.py"
alias wml="watchman-list"
alias wmd="watchman-delete"

alias wmt="watchman -- trigger"

watchman_trigger_list() {
    if [ $# -eq 0 ]; then
        set -- .
    fi
    watchman trigger-list "$@"
}
export -f watchman_trigger_list
alias wmtl=watchman_trigger_list
alias wmtd="watchman trigger-del"

alias wmw="watchman watch"
alias wmwl="watchman watch-list"
alias wmwd="watchman watch-del"
alias wmwp="watchman watch-project"

alias wss="watchman shutdown-server"
alias wq="watchman shutdown-server"

alias wv="watchman version"

watchman_log() {
    ps aux | grep watchman | grep -o -- '--logfile=\S*' | awk -F= '{ print $2 }'
}
export -f watchman_log
alias wmlg=watchman_log

watchman_wait() {
    if [ $# -eq 0 ]; then
        set -- .
    fi
    watchman-wait -m0 "$@"
}
export -f watchman_wait
alias ww=watchman_wait
alias ww0=watchman_wait

README.md

watchman-helpers

watchman-filter-exec.py

a.k.a. wmx:

watchman-filter-exec.py --help
# Usage: watchman-filter-exec.py [OPTIONS] [ARGS]...
# 
#   Filter changed files output by `watchman`; by default, only output Git-
#   tracked files.
# 
#   watchman-wait -m0 <dir> | watchman-filter-exec.py [-G/--no-git-filter] [-p/--prefix
#   <prefix>] [-v/--verbose...]
# 
# Options:
#   -G, --no-git-filter  Bypass filtering to Git-tracking files
#   -p, --prefix TEXT    Filter to relative paths beginning with this prefix
#                        (and strip the prefix)
#   -v, --verbose        1x: log to stderr when files pass filters and commands
#                        are run, and when the Git file listing is refreshed;
#                        2x: also log files that are skipped
#   --help               Show this message and exit.

Examples

# Print Git-tracked filenames as changes are made 
watchman-wait -m0 . | wmx

# Log to stderr:
# - commands run on Git-tracked files
# - Git file-list refreshes (whenever anything under `.git/` is changed)
# - skipped file paths (non-Git-tracked files; only in `-vv`, not `-v`) 
watchman-wait -m0 . | wmx -vv

Continuously sync Git worktree into a running Docker container

Create a .dockerignore that only includes Git-tracked files, using make-dockerignore.py:

make-dockerignore.py

Dockerfile that COPYs all Git-tracked files (thanks to generated .dockerignore above), and runs next dev auto-reloading server:

# Dockerfile
FROM node
COPY package.json package.json
RUN npm i
COPY . .
ARG PORT=3000
EXPOSE ${PORT}/tcp
ENV PATH="${PATH}:node_modules/.bin"
ENTRYPOINT ["next", "dev"]

Build image, run container w/ exposed webserver port:

port=3000
docker build --build-arg port=$port -t my-image .
docker run --rm -d -p $port:$port --name my-container my-image

Watch for changed files, copy changed Git-tracked files into my-container

watchman-wait -m0 . | wmx -v docker cp {} my-container:/{}

Leave this running, make changes, observe changed files to be copied into my-container:

Running: docker cp next.config.js my-container:/next.config.js
Refreshing Git file list (.git/index)
Running: docker cp package-lock.json my-container:/package-lock.json
…

watchman-delete

#!/usr/bin/env bash

if [ $# -eq 0 ]; then
  err "Usage: $0 <watched directory> [trigger name]"
elif [ $# -eq 1 ]; then
  watchman watch-del "$1"
else
  watchman trigger-del "$@"
fi

watchman-filter-exec.py

#!/usr/bin/env python
import click
import shlex
import sys
from subprocess import check_output, check_call


ARG_PLACEHOLDER = '{}'


@click.command('filter-files.py', help='Filter changed files output by `watchman`; by default, only output Git-tracked files.\n\n\twatchman-wait -m0 <dir> | filter-files.py [-G/--no-git-filter] [-p/--prefix <prefix>] [-v/--verbose...]')
@click.option('-G', '--no-git-filter', is_flag=True, help='Bypass filtering to Git-tracking files')
@click.option('-p', '--prefix', help='Filter to relative paths beginning with this prefix (and strip the prefix)')
@click.option('-v', '--verbose', count=True, help='1x: log to stderr when files pass filters and commands are run, and when the Git file listing is refreshed; 2x: also log files that are skipped')
@click.argument('args', nargs=-1)
def main(no_git_filter, prefix, verbose, args):
    args = list(args)
    if not args:
        args = ['echo']
    if all(ARG_PLACEHOLDER not in arg for arg in args):
        args += [ARG_PLACEHOLDER]

    do_git_filter = not no_git_filter
    if do_git_filter:
        git_files = list(filter(None, check_output(['git', 'ls-files']).decode().split('\n')))
    else:
        git_files = None

    def vlog(msg, level=1):
        if verbose >= level:
            sys.stderr.write(msg + '\n')

    for line in sys.stdin:
        file = line.rstrip('\n')
        if (no_git_filter or file in git_files) and (not prefix or file.startswith(prefix)):
            if prefix:
                file = file[len(prefix):]
            cmd = [
                arg.replace(ARG_PLACEHOLDER, file)
                for arg in args
            ]
            vlog(f'Running: {shlex.join(cmd)}')
            check_call(cmd)
        elif do_git_filter and file.startswith('.git/') and not file.startswith('.git/.watchman'):
            vlog(f'Refreshing Git file list ({file})')
            git_files = list(filter(None, check_output(['git', 'ls-files']).decode().split('\n')))
        else:
            vlog(f'Skipping: {file}', 2)


if __name__ == '__main__':
    main()

watchman-list

#!/usr/bin/env bash

if [ $# -eq 0 ]; then
  watchman watch-list
else
  watchman trigger-list "$@"
fi