PGO build scripts #705

dnkl commented 2021年08月31日 16:28:51 +02:00

Usage: meson-pgo.sh none|auto|partial|full|full-headless-sway <source-dir> <build-dir> <meson-options>

build-dir must not exist before the script is run (but if it does, the script will tell you so, and exit).

Supported compilers: gcc and clang

The script does not add any custom meson options, except configure -Db_pgo. It also does not run the final ninja call (since it's kind of a meson wrapper script). Thus, you probably want to call it like this:

meson-pgo.sh auto . /tmp/foot-pgo-build --buildtype=release -Db_lto=true && \
ninja -C /tmp/foot-pgo-build

Note the new full-headless-sway mode. This runs a full PGO build, but in a headless Sway instance. This means it can be used on e.g. sandboxed build servers (which would have to add sway as a build dependency).

auto will do a full-headless-sway build if sway is available. If not, it will do a full PGO build in the currently running compositor, if WAYLAND_DISPLAY is set. If no it falls back to a partial PGO build.

TODO

• script cleanup
• document in INSTALL.md
• verify performance of a full-headless build matches a "regular" full build

Closes #701

Usage: `meson-pgo.sh none|auto|partial|full|full-headless-sway <source-dir> <build-dir> <meson-options>` `build-dir` must *not* exist before the script is run (but if it does, the script will tell you so, and exit). Supported compilers: gcc and clang The script does *not* add _any_ custom meson options, except configure `-Db_pgo`. It also does **not** run the final `ninja` call (since it's kind of a meson wrapper script). Thus, you probably want to call it like this: ```sh meson-pgo.sh auto . /tmp/foot-pgo-build --buildtype=release -Db_lto=true && \ ninja -C /tmp/foot-pgo-build ``` Note the new `full-headless-sway` mode. This runs a full PGO build, but in a headless Sway instance. This means it can be used on e.g. sandboxed build servers (which would have to add sway as a build dependency). `auto` will do a full-headless-sway build if sway is available. If not, it will do a full PGO build in the currently running compositor, if `WAYLAND_DISPLAY` is set. If no it falls back to a partial PGO build. **TODO** * [x] script cleanup * [x] document in INSTALL.md * [x] verify performance of a full-headless build matches a "regular" full build Closes #701

Arnavion commented 2021年09月01日 08:50:03 +02:00

Running sway headless seems to require some more config.

• XDG_RUNTIME_DIR=$(mktemp -d), without which sway complains it doesn't know where to create the wayland socket.

• (削除) WLR_LIBINPUT_NO_DEVICES=1, because there aren't any input devices. (削除ここまで) This isn't necessary.

(Edit: Removed bit about WLR_RENDERER_ALLOW_SOFTWARE and WLR_RENDERER=pixman because they're not needed. My error with using them was PEBKAC; sway auto-selects the pixman renderer without either of them.)

• sway 1.6.1 crashes because of https://github.com/swaywm/sway/issues/6419 when the python script runs.

---

Running foot seems to require some more config.

• LC_CTYPE needs to be set to en_US.UTF-8 or similar, because the default could be C inside the chroot. I think it's fine if you don't want to do this since it doesn't seem appropriate to do it here; the package spec can do it.

• foot exits with exit code 230 if a font isn't installed:

  info: main.c:398: version: 1.9.0 -pgo +ime +graphemes
  info: main.c:405: arch: Linux x86_64/64-bit
  info: main.c:412: locale: en_US.UTF-8
  info: config.c:3053: loading configuration from /dev/null
  info: wayland.c:1270: HEADLESS-1: 1280x720+0x0@60Hz headless 0.00" scale=1 PPI=0x0 (physical) PPI=0x0 (logical), DPI=0.00
  info: wayland.c:1423: requesting SSD decorations
  info: fcft.c:267: fcft: 2.4.5 +graphemes +runs
  info: fcft.c:277: fontconfig: 2.13.1
  info: fcft.c:283: freetype: 2.11.0
  err: fcft.c:436: monospace:size=8.00:dpi=96.00:weight=bold: failed to match font
  err: fcft.c:436: monospace:size=8.00:dpi=96.00:slant=italic: failed to match font
  err: fcft.c:436: err: fcft.c:436: monospace:size=8.00:dpi=96.00:weight=bold:slant=italic: failed to match font
  monospace:size=8.00:dpi=96.00: failed to match font
  err: terminal.c:959: failed to load primary fonts
  err: fdm.c:215: no such FD: 6
  info: wayland.c:800: using SSD decorations
  warn: terminal.c:1702: slave exited with signal 1 (Hangup)
  info: main.c:579: goodbye
  

  I needed to install a monospace font. This also doesn't need any change in your script, just needs to be documented.

---

It would be useful to split this into three scripts (I converted them to bash and made them sway-specific for my testing, but you would of course devolve them back to sh and the other modes):

1. pgo-run.sh that does ./footclient --version && sway ...

   #!/bin/bash
   
   set -euxo pipefail
   sourcedir="1ドル"
   builddir="2ドル"
   cd "$builddir"
   ./footclient --version
   sway_conf="$(mktemp)"
   >"$sway_conf" echo "exec '$sourcedir/pgo-run-inner.sh' '$sourcedir' '$builddir'"
   xdg_runtime_dir="$(mktemp -d)"
   WLR_BACKENDS=headless \
    XDG_RUNTIME_DIR="$xdg_runtime_dir" \
    sway -d -c "$sway_conf"
   [ -f "$builddir/pgo-ok" ]
   

2. pgo-run-inner.sh that runs inside the sway exec (a separate script helps with readability)

   #!/bin/bash
   
   set -ux
   sourcedir="1ドル"
   builddir="2ドル"
   LC_CTYPE=en_US.UTF-8 "$builddir/foot" \
    -o tweak.render-timer=log \
    --config=/dev/null \
    --term=xterm \
    bash -c "
    set -euo pipefail
    tmp_file="$(mktemp)"
    '$sourcedir/scripts/generate-alt-random-writes.py' \
    --scroll --scroll-region \
    --colors-regular --colors-bright --colors-256 --colors-rgb \
    --attr-bold --attr-italic --attr-underline \
    --sixel \
    \"\$tmp_file\"
    cat \"\$tmp_file\"
    "
   rc="$?"
   >&2 echo "foot exited with $rc"
   if (( rc == 0 )); then
    touch "$builddir/pgo-ok"
   fi
   swaymsg exit
   

3. One that calls meson and pgo-run.sh

   meson -Db_pgo=generate
   meson compile
   "$sourcedir/pgo-run.sh" "$sourcedir" "$builddir"
   meson configure -C "$builddir" -Db_pgo=use
   

In my package spec, I can ignore (3) because it's encoded in the specfile (since it uses a macro for meson) but reuse the PGO measurement steps in (1) and (2), instead of having to do everything myself.

Running sway headless seems to require some more config. - `XDG_RUNTIME_DIR=$(mktemp -d)`, without which sway complains it doesn't know where to create the wayland socket. - ~~`WLR_LIBINPUT_NO_DEVICES=1`, because there aren't any input devices.~~ This isn't necessary. (Edit: Removed bit about `WLR_RENDERER_ALLOW_SOFTWARE` and `WLR_RENDERER=pixman` because they're not needed. My error with using them was PEBKAC; sway auto-selects the pixman renderer without either of them.) - sway 1.6.1 crashes because of https://github.com/swaywm/sway/issues/6419 when the python script runs. --- Running foot seems to require some more config. - `LC_CTYPE` needs to be set to `en_US.UTF-8` or similar, because the default could be `C` inside the chroot. I think it's fine if you don't want to do this since it doesn't seem appropriate to do it here; the package spec can do it. - foot exits with exit code 230 if a font isn't installed: ``` info: main.c:398: version: 1.9.0 -pgo +ime +graphemes info: main.c:405: arch: Linux x86_64/64-bit info: main.c:412: locale: en_US.UTF-8 info: config.c:3053: loading configuration from /dev/null info: wayland.c:1270: HEADLESS-1: 1280x720+0x0@60Hz headless 0.00" scale=1 PPI=0x0 (physical) PPI=0x0 (logical), DPI=0.00 info: wayland.c:1423: requesting SSD decorations info: fcft.c:267: fcft: 2.4.5 +graphemes +runs info: fcft.c:277: fontconfig: 2.13.1 info: fcft.c:283: freetype: 2.11.0 err: fcft.c:436: monospace:size=8.00:dpi=96.00:weight=bold: failed to match font err: fcft.c:436: monospace:size=8.00:dpi=96.00:slant=italic: failed to match font err: fcft.c:436: err: fcft.c:436: monospace:size=8.00:dpi=96.00:weight=bold:slant=italic: failed to match font monospace:size=8.00:dpi=96.00: failed to match font err: terminal.c:959: failed to load primary fonts err: fdm.c:215: no such FD: 6 info: wayland.c:800: using SSD decorations warn: terminal.c:1702: slave exited with signal 1 (Hangup) info: main.c:579: goodbye ``` I needed to install a monospace font. This also doesn't need any change in your script, just needs to be documented. --- It would be useful to split this into three scripts (I converted them to bash and made them sway-specific for my testing, but you would of course devolve them back to sh and the other modes): 1. `pgo-run.sh` that does `./footclient --version && sway ...` ```sh #!/bin/bash set -euxo pipefail sourcedir="1ドル" builddir="2ドル" cd "$builddir" ./footclient --version sway_conf="$(mktemp)" >"$sway_conf" echo "exec '$sourcedir/pgo-run-inner.sh' '$sourcedir' '$builddir'" xdg_runtime_dir="$(mktemp -d)" WLR_BACKENDS=headless \ XDG_RUNTIME_DIR="$xdg_runtime_dir" \ sway -d -c "$sway_conf" [ -f "$builddir/pgo-ok" ] ``` 1. `pgo-run-inner.sh` that runs inside the sway exec (a separate script helps with readability) ```sh #!/bin/bash set -ux sourcedir="1ドル" builddir="2ドル" LC_CTYPE=en_US.UTF-8 "$builddir/foot" \ -o tweak.render-timer=log \ --config=/dev/null \ --term=xterm \ bash -c " set -euo pipefail tmp_file="$(mktemp)" '$sourcedir/scripts/generate-alt-random-writes.py' \ --scroll --scroll-region \ --colors-regular --colors-bright --colors-256 --colors-rgb \ --attr-bold --attr-italic --attr-underline \ --sixel \ \"\$tmp_file\" cat \"\$tmp_file\" " rc="$?" >&2 echo "foot exited with $rc" if (( rc == 0 )); then touch "$builddir/pgo-ok" fi swaymsg exit ``` 1. One that calls meson and `pgo-run.sh` ```sh meson -Db_pgo=generate meson compile "$sourcedir/pgo-run.sh" "$sourcedir" "$builddir" meson configure -C "$builddir" -Db_pgo=use ``` In my package spec, I can ignore (3) because it's encoded in the specfile (since it uses a macro for `meson`) but reuse the PGO measurement steps in (1) and (2), instead of having to do everything myself.

dnkl commented 2021年09月01日 19:05:13 +02:00

XDG_RUNTIME_DIR=$(mktemp -d)

This is now being done for full-headless-sway

LC_CTYPE needs to be set to en_US.UTF-8 or similar
foot exits with exit code 230 if a font isn't installed:

Documented at the top of the script. I also added a comment mentioning Sway-1.6.2 (which obviously hasn't been released yet) is required for full-headless-sway.

Script refactoring is still pending.

> `XDG_RUNTIME_DIR=$(mktemp -d)` This is now being done for `full-headless-sway` > LC_CTYPE needs to be set to en_US.UTF-8 or similar > foot exits with exit code 230 if a font isn't installed: Documented at the top of the script. I also added a comment mentioning Sway-1.6.2 (which obviously hasn't been released yet) is required for `full-headless-sway`. Script refactoring is still pending.

dnkl commented 2021年09月04日 10:54:15 +02:00

meson-pgo.sh has been replaced by several script primitives, and a new top-level script, pgo.sh (all scripts located in the pgo directory).

The script primitives can hopefully be re-used by build servers, while pgo.sh both act as a user-friendly frontend for users doing manual PGO builds, as well as documenting how the script primitives should be used.

See 47b8fd7192 for details.

`meson-pgo.sh` has been replaced by several script primitives, and a new top-level script, `pgo.sh` (all scripts located in the `pgo` directory). The script primitives can hopefully be re-used by build servers, while `pgo.sh` both act as a user-friendly frontend for users doing manual PGO builds, as well as documenting how the script primitives should be used. See https://codeberg.org/dnkl/foot/commit/47b8fd719210f3defe3f946626afd75f49738294 for details.

dnkl commented 2021年09月04日 11:07:43 +02:00

Headless cage script added. Appears to work, but cage is spamming the output with

00:00:07.825 [types/wlr_output.c:720] Basic output test failed for HEADLESS-1
info: render.c:2598: frame rendered in 0s 94 μs (0s 0 μs double buffering)
00:00:07.841 [types/wlr_output.c:720] Basic output test failed for HEADLESS-1
info: render.c:2598: frame rendered in 0s 2211 μs (0s 0 μs double buffering)
00:00:07.857 [types/wlr_output.c:720] Basic output test failed for HEADLESS-1
info: render.c:2598: frame rendered in 0s 1328 μs (0s 0 μs double buffering)
00:00:07.874 [types/wlr_output.c:720] Basic output test failed for HEADLESS-1
info: render.c:2598: frame rendered in 0s 1314 μs (0s 0 μs double buffering)
00:00:07.890 [types/wlr_output.c:720] Basic output test failed for HEADLESS-1
info: render.c:2598: frame rendered in 0s 1234 μs (0s 0 μs double buffering)
00:00:07.906 [types/wlr_output.c:720] Basic output test failed for HEADLESS-1

Headless cage script added. Appears to work, but cage is spamming the output with ``` 00:00:07.825 [types/wlr_output.c:720] Basic output test failed for HEADLESS-1 info: render.c:2598: frame rendered in 0s 94 μs (0s 0 μs double buffering) 00:00:07.841 [types/wlr_output.c:720] Basic output test failed for HEADLESS-1 info: render.c:2598: frame rendered in 0s 2211 μs (0s 0 μs double buffering) 00:00:07.857 [types/wlr_output.c:720] Basic output test failed for HEADLESS-1 info: render.c:2598: frame rendered in 0s 1328 μs (0s 0 μs double buffering) 00:00:07.874 [types/wlr_output.c:720] Basic output test failed for HEADLESS-1 info: render.c:2598: frame rendered in 0s 1314 μs (0s 0 μs double buffering) 00:00:07.890 [types/wlr_output.c:720] Basic output test failed for HEADLESS-1 info: render.c:2598: frame rendered in 0s 1234 μs (0s 0 μs double buffering) 00:00:07.906 [types/wlr_output.c:720] Basic output test failed for HEADLESS-1 ```