Dev build in rootless Podman: "Failed to resolve package cache" #75

When building a custom recipe for pigz (see below) inside a rootless Podman container (also below), the build fails with the following message:

error build failed
 subject: pigz
 phase: build
 cause: failed to create package archive
 details: failed to resolve package cache; build_profile=/mere/dev/build/pigz-2.8-1-<hash>/profile

"Failed to create package archive" appears at /src/build_orchestrator/packaging.zig:513, and "failed to resolve package cache" is at /src/cli/commands/build.zig:187. I've not yet dug further to see from where these are called.

I also tested using the zstd recipe and got the same error pattern:

error build failed
 subject: zstd
 phase: build
 cause: failed to create package archive
 details: failed to resolve package cache; build_profile=/mere/dev/build/zstd-1.5.7-2-<hash>/profile

Should I attach cold-cache verbose output too? Compilation for pigz is quite short, but there's a fair bit of output from staging the build dependencies.

Podman run command (pwd has src/recipe_{pigz,zstd}.kdl):

podman run -v .:/test-mere:ro -it alpine-mere:proto bash

FROMalpine:3.23ARG \
	SRC_DIR=/src \
	ZIG_VERSION=0.16.0RUN <<EOF	## variables for installing Zig	## - PLATFORM :: e.g. ~x86_64-linux~	## - ZIG_TARFILE :: Zig download file	## - ZIG_TARDIR :: Zig tar file directory	## - ZIG_URL :: Zig download URL	## - ZIG_DIR :: output Zig directory	## - SRC_DIR :: e.g. Mere project source	PLATFORM="$(uname -mo | tr ' ' - | awk '{print tolower(0ドル)}')"	ZIG_TARFILE="zig-${PLATFORM}-${ZIG_VERSION}.tar.xz"	ZIG_TARDIR="${ZIG_TARFILE/\.tar\.xz/}"	ZIG_URL="https://ziglang.org/download/${ZIG_VERSION}/${ZIG_TARFILE}"	ZIG_DIR="/etc/zig/${ZIG_VERSION}"	## initialise Alpine package keeper	apk update	apk upgrade	## minimum packages:	## - build-base, curl, git, linux-headers, make :: common	## project download, compilation, and installation things	## - cmake :: extra for Mere build	## - bash, wget :: nicer general tools	apk add \
		bash \
		build-base \
		cmake \
		curl \
		git \
		linux-headers \
		make \
		wget	## clean package cache to minimise output image size	apk cache clean	apk cache purge	## download, unpack, and install the Zig distribution	## NOTE the APK version is out of date: v0.15.0; need v0.16.0	wget -q -O "${ZIG_TARFILE}" "${ZIG_URL}"	tar -xf "${ZIG_TARFILE}"	mkdir -p "$(dirname ${ZIG_DIR})"	mv "${ZIG_TARDIR}" "${ZIG_DIR}"	unlink "${ZIG_TARFILE}"	ln -s "${ZIG_DIR}/zig" /usr/bin/EOFRUN <<EOF	## clone and build Mere, then clear out the source directory	mkdir "${SRC_DIR}"	git clone https://codeberg.org/merelinux/mere.git "${SRC_DIR}/mere"	cd "${SRC_DIR}/mere"	zig build \
		install \
		--prefix /usr \
		--release=fast	cd -	rm -fr "${SRC_DIR}"EOFRUN <<EOF	## set up Mere	mere store init	wget -q -O /mere/config.kdl https://pkgs.merelinux.org/config.kdl	wget -q -O /mere/keys/mere.pub https://pkgs.merelinux.org/mere.pubEOF

recipe{	name"pigz"	description"Parallel implementation of gzip"	url"https://zlib.net/pigz/"	version"2.8"	release1	licenses"zlib"	archs"x86_64"	depends"busybox""llvm""make""musl-dev""zlib-ng-dev"	envCFLAGS="-O3 -flto=auto"}source"https://zlib.net/pigz/pigz-${recipe.version}.tar.gz"{	blake3"fa165f414a12851806d9d54920879dd989917b9aa410aec671c602b18773b236"}build{	scriptr#"
		make pigz CC=clang
	"#}check{	scriptr#"
		make test
	"#}install{	scriptr#"
		echo "${DESTDIR}"
		install \
			-Dm0755 pigz \
			-t "${DESTDIR}"/usr/bin
		ln -s pigz "${DESTDIR}"/usr/bin/unpigz
		install \
			-Dm0644 pigz.1 \
			-t "${DESTDIR}"/usr/share/man/man1
	"#}package"pigz"{	files\"usr/bin/pigz"\"usr/bin/unpigz"\"usr/share/man/man1/pigz.1"}

/root# mere dev -v build /test-mere/src/recipe_pigz.kdl
 debug: loading system config from: /mere/config.kdl
 debug: reading config file: /mere/config.kdl
 debug: parsing repository from kdl
 debug: validating name and url values
 debug: creating repo config with name=mere url=https://pkgs.merelinux.org/mere priority=100
 debug: successfully created repo config with name=mere enabled=true
 debug: validating 1 repos
 debug: checking repo 0 name=mere
 debug: validating unique name
 debug: validating name length
 debug: validating url length
 debug: validating url
 debug: done validating 1 repos
 debug: validating 1 repos
 debug: checking repo 0 name=mere
 debug: validating unique name
 debug: validating name length
 debug: validating url length
 debug: validating url
 debug: done validating 1 repos
prepare pigz-2.8
 fetch sources
 sources restored from cache: 4a425a98b70f17606ec8b84c7f57c6d3d77cef4398a74e6bce4f6b728caaaa41
 unpack
 source tree restored from cache: c8c4f88478fe54812ffcd4f6684ba3cb504b88b4c71a1c9deaabcb08704b73f9
build pigz-2.8
 create profile
 debug: creating isolated build profile
 debug: creating build profile at: /mere/dev/build/pigz-2.8-1-db2e996eab575cc6/profile
 debug: created build profile at: /mere/dev/build/pigz-2.8-1-db2e996eab575cc6/profile
 profile created: /mere/dev/build/pigz-2.8-1-db2e996eab575cc6/profile
 install dependencies
 dependency profile restored from cache: f6b7832269f41fdeed2ddb2b5f5014b2568172c98f80af0c888bb90314a3e9e6
 recipe execution
 debug: using profile for namespace: /mere/dev/build/pigz-2.8-1-db2e996eab575cc6/profile
 log writing: /mere/dev/build/pigz-2.8-1-db2e996eab575cc6/build.log
 build
 phase restored from cache: build (04f8f871bc33edb84dd8a9e51e8656121316a17798c417abbc31829d6c0fd2e9)
 check
 phase restored from cache: check (04f8f871bc33edb84dd8a9e51e8656121316a17798c417abbc31829d6c0fd2e9)
 install
 phase restored from cache: install (29f653447cb9c6bd8a4a82680832f1b050b91dd09b86bc308f5d0373003c2981)
 stage packages
 split staging restored from cache: 23ba678c53eae41ddcfb75e0b1d69610becf7a5a6be67a45f2c98d44ebfb1777
 package artifacts
 debug: compressed 1 man pages and rewrote 0 symlinks in /mere/dev/build/pigz-2.8-1-db2e996eab575cc6/pkg/pigz
 debug: stripped: usr/bin/pigz
 debug: stripped 1 files in /mere/dev/build/pigz-2.8-1-db2e996eab575cc6/pkg/pigz
 debug: cleaning up build profile: /mere/dev/build/pigz-2.8-1-db2e996eab575cc6/profile
error build failed
 subject: pigz
 phase: build
 cause: failed to create package archive
 details: failed to resolve package cache; build_profile=/mere/dev/build/pigz-2.8-1-db2e996eab575cc6/profile

Nice, thanks for the report! I'm fairly confident this is a rootless Podman + nested user namespace interaction.

mere dev build uses Linux user namespaces for build isolation, the build phases run inside a namespace where the build user is
mapped to look like root. On bare metal (or in a full-root container), this works fine because when the namespace exits,
everything is still owned by real root. But in rootless Podman, there's already one layer of UID mapping happening (your host user
-> container UID 0). Mere then creates a second nested namespace on top of that. When files get created inside the nested namespace
during build phases, their ownership may not map back cleanly to the outer container context. So when the packaging step runs
outside the build namespace and tries to write into directories that were set up by the namespace, it hits permission issues.

I haven't yet tested this specific scenario myself, but the code path and your error output line up with this theory. The fix
would be to ensure the packaging/cache-restore step can write to its output directories regardless of what happened inside the
build namespace, likely by explicitly ensuring the directories are writable before attempting the copy.

In the meantime, does running with --no-cache change anything?

No luck with --no-cache: the only difference is it re-compiles on each attempt, rather than using the cached stage outputs. The failure point and message are the same.

I think I know what you mean about nested namespaces, it's something I've run into before in a different context. There were possible ways to fix it -- probably a mapping shift applied by the outer or inner host -- but I didn't try those at the time for that project.

I tried adding a writable volume into the Podman container over /mere/dev. This writes the data out, the build and cache dirs are accessible on leaving the container, but it fails at the same point again with cold or warm cache (or no cache). The directory mapped into /mere/dev shows me as the owner and group.

I also tried alpine-chroot-install (currently running an Arch-based system) and I get an early failure instead, but this is almost certainly a different problem:

build phase 'build' failed with exit code 1; build_profile=<none>

But now there's a proper problem: I installed Mere directly into my system, but I still hit the same failure. I've tried running mere dev build with and without sudo, for both the pigz and zstd recipes. So I'm not sure what I've done to my system (or what dependencies I might be missing), but it's stopping Mere from resolving the package cache.

That is, I assume the fault is on my end -- I'd be curious to see if you or anyone else can reproduce! Is there any more information I should collect? I might just be missing a Mere set-up/config step.

mkdir ~/mere-dev
podman run \
	-v .:/test-mere \
	-v ${HOME}/mere-dev:/mere/dev \
	-it alpine-mere:proto \
	bash

Handful more attempts, all with the same "failed to resolve package cache" result:

• --userns=keep-id in the Podman call
• made the mounted mere-dev dir outside the project dir
• mount volumes with the :U,Z flags
• run Podman via sudo

ok, created a new release v0.14.0. This might not fix your issue, but it should make clearer where the issue is happening. Can you try please @oryktos and let me know? Thanks!

Shall do! I'll build a new container from the new release and update my local install, and report the result from trying both.

Also, apologies if my messages are a bit verbose! Please let me know if it's a bit much.

Okay, I updated the container and my local install to v0.14.0 and ran the pigz and zstd builds again. Both failed with "filesystem error reading staging or output directory". This seems to come from the function restore (build_orchestrator/cache_solver.zig:513), but I am not certain. This happens with cold, warm, or no cache.

Outside the container, I tried with and without sudo, but the result was the same. I also noticed the sticky bit is set on /mere/dev, it could have been there before but I didn't notice it.

Curious! It sounds like my systems and methods are mangled, rather than Mere. For the container, it's likely the user namespace; for my laptop, I couldn't say, but this machine has been through... a good bit, and it's not exactly high-end.

On my machine:

• /mere/dev — 0755, owned by root:root
• /mere/dev/cache — 1777 (sticky bit + world-writable), owned by root:root
• /mere/dev/cache/build — 1777, owned by root:root
• /mere/dev/cache/build/artifacts — 0755, owned by jeremy:jeremy
• /mere/dev/cache/build/keys — 0755, owned by jeremy:jeremy
• /mere/dev/cache/sources — 1777, owned by root:root, files owned by jeremy:jeremy

running sudo mere store init on the host system should fix the tree. You can use the -n flag to do a dry run first.

I ran sudo mere store init, I missed that when updating and re-installing. This is what I saw:

 Fixed permissions on /mere/dev/build (0755 -> 1777)
Created /mere/dev/outputs (mode 1777)
Created /mere/dev/repo (mode 1777)
Fixed permissions on /mere/dev/cache (0755 -> 1777)
Fixed permissions on /mere/dev/cache/sources (0755 -> 1777)
Fixed permissions on /mere/dev/cache/build (0755 -> 1777)
6 change(s) applied

Afterward, I had three ownership differences compared to your system:

Retrying gave the same error at this point, so I ran sudo chown -R oryktos:oryktos /mere/dev/cache/build/{artifact,key}s /mere/dev/cache/sources then tried again, but still the same error.

Wiping out /mere and starting again with a pre-compiled binary, following the contributing guide then fixing ownership differences, was also not successful (before and after chown).

I ran strace but I wouldn't know how to interpret it well. There's a lot of output! But it compresses well, or I could target specific types or areas. Would strace output help, and how should I prepare it?

Thanks for the extra details, very interesting. This sounds like some other underlying system config/setup that is in play. What kind of filesystem are you using? Are there any kernel messages that are showing errors? Maybe you could run dmesg right after the error. strace output would be fine too. You could attach the output here if you want. I'm also considering setting up a chat channel somewhere for easier discussion. Maybe Libera/IRC or Discord. Any preference?

I really do think it's my frankensystem, I made a few strange choices setting it up and it needs a rebuild. I'm running EndeavourOS, originally installed with XFCE but now running Hyprland. The filesystem is XFS. It's a cheap machine and the hardware isn't great.

sudo dmesg -H -T and sudo journalctl -x show nothing at all¹ around the time I ran mere dev -v build, but I probably haven't run the right command. Is there a filter or flag I should use?

As a side note, when I cleared out /mere earlier I needed to use chattr -i /mere/store within sudo bash -i before rm -fr /mere. sudo rm -fr /mere did not work, nor did sudo chattr -i /mere/store; it had to be sudo {shell} -i then chattr -i and rm -fr /mere in the elevated session.

Also, attached is strace output with a warm cache. All current tests are in the laptop, not the container. I see a few errors, the notable ones are two EINVAL errors on the recipe file, but that might not be unusual.

Re. a chat channel, I am not sure! I guess the main trouble with IRC would be sessions and async messaging. I do have a Discord account and others could join in that way too. I also use Element, but not tried public rooms. I've used Zulip too, that seems quite good for organisations but I realise it needs yet another account. Maybe Zulip, Element, or Discord are practical for the time-zone difference, given the session/history thing with IRC.

The chattr requirement is currently expected. mere intentionally sets directories in the store to be immutable after they're installed. I suppose it might be useful in some situations to have that as an optional config setting.

I'll take a look at the strace output, thanks.

I made a Discord server: https://discord.gg/Vs7QgdF9 let's see how useful that is.

No trouble, I wasn't sure if it was more of my mangled system and thought I should mention. I don't have a specific example of optional immutability, but I think it's a good default: there's less chance of wrecking the whole store. (Tangentially, I wonder why it's not more common to use immutability, e.g. if somehow accidentally running sudo rm -fr /...)

I'll jump in to the server this evening, my Discord username is different but I've introduced myself in the general channel!

I also realise I've diverted significantly from the ticket's original purpose! Should I test the error inside a container to confirm it's definitely the nested userns situation, and would I use a similar approach -- dmesg, journalctl, and strace? Or should I rename the ticket to reflect the non-containerised laptop debug?

Just looked at the strace log, this might actually explain both issues, in the rootless and on bare metal. It seems like you didn't generate a signing key for yourself?

openat(AT_FDCWD, "/home/oryktos/.mere/keys/mere.key", O_RDONLY|...) = -1 ENOENT (No such file or directory)

And in turn, that triggers a mere bug that doesn't bubble up the exact cause of the issue, that it couldn't sign the produced package. I'll look at fixing that part, but if you could test/confirm by doing this:

mere dev key generate

It should create a key at the right location

Oh gosh that's exactly what it was, for both the laptop and in the container. I am sorry! I saw it in the strace output but didn't chase it, for whatever reason...

I should have read slightly further in the contributing guide! But I did think the key was only needed for importing packages after a build. Maybe the note about the signing key could be moved to the setup part, after acquiring the repo config and key.

In any case, this is now fixed for rootless Podman and for a non-contained instance, so I will close the ticket. I quote what has recently become my catch-phrase: thank you, and I'm sorry!

I'll continue writing and testing new recipes. Are there any of specific interest? I don't have much experience building common software, but can port apkbuilds or solidify (recipefy?) some of the container ops I've put together for academic/scientific software, and adjust the pattern for other things (not just academic software).

No worries at all. This was helpful since it surfaced what I consider a bug: unhelpful error output.

Any recipe for software that you find genuinely useful is welcome, although I will say that I have no intention of ever adding support for systemd :)

Thanks again for your support, please keep the issues coming, they're very helpful.

I am glad it's helpful! I tend to go down rabbit holes a bit energetically, but not always the right one...

I'll have a think and note which software I use often, or which are noteworthy. Systemd is not on my radar, I will say! I haven't done much with service management, though openrc was easier to work with (could be my Alpine bias). I've read about but not used s6, I'll probably have a bit of a dig into that soon, too.

If there are things other than recipes to try too, please let me know! Late meeting last night so I didn't get a chance to check Discord, but I should be back on there soon too, probably tomorrow. I'm not sure what the time difference is -- for reference, I'm GMT+8, so it's currently 7:40am at the time of posting this.