Hi there, happy to talk a little about fw2tar - it's a pretty small utility focused on a very specific goal, building off the awesome work you all have done here (and also binwalk, though that project seems to have largely died over the last 3 years).

I'm wrapping up a PhD focused on firmware rehosting and dynamic analysis of firmware in general. Given a linux-based firmware image, I want to get it up and running under emulation with PANDA.re. My expertise (and the bulk of my research) is focused around runtime analysis and modification of a emulated guest. But a critical input to this is an accurate root filesystem for a given firmware image. Previously I've used binwalk/firmadyne's extractor but now it seems like unblob is the best tool around.

There are 3 goals I have with fw2tar that don't seem to overlap with unblob which is why I threw those utility scripts into a stand-alone repo instead of opening up PRs:

1. Identify linux-based root filesystems within extracted filesystems and package these up nicely
2. Maintain permissions within the filesystem
3. Support operation in high-performance-computing environments

Root filesystem detection This isn't anything too fancy - my scripts search for some standard linux directories and files to identify potential root filesystems. For each, we create a tar archive and try to exclude any recursive extraction artifacts (after prototyping a few approaches around multiple extraction passes with restricted depth), I ended up just excluding directories with _extract in the name). The generated archives are sorted by size from largest to smallest.

Maintain permissions: If we want to boot a system using an extracted filesystem, it causes all sorts of problems if the permissions are modified. One of my collaborators, @off-by-1-error opened an issue when we noticed no unblob-produced files were executable. That issue got fixed (thanks), but I recently noticed the {FILE,DIR}_PERMISSION_MASKS were also changing permissions so I disabled that and then got a simple permissions test to pass when having unblob extract a tar archive with a variety of permissions.

Support HPC environments: My research is focused on large-scale analyses of firmware - I'm working with thousands of firmware images and running them on a supercomputer where I don't have root access. I'm able to use singularity (basically a very-limited docker alternative) so I can install software and run experiments at scale. After I use fakeroot to run my modified unblob to preserve permissions and build an archive of a root filesystem, I then I feed the filesystem into PANDA.re with a custom kernel and emulate the target.

I don't have any particular asks for the unblob team - I'm grateful you all have tackled the hard problem of filesystem extraction and helped with the issues we've opened! If you have any interest in supporting any of the use cases I mentioned above, I'd certainly prefer your implementations over mine. Any analyses you'd like to build for identifying linux root filesystems or changes to adding the ability to produce tar archives with correct permissions would be awesome. But I'm not sure if those would be broadly useful to other users.

I do think I found an unblob bug around symlink handling and the MaliciousSymlinkRemoved error triggering far more often than it should, but I didn't yet have a chance to minimize a test case and open an issue. I removed that logic in my unblob fork and instead used a fork of the symlinks(8) utility to rewrite symlinks to be relative to a given directory during extraction. Unblob changes here. EDIT: I opened #761 to discuss some of the symlink bugs I'm seeing with unblob, but that's not actually related to the change here that just makes all the symlinks relative.

Let me flip the question around and ask if there's anything I can do to help you - I'm running both unblob and binwalk at scale on thousands of firmware images. I'm creating tar archives for everything that looks like a linux root filesystem and then I'm diffing the archives. I've done some spot checks and think unblob (with my patch to permission handling) is generally extracting files with correct permissions. I'm still investigating differences I see in files extracted between the tools. I haven't done much work to fix issues I've seen with binwalk and I know it's producing more files than it should right now (mostly due to recursive extraction going too deep). On my last test with ~5k FW I saw binwalk find 50,789 files that unblob didn't and unblob find 37,708 files that binwalk didn't. From a few manual spot checks, I believe unblob is producing significantly better output than binwalk.

I'm currently re-running my extraction at scale with #755 + #756 applied on my fork - I'll report back on how well those work and open new issues if I run into any other errors at runtime. But if there's anything else you'd like me to check at scale, let me know!

I like the direction you're exploring with permission as metadata. If that was working, I think we'd be able to just run unblob (with no permissions), consume the metadata and then build the filesystem archive from there.

I love projects that have a good scope and stick to it. It makes since that you wouldn't be interested in those use cases. I can collect some performance data and share it next time I run on the full corpus, I'd guess a median time for unblob is ~2s vs ~10s for binwalk. In one extreme case unblob extracted a filesystem in 10s that binwalk took 300s to do.

These symlinks bugs might be above my pay grade, I opened a PR with some fixes in #763, but I don't love what I built. In testing I found the symlinks utility (with patches to support rewriting links relative to a directory) wasn't working well - dangling symlinks weren't updated and would then point outside the extraction directory. I certainly won't be offended if you want to throw away all my code and build your own fix - a more unified interface + comprehensive tests for it would probably be a better design. If you're going to go that direction let me know and I can share some unit tests and minimal inputs I created while hacking on that PR.

As for the comparison between binwalk and unblob, I'm looking within identified root directories and comparing the files that are present. I don't have a list of inputs blobs that unblob didn't extract, just files that are/aren't present in the output. With all the changes on my branch (#755, #756 + an additional fix for directories, and #763) unblob is looking quite good - there are a bunch of files only present in the unblob extractions (15,527 in my last run) and the files produced by binwalk that don't map directly to a file in the unblob extraction are either:

1. Binwalk extracts files into the filesystem root directory while unblob has the file in a directory - spot checking makes me think unblob is correct here
2. Binwalk extracts files with non-printable characters in the name. Looks like junk.

Without #755, #756 + my additional fix for it, a few extractors were sometimes failing and many files were missing when that happened. Without #763 a large number of valid symlinks were missing.

I have a few non-public firmware corpora, but unfortunately none of them are mine so I can't redistribute them. I'm currently testing with the corpus from Greenhouse. But when I find failures I'm usually able to find the firmware online somewhere and share links to individual files.

Just wanted to say thanks for the close review of my PR and support for all the issues I'm opening. And sorry again for the confusion in that first big PR I opened, hopefully the smaller PR and issues with PoCs will be more useful for y'all.

I know you all have some different goals with extracted file permissions so I wanted to ask about this before I bother cleaning up code and opening PRs: if I'm expanding extractors and the filesystem class to better preserve permissions, would you be interested in getting any of those changes into unblob? Of course the permission bits you're adding would change the final permissions, but if other bits (e.g., o+w) were set in the original filesystem, that would produce a difference if the extractor handled it correctly.

For example, rehosting@2e4f43a expands Filesystem.write_chunks to take a mode arg and rehosting@5a538ed changes the yaffs extractor to create files with the mode as specified in the yaffs filesystem.

Unrelated to that, I also wanted to share some results from large-scale comparisons with Binwalk - in my analysis I'm running both a slightly forked Binwalk) and Unblob with my changes*, looking for directories that seem to be the root of a linux filesystem (e.g., checking for standard directories and at least a few executable files), then selecting the largest good-looking directory found by each extractor.

With this approach, I get:

19,946 byte-for-byte identical extractions
2,841 no filesystem identified in either extraction
2,358 filesystem only identified in unblob extraction
599 same files, distinct metadata (ownership, permissions, symlink destinations)
96 same files extracted but file sizes are different
95 filesystem only identified in binwalk extraction
32 different number of files between extractions (Unblob finding more)
23 different number of files (Binwalk finding more)

* My unblob changes now include swapping out a few 7z extractors for the format-specific extractors used by Binwalk as 7z unfortunately can't preserve file permissions

For example, rehosting@2e4f43a expands Filesystem.write_chunks to take a mode arg and rehosting@5a538ed changes the yaffs extractor to create files with the mode as specified in the yaffs filesystem.

I think both of these changes make sense. What's your take @e3krisztian ? We expose the mode for carve so we might as well for write_chunks.

• My unblob changes now include swapping out a few 7z extractors for the format-specific extractors used by Binwalk as 7z unfortunately can't preserve file permissions

Can you expand a bit ? Which format-specific extractors are you using ?

95 filesystem only identified in binwalk extraction

Do you know the kind of filesystems making these 95 entries ? I would suppose it's custom squashfs that our sasquatch fork can't cover but would be happy to know the exact details.

2,841 no filesystem identified in either extraction

Is it because they're encrypted or you're observing some filesystems that are not supported by the general public version of unblob ?