Releases - atomdrift/filefacts - Codeberg.org

Features

• New formats: WebAssembly, OCI container images, Python sdist, Android DEX; better npm/gem/oci/xbps/gpkg detection.
• Recognizes SVG, Cypher/CQL, and Python template files.
• Windowed entropy metrics surface a concealed high-entropy region (peak + offset) even when overall entropy is low.
• Facts now carry byte-range spans pinpointing each measurement's evidence.
• Registry metadata gains a security_hold flag and version_removed metric.

Performance

• Skips the XOR seed search on source/script files with no sign of XOR logic.
• Deduplicates stng strings; cache self-cleans (prunes to 90% of capacity instead of a 30-day TTL).

• Hermetic open for cache-less testing
• Add missing DMG support file

• Added external reference extraction: PURLs, URLs, evidence, offsets, hashes.
• Added package/lockfile parsing for npm, PyPI, Cargo, Go, Ruby, Composer, Arch.
• Added new file types: lockfiles, go.mod, go.sum, .SRCINFO, DMG.
• Added package metadata and reference output views.
• Added source call-target metrics, including obfuscated command targets.
• Improved JS/PHP/protobuf/package/DMG identification.
• Improved Go buildinfo, Mach-O slicing, and rizin symbol offsets.
• Added pre-commit hook support.

• Program Identity extraction — pulls publisher/package metadata from CRX, npm, nupkg, Rust
  crate, wheel, and xpi archives
• JCL (z/OS Job Control Language) detection; PKGBUILD classified as shell
• Panic-safe binary parsing — goblin wrapped to fail soft on malformed PE/Mach-O/ELF instead of
  aborting
• Binary fixes — corrected Mach-O FAT and PE offsets; richer PE version/debug info
• Scanning rework — consolidated string scanning, improved wide-char handling
• Dependency updates and cleanup

New Signals

• Added richer package ecosystem identification: Android/Alpine APK, npm, Cargo crate, Ruby gem, Debian package, NuGet, VSIX, IPA, conda, egg, Arch/FreeBSD/macOS pkg.
• Added .deb identity features: package, version, architecture, maintainer, dependencies, installed size.
• Added .gem identity features: name, version, platform, homepage, authors, licenses, runtime/development dependency counts.
• Added more PE malware-relevant metrics, such as CLR managed resource count, max entropy, max size.

Better Evidence Anchoring

• ELF dynamic imports now include offsets into .dynstr.
• Mach-O dylib references and code signatures now include offsets.
• Source member symbols now include byte offsets.

Detection Quality

• Improved package disambiguation, especially Android vs Alpine .apk, npm .tgz, Arch/FreeBSD/macOS .pkg.
• Better AppleScript and pacman/AUR install-scriptlet detection.
• Reduced false positives by classifying unsupported OCaml, Vim, Lisp, SQL, Smali, patches/diffs, CSS-like files, and TS baselines as text
  instead of weakly guessing JS/Kotlin/Batch.

Robustness For Large Corpora

• Hardened Tree-sitter extraction against panics, deep ASTs, huge query output, and scanner-risk cases.
• Added metrics such as ast.depth_capped, source.query_limited.*, and source.ast_unavailable so skipped/truncated AST extraction is explicit.
• Binary string extraction reuses existing PE/ELF/Mach-O parses, reducing duplicate parsing work.

Breaking/Schema Notes

• FileType::Pkg became specific variants like PkgMacos, PkgFreebsd, and PkgArch.
• Symbol::Member now has optional offset.
• New source error stages: SourceParse, SourceExtract, SourceAstWalk.
• pe.cert_table_size moved from values to metrics.

Features

• Added Debian package metadata: package, version, architecture, maintainer, dependencies, and installed size.
• Added RubyGems metadata: name, version, platform, authors, licenses, homepage, and runtime deps.
• Split package ecosystems into distinct types, including Android/Alpine APK, npm, crate, conda, NuGet, IPA,
  VSIX, FreeBSD pkg, and Arch pkg.
• Added PE/.NET signals for managed resource entropy/size, suspicious VERSIONINFO identity text, .reloc
  overhang, and certificate table size.
• Improved AppleScript detection for plaintext macOS stealer scripts with random or unknown extensions.

Performance

• Reuses parsed PE/ELF/Mach-O objects for string extraction instead of parsing binaries twice.
• Skips duplicate stng symbol/import extraction because filefacts already walks symbols itself.

Bugfixes

• Hardened AST analysis against deeply nested source chains that could overflow the worker stack.
• Emits ast.depth_capped when hostile or generated source exceeds analysis depth.
• Reduced false positives by treating Vim, Lisp, SQL, Smali, diffs, and TS baseline files as text.
• Fixed archive disambiguation so legitimate .apk and .pkg ecosystem variants do not look like extension
  mismatches.

New language & format support

• Clojure and Windows Batch AST analysis (tree-sitter)
• AppleScript source detection
• BEAM bytecode and FreeBSD zstd .pkg detection
• binding.gyp parsed as JSON; Meson, Arch package metadata, and R Markdown/Quarto recognized

New facts & metrics

• Comments exposed as a matchable string tier; JS/TS/Go backtick strings no longer misread as comments; Lisp and
  Batch comment styles added
• New obfuscation AST metrics: operator density, numeric arrays, XOR loops, const/string-return functions, identity
  functions, and more
• Extension/content mismatch metrics name the transition (e.g. binary_as_image), with carve-outs for benign
  conventions
• VBA identifier metrics: mean length, entropy, distinct trigger count
• binary.overall_entropy emitted for every file type
• Java constructors, C# qualified names, and import aliases (import x as y) surfaced

Detection accuracy

• Stylesheets treated as text, no longer misparsed as JavaScript
• JS/Kotlin scoring fixed: var/let/const count for JS; === ignored in separator runs
• Python relative imports keep their . prefix — no more false PyPI matches
• PEs that goblin can't parse now get rizin section/import recovery

Performance & memory

• jemalloc tuned to release freed pages immediately: peak RSS −~60%
• JSON output streams to stdout instead of building in memory
• Tree-sitter queries compiled once and cached

API & maintenance

• Symbol objects slimmed: dropped source, decl, and rare rizin fields
• Dependency updates

Features

• Electron ASAR and Clojure support — new .asar archive parser and Clojure source
  identification.
• Generic JSON support — .json files are parsed into top-level keys (with a 75 KB cap;
  larger files stay analyzable as raw text).
• Broader file identification — more extensions in the fileid map, a dedicated PE
  extension map, and improved GitHub Actions workflow detection.

Fixes

• Fixed a PDF /Length stream-end calculation bug that mis-bounded stream bodies when
  endstream had no preceding EOL.

Hardening

• Capped AST recursion depth at 1000 to stop pathologically nested
  (generated/adversarial) source from overflowing the worker stack; ast.max_depth now
  saturates at the cap as a "deeply nested" signal.

Internal

• Dependency updates and code cleanup.

New facts

• Markdown extractor — markdown.first_heading and deduped markdown.github_repos[]
• file.basename / file.stem emitted whenever a path is supplied
• Wheel outer-filename parsing: whl.filename.{name_prefix,version,build,python_tag,abi_tag,platform_tag} from the PEP 427 basename
• PDB derived names — pe.debug.pdb.basename and pe.debug.pdb.stem

Hardening

• Tree-sitter scanner audit. Per-grammar ScannerAudit table re-verified against locked crate sources;
• Python now models the actual combined delimiter + indent byte
• Mach-O fat-binary panic fix — misclassified CAFEBABE Java .class no longer slices OOB on bogus arch.offset.
• Mach-O code-signature caps — Requirement-expression depth 32, plist depth 64, entitlement XML 1 MiB.
• CHM bounds arithmetic — directory / namelist / system walks converted to checked_add / checked_mul.
• bytes_at::{u16,u32,u64}_{le,be} helpers replace unchecked try_into().unwrap() reads.
• CLI — --help / --version no longer process::exit; readdir errors surface per-entry

Breaking API

• unsafe_code = "deny" (was warn).
• FileId fields private — use file_type(), source(), extension_mismatch(). #[non_exhaustive].
• ParseError.kind/.stage are now ErrorKind / Stage enums (JSON wire form unchanged).
• ArchiveMember regrouped into ArchiveCompression / ArchiveOwnership / ArchiveOffsets sub-structs.
• VBA sentinel flattened to filefacts::VBA_NON_LITERAL_SENTINEL; rizin module no longer a re-export façade.
• ParsedFile is #[must_use]; ParsedFile::strings() renamed to text().
• Sections::from_iter_sections removed in favour of standard FromIterator.

Breaking — schema v4 → v6

• Unified symbol model. The peer Imports, Exports, Functions, and Ast collections are collapsed into a single tagged Symbols enum with kinds Import, Export, Function,
  Call, Member, Bind, and Identifier. Function.kind is renamed to decl; Function.calls is renamed to callees. (v5)
• Strings split. Strings is replaced by two peers: Text (byte-scan ASCII + UTF-16LE — the strings(1) tier) and Literals (parser-extracted source string literals). The
  StringCategory enum and ExtractedString.category field are removed. (v6)

New languages

Perl, Groovy, Zig, Elixir, and Makefile join the tree-sitter language set.

Source-analysis improvements

• Typed call arguments. Calls now emit an Arg enum carrying the actual literal value (String { value }, Number { text, value, radix }, Identifier { name }, Bool { value
  }, Template { value }, ...), so rules can match on the literal directly instead of correlating against literals by offset.
• New Identifier symbol kind. Every bare-identifier leaf (variables, parameters, type names, call targets) is captured with first-seen offset, enabling naming-pattern
  rules.
• Subscript-as-member. obj["constructor"]"constructor" now folds into a member chain instead of dropping out as a dynamic access, so JS sandbox-escape patterns
  land as Member symbols.
• Scanner-overflow guard. Files >2 MB, or Python sources whose estimated indent stack exceeds 450 levels, are skipped before parsing to avoid tree-sitter's
  external-scanner 1024-byte buffer abort in release builds.

CLI

• New view flags: --text, --literals, --symbols, --calls, --members, --binds, --identifiers.
• Removed: --strings, --ast (replaced by the above).