• the chat convo: https://chatgpt.com/share/68d0a6be-f704-800d-ac58-540a42b6bc35

1. Bootstrap with machine-readable sources

   • mime-db → seed MIME ↔ extensions (single JSON).
   • shared-mime-info → parse XML for categories + magic patterns.
   • PRONOM/DROID → unzip signature files; extract hex patterns + offsets + PUIDs.
   • libmagic → parse /usr/share/file/magic text; secondary support.
   • ffprobe → programmatically list containers/codecs from your own FFmpeg build (ffprobe -formats -codecs -protocols -of json).

2. Enrich with semi-structured sources

   • Wikidata → SPARQL queries for "software X supports extension Y", "format family", "developer", etc.
   • Wikipedia → per-page infobox scrape for missing descriptions/aliases (cache and manual review).

3. Human-curate deltas

   • Where sources disagree, keep priority order: IANA/mime-db > shared-mime > PRONOM/libmagic > Wikidata > Wikipedia > FileInfo.
   • Open a "review" sheet for oddities (e.g., .bin overlapping meanings).

4. Normalization rules

   • ext lowercase, no dot.
   • mime unique, lowercase; prefer mime-db entries.
   • category map from shared-mime-info "generic-icons" (image, video, audio, text, app → collapse to your enum).
   • magic store as normalized hex with wildcards; keep source in notes.
   • open_with: cap to 3–5 popular, per OS; source = Wikidata/vendor.

5. Versioning

   • Store raw snapshots in /sources/... with dates.
   • Build a deterministic pipeline (same inputs → same JSONs). Emit a build_id with timestamps + git SHA.

• mime-db (JS/JSON): gives you mime.full and extensions[]. Map into mime.schema.json and use reverse index to tee up filetype.mime.

• shared-mime-info (XML db):

  • Fields: <mime-type type="image/heic">, <glob pattern="*.heic"/>, <magic> with <match value="..." offset="...">.
  • Use this to fill filetype.magic[], category, and alternate extensions.

• PRONOM/DROID (signature files):

  • XML with byte sequences (Pos, ByteSequenceValue) and PUIDs.
  • Perfect for magic_signature.schema.json; map PUID → id, include extensions.

• ffprobe (your build):

  • ffprobe -hide_banner -formats -of json → mux/demux flags for containers.
  • ffprobe -hide_banner -codecs -of json → codec names + decoders/encoders.
  • ffprobe -hide_banner -protocols -of json → protocols list. Populate container, codec, and your taxonomy buckets 10–11–14.

• Wikidata SPARQL (JSON results, no scraping):

  • Query: apps that open a given extension; or file formats with filename extension ".heic".
  • Populates software.handles_extensions[] and filetype.open_with[].

• I/O: always save raw → /sources/{provider}/{date}/... (don’t parse in-place).
• Parser: pure functions from raw to normalized records matching your schemas.
• Joiners: merge by normalized keys (ext, mime.full) with priority rules.
• Emitted: one JSON per entity type (/build/filetypes.json, mimes.json, ...) and (optionally) one JSON-per-item for static page generation.

mime-db → mime + filetype seeds

import json, requests
db = requests.get("https://raw.githubusercontent.com/jshttp/mime-db/master/db.json").json()
mimes, ext_to_mimes = [], {}
for full, meta in db.items():
 exts = meta.get("extensions", [])
 mimes.append({"type": full.split("/")[0], "subtype": full.split("/")[1], "full": full, "extensions": exts})
 for e in exts:
 ext_to_mimes.setdefault(e.lower(), set()).add(full)
# seed filetypes from ext_to_mimes
filetypes = [{"id": ext, "ext": ext, "name": f".{ext.upper()} file", "category":"other", "mime": sorted(list(m))} for ext,m in ext_to_mimes.items()]

ffprobe → containers/codecs

ffprobe -hide_banner -formats -of json > formats.json
ffprobe -hide_banner -codecs -of json > codecs.json
ffprobe -hide_banner -protocols -of json > protocols.json

shared-mime-info XML → magic

from lxml import etree
root = etree.parse("freedesktop.org.xml")
sigs = []
for mt in root.findall(".//mime-type"):
 t = mt.get("type")
 for m in mt.findall(".//magic//match"):
 sigs.append({
 "id": f"{t}-{m.get('value')[:8]}",
 "hex": m.get("value").upper().replace("\\x"," ").strip(),
 "offset": int(m.get("offset","0")),
 "meaning": t,
 "extensions": [g.get("pattern").lstrip("*.") for g in mt.findall(".//glob")]
 })

Wikidata SPARQL (apps that open HEIC)

SELECT ?app ?appLabel WHERE {
 ?fmt wdt:P1195 "heic". # filename extension
 ?app wdt:P1072 ?fmt. # software supports file format
 SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}

• filetype.ext/name/mime/category/magic/open_with/related → mime-db + shared-mime-info + PRONOM + Wikidata
• codec.kind/profiles/common_containers → ffprobe + Wikipedia
• container.extensions/mime/streams_supported → ffprobe formats + specs
• magic_signature.hex/offset/meaning → PRONOM/libmagic/shared-mime-info
• software.platforms/handles_extensions → Wikidata + vendor docs
• support.browsers/oses → Can I Use/MDN (only for a handful like AVIF/WebP/HEVC)

• Licensing: PRONOM is free to use but credit; mime-db is MIT; shared-mime-info is LGPL-2.1 data; Wikipedia/Wikidata are CC-BY-SA/CC0. Attribute where required.
• Rate limits: cache requests, backoff; for Wikipedia/Wikidata use official APIs, not HTML.
• Consistency: extensions are many-to-many to MIME; your UI must handle multiple MIME per ext.
• Ambiguity: generic extensions (.bin, .dat) → keep but mark category: other, and do not auto-suggest risky openers.

• https://github.com/serpdownloaders/codecs