Chip'n Tell

Match user-uploaded audio samples from real Atari ST recordings back to the original .sndh files.

Usage tip: Use a program to monitor the sound output as an input and run live matching in the background while watching demo shows. On Linux you can do this in pavucontrol.

This is the code that runs https://cat.sync.wtf

The rest of this README concerns those who want to self-host or develop on top of this codebase.

Quick Start

# Start all services
podman-compose -f docker-compose.yaml up -d
# Initial setup
podman exec sndh_register_1 python3 /sndh/fingerprinter/__init__.py init

Running Things

# Start all services
podman-compose -f docker-compose.yaml up -d
# Start individual services
podman-compose -f docker-compose.yaml up -d db backend register
# Run fingerprinter inside container
podman exec sndh_register_1 python3 /sndh/fingerprinter/__init__.py <command>
# AudFPrint commands
podman exec sndh_register_1 python3 /sndh/fingerprinter/__init__.py init
podman exec sndh_register_1 python3 /sndh/fingerprinter/__init__.py fingerprint /wav/path/to/file.wav
podman exec sndh_register_1 python3 /sndh/fingerprinter/__init__.py match /wav/path/to/excerpt.wav
podman exec sndh_register_1 python3 /sndh/fingerprinter/__init__.py list
# Register all .sndh files (run inside register container)
podman exec sndh_register_1 /sndh/register.sh /sndh_lf # Sequential
podman exec sndh_register_1 /sndh/register.sh -j 8 /sndh_lf # 8 parallel jobs
podman exec sndh_register_1 /sndh/register.sh -j $(nproc) /sndh_lf # Use all cores
# After registration completes, build the hash table for fast matching
podman exec sndh_register_1 python3 /sndh/fingerprinter/__init__.py build-ht -o /wav/hash_table.pklz

Scraping New Songs from sndh.atari.org

New individual songs are published on sndh.atari.org between full archive releases. These are automatically scraped and registered separately from the main sndh_lf/ archive.

Automated Scraping (Cron)

The scraper runs automatically every day at 2 AM via cron in the register container. It:

1. Fetches the updates table from sndh.atari.org
2. Downloads new .sndh files to sndh_new/
3. Tracks which song IDs have already been processed

Manual Scraping

# Scrape and download new songs from sndh.atari.org
podman exec sndh_register_1 python3 /sndh/scrape_new_songs.py
# Register the scraped songs (runs psgplay, fingerprinting, and builds hash table)
podman exec sndh_register_1 /sndh/register.sh -j 8 /sndh_new
# Verify the songs are in the database
podman exec sndh_register_1 python3 /sndh/fingerprinter/__init__.py list

Cleanup Before Full Archive Updates

When a new full archive is published, clean out the individually scraped songs before re-registering:

# Remove all files from sndh_new/
podman exec sndh_register_1 /sndh/cleanup_sndh_new.sh
# Then re-register the full archive
podman exec sndh_register_1 /sndh/register.sh -j 12 /sndh_lf

How It Works

• New songs are downloaded with their original filenames (e.g., YQN_-_GemTOS_2026.sndh)
• Processed files go into sndh_new/ directory, separate from sndh_lf/
• The register.sh script automatically processes sndh_new/ alongside sndh_lf/
• Song IDs are tracked in sndh_new/.processed_ids.txt to avoid re-downloads
• Files are removed during full archive re-registration via cleanup_sndh_new.sh

Hash Table (.pklz)

Matching uses a pre-built AudFPrint hash table for speed. Without it, matching falls back to building the hash table from the database on every query (minutes instead of seconds).

Pipeline: register.sh stores fingerprints in DB → build-ht reads DB → writes /wav/hash_table.pklz → match loads .pklz

# Build hash table from DB contents
podman exec sndh_register_1 python3 /sndh/fingerprinter/__init__.py build-ht -o /wav/hash_table.pklz
# Match uses the pre-built table automatically; if missing, builds from DB (slow, logged as warning)
podman exec sndh_register_1 python3 /sndh/fingerprinter/__init__.py match /wav/path/to/excerpt.wav

Rebuild: After re-registering songs (e.g., after composer normalization changes), rebuild the hash table:

# 1. Delete sha256 markers to force re-registration
find /wav -name "*.sha256" -type f -delete
# 2. Re-run registration
podman exec sndh_register_1 /sndh/register.sh -j 12 /sndh_lf
# 3. Truncate DB (songs have UNIQUE constraint on name)
podman exec sndh_register_1 python3 -c "
import psycopg2, json
c = json.load(open('/sndh/fingerprinter/config.json'))
conn = psycopg2.connect(host=c['host'], user=c['user'], password=c['password'], dbname=c['database'])
conn.cursor().execute('TRUNCATE songs CASCADE'); conn.commit(); conn.close()
"
# 4. Re-register
podman exec sndh_register_1 /sndh/register.sh -j 12 /sndh_lf
# 5. Rebuild hash table
podman exec sndh_register_1 python3 /sndh/fingerprinter/__init__.py build-ht -o /wav/hash_table.pklz

Skip mechanism: register.sh uses .sha256 marker files in /wav/ to skip already-processed songs. If a marker file exists and contains the matching SHA256 hash of the .sndh file, the song is skipped. To force re-processing, delete the marker files.

JavaScript Linting

Linting runs in a container using oxlint (no Node.js installation required).

# Run linting
./lint-js.sh
# Via pre-commit (runs on staged files)
.venv/bin/pre-commit run js-lint
# Install pre-commit hooks
.venv/bin/pre-commit install

Pre-commit hooks are configured in .pre-commit-config.yaml. JavaScript files in backend/static/ are linted automatically on commit.

REST API

Health Check

GET /health

Response:

{
 "status": "ok",
 "songs_count": 9931
}

List/Search Songs

GET /api/songs?q=Nibbel&composer=Techno&limit=50&offset=0

Parameters:

• q - Search song name (case-insensitive)
• composer - Filter by composer (case-insensitive)
• limit - Max results (1-200, default 50)
• offset - Pagination offset (default 0)

Response:

{
 "songs": [
 {
 "id": 1,
 "name": "Techno/Dentro.sndh.1",
 "sndh_hash": "abc123...",
 "title": "Dentro",
 "composer": "Techno"
 }
 ],
 "total": 9931,
 "limit": 50,
 "offset": 0
}

Get Song Details

GET /api/songs/{id}

Response:

{
 "id": 1,
 "name": "Techno/Dentro.sndh.1",
 "sndh_hash": "abc123...",
 "title": "Dentro",
 "composer": "Techno",
 "source_path": "Techno/Dentro.sndh.1.in.wav",
 "created_at": "2025年05月22日T10:00:00"
}

Upload Audio for Matching

POST /api/upload
Content-Type: multipart/form-data
file: <audio file>
min_matches: 10 (optional, default 10)

Accepts any format ffmpeg can decode (MP3, WAV, OGG, FLAC, etc.) Max file size: 50MB Rate limit: 5 uploads/minute per IP

Response:

{
 "matches": [
 {
 "song_name": "Techno/Dentro.sndh.1",
 "song_id": 1,
 "title": "Dentro",
 "composer": "Techno",
 "matches": 4080,
 "match_rate": 99.0,
 "coverage": 85.3,
 "confidence": 93.5,
 "time_offset": 0
 }
 ],
 "file_name": "recording.mp3",
 "file_size": 2189725
}

Matching quality indicators:

• Strong match: 20+ aligned hashes, 0.5%+ rate, 25%+ confidence
• Weak match: <10 aligned hashes, <0.1% rate (likely false positive)
• Clean WAV self-match: 100% rate, thousands of aligned hashes

Clicking the song title or path in the UI opens the song's search results on sndh.atari.org.

Database Schema

songs table

fingerprints table

AudFPrint Configuration

Key parameters and their recommended values:

--density 20.0 # Target hashes per second
--hashbits 20 # Bits per hash (1M distinct fingerprints)
--bucketsize 100 # Entries per bucket
--maxtimebits 14 # Time range up to ~6 minutes
--min-count 10 # Minimum matching hashes for a match (MIN_MATCHES)
--match-win 2 # Maximum frame skew for matches (matcher.window)

Application-level thresholds:

MIN_MATCHES = 10 # Minimum aligned hashes to consider a match
MIN_MATCH_RATE = 0.001 # Minimum match rate (0.1%) to consider a match

Matching quality indicators:

• Clean WAV self-match: 100% rate, thousands of aligned hashes
• Real recording match: 20+ aligned hashes, 0.5%+ rate
• False positive threshold: <10 aligned hashes, <0.1% rate

Protection Limits

• File size: 50MB max
• Rate limit: 5 uploads/minute per IP
• Disk space: 100MB free required
• Database capacity: 10,000 songs max (configurable)

GPU Acceleration

The system automatically uses GPU-accelerated AudFPrint modules when available, providing 2-15x speedup for fingerprinting and matching operations.

Requirements

Host System (for local development)

• NVIDIA GPU with CUDA support
• NVIDIA drivers installed
• NVIDIA Container Toolkit installed

Install NVIDIA Container Toolkit (Ubuntu/openSUSE):

# Add NVIDIA package repository
distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
 && curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - \
 && curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
# Install NVIDIA Container Toolkit
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo systemctl restart docker

For openSUSE:

sudo zypper install nvidia-container-toolkit
sudo systemctl restart docker

Verify installation:

nvidia-smi # Should show GPU information
docker run --rm --gpus all nvidia/cuda:13.2.2-base nvidia-smi # Should show GPU info in container

Container Images

The backend and register images now use nvidia/cuda:13.2.2-base-ubuntu22.04 as base and include:

• CUDA 13.2.2 runtime (compatible with CUDA 13.0-13.3)
• CuPy (cupy-cuda13x) for GPU-accelerated NumPy
• cuSignal for GPU-accelerated signal processing

Running with GPU Support

Local Development (Podman)

# Enable GPU in podman (may require root or user configuration)
podman system connection default --set-default
# Start services with GPU access
podman-compose -f docker-compose.yaml up -d
# Verify GPU is accessible in container
podman exec sndh_backend_1 nvidia-smi
podman exec sndh_register_1 nvidia-smi

Note: Podman may require additional configuration for GPU access. See Podman GPU documentation.

Production (Docker on 192.168.0.2)

The production host (192.168.0.2) uses rootless Docker. To enable GPU support:

1. Install NVIDIA Container Toolkit:

ssh -i syncasm -o IdentitiesOnly=yes syncasm@192.168.0.2
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo systemctl restart docker

2. Configure Docker daemon for GPU:

# Edit /etc/docker/daemon.json (create if doesn't exist)
sudo mkdir -p /etc/docker
sudo nano /etc/docker/daemon.json

Add the following:

{
 "runtimes": {
 "nvidia": {
 "path": "nvidia-container-runtime",
 "runtimeArgs": []
 }
 },
 "default-runtime": "nvidia"
}

3. Restart Docker:

pkill -u syncasm dockerd
pkill -u syncasm rootlesskit
sleep 3
dockerd-rootless.sh &

4. Start services:

cd /home/syncasm/sndh
docker compose -f docker-compose.prod.yaml up -d

5. Verify GPU access:

docker exec sndh-backend-1 nvidia-smi
docker exec sndh-register-1 nvidia-smi

GPU Configuration Options

The GPU acceleration can be controlled via environment variables:

Example - Limit to GPU 0 only:

# In docker-compose.yamlenvironment:- NVIDIA_VISIBLE_DEVICES=0

Monitoring GPU Usage

Check GPU utilization:

# On host
nvidia-smi
# In container
podman exec sndh_backend_1 nvidia-smi
docker exec sndh-backend-1 nvidia-smi

Check if GPU acceleration is active:

# Check fingerprinter logs
podman logs sndh_backend_1 | grep -i gpu
# Or check directly
podman exec sndh_backend_1 python3 -c "import sys; sys.path.insert(0, '/app/fingerprinter'); from fingerprinter import GPU_AVAILABLE; print('GPU:', GPU_AVAILABLE)"

VRAM Requirements

• Minimum: 500MB free VRAM (configurable in fingerprinter/__init__.py)
• Recommended: 1GB+ for production dataset (9,951 songs, 46M fingerprints)
• Hash table size: ~350-400MB in GPU memory

The system automatically falls back to CPU if insufficient VRAM is available.

Fallback Behavior

If GPU is not available (no NVIDIA drivers, no CuPy, or insufficient VRAM):

• System automatically uses CPU-only AudFPrint modules
• All functionality works normally (just slower)
• Logs a warning message: Using CPU-only AudFPrint modules

No manual configuration is required - the fallback is completely transparent.