Dynamic Filesystem Transformation Layer

A FUSE-based filesystem that provides dynamic filtering, on-the-fly content transformation, and virtual organizational views over existing filesystems—all without copying files or consuming additional storage.

Tests Coverage Python License

ShadowFS creates a "shadow layer" over your existing filesystems, enabling:

• Dynamic Filtering: Show/hide files based on runtime-configurable rules
• On-the-Fly Transformation: Convert, compress, encrypt, or template files transparently during read
• Virtual Organization: Create multiple directory structures (by type, date, tags) over the same files
• Zero Storage Overhead: Files remain in place—no copying, no duplication

Think of it as "virtual filesystem middleware" that sits between applications and your files, providing programmable views and transformations.

# Install dependencies
pip install fusepy pyyaml jinja2
# Clone and install
git clone https://github.com/andronics/shadowfs.git
cd shadowfs
pip install -e .
# Run tests
pytest tests/ -v
# Mount with configuration
shadowfs --config shadowfs.yaml --mount /mnt/shadowfs

Hide build artifacts and clutter:

rules:
 - name: "Hide build files"
 type: exclude
 patterns:
 - "**/__pycache__/**"
 - "**/node_modules/**"
 - "**/dist/**"

Read Markdown as HTML on-the-fly:

transforms:
 - name: "Markdown to HTML"
 pattern: "**/*.md"
 type: convert
 from: markdown
 to: html

Organize by date, camera, tags simultaneously:

virtual_layers:
 - name: by-date
 type: date
 date_field: ctime
 - name: by-camera
 type: classifier
 classifier: exif

Current Phase: Phase 3 Complete (Integration - Transforms)

• ✅ Phase 0: Development Infrastructure
• ✅ Phase 1: Foundation Layer (path utilities, file operations, validators)
• ✅ Phase 2: Infrastructure Layer (config, cache, logging, metrics)
• ✅ Phase 3: Integration - Transforms (compression, format conversion, templates)
• 🚧 Phase 4: Integration - Virtual Layers (in progress)
• ⏳ Phase 5: Application Layer (FUSE operations)
• ⏳ Phase 6: Production Readiness

Test Coverage: 187 tests passing with ~97% average coverage on completed phases.

ShadowFS follows a 4-layer architecture compliant with Meta-Architecture v1.0.0:

┌─────────────────────────────────────────┐
│ Layer 4: Application │
│ (FUSE operations, CLI, control API) │
├─────────────────────────────────────────┤
│ Layer 3: Integration │
│ (transforms, rules, virtual layers) │
├─────────────────────────────────────────┤
│ Layer 2: Infrastructure │
│ (config, cache, logging, metrics) │
├─────────────────────────────────────────┤
│ Layer 1: Foundation │
│ (path utils, file ops, validators) │
└─────────────────────────────────────────┘

Key Components:

• Transform Pipeline: Chain multiple transforms (compression → encryption → conversion)
• Rule Engine: Pattern-based file visibility control
• Virtual Layers: Multiple organizational views over same files
• Multi-Level Caching: 3-tier cache (attributes, content, transforms)

shadowfs:
 version: "1.0"
 sources:
 - path: /data/documents
 priority: 1
 readonly: true
 rules:
 - name: "Hide hidden files"
 type: exclude
 pattern: "**/.*"
 transforms:
 - name: "Auto-decompress"
 pattern: "**/*.gz"
 type: decompress
 algorithm: gzip
 virtual_layers:
 - name: by-type
 type: classifier
 classifier: extension
 cache:
 enabled: true
 max_size_mb: 512
 ttl_seconds: 300

• Compression: gzip, bz2, lzma (compress/decompress)
• Format Conversion: Markdown→HTML, CSV↔JSON, YAML→JSON
• Templates: Jinja2 template rendering with context
• Extensible: Plugin architecture for custom transforms

• Classifier: Organize by extension, size, MIME type
• Date-based: Hierarchical date structures (YYYY/MM/DD)
• Tag-based: Organize by metadata tags
• Pattern-based: Rule-driven classification

• 3-tier caching: Attributes (60s), Content (300s), Transforms (600s)
• Async operations: Thread pool for I/O
• Lazy evaluation: Transforms only applied when accessed
• Connection pooling: Reuse file handles

• CLAUDE.md : Complete project documentation
• PLAN.md : Implementation roadmap and progress
• docs/architecture.md : Technical architecture (Meta-Architecture v1.0.0 compliant)
• docs/virtual-layers.md : Virtual layer system design
• docs/middleware-ideas.md : Future middleware extensions
• docs/typescript-type-discovery.md : Conceptual foundation

# Run all tests
pytest tests/ -v
# Run with coverage
pytest tests/ --cov=shadowfs --cov-report=html
# Run specific test suite
pytest tests/transforms/ -v
pytest tests/integration/ -v
# Run with specific coverage target
pytest tests/transforms/ --cov=shadowfs/transforms --cov-report=term-missing

Current Test Stats:

• Foundation tests: Coming in Phase 1 completion
• Infrastructure tests: Coming in Phase 2 completion
• Transform tests: 120 tests, ~97% coverage
• Pipeline tests: 34 tests, 99.37% coverage
• Base transform tests: 33 tests, 98.82% coverage

• Python 3.11+
• FUSE (libfuse3-dev on Debian/Ubuntu)
• Optional: markdown, pyyaml, jinja2 (for specific transforms)

# Create virtual environment
python -m venv venv
source venv/bin/activate
# Install development dependencies
pip install -e .
pip install -r requirements-dev.txt
# Install pre-commit hooks
pre-commit install
# Run tests
pytest tests/ -v

• Black: Code formatting
• isort: Import sorting
• flake8: Linting
• mypy: Type checking
• pytest: 100% coverage target
• pre-commit: Automated checks

The "shadow filesystem" concept was inspired by TypeScript's .d.ts type discovery system—just as TypeScript creates a type layer over JavaScript, ShadowFS creates organizational and transformation layers over filesystems.

• FUSE : Filesystem in Userspace
• fusepy : Python FUSE bindings
• UnionFS : Union filesystem (view composition inspiration)
• EncFS : Encrypted filesystem (transform inspiration)
• TagFS : Tag-based filesystem (virtual layer inspiration)

Contributions welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Write tests for new functionality
4. Ensure all tests pass with good coverage
5. Follow code quality standards (black, isort, flake8, mypy)
6. Submit a pull request

MIT License - See LICENSE file for details.

Project Maintainer: Stephen Cox (andronics)

Repository: https://github.com/andronics/shadowfs

Status: Design & Implementation Phase Version: 0.1.0-alpha Last Updated: 2025年11月11日