cl-uastc

A Common Lisp library for decoding and transcoding UASTC GPU texture blocks.

UASTC is a 19-mode, 4x4 pixel, LDR-only subset of ASTC designed for universal transcoding to GPU-native compressed formats. This library operates on raw 16-byte blocks and includes optional container parsing for KTX2 and .basis files.

Status

All transcode targets complete.

The RGBA8 decode path produces pixels directly. The ASTC transcoding path produces hardware-native ASTC ×ばつ4 blocks that decode to identical pixels, with no quality loss, suitable for GPU upload on devices with ASTC hardware support.

The BC7 transcoding path produces hardware-native BC7 blocks suitable for GPU upload on all desktop GPUs (and most mobile GPUs via Vulkan). Output is byte-exact against the Basis Universal reference transcoder.

Supported Modes

All 19 UASTC modes are decoded and transcoded:

• Modes 0, 1, 2, 5, 18 - 1-subset RGB (trit/quint/bit endpoints)
• Modes 3, 7 - Multi-subset RGB (2 or 3 subsets, partition patterns)
• Mode 4 - 2-subset RGB with precomputed partition patterns
• Mode 6 - 1-subset RGB, dual-plane (component select)
• Modes 9, 10, 11, 12, 13, 14 - RGBA modes (single/dual-plane, various quantizations)
• Modes 15, 16, 17 - Luminance-Alpha (single/dual-plane)
• Mode 8 - Solid color (void-extent)

Usage

(ql:quickload "cl-uastc")
;; Decode a single 16-byte UASTC block to 16 RGBA8 pixels (64 bytes)
(let ((block (make-array 16 :element-type '(unsigned-byte 8))))
 ;; ... fill block from your container parser ...
 (cl-uastc:decode-uastc-block-rgba8 block))
;; Transcode a single 16-byte UASTC block to a 16-byte ASTC 4x4 block
(cl-uastc:transcode-uastc-block-astc block)
;; Transcode a single 16-byte UASTC block to a 16-byte BC7 block
(cl-uastc:transcode-uastc-block-bc7 block)
;; Transcode/decode a whole image (raw block data, no container)
(cl-uastc:transcode-uastc-image block-data width height :target :astc) ; or :bc7, :rgba8

The input is a flat octet vector of 16-byte UASTC blocks in raster order. For RGBA8 output, the result is a flat octet vector of pixels (4 bytes per pixel) in raster order. For ASTC/BC7 output, the result is a flat octet vector of 16-byte compressed blocks in raster order.

Loading from Container Files

KTX2 and .basis containers are supported directly:

;; Load and transcode a KTX2 or .basis file (auto-detected)
(let ((file (cl-uastc:load-uastc-file #p"/path/to/texture.ktx2")))
 (format t "~Dx~D, ~D mip levels~%"
 (cl-uastc:uastc-file-width file)
 (cl-uastc:uastc-file-height file)
 (length (cl-uastc:uastc-file-levels file)))
 ;; Transcode mip 0 to BC7
 (cl-uastc:transcode-uastc-file file :target :bc7))
;; Access raw block data for a specific mip level
(multiple-value-bind (data width height)
 (cl-uastc:uastc-file-mip-data file 0)
 (cl-uastc:transcode-uastc-image data width height :target :astc))

Supported containers:

• KTX2 - Khronos standard (.ktx2); supercompression must be 0 (uncompressed)
• .basis - Basis Universal format; UASTC mode only (not ETC1S)

Testing

Decode correctness is verified against the Basis Universal reference transcoder using multiple test images covering 17 of 19 modes (modes 14 and 16 are extremely rare and use the same generic code path).

ASTC Transcoding

ASTC transcoding is verified by round-tripping: transcode UASTC → ASTC, decode with ARM's astcenc, then compare against our RGBA8 decode.

=== GLOBAL SUMMARY (4 test images, 61440 blocks) ===
 Mode 0: 827968 ch, mean=0.1260 max=1
 Mode 1: 20672 ch, mean=0.0440 max=1
 Mode 2: 2816 ch, mean=0.0969 max=1
 Mode 3: 3264 ch, mean=0.0643 max=1
 Mode 4: 72832 ch, mean=0.0862 max=1
 Mode 5: 200192 ch, mean=0.0901 max=1
 Mode 6: 1057856 ch, mean=0.0718 max=1
 Mode 7: 1728 ch, mean=0.0839 max=1
 Mode 8: 1920 ch, mean=0.0000 max=0
 Mode 9: 20352 ch, mean=0.0114 max=1
 Mode 10: 1004928 ch, mean=0.1273 max=1
 Mode 11: 156096 ch, mean=0.0408 max=1
 Mode 12: 83968 ch, mean=0.1760 max=1
 Mode 13: 139136 ch, mean=0.0000 max=0
 Mode 15: 2880 ch, mean=0.1240 max=1
 Mode 17: 64 ch, mean=0.1406 max=1
 Mode 18: 335488 ch, mean=0.0964 max=1

Max error ≤ 1 for all modes. The ±1 difference is integer rounding in endpoint unquantization; the transcoded ASTC blocks are bit-exact matches to basisu's reference ASTC output.

BC7 Transcoding

BC7 transcoding is verified by byte-exact comparison against Basis Universal's reference BC7 output across 36,864 blocks from 3 test images, covering all UASTC modes that appear in practice:

test_rgba.basis (256x256): 4096 blocks, 0 mismatches
test_complex_rgb.basis (512x512): 16384 blocks, 0 mismatches
test_rgba2.basis (512x512): 16384 blocks, 0 mismatches

To reproduce:

sbcl --load tests/transcode-astc-test.lisp # kodim23 pixel-exact test
sbcl --load tests/transcode-all-modes-test.lisp # multi-image all-modes test
sbcl --load tests/bc7-smoke-test.lisp # BC7 vs basisu byte-exact test

Performance

Benchmarks run on real test images covering all UASTC modes present in practice.

All numbers below were measured single-threaded on the perf/scalar-and-simd branch. It builds on the dependency-free optimizations already in master (per-file block compilation, a byte-gather extract-bits, comprehensive type declarations across the lookup tables and hot paths, and a table-driven byte reversal in the ASTC weight packer) with three further bit-exact scalar improvements: a single-pass word-wise bit-buffer in the ASTC weight packer and sequential forward bit-readers in the weight extractors. Net effect vs master (min-of-5, single-thread): ASTC ×ばつ faster, RGBA8 ~12%, BC7 ~6% (output byte-identical, all 100 tests pass). An AVX2 sb-simd decode path was also evaluated and rejected (see Design Notes).

Throughput

Throughput

• RGBA8 (full decode + interpolation): ~19-29 Mpix/s (single-thread)
• ASTC (lossless repack): ~23-26 Mpix/s (single-thread)
• BC7 (near-lossless, endpoint requantization): ~12-13 Mpix/s (single-thread)

Per-Mode Transcode Time

Per-mode transcode time

Most modes transcode in 1-3 μs/block. Multi-subset modes (3, 4, 7) are slightly slower due to partition pattern handling.

Allocation Pressure

Allocation per block

All three paths use dynamic-extent stack allocation for temporary buffers (weight arrays, endpoint arrays, quantization intermediates). Per-block allocation is minimal for common modes: ASTC ~0.03 KB, BC7 ~0.03-0.3 KB, RGBA8 ~0.07-0.3 KB. Multi-subset modes allocate more due to partition handling. The RGBA8 path uses pre-allocated buffers instead of struct allocation for the interpolated-block decode.

Full Image Transcode

Full image transcode

Results Table

Size Scaling

Size scaling

cl-uastc RGBA8 decode scales linearly with pixel count and is consistently faster than pngload: roughly ×ばつ faster at small/medium sizes, narrowing ×ばつ at multi-megapixel sizes but staying ahead across the full range (single-threaded). The values below are the mean across all five content patterns at each size; pngload decodes the matching PNG of identical content. UASTC's per-block cost is bounded by its mode mix, whereas PNG decode time swings widely with content (see the per-pattern chart below).

(All times single-threaded, mean of the five content patterns per size. RGBA8-vs-PNG = mean pngload time / mean RGBA8 time. pngload decodes a PNG of the same content; its time is highly content-dependent (e.g. at 81922 it ranges from ~1.1 s on noise to ~3.9 s on smooth-worley).)

Comparison: cl-uastc vs PNG Loading

cl-uastc RGBA8 decode is consistently faster than pngload (×ばつ) across the full size range tested (single-threaded). The more durable advantages are that UASTC decode is content-stable (cost bounded by mode mix, not compressibility) and that it produces GPU-native compressed formats (ASTC/BC7) that PNG cannot provide.

Size & Content Scaling

A second benchmark measures how single-threaded transcode time scales with image size (322 ... 81922) across five procedurally-generated content patterns (black, gradient, noise, perlin, smooth-worley). The generator (bench/gen-test-data.lisp) is adapted from a script by Nicolas Hafner (shinmera); it renders each pattern to SF3, converts to PNG via ImageMagick, then encodes UASTC KTX2 with the Khronos ktx CLI.

Transcode time vs image size

Time scales linearly with pixel count. Against pngload, RGBA8 decode is faster across sizes, with the largest margin on cheaper/smaller content, while the compressed targets (ASTC/BC7) trade some decode speed for GPU-native output that PNG cannot produce.

Transcode time by content pattern

Content matters as much as size: at 81922, uniform black transcodes in ~0.6 s (ASTC) while high-entropy noise takes ~2.8 s, because the UASTC mode mix (and thus per-block work) differs sharply between flat and detailed regions. This variety is what makes the set useful for exercising every mode path.

Running Benchmarks

sbcl --eval '(push #p"./" asdf:*central-registry*)' \
 --eval '(asdf:load-system "cl-uastc-bench")' \
 --eval '(cl-uastc-bench:run-benchmarks :output-dir #p"bench/results/")'
# Generate charts (requires matplotlib + seaborn)
python3 bench/plot-charts.py bench/results/bench-results.csv
# Or with gnuplot
cd bench && gnuplot plot-gnuplot.gp

Size & content scaling benchmark (requires the ktx CLI, ImageMagick with SF3 support, and the random-noise, nibbles, and pngload Quicklisp systems):

# 1. Generate the test images into bench/images/ (~2.2 GB for the full 322..81922 set)
sbcl --dynamic-space-size 8192 --non-interactive \
 --load bench/gen-test-data.lisp --eval '(run)'
# 2. Benchmark every pattern x size -> bench/results/bench-scaling.csv
sbcl --dynamic-space-size 10240 --non-interactive --load bench/bench-scaling.lisp
# 3. Generate the scaling + content charts
python3 bench/plot-scaling.py

Design Notes: Pure Scalar, Single-Threaded

cl-uastc is deliberately dependency-free, single-threaded, and free of SIMD intrinsics. Two avenues were evaluated and rejected:

• SIMD (sb-simd/AVX2). An AVX2 RGBA8 interpolation path was prototyped (all 16 texels of a channel in one 16-lane s16.16 vector, bit-exact, behind a feature flag). It ran ×ばつ slower than the scalar loop. UASTC's ×ばつ4 blocks are too small: the per-block cost of lane construction (make-s16.16), value extraction, and simd-pack boxing dwarfs the arithmetic it saves. It also coupled the build to an SBCL-specific contrib. Removed.
• Multithreading. Intentionally not used. The transcoder is a pure, embeddable per-block function; threading is left to the caller, who knows their own workload and scheduler.

The performance gains here come entirely from portable scalar techniques: per-file block compilation, byte-gather bit extraction, sequential forward bit readers, a table-driven byte-reversal weight packer, and comprehensive type declarations on the hot integer paths.

Project Structure

cl-uastc/
 cl-uastc.asd ; system definition
 cl-uastc-tests.asd ; test system definition
 cl-uastc-bench.asd ; benchmark system definition
 src/
 package.lisp ; package + exports
 bits.lisp ; bit extraction, replication, reversal
 tables.lisp ; spec tables, partition patterns, anchors
 uastc-block.lisp ; block parsing, mode decode, endpoints
 transcode-astc.lisp ; UASTC -> ASTC 4x4 lossless transcoding
 transcode-bc7.lisp ; UASTC -> BC7 near-lossless transcoding
 decode-rgba.lisp ; UASTC -> RGBA8 interpolation
 image.lisp ; whole-image transcode/decode convenience
 container.lisp ; KTX2 and .basis file parsing
 bench/
 package.lisp ; benchmark package
 bench.lisp ; benchmark suite (all modes, real images)
 gen-test-data.lisp ; procedural test-image generator (SF3->PNG->KTX)
 bench-scaling.lisp ; size + content scaling benchmark suite
 profile-sprof.lisp ; sb-sprof statistical profiler script
 plot-charts.py ; matplotlib/seaborn chart generator
 plot-scaling.py ; chart generator for size + content scaling
 plot-gnuplot.gp ; gnuplot chart script
 plot-interactive.html ; interactive Chart.js dashboard
 images/ ; generated test images (gitignored)
 charts/ ; committed chart PNGs for README
 tests/
 package.lisp ; test package definition
 bits-test.lisp ; bit utility unit tests
 block-test.lisp ; block parsing unit tests
 container-test.lisp ; KTX2 and .basis parsing unit tests
 census.lisp ; mode distribution analysis
 decode-test.lisp ; RGBA8 decode vs reference PNG
 transcode-astc-test.lisp ; pixel-exact ASTC roundtrip (kodim23)
 transcode-all-modes-test.lisp ; full ASTC roundtrip across all modes
 bc7-smoke-test.lisp ; BC7 byte-exact test vs basisu reference
 kodim23.basis ; test image (UASTC+KTX2)
 kodim23_raw.basis ; test image (raw UASTC+KTX2)
 kodim23_reference_rgba.png ; reference RGBA8 decode output
 kodim23_transcoded.astc ; ASTC transcode output (roundtrip test)
 kodim23_astc_decoded.png ; astcenc decode of transcoded ASTC
 test_rgba.basis ; 256x256 RGBA test image
 test_rgba.png ; reference PNG for test_rgba
 test_rgba2.basis ; 512x512 RGBA test image
 test_rgba2.png ; reference PNG for test_rgba2
 test_complex_rgb.basis ; 512x512 complex RGB test image
 test_complex_rgb.png ; reference PNG for test_complex_rgb

Scope

The primary transcode targets (ASTC and BC7) are implemented and verified. Together these cover virtually all modern GPU hardware: ASTC for mobile/embedded (ARM, Apple, Qualcomm) and BC7 for desktop (all DX11+ and Vulkan GPUs). Container parsing supports KTX2 and .basis files directly. The only transcode formats not implemented would be lower-priority ones like ETC1/2, BC1-5, or PVRTC, which are less commonly needed when you have BC7 + ASTC coverage.

References

• Khronos Data Format Spec v1.4, Ch.25 - UASTC format definition
• Basis Universal Wiki - UASTC LDR Spec - Partition tables, anchor indices, mode details
• Basis Universal - Reference C++ implementation

License

MIT