Yes, that has been attempted in the past, it is very slow.

For generating tokens you're I/O bound. Loading the data from RAM to VRAM and then from VRAM into the GPU is going to be slower than just loading the weights from RAM into the CPU.

Would on-device dynamic decompression be worth looking into? Model parameters, KV etc. are fairly compressible even before quantization. So one would load a compressed model to VRAM (faster I/O) and dynamically decompress the next layer weights & cached KV in parallel to the inference step. For batches sized N, keep up to N decompressed layers in a circular buffer as values propagate, given the data hazard between layers. Futhermore, a just-in-time prefetcher thats loads RAM->VRAM the next compressed portion of a large model, overwriting stale data, could compensate for the I/O latency. Does this make any sense?