Thank you! I was thinking about the group_size/block... This is exactly what I was hoping for. If you don't mind having this as an experimental/academic feature, could this please be added? It would be very nice to be able to experiment with your quantizations with GGUF. And, being able to quantize and export the de-quantized model weights that are already "set up", using GPTQ, GPTQ-v2, etc, and converting and quantizing them to GGUF would make this a reality.

Back in the LLaMA-2 days, I was able to use OmniQuant's de-quantized ("fake") model-weights successfully in conversion and quantization to GGUF - also: at that time, GGUF had a built-in AWQ conversion script, so we also were able to convert models to GGUF using AWQ scales (this unfortunately is no longer supported, as no one used the feature but me...). But, I digress...

I know GGUF is compatible with other quantization schemes like GPTQ and AWQ as long as the group_size and block size are the same as that which GGUF uses. Being able to use GPTQ, GPTQ-v2, etc with GGUF would open the doors to much more accurate GGUF models. I really hope this can be added as an experimental/academic feature. Because, I really like llama.cpp's system agnostic approach, but I crave more accurate quantizations like GPTQ, GPTQ-v2, etc, and I know I am not alone. 😋

Exactly, as long as GGUF quantization (packing) is aligned with the gptq format, then I see very low to no loss. For example, don't convert gptq 4bit group size=128 to gguf 3bit and group_size=32. The bit and group_size should be aligned as much possible.

So, just to clarify: we can export the de-quantized GPTQ model-weights to BF16 via saving/exporting to MLX to use with other compatible quantization frameworks, such as GGUF. So, the academic feature request is, technically, already implicitly supported? 🤔👀

@joseph777111 If you are up to the task, please check the mlx export code. We already do full BF16 dequantzization. So you just need to copy the mlx export code and replace the mlx part with gguf equivalent! This would be a great feature/PR to have. Let me know. I can assist with you on the PR if you run into any problem.

Sorry, I never saw and was never notified that you messaged me. Please forgive my ineptness to reply. That sounds like a great idea. Unfortunately, I don't have time right now. But, if I ever find the time, I would love to pursue this. Thank you for suggesting this. 😋