@antirez could you have a look at it ?

Hi, the PR itself has a few quality issues but especially it is not clear why it would be useful for the proejct as a whole given that we convert from DS4 hugging face formats.

Quantization is successful.

@antirez Thank you for the quick response, let me explain.

• our sft/RL model of deepseek v4 has embedding layer (bf16 or int32), while deepseek model has embedding with type int64

• since we are running in Hopper platform , our expert weight stored with E4M3 FP8 weight and weight scale stored with FP32 for best performance (which can verified in SGLang):

  sglang fp8 serving

  Customer DSV4 sglang fp8 serving in Hopper platform with identity injectioin, private/public knowledge injection and enhanced security shield module
  

• Huggingface model is not SGLang compatible version, while our version is; and huggingface does not consider convert SFT/RL model from Bf16 to FP8 variants

The model is tuned specifically to handle Candonese, Chinese madarin and English efficiently.

Hi @antirez I can make sure the modification can generate correct model checkpoint for ds4. Wish your attention.

GB10 (2 bit dsv4-sft-rl, 15 toks/sec) :

Serving with raw model (no system prompt):