-
Notifications
You must be signed in to change notification settings - Fork 75
Open
@Espere-1119-Song
Description
Describe the bug
When I try to apply tri_mix on qwen2.5-7B, I meet the error: Failed to apply TriangleMix: The attn_type tri_mix you specified is not supported.
I test with MInferenceConfig.get_available_attn_types(), but also miss the tri_mix.
How can I apply tri_mix on my LLM?
>>> from minference import MInferenceConfig
To benefit from fast kernel implementations, we recommend installing SGLang or vllm.
>>> supported_attn_types = MInferenceConfig.get_available_attn_types()
>>> supported_attn_types
['minference', 'vllm_minference', 'hf', 'vllm', 'dense', 'static', 'dilated1', 'dilated2', 'a_shape', 'tri_shape', 'vllm_a_shape', 'vllm_tri_shape', 'inf_llm', 'flexprefill', 'vllm_flexprefill', 'xattention']
>>>
Steps to reproduce
No response
Expected Behavior
No response
Logs
No response
Additional Information
No response