Pinned Discussions

• youkaichao

  GitHub discussion is not used anymore, please use the forum for discussion.

  📣 Announcements · youkaichao

• 25 You must be logged in to vote

  💡

  /v1/embeddings please

  feature request New feature or request

  yuhai-china announced Jun 21, 2023 in Ideas

  @yuhai-china @surak @adrianlyjak

  @yuhai-china @surak @adrianlyjak

  29
• 1 You must be logged in to vote

  💡

  Explicit Cache documents with reference ids?

  yupbank announced Mar 27, 2025 in Ideas

  @yupbank @DarkLight1337

  @yupbank @DarkLight1337

  1
• 3 You must be logged in to vote

  💡

  Exploring Q-Filters for KV Cache Compression (32x Compression with FlashAttention)

  MillionthOdin16 announced Mar 6, 2025 in Ideas

  @MillionthOdin16 @NathanGodey

  @MillionthOdin16 @NathanGodey

  1
• 3 You must be logged in to vote

  💡

  Enhancing LLM Serving with ZenTorch on AMD Gen5 CPUs

  Manoj-red-hat announced Feb 12, 2025 in Ideas

  @Manoj-red-hat

  @Manoj-red-hat

  0
• 5 You must be logged in to vote

  💡

  vllm service profiler

  wyhhyw123 announced Oct 31, 2024 in Ideas

  @wyhhyw123 @Wesley-Jzy @ShoupingShan

  @wyhhyw123 @Wesley-Jzy @ShoupingShan

  4
• 18 You must be logged in to vote

  💡

  Support for apple/OpenELM-3B-Instruct

  bks5881 announced Apr 25, 2024 in Ideas

  @bks5881 @stas00 @youkaichao

  @bks5881 @stas00 @youkaichao

  5
• 1 You must be logged in to vote

  💡

  Optimizing of prefix caching (V2)

  brotherchen announced Oct 25, 2024 in Ideas

  @brotherchen

  @brotherchen

  0
• 1 You must be logged in to vote

  💡

  Samplers Order support

  ZeroYuJie announced Oct 17, 2024 in Ideas

  @ZeroYuJie

  @ZeroYuJie

  0
• 1 You must be logged in to vote

  💡

  Change models without stopping openai server

  g-eoj announced Oct 16, 2024 in Ideas

  @g-eoj

  @g-eoj

  0
• 1 You must be logged in to vote

  💡

  Can the docker image load a gradio web server?

  mglover-bb announced Sep 16, 2024 in Ideas

  @mglover-bb

  @mglover-bb

  0
• 17 You must be logged in to vote

  💡

  RFC: FP8 in vLLM

  HaiShaw announced Jan 17, 2024 in Ideas

  @HaiShaw @markab21 @pedrito87

  @HaiShaw @markab21 @pedrito87

  5
• 2 You must be logged in to vote

  💡

  The Phenomenon of Latency Jumps in Inference

  UltraTar0 announced Sep 5, 2024 in Ideas

  @UltraTar0 @SeungminHeo

  @UltraTar0 @SeungminHeo

  2
• 2 You must be logged in to vote

  💡

  Can Paged Attention be used in training to reduce memory usage?

  CACACAZ announced Sep 9, 2024 in Ideas

  @CACACAZ

  @CACACAZ

  0
• 1 You must be logged in to vote

  💡

  Bucketing with vllm

  hbikki announced Aug 21, 2024 in Ideas

  @hbikki

  @hbikki

  1
• 1 You must be logged in to vote

  💡

  How to support re-compute kv-cache after certain decoded token

  jiazhan-msft announced Jul 29, 2024 in Ideas

  @jiazhan-msft

  @jiazhan-msft

  0
• 2 You must be logged in to vote

  💡

  RFC: FP8 Quantization Schema in vLLM update

  HaiShaw announced Jun 24, 2024 in Ideas

  @HaiShaw @mgoin @comaniac

  @HaiShaw @mgoin @comaniac

  6
• 9 You must be logged in to vote

  💡

  Is it possible to implement distributed CPUs inference?

  hyperbolic-c announced Apr 20, 2024 in Ideas

  @hyperbolic-c

  @hyperbolic-c

  0
• 3 You must be logged in to vote

  💡

  Support for https://huggingface.co/CohereForAI/c4ai-command-r-plus

  bks5881 announced Apr 9, 2024 in Ideas

  @bks5881 @martin416

  @bks5881 @martin416

  2
• 3 You must be logged in to vote

  💡

  🤝🏻vllm support for pipeline huggingface

  aleks3ewe announced Apr 5, 2024 in Ideas

  @aleks3ewe

  @aleks3ewe

  0
• 6 You must be logged in to vote

  💡

  RFC: FP8 Quantization Schema in vLLM

  HaiShaw announced Mar 6, 2024 in Ideas

  @HaiShaw

  @HaiShaw

  0
• 1 You must be logged in to vote

  💡

  Loading models from an S3 location instead of local path

  petrosbaltzis announced Feb 28, 2024 in Ideas

  @petrosbaltzis @simon-mo @ikalista

  @petrosbaltzis @simon-mo @ikalista

  2
• 3 You must be logged in to vote

  💡

  Any possible chance to add a context-free grammar constraint language ?

  nova-land announced Sep 30, 2023 in Ideas

  @nova-land @mhillebrand @jpeig

  @nova-land @mhillebrand @jpeig

  4
• 4 You must be logged in to vote

  💡

  support for Optimum-NVIDIA

  dinchu announced Dec 6, 2023 in Ideas

  @dinchu

  @dinchu

  0
• 2 You must be logged in to vote

  💡

  LM-Infinite

  bitnom announced Sep 1, 2023 in Ideas

  @bitnom @kyegomez

  @bitnom @kyegomez

  1
• 1 You must be logged in to vote

  💡

  Kernel fusion for Llama-v2

  TejaGollapudi announced Jul 21, 2023 in Ideas

  @TejaGollapudi

  @TejaGollapudi

  0