This repository was archived by the owner on Sep 10, 2025. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 249
This repository was archived by the owner on Sep 10, 2025. It is now read-only.
What's the future plan for torchchat serving #1491
Open
Assignees
@jenniew
Description
I see current torchchat serving provides basic serving function. I'm wondering what the future plan for serving. What's the target of torchchat serve? Will it provide more optimized and high performance serving features(like Continuous batching, prefix-caching, chunked prefill, etc.)