- 
  Notifications
 
You must be signed in to change notification settings  - Fork 186
 
[WIP] Add the streaming response to prefix cache #1730
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: zetxqx 
Once this PR has been reviewed and has the lgtm label, please assign danehans for approval. For more information see the Code Review Process.
The full list of commands accepted by this bot can be found here.
Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment
 ✅ Deploy Preview for gateway-api-inference-extension ready!
 To edit notification comments on pull requests, go to your Netlify project configuration.  | 
 
4857c71 to
 1af87ef  
 Compare
 
 f0393b3 to
 07d2e2a  
 Compare
 
 /retest
07d2e2a to
 8e8df08  
 Compare
 
 ...ethod of Messages in ChatCompletions.
8e8df08 to
 368d54f  
 Compare
 
 
Uh oh!
There was an error while loading. Please reload this page.
What type of PR is this?
/kind feature
What this PR does / why we need it:
This PR introduces the capability to process streaming responses from the model server and add to the approximate prefix cache. It adds a new parser for streaming responses that can handle both chat and legacy completion formats.
Which issue(s) this PR fixes:
Fixes #971
Does this PR introduce a user-facing change?: