-
Notifications
You must be signed in to change notification settings - Fork 1.1k
fix: engine cannot give response to the second user request with stre... #701
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
thanks a lot for the contribution. Would it be possible for you to provide a script for reproducing the issue / elaborate on the issue? Thank you!
@Copilot
Copilot
AI
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR restructures the streaming generation method in MLCEngine to ensure the model lock is always released once, fixing an issue where the engine could not handle a second streaming request.
- Flattened multiple nested
try/catchblocks into a singletry+finallyaround the main logic. - Converted inner helper functions (
_countTrailingReplacementChar,_getChunk) to arrow function expressions. - Removed redundant
lock.release()calls spread across catches; now released once infinally.
Comments suppressed due to low confidence (2)
src/engine.ts:706
- This TODO is still open—either implement the usage support for non-chat completions or file a tracking issue to ensure it isn't overlooked.
// TODO(Charlie): support usage for completion
src/engine.ts:483
- Add a test case that performs two consecutive streaming requests (with and without errors) to verify the lock is always released and reacquired correctly.
genConfig: GenerationConfig,
Copilot
AI
Jun 24, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Consider adding a brief comment above this try/finally to clarify its scope is for lock acquisition and release, improving future readability.
Copilot uses AI. Check for mistakes.
thanks a lot for the contribution. Would it be possible for you to provide a script for reproducing the issue / elaborate on the issue? Thank you!
Hey there! So sorry for the super late reply! 😊
I've just pushed all the updates to the forked project right here: https://github.com/lihaiyin88/web-llm/tree/demo_for_issue
If you have any questions at all, don't hesitate to let me know! I'm all ears! 😎
Uh oh!
There was an error while loading. Please reload this page.
see title