This repository was archived by the owner on Mar 6, 2024. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 472
Prompt limit issue fixed #501
Open
+383
−46
Open
Changes from 5 commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
6f45f40
Feature/prompt limit issue (#3) -> Prompt limit increase with split t...
Infracloud-harsh a49ec52
Merge branch 'coderabbitai:main' into main
Infracloud-harsh 11f2846
revert syntax changes openai-review.yml
Infracloud-harsh 44a12fb
Update tokenizer.ts
Infracloud-harsh 4cf4e4d
Update tokenizer.ts as per the codereviwer ai comment
Infracloud-harsh 9642c45
Update tokenizer.ts
Infracloud-harsh File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
47 changes: 47 additions & 0 deletions
__tests__/tokenizer.test.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| import {splitPrompt} from './../src/tokenizer' // Import your module with the splitPrompt function | ||
|
|
||
| describe('splitPrompt function', () => { | ||
| it('should split a prompt into smaller pieces', async () => { | ||
| const maxTokens = 10 // Adjust this as needed | ||
| const prompt = 'This is a test prompt for splitting into smaller pieces.' | ||
|
|
||
| const result = await splitPrompt(maxTokens, prompt) | ||
|
|
||
| // Calculate the expected output based on the maxTokens value | ||
| const expectedOutput = [ | ||
| 'This is a', | ||
| 'test', | ||
| 'prompt for', | ||
| 'splitting', | ||
| 'into', | ||
| 'smaller', | ||
| 'pieces.' | ||
| ] | ||
|
|
||
| expect(result).toEqual(expectedOutput) | ||
| }) | ||
Infracloud-harsh marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| it('should handle a prompt smaller than maxTokens', async () => { | ||
| const maxTokens = 100 // A large value | ||
| const prompt = 'A very short prompt.' | ||
|
|
||
| const result = await splitPrompt(maxTokens, prompt) | ||
|
|
||
| // The prompt is already smaller than maxTokens, so it should return an array with the entire prompt. | ||
| const expectedOutput = 'A very short prompt.' | ||
|
|
||
| expect(result).toEqual(expectedOutput) | ||
| }) | ||
Infracloud-harsh marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| it('should handle an empty prompt', async () => { | ||
| const maxTokens = 10 | ||
| const prompt = '' | ||
|
|
||
| const result = await splitPrompt(maxTokens, prompt) | ||
|
|
||
| // An empty prompt should result in an empty array. | ||
| const expectedOutput: string[] | string = '' | ||
|
|
||
| expect(result).toEqual(expectedOutput) | ||
| }) | ||
Infracloud-harsh marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| }) | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.