The command seems to work great but the keyword not so great, but will give it another try.

Do you mean that it does not always recognise the "key phrase" at the start, but after it recognises it, then the commands are processed OK?

The "key phrase" can be easily changed in the source code:

https://github.com/ggerganov/whisper.cpp/blob/4698dcdb5238748951a087a5b26309c6b2826cc0/examples/command/command.cpp#L545-L547

The implemented approach does not depend on the timestamps.
It simply matches the transcribed text and picks everything after the "key phrase".

I'm wondering which approach makes more sense, as using whisper and a 30 second window seems like a sledge hammer for simple wake word detection.
Two options could be;

1. just use an off-the-shelf wake word system, to start using whisper until VAD detects a pause, or
2. use streaming and the VAD detection as a extra token to help parse the text. i.e. we use the whisper streaming mode, and push text to a file, but when the VAD detects a pause, we add a token to the text, like a ';'.
   It's pretty trivial to then parse the text in python, splitting on the VAD token, and checking if the first words after a split is the wake word/phrase.

Does it not need a param of minimum prob-threshold?

The problem is an off-the-shelf wake word system doesn't really exist there are quite a few but fixed KW whilst its great to be able to choose without commercial limits such as picovoice.
The beamsearch does give quite an element of latency for just a command and before I forget in the commands.txt should there be a paired whitespace sperated minimum probility threshold for each command or just an overall param to the ./command -prob .6

I like how it stems and breaks up the commands with a probility matrix but for simple commands the latency of the beamsearch is a bit much.
I have been thinking of late that you can drop KWS and have remote VAD to a central whisper that uses KW to authenticate a command.
Also generally a command has a predicate and subject and often a level.
I will get back to distributed streaming VAD later but 'Hey Whisper, Turn on the Light' is an example of KW, Predicate, Level, Subject and it fills up that beamsearch so the latency is less of an issue.
The only single word command I can think of is 'Stop' as the subject is whatever it is already doing.

So yeah I like some elements of what has been done but because of the latency I am thinking a hybrid would be great, but the KW is an audio authentication prefix to a commnd and should allows be there to stop 3rd party spoken words such as TV or media causing havoc.

Going back to VAD its such a simple model I get great results with a CRNN KW where the Noise classification is actually really accurate as a inverse VAD as if its not current argmax the input is spoken.
Even a simple CNN could make a great VAD and you can capture spoken word and deliberate overfit to user(s) voice or even a Speaker diarisation model just to kickstart broadcast where the Whisper.ccp VAD will work for end of sentence.
https://github.com/pirxus/personalVAD

Then you end up with a targetted voice broadcast, kw authentication and a probability stemmed command sentence where the latency of the overall sentence is more tolerable.

OK, this will take me some time, as I'm a Python programmer, and my C++ is very very rusty... this will take a long time...

But here is the plan:
You pass the wake phrase as an argument
./command --wake_phrase 'hey whisper'

Count the number of words in the wake phrase
phrase_length = len(wake_phrase.split())

Use the "process_general_transcription" in "have_prompt" mode to get a block of text, separated by the VAD, something like:
sample = 'hey, whisper, what's the time?'

Extract the first words, with the same length as the wake phrase:
potential_wake_phrase = ' '.join(sample.split()[:phrase_length])

Compute the similarity between two strings using Levenshtein distance:
similarity = levenshtein(wake_phrase, potential_wake_phrase)

If the similarity is good, output the sample, minus the initial wake phrase:

if similarity > threshold:
 print(' '.join(sample.split()[phrase_length:])

I think that would fit my use case.

EDIT: OK, using ChatGTP makes this much easier that I expected, should be done today!

Screenshot 2023年01月07日 at 13 28 02

all done, my first C++ code in 6 years, thanks ChatGTP! Pull request made!

I am planning to improve the guided mode of the command example to support 2, 3 and more words.
You would be able to specify a set of allowed words for the first word, another set of words for the second word, etc.
This can be implemented with a single Encoder pass and then using a proper decoding strategy that conditions the token probabilities based on the allowed words.

I plan to demonstrate this with a simple Chess application that allows you to input moves with voice (#428):

"pawn to d4"
"bishop to e5"
"rook takes knight"
etc.

I think this is very similar to your use case, so keep an eye on this development.
No ETA for the moment.

Awesome!

Hello @ggerganov this is a great starting point and I have reviewed the current code in main branch.
I have some questions:
a) there is the initial sentence for the activation: if one want to use a single word how is the performance usually affected? Imagine playing a videogame which requires some fast actions, I don't really want to say an entire sentence to then issue a command. What if we allow the user to choose a single but low probability word? This can be for example a latin world like "vox"
b) for the multiple keywords can we have a simple BNF notation or similar to allow things like more generic:
move [player|troops] [left|right|north|south]
drop (all) inventory
jump on *
So basically some form of simple regex to allow: set of possible words, optional words, any words.
I think most of the logic is in this function: https://github.com/ggerganov/whisper.cpp/blob/e27fd6f0c0c14d51ff7035499c2c94d91e090f4d/examples/command/command.cpp#L254 right?

BNF grammar is already supported. See the grammar-based examples and the wchess example.

https://github.com/ggerganov/whisper.cpp/tree/master/examples/command

Thank you for your assistance, but there is nothing there about building for Windows.

Have you already successfully built the basic whisper.cpp package?

It worked for me using CMAKE and the MSVC build using the community version of the Microsoft compiler and tools. You will probably need CMAKE installed and on the path.

Follow the quickstart instructions: https://github.com/ggerganov/whisper.cpp/tree/master

There's a download-ggml-model.cmd for downloading the model instead of the sh command.

If this works you can then try and build the command tools. These require the SDL2 library. I use vcpckg under Windows to make this and many other ported libraries available to MSVC. (install vcpckg, then use this to install SDL2)

For some unknown reason, the CMake Find_package(SDL2) did not work for me. I had to manually tell cmake where SDL2 is installed with the command

setx SDL2_DIR "C:\prog\vcpkg\installed\x64-windows\share\sdl2\"

Change the command path to suit your installation.

then

cmake -B build -DWHISPER_SDL2=ON
cmake --build build --config Release

worked fine for me (Quite a few compiler warnings though). Binaries then available at ./build/bin/Release/