--voice in whisper-enabled builds initialized Whisper successfully but failed at capture time with whisper audio capture not yet implemented on this platform (reported on Linux amd64). This change wires real microphone capture into the Whisper path using a cross-platform backend so voice input works consistently on Linux/macOS/Windows.
-
Whisper voice path now performs real capture + inference
- Replaced placeholder return in
RecordAndTranscribe() with:
- microphone capture at 16kHz / mono / S16
- Whisper inference via
whisper_full(...)
- transcript assembly from Whisper segments
- Preserved model bootstrap behavior (
ZOP_WHISPER_MODEL / auto-download).
-
Cross-platform capture backend
- Added
github.com/gen2brain/malgo as the audio capture layer (miniaudio bindings).
- Implemented capture lifecycle with bounded recording and termination conditions:
- max duration cap
- silence-based stop after detected speech
- interrupt-aware cancellation.
-
Audio normalization + speech detection helpers
- Added focused helper functions for:
- byte → int16 conversion
- int16 amplitude threshold speech detection
- int16 → normalized float PCM conversion expected by Whisper.
- Added targeted unit tests for these helpers and edge cases (including
-32768 handling).
Example of the new inference handoff:
pcm,err:=captureAudioPCM(recordCtx,captureMaxDuration,captureStopAfterSilence)iferr!=nil{return"",fmt.Errorf("capturing audio: %w",err)}wparams:=C.whisper_full_default_params(C.WHISPER_SAMPLING_GREEDY)wparams.print_progress=falsewparams.print_realtime=falsewparams.print_timestamps=falseifrc:=C.whisper_full(ctx,wparams,(*C.float)(unsafe.Pointer(&pcm[0])),C.int(len(pcm)));rc!=0{return"",fmt.Errorf("whisper inference failed (code %d)",int(rc))}
Original prompt
This section details on the original issue you should resolve
<issue_title>Whisper audio capture did not work on linux amd64</issue_title>
<issue_description>```
$ uname -a
Linux thinkpaddy 6.17.0-14-generic peterwwillis/zop#14~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Jan 15 15:52:10 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.4 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo
$ arecord -l
**** List of CAPTURE Hardware Devices ****
card 1: Generic_1 [HD-Audio Generic], device 0: ALC257 Analog [ALC257 Analog]
Subdevices: 1/1
Subdevice peterwwillis/zop#0: subdevice peterwwillis/zop#0
card 2: acp63 [acp63], device 0: DMIC capture dmic-hifi-0 []
Subdevices: 1/1
Subdevice peterwwillis/zop#0: subdevice peterwwillis/zop#0
$ arecord -t wav --max-file-time 30 mon.wav
Warning: Some sources (like microphones) may produce inaudible results
with 8-bit sampling. Use '-f' argument to increase resolution
e.g. '-f S16_LE'.
Recording WAVE 'mon.wav' : Unsigned 8 bit, Rate 8000 Hz, Mono
^CAborted by signal Interrupt...
peterw@thinkpaddy:~/bin$ aplay mon.wav
Playing WAVE 'mon.wav' : Unsigned 8 bit, Rate 8000 Hz, Mono
$ ./zop --voice
[zop] Whisper model not found at "/home/peterw/.local/share/zop/whisper/ggml-base.en.bin" – downloading from https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.en.bin ...
[zop] Whisper model saved to "/home/peterw/.local/share/zop/whisper/ggml-base.en.bin"
whisper_init_from_file_with_params_no_state: loading model from '/home/peterw/.local/share/zop/whisper/ggml-base.en.bin'
whisper_init_with_params_no_state: use gpu = 1
whisper_init_with_params_no_state: flash attn = 1
whisper_init_with_params_no_state: gpu_device = 0
whisper_init_with_params_no_state: dtw = 0
whisper_init_with_params_no_state: devices = 1
whisper_init_with_params_no_state: backends = 1
whisper_model_load: loading model
whisper_model_load: n_vocab = 51864
whisper_model_load: n_audio_ctx = 1500
whisper_model_load: n_audio_state = 512
whisper_model_load: n_audio_head = 8
whisper_model_load: n_audio_layer = 6
whisper_model_load: n_text_ctx = 448
whisper_model_load: n_text_state = 512
whisper_model_load: n_text_head = 8
whisper_model_load: n_text_layer = 6
whisper_model_load: n_mels = 80
whisper_model_load: ftype = 1
whisper_model_load: qntvr = 0
whisper_model_load: type = 2 (base)
whisper_model_load: adding 1607 extra tokens
whisper_model_load: n_langs = 99
whisper_model_load: CPU total size = 147.37 MB
whisper_model_load: model size = 147.37 MB
whisper_backend_init_gpu: device 0: CPU (type: 0)
whisper_backend_init_gpu: no GPU found
whisper_init_state: kv self size = 6.29 MB
whisper_init_state: kv cross size = 18.87 MB
whisper_init_state: kv pad size = 3.15 MB
whisper_init_state: compute buffer (conv) = 16.28 MB
whisper_init_state: compute buffer (encode) = 23.09 MB
whisper_init_state: compute buffer (cross) = 4.66 MB
whisper_init_state: compute buffer (decode) = 96.37 MB
Recording... press Ctrl-C or wait for silence.
Error: voice input: whisper audio capture not yet implemented on this platform
Make the audio capture work (and on all supported platforms, not just Linux; macos, windows, etc)</issue_description>
## Comments on the Issue (you are @copilot in this section)
<comments>
</comments>
✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.
`--voice` in whisper-enabled builds initialized Whisper successfully but failed at capture time with `whisper audio capture not yet implemented on this platform` (reported on Linux amd64). This change wires real microphone capture into the Whisper path using a cross-platform backend so voice input works consistently on Linux/macOS/Windows.
- **Whisper voice path now performs real capture + inference**
- Replaced placeholder return in `RecordAndTranscribe()` with:
- microphone capture at **16kHz / mono / S16**
- Whisper inference via `whisper_full(...)`
- transcript assembly from Whisper segments
- Preserved model bootstrap behavior (`ZOP_WHISPER_MODEL` / auto-download).
- **Cross-platform capture backend**
- Added `github.com/gen2brain/malgo` as the audio capture layer (miniaudio bindings).
- Implemented capture lifecycle with bounded recording and termination conditions:
- max duration cap
- silence-based stop after detected speech
- interrupt-aware cancellation.
- **Audio normalization + speech detection helpers**
- Added focused helper functions for:
- byte → int16 conversion
- int16 amplitude threshold speech detection
- int16 → normalized float PCM conversion expected by Whisper.
- Added targeted unit tests for these helpers and edge cases (including `-32768` handling).
Example of the new inference handoff:
```go
pcm, err := captureAudioPCM(recordCtx, captureMaxDuration, captureStopAfterSilence)
if err != nil {
return "", fmt.Errorf("capturing audio: %w", err)
}
wparams := C.whisper_full_default_params(C.WHISPER_SAMPLING_GREEDY)
wparams.print_progress = false
wparams.print_realtime = false
wparams.print_timestamps = false
if rc := C.whisper_full(ctx, wparams, (*C.float)(unsafe.Pointer(&pcm[0])), C.int(len(pcm))); rc != 0 {
return "", fmt.Errorf("whisper inference failed (code %d)", int(rc))
}
```
<!-- START COPILOT ORIGINAL PROMPT -->
<details>
<summary>Original prompt</summary>
>
> ----
>
> *This section details on the original issue you should resolve*
>
> <issue_title>Whisper audio capture did not work on linux amd64</issue_title>
> <issue_description>```
> $ uname -a
> Linux thinkpaddy 6.17.0-14-generic peterwwillis/zop#14~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Jan 15 15:52:10 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
> $ cat /etc/os-release
> PRETTY_NAME="Ubuntu 24.04.4 LTS"
> NAME="Ubuntu"
> VERSION_ID="24.04"
> VERSION="24.04.4 LTS (Noble Numbat)"
> VERSION_CODENAME=noble
> ID=ubuntu
> ID_LIKE=debian
> HOME_URL="https://www.ubuntu.com/"
> SUPPORT_URL="https://help.ubuntu.com/"
> BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
> PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
> UBUNTU_CODENAME=noble
> LOGO=ubuntu-logo
>
> $ arecord -l
> **** List of CAPTURE Hardware Devices ****
> card 1: Generic_1 [HD-Audio Generic], device 0: ALC257 Analog [ALC257 Analog]
> Subdevices: 1/1
> Subdevice peterwwillis/zop#0: subdevice peterwwillis/zop#0
> card 2: acp63 [acp63], device 0: DMIC capture dmic-hifi-0 []
> Subdevices: 1/1
> Subdevice peterwwillis/zop#0: subdevice peterwwillis/zop#0
> $ arecord -t wav --max-file-time 30 mon.wav
> Warning: Some sources (like microphones) may produce inaudible results
> with 8-bit sampling. Use '-f' argument to increase resolution
> e.g. '-f S16_LE'.
> Recording WAVE 'mon.wav' : Unsigned 8 bit, Rate 8000 Hz, Mono
> ^CAborted by signal Interrupt...
> peterw@thinkpaddy:~/bin$ aplay mon.wav
> Playing WAVE 'mon.wav' : Unsigned 8 bit, Rate 8000 Hz, Mono
>
>
>
> $ ./zop --voice
> [zop] Whisper model not found at "/home/peterw/.local/share/zop/whisper/ggml-base.en.bin" – downloading from https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.en.bin ...
> [zop] Whisper model saved to "/home/peterw/.local/share/zop/whisper/ggml-base.en.bin"
> whisper_init_from_file_with_params_no_state: loading model from '/home/peterw/.local/share/zop/whisper/ggml-base.en.bin'
> whisper_init_with_params_no_state: use gpu = 1
> whisper_init_with_params_no_state: flash attn = 1
> whisper_init_with_params_no_state: gpu_device = 0
> whisper_init_with_params_no_state: dtw = 0
> whisper_init_with_params_no_state: devices = 1
> whisper_init_with_params_no_state: backends = 1
> whisper_model_load: loading model
> whisper_model_load: n_vocab = 51864
> whisper_model_load: n_audio_ctx = 1500
> whisper_model_load: n_audio_state = 512
> whisper_model_load: n_audio_head = 8
> whisper_model_load: n_audio_layer = 6
> whisper_model_load: n_text_ctx = 448
> whisper_model_load: n_text_state = 512
> whisper_model_load: n_text_head = 8
> whisper_model_load: n_text_layer = 6
> whisper_model_load: n_mels = 80
> whisper_model_load: ftype = 1
> whisper_model_load: qntvr = 0
> whisper_model_load: type = 2 (base)
> whisper_model_load: adding 1607 extra tokens
> whisper_model_load: n_langs = 99
> whisper_model_load: CPU total size = 147.37 MB
> whisper_model_load: model size = 147.37 MB
> whisper_backend_init_gpu: device 0: CPU (type: 0)
> whisper_backend_init_gpu: no GPU found
> whisper_init_state: kv self size = 6.29 MB
> whisper_init_state: kv cross size = 18.87 MB
> whisper_init_state: kv pad size = 3.15 MB
> whisper_init_state: compute buffer (conv) = 16.28 MB
> whisper_init_state: compute buffer (encode) = 23.09 MB
> whisper_init_state: compute buffer (cross) = 4.66 MB
> whisper_init_state: compute buffer (decode) = 96.37 MB
> Recording... press Ctrl-C or wait for silence.
> Error: voice input: whisper audio capture not yet implemented on this platform
> ```
>
> Make the audio capture work (and on all supported platforms, not just Linux; macos, windows, etc)</issue_description>
>
> ## Comments on the Issue (you are @copilot in this section)
>
> <comments>
> </comments>
>
</details>
<!-- START COPILOT CODING AGENT SUFFIX -->
- Fixes peterwwillis/zop#23
<!-- START COPILOT CODING AGENT TIPS -->
---
✨ Let Copilot coding agent [set things up for you](https://github.com/peterwwillis/zop/issues/new?title=✨+Set+up+Copilot+instructions&body=Configure%20instructions%20for%20this%20repository%20as%20documented%20in%20%5BBest%20practices%20for%20Copilot%20coding%20agent%20in%20your%20repository%5D%28https://gh.io/copilot-coding-agent-tips%29%2E%0A%0A%3COnboard%20this%20repo%3E&assignees=copilot) — coding agent works faster and does higher quality work when set up for your repo.
