This is a script that I worked on to help empower physicians to alleviate the burden of documentation by utilizing a medical scribe to create SOAP notes. Expensive solutions could potentially share personal health information with their cloud-based operations. It utilizes Koboldcpp and Whisper on a local server that is concurrently running the Server.py script. The Client.py script can then be used by physicians on their device to record patient-physician conversations after a signed consent is obtained and process the result into a SOAP note.

Regards,

Braedon Hendy

Youtube Demo

• 2024年03月17日 - updated client.py to allow for OpenAI token access when GPT button is selected. A prompt will show to allow for scrubbing of any personal health information.
• 2024年03月28日 - updated client.py to allow for Whisper to run locally when set to True in the settings.
• 2024年03月29日 - added Scrubadub to be used to remove personal information prior to OpenAI token access.
• 2024年04月26日 - added alternative server file to use Faster-Whisper
• 2024年05月03日 - added alternative server file to use WhisperX
• 2024年05月06日 - added real-time Whisper processing
• 2024年05月13日 - added SSL and OHIP scrubbing
• 2024年05月14日 - GPT model selection
• 2024年06月01日 - template options and further fine-tuning for local and remote real-time Whisper

Example instructions for running on a single machine:

I will preface that this will run slowly if you are not using a GPU but will demonstrate the capability.

Install Python 3.10.9 HERE. (if the hyperlink doesn't work https://www.python.org/downloads/release/python-3109/). Make sure you click the checkbox to select "Add Python to Path".

Next, you need to install software to convert the audio file to be processed. Press Windows key + R, you can run the command line by typing powershell. Copy/type the following:

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser
Invoke-RestMethod -Uri https://get.scoop.sh | Invoke-Expression
scoop install ffmpeg

If this was successful, you need to download the files that I wrote HERE. Unzip the files (if the hyperlink doesn't work https://github.com/1984Doc/AI-Scribe).

Run the client.py (it may prompt for installation of various dependencies via pip)

I would recommend using the GPT option using an API key. The cost for running each model may determine the overall choice and can be selected in the Settings menu of the program.

Example instructions for running on a server with a GPU:

Install Python 3.10.9 HERE. (if the hyperlink doesn't work https://www.python.org/downloads/release/python-3109/). Make sure you click the checkbox to select "Add Python to Path".

Press Windows key + R, you can run the command line by typing cmd. Copy/type the following, running each line by pressing Enter:

pip install openai-whisper

Now you need to download the AI model (it is large). I recommend the Mistral 7B v0.2 or Meta Llama 3 models. These can be found on HuggingFace.

You now need to launch the AI model with the following software that you can download HERE. It will download automatically and you will need to open it (if hyperlink doesn't work https://github.com/LostRuins/koboldcpp/releases). If you have an NVidia RTX-based card, the below instructions can be modified using Koboldcpp.exe rather than koboldcpp_nocuda.exe.

Once the Koboldcpp.exe is opened, click the Browse button and select the model downloaded. Now click the Launch button.

You should see a window open and can ask it questions to test!

If this was successful, you need to download the files that I wrote HERE. Unzip the files (if the hyperlink doesn't work https://github.com/1984Doc/AI-Scribe).

Run the server.py file. This will download the files to help organize the text after converting from audio.

Run the client.py file and edit the IP addresses in the Settings menu.

1. Download and install janAI and configure with your LLM of choice.
2. Start the JanAI server.
3. Open the python client applications and set the Model Endpoint to your settings in the JanAI (Typically http://localhost:1337/v1/ by default)
4. Set your model to the one of choice (Gemma 2 2b recommended Model ID: "gemma-2-2b-it")
5. Save the settings
6. Click the KoboldCPP button to enable custom endpoint.