URL: https://linuxfr.org/users/iman/journaux/decrire-une-une-image-avec-une-ia-locale Title: décrire une une image avec une iA locale Authors: i M@N Date: 2024年05月08日T10:41:01+02:00 License: CC By-SA Tags: hugging_face, reconnaissance_forme, llama, llava, python3, intelligence_artificielle et grands_modèles_de_langage Score: 20 Aujourd'hui c'est fourienTM, petit tuto sans prétention! Pour décrire des images en utilisant une iA localement j'utilise LLaVA qui fait partie de LLaMA C++ ([llama.cpp](https://github.com/ggerganov/llama.cpp)) prérequis : - créer un dossier image_summary et ses sous dossiers ```mkdir -p image_summary/bin image_summary/models image_summary/data/img image_summary/data/txt``` - créer un venv (j'utilise Python 3.10.6) ```python -m venv ./image_summary/venv/``` - activer l'environnement ```source ./image_summary/venv/bin/activate``` - mettre à jour pip ```pip install --upgrade pip``` - installer les dépendances ```pip install 'glob2==0.7'``` - désactiver l'environnement ```deactivate``` - télécharger le [code source de llama.cpp de cette release](https://github.com/ggerganov/llama.cpp/releases/tag/b2356) (llava a été temporairement retiré afin de refactoriser le code) - décompresser l'archive du code source dans un répertoire ```mkdir llama.cpp-b2356 && tar -zvxf llama.cpp-b2356.tar.gz --strip-components=1 -C ./llama.cpp-b2356/``` - compiler les binaires de llama.cpp pour avoir llava-cli ```cd llama.cpp-b2356 cmake -Bbuild # basique CPU + RAM cmake -Bbuild -DLLAMA_CUBLAS=ON # pour utiliser CUDA avec une carte NViDiA cmake --build build --config Release``` - copier ./build/bin/llava-cli dans image_summary/bin/ - télécharger les [fichiers mmjprog et ggml](https://huggingface.co/mys/ggml_llava-v1.5-7b/tree/main) [ggml-model-q4_k.gguf](https://huggingface.co/mys/ggml_llava-v1.5-7b/resolve/main/ggml-model-q4_k.gguf?download=true) (LLM 4Go) et [mmproj-model-f16.gguf](https://huggingface.co/mys/ggml_llava-v1.5-7b/resolve/main/mmproj-model-f16.gguf?download=true) (la partie multimodale qui "voit" 600Mo) dans image_summary/models/ - placer ce [script python](https://gist.github.com/ali0une/f9a96f6707a743de50361dd332597e14) dans image_summary/image_summary.py ```python # mon ""code"" python avec coloration syntaxique # (désolé je suis pas un grand codeur j'accepte les pull requests ^^) from pathlib import Path import glob import subprocess import os LLAVA_EXEC_PATH = "./bin/llava-cli" MODEL_PATH = "./models/ggml-model-f16.gguf" MMPROJ_PATH = "./models/mmproj-model-f16.gguf" DATA_DIR = "data" IMAGE_DIR = Path(DATA_DIR, "img") TXT_DIR = Path(DATA_DIR, "txt") types = ('*.jpg', '*.png') # the tuple of file types image_paths = [] for files in types: image_paths.extend(sorted(glob.glob(str(IMAGE_DIR.joinpath(files))))) #print(image_paths) txt_paths = sorted(glob.glob(str(TXT_DIR.joinpath("*.txt")))) TEMP = 0.1 ## for llava 1.5 PROMPT = "You are an assistant who perfectly describes images." bash_command = f"{LLAVA_EXEC_PATH} -m {MODEL_PATH} --mmproj {MMPROJ_PATH} --temp {TEMP} -p '{PROMPT}' --ctx-size 0" #print(bash_command) # Bash command output # ./bin/llava-cli -m ./models/ggml-model-f16.gguf --mmproj ./models/mmproj-model-f16.gguf --temp 0.1 -p "Describe the image." --ctx-size 0 for image_path in image_paths: image_name = Path(image_path).stem image_summary_path = TXT_DIR.joinpath(image_name + ".txt") if not os.path.exists(image_summary_path): print(f"Processing {image_path}") # add input image and output txt filenames to bash command bash_command_cur = f"{bash_command} --image '{image_path}'> '{image_summary_path}'" print(bash_command_cur) # run the bash command process = subprocess.Popen( bash_command_cur, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE ) # get the output and error from the command output, error = process.communicate() # commment output and error for less verbose output # print("Output:") # print(output.decode("utf-8")) # print("Error:") # print(error.decode("utf-8")) # return the code of the command return_code = process.returncode # print(f"Return code: {return_code}") # print() print("Done") # clean txt files bash_command_sed = f"sed -i '/_/d' '{image_summary_path}' && sed -i '/^[[:space:]]*$/d' '{image_summary_path}'" print(bash_command_sed) # run the bash command process = subprocess.Popen( bash_command_sed, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE ) print("txt files cleaned") else: print(f"Already processed {image_summary_path}") ``` - placer 2 ou 3 images dans image_summary/img/ pour tester - lancer le script ```./image_summary/venv/bin/python image_summary.py``` - profit dans image_summary/txt/ le résultat, ça prend environ 1 minute par image sur mon processeur i5-12600K (beaucoup moins avec CUDA) références : https://github.com/ggerganov/llama.cpp https://huggingface.co/mys/ggml_tree/main https://plainenglish.io/community/generate-a-summary-of-an-image-with-an-llm-in-python-0fc069

AltStyle によって変換されたページ (->オリジナル) /