Cannot calculate confusion matrix utilizing supervision from roboflow for Yolov8 model

Question

I am trying to calculate the confusion matrix for my yolov8 (or yolov11) model utilizing supervision from roboflow. I found some instructions but they do not seem to be crystal clear. For example these instructions are a bit vague:

import supervision as sv
from ultralytics import YOLO
dataset = sv.DetectionDataset.from_yolo(...)
model = YOLO(...)
def callback(image: np.ndarray) -> sv.Detections:
 result = model(image)[0]
 return sv.Detections.from_ultralytics(result)
confusion_matrix = sv.ConfusionMatrix.benchmark(
 dataset = dataset,
 callback = callback
)
confusion_matrix.plot()

From other sites (e.g. from this issue report which seem to address the same issue with me) I found out that there are these parameters for the dataset: images_directory_path, annotations_directory_path and data_yaml_path

I modified the code to include this and point to my validation image folder (the subfolder only containing the val images -although pointing to the generic image folder did not help either-), to my annotations (in yolo format, also the subfolder containing only the valiadation annotations) validation folder and to the yaml file of the dataset (to acquire the classes I guess).

dataset = sv.DetectionDataset.from_yolo(images_directory_path=IMAGES_DIR,
 annotations_directory_path=ANNOT_DIR,
 data_yaml_path=YAML_PATH)

The model points to the actual model weights in pytorch format.

Unfortunately, this did not produce any valid results. My confusion matric is empty but the classes seem correct. So, I suspect my dataset is not read correctly while the yaml file is read correctly.

Does anyone know how to correctly insert the parameters in the dataset object?

Edit:

I tried to debug the issue. First, I have installed version supervision==0.26.1. I also followed the instruction in this notebook where I tried to load my dataset.

This seems to work up to the point of print(dataset.classes)

['car', 'person', 'truck', 'motor', 'bus', 'bike', 'fire']

but the next one fails:

IMAGE_NAME = next(iter(dataset.images.keys()))

AttributeError: 'DetectionDataset' object has no attribute 'images'

There seem to be same changes to the interface which prevent the loading of the dataset.

• colonel: I guess, supervision version in the notebook you are referring and supervision==0.26.1 which you manually installed are different. that is why it fails because dataset.images doesn’t exist and gives you attribute error. I can give sample answer to your question.

Answer 1

Replace this

IMAGE_NAME = next(iter(dataset.images.keys()))

with

IMAGE_NAME = dataset.image_paths[0]

or if you want to iterate

IMAGE_NAME = next(iter(dataset.image_paths))

Hope this will fix the Attribute error