How to physically test a machine learning model?

Question

I am new to machine learning. I have implemented a machine learning model which detects sound (for example: horn, siren, hammer etc) and predicts the type of sound. I have to physically test the model on actual sound of any object. How do I go about it?

• Robert Harvey: By presenting your model with various sound waves to see if it produces the responses you expect.
• Hans-Martin Mosner: Wear ear protection and produce all kinds of noises your model should know. Don't forget to add a microphone so it hears the sounds you make.
• Neel Nath: I have the whole code in jupyter notebook. How do I present sound waves to jupyter notebook? Is there some API or something which will help me?
• amon: What do you mean with "physically"? A model is just code/data. A recording of sound waves is also just data. Are you asking how you can continually listen to surrounding sounds via a microphone?
• Neel Nath: Yes, how can I make the model continuously listen to the sound?

Answer 1

Model validation is an important part of any machine learning application. To validate your model, you have to split your available data into a training set and a validation set. You then train your model on the training set, and use the validation set to calculate performance statistics.

It seems you have a classification problem (given a sound, is it a horn, siren, or hammer?). For the samples in the validation set, you will have to know the true classification. You can then compare the true classification to the predicted classification of your model, and can calculate error statistics like false positives (a sound was classified as a siren but wasn't) or false negatives (a sound was classified as something else but was in fact a siren). If your classification model outputs continuous predictions (i.e. performs fuzzy classification), then you can also use measures like the mean squared error as an indicator of model performance.

A problem with the training set – validation set split is that the training set necessarily becomes smaller, and that the model is not trained on all available samples. There are cross-validation approaches like Leave-One-Out Cross Validation or k-Fold Cross Validation that use the available data more efficiently. Techniques like cross-validation or bootstrapping can also be used to calculate confidence intervals for your performance measures.

• Neel Nath: I have trained and validated the model. How can I physically test the model after everything is done?
• John Wu: OP, what do you see as the difference between validation and testing?

Answer 2

I have trained and validated the model. How can I physically test the model after everything is done?

[comment by the OP to @anon's answer]

To test the model, you should set up an environment (physical and computational) that resembles the intended production environment as closely as possible.

I have to physically test the model on actual sound of any object. How do I go about it?

[from the OP]

That's not much information for us to be able to answer completely. In particular, how large are these objects? An appropriate answer is likely to be very different if the object is the size of an apple, than if the object is the size of a car. Also, we don't really know what the production / real-world environment will look like. Anyway, here are my thoughts for a fairly generalised approach:

1. Define Test Scenarios: Decide whether you will run the tests in a controlled (eg., quiet room) or realistic (eg., outdoor/street) environments. Testing in diverse conditions will show how the model handles background noise and variability.

2. Gather Sample Sounds: Record real-world sounds of target objects (horns, sirens, hammers etc) in conditions similar to the expected use scenario. If direct recordings aren’t possible, use high-quality sound libraries. Capture a variety of sounds with different distances, angles, and background noises.

3. Set Up Real-Time Testing: Deploy the model on a device or app (e.g., smartphone or microcontroller) for real-time input. This setup helps you evaluate response time, latency, and the model’s ability to handle continuous sound.

4. Evaluate Model Performance: I won't go into detail here as there is voluminous discussions, and often not a great deal of consensus. I will just say that my advice is to stay away for the ever-popular precision, recall, accuracy and F1-score, and instead a method that is a "proper scoring rule" such as Brier Score and log-loss. I highly recommend reading the following threads over at CrossValidated:

Why is accuracy not the best measure for assessing classification models ?

Academic reference on the drawbacks of accuracy, F1 score, sensitivity and/or specificity

5. Iterate: Based on performance, adjust the model parameters or retrain with additional data if needed. You may add synthetic noise or use data augmentation to expand training diversity.