Purpose of validation set

Question

For example, I want to know what is the best k neighbours for my KNN model. It is recommended to have a separate dataset for finding the k with which your model gives the best performance, and then confirm it with test set.

Why would I need to use validation set if I can use for this purpose test set? Can't we just train n models with different k parameter and see which one performed the best using test set to chose the best k?

• ninadepina: You shouldn’t use the test set to pick hyperparameters (like k) because the test set is meant to simulate unseen data for final evaluation. If you pick k based on test performance, you’re indirectly training on the test set, which biases your evaluation.
• Peko: So this involves that I should treat the test dataset as a dataset that will come after the model is trained and validated?
• ninadepina: Exactly! Think of the test set as new, unseen data that only comes into play after you’ve trained your model and chosen all hyperparameters (like k in KNN) using the training and validation sets. You never look at the test set while tuning your model, because it’s meant to give a realistic estimate of how your model will perform in the real world.
• Dr. Snoopy: This is not a programming question! why are you even asking this in a programming site?

Answer 1

You shouldn't use the test set since its no longer unseen anymore, it is supposed to only be used once at the end to see the true test of how well it generalizes.

Instead validation sets are used to tune k, which will help the model improve, however afterwards you only use the test set once at the end. Hope this helped!