Note

Go to the end to download the full example code. or to run this example in your browser via JupyterLite or Binder

Comparing randomized search and grid search for hyperparameter estimation#

Compare randomized search and grid search for optimizing hyperparameters of a linear SVM with SGD training. All parameters that influence the learning are searched simultaneously (except for the number of estimators, which poses a time / quality tradeoff).

The randomized search and the grid search explore exactly the same space of parameters. The result in parameter settings is quite similar, while the run time for randomized search is drastically lower.

The performance is may slightly worse for the randomized search, and is likely due to a noise effect and would not carry over to a held-out test set.

Note that in practice, one would not search over this many different parameters simultaneously using grid search, but pick only the ones deemed most important.

RandomizedSearchCV took 0.58 seconds for 15 candidates parameter settings.
Model with rank: 1
Mean validation score: 0.987 (std: 0.011)
Parameters: {'alpha': np.float64(0.01001911984591966), 'average': False, 'l1_ratio': np.float64(0.7665012035905148)}
Model with rank: 2
Mean validation score: 0.987 (std: 0.011)
Parameters: {'alpha': np.float64(0.40134964872774576), 'average': False, 'l1_ratio': np.float64(0.05033776045421079)}
Model with rank: 3
Mean validation score: 0.983 (std: 0.011)
Parameters: {'alpha': np.float64(0.1352374671440465), 'average': False, 'l1_ratio': np.float64(0.6719936995475292)}
GridSearchCV took 3.35 seconds for 60 candidate parameter settings.
Model with rank: 1
Mean validation score: 0.994 (std: 0.005)
Parameters: {'alpha': np.float64(0.1), 'average': False, 'l1_ratio': np.float64(0.1111111111111111)}
Model with rank: 2
Mean validation score: 0.991 (std: 0.008)
Parameters: {'alpha': np.float64(0.1), 'average': False, 'l1_ratio': np.float64(0.0)}
Model with rank: 3
Mean validation score: 0.989 (std: 0.018)
Parameters: {'alpha': np.float64(1.0), 'average': False, 'l1_ratio': np.float64(0.0)}

# Authors: The scikit-learn developers
# SPDX-License-Identifier: BSD-3-Clause
fromtimeimport time
importnumpyasnp
importscipy.statsasstats
fromsklearn.datasetsimport load_digits
fromsklearn.linear_modelimport SGDClassifier
fromsklearn.model_selectionimport GridSearchCV , RandomizedSearchCV
# get some data
X, y = load_digits (return_X_y=True, n_class=3)
# build a classifier
clf = SGDClassifier (loss="hinge", penalty="elasticnet", fit_intercept=True)
# Utility function to report best scores
defreport(results, n_top=3):
 for i in range(1, n_top + 1):
 candidates = np.flatnonzero (results["rank_test_score"] == i)
 for candidate in candidates:
 print("Model with rank: {0}".format(i))
 print(
 "Mean validation score: {0:.3f} (std: {1:.3f})".format(
 results["mean_test_score"][candidate],
 results["std_test_score"][candidate],
 )
 )
 print("Parameters: {0}".format(results["params"][candidate]))
 print("")
# specify parameters and distributions to sample from
param_dist = {
 "average": [True, False],
 "l1_ratio": stats.uniform (0, 1),
 "alpha": stats.loguniform (1e-2, 1e0),
}
# run randomized search
n_iter_search = 15
random_search = RandomizedSearchCV (
 clf, param_distributions=param_dist, n_iter=n_iter_search
)
start = time ()
random_search.fit(X, y)
print(
 "RandomizedSearchCV took %.2f seconds for %d candidates parameter settings."
 % ((time () - start), n_iter_search)
)
report(random_search.cv_results_)
# use a full grid over all parameters
param_grid = {
 "average": [True, False],
 "l1_ratio": np.linspace (0, 1, num=10),
 "alpha": np.power (10, np.arange (-2, 1, dtype=float)),
}
# run grid search
grid_search = GridSearchCV (clf, param_grid=param_grid)
start = time ()
grid_search.fit(X, y)
print(
 "GridSearchCV took %.2f seconds for %d candidate parameter settings."
 % (time () - start, len(grid_search.cv_results_["params"]))
)
report(grid_search.cv_results_)