How to Tune the Hyperparameters of a Model?

Hyper-parameters are parameters that are not directly learnt within estimators. In scikit-learn they are passed as arguments to the constructor of the estimator classes. Typical examples include C, kernel and gamma for Support Vector Classifier, alpha for Lasso, etc.

It is possible and recommended to search the hyper-parameter space for the best cross validation score.

Any parameter provided when constructing an estimator may be optimized in this manner. Specifically, to find the names and current values for all parameters for a given estimator, use:

estimator.get_params()

A search consists of:

• an estimator (regressor or classifier such as sklearn.svm.SVC());

• a parameter space;

• a method for searching or sampling candidates;

• a cross-validation scheme; and

• a score function.

Some models allow for specialized, efficient parameter search strategies. Two generic approaches to sampling search candidates are provided in scikit-learn: for given values, GridSearchCV exhaustively considers all parameter combinations, while RandomizedSearchCV can sample a given number of candidates from a parameter space with a specified distribution.

To learn more about hyperparameter tuning in scikit-learn, please visit this link.

A code example of Parameter estimation using grid search with cross-validation is provided at this link.