Feature transformations with ensembles of trees#

Transform your features into a higher dimensional, sparse space. Then train a linear model on these features.

First fit an ensemble of trees (totally random trees, a random forest, or gradient boosted trees) on the training set. Then each leaf of each tree in the ensemble is assigned a fixed arbitrary feature index in a new feature space. These leaf indices are then encoded in a one-hot fashion.

Each sample goes through the decisions of each tree of the ensemble and ends up in one leaf per tree. The sample is encoded by setting feature values for these leaves to 1 and the other feature values to 0.

The resulting transformer has then learned a supervised, sparse, high-dimensional categorical embedding of the data.

# Authors: The scikit-learn developers
# SPDX-License-Identifier: BSD-3-Clause

First, we will create a large dataset and split it into three sets:

• a set to train the ensemble methods which are later used to as a feature engineering transformer;

• a set to train the linear model;

• a set to test the linear model.

It is important to split the data in such way to avoid overfitting by leaking data.

fromsklearn.datasetsimport make_classification
fromsklearn.model_selectionimport train_test_split
X, y = make_classification (n_samples=80_000, random_state=10)
X_full_train, X_test, y_full_train, y_test = train_test_split (
 X, y, test_size=0.5, random_state=10
)
X_train_ensemble, X_train_linear, y_train_ensemble, y_train_linear = train_test_split (
 X_full_train, y_full_train, test_size=0.5, random_state=10
)

For each of the ensemble methods, we will use 10 estimators and a maximum depth of 3 levels.

n_estimators = 10
max_depth = 3

First, we will start by training the random forest and gradient boosting on the separated training set

fromsklearn.ensembleimport GradientBoostingClassifier , RandomForestClassifier
random_forest = RandomForestClassifier (
 n_estimators=n_estimators, max_depth=max_depth, random_state=10
)
random_forest.fit(X_train_ensemble, y_train_ensemble)
gradient_boosting = GradientBoostingClassifier (
 n_estimators=n_estimators, max_depth=max_depth, random_state=10
)
_ = gradient_boosting.fit(X_train_ensemble, y_train_ensemble)

Notice that HistGradientBoostingClassifier is much faster than GradientBoostingClassifier starting with intermediate datasets (n_samples >= 10_000), which is not the case of the present example.

The RandomTreesEmbedding is an unsupervised method and thus does not required to be trained independently.

fromsklearn.ensembleimport RandomTreesEmbedding
random_tree_embedding = RandomTreesEmbedding (
 n_estimators=n_estimators, max_depth=max_depth, random_state=0
)

Now, we will create three pipelines that will use the above embedding as a preprocessing stage.

The random trees embedding can be directly pipelined with the logistic regression because it is a standard scikit-learn transformer.

fromsklearn.linear_modelimport LogisticRegression
fromsklearn.pipelineimport make_pipeline
rt_model = make_pipeline (random_tree_embedding, LogisticRegression (max_iter=1000))
rt_model.fit(X_train_linear, y_train_linear)

Pipeline(steps=[('randomtreesembedding',
 RandomTreesEmbedding(max_depth=3, n_estimators=10,
 random_state=0)),
 ('logisticregression', LogisticRegression(max_iter=1000))])