Measures of variable importance in random forests

Question

I've been playing around with random forests for regression and am having difficulty working out exactly what the two measures of importance mean, and how they should be interpreted.

The importance() function gives two values for each variable: %IncMSE and IncNodePurity. Is there simple interpretations for these 2 values?

For IncNodePurity in particular, is this simply the amount the RSS increase following the removal of that variable?

• Nick Sabbe: Have you looked at ?importance ? There's an explanation there on what both measures mean...
• dcl: @Nick Sabbe, I have, and am trying to wrap my head around them. I was wondering if there are any nice intuitive interpretations for them.

Answer 1

The first one can be 'interpreted' as follows: if a predictor is important in your current model, then assigning other values for that predictor randomly but 'realistically' (i.e.: permuting this predictor's values over your dataset), should have a negative influence on prediction, i.e.: using the same model to predict from data that is the same except for the one variable, should give worse predictions.

So, you take a predictive measure (MSE) with the original dataset and then with the 'permuted' dataset, and you compare them somehow. One way, particularly since we expect the original MSE to always be smaller, the difference can be taken. Finally, for making the values comparable over variables, these are scaled.

For the second one: at each split, you can calculate how much this split reduces node impurity (for regression trees, indeed, the difference between RSS before and after the split). This is summed over all splits for that variable, over all trees.

Note: a good read is Elements of Statistical Learning by Hastie, Tibshirani and Friedman...

• dcl: Cheers, I actually have that book open now :)
• DavideChicco.it: What does RSS mean?
• Barker: RSS is Residual Sum of Squares
• InColorado: The link to Hastie et al. above is broken. This one works as of Feb 2025: hastie.su.domains/ElemStatLearn

Answer 2

Random Forest importance metrics as implemented in the randomForest package in R have quirks in that correlated predictors get low importance values.

http://bioinformatics.oxfordjournals.org/content/early/2010/04/12/bioinformatics.btq134.full.pdf

I have a modified implementation of random forests out on CRAN which implements their approach of estimating empirical p values and false discovery rates, here

http://cran.r-project.org/web/packages/pRF/index.html

• Agile Bean: does this explain the different output of variable importance if you use randomForest with the caret package like caret::train(method="rf", importance = TRUE, ...) ??