Working on caret based models #446

Can you give more information, perhaps an example of what you are looking for?

Non-parametric models (like xgboot) don't really lend themselves to predictor specific effect sizes. There are variable importance measures you might want to look at:
https://topepo.github.io/caret/variable-importance.html

yes variable importance is part of the solution. However conceptually if one could consider R-square of 100% to be a model that perfectly explains the dependent variable, then an R-square of 10% means 90% is unexplained effect. So hoping for a way to get model agnostic effect distribution of predictors

You can get model-wise diagnostics.

But predictor-wise is much harder, and in any case would not be model agnostic.

As an example, here is a simple linear model - we can compare the unique contribution of each predictor by leaving it out and seeing the change in R square. And yet...

m <- lm(mpg ~ cyl + am + hp,
 data = mtcars)
m_drop_cyl <- update(m, formula. = . ~ . - cyl)
m_drop_am <- update(m, formula. = . ~ . - am)
m_drop_hp <- update(m, formula. = . ~ . - hp)
R2_total <- performance::r2(m)[[1]]

The difference should be the unique contribution of each model

R2_delta_cyl <- R2_total - performance::r2(m_drop_cyl)[[1]]
R2_delta_am <- R2_total - performance::r2(m_drop_am)[[1]]
R2_delta_hp <- R2_total - performance::r2(m_drop_hp)[[1]]
c(R2_total = R2_total,
 Sum_unique_R2 = R2_delta_cyl + R2_delta_am + R2_delta_hp)
#> R2_total.R2 Sum_unique_R2.R2 
#> 0.8041352 0.1306490

^{Created on 2022年05月04日 by the reprex package (v2.0.1)}

In more complex models this becomes even more prominent - e.g., in tree based models or KNN, predictors interact in complex ways...