Combine objects of class 'testargs'

Description

Combines an arbitrary number of 'testargs' objects

Usage

## S4 method for signature 'testargs'
c(x, ...)

Arguments

Details

If the argument and diagnostic names are inconsistent across objects, the combined 'testargs' object is constructed by simply taking the union of all argument and diagnostic names. Then, rbind.fill() is used to combine the diagnostic data, producing intentional NA values where appropriate.

Value

An object of class 'testargs', the result of combining x and ...

Find the optimal combinations of arguments for each diagnostic

Description

The measure of optimality is typically diagnostic dependent; for example, we wish to minimise the RMSE and run time, but we want coverage to be as close to the purported value as possible. Hence, optimal_arguments() allows one to set the optimality criteria individually for each diagnostic.

Usage

optimal_arguments(object, optimality_criterion = which.min)

Arguments

Value

A data.frame; each row corresponds to one of the diagnostics (specified by the row names), and the columns contain the argument values that optimise the corresponding diagnostic. The diagnostics at each of these optimal argument combinations are also included

Examples

## See ?test_arguments

Visualise diagnostics across the tested arguments

Description

Using various aesthetics, plot_diagnostics() can visualise the performance of all combinations of up to 4 different arguments simultaneously.

Usage

plot_diagnostics(
 object,
 focused_args = NULL,
 average_out_non_focused_args = TRUE,
 plot_order = NULL
)

Arguments

Value

a facetted 'ggplot' object, where:

• the columns of the facet are split by the diagnostics

• the y-axis corresponds to the values of the diagnostics

• the x-axis corresponds to the first argument

• the colour scale and grouping correspond to the second argument (if present)

• if a third argument is present, facet_grid() is used, whereby columns correspond to levels of the third argument, and rows correspond to diagnostics. Note that facet_grid() forces a given row to share a common y-scale, so the plot would be misleading if diagnostics were kept as columns

• the shape of the points corresponds to the fourth argument (if present)

Examples

## See ?test_arguments

Test (multiple) arguments of a prediction algorithm

Description

Test the performance of a prediction algorithm over a range of argument values. Multiple arguments can be tested simultaneously.

Usage

test_arguments(pred_fun, df_train, df_test, diagnostic_fun, arguments)

Arguments

Details

For each combination of the supplied argument levels, the value of pred_fun() is combined with df_test using cbind(), which is then passed into diagnostic_fun() to compute the diagnostics. Since the number of columns in the returned value of pred_fun() is arbitrary, one can test both predictions and uncertainty quantification of the predictions (e.g., by including prediction standard errors or predictive interval bounds)

Value

an object of class 'testargs' containing all information from the testing procedure

See Also

plot_diagnostics , optimal_arguments

Examples

library("testarguments")
## Simulate training and testing data
RNGversion("3.6.0"); set.seed(1)
n <- 1000 # sample size
x <- seq(-1, 1, length.out = n) # covariates
mu <- exp(3 + 2 * x * (x - 1) * (x + 1) * (x - 2)) # polynomial function in x
Z <- rpois(n, mu) # simulate data
df <- data.frame(x = x, Z = Z, mu = mu)
train_id <- sample(1:n, n/2, replace = FALSE)
df_train <- df[train_id, ]
df_test <- df[-train_id, ]
## Algorithm that uses df_train to predict over df_test. We use glm(), and
## test the degree of the regression polynomial and the link function.
pred_fun <- function(df_train, df_test, degree, link) {
 M <- glm(Z ~ poly(x, degree), data = df_train,
 family = poisson(link = as.character(link)))
 ## Predict over df_test
 pred <- as.data.frame(predict(M, df_test, type = "link", se.fit = TRUE))
 ## Compute response level predictions and 90% prediction interval
 inv_link <- family(M)$linkinv
 fit_Y <- pred$fit
 se_Y <- pred$se.fit
 pred <- data.frame(fit_Z = inv_link(fit_Y),
 upr_Z = inv_link(fit_Y + 1.645 * se_Y),
 lwr_Z = inv_link(fit_Y - 1.645 * se_Y))
 return(pred)
}
## Define diagnostic function. Should return a named vector
diagnostic_fun <- function(df) {
 with(df, c(
 RMSE = sqrt(mean((Z - fit_Z)^2)),
 MAE = mean(abs(Z - fit_Z)),
 coverage = mean(lwr_Z < mu & mu < upr_Z)
 ))
}
## Compute the user-defined diagnostics over a range of argument levels
testargs_object <- test_arguments(
 pred_fun, df_train, df_test, diagnostic_fun,
 arguments = list(degree = 1:6, link = c("log", "sqrt"))
)
## Visualise the performance across all combinations of the supplied arguments
plot_diagnostics(testargs_object)
## Focus on a subset of the tested arguments
plot_diagnostics(testargs_object, focused_args = "degree")
## Compute the optimal arguments for each diagnostic
optimal_arguments(
 testargs_object,
 optimality_criterion = list(coverage = function(x) which.min(abs(x - 0.90)))
)

'testargs' class

Description

This is the central class definition of the testarguments package, containing all information from a call to test_arguments

Slots

diagnostics_df

a data.frame containing the diagnostics for each combination of the supplied arguments

arg_names

the argument names

diagnostic_names

the diagnostic names