Experimental Evaluation of Algorithm-Assisted Human Decision-Making

Description

Provides statistical methods for analyzing experimental evaluation of the causal impacts of algorithmic recommendations on human decisions developed by Imai, Jiang, Greiner, Halen, and Shin (2023) <doi:10.1093/jrsssa/qnad010> and Ben-Michael, Greiner, Huang, Imai, Jiang, and Shin (2024) <doi:10.48550/arXiv.2403.12108>. The data used for this paper, and made available here, are interim, based on only half of the observations in the study and (for those observations) only half of the study follow-up period. We use them only to illustrate methods, not to draw substantive conclusions.

Package Content

Index of help topics:

APCEsummary Summary of APCE
APCEsummaryipw Summary of APCE for frequentist analysis
AiEvalmcmc Gibbs sampler for the main analysis
BootstrapAPCEipw Bootstrap for estimating variance of APCE
BootstrapAPCEipwRE Bootstrap for estimating variance of APCE with
 random effects
BootstrapAPCEipwREparallel
 Bootstrap for estimating variance of APCE with
 random effects
CalAPCE Calculate APCE
CalAPCEipw Compute APCE using frequentist analysis
CalAPCEipwRE Compute APCE using frequentist analysis with
 random effects
CalAPCEparallel Calculate APCE using parallel computing
CalDIM Calculate diff-in-means estimates
CalDIMsubgroup Calculate diff-in-means estimates
CalDelta Calculate the delta given the principal stratum
CalFairness Calculate the principal fairness
CalOptimalDecision Calculate optimal decision & utility
CalPS Calculate the proportion of principal strata
 (R)
FTAdata Interim Dane data with failure to appear (FTA)
 as an outcome
HearingDate Interim court event hearing date
NCAdata Interim Dane data with new criminal activity
 (NCA) as an outcome
NVCAdata Interim Dane data with new violent criminal
 activity (NVCA) as an outcome
PSAdata Interim Dane PSA data
PlotAPCE Plot APCE
PlotDIMdecisions Plot diff-in-means estimates
PlotDIMoutcomes Plot diff-in-means estimates
PlotFairness Plot the principal fairness
PlotOptimalDecision Plot optimal decision
PlotPS Plot the proportion of principal strata (R)
PlotSpilloverCRT Plot conditional randomization test
PlotSpilloverCRTpower Plot power analysis of conditional
 randomization test
PlotStackedBar Stacked barplot for the distribution of the
 decision given psa
PlotStackedBarDMF Stacked barplot for the distribution of the
 decision given DMF recommendation
PlotUtilityDiff Plot utility difference
PlotUtilityDiffCI Plot utility difference with 95
 interval
SpilloverCRT Conduct conditional randomization test
SpilloverCRTpower Conduct power analysis of conditional
 randomization test
TestMonotonicity Test monotonicity
TestMonotonicityRE Test monotonicity with random effects
aihuman-package Experimental Evaluation of Algorithm-Assisted
 Human Decision-Making
compute_bounds_aipw Compute Risk (AI v. Human)
compute_nuisance_functions
 Fit outcome/decision and propensity score
 models
compute_nuisance_functions_ai
 Fit outcome/decision and propensity score
 models conditioning on the AI recommendation
compute_stats Compute Risk (Human+AI v. Human)
compute_stats_agreement
 Agreement of Human and AI Decision Makers
compute_stats_aipw Compute Risk (Human+AI v. Human)
compute_stats_subgroup
 Compute Risk (Human+AI v. Human) for a Subgroup
 Defined by AI Recommendation
crossfit Crossfitting for nuisance functions
g_legend Pulling ggplot legend
hearingdate_synth Synthetic court event hearing date
plot_agreement Visualize Agreement
plot_diff_ai_aipw Visualize Difference in Risk (AI v. Human)
plot_diff_human Visualize Difference in Risk (Human+AI v.
 Human)
plot_diff_human_aipw Visualize Difference in Risk (Human+AI v.
 Human)
plot_diff_subgroup Visualize Difference in Risk (Human+AI v.
 Human) for a Subgroup Defined by AI
 Recommendation
plot_preference Visualize Preference
psa_synth Synthetic PSA data
synth Synthetic data
table_agreement Table of Agreement

Maintainer

Sooahn Shin <sooahnshin@g.harvard.edu>

Author(s)

Sooahn Shin [aut, cre] (<https://orcid.org/0000-0001-6213-2197>), Zhichao Jiang [aut], Kosuke Imai [aut]

Summary of APCE

Description

Summary of average principal causal effects (APCE) with ordinal decision.

Usage

APCEsummary(apce.mcmc)

Arguments

Value

A data.frame that consists of mean and quantiles (2.5

References

Imai, K., Jiang, Z., Greiner, D.J., Halen, R., and Shin, S. (2023). "Experimental evaluation of algorithm-assisted human decision-making: application to pretrial public safety assessment." Journal of the Royal Statistical Society: Series A. <DOI:10.1093/jrsssa/qnad010>.

Examples

data(synth)
sample_mcmc <- AiEvalmcmc(data = synth, n.mcmc = 10)
subgroup_synth <- list(
 1:nrow(synth), which(synth$Sex == 0), which(synth$Sex == 1),
 which(synth$Sex == 1 & synth$White == 0), which(synth$Sex == 1 & synth$White == 1)
)
sample_apce <- CalAPCE(data = synth, mcmc.re = sample_mcmc, subgroup = subgroup_synth)
sample_apce_summary <- APCEsummary(sample_apce[["APCE.mcmc"]])

Summary of APCE for frequentist analysis

Description

Summary of average principal causal effects (APCE) with ordinal decision with frequentist results.

Usage

APCEsummaryipw(
 G1_est,
 G2_est,
 G3_est,
 G4_est,
 G5_est,
 G1_boot,
 G2_boot,
 G3_boot,
 G4_boot,
 G5_boot,
 name.group = c("Overall", "Female", "Male", "Non-white\nMale", "White\nMale")
)

Arguments

Value

A data.frame that consists of mean and quantiles (2.5

Examples

data(synth)
synth$SexWhite <- synth$Sex * synth$White
freq_apce <- CalAPCEipw(synth)
boot_apce <- BootstrapAPCEipw(synth, rep = 10)
# subgroup analysis
data_s0 <- subset(synth, synth$Sex == 0, select = -c(Sex, SexWhite))
freq_s0 <- CalAPCEipw(data_s0)
boot_s0 <- BootstrapAPCEipw(data_s0, rep = 10)
data_s1 <- subset(synth, synth$Sex == 1, select = -c(Sex, SexWhite))
freq_s1 <- CalAPCEipw(data_s1)
boot_s1 <- BootstrapAPCEipw(data_s1, rep = 10)
data_s1w0 <- subset(synth, synth$Sex == 1 & synth$White == 0, select = -c(Sex, White, SexWhite))
freq_s1w0 <- CalAPCEipw(data_s1w0)
boot_s1w0 <- BootstrapAPCEipw(data_s1w0, rep = 10)
data_s1w1 <- subset(synth, synth$Sex == 1 & synth$White == 1, select = -c(Sex, White, SexWhite))
freq_s1w1 <- CalAPCEipw(data_s1w1)
boot_s1w1 <- BootstrapAPCEipw(data_s1w1, rep = 10)
freq_apce_summary <- APCEsummaryipw(
 freq_apce, freq_s0, freq_s1, freq_s1w0, freq_s1w1,
 boot_apce, boot_s0, boot_s1, boot_s1w0, boot_s1w0
)
PlotAPCE(freq_apce_summary,
 y.max = 0.25, decision.labels = c(
 "signature", "small cash",
 "middle cash", "large cash"
 ), shape.values = c(16, 17, 15, 18),
 col.values = c("blue", "black", "red", "brown", "purple"), label = FALSE
)

Llama3 Recommendations (internal)

Description

Llama3 recommendations example for the illustration purpose.

Usage

A_llama

Format

An object of class numeric of length 1891.

Value

A numeric vector of 0 or 1.

Gibbs sampler for the main analysis

Description

See Appendix S5 for more details.

Usage

AiEvalmcmc(
 data,
 rho = 0,
 Sigma0.beta.inv = NULL,
 Sigma0.alpha.inv = NULL,
 sigma0 = NULL,
 beta = NULL,
 alpha = NULL,
 theta = NULL,
 delta = NULL,
 n.mcmc = 5 * 10,
 verbose = FALSE,
 out.length = 10,
 beta.zx.off = FALSE,
 theta.z.off = FALSE
)

Arguments

Value

An object of class mcmc containing the posterior samples.

Examples

data(synth)
sample_mcmc <- AiEvalmcmc(data = synth, n.mcmc = 2)

Bootstrap for estimating variance of APCE

Description

Estimate variance of APCE for frequentist analysis using bootstrap. See S7 for more details.

Usage

BootstrapAPCEipw(data, rep = 1000)

Arguments

Value

An object of class list with the following elements:

Examples

data(synth)
set.seed(123)
boot_apce <- BootstrapAPCEipw(synth, rep = 100)

Bootstrap for estimating variance of APCE with random effects

Description

Estimate variance of APCE for frequentist analysis with random effects using bootstrap. See S7 for more details.

Usage

BootstrapAPCEipwRE(
 data,
 rep = 1000,
 fixed,
 random,
 CourtEvent_HearingDate,
 nAGQ = 1
)

Arguments

Value

An object of class list with the following elements:

References

Imai, K., Jiang, Z., Greiner, D.J., Halen, R., and Shin, S. (2023). "Experimental evaluation of algorithm-assisted human decision-making: application to pretrial public safety assessment." Journal of the Royal Statistical Society: Series A. <DOI:10.1093/jrsssa/qnad010>.

Examples

data(synth)
data(hearingdate_synth)
synth$CourtEvent_HearingDate <- hearingdate_synth
set.seed(123)
boot_apce_re <- BootstrapAPCEipwRE(synth,
 fixed = "Y ~ Sex + White + Age +
 CurrentViolentOffense + PendingChargeAtTimeOfOffense +
 PriorMisdemeanorConviction + PriorFelonyConviction +
 PriorViolentConviction + D",
 random = "~ 1|CourtEvent_HearingDate"
)

Bootstrap for estimating variance of APCE with random effects

Description

Estimate variance of APCE for frequentist analysis with random effects using bootstrap. See S7 for more details.

Usage

BootstrapAPCEipwREparallel(data, rep = 1000, fixed, random, nAGQ = 1, size = 5)

Arguments

Value

An object of class list with the following elements:

References

Imai, K., Jiang, Z., Greiner, D.J., Halen, R., and Shin, S. (2023). "Experimental evaluation of algorithm-assisted human decision-making: application to pretrial public safety assessment." Journal of the Royal Statistical Society: Series A. <DOI:10.1093/jrsssa/qnad010>.

Examples

data(synth)
data(hearingdate_synth)
synth$CourtEvent_HearingDate <- hearingdate_synth
set.seed(123)
boot_apce_re <- BootstrapAPCEipwREparallel(synth,
 fixed = "Y ~ Sex + White + Age +
 CurrentViolentOffense + PendingChargeAtTimeOfOffense +
 PriorMisdemeanorConviction + PriorFelonyConviction +
 PriorViolentConviction + D",
 random = "~ 1|CourtEvent_HearingDate",
 size = 1
) # adjust the size

Calculate APCE

Description

Calculate average principal causal effects (APCE) with ordinal decision. See Section 3.4 for more details.

Usage

CalAPCE(
 data,
 mcmc.re,
 subgroup,
 name.group = c("overall", "Sex0", "Sex1", "Sex1 White0", "Sex1 White1"),
 rho = 0,
 burnin = 0,
 out.length = 500,
 c0 = 0,
 c1 = 0,
 ZX = NULL,
 save.individual.optimal.decision = FALSE,
 parallel = FALSE,
 optimal.decision.only = FALSE,
 dmf = NULL,
 fair.dmf.only = FALSE
)

Arguments

Value

An object of class list with the following elements:

References

Imai, K., Jiang, Z., Greiner, D.J., Halen, R., and Shin, S. (2023). "Experimental evaluation of algorithm-assisted human decision-making: application to pretrial public safety assessment." Journal of the Royal Statistical Society: Series A. <DOI:10.1093/jrsssa/qnad010>.

Examples

data(synth)
sample_mcmc <- AiEvalmcmc(data = synth, n.mcmc = 2)
subgroup_synth <- list(
 1:nrow(synth), which(synth$Sex == 0), which(synth$Sex == 1),
 which(synth$Sex == 1 & synth$White == 0), which(synth$Sex == 1 & synth$White == 1)
)
sample_apce <- CalAPCE(data = synth, mcmc.re = sample_mcmc, subgroup = subgroup_synth)

Compute APCE using frequentist analysis

Description

Estimate propensity score and use Hajek estimator to compute APCE. See S7 for more details.

Usage

CalAPCEipw(data)

Arguments

Value

An object of class list with the following elements:

Examples

data(synth)
freq_apce <- CalAPCEipw(synth)

Compute APCE using frequentist analysis with random effects

Description

Estimate propensity score and use Hajek estimator to compute APCE. See S7 for more details.

Usage

CalAPCEipwRE(data, fixed, random, nAGQ = 1)

Arguments

Value

An object of class list with the following elements:

References

Imai, K., Jiang, Z., Greiner, D.J., Halen, R., and Shin, S. (2023). "Experimental evaluation of algorithm-assisted human decision-making: application to pretrial public safety assessment." Journal of the Royal Statistical Society: Series A. <DOI:10.1093/jrsssa/qnad010>.

Examples

data(synth)
data(hearingdate_synth)
synth$CourtEvent_HearingDate <- hearingdate_synth
freq_apce_re <- CalAPCEipwRE(synth,
 fixed = "Y ~ Sex + White + Age +
 CurrentViolentOffense + PendingChargeAtTimeOfOffense +
 PriorMisdemeanorConviction + PriorFelonyConviction +
 PriorViolentConviction + D",
 random = "~ 1|CourtEvent_HearingDate"
)

Calculate APCE using parallel computing

Description

Calculate average principal causal effects (APCE) with ordinal decision using parallel computing. See Section 3.4 for more details.

Usage

CalAPCEparallel(
 data,
 mcmc.re,
 subgroup,
 name.group = c("overall", "Sex0", "Sex1", "Sex1 White0", "Sex1 White1"),
 rho = 0,
 burnin = 0,
 out.length = 500,
 c0 = 0,
 c1 = 0,
 ZX = NULL,
 save.individual.optimal.decision = FALSE,
 optimal.decision.only = FALSE,
 dmf = NULL,
 fair.dmf.only = FALSE,
 size = 5
)

Arguments

Value

An object of class list with the following elements:

References

Imai, K., Jiang, Z., Greiner, D.J., Halen, R., and Shin, S. (2023). "Experimental evaluation of algorithm-assisted human decision-making: application to pretrial public safety assessment." Journal of the Royal Statistical Society: Series A. <DOI:10.1093/jrsssa/qnad010>.

Examples

data(synth)
sample_mcmc <- AiEvalmcmc(data = synth, n.mcmc = 10)
subgroup_synth <- list(
 1:nrow(synth), which(synth$Sex == 0), which(synth$Sex == 1),
 which(synth$Sex == 1 & synth$White == 0), which(synth$Sex == 1 & synth$White == 1)
)
sample_apce <- CalAPCEparallel(
 data = synth, mcmc.re = sample_mcmc,
 subgroup = subgroup_synth,
 size = 1
) # adjust the size

Calculate diff-in-means estimates

Description

Calculate average causal effect based on diff-in-means estimator.

Usage

CalDIM(data)

Arguments

Value

A data.frame of diff-in-means estimates effect for each value of D and Y.

Examples

data(synth)
CalDIM(synth)

Calculate diff-in-means estimates

Description

Calculate average causal effect based on diff-in-means estimator.

Usage

CalDIMsubgroup(
 data,
 subgroup,
 name.group = c("Overall", "Female", "Male", "Non-white\nMale", "White\nMale")
)

Arguments

Value

A data.frame of diff-in-means estimates for each value of D and Y for each subgroup.

Examples

data(synth)
subgroup_synth <- list(
 1:nrow(synth), which(synth$Sex == 0), which(synth$Sex == 1),
 which(synth$Sex == 1 & synth$White == 0), which(synth$Sex == 1 & synth$White == 1)
)
CalDIMsubgroup(synth, subgroup = subgroup_synth)

Calculate the delta given the principal stratum

Description

Calculate the maximal deviation of decisions probability among the distributions for different groups (delta) given the principal stratum (R).

Usage

CalDelta(r, k, pd0, pd1, attr)

Arguments

Value

A data.frame of the delta.

Examples

data(synth)
subgroup_synth <- list(
 1:nrow(synth), which(synth$Sex == 0), which(synth$Sex == 1),
 which(synth$Sex == 1 & synth$White == 0), which(synth$Sex == 1 & synth$White == 1)
)
sample_mcmc <- AiEvalmcmc(data = synth, n.mcmc = 10)
sample_apce <- CalAPCE(
 data = synth, mcmc.re = sample_mcmc, subgroup = subgroup_synth,
 burnin = 0
)
CalDelta(0, 3, sample_apce[["P.D0.mcmc"]], sample_apce[["P.D1.mcmc"]], c(2, 3))

Calculate the principal fairness

Description

See Section 3.6 for more details.

Usage

CalFairness(apce, attr = c(2, 3))

Arguments

Value

A data.frame of the delta.

Examples

data(synth)
subgroup_synth <- list(
 1:nrow(synth), which(synth$Sex == 0), which(synth$Sex == 1),
 which(synth$Sex == 1 & synth$White == 0), which(synth$Sex == 1 & synth$White == 1)
)
sample_mcmc <- AiEvalmcmc(data = synth, n.mcmc = 10)
sample_apce <- CalAPCE(
 data = synth, mcmc.re = sample_mcmc, subgroup = subgroup_synth,
 burnin = 0
)
CalFairness(sample_apce)

Calculate optimal decision & utility

Description

(1) Calculate optimal decision for each observation given each of c0 (cost of an outcome) and c1 (cost of an unnecessarily harsh decision) from the lists. (2) Calculate difference in the expected utility between binary version of judge's decisions and DMF recommendations given each of c0 (cost of an outcome) and c1 (cost of an unnecessarily harsh decision) from the lists.

Usage

CalOptimalDecision(
 data,
 mcmc.re,
 c0.ls,
 c1.ls,
 dmf = NULL,
 rho = 0,
 burnin = 0,
 out.length = 500,
 ZX = NULL,
 size = 5,
 include.utility.diff.mcmc = FALSE
)

Arguments

Value

A data.frame of (1) the probability that the optimal decision for each observation being d in (0,1,...,k), (2) expected utility of binary version of judge's decision (g_d), (3) expected utility of binary DMF recommendation, and (4) the difference between (2) and (3). If include.utility.diff.mcmc = TRUE, returns a list of such data.frame and a data.frame that includes the result for mean and quantile of Utility.diff.control.mcmc and Utility.diff.treated.mcmc across mcmc samples.

Examples

data(synth)
sample_mcmc <- AiEvalmcmc(data = synth, n.mcmc = 10)
sample_optd <- CalOptimalDecision(
 data = synth, mcmc.re = sample_mcmc,
 c0.ls = seq(0, 5, 1), c1.ls = seq(0, 5, 1),
 size = 1
) # adjust the size

Calculate the proportion of principal strata (R)

Description

Calculate the proportion of each principal stratum (R).

Usage

CalPS(
 p.r.mcmc,
 name.group = c("Overall", "Female", "Male", "Non-white\nMale", "White\nMale")
)

Arguments

Value

A data.frame of the proportion of each principal stratum.

Examples

data(synth)
sample_mcmc <- AiEvalmcmc(data = synth, n.mcmc = 10)
subgroup_synth <- list(
 1:nrow(synth), which(synth$Sex == 0), which(synth$Sex == 1),
 which(synth$Sex == 1 & synth$White == 0), which(synth$Sex == 1 & synth$White == 1)
)
sample_apce <- CalAPCE(
 data = synth, mcmc.re = sample_mcmc,
 subgroup = subgroup_synth
)
CalPS(sample_apce[["P.R.mcmc"]])

Interim Dane data with failure to appear (FTA) as an outcome

Description

An interim dataset containing pre-treatment covariates, a binary treatment (Z), an ordinal decision (D), and an outcome variable (Y). The data used for the paper, and made available here, are interim, based on only half of the observations in the study and (for those observations) only half of the study follow-up period. We use them only to illustrate methods, not to draw substantive conclusions.

Usage

FTAdata

Format

A data frame with 1891 rows and 19 variables:

Z

binary treatment

D

ordinal decision

Y

outcome

Sex

male or female

White

white or non-white

SexWhite

the interaction between gender and race

Age

age

PendingChargeAtTimeOfOffense

binary variable for pending charge (felony, misdemeanor, or both) at the time of offense

NCorNonViolentMisdemeanorCharge

binary variable for current non-violent felony charge

ViolentMisdemeanorCharge

binary variable for current violent misdemeanor charge

ViolentFelonyCharge

binary variable for current violent felony charge

NonViolentFelonyCharge

binary variable for current non-violent felony charge

PriorMisdemeanorConviction

binary variable for prior conviction of misdemeanor

PriorFelonyConviction

binary variable for prior conviction of felony

PriorViolentConviction

four-level ordinal variable for prior violent conviction

PriorSentenceToIncarceration

binary variable for prior sentence to incarceration

PriorFTAInPast2Years

three-level ordinal variable for FTAs from past two years

PriorFTAOlderThan2Years

binary variable for FTAs from over two years ago

Staff_ReleaseRecommendation

four-level ordinal variable for the DMF recommendation

Interim court event hearing date

Description

An Interim Dane court event hearing date of Dane data in factor format. The data used for the paper, and made available here, are interim, based on only half of the observations in the study and (for those observations) only half of the study follow-up period. We use them only to illustrate methods, not to draw substantive conclusions.

Usage

HearingDate

Format

A date variable in factor format.

Interim Dane data with new criminal activity (NCA) as an outcome

Description

An interim dataset containing pre-treatment covariates, a binary treatment (Z), an ordinal decision (D), and an outcome variable (Y). The data used for the paper, and made available here, are interim, based on only half of the observations in the study and (for those observations) only half of the study follow-up period. We use them only to illustrate methods, not to draw substantive conclusions.

Usage

NCAdata

Format

A data frame with 1891 rows and 19 variables:

Z

binary treatment

D

ordinal decision

Y

outcome

Sex

male or female

White

white or non-white

SexWhite

the interaction between gender and race

Age

age

PendingChargeAtTimeOfOffense

binary variable for pending charge (felony, misdemeanor, or both) at the time of offense

NCorNonViolentMisdemeanorCharge

binary variable for current non-violent felony charge

ViolentMisdemeanorCharge

binary variable for current violent misdemeanor charge

ViolentFelonyCharge

binary variable for current violent felony charge

NonViolentFelonyCharge

binary variable for current non-violent felony charge

PriorMisdemeanorConviction

binary variable for prior conviction of misdemeanor

PriorFelonyConviction

binary variable for prior conviction of felony

PriorViolentConviction

four-level ordinal variable for prior violent conviction

PriorSentenceToIncarceration

binary variable for prior sentence to incarceration

PriorFTAInPast2Years

three-level ordinal variable for FTAs from past two years

PriorFTAOlderThan2Years

binary variable for FTAs from over two years ago

Staff_ReleaseRecommendation

four-level ordinal variable for the DMF recommendation

Interim Dane data with new violent criminal activity (NVCA) as an outcome

Description

An interim dataset containing pre-treatment covariates, a binary treatment (Z), an ordinal decision (D), and an outcome variable (Y). The data used for the paper, and made available here, are interim, based on only half of the observations in the study and (for those observations) only half of the study follow-up period. We use them only to illustrate methods, not to draw substantive conclusions.

Usage

NVCAdata

Format

A data frame with 1891 rows and 19 variables:

Z

binary treatment

D

ordinal decision

Y

outcome

Sex

male or female

White

white or non-white

SexWhite

the interaction between gender and race

Age

age

PendingChargeAtTimeOfOffense

binary variable for pending charge (felony, misdemeanor, or both) at the time of offense

NCorNonViolentMisdemeanorCharge

binary variable for current non-violent felony charge

ViolentMisdemeanorCharge

binary variable for current violent misdemeanor charge

ViolentFelonyCharge

binary variable for current violent felony charge

NonViolentFelonyCharge

binary variable for current non-violent felony charge

PriorMisdemeanorConviction

binary variable for prior conviction of misdemeanor

PriorFelonyConviction

binary variable for prior conviction of felony

PriorViolentConviction

four-level ordinal variable for prior violent conviction

PriorSentenceToIncarceration

binary variable for prior sentence to incarceration

PriorFTAInPast2Years

three-level ordinal variable for FTAs from past two years

PriorFTAOlderThan2Years

binary variable for FTAs from over two years ago

Staff_ReleaseRecommendation

four-level ordinal variable for the DMF recommendation

Interim Dane PSA data

Description

An interim dataset containing a binary treatment (Z), ordinal decision (D), three PSA variables (FTAScore, NCAScore, and NVCAFlag), DMF recommendation, and two pre-treatment covariates (binary indicator for gender; binary indicator for race). The data used for the paper, and made available here, are interim, based on only half of the observations in the study and (for those observations) only half of the study follow-up period. We use them only to illustrate methods, not to draw substantive conclusions.

Usage

PSAdata

Format

A data frame with 1891 rows and 7 variables:

Z

binary treatment

D

ordinal decision

FTAScore

FTA score

NCAScore

NCA score

NVCAFlag

NVCA flag

DMF

DMF recommendation

Sex

male or female

White

white or non-white

Plot APCE

Description

See Figure 4 for example.

Usage

PlotAPCE(
 res,
 y.max = 0.1,
 decision.labels = c("signature bond", "small cash bond", "large cash bond"),
 shape.values = c(16, 17, 15),
 col.values = c("blue", "black", "red", "brown"),
 label = TRUE,
 r.labels = c("safe", "easily\npreventable", "prevent-\nable", "risky\n"),
 label.position = c("top", "top", "top", "top"),
 top.margin = 0.01,
 bottom.margin = 0.01,
 name.group = c("Overall", "Female", "Male", "Non-white\nMale", "White\nMale"),
 label.size = 4
)

Arguments

Value

A ggplot.

Examples

data(synth)
sample_mcmc <- AiEvalmcmc(data = synth, n.mcmc = 10)
subgroup_synth <- list(
 1:nrow(synth), which(synth$Sex == 0), which(synth$Sex == 1),
 which(synth$Sex == 1 & synth$White == 0), which(synth$Sex == 1 & synth$White == 1)
)
sample_apce <- CalAPCE(
 data = synth, mcmc.re = sample_mcmc,
 subgroup = subgroup_synth
)
sample_apce_summary <- APCEsummary(sample_apce[["APCE.mcmc"]])
PlotAPCE(sample_apce_summary,
 y.max = 0.25, decision.labels = c(
 "signature", "small cash",
 "middle cash", "large cash"
 ), shape.values = c(16, 17, 15, 18),
 col.values = c("blue", "black", "red", "brown", "purple"), label = FALSE
)

Plot diff-in-means estimates

Description

See Figure 2 for example.

Usage

PlotDIMdecisions(
 res,
 y.max = 0.2,
 decision.labels = c("signature bond ", "small cash bond ", "large cash bond"),
 col.values = c("grey60", "grey30", "grey6"),
 shape.values = c(16, 17, 15)
)

Arguments

Value

A ggplot.

Examples

data(synth)
subgroup_synth <- list(
 1:nrow(synth), which(synth$Sex == 0), which(synth$Sex == 1),
 which(synth$Sex == 1 & synth$White == 0), which(synth$Sex == 1 & synth$White == 1)
)
res_dec <- CalDIMsubgroup(synth, subgroup = subgroup_synth)
PlotDIMdecisions(res_dec,
 decision.labels = c("signature", "small cash", "middle cash", "large cash"),
 col.values = c("grey60", "grey30", "grey6", "grey1"),
 shape.values = c(16, 17, 15, 18)
)

Plot diff-in-means estimates

Description

See Figure 2 for example.

Usage

PlotDIMoutcomes(
 res.fta,
 res.nca,
 res.nvca,
 label.position = c("top", "top", "top"),
 top.margin = 0.01,
 bottom.margin = 0.01,
 y.max = 0.2,
 label.size = 7,
 name.group = c("Overall", "Female", "Male", "Non-white\nMale", "White\nMale")
)

Arguments

Value

A ggplot.

Examples

data(synth)
subgroup_synth <- list(
 1:nrow(synth), which(synth$Sex == 0), which(synth$Sex == 1),
 which(synth$Sex == 1 & synth$White == 0), which(synth$Sex == 1 & synth$White == 1)
)
synth_fta <- synth_nca <- synth_nvca <- synth
set.seed(123)
synth_fta$Y <- sample(0:1, 1000, replace = TRUE)
synth_nca$Y <- sample(0:1, 1000, replace = TRUE)
synth_nvca$Y <- sample(0:1, 1000, replace = TRUE)
res_fta <- CalDIMsubgroup(synth_fta, subgroup = subgroup_synth)
res_nca <- CalDIMsubgroup(synth_nca, subgroup = subgroup_synth)
res_nvca <- CalDIMsubgroup(synth_nvca, subgroup = subgroup_synth)
PlotDIMoutcomes(res_fta, res_nca, res_nvca)

Plot the principal fairness

Description

See Figure 5 for example.

Usage

PlotFairness(
 res,
 top.margin = 0.01,
 y.max = 0.2,
 y.min = -0.1,
 r.labels = c("Safe", "Easily\nPreventable", "Preventable", "Risky"),
 label = TRUE
)

Arguments

Value

A data.frame of the delta.

Examples

data(synth)
subgroup_synth <- list(
 1:nrow(synth), which(synth$Sex == 0), which(synth$Sex == 1),
 which(synth$Sex == 1 & synth$White == 0), which(synth$Sex == 1 & synth$White == 1)
)
sample_mcmc <- AiEvalmcmc(data = synth, n.mcmc = 10)
sample_apce <- CalAPCE(
 data = synth, mcmc.re = sample_mcmc, subgroup = subgroup_synth,
 burnin = 0
)
sample_fair <- CalFairness(sample_apce)
PlotFairness(sample_fair, y.max = 0.4, y.min = -0.4, r.labels = c(
 "Safe", "Preventable 1",
 "Preventable 2", "Preventable 3", "Risky"
))

Plot optimal decision

Description

See Figure 6 for example.

Usage

PlotOptimalDecision(res, colname.d, idx = NULL)

Arguments

Value

A ggplot.

Examples

data(synth)
sample_mcmc <- AiEvalmcmc(data = synth, n.mcmc = 10)
sample_optd <- CalOptimalDecision(
 data = synth, mcmc.re = sample_mcmc,
 c0.ls = seq(0, 5, 1), c1.ls = seq(0, 5, 1),
 size = 1
) # adjust the size
sample_optd$cash <- sample_optd$d1 + sample_optd$d2 + sample_optd$d3
PlotOptimalDecision(sample_optd, "cash")

Plot the proportion of principal strata (R)

Description

See Figure 3 for example.

Usage

PlotPS(
 res,
 y.min = 0,
 y.max = 0.75,
 col.values = c("blue", "black", "red", "brown"),
 label = TRUE,
 r.labels = c("safe", " easily \n preventable ",
 "\n preventable\n", " risky"),
 label.position = c("top", "top", "top", "bottom"),
 top.margin = 0.02,
 bottom.margin = 0.02,
 label.size = 6.5
)

Arguments

Value

A ggplot.

Examples

data(synth)
sample_mcmc <- AiEvalmcmc(data = synth, n.mcmc = 10)
subgroup_synth <- list(
 1:nrow(synth), which(synth$Sex == 0), which(synth$Sex == 1),
 which(synth$Sex == 1 & synth$White == 0), which(synth$Sex == 1 & synth$White == 1)
)
sample_apce <- CalAPCE(
 data = synth, mcmc.re = sample_mcmc,
 subgroup = subgroup_synth
)
sample_ps <- CalPS(sample_apce[["P.R.mcmc"]])
PlotPS(sample_ps, col.values = c("blue", "black", "red", "brown", "purple"), label = FALSE)

Plot conditional randomization test

Description

See Figure S8 for example.

Usage

PlotSpilloverCRT(res)

Arguments

Value

A ggplot

Examples

data(synth)
data(hearingdate_synth)
crt <- SpilloverCRT(D = synth$D, Z = synth$Z, CourtEvent_HearingDate = hearingdate_synth)
PlotSpilloverCRT(crt)

Plot power analysis of conditional randomization test

Description

See Figure S9 for example.

Usage

PlotSpilloverCRTpower(res)

Arguments

Value

A ggplot

Examples

data(synth)
data(hearingdate_synth)
crt_power <- SpilloverCRTpower(
 D = synth$D, Z = synth$Z,
 CourtEvent_HearingDate = hearingdate_synth,
 size = 1
) # adjust the size
PlotSpilloverCRTpower(crt_power)

Stacked barplot for the distribution of the decision given psa

Description

See Figure 1 for example.

Usage

PlotStackedBar(
 data,
 fta.label = "FTAScore",
 nca.label = "NCAScore",
 nvca.label = "NVCAFlag",
 d.colors = c("grey60", "grey30", "grey10"),
 d.labels = c("signature bond", "small cash bond", "large cash bond"),
 legend.position = "none"
)

Arguments

Value

A list of three ggplots.

Examples

data(psa_synth)
# Control group (PSA not provided)
PlotStackedBar(psa_synth[psa_synth$Z == 0, ], d.colors = c(
 "grey80", "grey60",
 "grey30", "grey10"
), d.labels = c(
 "signature", "small",
 "middle", "large"
))
# Treated group (PSA provided)
PlotStackedBar(psa_synth[psa_synth$Z == 0, ], d.colors = c(
 "grey80", "grey60",
 "grey30", "grey10"
), d.labels = c(
 "signature", "small",
 "middle", "large"
))

Stacked barplot for the distribution of the decision given DMF recommendation

Description

See Figure 1 for example.

Usage

PlotStackedBarDMF(
 data,
 dmf.label = "dmf",
 dmf.category = NULL,
 d.colors = c("grey60", "grey30", "grey10"),
 d.labels = c("signature bond", "small cash bond", "large cash bond"),
 legend.position = "none"
)

Arguments

Value

A list of three ggplots.

Examples

data(psa_synth)
PlotStackedBarDMF(psa_synth, dmf.label = "DMF", d.colors = c(
 "grey80",
 "grey60", "grey30", "grey10"
), d.labels = c(
 "signature",
 "small", "middle", "large"
))

Plot utility difference

Description

See Figure 7 for example.

Usage

PlotUtilityDiff(res, idx = NULL)

Arguments

Value

A ggplot.

Examples

data(synth)
sample_mcmc <- AiEvalmcmc(data = synth, n.mcmc = 10)
synth_dmf <- sample(0:1, nrow(synth), replace = TRUE) # random dmf recommendation
sample_utility <- CalOptimalDecision(
 data = synth, mcmc.re = sample_mcmc,
 c0.ls = seq(0, 5, 1), c1.ls = seq(0, 5, 1),
 dmf = synth_dmf, size = 1
) # adjust the size
PlotUtilityDiff(sample_utility)

Plot utility difference with 95% confidence interval

Description

See Figure S17 for example.

Usage

PlotUtilityDiffCI(res)

Arguments

Value

A ggplot.

Examples

data(synth)
sample_mcmc <- AiEvalmcmc(data = synth, n.mcmc = 10)
synth_dmf <- sample(0:1, nrow(synth), replace = TRUE) # random dmf recommendation
sample_utility <- CalOptimalDecision(
 data = synth, mcmc.re = sample_mcmc,
 c0.ls = seq(0, 5, 1), c1.ls = seq(0, 5, 1),
 dmf = synth_dmf, size = 1, # adjust the size
 include.utility.diff.mcmc = TRUE
)
PlotUtilityDiffCI(sample_utility$res.mcmc)

Conduct conditional randomization test

Description

See S3.1 for more details.

Usage

SpilloverCRT(D, Z, CourtEvent_HearingDate, n = 100, seed.number = 12345)

Arguments

Value

A list of the observed and permuted test statistics and its p-value.

Examples

data(synth)
data(hearingdate_synth)
crt <- SpilloverCRT(D = synth$D, Z = synth$Z, CourtEvent_HearingDate = hearingdate_synth)

Conduct power analysis of conditional randomization test

Description

See S3.2 for more details.

Usage

SpilloverCRTpower(
 D,
 Z,
 CourtEvent_HearingDate,
 n = 4,
 m = 4,
 size = 2,
 cand_omegaZtilde = seq(-1.5, 1.5, by = 0.5)
)

Arguments

Value

A data.frame of the result of power analysis.

Examples

data(synth)
data(hearingdate_synth)
crt_power <- SpilloverCRTpower(
 D = synth$D, Z = synth$Z,
 CourtEvent_HearingDate = hearingdate_synth,
 size = 1
) # adjust the size

Test monotonicity

Description

Test monotonicity using frequentist analysis

Usage

TestMonotonicity(data)

Arguments

Value

Message indicating whether the monotonicity assumption holds.

Examples

data(synth)
TestMonotonicity(synth)

Test monotonicity with random effects

Description

Test monotonicity using frequentist analysis with random effects for the hearing date of the case.

Usage

TestMonotonicityRE(data, fixed, random)

Arguments

Value

Message indicating whether the monotonicity assumption holds.

References

Imai, K., Jiang, Z., Greiner, D.J., Halen, R., and Shin, S. (2023). "Experimental evaluation of algorithm-assisted human decision-making: application to pretrial public safety assessment." Journal of the Royal Statistical Society: Series A. <DOI:10.1093/jrsssa/qnad010>.

Examples

data(synth)
data(hearingdate_synth)
synth$CourtEvent_HearingDate <- hearingdate_synth
TestMonotonicityRE(synth,
 fixed = "Y ~ Sex + White + Age +
 CurrentViolentOffense + PendingChargeAtTimeOfOffense +
 PriorMisdemeanorConviction + PriorFelonyConviction +
 PriorViolentConviction + D",
 random = "~ 1|CourtEvent_HearingDate"
)

Compute Risk (AI v. Human)

Description

Compute the difference in risk between AI and human decision makers using AIPW estimators.

Usage

compute_bounds_aipw(
 Y,
 A,
 D,
 Z,
 X = NULL,
 nuis_funcs,
 nuis_funcs_ai,
 true.pscore = NULL,
 l01 = 1
)

Arguments

Value

A tibble the following columns:

• Z_focal: The focal treatment indicator. '1' indicates the treatment group.

• Z_compare: The comparison treatment indicator. '0' indicates the control group.

• X: Pretreatment covariate (if provided).

• fn_diff_lb: The lower bound of difference in false negatives

• fn_diff_ub: The upper bound of difference in false negatives

• fp_diff_lb: The lower bound of difference in false positives

• fp_diff_ub: The upper bound of difference in false positives

• loss_diff_lb: The lower bound of difference in loss

• loss_diff_ub: The upper bound of difference in loss

• fn_diff_lb_se: The standard error of the difference in false negatives

• fn_diff_ub_se: The standard error of the difference in false negatives

• fp_diff_lb_se: The standard error of the difference in false positives

• fp_diff_ub_se: The standard error of the difference in false positives

• loss_diff_lb_se: The standard error of the difference in loss

• loss_diff_ub_se: The standard error of the difference in loss

Examples

compute_bounds_aipw(
 Y = NCAdata$Y,
 A = PSAdata$DMF,
 D = ifelse(NCAdata$D == 0, 0, 1),
 Z = NCAdata$Z,
 nuis_funcs = nuis_func,
 nuis_funcs_ai = nuis_func_ai,
 true.pscore = rep(0.5, nrow(NCAdata)),
 X = NULL,
 l01 = 1
)

Fit outcome/decision and propensity score models

Description

Fit (1) the decision model m^{D}(z, X_i) := \Pr(D = 1 \mid Z = z, X = X_i) and (2) the outcome model m^{Y}(z, X_i) := \Pr(Y = 1 \mid D = 0, Z = z, X = X_i) for each treatment group z \in \{0,1\} and (3) the propensity score model e(1, X_i) := \Pr(Z = 1 \mid X = X_i).

Usage

compute_nuisance_functions(
 Y,
 D,
 Z,
 V,
 d_form = D ~ .,
 y_form = Y ~ .,
 ps_form = Z ~ .,
 distribution = "bernoulli",
 n.trees = 1000,
 shrinkage = 0.01,
 interaction.depth = 1,
 ...
)

Arguments

Value

A list with the following components:

z_models

A data.frame with the following columns:

idx

Index of observation.

d_pred

Predicted probability of decision.

y_pred

Predicted probability of outcome.

Z

Treatment group.

pscore

A vector of predicted propensity scores.

Examples

compute_nuisance_functions(
 Y = NCAdata$Y,
 D = ifelse(NCAdata$D == 0, 0, 1),
 Z = NCAdata$Z,
 V = NCAdata[, c("Sex", "White", "Age")],
 shrinkage = 0.01,
 n.trees = 1000
)

Fit outcome/decision and propensity score models conditioning on the AI recommendation

Description

Fit (1) the decision model m^{D}(z, a, X_i) := \Pr(D = 1 \mid Z = z, A = a, X = X_i) and (2) the outcome model m^{Y}(z, a, X_i) := \Pr(Y = 1 \mid D = 0, Z = z, A = a, X = X_i) for each treatment group z \in \{0,1\} and AI recommendation a \in \{0,1\}, and (3) the propensity score model e(1, X_i) := \Pr(Z = 1 \mid X = X_i).

Usage

compute_nuisance_functions_ai(
 Y,
 D,
 Z,
 A,
 V,
 d_form = D ~ .,
 y_form = Y ~ .,
 ps_form = Z ~ .,
 distribution = "bernoulli",
 n.trees = 1000,
 shrinkage = 0.01,
 interaction.depth = 1,
 ...
)

Arguments

Value

A list with the following components:

z_models

A data.frame with the following columns:

idx

Index of observation.

d_pred

Predicted probability of decision.

y_pred

Predicted probability of outcome.

Z

Treatment group.

A

AI recommendation.

pscore

A vector of predicted propensity scores.

Examples

compute_nuisance_functions_ai(
 Y = NCAdata$Y,
 D = ifelse(NCAdata$D == 0, 0, 1),
 Z = NCAdata$Z,
 A = PSAdata$DMF,
 V = NCAdata[, c("Sex", "White", "Age")],
 shrinkage = 0.01,
 n.trees = 1000
)

Compute Risk (Human+AI v. Human)

Description

Compute the difference in risk between human+AI and human decision makers using difference-in-means estimators.

Usage

compute_stats(Y, D, Z, X = NULL, l01 = 1)

Arguments

Value

A tibble the following columns:

• Z_focal: The focal treatment indicator. '1' indicates the treatment group.

• Z_compare: The comparison treatment indicator. '0' indicates the control group.

• X: Pretreatment covariate (if provided).

• loss_diff: The difference in loss between human+AI and human decision

• loss_diff_se: The standard error of the difference in loss

• fn_diff: The difference in false negatives between human+AI and human decision

• fn_diff_se: The standard error of the difference in false negatives

• fp_diff: The difference in false positives between human+AI and human decision

• fp_diff_se: The standard error of the difference in false positives

Examples

compute_stats(
 Y = NCAdata$Y,
 D = ifelse(NCAdata$D == 0, 0, 1),
 Z = NCAdata$Z,
 X = NULL,
 l01 = 1
)

Agreement of Human and AI Decision Makers

Description

Estimate the impact of AI recommendations on the agreement between human decisions and AI recommendations using a difference-in-means estimator of an indicator 1\{D_i = A_i\}.

Usage

compute_stats_agreement(Y, D, Z, A, X = NULL)

Arguments

Value

A tibble with the following columns:

• X: Pretreatment covariate (if provided).

• agree_diff: Difference in agreement between human decisions and AI recommendations.

• agree_diff_se: Standard error of the difference in agreement.

Examples

compute_stats_agreement(
 Y = NCAdata$Y,
 D = ifelse(NCAdata$D == 0, 0, 1),
 Z = NCAdata$Z,
 A = PSAdata$DMF
)

Compute Risk (Human+AI v. Human)

Description

Compute the difference in risk between human+AI and human decision makers using AIPW estimators.

Usage

compute_stats_aipw(Y, D, Z, nuis_funcs, true.pscore = NULL, X = NULL, l01 = 1)

Arguments

Value

A tibble the following columns:

• Z_focal: The focal treatment indicator. '1' indicates the treatment group.

• Z_compare: The comparison treatment indicator. '0' indicates the control group.

• X: Pretreatment covariate (if provided).

• loss_diff: The difference in loss between human+AI and human decision

• loss_diff_se: The standard error of the difference in loss

• fn_diff: The difference in false negatives between human+AI and human decision

• fn_diff_se: The standard error of the difference in false negatives

• fp_diff: The difference in false positives between human+AI and human decision

• fp_diff_se: The standard error of the difference in false positives

Examples

compute_stats_aipw(
 Y = NCAdata$Y,
 D = ifelse(NCAdata$D == 0, 0, 1),
 Z = NCAdata$Z,
 nuis_funcs = nuis_func,
 true.pscore = rep(0.5, nrow(NCAdata)),
 X = NULL,
 l01 = 1
)

Compute Risk (Human+AI v. Human) for a Subgroup Defined by AI Recommendation

Description

Compute the difference in risk between human+AI and human decision makers, for a subgroup \{A_i = a\}, using AIPW estimators. This can be used for computing how the decision maker overrides the AI recommendation.

Usage

compute_stats_subgroup(
 Y,
 D,
 Z,
 A,
 a = 1,
 nuis_funcs,
 true.pscore = NULL,
 X = NULL,
 l01 = 1
)

Arguments

Value

A tibble the following columns:

• Z_focal: The focal treatment indicator. '1' indicates the treatment group.

• Z_compare: The comparison treatment indicator. '0' indicates the control group.

• X: Pretreatment covariate (if provided).

• loss_diff: The difference in loss between human+AI and human decision

• loss_diff_se: The standard error of the difference in loss

• tn_fn_diff: The difference in true negatives and false negatives between human+AI and human decision

• tn_fn_diff_se: The standard error of the difference in true negatives and false negatives

• tp_diff: The difference in true positives between human+AI and human decision

• tp_diff_se: The standard error of the difference in true positives

• tn_diff: The difference in true negatives between human+AI and human decision

• tn_diff_se: The standard error of the difference in true negatives

• fn_diff: The difference in false negatives between human+AI and human decision

• fn_diff_se: The standard error of the difference in false negatives

• fp_diff: The difference in false positives between human+AI and human decision

• fp_diff_se: The standard error of the difference in false positives

Examples

compute_stats_subgroup(
 Y = NCAdata$Y,
 D = ifelse(NCAdata$D == 0, 0, 1),
 Z = NCAdata$Z,
 A = PSAdata$DMF,
 a = 1,
 nuis_funcs = nuis_func,
 true.pscore = rep(0.5, nrow(NCAdata)),
 X = NULL,
 l01 = 1
)

Crossfitting for nuisance functions

Description

Implement crossfitting with boosting methods and get predicted values for outcome/decision regression or propensity score models

Usage

crossfit(data, include_for_fit, form, ...)

Arguments

Value

A vector of predicted values

Pulling ggplot legend

Description

Pulling ggplot legend

Usage

g_legend(p)

Arguments

Value

A ggplot legend.

Synthetic court event hearing date

Description

A synthetic court event hearing date

Usage

hearingdate_synth

Format

A date variable.

NCA follow policy (internal; increasing monotonicity)

Description

Optimal policy whether to follow PSA recommendation for NCA as an outcome and policy class with an increasing monotonicity constraint. Used for the replication in the vignette.

Usage

nca_follow_policy

Format

A data frame with 41 rows and 6 variables:

FTAScore

FTA score

NCAScore

NCA score

NVCAFlag

NVCA flag

wts

Weight for each observation

n

Number of such cases

policy

Optimal policy

NCA follow policy (internal; decreasing monotonicity)

Description

Optimal policy whether to follow PSA recommendation for NCA as an outcome and policy class with an decreasing monotonicity constraint. Used for the replication in the vignette.

Usage

nca_follow_policy_dec

Format

A data frame with 41 rows and 6 variables:

FTAScore

FTA score

NCAScore

NCA score

NVCAFlag

NVCA flag

wts

Weight for each observation

n

Number of such cases

policy

Optimal policy

NCA provide policy (internal; increasing monotonicity)

Description

Optimal policy whether to provide PSA recommendation for NCA as an outcome and policy class with an increasing monotonicity constraint. Used for the replication in the vignette.

Usage

nca_provide_policy

Format

A data frame with 41 rows and 6 variables:

FTAScore

FTA score

NCAScore

NCA score

NVCAFlag

NVCA flag

wts

Weight for each observation

n

Number of such cases

policy

Optimal policy

NCA provide policy (internal; decreasing monotonicity)

Description

Optimal policy whether to provide PSA recommendation for NCA as an outcome and policy class with an decreasing monotonicity constraint. Used for the replication in the vignette.

Usage

nca_provide_policy_dec

Format

A data frame with 41 rows and 6 variables:

FTAScore

FTA score

NCAScore

NCA score

NVCAFlag

NVCA flag

wts

Weight for each observation

n

Number of such cases

policy

Optimal policy

Nuisance functions (internal)

Description

Nuisance functions generated with 'compute_nuisance_functions' for the illustration purpose.

Usage

nuis_func

Format

An object of class nuisance_functions of length 2.

Value

A list with the following components:

z_models

A data.frame with the following columns:

idx

Index of observation.

d_pred

Predicted probability of decision.

y_pred

Predicted probability of outcome.

Z

Treatment group.

pscore

A vector of predicted propensity scores.

Nuisance functions conditioning on AI (internal)

Description

Nuisance functions generated with 'compute_nuisance_functions_ai' for the illustration purpose.

Usage

nuis_func_ai

Format

An object of class nuisance_functions of length 2.

Value

A list with the following components:

z_models

A data.frame with the following columns:

idx

Index of observation.

d_pred

Predicted probability of decision.

y_pred

Predicted probability of outcome.

Z

Treatment group.

A

AI recommendation.

pscore

A vector of predicted propensity scores.

Visualize Agreement

Description

Visualize the agreement between human decisions and AI recommendations using a difference-in-means estimator of an indicator 1\{D_i = A_i\}. Generate a plot based on the overall agreement and subgroup-specific agreement.

Usage

plot_agreement(
 Y,
 D,
 Z,
 A,
 subgroup1,
 subgroup2,
 label.subgroup1 = "Subgroup 1",
 label.subgroup2 = "Subgroup 2",
 x.order = NULL,
 p.title = NULL,
 p.lb = -0.3,
 p.ub = 0.3,
 y.lab = "Impact of PSA"
)

Arguments

Value

A ggplot object.

Examples

plot_agreement(
 Y = NCAdata$Y,
 D = ifelse(NCAdata$D == 0, 0, 1),
 Z = NCAdata$Z,
 A = PSAdata$DMF,
 subgroup1 = ifelse(NCAdata$White == 1, "White", "Non-white"),
 subgroup2 = ifelse(NCAdata$Sex == 1, "Male", "Female"),
 label.subgroup1 = "Race",
 label.subgroup2 = "Gender",
 x.order = c("Overall", "Non-white", "White", "Female", "Male")
)

Visualize Difference in Risk (AI v. Human)

Description

Visualize the difference in risk between AI and human decision makers using AIPW estimators. Generate a plot based on the overall and subgroup-specific results.

Usage

plot_diff_ai_aipw(
 Y,
 D,
 Z,
 V = NULL,
 A,
 z_compare = 0,
 l01 = 1,
 nuis_funcs = NULL,
 nuis_funcs_ai = NULL,
 true.pscore = NULL,
 subgroup1,
 subgroup2,
 label.subgroup1 = "Subgroup 1",
 label.subgroup2 = "Subgroup 2",
 x.order = NULL,
 zero.line = TRUE,
 arrows = TRUE,
 y.min = -Inf,
 p.title = NULL,
 p.lb = -1,
 p.ub = 1,
 y.lab = "PSA versus Human",
 p.label = c("PSA worse", "PSA better")
)

Arguments

Value

A ggplot object.

Examples

plot_diff_ai_aipw(
 Y = NCAdata$Y,
 D = ifelse(NCAdata$D == 0, 0, 1),
 Z = NCAdata$Z,
 A = PSAdata$DMF,
 z_compare = 0,
 nuis_funcs = nuis_func,
 nuis_funcs_ai = nuis_func_ai,
 true.pscore = rep(0.5, nrow(NCAdata)),
 l01 = 1,
 subgroup1 = ifelse(NCAdata$White == 1, "White", "Non-white"),
 subgroup2 = ifelse(NCAdata$Sex == 1, "Male", "Female"),
 label.subgroup1 = "Race",
 label.subgroup2 = "Gender",
 x.order = c("Overall", "Non-white", "White", "Female", "Male"),
 zero.line = TRUE, arrows = TRUE, y.min = -Inf,
 p.title = NULL, p.lb = -0.3, p.ub = 0.3
)

Visualize Difference in Risk (Human+AI v. Human)

Description

Visualize the the difference in risk between human+AI and human decision makers using difference-in-means estimators. Generate a plot based on the overall agreement and subgroup-specific agreement.

Usage

plot_diff_human(
 Y,
 D,
 Z,
 l01 = 1,
 subgroup1,
 subgroup2,
 label.subgroup1 = "Subgroup 1",
 label.subgroup2 = "Subgroup 2",
 x.order = NULL,
 p.title = NULL,
 p.lb = -1,
 p.ub = 1,
 y.lab = "Impact of PSA",
 p.label = c("PSA harms", "PSA helps")
)

Arguments

Value

A ggplot object.

Examples

plot_diff_human(
 Y = NCAdata$Y,
 D = ifelse(NCAdata$D == 0, 0, 1),
 Z = NCAdata$Z,
 l01 = 1,
 subgroup1 = ifelse(NCAdata$White == 1, "White", "Non-white"),
 subgroup2 = ifelse(NCAdata$Sex == 1, "Male", "Female"),
 label.subgroup1 = "Race",
 label.subgroup2 = "Gender",
 x.order = c("Overall", "Non-white", "White", "Female", "Male"),
 p.title = NULL, p.lb = -0.3, p.ub = 0.3
)

Visualize Difference in Risk (Human+AI v. Human)

Description

Visualize the difference in risk between human+AI and human decision makers using AIPW estimators. Generate a plot based on the overall agreement and subgroup-specific agreement.

Usage

plot_diff_human_aipw(
 Y,
 D,
 Z,
 V = NULL,
 l01 = 1,
 nuis_funcs = NULL,
 true.pscore = NULL,
 subgroup1,
 subgroup2,
 label.subgroup1 = "Subgroup 1",
 label.subgroup2 = "Subgroup 2",
 x.order = NULL,
 p.title = NULL,
 p.lb = -1,
 p.ub = 1,
 y.lab = "Impact of PSA",
 p.label = c("PSA harms", "PSA helps")
)

Arguments

Value

A ggplot object.

Examples

plot_diff_human_aipw(
 Y = NCAdata$Y,
 D = ifelse(NCAdata$D == 0, 0, 1),
 Z = NCAdata$Z,
 nuis_funcs = nuis_func,
 true.pscore = rep(0.5, nrow(NCAdata)),
 l01 = 1,
 subgroup1 = ifelse(NCAdata$White == 1, "White", "Non-white"),
 subgroup2 = ifelse(NCAdata$Sex == 1, "Male", "Female"),
 label.subgroup1 = "Race",
 label.subgroup2 = "Gender",
 x.order = c("Overall", "Non-white", "White", "Female", "Male"),
 p.title = NULL, p.lb = -0.3, p.ub = 0.3
)

Visualize Difference in Risk (Human+AI v. Human) for a Subgroup Defined by AI Recommendation

Description

Visualize the the difference in risk between human+AI and human decision makers using AIPW estimators, for a subgroup defined by AI recommendation. Generate a plot based on the overall agreement and subgroup-specific agreement.

Usage

plot_diff_subgroup(
 Y,
 D,
 Z,
 A,
 a = 1,
 V = NULL,
 l01 = l01,
 nuis_funcs = NULL,
 true.pscore = NULL,
 subgroup1,
 subgroup2,
 label.subgroup1 = "Subgroup 1",
 label.subgroup2 = "Subgroup 2",
 x.order = NULL,
 p.title = NULL,
 p.lb = -1,
 p.ub = 1,
 y.lab = "Impact of PSA",
 p.label = c("Human correct", "PSA correct"),
 label = "TNP - FNP",
 metrics = c("Misclassification Rate", "False Negative Proportion",
 "False Positive Proportion")
)

Arguments

Value

A ggplot object.

Examples

plot_diff_subgroup(
 Y = NCAdata$Y,
 D = ifelse(NCAdata$D == 0, 0, 1),
 Z = NCAdata$Z,
 A = PSAdata$DMF,
 a = 1,
 l01 = 1,
 nuis_funcs = nuis_func,
 true.pscore = rep(0.5, nrow(NCAdata)),
 subgroup1 = ifelse(NCAdata$White == 1, "White", "Non-white"),
 subgroup2 = ifelse(NCAdata$Sex == 1, "Male", "Female"),
 label.subgroup1 = "Race",
 label.subgroup2 = "Gender",
 x.order = c("Overall", "Non-white", "White", "Female", "Male"),
 p.title = NULL, p.lb = -0.5, p.ub = 0.5,
 label = "TNP - FNP",
 metrics = c("True Negative Proportion (TNP)", "False Negative Proportion (FNP)", "TNP - FNP")
)

Visualize Preference

Description

Compute the difference in risk between AI and human decision makers using AIPW estimators over a set of loss ratios, and then visualize when we prefer human over AI decision makers. Generate a plot based on the overall and subgroup-specific results.

Usage

plot_preference(
 Y,
 D,
 Z,
 V = NULL,
 A,
 z_compare = 0,
 true.pscore = NULL,
 nuis_funcs = NULL,
 nuis_funcs_ai = NULL,
 l01_seq = 10^seq(-2, 2, length.out = 100),
 alpha = 0.05,
 subgroup1,
 subgroup2,
 label.subgroup1 = "Subgroup 1",
 label.subgroup2 = "Subgroup 2",
 x.order = NULL,
 p.title = NULL,
 legend.position = "none",
 p.label = c("AI-alone preferred", "Human-alone preferred", "Ambiguous")
)

Arguments

Value

A ggplot object.

Examples

plot_preference(
 Y = NCAdata$Y,
 D = ifelse(NCAdata$D == 0, 0, 1),
 Z = NCAdata$Z,
 A = PSAdata$DMF,
 z_compare = 0,
 nuis_funcs = nuis_func,
 nuis_funcs_ai = nuis_func_ai,
 true.pscore = rep(0.5, nrow(NCAdata)),
 l01_seq = 10^seq(-2, 2, length.out = 10),
 alpha = 0.05,
 subgroup1 = ifelse(NCAdata$White == 1, "White", "Non-white"),
 subgroup2 = ifelse(NCAdata$Sex == 1, "Male", "Female"),
 label.subgroup1 = "Race",
 label.subgroup2 = "Gender",
 x.order = c("Overall", "Non-white", "White", "Female", "Male"),
 p.title = NULL, legend.position = "none",
 p.label = c("AI-alone preferred", "Human-alone preferred", "Ambiguous")
)

Synthetic PSA data

Description

A synthetic dataset containing a binary treatment (Z), ordinal decision (D), three PSA variables (FTAScore, NCAScore, and NVCAFlag), and DMF recommendation.

Usage

psa_synth

Format

A data frame with 1000 rows and 4 variables:

Z

binary treatment

D

ordinal decision

FTAScore

FTA score

NCAScore

NCA score

NVCAFlag

NVCA flag

DMF

DMF recommendation

Synthetic data

Description

A synthetic dataset containing pre-treatment covariates, a binary treatment (Z), an ordinal decision (D), and an outcome variable (Y).

Usage

synth

Format

A data frame with 1000 rows and 11 variables:

Z

binary treatment

D

ordinal decision

Y

outcome

Sex

male or female

White

white or non-white

Age

age

CurrentViolentOffense

binary variable for current violent offense

PendingChargeAtTimeOfOffense

binary variable for pending charge (felony, misdemeanor, or both) at the time of offense

PriorMisdemeanorConviction

binary variable for prior conviction of misdemeanor

PriorFelonyConviction

binary variable for prior conviction of felony

PriorViolentConviction

four-level ordinal variable for prior violent conviction

Table of Agreement

Description

Estimate the impact of AI recommendations on the agreement between human decisions and AI recommendations using a difference-in-means estimator of an indicator 1\{D_i = A_i\}. Generate a table based on the overall agreement and subgroup-specific agreement.

Usage

table_agreement(
 Y,
 D,
 Z,
 A,
 subgroup1,
 subgroup2,
 label.subgroup1 = "Subgroup 1",
 label.subgroup2 = "Subgroup 2"
)

Arguments

Value

A tibble with the following columns:

• cov: Subgroup label.

• X: Subgroup value.

• agree_diff: Difference in agreement between human decisions and AI recommendations.

• agree_diff_se: Standard error of the difference in agreement.

• ci_lb: Lower bound of the 95% confidence interval.

• ci_ub: Upper bound of the 95% confidence interval.

Examples

table_agreement(
 Y = NCAdata$Y,
 D = ifelse(NCAdata$D == 0, 0, 1),
 Z = NCAdata$Z,
 A = PSAdata$DMF,
 subgroup1 = ifelse(NCAdata$White == 1, "White", "Non-white"),
 subgroup2 = ifelse(NCAdata$Sex == 1, "Male", "Female"),
 label.subgroup1 = "Race",
 label.subgroup2 = "Gender"
)

Visualize Agreement (internal)

Description

Internal function to visualize the agreement between human decisions and AI recommendations using a difference-in-means estimator of an indicator 1\{D_i = A_i\}.

Usage

vis_agreement(
 df,
 p.title = NULL,
 p.lb = -0.2,
 p.ub = 0.2,
 y.lab = "Impact of PSA"
)

Arguments

Value

A ggplot object.

Visualize Risk (AI v. Human; internal)

Description

Internal function to visualize the difference in risk between AI and human decision makers.

Usage

vis_diff_ai(
 df,
 label.subgroup1 = "Subgroup 1",
 label.subgroup2 = "Subgroup 2",
 x.order = NULL,
 zero.line = TRUE,
 arrows = TRUE,
 y.min = -Inf,
 p.title = NULL,
 p.lb = -1,
 p.ub = 1,
 y.lab = "PSA versus Human",
 p.label = c("PSA worse", "PSA better")
)

Arguments

Value

A ggplot object.

Visualize Risk (Human+AI v. Human; internal)

Description

Internal function to visualize the difference in risk between human+AI and human decision makers.

Usage

vis_diff_human(
 df,
 label.subgroup1 = "Subgroup 1",
 label.subgroup2 = "Subgroup 2",
 x.order = NULL,
 p.title = NULL,
 p.lb = -0.2,
 p.ub = 0.2,
 y.lab = "Impact of PSA",
 p.label = c("PSA harms", "PSA helps")
)

Arguments

Value

A ggplot object.

Visualize Risk (Human+AI v. Human; internal)

Description

Internal function to visualize the difference in risk between human+AI and human decision makers, for a subgroup defined by AI recommendation.

Usage

vis_diff_subgroup(
 df,
 label.subgroup1 = "Subgroup 1",
 label.subgroup2 = "Subgroup 2",
 x.order = NULL,
 p.title = NULL,
 p.lb = -0.2,
 p.ub = 0.2,
 y.lab = "Impact of PSA",
 p.label = c("Human correct", "PSA correct"),
 metrics = c("Misclassification Rate", "False Negative Proportion",
 "False Positive Proportion")
)

Arguments

Value

A ggplot object.

Visualize Preference (internal)

Description

Internal function to visualize preference for Human over AI decision makers.

Usage

vis_preference(
 df,
 p.title = NULL,
 legend.position = "none",
 p.label = c("AI-alone preferred", "Human-alone preferred", "Ambiguous")
)

Arguments

Value

A ggplot object.