Sample size, P-values (its relationship), and data visualization with plots. ggplot and T test

Question

We know that P-values (within t-test context as an example..) is highly sensitive to sample size. A larger sample will yield a smaller p-value remaining everything else constant. On the other hand, Cohen ́s d effect size remains the same.

Sample size and P values

I'm inspired in this code here, but I ́ve changed some parts to make the difference between means constant, instead of creating a random variable based on a normal distribution.

Although everything is working, I do imagine that some of the experts in this community could improve my syntax.

library(tidyverse)
ctrl_mean <- 8
ctrl_sd <- 1
treated_mean <- 7.9
treated_sd <- 1.2
sample <- numeric() #criar vetor para grupar resultados
nsim <- 1000 #criar variavel
t_result <- numeric()
for (i in 1:nsim) { 
 set.seed(123) 
 t_result[i] <- (mean(ctrl_mean)-mean(treated_mean))/sqrt((ctrl_sd^2/(i))+(treated_sd^2/(i))) #manual t test
 sample[i] <- i # number of participants
}
ds <- data.frame(
 sample = sample, #assign the sample size
 t_result = round(t_result,3), #get the t test result
 degrees = sample*2-2) #compute the degrees of freedom
ds %>% 
 filter(sample>1) %>% 
 mutate(P_Value = 2*pt(abs(t_result), df=degrees,lower.tail=FALSE)) %>% 
 left_join(ds,.) -> ds
#plot 
ggplot(ds, aes(x=sample, y=P_Value)) +
 geom_line() +
 annotate("segment", x = 1, xend=sample, y = 0.05, yend = 0.05, colour = "purple", linetype = "dashed") +
 annotate("segment", x = 1, xend=sample, y = 0.01, yend = 0.01, colour = "red", linetype = "dashed") +
 annotate("text", x = c(1,1), y=c(.035,.001), label = c("p < 0.05", "p < 0.01"))

Answer 1

In this code you do not need the loop:

sample <- 1:nsim
t_result <- (mean(ctrl_mean)-mean(treated_mean)) /
 sqrt((ctrl_sd^2/(sample))+(treated_sd^2/(sample)))
# OR:
t_result <- (mean(ctrl_mean) - mean(treated_mean)) /
 sqrt((ctrl_sd^2 + treated_sd^2) / sample)

why the set seed?

• Luis: Thank you. Easier than I was imagining. Set.seed, in this case, was an error, thanks for highlighting that.