R, 87 bytes

Or R>=4.1, 80 bytes by replacing the word function with a \.

function(l,a,b,`[`=lapply)cbind(rep(seq(l),lengths(w<-l[`%in%`,a:b][which])),unlist(w))

Try it online!

Solution shorter in R>=4.1:

R, 92 bytes

Or R>=4.1, 78 bytes by replacing two function occurrences with \s.

function(l,a,b)cbind(rep(seq(l),lengths(w<-lapply(l,function(x)which(x%in%a:b)))),unlist(w))

Try it online!

R, 87 bytes

Or R>=4.1, 80 bytes by replacing the word function with a \.

function(l,a,b,`[`=lapply)cbind(rep(seq(l),lengths(w<-l[`%in%`,a:b][which])),unlist(w))

Try it online!

Solution shorter in R>=4.1:

R, 92 bytes

Or R>=4.1, 78 bytes by replacing two function occurrences with \s.

function(l,a,b)cbind(rep(seq(l),lengths(w<-lapply(l,function(x)which(x%in%a:b)))),unlist(w))

Try it online!

Outgolfed by @Giuseppe . See that answer for a detailed explanation and comparison of our approaches.