Enhancement: Add notes of R versions when functions were added or behavior changed in backwards incompatible way #1

eshom commented 2024年08月09日 19:54:04 +02:00

This would be nice to have, especially for backwards incompatible changes. It would also be a good thing for package developers who want to maximize backwards compatibility.

I've found myself needing this today when a test I wrote for Rf_asInteger in R-4.4.1 failed. In April 17th 2024 support for coercing RAWSXP to int was added. Referencing the work in progress API documentation, it mentions that RAWSXP should be supported, but it's a new unreleased addition to Rf_asInteger behavior. So this behavior is not supported for R <= 4.4.1.

I think this would require going through all C-LEVEL FACILITIES sections, and maybe with help of svn blame for cases of undocumented changes.

This would be nice to have, especially for backwards incompatible changes. It would also be a good thing for package developers who want to maximize backwards compatibility. I've found myself needing this today when a test I wrote for `Rf_asInteger` in `R-4.4.1` failed. In April 17th 2024 support for coercing `RAWSXP` to `int` was added. Referencing the work in progress API documentation, it mentions that `RAWSXP` should be supported, but it's a new unreleased addition to `Rf_asInteger` behavior. So this behavior is not supported for `R <= 4.4.1`. I think this would require going through all `C-LEVEL FACILITIES` sections, and maybe with help of `svn blame` for cases of undocumented changes.

aitap commented 2024年08月10日 18:25:23 +02:00

That would be quite useful to have, thanks for letting me know. I've been following R-devel so far, so there are more unmarked R ≥ 4.5.0 functions.

Would you prefer the version information in a separate subsection (like "arguments" and "return value and side effects" right now)? Should I instead put the version differences in the middle of the text itself?

That would be quite useful to have, thanks for letting me know. I've been following R-devel so far, so there are more unmarked R ≥ 4.5.0 functions. Would you prefer the version information in a separate subsection (like "arguments" and "return value and side effects" right now)? Should I instead put the version differences in the middle of the text itself?

eshom commented 2024年08月10日 18:42:40 +02:00

In my opinion, I don't think a subsection is needed. In the middle of the text works well enough for python docs (which I used as style reference).

For example in the case I mentioned something like this is sufficient:

Rf_asInteger
 RAWSXP (New in 4.5.0), LGLSXP, INTSXP, REALSXP (may warn), CPLXSXP (may warn), STRSXP (may warn), CHARSXP (may warn) 

In my opinion, I don't think a subsection is needed. In the middle of the text works well enough for python docs (which I used as style reference). For example in the case I mentioned something like this is sufficient: ``` Rf_asInteger RAWSXP (New in 4.5.0), LGLSXP, INTSXP, REALSXP (may warn), CPLXSXP (may warn), STRSXP (may warn), CHARSXP (may warn) ```

aitap commented 2024年08月11日 09:32:46 +02:00

Thanks! And if a function as a whole has been introduced in a recent version of R, I'll add a paragraph at the end of the page, like here (subject to the limitations of Texinfo).

Want to become a committer? I've been rebasing my branches (even trunk, one last time before I shared the link on the list), but I don't have to keep doing that if you would like to work with them.

Thanks! And if a function as a whole has been introduced in a recent version of R, I'll add a paragraph at the end of the page, like [here](https://docs.python.org/3/library/functions.html#breakpoint) (subject to the limitations of Texinfo). Want to become a committer? I've been rebasing my branches (even `trunk`, one last time before I shared the link on the list), but I don't have to keep doing that if you would like to work with them.

eshom commented 2024年08月11日 17:43:17 +02:00

Looks nice so far, I can try to contribute a bit if I can. I don't know the R API as well (I'm still learning slowly for my own bindings project) but I can look up in R's NEWS for functions / changes you are less sure of. Let me know.

Looks nice so far, I can try to contribute a bit if I can. I don't know the R API as well (I'm still learning slowly for my own bindings project) but I can look up in R's NEWS for functions / changes you are less sure of. Let me know.

aitap commented 2024年08月11日 18:58:48 +02:00

We're only human, and trying to create understanding of a system that has been evolving for more than 30 years. Your tests caught something I didn't notice. I'm afraid it's almost a given that there's more to catch.

We're only human, and trying to create understanding of a system that has been evolving for more than 30 years. Your tests caught something I didn't notice. I'm afraid it's almost a given that there's more to catch.

aitap commented 2024年08月17日 15:33:23 +02:00

Speaking of R versions, out of 21158 packages in tools::CRAN_package_db(), 67% declare a dependency on an R version:

tools::CRAN_package_db() -> db
db$Depends |>
 regexec('\\<R\\s*\\(\\s*>=*\\s*([0-9.]+)', text = _) |>
 regmatches(x = db$Depends) |>
 Filter(f = length) |>
 vapply(`[[`, '', 2) |> table() -> R_versions
plot(R_versions[order(numeric_version(names(R_versions)))])

rver.png

Not all of them are valid. R CMD build automatically adds a dependency on R >= 2.10 if it sees data files compressed with xz or bzip2 and R >= 3.5.0 if it sees data files in serialization format 3, which accounts for the two highest peaks. Some of the declared versions (3.7.0, 3.8, 3.10, 3.10.0, 3.21, 3.50, 3.60, 3.63) never existed at all. Almost nobody actually tests their packages with the stated R version; a few exceptions I know are data.table (3.1.0) and this.path (2.15.0!).

Speaking of R versions, out of 21158 packages in `tools::CRAN_package_db()`, 67% declare a dependency on an R version: ``` tools::CRAN_package_db() -> db db$Depends |> regexec('\\<R\\s*\\(\\s*>=*\\s*([0-9.]+)', text = _) |> regmatches(x = db$Depends) |> Filter(f = length) |> vapply(`[[`, '', 2) |> table() -> R_versions plot(R_versions[order(numeric_version(names(R_versions)))]) ```  Not all of them are valid. `R CMD build` automatically adds a dependency on `R >= 2.10` if it sees data files compressed with `xz` or `bzip2` and `R >= 3.5.0` if it sees data files in serialization format 3, which accounts for the two highest peaks. Some of the declared versions (3.7.0, 3.8, 3.10, 3.10.0, 3.21, 3.50, 3.60, 3.63) never existed at all. Almost nobody actually tests their packages with the stated R version; a few exceptions I know are [data.table](https://cran.r-project.org/package=data.table) (3.1.0) and [this.path](https://cran.r-project.org/package=this.path) (2.15.0!).

eshom commented 2024年08月17日 16:27:26 +02:00

Interesting.

It also makes sense to see the distribution of the subset of packages that need compilation (4810), as they are ones that directly or indirectly (Rcpp) use the API. Without Rcpp (when not mentioned in Imports) we are left with 2175 packages.

tools::CRAN_package_db() -> db
par(mfrow = c(3, 1))
db$Depends |>
 regexec("\\<R\\s*\\(\\s*>=*\\s*([0-9.]+)", text = _) |>
 regmatches(x = db$Depends) |>
 Filter(f = length) |>
 vapply(`[[`, "", 2) |>
 table() -> R_versions
plot(R_versions[order(numeric_version(names(R_versions)))],
 xlab = "Declared R version dependency", ylab = "Package Count",
 main = "All"
)
db_c <- subset(db, NeedsCompilation == "yes")
db_c$Depends |>
 regexec("\\<R\\s*\\(\\s*>=*\\s*([0-9.]+)", text = _) |>
 regmatches(x = db_c$Depends) |>
 Filter(f = length) |>
 vapply(`[[`, "", 2) |>
 table() -> R_versions
plot(R_versions[order(numeric_version(names(R_versions)))],
 xlab = "Declared R version dependency", ylab = "Package Count",
 main = "NeedsCompilation = yes"
)
db_no_rcpp <- subset(db, !grepl(".*Rcpp.*", Imports) & NeedsCompilation == "yes", select = "Package", drop = TRUE)
db_c$Depends |>
 regexec("\\<R\\s*\\(\\s*>=*\\s*([0-9.]+)", text = _) |>
 regmatches(x = db_c$Depends) |>
 Filter(f = length) |>
 vapply(`[[`, "", 2) |>
 table() -> R_versions
plot(R_versions[order(numeric_version(names(R_versions)))],
 xlab = "Declared R version dependency", ylab = "Package Count",
 main = "NeedsCompilation = yes & No Rcpp"
)

image

Interesting. It also makes sense to see the distribution of the subset of packages that need compilation (4810), as they are ones that directly or indirectly (`Rcpp`) use the API. Without `Rcpp` (when not mentioned in `Imports`) we are left with 2175 packages. ```R tools::CRAN_package_db() -> db par(mfrow = c(3, 1)) db$Depends |> regexec("\\<R\\s*\\(\\s*>=*\\s*([0-9.]+)", text = _) |> regmatches(x = db$Depends) |> Filter(f = length) |> vapply(`[[`, "", 2) |> table() -> R_versions plot(R_versions[order(numeric_version(names(R_versions)))], xlab = "Declared R version dependency", ylab = "Package Count", main = "All" ) db_c <- subset(db, NeedsCompilation == "yes") db_c$Depends |> regexec("\\<R\\s*\\(\\s*>=*\\s*([0-9.]+)", text = _) |> regmatches(x = db_c$Depends) |> Filter(f = length) |> vapply(`[[`, "", 2) |> table() -> R_versions plot(R_versions[order(numeric_version(names(R_versions)))], xlab = "Declared R version dependency", ylab = "Package Count", main = "NeedsCompilation = yes" ) db_no_rcpp <- subset(db, !grepl(".*Rcpp.*", Imports) & NeedsCompilation == "yes", select = "Package", drop = TRUE) db_c$Depends |> regexec("\\<R\\s*\\(\\s*>=*\\s*([0-9.]+)", text = _) |> regmatches(x = db_c$Depends) |> Filter(f = length) |> vapply(`[[`, "", 2) |> table() -> R_versions plot(R_versions[order(numeric_version(names(R_versions)))], xlab = "Declared R version dependency", ylab = "Package Count", main = "NeedsCompilation = yes & No Rcpp" ) ```