Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Sign up
Appearance settings

Commit cc62c23

Browse files
[feat] Add EsgfQuery class to wrap ESGF search RESTful APIs (#73)
1 parent ce7dadb commit cc62c23

25 files changed

+3906
-272
lines changed

‎DESCRIPTION‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
Package: epwshiftr
22
Title: Create Future 'EnergyPlus' Weather Files using 'CMIP6' Data
3-
Version: 0.1.3.9011
3+
Version: 0.1.3.9012
44
Authors@R: c(
55
person(given = "Hongyuan",
66
family = "Jia",
@@ -48,7 +48,7 @@ Encoding: UTF-8
4848
URL: https://github.com/ideas-lab-nus/epwshiftr
4949
BugReports: https://github.com/ideas-lab-nus/epwshiftr/issues
5050
Roxygen: list(markdown = TRUE)
51-
RoxygenNote: 7.2.0
51+
RoxygenNote: 7.2.1
5252
Collate:
5353
'coord.R'
5454
'dict.R'
@@ -58,5 +58,7 @@ Collate:
5858
'gh.R'
5959
'morph.R'
6060
'netcdf.R'
61+
'query.R'
6162
VignetteBuilder: knitr
6263
Config/testthat/start-first: utils, esgf, coord, netcdf, morph
64+
Config/testthat/edition: 3

‎NAMESPACE‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,9 @@
22

33
S3method(print,CMIP6CV)
44
S3method(print,CMIP6DReq)
5-
export(CMIP6Dict)
5+
S3method(print,EsgfQueryParam)
6+
export(Cmip6Dict)
7+
export(EsgfQuery)
68
export(cmip6_dict)
79
export(esgf_query)
810
export(extract_data)
@@ -13,6 +15,7 @@ export(init_cmip6_index)
1315
export(load_cmip6_index)
1416
export(match_coord)
1517
export(morphing_epw)
18+
export(query_esgf)
1619
export(set_cmip6_index)
1720
export(summary_database)
1821
importFrom(PCICt,as.PCICt)

‎NEWS.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,7 @@
7272
be able to capture the average of future climate (#41).
7373
* Now epwshiftr is able to download, parse and store CMIP6 Controlled
7474
Vocabularies (CVs) and Data Request data using the newly introduced class
75-
`CMIP6Dict`. Please see `?CMIP6Dict` for details (#53).
75+
`Cmip6Dict`. Please see `?Cmip6Dict` for details (#53).
7676
* A new option `epwshiftr.threshold_alpha` has been added to set the threshold
7777
of the absolute value for alpha, i.e. monthly-mean fractional change when
7878
performing morphing operations. The default value is set to `3`. If the
@@ -82,6 +82,8 @@
8282
* Now HDF5 format is supported (#60).
8383
* Now `replica` can be `NULL` in `esgf_query()` and `init_cmip6_index()`. In
8484
this case, both the master record and replicas are all returned (#61).
85+
* New class `EsgfQuery` is added to support more flexible. Please see
86+
`?EsgfQuery` for details (#63).
8587

8688
## Bug fixes
8789

@@ -96,7 +98,9 @@
9698
`morphing_epw()` (#25).
9799

98100
## Internal refactor
101+
99102
* `fields` parameter is used to directly filter the ESGF query responses (#66).
103+
* Improve URL encoding (#62).
100104

101105
# epwshiftr 0.1.3
102106

‎R/dict.R‎

Lines changed: 30 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
#' CMIP6 Controlled Vocabularies (CVs) and Data Request Dictionary
22
#'
3-
#' The `CMIP6Dict` object provides functionalities to fetch the latested CMIP6
3+
#' The `Cmip6Dict` object provides functionalities to fetch the latested CMIP6
44
#' Controlled Vocabularies (CVs) and Data Request (DReq) information.
55
#'
66
#' The CMIP6 CVs gives a well-defined set of global attributes that are recorded
@@ -15,7 +15,7 @@
1515
#' endorsed MIP. The raw data of DReq is stored a Microsoft Excel file
1616
#' (`CMIP6_MIP_tables.xlsx`) in a
1717
#' [Subversion repo](http://proj.badc.rl.ac.uk/svn/exarch/CMIP6dreq/trunk).
18-
#' The `CMIP6Dict` object uses the parsed DReq data that is stored in the
18+
#' The `Cmip6Dict` object uses the parsed DReq data that is stored in the
1919
#' [GitHub Repo](https://github.com/PCMDI/cmip6-cmor-tables).
2020
#'
2121
#' For more information, please see:
@@ -26,10 +26,10 @@
2626
#' @examples
2727
#' \dontrun{
2828
#'
29-
#' # create a new CMIP6Dict object
29+
#' # create a new Cmip6Dict object
3030
#' dict <- cmip6_dict()
3131
#'
32-
#' # by default, there is no data when the CMIP6Dict was created
32+
#' # by default, there is no data when the Cmip6Dict was created
3333
#' dict$is_empty()
3434
#'
3535
#' # fetch and parse all CVs and Data Request data
@@ -71,16 +71,16 @@
7171
#' @author Hongyuan Jia
7272
#'
7373
#' @importFrom R6 R6Class
74-
#' @name CMIP6Dict
74+
#' @name Cmip6Dict
7575
#' @export
7676
cmip6_dict <- function() {
77-
EPWSHIFTR_ENV$dict <- CMIP6Dict$new()
78-
EPWSHIFTR_ENV$dict
77+
this$dict <- Cmip6Dict$new()
78+
this$dict
7979
}
8080

81-
#' @name CMIP6Dict
81+
#' @name Cmip6Dict
8282
#' @export
83-
CMIP6Dict <- R6::R6Class("CMIP6Dict",
83+
Cmip6Dict <- R6::R6Class("Cmip6Dict",
8484
cloneable = FALSE, lock_class = TRUE,
8585
public = list(
8686
#' @description
@@ -96,9 +96,9 @@ CMIP6Dict <- R6::R6Class("CMIP6Dict",
9696
},
9797

9898
#' @description
99-
#' Is it an empty CMIP6Dict?
99+
#' Is it an empty Cmip6Dict?
100100
#'
101-
#' `$is_empty()` checks if this `CMIP6Dict` is empty, i.e. the `$build()
101+
#' `$is_empty()` checks if this `Cmip6Dict` is empty, i.e. the `$build()
102102
#' ` or `$load()` method hasn't been called yet and there is no data of
103103
#' CVs and Data Request.
104104
#'
@@ -144,8 +144,17 @@ CMIP6Dict <- R6::R6Class("CMIP6Dict",
144144
#' REST APIs. If `NULL`, `GITHUB_PAT` or `GITHUB_TOKEN`
145145
#' environment variable will be used if exists. Default: `NULL`.
146146
#'
147-
#' @return The updated `CMIP6Dict` itself.
148-
build = function(token = NULL) {
147+
#' @param force Whether to force to rebuild the dict when it has been
148+
#' already built before. Default: `FALSE`.
149+
#'
150+
#' @return The updated `Cmip6Dict` itself.
151+
build = function(token = NULL, force = FALSE) {
152+
assert_flag(force)
153+
154+
if (self$is_empty()) force <- TRUE
155+
156+
if (!force) return(self)
157+
149158
dict <- cmip6dict_build(cmip6dict_fetch())
150159
for (nm in names(dict)) private[[paste0("m_", nm)]] <- dict[[nm]]
151160
self
@@ -192,12 +201,12 @@ CMIP6Dict <- R6::R6Class("CMIP6Dict",
192201
},
193202

194203
#' @description
195-
#' Save the CMIP6Dict object
204+
#' Save the Cmip6Dict object
196205
#'
197-
#' `$save()` stores all the core data of current `CMIP6Dict` object into
206+
#' `$save()` stores all the core data of current `Cmip6Dict` object into
198207
#' an [RDS][saveRDS()] file named `CMIP6DICT` in the specified folder.
199208
#' This file can be reloaded via `$load()` method to restore the last
200-
#' state of current `CMIP6Dict` object.
209+
#' state of current `Cmip6Dict` object.
201210
#'
202211
#' @param dir A single string giving the directory to save the RDS file.
203212
#' Default is set to the global option `epwshiftr.dir`. The
@@ -218,7 +227,7 @@ CMIP6Dict <- R6::R6Class("CMIP6Dict",
218227
},
219228

220229
#' @description
221-
#' Load the saved CMIP6Dict object from file
230+
#' Load the saved Cmip6Dict object from file
222231
#'
223232
#' `$load()` loads the RDS file named `CMIP6DICT` that is created using
224233
#' `$save()` method.
@@ -248,12 +257,12 @@ CMIP6Dict <- R6::R6Class("CMIP6Dict",
248257
},
249258

250259
#' @description
251-
#' Print a summary of the current `CMIP6Dict` object
260+
#' Print a summary of the current `Cmip6Dict` object
252261
#'
253-
#' `$print()` gives the summary of current `CMIP6Dict` object including
262+
#' `$print()` gives the summary of current `Cmip6Dict` object including
254263
#' the version of CVs and Data Request, and the last built time.
255264
#'
256-
#' @return The `CMIP6Dict` object itself, invisibly.
265+
#' @return The `Cmip6Dict` object itself, invisibly.
257266
print = function() {
258267
d <- cli::cli_div(
259268
theme = list(rule = list("line-type" = "double"))
@@ -381,7 +390,7 @@ cmip6dict_download_cv_file <- function(tag, dir = tempdir(), token = NULL) {
381390

382391
file <- ""
383392
cli::cli_progress_step(
384-
"Downloading data of {.strong CMIP6 CVs} ['{.file {file}}']...",
393+
"Downloading data of {.strong CMIP6 CVs} [{.file {file}}]...",
385394
"Downloaded data of {.strong CMIP6 CVs} successfully.",
386395
"Failed to download data of {.strong CMIP6 CVs}.",
387396
spinner = TRUE

‎R/epwshiftr-package.R‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -25,9 +25,14 @@
2525
"_PACKAGE"
2626

2727
# package internal environment
28-
EPWSHIFTR_ENV <- new.env(parent = emptyenv())
29-
EPWSHIFTR_ENV$index_db <- NULL
30-
EPWSHIFTR_ENV$dict <- NULL
28+
this <- new.env(parent = emptyenv())
29+
this$index_db <- NULL
30+
this$dict <- NULL
31+
this$cache <- list()
32+
33+
# nocov start
34+
attach_cache <- function(cache) this$cache <- cache
35+
# nocov end
3136

3237
## usethis namespace: start
3338
#' @importFrom checkmate assert_string

‎R/esgf.R‎

Lines changed: 44 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -121,10 +121,10 @@ RES_FILE <- c(
121121
#' described below. Default: `"r1i1p1f1"`.
122122
#' If `NULL`, all possible variants are returned.
123123
#'
124-
#' * `r`: realization_index (<k>) = realization number (integer >0)
125-
#' * `i`: initialization_index (<l>) = index for variant of initialization method (integer >0)
126-
#' * `p`: physics_index (<m>) = index for model physics variant (integer >0)
127-
#' * `f`: forcing_index (<n>) = index for variant of forcing (integer >0)
124+
#' * `r`: realization_index (`<k>`) = realization number (integer >0)
125+
#' * `i`: initialization_index (`<l>`) = index for variant of initialization method (integer >0)
126+
#' * `p`: physics_index (`<m>`) = index for model physics variant (integer >0)
127+
#' * `f`: forcing_index (`<n>`) = index for variant of forcing (integer >0)
128128
#'
129129
#' @param replica Whether the record is the "master" copy, or a replica. Use
130130
#' `FALSE` to return only originals and `TRUE` to return only replicas.
@@ -264,55 +264,71 @@ esgf_query <- function(activity = "ScenarioMIP",
264264
variant = "variant_label"
265265
)
266266

267-
pair <- function(x, first = FALSE) {
267+
pair <- function(x, encode = TRUE) {
268+
checkmate::assert_vector(x, TRUE, null.ok = TRUE)
269+
268270
# get name
269271
var <- deparse(substitute(x))
272+
270273
# skip if empty
271274
if (is.null(x) || length(x) == 0) {
272275
return()
273276
}
274277
# get key name
275278
key <- dict[names(dict) == var]
276279
if (!length(key)) key <- var
280+
277281
if (is.logical(x)) x <- tolower(x)
278-
s <- paste0(key, "=", paste0(x, collapse = "%2C")) # %2C = ","
279-
if (first) s else paste0("&", s)
282+
283+
if (encode) x <- query_param_encode(as.character(x))
284+
285+
paste0(key, "=", paste0(x, collapse = query_param_encode(",")))
280286
}
281287

282-
`%and%` <- function(lhs, rhs) if (is.null(rhs)) lhs else paste0(lhs, rhs)
288+
`%and%` <- function(lhs, rhs) {
289+
if (is.null(rhs)) {
290+
lhs
291+
} else if (lhs == url_base) {
292+
paste(lhs, rhs, sep = "", collapse = "")
293+
} else {
294+
paste(lhs, rhs, sep = "&", collapse = "&")
295+
}
296+
}
283297

284298
project <- "CMIP6"
285-
format <- "application%2Fsolr%2Bjson"
299+
format <- "application/solr+json"
300+
offset <- 0L
286301

287302
resolution <- c(
288303
gsub(" ", "", resolution, fixed = TRUE),
289304
gsub(" ", "+", resolution, fixed = TRUE)
290305
)
291306

307+
# use `fileds` to directly subset data from responses
308+
if (type == "Dataset") {
309+
fields <- RES_DATASET
310+
} else if (type == "File") {
311+
fields <- RES_FILE
312+
}
313+
292314
q <- url_base %and%
293-
pair(project, TRUE) %and%
315+
pair(offset) %and%
316+
pair(limit) %and%
317+
pair(type) %and%
318+
pair(replica) %and%
319+
pair(latest) %and%
320+
pair(project) %and%
294321
pair(activity) %and%
295322
pair(experiment) %and%
296323
pair(source) %and%
297324
pair(variable) %and%
298-
pair(resolution) %and%
325+
pair(resolution, FALSE) %and%
299326
pair(variant) %and%
300327
pair(data_node) %and%
301328
pair(frequency) %and%
302-
pair(replica) %and%
303-
pair(latest) %and%
304-
pair(type) %and%
305-
pair(limit) %and%
329+
pair(fields) %and%
306330
pair(format)
307331

308-
# use `fileds` to directly subset data from responses
309-
if (type == "Dataset") {
310-
fields <- RES_DATASET
311-
} else if (type == "File") {
312-
fields <- RES_FILE
313-
}
314-
q <- q %and% pair(fields)
315-
316332
q <- tryCatch(jsonlite::read_json(q), warning = function(w) w, error = function(e) e)
317333

318334
# nocov start
@@ -573,7 +589,7 @@ init_cmip6_index <- function(activity = "ScenarioMIP",
573589
data.table::fwrite(dt, file.path(.data_dir(TRUE), "cmip6_index.csv"))
574590
verbose("Data file index saved to '", normalizePath(file.path(.data_dir(TRUE), "cmip6_index.csv")), "'")
575591

576-
EPWSHIFTR_ENV$index_db <- data.table::copy(dt)
592+
this$index_db <- data.table::copy(dt)
577593
}
578594

579595
dt
@@ -596,10 +612,10 @@ init_cmip6_index <- function(activity = "ScenarioMIP",
596612
#' @importFrom data.table copy fread
597613
#' @export
598614
load_cmip6_index <- function(force = FALSE) {
599-
if (is.null(EPWSHIFTR_ENV$index_db)) force <- TRUE
615+
if (is.null(this$index_db)) force <- TRUE
600616

601617
if (!force) {
602-
idx <- data.table::copy(EPWSHIFTR_ENV$index_db)
618+
idx <- data.table::copy(this$index_db)
603619
} else {
604620
f <- normalizePath(file.path(.data_dir(force = FALSE), "cmip6_index.csv"), mustWork = FALSE)
605621
if (!file.exists(f)) {
@@ -661,7 +677,7 @@ load_cmip6_index <- function(force = FALSE) {
661677
}
662678

663679
# udpate package internal stored file index
664-
EPWSHIFTR_ENV$index_db <- data.table::copy(idx)
680+
this$index_db <- data.table::copy(idx)
665681

666682
idx[]
667683
}
@@ -707,7 +723,7 @@ set_cmip6_index <- function(index, save = FALSE) {
707723
}
708724

709725
# udpate package internal stored file index
710-
EPWSHIFTR_ENV$index_db <- data.table::copy(index)
726+
this$index_db <- data.table::copy(index)
711727

712728
invisible(index)
713729
}

‎R/morph.R‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -157,15 +157,15 @@ remove_units <- function (data, var) {
157157
#'
158158
#' If using a shift, for each month, a shift \eqn{\Delta x_m} is applied to
159159
#' \eqn{x_0}. \eqn{\Delta x_m} is the absolute change in the monthly mean value
160-
#' of the variable for the month \eqn{m}, i.e. \eqn{\Delta x_m = <x_0>_m - <x>_m}
161-
#' . Here the monthly variance of the variable
162-
#' is unchanged.
160+
#' of the variable for the month \eqn{m},
161+
#' i.e. \eqn{\Delta x_m = <x_0>_m - <x>_m}. Here the monthly variance of the
162+
#' variable is unchanged.
163163
#'
164164
#' ## Stretch:
165165
#'
166166
#' If using a stretch, for each month, a stretch \eqn{\alpha _m} is applied to
167167
#' \eqn{x_0}, where \eqn{\alpha _m} is the fractional change in the monthly-mean
168-
#' value of a variable, i.e. \eqn{\alpha _m} = <x>_m / <x_0>_m. In this case,
168+
#' value of a variable, i.e. \eqn{\alpha _m = <x>_m / <x_0>_m}. In this case,
169169
#' the variance will be multiplied by to \eqn{alpha^2_m}
170170
#'
171171
#' ## Combined Shift and Stretch:

‎R/netcdf.R‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -406,7 +406,7 @@ summary_database <- function (
406406
}
407407

408408
# update index
409-
EPWSHIFTR_ENV$index_db <- copy(idx)
409+
this$index_db <- copy(idx)
410410

411411
if (update) {
412412
# save database into the app data directory

0 commit comments

Comments
(0)

AltStyle によって変換されたページ (->オリジナル) /