Testing your code#

Hypothesis testing#

Note

Testing with hypothesis is a fairly advanced topic. Before reading this section it is recommended that you take a look at our guide to xarray’s Data Structures, are familiar with conventional unit testing in pytest, and have seen the hypothesis library documentation.

The hypothesis library is a powerful tool for property-based testing. Instead of writing tests for one example at a time, it allows you to write tests parameterized by a source of many dynamically generated examples. For example you might have written a test which you wish to be parameterized by the set of all possible integers via hypothesis.strategies.integers().

Property-based testing is extremely powerful, because (unlike more conventional example-based testing) it can find bugs that you did not even think to look for!

Strategies#

Each source of examples is called a "strategy", and xarray provides a range of custom strategies which produce xarray data structures containing arbitrary data. You can use these to efficiently test downstream code, quickly ensuring that your code can handle xarray objects of all possible structures and contents.

These strategies are accessible in the xarray.testing.strategies module, which provides

testing.strategies.supported_dtypes()

Generates only those numpy dtypes which xarray can handle.

testing.strategies.names()

Generates arbitrary string names for dimensions / variables.

testing.strategies.dimension_names(*[, ...])

Generates an arbitrary list of valid dimension names.

testing.strategies.dimension_sizes(*[, ...])

Generates an arbitrary mapping from dimension names to lengths.

testing.strategies.attrs()

Generates arbitrary valid attributes dictionaries for xarray objects.

testing.strategies.variables(*[, ...])

Generates arbitrary xarray.Variable objects.

testing.strategies.unique_subset_of(objs, *)

Return a strategy which generates a unique subset of the given objects.

These build upon the numpy and array API strategies offered in hypothesis.extra.numpy and hypothesis.extra.array_api:

importhypothesis.extra.numpyasnpst

Generating Examples#

To see an example of what each of these strategies might produce, you can call one followed by the .example() method, which is a general hypothesis method valid for all strategies.

importxarray.testing.strategiesasxrst
xrst.variables().example()

<xarray.Variable (ŗõ: 1, Eż: 1)> Size: 8B
array([[-9223372036854775670]])
Attributes:
 : [b'\xebq\xfe\xbd\xb7DiV\xb4\xbb' b'\x01']
 s: False
 ś: None

xarray.Variable

• ŗõ: 1
• Eż: 1

• -9223372036854775670

  array([[-9223372036854775670]])

• Attributes: (3)

  :

  [b'\xebq\xfe\xbd\xb7DiV\xb4\xbb' b'\x01']

  s :

  False

  ś :

  None

xrst.variables().example()

<xarray.Variable (ăŸI: 6, sĠz: 4)> Size: 384B
array([[ 8.07621540e+163-1.88216603e+016j,
 8.07621540e+163-1.88216603e+016j,
 8.07621540e+163-1.88216603e+016j,
 8.07621540e+163-1.88216603e+016j],
 [ 8.07621540e+163-1.88216603e+016j,
 8.07621540e+163-1.88216603e+016j,
 8.07621540e+163-1.88216603e+016j,
 8.07621540e+163-1.88216603e+016j],
 [ 8.07621540e+163-1.88216603e+016j,
 8.07621540e+163-1.88216603e+016j,
 8.07621540e+163-1.88216603e+016j,
 8.07621540e+163-1.88216603e+016j],
 [ 8.07621540e+163-1.88216603e+016j,
 8.07621540e+163-1.88216603e+016j,
 -6.69873801e+016-2.09143316e-203j,
 8.07621540e+163-1.88216603e+016j],
 [ 8.07621540e+163-1.88216603e+016j,
 8.07621540e+163-1.88216603e+016j,
 8.07621540e+163-1.88216603e+016j,
 8.07621540e+163-1.88216603e+016j],
 [ 8.07621540e+163-1.88216603e+016j,
 8.07621540e+163-1.88216603e+016j,
 8.07621540e+163-1.88216603e+016j,
 8.07621540e+163-1.88216603e+016j]])

xarray.Variable

• ăŸI: 6
• sĠz: 4

• (8.076215403713818e+163-1.8821660279064784e+16j) ... (8.07621540371...

  array([[ 8.07621540e+163-1.88216603e+016j,
   8.07621540e+163-1.88216603e+016j,
   8.07621540e+163-1.88216603e+016j,
   8.07621540e+163-1.88216603e+016j],
   [ 8.07621540e+163-1.88216603e+016j,
   8.07621540e+163-1.88216603e+016j,
   8.07621540e+163-1.88216603e+016j,
   8.07621540e+163-1.88216603e+016j],
   [ 8.07621540e+163-1.88216603e+016j,
   8.07621540e+163-1.88216603e+016j,
   8.07621540e+163-1.88216603e+016j,
   8.07621540e+163-1.88216603e+016j],
   [ 8.07621540e+163-1.88216603e+016j,
   8.07621540e+163-1.88216603e+016j,
   -6.69873801e+016-2.09143316e-203j,
   8.07621540e+163-1.88216603e+016j],
   [ 8.07621540e+163-1.88216603e+016j,
   8.07621540e+163-1.88216603e+016j,
   8.07621540e+163-1.88216603e+016j,
   8.07621540e+163-1.88216603e+016j],
   [ 8.07621540e+163-1.88216603e+016j,
   8.07621540e+163-1.88216603e+016j,
   8.07621540e+163-1.88216603e+016j,
   8.07621540e+163-1.88216603e+016j]])

xrst.variables().example()

<xarray.Variable (0: 1)> Size: 1B
array([0], dtype=int8)

xarray.Variable

• 0: 1

• 0

  array([0], dtype=int8)

You can see that calling .example() multiple times will generate different examples, giving you an idea of the wide range of data that the xarray strategies can generate.

In your tests however you should not use .example() - instead you should parameterize your tests with the hypothesis.given() decorator:

fromhypothesisimport given

@given(xrst.variables())
deftest_function_that_acts_on_variables(var):
 assert func(var) == ...

Chaining Strategies#

Xarray’s strategies can accept other strategies as arguments, allowing you to customise the contents of the generated examples.

# generate a Variable containing an array with a complex number dtype, but all other details still arbitrary
fromhypothesis.extra.numpyimport complex_number_dtypes
xrst.variables(dtype=complex_number_dtypes()).example()

<xarray.Variable (0: 1)> Size: 8B
array([0.+0.j], dtype='>c8')

xarray.Variable

• 0: 1

• 0j

  array([0.+0.j], dtype='>c8')

This also works with custom strategies, or strategies defined in other packages. For example you could imagine creating a chunks strategy to specify particular chunking patterns for a dask-backed array.

Fixing Arguments#

If you want to fix one aspect of the data structure, whilst allowing variation in the generated examples over all other aspects, then use hypothesis.strategies.just().

importhypothesis.strategiesasst
# Generates only variable objects with dimensions ["x", "y"]
xrst.variables(dims=st.just(["x", "y"])).example()

<xarray.Variable (x: 1, y: 1)> Size: 2B
array([[0.]], dtype=float16)

xarray.Variable

• x: 1
• y: 1

• 0.0

  array([[0.]], dtype=float16)

(This is technically another example of chaining strategies - hypothesis.strategies.just() is simply a special strategy that just contains a single example.)

To fix the length of dimensions you can instead pass dims as a mapping of dimension names to lengths (i.e. following xarray objects’ .sizes() property), e.g.

# Generates only variables with dimensions ["x", "y"], of lengths 2 & 3 respectively
xrst.variables(dims=st.just({"x": 2, "y": 3})).example()

<xarray.Variable (x: 2, y: 3)> Size: 24B
array([[-0.00000000e+00, -inf, -3.22830410e-35],
 [ 1.00000001e-01, -1.06156635e+14, -2.22044605e-16]], dtype=float32)
Attributes:
 ïÓ: {}

xarray.Variable

• x: 2
• y: 3

• -0.0 -inf -3.228e-35 0.1 -1.062e+14 -2.22e-16

  array([[-0.00000000e+00, -inf, -3.22830410e-35],
   [ 1.00000001e-01, -1.06156635e+14, -2.22044605e-16]], dtype=float32)

• Attributes: (1)

  ïÓ :

  {}

You can also use this to specify that you want examples which are missing some part of the data structure, for instance

# Generates a Variable with no attributes
xrst.variables(attrs=st.just({})).example()

<xarray.Variable (ĝä: 4, Ŗ: 4)> Size: 64B
array([[ 193, 55111, 4, 4287757306],
 [ 283005341, 18587, 116, 55111],
 [ 56820, 55111, 51267, 37770],
 [ 55111, 154853659, 55111, 55111]], dtype=uint32)

xarray.Variable

• ĝä: 4
• Ŗ: 4

• 193 55111 4 4287757306 283005341 ... 37770 55111 154853659 55111 55111

  array([[ 193, 55111, 4, 4287757306],
   [ 283005341, 18587, 116, 55111],
   [ 56820, 55111, 51267, 37770],
   [ 55111, 154853659, 55111, 55111]], dtype=uint32)

Through a combination of chaining strategies and fixing arguments, you can specify quite complicated requirements on the objects your chained strategy will generate.

fixed_x_variable_y_maybe_z = st.fixed_dictionaries(
 {"x": st.just(2), "y": st.integers(3, 4)}, optional={"z": st.just(2)}
)
fixed_x_variable_y_maybe_z.example()

{'x': 2, 'y': 3, 'z': 2}

special_variables = xrst.variables(dims=fixed_x_variable_y_maybe_z)
special_variables.example()

<xarray.Variable (x: 2, y: 3, z: 2)> Size: 48B
array([[[ 7.8677863e+15, -3.8497879e+16],
 [-4.4211414e+16, 6.7641650e+16],
 [ 1.4012985e-45, 2.3187232e-17]],
 [[-6.6607113e+16, 1.1920929e-07],
 [ 2.3187232e-17, -1.0000000e-01],
 [-6.9295788e+16, 4.3072831e+16]]], dtype=float32)
Attributes:
 DžKŻ2: ['\x8eÇ\U0007795d\x19\x8d\U0004ce1eÄ\x07' 'o㒦¤']
 ä: fţÉż9
 L·ĉËÆĂ: None

xarray.Variable

• x: 2
• y: 3
• z: 2

• 7.868e+15 -3.85e+16 -4.421e+16 6.764e+16 ... -0.1 -6.93e+16 4.307e+16

  array([[[ 7.8677863e+15, -3.8497879e+16],
   [-4.4211414e+16, 6.7641650e+16],
   [ 1.4012985e-45, 2.3187232e-17]],
   [[-6.6607113e+16, 1.1920929e-07],
   [ 2.3187232e-17, -1.0000000e-01],
   [-6.9295788e+16, 4.3072831e+16]]], dtype=float32)

• Attributes: (3)

  DžKŻ2 :

  ['\x8eÇ\U0007795d\x19\x8d\U0004ce1eÄ\x07' 'o㒦¤']

  ä :

  fţÉż9

  L·ĉËÆĂ :

  None

special_variables.example()

<xarray.Variable (x: 2, y: 3, z: 2)> Size: 96B
array([[[ 21605, 33764],
 [ 41479, 27518],
 [ 217440412, 10000000000001]],
 [[ 26044, 9772],
 [ 71, 98],
 [ 50451, 3344109754]]], dtype=uint64)

xarray.Variable

• x: 2
• y: 3
• z: 2

• 21605 33764 41479 27518 217440412 ... 9772 71 98 50451 3344109754

  array([[[ 21605, 33764],
   [ 41479, 27518],
   [ 217440412, 10000000000001]],
   [[ 26044, 9772],
   [ 71, 98],
   [ 50451, 3344109754]]], dtype=uint64)

Here we have used one of hypothesis’ built-in strategies hypothesis.strategies.fixed_dictionaries() to create a strategy which generates mappings of dimension names to lengths (i.e. the size of the xarray object we want). This particular strategy will always generate an x dimension of length 2, and a y dimension of length either 3 or 4, and will sometimes also generate a z dimension of length 2. By feeding this strategy for dictionaries into the dims argument of xarray’s variables() strategy, we can generate arbitrary Variable objects whose dimensions will always match these specifications.

Generating Duck-type Arrays#

Xarray objects don’t have to wrap numpy arrays, in fact they can wrap any array type which presents the same API as a numpy array (so-called "duck array wrapping", see wrapping numpy-like arrays).

Imagine we want to write a strategy which generates arbitrary Variable objects, each of which wraps a sparse.COO array instead of a numpy.ndarray. How could we do that? There are two ways:

1. Create an xarray object with numpy data and use the hypothesis’ .map() method to convert the underlying array to a different type:

importsparse

defconvert_to_sparse(var):
 return var.copy(data=sparse.COO.from_numpy(var.to_numpy()))

sparse_variables = xrst.variables(dims=xrst.dimension_names(min_dims=1)).map(
 convert_to_sparse
)
sparse_variables.example()

<xarray.Variable (ž: 4)> Size: 96B
<COO: shape=(4,), dtype=complex128, nnz=4, fill_value=0j>

xarray.Variable

• ž: 4

• <COO: nnz=4, fill_value=0j>

  Format	coo
  Data Type	complex128
  Shape	(4,)
  nnz	4
  Density	1.0
  Read-only	True
  Size	96
  Storage ratio	1.50

sparse_variables.example()

<xarray.Variable (s: 6)> Size: 96B
<COO: shape=(6,), dtype=complex64, nnz=6, fill_value=0j>
Attributes:
 ÄŶ: None

xarray.Variable

• s: 6

• <COO: nnz=6, fill_value=0j>

  Format	coo
  Data Type	complex64
  Shape	(6,)
  nnz	6
  Density	1.0
  Read-only	True
  Size	96
  Storage ratio	2.00

• Attributes: (1)

  ÄŶ :

  None

2. Pass a function which returns a strategy which generates the duck-typed arrays directly to the array_strategy_fn argument of the xarray strategies:

defsparse_random_arrays(shape: tuple[int, ...]) -> sparse._coo.core.COO:
"""Strategy which generates random sparse.COO arrays"""
 if shape is None:
 shape = npst.array_shapes()
 else:
 shape = st.just(shape)
 density = st.integers(min_value=0, max_value=1)
 # note sparse.random does not accept a dtype kwarg
 return st.builds(sparse.random, shape=shape, density=density)
defsparse_random_arrays_fn(
 *, shape: tuple[int, ...], dtype: np.dtype
) -> st.SearchStrategy[sparse._coo.core.COO]:
 return sparse_random_arrays(shape=shape)

sparse_random_variables = xrst.variables(
 array_strategy_fn=sparse_random_arrays_fn, dtype=st.just(np.dtype("float64"))
)
sparse_random_variables.example()

<xarray.Variable (oŽ: 3, ŽÎu: 4)> Size: 0B
<COO: shape=(3, 4), dtype=float64, nnz=0, fill_value=0.0>
Attributes:
 ŻĠIĩŽ: [b'\xa2\xb1\xa2\x9b\xa0\xca+z\x96!e' b"j4\x85EX\xcb\xe9\x18\xad...
 ŴŻďwę: ŤÕ
 ÔÖžĖs: False
 ġł4ĄŽ: ŸĮ
 1⁄4: True
 ĴŲYłŬ: 
 ž: None

xarray.Variable

• oŽ: 3
• ŽÎu: 4

• <COO: nnz=0, fill_value=0.0>

  Format	coo
  Data Type	float64
  Shape	(3, 4)
  nnz	0
  Density	0.0
  Read-only	True
  Size	0
  Storage ratio	0.00

• Attributes: (7)

  ŻĠIĩŽ :

  [b'\xa2\xb1\xa2\x9b\xa0\xca+z\x96!e' b"j4\x85EX\xcb\xe9\x18\xad'\x94"]

  ŴŻďwę :

  ŤÕ

  ÔÖžĖs :

  False

  ġł4ĄŽ :

  ŸĮ

  1⁄4 :

  True

  ĴŲYłŬ :

  ž :

  None

Either approach is fine, but one may be more convenient than the other depending on the type of the duck array which you want to wrap.

Compatibility with the Python Array API Standard#

Xarray aims to be compatible with any duck-array type that conforms to the Python Array API Standard (see our docs on Array API Standard support).

Warning

The strategies defined in testing.strategies are not guaranteed to use array API standard-compliant dtypes by default. For example arrays with the dtype np.dtype('float16') may be generated by testing.strategies.variables() (assuming the dtype kwarg was not explicitly passed), despite np.dtype('float16') not being in the array API standard.

If the array type you want to generate has an array API-compliant top-level namespace (e.g. that which is conventionally imported as xp or similar), you can use this neat trick:

importnumpyasxp # compatible in numpy 2.0
# use `import numpy.array_api as xp` in numpy>=1.23,<2.0
fromhypothesis.extra.array_apiimport make_strategies_namespace
xps = make_strategies_namespace(xp)
xp_variables = xrst.variables(
 array_strategy_fn=xps.arrays,
 dtype=xps.scalar_dtypes(),
)
xp_variables.example()

<xarray.Variable (Ŋ: 5, 0ŻL·ĀO: 1)> Size: 40B
array([[ 5932030191829622144],
 [-9223372036854775775],
 [-9223372036854775685],
 [ 4668046844678045748],
 [ 7156817483075960783]])
Attributes: (12/17)
 ýŸŬŦ: 
 Ê: 1
 įÔĄżĪ: None
 Ÿďl: Ũ
 ğÛŠôJ: True
 l·ö: False
 ... ...
 Yžu: None
 ńĖũÁ1⁄4: False
 3ʼntl·Ĝ: [[b'\xaditd\x8a\x04']\n [b'']]
 aĆBÃŪ: ÇŽŞī
 ĦżUHŻ: None
 sŏŷĸĵ: [[15467]\n [ 1582]]

xarray.Variable

• Ŋ: 5
• 0ŻL·ĀO: 1

• 5932030191829622144 -9223372036854775775 ... 7156817483075960783

  array([[ 5932030191829622144],
   [-9223372036854775775],
   [-9223372036854775685],
   [ 4668046844678045748],
   [ 7156817483075960783]])

• Attributes: (17)

  ýŸŬŦ :

  Ê :

  1

  įÔĄżĪ :

  None

  Ÿďl :

  Ũ

  ğÛŠôJ :

  True

  l·ö :

  False

  žsŔ :

  True

  :

  False

  ú :

  None

  ÐÍŽŻ :

  True

  ž :

  None

  Yžu :

  None

  ńĖũÁ1⁄4 :

  False

  3ʼntl·Ĝ :

  [[b'\xaditd\x8a\x04'] [b'']]

  aĆBÃŪ :

  ÇŽŞī

  ĦżUHŻ :

  None

  sŏŷĸĵ :

  [[15467] [ 1582]]

Another array API-compliant duck array library would replace the import, e.g. import cupy as cp instead.

Testing over Subsets of Dimensions#

A common task when testing xarray user code is checking that your function works for all valid input dimensions. We can chain strategies to achieve this, for which the helper strategy unique_subset_of() is useful.

It works for lists of dimension names

dims = ["x", "y", "z"]
xrst.unique_subset_of(dims).example()

['z', 'x']

xrst.unique_subset_of(dims).example()

['y', 'z']

as well as for mappings of dimension names to sizes

dim_sizes = {"x": 2, "y": 3, "z": 4}
xrst.unique_subset_of(dim_sizes).example()

{'y': 3, 'x': 2}

xrst.unique_subset_of(dim_sizes).example()

{'z': 4, 'x': 2}

This is useful because operations like reductions can be performed over any subset of the xarray object’s dimensions. For example we can write a pytest test that tests that a reduction gives the expected result when applying that reduction along any possible valid subset of the Variable’s dimensions.

importnumpy.testingasnpt
@given(st.data(), xrst.variables(dims=xrst.dimension_names(min_dims=1)))
deftest_mean(data, var):
"""Test that the mean of an xarray Variable is always equal to the mean of the underlying array."""
 # specify arbitrary reduction along at least one dimension
 reduction_dims = data.draw(xrst.unique_subset_of(var.dims, min_size=1))
 # create expected result (using nanmean because arrays with Nans will be generated)
 reduction_axes = tuple(var.get_axis_num(dim) for dim in reduction_dims)
 expected = np.nanmean(var.data, axis=reduction_axes)
 # assert property is always satisfied
 result = var.mean(dim=reduction_dims).data
 npt.assert_equal(expected, result)

On this page

Edit on GitHub