GroupBy: Group and Bin Data#

Often we want to bin or group data, produce statistics (mean, variance) on the groups, and then return a reduced data set. To do this, Xarray supports "group by" operations with the same API as pandas to implement the split-apply-combine strategy:

Split your data into multiple independent groups.
Apply some function to each group.
Combine your groups back into a single data object.

Group by operations work on both Dataset and DataArray objects. Most of the examples focus on grouping by a single one-dimensional variable, although support for grouping over a multi-dimensional variable has recently been implemented. Note that for one-dimensional data, it is usually faster to rely on pandas’ implementation of the same pipeline.

Tip

Install the flox package to substantially improve the performance of GroupBy operations, particularly with dask. flox extends Xarray’s in-built GroupBy capabilities by allowing grouping by multiple variables, and lazy grouping by dask arrays. If installed, Xarray will automatically use flox by default.

Binning#

Sometimes you don’t want to use all the unique values to determine the groups but instead want to "bin" the data into coarser groups. You could always create a customized coordinate, but xarray facilitates this via the Dataset.groupby_bins() method.

x_bins = [0, 25, 50]
ds.groupby_bins("x", x_bins).groups

{Interval(0, 25, closed='right'): [0, 1],
 Interval(25, 50, closed='right'): [2, 3]}

The binning is implemented via pandas.cut(), whose documentation details how the bins are assigned. As seen in the example above, by default, the bins are labeled with strings using set notation to precisely identify the bin limits. To override this behavior, you can specify the bin labels explicitly. Here we choose float labels which identify the bin centers:

x_bin_labels = [12.5, 37.5]
ds.groupby_bins("x", x_bins, labels=x_bin_labels).groups

{np.float64(12.5): [0, 1], np.float64(37.5): [2, 3]}

First and last#

There are two special aggregation operations that are currently only found on groupby objects: first and last. These provide the first or last example of values for group along the grouped dimension:

ds.groupby("letters").first(...)

<xarray.Dataset> Size: 64B
Dimensions: (letters: 2, y: 3)
Coordinates:
 * letters (letters) object 16B 'a' 'b'
Dimensions without coordinates: y
Data variables:
 foo (letters, y) float64 48B 0.127 0.9667 0.2605 0.8972 0.3767 0.3362

xarray.Dataset

Dimensions:
- letters: 2
- y: 3
Coordinates: (1)
- letters
  (letters)
  object
  'a' 'b'
```
array(['a', 'b'], dtype=object)
```

Data variables: (1)

foo

(letters, y)

float64

0.127 0.9667 ... 0.3767 0.3362

array([[0.12696983, 0.96671784, 0.26047601],
 [0.89723652, 0.37674972, 0.33622174]])

By default, they skip missing values (control this with skipna).

Grouper Objects#

Both groupby_bins and resample are specializations of the core groupby operation for binning, and time resampling. Many problems demand more complex GroupBy application: for example, grouping by multiple variables with a combination of categorical grouping, binning, and resampling; or more specializations like spatial resampling; or more complex time grouping like special handling of seasons, or the ability to specify custom seasons. To handle these use-cases and more, Xarray is evolving to providing an extension point using Grouper objects.

Tip

See the grouper design doc for more detail on the motivation and design ideas behind Grouper objects.

For now Xarray provides three specialized Grouper objects:

groupers.UniqueGrouper for categorical grouping
groupers.BinGrouper for binned grouping
groupers.TimeResampler for resampling along a datetime coordinate

These provide functionality identical to the existing groupby, groupby_bins, and resample methods. That is,

ds.groupby("x")

is identical to

fromxarray.groupersimport UniqueGrouper
ds.groupby(x=UniqueGrouper())

Similarly,

ds.groupby_bins("x", bins=bins)

is identical to

fromxarray.groupersimport BinGrouper
ds.groupby(x=BinGrouper(bins))

and

ds.resample(time="ME")

is identical to

fromxarray.groupersimport TimeResampler
ds.resample(time=TimeResampler("ME"))

The groupers.UniqueGrouper accepts an optional labels kwarg that is not present in DataArray.groupby() or Dataset.groupby(). Specifying labels is required when grouping by a lazy array type (e.g. dask or cubed). The labels are used to construct the output coordinate (say for a reduction), and aggregations will only be run over the specified labels. You may use labels to also specify the ordering of groups to be used during iteration. The order will be preserved in the output.

Time Grouping and Resampling#

Shuffling#

Shuffling is a generalization of sorting a DataArray or Dataset by another DataArray, named label for example, that follows from the idea of grouping by label. Shuffling reorders the DataArray or the DataArrays in a Dataset such that all members of a group occur sequentially. For example, Shuffle the object using either DatasetGroupBy or DataArrayGroupBy as appropriate.

da = xr.DataArray(
 dims="x",
 data=[1, 2, 3, 4, 5, 6],
 coords={"label": ("x", "a b c a b c".split(" "))},
)
da.groupby("label").shuffle_to_chunks()

<xarray.DataArray (x: 6)> Size: 48B
array([1, 4, 2, 5, 3, 6])
Coordinates:
 label (x) <U1 24B 'a' 'a' 'b' 'b' 'c' 'c'
Dimensions without coordinates: x

xarray.DataArray

x: 6

1 4 2 5 3 6
```
array([1, 4, 2, 5, 3, 6])
```

Coordinates: (1)

label

(x)

<U1

'a' 'a' 'b' 'b' 'c' 'c'

array(['a', 'a', 'b', 'b', 'c', 'c'], dtype='<U1')

For chunked array types (e.g. dask or cubed), shuffle may result in a more optimized communication pattern when compared to direct indexing by the appropriate indexer. Shuffling also makes GroupBy operations on chunked arrays an embarrassingly parallel problem, and may significantly improve workloads that use DatasetGroupBy.map() or DataArrayGroupBy.map().

GroupBy: Group and Bin Data#

Split#

Binning#

Apply#

First and last#

Grouped arithmetic#

Multidimensional Grouping#

Grouper Objects#

Grouping by multiple variables#

Time Grouping and Resampling#

Shuffling#