import numpy as np
# creates an expression (nothing is computed)
r = np.sqrt(ds.x**2 + ds.y**2 + ds.z**2)
r # for convenience, we print out some values

<vaex.expression.Expression(expressions='sqrt((((x ** 2) + (y ** 2)) + (z ** 2)))')> instance at 0x11bcc4780 values=[2.9655450396553587, 5.77829281049018, 6.99079603950256, 9.431842752707537, 0.8825613121347967 ... (total 330000 values) ... 7.453831761514681, 15.398412491068198, 8.864250273925633, 17.601047186042507, 14.540181524970293]

These expressions can be added to the dataset, creating what we call a virtual column. These virtual columns are simular to normal columns, except they do not waste memory.

ds['r'] = r # add a (virtual) column that will be computed on the fly
ds.mean(ds.x), ds.mean(ds.r) # calculate statistics on normal and virtual columns

(-0.06713149126400597, 9.407082338299773)

One of the core features of vaex is its ability to calculate statistics on a regular (N-dimensional) grid. The dimensions of the grid are specified by the binby argument (analogous to SQL's grouby), and the shape and limits.

ds.mean(ds.r, binby=ds.x, shape=32, limits=[-10, 10]) # create statistics on a regular grid (1d)

array([15.01058183, 14.43693006, 13.72923338, 12.90294499, 11.86615103,
 11.03563695, 10.12162553, 9.2969267 , 8.58250973, 7.86602644,
 7.19568442, 6.55738773, 6.01942499, 5.51462457, 5.15798991,
 4.8274218 , 4.7346551 , 5.1343761 , 5.46017944, 6.02199777,
 6.54132124, 7.27025256, 7.99780777, 8.55188217, 9.30286584,
 9.97067561, 10.81633293, 11.60615795, 12.33813552, 13.10488982,
 13.86868565, 14.60577266])

ds.mean(ds.r, binby=[ds.x, ds.y], shape=32, limits=[-10, 10]) # or 2d
ds.count(ds.r, binby=[ds.x, ds.y], shape=32, limits=[-10, 10]) # or 2d counts/histogram

array([[22., 33., 37., ..., 58., 38., 45.],
 [37., 36., 47., ..., 52., 36., 53.],
 [34., 42., 47., ..., 59., 44., 56.],
 ...,
 [73., 73., 84., ..., 41., 40., 37.],
 [53., 58., 63., ..., 34., 35., 28.],
 [51., 32., 46., ..., 47., 33., 36.]])