Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Parallelization problem for 3D tensor #600

Unanswered
ppanzx asked this question in Q&A
Discussion options

How can I leverage the code to compute the EMD btween a pair of 3D-tensor, and return a 4D-tensor as the EMD matrix?

You must be logged in to vote

Replies: 1 comment 1 reply

Comment options

Hello EMD is already optimized C++ and is not paralelizable directly but you can use jobli to comptue multiple emd in parallel. here is some code that was sent to me by the great @tomMoral

# %%
import numpy as np
import ot
from joblib import Parallel, delayed
# %%
n = 1000
k = 50
M = np.random.rand(k, n, n)
a = np.ones(n) / n
# %% loop
def emd(M, axes=None):
 return ot.emd(a, a, M)
R_loop = np.zeros((k, n, n))
ot.tic()
for i in range(k):
 R_loop[i] = emd(M[i])
ot.toc()
# %% numpy take/stack
def apply_across_axis(func, M, axis=0):
 return np.stack([
 func(M.take(i, axis))
 for i in range(M.shape[axis])
 ], axis=axis)
ot.tic()
R_numpy = apply_across_axis(emd, M, 0)
ot.toc()
# %% joblib?
def apply_across_axis_joblib(func, M, axis=0, n_jobs=4):
 res = Parallel(n_jobs=n_jobs, max_nbytes=None)(
 delayed(func)(M.take(i, axis))
 for i in range(M.shape[axis])
 )
 return np.stack(res, axis=axis)
R_joblib = apply_across_axis_joblib(emd, M[:4], 0)
ot.tic()
R_joblib = apply_across_axis_joblib(emd, M, 0)
ot.toc()

I think it can be easily adapted to your problem (I dont know what is going to happen with torch tensors though)

You must be logged in to vote
1 reply
Comment options

Are there any torch-like codes?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
Q&A
Labels
2 participants
Converted from issue

This discussion was converted from issue #598 on January 15, 2024 12:44.

AltStyle によって変換されたページ (->オリジナル) /