Parallelize array creation

Question

F#

[|for index in 1 .. items.Count()-1 -> (* create object here - complex operations *)|]

C#

Object[] newItemArray= new Object[items.Count];
Parallel.For(0, items.Count, index=>
{
 /*complex stuff here*/
 newItemArray[index] = new Object();
});

I have the above C# and F# doing the same thing. Without the Parallel.For the F# is slightly faster. With the Parallel.For the C# takes half as long to execute. How do I properly parallelize the F# to get the same performance boost as the C#?

The way I've tried so far was Array.Parallel.Iteri, so I could use the same index into an array trick I used in C#, but it slowed it down instead of speeding it up.

Edit:

More details on what I'm doing:

I have an enumerable of byte array array array. I have another byte array array array to which I'm comparing the others. I'm sorting the enumerable on % similarity and returning the first 500.

In both the F# and the C# I'm doing a simple nested for loop which increments a counter. When done looping over a particular item in my enumerable, I create a tuple of (item, counter). Once done creating my new enumerable of (item, counter), I sort it on the counter variable, grab the first 500 and then convert back to just an enumerable of item.

The part which I'm putting inside the Parallel.For is the creation of the IEnumerable<Tuple<item, int>>

• N_A: I guess it's time to start comparing IL...
• pad: I know it's not easy but can you give a reproducible example? You Edit doesn't help that much.

Answer 1

Array.Parallel.init items.Count (fun index ->
 (* create object here - complex operations *))

Official documentation: Parallel.init<'T> Function (F#)

• ildjarn: @pad : Not necessarily faster, just lacking in details. :-P
• N_A: For some reason this takes 3x as long as my sequential solution and 7 times as long as my parallel C# solution.
• ildjarn: @mydogisbox : You're profiling a release build, not debug, right?
• N_A: Its release and "Start without debugging" with the "DEBUG" constant not defined.
• N_A: To be clear, I'm using a Stopwatch to time the runs and then just printing the elapsed time to the console.

Answer 2

You shouldn't use array comprehension in this case. It's a bit slower than high-order functions and can't be parallelized.

Though I would prefer @ildjarn's solution, here is an equivalent solution to your C# one:

// You need an uninitialized array to fill in later
let newItemArray = Array.zeroCreate items.Count
// Fill in the array in a parallel manner
Array.Parallel.iteri (fun i v ->
 (* create object here - complex operations *)) newItemArray

You can also use Parallel.For directly:

let newItemArray = Array.zeroCreate items.Count 
Parallel.For(0, items.Count, 
 (fun index ->
 (* complex stuff here *)
 newItemArray.[index] <- Object())
) |> ignore

It's more verbose but gives you more control over degree of parallelism.

• N_A: iteri is the first method I tried and is only slightly slower than my sequential solution. The Parallel.For solution takes about 3.5 times longer than my sequential solution.
• pad: Can you quickly compare parallel vs sequential solutions in F# Interactive with #time "on";;? It's hard to guess without more details.
• N_A: I have a stopwatch around them. Times are: 1.025 for the Parallel.For solution and .436 for the iteri solution. My sequential F# solution runs at about .360 every time.
• pad: We need more details on the F# and C# functions. It's weird that F# versions don't scale when C# versions do.
• N_A: I figured it out. I was using some ref types to get around some parallelism issues and it turned out that was killing my performance. Removed some of that and then used Array.Parallel.mapi and it works like a charm.