Adding to array in parallel

Question

I'm using Grand Central Dispatch to transforms elements of one array into another. I call dispatch_apply on the source array, transform it into zero or more items, then add them to the destination array. This is a simplified example:

let src = Array(0..<1000)
var dst = [UInt32]()
let queue = dispatch_queue_create("myqueue", DISPATCH_QUEUE_CONCURRENT)
dispatch_apply(src.count, queue) { i in
 dst.append(arc4random_uniform(UInt32(i))) // <-- potential error here
}
print(dst)

I sometimes get an error on the append line. The error is always one of:

1. malloc: *** error for object 0x107508f00: pointer being freed was not allocated
2. fatal error: UnsafeMutablePointer.destroy with negative count
3. fatal error: Can't form Range with end < start

I guess this is due to append not being thread-safe. What did I do wrong and how to fix it?

• Martin R: Compare stackoverflow.com/questions/26693838/….
• Jenny: Thanks @MartinR. That question alter the existing value of the array rather than append to it. withUnsafeMutablePointer does not have an append method
• Martin R: As you already noticed, append() is not thread-safe. You probably have to create the array with the necessary size first. Then you can use the methods from the referenced Q&A to fill the array in parallel from multiple threads.
• user3441734: sometimes? man, you are lucky!. anyway, as you wrote, swift types are not thread-safe. i thing, that this is just very simplified example of something what you are trying to do. try to avoid this approach. even though you will have a thread-safe version of an array, this approach will be real performance killer. if you need just separate a long running 'transformation' from your main loop, call the whole 'transformation' asynchronously, but as one or few separate blocks

Answer 1

As you have discovered, Swift may relocate the array for memory efficiency. When you run multiple append, it's bound to happen. My guess is that appending to the array is a very low cost operation compared to your transformation. You can create a serial queue just to extend the dst array:

let src = Array(0..<1000)
var dst = [UInt32]()
let queue = dispatch_queue_create("myqueue", DISPATCH_QUEUE_CONCURRENT)
let serialQueue = dispatch_queue_create("mySerialQueue", DISPATCH_QUEUE_SERIAL)
let serialGroup = dispatch_group_create()
dispatch_apply(src.count, queue) { i in
 let result = arc4random_uniform(UInt32(i)) // Your long-running transformation here
 dispatch_group_async(serialGroup, serialQueue) {
 dst.append(result)
 }
}
// Wait until all append operations are complete
dispatch_group_wait(serialGroup, DISPATCH_TIME_FOREVER)

• Jenny: Why did you use dispatch_group_async instead of group_async?
• Code Different: I think you meant dispatch_group_async vs. dispatch_async. The reason is that you can wait on the first, but not the second function.
• A'sa Dickens: I have this same issue, but i didn't think it would be something so complex XD thanks for the solution !

Answer 2

In case anyone is interested in Swift 4 (I suppose Swift 3 onwards) solution (slightly different approach):

let source = Array(0..<1000)
// If inside class/struct, make this lazy or execute the block inside init
var destination: [UInt32] = {
 var copy = Array<UInt32>.init(repeating: 0, count: source.count)
 DispatchQueue.concurrentPerform(iterations: source.count) { i in
 // Do stuff to get the i'th element 
 copy[i] = UInt32.random(in: 0..<i) 
 }
 return copy
}()

If you choose to avoid the tie between your array size and the number of concurrent operations:

(e.g. you may choose to do it in two iterations 0..<500 in parallel with 500..<1000)

let iterations = 2 
DispatchQueue.concurrentPerform(iterations: iterations) { index in
 // First iteration would start at 0 whereas second (index = 1) starts at 500
 let start = index * source.count / iterations
 // First and second iterations will have 500 and 1000 upper bounds respectively 
 let end = (index + 1) * source.count / iterations
 for i in start..<end {
 // Do stuff to get the i'th element
 copy[i] = UInt32.random(in: 0..<source.count)
 }
}