AFAIK there's not much you can do to further improve performances CPU side, but as suggested in the comments this seems like a typical application where GPU processing could give you significant performace improvement. However doing image processing with OpenGL or Metal is in general not an easy task, specially if you're completely new to it.
Fortunately starting from iOS8 things got a little bit easier since we can leverage the power of GPU to create custom filters, in a fairly straight forward fashion, using the Core Image framework. Anecdotally the documentation that states that custom filters do not work on iOS does not reflect iOS 8 release notes which tell us the opposite.
To get you started take a moment to get through Apple's Core Image custom filter paragraph, but oversimplifying you can think of these custom filters as custom calculations that are performed on every single pixel of the image at the same time.
Each Core Image Filter consists of 2 files:
- Filter implementation: a
CIFiltersubclass - Kernel: the aformentioned calculation, written in a variant of the OpenGL Shading Language (docs, wiki) which is a language based on C used to program the pipeline of the GPU.
In our case:
OneColorFocusCoreImageFilter.swift
class OneColorFocusCoreImageFilter: CIFilter {
private static var kernel: CIColorKernel?
private static var context: CIContext?
private var _inputImage: CIImage?
private var inputImage: CIImage? {
get { return _inputImage }
set { _inputImage = newValue }
}
private var focusColor: CIColor?
init(image: UIImage, focusColorRed: Int, focusColorGreen: Int, focusColorBlue: Int) {
super.init()
OneColorFocusCoreImageFilter.preload()
inputImage = CIImage(image: image)
focusColor = CIColor(red: CGFloat(focusColorRed) / 255.0, green: CGFloat(focusColorGreen) / 255.0, blue: CGFloat(focusColorBlue) / 255.0)
}
required init?(coder aDecoder: NSCoder) {
super.init(coder: aDecoder)
OneColorFocusCoreImageFilter.preload()
}
override var outputImage : CIImage! {
if let inputImage = inputImage,
let kernel = OneColorFocusCoreImageFilter.kernel,
let fc = focusColor {
return kernel.applyWithExtent(inputImage.extent, roiCallback: { (_, _) -> CGRect in return inputImage.extent }, arguments: [inputImage, fc]) // to support iOS8
// return kernel.applyWithExtent(inputImage.extent, arguments: [inputImage, fc]) // iOS9 and newer
}
return nil
}
func outputUIImage() -> UIImage {
let ciimage = self.outputImage
return UIImage(CGImage: OneColorFocusCoreImageFilter.context!.createCGImage(ciimage, fromRect: ciimage.extent))
}
private class func createKernel() -> CIColorKernel {
let kernelString = try! String(contentsOfFile: NSBundle.mainBundle().pathForResource("OneColorFocusCoreImageFilter", ofType: "cikernel")!, encoding: NSUTF8StringEncoding)
return CIColorKernel(string: kernelString)!
}
class func preload() {
// preloading kernel speeds up first execution of filter
if kernel != nil {
return
}
kernel = createKernel()
context = CIContext(options: [kCIContextWorkingColorSpace: NSNull()])
}
}
OneColorFocusCoreImageFilter.cikernel
kernel vec4 OneColorFocusCoreImageFilter(sampler source, __color focusColor)
{
vec4 pixel = sample(source, samplerCoord(source));
const float cLinearThreshold = 0.0031308;
const float powE = 1.0 / 2.4;
const float focusColorThreshold = 70.0 / 255.0;
vec4 diff = abs(pixel - focusColor);
bool pixelShouldBeInOriginalColor = (diff.r < focusColorThreshold && diff.g < focusColorThreshold && diff.b < focusColorThreshold);
float Y = dot(pixel.rgb, vec3(0.2126, 0.7152, 0.0722));
/*
if (Y <= cLinearThreshold) {
Y *= 12.92;
} else {
Y = 1.055 * pow(Y, powE) - 0.055;
}
Can be rewritten as follows to avoid branches
*/
bool belowThreshold = (Y <= cLinearThreshold);
Y = Y * 12.92 * float(belowThreshold) + (1.055 * pow(Y, powE) - 0.055) * float(!belowThreshold);
return pixel.rgba * float(pixelShouldBeInOriginalColor) + vec4(vec3(Y), 1.0) * float(!pixelShouldBeInOriginalColor);
}
Kernel optimization
I'm by no means an expert in GLSL but it is known that branches (if, loops, etc) have severe impacts on the kernel performaces. Therefore I included in the comment an example of how you can rewrite the branch.
Filter benchmarksts
On my iPhone 5S on a 1537 ×ばつ 667 pixels image I'm getting approximately a 5x speedup
- CPU ~ 120
- GPU ~ 25ms
On a 375 ×ばつ 500 pixels image we have a 3x speedup
- CPU ~ 40ms
- GPU ~ 15ms
Profiling
Profiling the GPU version of the filter shows that the filter by itself is very fast to be executed, while the true bottleneck is caused by the conversion to UIImage -> CIImage -> UIImage (we have a memory bandwidth constraint). This is most probably caused because we have to copy the Image buffer to a GPU texture an vice versa.
The numbers do also tell us that the Swift compiler is doing a great job optimizing the CPU version of the filter which has overall pretty decent performances (did you turn on the compilation optimization level?)
Further notes
Depending on your application there might be situations where you could get even better performances using the Core Image version of the filter. For example if you're getting samples from your device's camera there are some specifically methods to convert to CIImage (where the copy to GPU is optimized) like for example CIImage(CVImageBuffer:).
Sample code
You can find a working sample project https://github.com/tcamin/CustomCoreImageFilteringDemo
- 166
- 3