SlideShare a Scribd company logo

UIImageView vs Metal [日本語版] #tryswiftconf

•
•
Shuichi Tsutsumi
Shuichi Tsutsumi

try! Swift Tokyo 2018 での発表資料です。Metalの使い方の話「ではなく」、Metalを通じて**普段意識する機会の少ないGPUレイヤに目を向けてみる**という内容となっておりますので、Metal使わないし興味ないという方々もぜひ。 英語版は https://www.slideshare.net/t26v0748/uiimageview-vs-metal-89418399/ にあります。 補足記事を書きました： http://shu223.hatenablog.com/entry/2018/03/05/124639 【概要】 MetalはGPUへのアクセスを提供するAPIで、OpenGLより10倍速いという謳い文句で登場しました。本セッションではMetalの基礎を解説しつつ、iOSにおけるグラフィックス描画性能をUIImageViewと比較してみます。 MetalのAPIを直接利用する機会がなくても、Metalはあなたのアプリの裏で暗躍しています。身近なクラスとの比較を通じて、普段我々が意識することのないGPUのレイヤで何が起きているのか、目を向けてみるきっかけになればと思います。

1 of 74
Download to read offline
try! Swift Tokyo 2018 Shuichi Tsutsumi @shu223 UIImageView vs Metal [ ]
Shuichi Tsutsumi @shu223 • iOS Developer - @Fyusion Inc. - @Freelance
• Metal • GPU
Agenda • UIImageView vs Metal → GPU 1. UIKit is optimized well with GPU. 2. Consider also the GPU, when measuring the performance. 3. Pay attention to the processing flow between CPU and GPU. 4. Be careful where the resource is.
imageView.image = image
UIImageView vs Metal [日本語版] #tryswiftconf
UIImageView vs Metal [日本語版] #tryswiftconf

 1 1 60 

CPU GPU CPU • • GPU • • CPU
• CPU - 100% • GPU GPU

 1 60 • CPU • OK GPU 1
What’s ?
GPU GPU Your app ???
OpenGL
OpenGL • API • • Apple
Metal • Apple • Apple • OpenGL 10
Sounds great!
Metal
imageView.image = image
UIImageView vs Metal [日本語版] #tryswiftconf
Metal
func draw(in view: MTKView) { guard let drawable = view.currentDrawable else {return} guard let commandBuffer = commandQueue.makeCommandBuffer() else {fatalError()} guard let blitEncoder = commandBuffer.makeBlitCommandEncoder() else {fatalError()} let w = min(texture.width, drawable.texture.width) let h = min(texture.height, drawable.texture.height) blitEncoder.copy(from: texture, sourceSlice: 0, sourceLevel: 0, sourceOrigin: MTLOrigin(x: 0, y: 0, z: 0), sourceSize: MTLSizeMake(w, h, texture.depth), to: drawable.texture, destinationSlice: 0, destinationLevel: 0, destinationOrigin: MTLOrigin(x: 0, y: 0, z: 0)) blitEncoder.endEncoding() commandBuffer.present(drawable) commandBuffer.commit() commandBuffer.waitUntilCompleted() } private let device = MTLCreateSystemDefaultDevice()! 
 private func setup() { commandQueue = device.makeCommandQueue() let textureLoader = MTKTextureLoader(device: device) texture = try! textureLoader.newTexture( name: "highsierra", scaleFactor: view.contentScaleFactor, bundle: nil) mtkView.device = device mtkView.delegate = self mtkView.colorPixelFormat = texture.pixelFormat }
UIImageView vs Metal [日本語版] #tryswiftconf
private let device = MTLCreateSystemDefaultDevice()! 
 private func setup() { commandQueue = device.makeCommandQueue() let textureLoader = MTKTextureLoader(device: device) texture = try! textureLoader.newTexture( name: "highsierra", scaleFactor: view.contentScaleFactor, bundle: nil) mtkView.device = device mtkView.delegate = self mtkView.colorPixelFormat = texture.pixelFormat } func draw(in view: MTKView) { guard let drawable = view.currentDrawable else {return} guard let commandBuffer = commandQueue.makeCommandBuffer() else {fatalE guard let blitEncoder = commandBuffer.makeBlitCommandEncoder() else {fa let w = min(texture.width, drawable.texture.width) let h = min(texture.height, drawable.texture.height) blitEncoder.copy(from: texture, sourceSlice: 0, sourceLevel: 0, sourceOrigin: MTLOrigin(x: 0, y: 0, z: 0), sourceSize: MTLSizeMake(w, h, texture.depth), to: drawable.texture, destinationSlice: 0, destinationLevel: 0, destinationOrigin: MTLOrigin(x: 0, y: 0, z: 0)) blitEncoder.endEncoding() commandBuffer.present(drawable) commandBuffer.commit() commandBuffer.waitUntilCompleted() } imageView.image = image
private let device = MTLCreateSystemDefaultDevice()! 
 private func setup() { commandQueue = device.makeCommandQueue() let textureLoader = MTKTextureLoader(device: device) texture = try! textureLoader.newTexture( name: "highsierra", scaleFactor: view.contentScaleFactor, bundle: nil) mtkView.device = device mtkView.delegate = self mtkView.colorPixelFormat = texture.pixelFormat } func draw(in view: MTKView) { guard let drawable = view.currentDrawable else {return} guard let commandBuffer = commandQueue.makeCommandBuffer() else {fatalE guard let blitEncoder = commandBuffer.makeBlitCommandEncoder() else {fa let w = min(texture.width, drawable.texture.width) let h = min(texture.height, drawable.texture.height) blitEncoder.copy(from: texture, sourceSlice: 0, sourceLevel: 0, sourceOrigin: MTLOrigin(x: 0, y: 0, z: 0), sourceSize: MTLSizeMake(w, h, texture.depth), to: drawable.texture, destinationSlice: 0, destinationLevel: 0, destinationOrigin: MTLOrigin(x: 0, y: 0, z: 0)) blitEncoder.endEncoding() commandBuffer.present(drawable) commandBuffer.commit() commandBuffer.waitUntilCompleted() } imageView.image = image 💡
My Idea: Metal ✓ Easy to use as UIImageView ✓ Metal Accelerated “MetalImageView” metalImageView.texture = texture
Powered by
UIImageView vs Metal
• - 5120 x 3200 (elcapitan.jpg) - 1245 x 1245 (sierra.png)
Measuring Code let time1 = CACurrentMediaTime() if isMetal { let metalCell = cell as! MetalTableViewCell metalCell.metalImageView.textureName = name } else { let uikitCell = cell as! TableViewCell uikitCell.uiImageView.image = UIImage(named: name) } let time2 = CACurrentMediaTime() print("time:(time2-time1)") Time Interval Render with UIImageView Render with Metal
Results • Metal is 10x - 20x faster! Time to render an image UIImageView 0.4 - 0.6 msec Metal 0.02 - 0.05 msec iPhone 6s
• Metal UIImageView Metal
Measuring Code let time1 = CACurrentMediaTime() if isMetal { let metalCell = cell as! MetalTableViewCell metalCell.metalImageView.textureName = name } else { let uikitCell = cell as! TableViewCell uikitCell.uiImageView.image = UIImage(named: name) } let time2 = CACurrentMediaTime() print("time:(time2-time1)")
CPU GPU
CPU GPU 
 CPU/GPU GPU GPU
let time1 = CACurrentMediaTime() if isMetal { let metalCell = cell as! MetalTableViewCell metalCell.metalImageView.textureName = name } else { let uikitCell = cell as! TableViewCell uikitCell.uiImageView.image = UIImage(named: name) } let time2 = CACurrentMediaTime() print("time:(time2-time1)")
CPU GPU 
 CPU/GPU GPU GPU
• GPU func draw(in view: MTKView) { // Prepare the command buffer ... // Push the command buffer commandBuffer.commit() // Wait commandBuffer.waitUntilCompleted() // Measure let endTime = CACurrentMediaTime() print(“Time: (endTime - startTime)”) } GPU GPU
Results • Metal !? - 30 fps → • UIImageView Time to render an image UIImageView 0.4 - 0.6 msec Metal 40 - 200 msec
UIImageView
※WWDC17 Platforms State of the Union UIKit Metal
• UIKit •
Point 1: UIKit is optimized well with GPU
Point 2: Consider also the GPU, 
 when measuring the performance
MetalImageView Metal
Profile using Instruments Metal System Trace
UIImageView vs Metal [日本語版] #tryswiftconf
On CPU On GPU Create command buffers etc. on CPU Submit command buffers etc. on CPU Process shaders on GPU
On CPU On GPU
Problem 1
Resize (MPSImageLanczosScale) Render (MTLBlitCommandEncoder) Unexpected interval Measuring Time
• MPSImageLanczosScale • setNeedsDisplay() • MTKViewDelegate draw(in:) • draw(in:) Problem
GPU CPU On CPU On GPU
UIImageView vs Metal [日本語版] #tryswiftconf
Metal 4: GPU
Resize Render 2. CPU creates GPU commands 
 as a command buffer 4. GPU processes the commands 3. Push it to GPU
GPU • • GPU
2. CPU creates GPU commands 
 as a command buffer 4. GPU processes the commands 3. Push it to GPU Resize Render
Resize Render
Unexpected interval Combine Resize Render Resize+Render
Point 3: Pay attention to the processing flow between CPU and GPU
Problem 2
UIImageView vs Metal [日本語版] #tryswiftconf
CPU/GPU
• 20 - 500 msec • → let startTime = CACurrentMediaTime() textureLoader.newTexture(name: name, scaleFactor: scaleFactor, bundle: nil) { (texture, error) in let endTime = CACurrentMediaTime() print("Time to load (name): (endTime - startTime)") // Do something ... }
• UIImage(named:) • Metal/GPU
Metal/GPU : private var cachedTextures: [String: MTLTexture] = [:]OK private var cachedImages: [String: UIImage] = [:]NG
After adopting Cache
Point 4: Be careful where the resource is.
UIImageView vs Metal [日本語版] #tryswiftconf
UIImageView vs Metal [日本語版] #tryswiftconf
• Metal • GPU
• UIImageView vs Metal → GPU 1. UIKit is optimized well with GPU. 2. Consider also the GPU, when measuring the performance. 3. Pay attention to the processing flow between CPU and GPU. 4. Be careful where the resource is.
Thank you! https://github.com/shu223

More Related Content

UIImageView vs Metal [日本語版] #tryswiftconf

  • 1. try! Swift Tokyo 2018 Shuichi Tsutsumi @shu223 UIImageView vs Metal [ ]
  • 2. Shuichi Tsutsumi @shu223 • iOS Developer - @Fyusion Inc. - @Freelance
  • 4. Agenda • UIImageView vs Metal → GPU 1. UIKit is optimized well with GPU. 2. Consider also the GPU, when measuring the performance. 3. Pay attention to the processing flow between CPU and GPU. 4. Be careful where the resource is.
  • 18. Metal
  • 21. Metal
  • 22. func draw(in view: MTKView) { guard let drawable = view.currentDrawable else {return} guard let commandBuffer = commandQueue.makeCommandBuffer() else {fatalError()} guard let blitEncoder = commandBuffer.makeBlitCommandEncoder() else {fatalError()} let w = min(texture.width, drawable.texture.width) let h = min(texture.height, drawable.texture.height) blitEncoder.copy(from: texture, sourceSlice: 0, sourceLevel: 0, sourceOrigin: MTLOrigin(x: 0, y: 0, z: 0), sourceSize: MTLSizeMake(w, h, texture.depth), to: drawable.texture, destinationSlice: 0, destinationLevel: 0, destinationOrigin: MTLOrigin(x: 0, y: 0, z: 0)) blitEncoder.endEncoding() commandBuffer.present(drawable) commandBuffer.commit() commandBuffer.waitUntilCompleted() } private let device = MTLCreateSystemDefaultDevice()! 
 private func setup() { commandQueue = device.makeCommandQueue() let textureLoader = MTKTextureLoader(device: device) texture = try! textureLoader.newTexture( name: "highsierra", scaleFactor: view.contentScaleFactor, bundle: nil) mtkView.device = device mtkView.delegate = self mtkView.colorPixelFormat = texture.pixelFormat }
  • 24. private let device = MTLCreateSystemDefaultDevice()! 
 private func setup() { commandQueue = device.makeCommandQueue() let textureLoader = MTKTextureLoader(device: device) texture = try! textureLoader.newTexture( name: "highsierra", scaleFactor: view.contentScaleFactor, bundle: nil) mtkView.device = device mtkView.delegate = self mtkView.colorPixelFormat = texture.pixelFormat } func draw(in view: MTKView) { guard let drawable = view.currentDrawable else {return} guard let commandBuffer = commandQueue.makeCommandBuffer() else {fatalE guard let blitEncoder = commandBuffer.makeBlitCommandEncoder() else {fa let w = min(texture.width, drawable.texture.width) let h = min(texture.height, drawable.texture.height) blitEncoder.copy(from: texture, sourceSlice: 0, sourceLevel: 0, sourceOrigin: MTLOrigin(x: 0, y: 0, z: 0), sourceSize: MTLSizeMake(w, h, texture.depth), to: drawable.texture, destinationSlice: 0, destinationLevel: 0, destinationOrigin: MTLOrigin(x: 0, y: 0, z: 0)) blitEncoder.endEncoding() commandBuffer.present(drawable) commandBuffer.commit() commandBuffer.waitUntilCompleted() } imageView.image = image
  • 25. private let device = MTLCreateSystemDefaultDevice()! 
 private func setup() { commandQueue = device.makeCommandQueue() let textureLoader = MTKTextureLoader(device: device) texture = try! textureLoader.newTexture( name: "highsierra", scaleFactor: view.contentScaleFactor, bundle: nil) mtkView.device = device mtkView.delegate = self mtkView.colorPixelFormat = texture.pixelFormat } func draw(in view: MTKView) { guard let drawable = view.currentDrawable else {return} guard let commandBuffer = commandQueue.makeCommandBuffer() else {fatalE guard let blitEncoder = commandBuffer.makeBlitCommandEncoder() else {fa let w = min(texture.width, drawable.texture.width) let h = min(texture.height, drawable.texture.height) blitEncoder.copy(from: texture, sourceSlice: 0, sourceLevel: 0, sourceOrigin: MTLOrigin(x: 0, y: 0, z: 0), sourceSize: MTLSizeMake(w, h, texture.depth), to: drawable.texture, destinationSlice: 0, destinationLevel: 0, destinationOrigin: MTLOrigin(x: 0, y: 0, z: 0)) blitEncoder.endEncoding() commandBuffer.present(drawable) commandBuffer.commit() commandBuffer.waitUntilCompleted() } imageView.image = image 💡
  • 26. My Idea: Metal ✓ Easy to use as UIImageView ✓ Metal Accelerated “MetalImageView” metalImageView.texture = texture
  • 29. • - 5120 x 3200 (elcapitan.jpg) - 1245 x 1245 (sierra.png)
  • 30. Measuring Code let time1 = CACurrentMediaTime() if isMetal { let metalCell = cell as! MetalTableViewCell metalCell.metalImageView.textureName = name } else { let uikitCell = cell as! TableViewCell uikitCell.uiImageView.image = UIImage(named: name) } let time2 = CACurrentMediaTime() print("time:(time2-time1)") Time Interval Render with UIImageView Render with Metal
  • 31. Results • Metal is 10x - 20x faster! Time to render an image UIImageView 0.4 - 0.6 msec Metal 0.02 - 0.05 msec iPhone 6s
  • 33. Measuring Code let time1 = CACurrentMediaTime() if isMetal { let metalCell = cell as! MetalTableViewCell metalCell.metalImageView.textureName = name } else { let uikitCell = cell as! TableViewCell uikitCell.uiImageView.image = UIImage(named: name) } let time2 = CACurrentMediaTime() print("time:(time2-time1)")
  • 36. let time1 = CACurrentMediaTime() if isMetal { let metalCell = cell as! MetalTableViewCell metalCell.metalImageView.textureName = name } else { let uikitCell = cell as! TableViewCell uikitCell.uiImageView.image = UIImage(named: name) } let time2 = CACurrentMediaTime() print("time:(time2-time1)")
  • 38. • GPU func draw(in view: MTKView) { // Prepare the command buffer ... // Push the command buffer commandBuffer.commit() // Wait commandBuffer.waitUntilCompleted() // Measure let endTime = CACurrentMediaTime() print(“Time: (endTime - startTime)”) } GPU GPU
  • 39. Results • Metal !? - 30 fps → • UIImageView Time to render an image UIImageView 0.4 - 0.6 msec Metal 40 - 200 msec
  • 41. ※WWDC17 Platforms State of the Union UIKit Metal
  • 43. Point 1: UIKit is optimized well with GPU
  • 44. Point 2: Consider also the GPU, 
 when measuring the performance
  • 48. On CPU On GPU Create command buffers etc. on CPU Submit command buffers etc. on CPU Process shaders on GPU
  • 52. • MPSImageLanczosScale • setNeedsDisplay() • MTKViewDelegate draw(in:) • draw(in:) Problem
  • 56. Resize Render 2. CPU creates GPU commands 
 as a command buffer 4. GPU processes the commands 3. Push it to GPU
  • 58. 2. CPU creates GPU commands 
 as a command buffer 4. GPU processes the commands 3. Push it to GPU Resize Render
  • 61. Point 3: Pay attention to the processing flow between CPU and GPU
  • 65. • 20 - 500 msec • → let startTime = CACurrentMediaTime() textureLoader.newTexture(name: name, scaleFactor: scaleFactor, bundle: nil) { (texture, error) in let endTime = CACurrentMediaTime() print("Time to load (name): (endTime - startTime)") // Do something ... }
  • 67. Metal/GPU : private var cachedTextures: [String: MTLTexture] = [:]OK private var cachedImages: [String: UIImage] = [:]NG
  • 69. Point 4: Be careful where the resource is.
  • 73. • UIImageView vs Metal → GPU 1. UIKit is optimized well with GPU. 2. Consider also the GPU, when measuring the performance. 3. Pay attention to the processing flow between CPU and GPU. 4. Be careful where the resource is.