SlideShare a Scribd company logo

iOS Visual F/X Using GLSL

•
•
Douglass Turner
Douglass Turner

The document provides an overview of iOS visual effects capabilities and limitations. It discusses Cocoa Touch's reliance on CALayers for 3D capabilities despite UIViews living in 2D space. It introduces OpenGL and the power of shaders for advanced visual effects. Code examples are provided of various GLSL shaders for effects like color remapping, hue shifting, and edge detection. The document emphasizes the parallel processing capabilities of shaders for complex effects.

1 of 73
Download to read offline
iOS Visual F/X Douglass Turner Elastic Image Software email: douglass.turner@gmail.com tweets: @dugla cell: 781 775 3708
• Cocoa Touch Limitations • Visual f/x Idioms • Show and Tell
Cocoa Touch and its realization in Objective-C is about: Abstraction. Pattern. Messaging. Generality. Simplicity.
Containers: NSArray. NSDictionary. NSSet. Data: NSData. NSString. CoreData. Patterns: MVC. Target/Action. Delegation.
The Objective-C Runtime is Powerful: NSClassFromString | NSStringFromClass.
When it comes to visual expressive power the limitations of Cocoa Touch become apparent.
UIView lives in flatland ... // UIView struct CGRect { CGPoint origin; CGSize size; }; struct CGPoint { CGFloat x; CGFloat y; }; struct CGAffineTransform { CGFloat a, b, c, d; CGFloat tx, ty; }; @property(nonatomic) CGRect frame; @property(nonatomic) CGRect bounds; @property(nonatomic) CGPoint center; @property(nonatomic) CGAffineTransform transform; - (CGPoint)convertPoint:(CGPoint)point toView:(UIView *)view; - (CGPoint)convertPoint:(CGPoint)point fromView:(UIView *)view; - (CGRect)convertRect:(CGRect)rect toView:(UIView *)view; - (CGRect)convertRect:(CGRect)rect fromView:(UIView *)view;
... but relies on CALayer which lives in 3D. struct CATransform3D { CGFloat m11, m12, m13, m14; CGFloat m21, m22, m23, m24; CGFloat m31, m32, m33, m34; CGFloat m41, m42, m43, m44; }; // CALayer @property CGFloat zPosition; @property CGFloat anchorPointZ; @property CATransform3D transform;
iOS would prefer you ignore pixels and think about resolution independent points instead
The level of visual abstraction is the asset: Image. Video. Audio.
As long as you are willing to remain at this high level of abstraction you can get a lot done using UIViewAnimation, CIImage, etc.
But perhaps you are interested in going a bit deeper ...
Hello OpenGL You take the red pill – you stay in Wonderland and I show you how deep the rabbit-hole goes." -Morpheus
OpenGL Cocoa
OpenGL is old school
You are flipping levers on a large state machine.
OpenGL is a dragster. Not a Prius. Just buckle up ...
... and enjoy the ride, baby!
OpenGL websites are a bit different than the Cocoa websites/blogs you are familiar with ...
Isn’t that sweet ...
Um... WTF?
Can you feel the love?
Dude. I think my eyes are bleeding.
Our focus is GLSL the OpenGL Shading Language
The Shader Backstory • Pixar creates Reyes (Render Everything You Ever Saw) • Pixar creates RenderMan: C-like language for describing a 3D look at the sample (pixel) level. • Boom! Everything changes. • GLSL created in the spirit of RenderMan.
Shaders allow us to ignore the rest of a software system, freeing us to focus on achieving a specific look.
EISParticleSystem http://github.com/turner/EISParticleSystem
Texture Atlas Hacks http://github.com/turner/EISParticleSystem
t s Texture Space
t s Texture Space
Accelerometer Hacks http://github.com/turner/EISParticleSystem
Initialize model state touchesBegan:withEvent: Evolve model state touchesMoved:withEvent: Clean up model state touchesEnded:withEvent: touchesCancelled:withEvent:
Demo
GLSL. Think C-lite.
TextureMapShader varying! vec2 v_st; uniform sampler2D hero; void main() { ! ! gl_FragColor = (heroChannels == 1) ? vec4(texture2D(hero, v_st).a) : texture2D(hero, v_st); ! } varying! vec2 v_st; uniform sampler2D hero; void main() { ! ! gl_FragColor = (heroChannels == 1) ? vec4(texture2D(hero, v_st).a) : texture2D(hero, v_st); ! }
InvertColorShader vec2 v_st; uniform int heroChannels; uniform sampler2D hero; void main() { ! ! vec3 rgb; rgb = (heroChannels == 1) ? 1.0 - vec3(raw.a) : 1.0 - raw.rgb; ! gl_FragColor = vec4(rgb, raw.a);! }
iOS Visual F/X Using GLSL
iOS Visual F/X Using GLSL
LuminanceShader varying vec2 v_st; uniform int heroChannels; uniform sampler2D hero; void main() { ! vec4 raw = (heroChannels == 1) ? vec4(texture2D(hero, v_st).a) : texture2D(hero, v_st); ! vec3 rgb; ! if (heroChannels == 1) { ! ! ! rgb = vec3(texture2D(hero, v_st).a);! ! ! } else { ! ! ! vec3 luminance_weights = vec3(0.30, 0.59, 0.11);!! ! ! rgb = vec3( dot( luminance_weights, texture2D(hero, v_st).rgb ) ); ! } ! ! gl_FragColor = vec4(rgb, raw.a); }
iOS Visual F/X Using GLSL
iOS Visual F/X Using GLSL
MixShader varying! vec2 v_st; uniform vec3 overRGB;! uniform vec3 underRGB; uniform int heroChannels; uniform sampler2D hero; void main() { ! float mixer;! ! mixer = (heroChannels == 1) ? texture2D(hero, v_st).a : texture2D(hero, v_st).g; ! ! gl_FragColor = vec4(mix(underRGB, overRGB, mixer), texture2D(hero, v_st).a); ! } varying! vec2 v_st; uniform vec3 overRGB;! uniform vec3 underRGB; uniform int heroChannels; uniform sampler2D hero; void main() { ! float mixer;! ! mixer = (heroChannels == 1) ? texture2D(hero, v_st).a : texture2D(hero, v_st).g; ! ! gl_FragColor = vec4(mix(underRGB, overRGB, mixer), texture2D(hero, v_st).a); ! }
iOS Visual F/X Using GLSL
The GPU is a parallel processing beast. It uses a SIMD architecture (Single Instruction Multiple Data) to achieve massive processing power. A GLSL shader is a SIMD program. The GPU takes a shader and evaluates it simultaneously - but with different data - at every sample (pixel) in parallel: On iPad that is: 2048 x 1536 * 30 fps = 94,371,840 shader evaluations per sec.
What is exciting about this level of performance on an iOS device is the ability to take tasks previously thought of as desktop tasks done with Photoshop, Final Cut, etc. and do them live in a handheld device equipped with camera, mic, and other sensors.
Demo. GLSL Powered Apps • Beautiful Panoramas. App Store: http://bit.ly/9KJBLA • BMW Interior. Panoramic hotspots prototype. • RadPad. iPad radiology prototype.
Shader Idioms To fully “get” the power of shaders and their style of use it helps know the idioms guiding their use: • Multiple Passes • Iteration aka Ping/Pong • Buffers/Channels • Indirection/Remapping
Shader Idioms These idioms derive directly from Hollywood film production workflows and practices that enable: • complex problem decomposition. Tractability. • rapid turnaround. • maximum flexibility and tweek-ability. • responsiveness to director’s whims
Demo. ElasticImage.
ElasticImage highlights. • Rapid shader creation & deployment • Shaders and gesture declared in plist • Cocoa Touch for gestures & the usual. • C++ for 3D glue code.
iOS Visual F/X Using GLSL
Shader Example: ColourLovers
iOS Visual F/X Using GLSL
iOS Visual F/X Using GLSL
I selected interesting palettes and created a texture. Horizontal axis is the color palette. Vertical axis selects between color palettes. At runtime this palette texture is loaded and attached to the ColourLoverShader.
EISColourLoversShader.fsh varying! mediump vec2 v_st; // Palette selector uniform float paletteDial; // Contribution of remapped color uniform float strengthDial; // Palettes are ganged together into a single texture. // A specific palette is selected with the paletteDial uniform sampler2D colourLoversPalettes; // hero uniform int heroChannels; uniform sampler2D hero; void main() { ! ! vec4 raw = (heroChannels == 1) ? vec4(texture2D(hero, v_st).a) : texture2D(hero, v_st); ! ! vec3 cooked; ! cooked.r = texture2D(colourLoversPalettes, vec2(raw.r, paletteDial)).r; ! cooked.g = texture2D(colourLoversPalettes, vec2(raw.g, paletteDial)).g; ! cooked.b = texture2D(colourLoversPalettes, vec2(raw.b, paletteDial)).b; ! ! gl_FragColor = vec4(mix(raw.rgb, cooked, strengthDial), raw.a); }
EISColourLoversShader.fsh varying! mediump vec2 v_st; // Palette selector uniform float paletteDial; // Contribution of remapped color uniform float strengthDial; // Palettes are ganged together into a single texture. // A specific palette is selected with the paletteDial uniform sampler2D colourLoversPalettes; // hero uniform int heroChannels; uniform sampler2D hero; void main() { ! ! vec4 raw = (heroChannels == 1) ? vec4(texture2D(hero, v_st).a) : texture2D(hero, v_st); ! ! vec3 cooked; ! cooked.r = texture2D(colourLoversPalettes, vec2(raw.r, paletteDial)).r; ! cooked.g = texture2D(colourLoversPalettes, vec2(raw.g, paletteDial)).g; ! cooked.b = texture2D(colourLoversPalettes, vec2(raw.b, paletteDial)).b; ! ! gl_FragColor = vec4(mix(raw.rgb, cooked, strengthDial), raw.a); }
Demo ColourLovers Shader
Hue Shift Shader RGB HSB
Conceptually simple idea. Use one channel of a photo to select the hue of a resultant image.
Demo Hue Shift Shader
Quantize ST Shader
Sampling a texture at a low rate results in quantization. Beyond Photography - The Digital Darkroom by Gerard J. Holzmann
Sampling in polar coordinates rather then cartesian is a bit more interesting. Beyond Photography - The Digital Darkroom by Gerard J. Holzmann
Demo Quantize ST Shader
BurnShader
iOS Visual F/X Using GLSL
iOS Visual F/X Using GLSL
red channel = drop shadow green channel = cut out final rgb texture
Demo BurnShader
Links A previous meetup talk I gave on iOS OpenGL • iOS OpenGL - http://slidesha.re/Y1MW8 Github - code • HelloGLSL - http://bit.ly/JCcMju • EISRenderHelpful. Helpful 3D rendering glue code - http://bit.ly/JZ4HW3 Github - people to follow • Philip Rideout - https://github.com/prideout • Raphael Sebbe - https://github.com/rsebbe • Brad Larson - https://github.com/BradLarson • Jeff LaMarche - https://github.com/jlamarche Elsewhere: • Martins Upitis - http://devlog-martinsh.blogspot.com/ • Ole Begemann - http://bit.ly/srlCBV • Daniel Rakos - http://bit.ly/a3QATn
Thank You!
Douglass Turner Elastic Image Software email: douglass.turner@gmail.com tweets: @dugla cell: 781 775 3708 Copyright © Douglass Turner

More Related Content

iOS Visual F/X Using GLSL

  • 1. iOS Visual F/X Douglass Turner Elastic Image Software email: douglass.turner@gmail.com tweets: @dugla cell: 781 775 3708
  • 2. • Cocoa Touch Limitations • Visual f/x Idioms • Show and Tell
  • 3. Cocoa Touch and its realization in Objective-C is about: Abstraction. Pattern. Messaging. Generality. Simplicity.
  • 4. Containers: NSArray. NSDictionary. NSSet. Data: NSData. NSString. CoreData. Patterns: MVC. Target/Action. Delegation.
  • 5. The Objective-C Runtime is Powerful: NSClassFromString | NSStringFromClass.
  • 6. When it comes to visual expressive power the limitations of Cocoa Touch become apparent.
  • 7. UIView lives in flatland ... // UIView struct CGRect { CGPoint origin; CGSize size; }; struct CGPoint { CGFloat x; CGFloat y; }; struct CGAffineTransform { CGFloat a, b, c, d; CGFloat tx, ty; }; @property(nonatomic) CGRect frame; @property(nonatomic) CGRect bounds; @property(nonatomic) CGPoint center; @property(nonatomic) CGAffineTransform transform; - (CGPoint)convertPoint:(CGPoint)point toView:(UIView *)view; - (CGPoint)convertPoint:(CGPoint)point fromView:(UIView *)view; - (CGRect)convertRect:(CGRect)rect toView:(UIView *)view; - (CGRect)convertRect:(CGRect)rect fromView:(UIView *)view;
  • 8. ... but relies on CALayer which lives in 3D. struct CATransform3D { CGFloat m11, m12, m13, m14; CGFloat m21, m22, m23, m24; CGFloat m31, m32, m33, m34; CGFloat m41, m42, m43, m44; }; // CALayer @property CGFloat zPosition; @property CGFloat anchorPointZ; @property CATransform3D transform;
  • 9. iOS would prefer you ignore pixels and think about resolution independent points instead
  • 10. The level of visual abstraction is the asset: Image. Video. Audio.
  • 11. As long as you are willing to remain at this high level of abstraction you can get a lot done using UIViewAnimation, CIImage, etc.
  • 12. But perhaps you are interested in going a bit deeper ...
  • 13. Hello OpenGL You take the red pill – you stay in Wonderland and I show you how deep the rabbit-hole goes." -Morpheus
  • 14. OpenGL Cocoa
  • 15. OpenGL is old school
  • 16. You are flipping levers on a large state machine.
  • 17. OpenGL is a dragster. Not a Prius. Just buckle up ...
  • 18. ... and enjoy the ride, baby!
  • 19. OpenGL websites are a bit different than the Cocoa websites/blogs you are familiar with ...
  • 22. Can you feel the love?
  • 23. Dude. I think my eyes are bleeding.
  • 24. Our focus is GLSL the OpenGL Shading Language
  • 25. The Shader Backstory • Pixar creates Reyes (Render Everything You Ever Saw) • Pixar creates RenderMan: C-like language for describing a 3D look at the sample (pixel) level. • Boom! Everything changes. • GLSL created in the spirit of RenderMan.
  • 26. Shaders allow us to ignore the rest of a software system, freeing us to focus on achieving a specific look.
  • 29. t s Texture Space
  • 30. t s Texture Space
  • 32. Initialize model state touchesBegan:withEvent: Evolve model state touchesMoved:withEvent: Clean up model state touchesEnded:withEvent: touchesCancelled:withEvent:
  • 33. Demo
  • 35. TextureMapShader varying! vec2 v_st; uniform sampler2D hero; void main() { ! ! gl_FragColor = (heroChannels == 1) ? vec4(texture2D(hero, v_st).a) : texture2D(hero, v_st); ! } varying! vec2 v_st; uniform sampler2D hero; void main() { ! ! gl_FragColor = (heroChannels == 1) ? vec4(texture2D(hero, v_st).a) : texture2D(hero, v_st); ! }
  • 36. InvertColorShader vec2 v_st; uniform int heroChannels; uniform sampler2D hero; void main() { ! ! vec3 rgb; rgb = (heroChannels == 1) ? 1.0 - vec3(raw.a) : 1.0 - raw.rgb; ! gl_FragColor = vec4(rgb, raw.a);! }
  • 39. LuminanceShader varying vec2 v_st; uniform int heroChannels; uniform sampler2D hero; void main() { ! vec4 raw = (heroChannels == 1) ? vec4(texture2D(hero, v_st).a) : texture2D(hero, v_st); ! vec3 rgb; ! if (heroChannels == 1) { ! ! ! rgb = vec3(texture2D(hero, v_st).a);! ! ! } else { ! ! ! vec3 luminance_weights = vec3(0.30, 0.59, 0.11);!! ! ! rgb = vec3( dot( luminance_weights, texture2D(hero, v_st).rgb ) ); ! } ! ! gl_FragColor = vec4(rgb, raw.a); }
  • 42. MixShader varying! vec2 v_st; uniform vec3 overRGB;! uniform vec3 underRGB; uniform int heroChannels; uniform sampler2D hero; void main() { ! float mixer;! ! mixer = (heroChannels == 1) ? texture2D(hero, v_st).a : texture2D(hero, v_st).g; ! ! gl_FragColor = vec4(mix(underRGB, overRGB, mixer), texture2D(hero, v_st).a); ! } varying! vec2 v_st; uniform vec3 overRGB;! uniform vec3 underRGB; uniform int heroChannels; uniform sampler2D hero; void main() { ! float mixer;! ! mixer = (heroChannels == 1) ? texture2D(hero, v_st).a : texture2D(hero, v_st).g; ! ! gl_FragColor = vec4(mix(underRGB, overRGB, mixer), texture2D(hero, v_st).a); ! }
  • 44. The GPU is a parallel processing beast. It uses a SIMD architecture (Single Instruction Multiple Data) to achieve massive processing power. A GLSL shader is a SIMD program. The GPU takes a shader and evaluates it simultaneously - but with different data - at every sample (pixel) in parallel: On iPad that is: 2048 x 1536 * 30 fps = 94,371,840 shader evaluations per sec.
  • 45. What is exciting about this level of performance on an iOS device is the ability to take tasks previously thought of as desktop tasks done with Photoshop, Final Cut, etc. and do them live in a handheld device equipped with camera, mic, and other sensors.
  • 46. Demo. GLSL Powered Apps • Beautiful Panoramas. App Store: http://bit.ly/9KJBLA • BMW Interior. Panoramic hotspots prototype. • RadPad. iPad radiology prototype.
  • 47. Shader Idioms To fully “get” the power of shaders and their style of use it helps know the idioms guiding their use: • Multiple Passes • Iteration aka Ping/Pong • Buffers/Channels • Indirection/Remapping
  • 48. Shader Idioms These idioms derive directly from Hollywood film production workflows and practices that enable: • complex problem decomposition. Tractability. • rapid turnaround. • maximum flexibility and tweek-ability. • responsiveness to director’s whims
  • 50. ElasticImage highlights. • Rapid shader creation & deployment • Shaders and gesture declared in plist • Cocoa Touch for gestures & the usual. • C++ for 3D glue code.
  • 55. I selected interesting palettes and created a texture. Horizontal axis is the color palette. Vertical axis selects between color palettes. At runtime this palette texture is loaded and attached to the ColourLoverShader.
  • 56. EISColourLoversShader.fsh varying! mediump vec2 v_st; // Palette selector uniform float paletteDial; // Contribution of remapped color uniform float strengthDial; // Palettes are ganged together into a single texture. // A specific palette is selected with the paletteDial uniform sampler2D colourLoversPalettes; // hero uniform int heroChannels; uniform sampler2D hero; void main() { ! ! vec4 raw = (heroChannels == 1) ? vec4(texture2D(hero, v_st).a) : texture2D(hero, v_st); ! ! vec3 cooked; ! cooked.r = texture2D(colourLoversPalettes, vec2(raw.r, paletteDial)).r; ! cooked.g = texture2D(colourLoversPalettes, vec2(raw.g, paletteDial)).g; ! cooked.b = texture2D(colourLoversPalettes, vec2(raw.b, paletteDial)).b; ! ! gl_FragColor = vec4(mix(raw.rgb, cooked, strengthDial), raw.a); }
  • 57. EISColourLoversShader.fsh varying! mediump vec2 v_st; // Palette selector uniform float paletteDial; // Contribution of remapped color uniform float strengthDial; // Palettes are ganged together into a single texture. // A specific palette is selected with the paletteDial uniform sampler2D colourLoversPalettes; // hero uniform int heroChannels; uniform sampler2D hero; void main() { ! ! vec4 raw = (heroChannels == 1) ? vec4(texture2D(hero, v_st).a) : texture2D(hero, v_st); ! ! vec3 cooked; ! cooked.r = texture2D(colourLoversPalettes, vec2(raw.r, paletteDial)).r; ! cooked.g = texture2D(colourLoversPalettes, vec2(raw.g, paletteDial)).g; ! cooked.b = texture2D(colourLoversPalettes, vec2(raw.b, paletteDial)).b; ! ! gl_FragColor = vec4(mix(raw.rgb, cooked, strengthDial), raw.a); }
  • 60. Conceptually simple idea. Use one channel of a photo to select the hue of a resultant image.
  • 61. Demo Hue Shift Shader
  • 63. Sampling a texture at a low rate results in quantization. Beyond Photography - The Digital Darkroom by Gerard J. Holzmann
  • 64. Sampling in polar coordinates rather then cartesian is a bit more interesting. Beyond Photography - The Digital Darkroom by Gerard J. Holzmann
  • 69. red channel = drop shadow green channel = cut out final rgb texture
  • 71. Links A previous meetup talk I gave on iOS OpenGL • iOS OpenGL - http://slidesha.re/Y1MW8 Github - code • HelloGLSL - http://bit.ly/JCcMju • EISRenderHelpful. Helpful 3D rendering glue code - http://bit.ly/JZ4HW3 Github - people to follow • Philip Rideout - https://github.com/prideout • Raphael Sebbe - https://github.com/rsebbe • Brad Larson - https://github.com/BradLarson • Jeff LaMarche - https://github.com/jlamarche Elsewhere: • Martins Upitis - http://devlog-martinsh.blogspot.com/ • Ole Begemann - http://bit.ly/srlCBV • Daniel Rakos - http://bit.ly/a3QATn
  • 73. Douglass Turner Elastic Image Software email: douglass.turner@gmail.com tweets: @dugla cell: 781 775 3708 Copyright © Douglass Turner