SlideShare a Scribd company logo

JS Days Mobile Meow

•
•
Greg Schechter
Greg Schechter

This is the story of the glory and struggle of bringing a high quality YouTube experience to the mobile web. Once upon a time there was a web developer who wanted to play videos on the web. So he filmed a cat and wrote a Flash and a HTML5 player. He spent many hours making it work on his favorite desktop browsers and even the one his grandfather still used. People could watch his cat video, he smiled, and the world was good. Then one day someone put a browser in a phone and soon there were many phones with many different browsers. This new set of environments were even harder to develop for and had a slew of new terrifying bugs. The web developer was miserable knowing people couldn’t watch his cat video. With much time and effort he figured out many of the secrets needed to combat the evils of the different mobile platforms. Once again people could watch his cat video, he smiled, and the world was good. The end.

Related slideshows

HTML5 Video Player - HTML5 Dev Conf 2012
HTML5 Video Player - HTML5 Dev Conf 2012HTML5 Video Player - HTML5 Dev Conf 2012
HTML5 Video Player - HTML5 Dev Conf 2012
 
HTML5 Multimedia Accessibility
HTML5 Multimedia AccessibilityHTML5 Multimedia Accessibility
HTML5 Multimedia Accessibility
 
State of Media Accessibility in HTML5
State of Media Accessibility in HTML5State of Media Accessibility in HTML5
State of Media Accessibility in HTML5
 
1 of 97
Download to read offline
Mobile Meow Bring YouTube to the Mobile Web By YouTube's Greg Schechter
Greg Schechter The Web Warrior schechter@google.com
Video
Video
JS Days Mobile Meow
JS Days Mobile Meow
A Brief History
Video
Flash
<video>
Video
Playback Stats
Flash Native <video>
Flash Flash == Flash
Flash Flash
Native
Native ○ For fallbacks, use rtsp:// protocol (serving .3pg) if the device won't support HTML5 ○ Use custom protocol / URL scheme to launch your own Android, iOS native app (no Windows Phone yet) ■ youtube://video_id
<video>
The Different Browsers Safari Android
The Different Browsers Opera Safari Android Chrome Silk Mini Firefox Opera IE
The Different Browsers Opera Safari Android Chrome Silk Mini Firefox Opera IE AMP LE S
By brownpau at http://www.flickr.com/photos/brownpau/533267369/
How do we start <video>
The Source <video src="funny_cat_video">
The Source <video src="funny_cat_video.webm/mp4/ogg">
Mobile Formats Support Chrome Safari Firefox Opera IE Android H.264 WebM Android 4+ Android 4+ Android 4+ HLS Android 3+ Platform Versions and Distribution
The Source <video> <source src="funny_cat_video.mp4"> <source src="funny_cat_video.webm"> </video>
HLS = HTTP Live Streaming <video src="playlist.m3u8">
HLS video-0-1.ts format-0.m3u8 video-0-2.ts ... video-1-1.ts index.m3u8 format-0.m3u8 video-1-2.ts ... video-2-1.ts format-0.m3u8 video-2-2.ts ...
HLS Big Features ○ Live and Adaptive Streaming ○ Built in CDN failover ○ Metadata + Cue Points Apple App Requirements ○ > 10 minutes in duration or 5 mb in 5 minutes ○ at least one 64 kbps or lower stream required Tools and more information on Apple's website: Apple's Using HTTP Live Streaming
Power
Power Consumption: H264 vs WebM **fullscreen flash **windowed flash Research done by Yossi Oren For more information visit http://iss.oy.ne.ro/
Stick it in a page
Basic video tag Safari ○ Renders a thumbnail poster and play button (flaky) ○ Background is set to black and cannot be overwritten ○ If a parent element has display:none set at any point the video fails to playback properly Opera ○ Gives video dimentions black background if the information is available ○ Has issues with stretching WebM IE and Android ○ Play button/film icon for all video tags regardless if browser can play the format ○ Background is set to black/grey and cannot be overwritten
Give it a poster and click it
Poster Attribute Use the poster attribute to get a thumbnail Safari, Chrome, Android, and Opera ○ Preserve aspect ratio of poster image IE and Firefox ○ Stretch image to size of video tag IE ○ Keeps poster as paused state All Others ○ Replace video with the paused frame
Basic video tag Chrome ○ Default click to toggle play/pause ○ Remove with an empty onclick handler Safari ○ Will launch any supported video with or without an onclick handler Android and IE ○ Shows click interactions but does not play with empty onclick
Basic video tag Pro tips: ○ Create your own cued state to convey you can play the video ○ Set the background color to black for continuity across platforms
Controls <video controls>
Controls Chrome Safari Firefox Opera IE Android Play/pause Button Volume Only in Mute toggle Don't work in Controls Fullscreen only Android 4 Seek bar Fullscreen Button Gesture Button button Playback Both Fullscreen Inline only Determined Fullscreen Both Type only by hardware only
Custom Controls ● Preserve your brand ● Unified experience across platforms and browsers
Custom Controls Lets get some custom controls
Custom Controls ● Allows us to expand the set of controls and add our own ○ annotations ○ playlist ○ captions ○ more
Custom Controls Pro Tips ● User expect to be able to drag the progress bar ○ Need to remender to prevent scroll on touchstart ● Volume can't be set everywhere and users are accustomed to using device controls ○ So don't build controls for it
Custom Controls Pro Tips ● Fingers are fat ○ Average finger is 11mm so make targets at least 40px with 10px padding ○ Use SVG's so icons can be scaled and shared with desktop application ○ Do what you can in css
Custom Controls Pro Tips ● Don't trigger content with hover ● Fullscreen ○ The browsing context is always fullscreen so fake it
Fullscreen
requestFullScreen
requestFullScreen ● Desktop Only ○ Firefox, Safari, and Chrome ○ Vendor Prefixed
What about Mobile?
Isn't it already fullscreen?
Yes (sort of)
Mobile Fullscreen ● Open New Tab ● webkitEnterFullscreen ■ Webkit only ■ Video element only ■ metadata must be loaded
Mobile Fullscreen ● Open New Tab ● webkitEnterFullscreen ■ Webkit only ■ Video element only ■ metadata must be loaded ● Pro Tip: Remember size context changes so use viewport to scale icons and controls
@viewport { width: device-width; }
autoplay <video autoplay>
JS Days Mobile Meow
autoplay In Safari on iOS (for all devices, including iPad), where the user may be on a cellular network and be charged per data unit, preload and autoplay are disabled. No data is loaded until the user initiates it.
What about everyone else?
Autoplay Chrome Safari Firefox Opera IE Android Attribute Scripted Buggy
autoplay
autoplay <video onclick="this.play()">
JS Days Mobile Meow
JS Days Mobile Meow
autoplay function someClickEvent(evt) { // In a user initiated thread. myVideoElement.load(); getVideoData(); // Triggers an ajax call. } function onGetVideoDataReturned(data) { // Not in a user initiated thread. setVideoElementSrc(data); myVideoElement.load(); myVideoElement.play(); }
autoplay function someClickEvent(evt) { // In a user initiated thread. myVideoElement.load(); getVideoData(); // Triggers an ajax call. } function onGetVideoDataReturned(data) { // Not in a user initiated thread. setVideoElementSrc(data); myVideoElement.load(); // For Android window.setTimeout(function() { myVideoElement.play(); }, 0); }
Embeds
Embeds <script>
Embeds <script> ○ We need our content to be sandboxed ○ More than just a video tag
Embeds <script> ○ We need our content to be sandboxed ○ More than just a video tag <object>
Embeds <script> ○ We need our content to be sandboxed ○ More than just a video tag <object> ○ Can load content with the data attribute ○ But no way to interact with it via JavaScript
Embeds <iframe>
Embeds <iframe> ○ Allows our content to be sandboxed ○ JavaScript API communication
Embeds <iframe type="text/html" width="640" height="385" frameborder="0" src="http://www.youtube.com/embed/VIDEO_ID" allowfullscreen> </iframe>
Embeds Pro tip: Plan for the future (if you can) <iframe type="text/html" width="640" height="385" frameborder="0" src="http://www.youtube.com/embed/VIDEO_ID" allowfullscreen> </iframe>
Embeds Pro Tips: html { /** Hack to fix iPhone resizing. */ overflow: hidden; }
Embeds Pro Tips: body { /** Dymanic Resizing **/ background-color: #000; height: 100%; width: 100%; /** Remove highlight when use clicks **/ -webkit-tap-highlight-color: rgba(0, 0, 0, 0); }
Testing
Testing: The Old Fashioned Way ● Device lab ○ accurate ○ costly ○ space inefficient ○ boring
Testing: With Software ● Hardware Emulators / Simulators ○ available for major systems ○ approximation varies ○ still boring ● Android Emulator ○ bulky, slow, no video codecs ● iOs Simulator ○ both tablet and phone
Testing: With Software ● Browser Simulators ○ Poorer approximation ● Opera Mobile Simulator ○ missing video tag support ● Fennec (FF) Simulator ○ poor touch control mapping ○ sends desktop user agent
Testing: Automation ● Selenium! ○ use with simulated or real devices ○ not boring
Testing: Automation ● Android Webdriver ○ Still no video support ● iOS Webdriver ○ Need to register as ios dev ○ Intermittent issues with playback ● IE ○ No webdriver APIs for mobile
Testing: Automation ● Chrome, Opera ○ Driver APIs built-in ○ Remote debugging ● FFMobile ○ No webdriver APIs
Testing: Automation Strategy ● Test API methods in mobile context first ○ cheap ○ can approximate video playback by video. currentTime
Testing: Automation Strategy ● More sophisticated tests to follow ○ screenreader ● Screen cap processing ○ check for distortion ○ playback accuracy
Testing: Automation Strategy ● Screen cap processing
Testing: Hardware Assistance for more information check out http://bitbeam.org/
The Future Cats in Space by WF&TD
JS Days Mobile Meow
Or
PointerEvents http://msdn.microsoft.com/en-us/library/ie/hh673557(v=vs.85).aspx
http://imgs.xkcd.com/comics/in_ur_reality.png
Track <video> <track src="cats_meow" kind="subtitles"> </video>
Camera Access with getUserMedia
More to learn ● HTML5 Video Spec ○ http://dev.w3.org/html5/spec-author-view/video.html ● Browser Blogs ○ http://dev.opera.com/ ○ http://hacks.mozilla.org/ ○ http://blogs.msdn.com/b/ie/ ○ http://peter.sh/ ● Documentation ○ https://developer.mozilla.org/en-US/ ● Other ○ http://developer.apple. com/library/safari/#documentation/AudioVideo/Conceptual/Using_HT ML5_Audio_Video/Device-SpecificConsiderations/Device- SpecificConsiderations.html
Questions? can haz question? By cloudzilla http://www.flickr.com/photos/cloudzilla/378829651/ schechter@google.com

More Related Content

JS Days Mobile Meow

  • 1. Mobile Meow Bring YouTube to the Mobile Web By YouTube's Greg Schechter
  • 2. Greg Schechter The Web Warrior schechter@google.com
  • 11. Video
  • 13. Flash Native <video>
  • 14. Flash Flash == Flash
  • 15. Flash Flash
  • 17. Native ○ For fallbacks, use rtsp:// protocol (serving .3pg) if the device won't support HTML5 ○ Use custom protocol / URL scheme to launch your own Android, iOS native app (no Windows Phone yet) ■ youtube://video_id
  • 19. The Different Browsers Safari Android
  • 20. The Different Browsers Opera Safari Android Chrome Silk Mini Firefox Opera IE
  • 21. The Different Browsers Opera Safari Android Chrome Silk Mini Firefox Opera IE AMP LE S
  • 22. By brownpau at http://www.flickr.com/photos/brownpau/533267369/
  • 23. How do we start <video>
  • 24. The Source <video src="funny_cat_video">
  • 26. Mobile Formats Support Chrome Safari Firefox Opera IE Android H.264 WebM Android 4+ Android 4+ Android 4+ HLS Android 3+ Platform Versions and Distribution
  • 27. The Source <video> <source src="funny_cat_video.mp4"> <source src="funny_cat_video.webm"> </video>
  • 28. HLS = HTTP Live Streaming <video src="playlist.m3u8">
  • 29. HLS video-0-1.ts format-0.m3u8 video-0-2.ts ... video-1-1.ts index.m3u8 format-0.m3u8 video-1-2.ts ... video-2-1.ts format-0.m3u8 video-2-2.ts ...
  • 30. HLS Big Features ○ Live and Adaptive Streaming ○ Built in CDN failover ○ Metadata + Cue Points Apple App Requirements ○ > 10 minutes in duration or 5 mb in 5 minutes ○ at least one 64 kbps or lower stream required Tools and more information on Apple's website: Apple's Using HTTP Live Streaming
  • 31. Power
  • 32. Power Consumption: H264 vs WebM **fullscreen flash **windowed flash Research done by Yossi Oren For more information visit http://iss.oy.ne.ro/
  • 33. Stick it in a page
  • 34. Basic video tag Safari ○ Renders a thumbnail poster and play button (flaky) ○ Background is set to black and cannot be overwritten ○ If a parent element has display:none set at any point the video fails to playback properly Opera ○ Gives video dimentions black background if the information is available ○ Has issues with stretching WebM IE and Android ○ Play button/film icon for all video tags regardless if browser can play the format ○ Background is set to black/grey and cannot be overwritten
  • 35. Give it a poster and click it
  • 36. Poster Attribute Use the poster attribute to get a thumbnail Safari, Chrome, Android, and Opera ○ Preserve aspect ratio of poster image IE and Firefox ○ Stretch image to size of video tag IE ○ Keeps poster as paused state All Others ○ Replace video with the paused frame
  • 37. Basic video tag Chrome ○ Default click to toggle play/pause ○ Remove with an empty onclick handler Safari ○ Will launch any supported video with or without an onclick handler Android and IE ○ Shows click interactions but does not play with empty onclick
  • 38. Basic video tag Pro tips: ○ Create your own cued state to convey you can play the video ○ Set the background color to black for continuity across platforms
  • 39. Controls <video controls>
  • 40. Controls Chrome Safari Firefox Opera IE Android Play/pause Button Volume Only in Mute toggle Don't work in Controls Fullscreen only Android 4 Seek bar Fullscreen Button Gesture Button button Playback Both Fullscreen Inline only Determined Fullscreen Both Type only by hardware only
  • 41. Custom Controls ● Preserve your brand ● Unified experience across platforms and browsers
  • 42. Custom Controls Lets get some custom controls
  • 43. Custom Controls ● Allows us to expand the set of controls and add our own ○ annotations ○ playlist ○ captions ○ more
  • 44. Custom Controls Pro Tips ● User expect to be able to drag the progress bar ○ Need to remender to prevent scroll on touchstart ● Volume can't be set everywhere and users are accustomed to using device controls ○ So don't build controls for it
  • 45. Custom Controls Pro Tips ● Fingers are fat ○ Average finger is 11mm so make targets at least 40px with 10px padding ○ Use SVG's so icons can be scaled and shared with desktop application ○ Do what you can in css
  • 46. Custom Controls Pro Tips ● Don't trigger content with hover ● Fullscreen ○ The browsing context is always fullscreen so fake it
  • 49. requestFullScreen ● Desktop Only ○ Firefox, Safari, and Chrome ○ Vendor Prefixed
  • 51. Isn't it already fullscreen?
  • 53. Mobile Fullscreen ● Open New Tab ● webkitEnterFullscreen ■ Webkit only ■ Video element only ■ metadata must be loaded
  • 54. Mobile Fullscreen ● Open New Tab ● webkitEnterFullscreen ■ Webkit only ■ Video element only ■ metadata must be loaded ● Pro Tip: Remember size context changes so use viewport to scale icons and controls
  • 55. @viewport { width: device-width; }
  • 56. autoplay <video autoplay>
  • 58. autoplay In Safari on iOS (for all devices, including iPad), where the user may be on a cellular network and be charged per data unit, preload and autoplay are disabled. No data is loaded until the user initiates it.
  • 60. Autoplay Chrome Safari Firefox Opera IE Android Attribute Scripted Buggy
  • 62. autoplay <video onclick="this.play()">
  • 65. autoplay function someClickEvent(evt) { // In a user initiated thread. myVideoElement.load(); getVideoData(); // Triggers an ajax call. } function onGetVideoDataReturned(data) { // Not in a user initiated thread. setVideoElementSrc(data); myVideoElement.load(); myVideoElement.play(); }
  • 66. autoplay function someClickEvent(evt) { // In a user initiated thread. myVideoElement.load(); getVideoData(); // Triggers an ajax call. } function onGetVideoDataReturned(data) { // Not in a user initiated thread. setVideoElementSrc(data); myVideoElement.load(); // For Android window.setTimeout(function() { myVideoElement.play(); }, 0); }
  • 69. Embeds <script> ○ We need our content to be sandboxed ○ More than just a video tag
  • 70. Embeds <script> ○ We need our content to be sandboxed ○ More than just a video tag <object>
  • 71. Embeds <script> ○ We need our content to be sandboxed ○ More than just a video tag <object> ○ Can load content with the data attribute ○ But no way to interact with it via JavaScript
  • 73. Embeds <iframe> ○ Allows our content to be sandboxed ○ JavaScript API communication
  • 74. Embeds <iframe type="text/html" width="640" height="385" frameborder="0" src="http://www.youtube.com/embed/VIDEO_ID" allowfullscreen> </iframe>
  • 75. Embeds Pro tip: Plan for the future (if you can) <iframe type="text/html" width="640" height="385" frameborder="0" src="http://www.youtube.com/embed/VIDEO_ID" allowfullscreen> </iframe>
  • 76. Embeds Pro Tips: html { /** Hack to fix iPhone resizing. */ overflow: hidden; }
  • 77. Embeds Pro Tips: body { /** Dymanic Resizing **/ background-color: #000; height: 100%; width: 100%; /** Remove highlight when use clicks **/ -webkit-tap-highlight-color: rgba(0, 0, 0, 0); }
  • 79. Testing: The Old Fashioned Way ● Device lab ○ accurate ○ costly ○ space inefficient ○ boring
  • 80. Testing: With Software ● Hardware Emulators / Simulators ○ available for major systems ○ approximation varies ○ still boring ● Android Emulator ○ bulky, slow, no video codecs ● iOs Simulator ○ both tablet and phone
  • 81. Testing: With Software ● Browser Simulators ○ Poorer approximation ● Opera Mobile Simulator ○ missing video tag support ● Fennec (FF) Simulator ○ poor touch control mapping ○ sends desktop user agent
  • 82. Testing: Automation ● Selenium! ○ use with simulated or real devices ○ not boring
  • 83. Testing: Automation ● Android Webdriver ○ Still no video support ● iOS Webdriver ○ Need to register as ios dev ○ Intermittent issues with playback ● IE ○ No webdriver APIs for mobile
  • 84. Testing: Automation ● Chrome, Opera ○ Driver APIs built-in ○ Remote debugging ● FFMobile ○ No webdriver APIs
  • 85. Testing: Automation Strategy ● Test API methods in mobile context first ○ cheap ○ can approximate video playback by video. currentTime
  • 86. Testing: Automation Strategy ● More sophisticated tests to follow ○ screenreader ● Screen cap processing ○ check for distortion ○ playback accuracy
  • 87. Testing: Automation Strategy ● Screen cap processing
  • 88. Testing: Hardware Assistance for more information check out http://bitbeam.org/
  • 89. The Future Cats in Space by WF&TD
  • 91. Or
  • 94. Track <video> <track src="cats_meow" kind="subtitles"> </video>
  • 95. Camera Access with getUserMedia
  • 96. More to learn ● HTML5 Video Spec ○ http://dev.w3.org/html5/spec-author-view/video.html ● Browser Blogs ○ http://dev.opera.com/ ○ http://hacks.mozilla.org/ ○ http://blogs.msdn.com/b/ie/ ○ http://peter.sh/ ● Documentation ○ https://developer.mozilla.org/en-US/ ● Other ○ http://developer.apple. com/library/safari/#documentation/AudioVideo/Conceptual/Using_HT ML5_Audio_Video/Device-SpecificConsiderations/Device- SpecificConsiderations.html
  • 97. Questions? can haz question? By cloudzilla http://www.flickr.com/photos/cloudzilla/378829651/ schechter@google.com